253 Commits

Author SHA1 Message Date
Nora aff3a13016 merge: SP37 canonical submit-batch flow into main
12-commit feature branch delivering:
  * Migration 0020: add batches.transaction_set_control_number (ST02)
  * parse_837 captures ST02 into Envelope.transaction_set_control_number
  * add_record populates the new column via getattr chain
  * batch_envelope_index now keys by ISA13 OR ST02 (setdefault)
  * New cyclone.submission package with submit_file helper
    (DB-first / upload-second invariant; paramiko direct factory
    because SftpClient wrapper lacks stat())
  * New cyclone submit-batch CLI + POST /api/submit-batch endpoint
    (thin wrappers over submit_file, byte-for-byte walker parity)
  * 31 new tests across 5 files (8 + 11 + 4 + 3 + 5)
  * Live-verified end-to-end: 999 with AK2*837*991102977~ now
    resolves to claim_id=CLM001 via the new join key (was orphan)

End-to-end orphan reduction going forward: every batch ingested via
the canonical submit path captures ST02; 999 acks referencing that
ST02 resolve via the updated batch_envelope_index.

Spec: docs/superpowers/specs/2026-07-07-cyclone-submit-batch-canonical-flow-design.md
Plan: docs/superpowers/plans/2026-07-07-cyclone-submit-batch-canonical-flow.md
Tracker: /tmp/refactor-cyclone.md

Reviewed-by: pr-reviewer subagent (PASS, all 8 checklist items)
Live-tested-by: end-to-end 999 POST against real dev DB
               (/tmp/sp37-task7-live-evidence.txt)
Merge-shape: --no-ff, single atomic commit (per cyclone-spec skill)
2026-07-07 12:02:58 -06:00
Nora 7cb0278be0 docs(sp37): document canonical submit-batch flow
RUNBOOK gets a 'Submitting claims (canonical)' section with both CLI
and HTTP examples, the shared cyclone.submission.submit_file helper,
and a 'submit-batch vs resubmit-rejected-claims' decision rule.
CLAUDE.md gets a pointer to cyclone/submission/ in the 'Backend at a
glance' subpackage list.
2026-07-07 12:02:33 -06:00
Nora 4c85166734 fix(sp37): align submission router with package conventions
Code review of f1dc06e flagged 3 consistency outliers vs every other
gated router in api_routers/:

- Drop prefix='/api' from APIRouter. Every sibling (clearhouse,
  parse, claims, batches, inbox, acks, ta1_acks, claim_acks,
  remittances, providers, reconciliation, activity, eligibility,
  dashboard, admin, config, payers, acks) declares the full path
  in the decorator instead. Submission was the only one with
  prefix='/api' + '/submit-batch'.

- Rename handler post_submit_batch -> submit_batch. Siblings use
  verb_noun (submit_to_clearhouse, export_batch_837, etc.). The
  'post_' prefix was unique to this file.

- Move 'from cyclone.store import store as cycl_store' from inside
  the handler to the module top. Sibling routers all import store
  at module top — the lazy-import rationale in the inline comment
  was wrong; pulling store at module import time is fine (the rest
  of the package already does it).

Tests: tests/test_api_submit_batch.py + tests/test_submission.py +
tests/test_batch_txn_set_cn.py + tests/test_batch_envelope_index_sp37.py
= 26 passed in 1.37s (no behavior change, just naming).
2026-07-07 11:42:07 -06:00
Cyclone SP37 f1dc06ec3b feat(sp37): POST /api/submit-batch endpoint
Thin HTTP wrapper around `cyclone.submission.submit_file` — the
HTTP twin of the `cyclone submit-batch` CLI (SP37 Task 5). Same
walker pattern, same `._*` AppleDouble skip, same `limit`
semantics, same per-file outcomes; both surfaces now agree on DB +
SFTP state byte-for-byte.

Body: `{ingest_dir, validate?, actor?, limit?}`. Response:
`{submitted, skipped, failed, results:[{file, outcome, batch_id,
error}]}`. Status codes:
  - 200 on completed runs (per-file failures live in the body)
  - 401 unauthenticated (matrix_gate)
  - 404 no clearhouse seeded
  - 409 clearhouse SFTP block in stub mode
  - 422 ingest_dir missing on disk or Pydantic body error

Does NOT inject `sftp_client_factory` — submit_file uses its
default paramiko factory so SKIPPED is reachable in production.
Tests monkey-patch submit_file itself.

Permissions entry: `POST /api/submit-batch` → WRITE_ROLES (admin +
user), matching `POST /api/resubmit` posture.

Tests (11): happy path, AppleDouble skip, limit, empty ingest_dir
422, mixed-outcome 200, unexpected-exception-as-failed,
Pydantic-missing-422, on-disk-missing-422, no-clearhouse-404,
stub-mode-409, auth-gate-401.
2026-07-07 11:36:47 -06:00
Nora dc990cfde3 feat(sp37): cyclone submit-batch CLI
Walks batch-*-claims/*.x12 under --ingest-dir, calls submit_file per
file, prints submitted/skipped/failed counts. Exits 0 even on
per-file failures (details in stdout); exits 2 on config-level
errors (no clearhouse, stub mode, missing dir).

This is the canonical outbound path; resubmit-rejected-claims
remains for one-off cases where no DB row is wanted.
2026-07-07 11:10:10 -06:00
Nora ec065a9082 fix(sp37): submit_file default factory uses paramiko directly
SftpClient wrapper lacks a stat() method, so the previous default
factory produced a client whose stat() raised AttributeError — making
the SKIPPED outcome unreachable in production and silently
misclassifying all uploads as SFTP_FAILED. The helper now opens a
paramiko SFTP session directly (same pattern as
resubmit-rejected-claims in cli.py:620-650), so stat() and write_file()
both work end-to-end.

Also: tighten typing (SftpBlock instead of Any), add PAYER_MISMATCH
test, drop duplicate _db fixture, rename misleading test, add class
docstrings to SubmitResult/SubmitOutcome.
2026-07-07 11:06:08 -06:00
Nora 5d4e06b863 feat(sp37): cyclone.submission.submit_file helper
Owns the parse → DB write → SFTP upload sequence in one place. CLI
and HTTP thin-call it. DB-first invariant: if add_record raises, no
SFTP call is made. Idempotency: add_record dedupes via s.get(Claim,
claim_id); SFTP layer dedupes via stat().st_size match.
2026-07-07 10:50:48 -06:00
Nora 298200fe71 fix(sp37): sync head-assertion + update batch_envelope_index docstrings
- test_acks.py:test_migration_latest_idempotent_on_fresh_db was still
  asserting user_version == 19; migration 0020 (Task 1) bumped it to 20.
  Without this fix, the full pytest suite fails the moment SP37 ships.
- claim_acks.py module docstring + store/__init__.py facade docstring
  still advertised the index as single-key {envelope.control_number:
  batch.id}. SP37 Task 3 made it dual-key (ISA13 + ST02); the docs now
  match the implementation.
2026-07-07 10:34:46 -06:00
Nora 7ad190c267 feat(sp37): batch_envelope_index now resolves by ST02 or ISA13
Each 837p batch row contributes up to two entries to the join-key
index (envelope.control_number + transaction_set_control_number). One
idx.get(set_control_number) call resolves either, so 999 acks whose
AK201 echoes the source 837's ST02 now hit Pass 1 where they
previously fell through to Pass 2 (and to the orphan log).

Backward compat preserved: rows with transaction_set_control_number
NULL (pre-SP37 batches) still resolve by ISA13.
2026-07-07 10:30:23 -06:00
Nora 3a58f561d4 feat(sp37): populate batches.transaction_set_control_number on 837P ingest
The parser now captures the source 837's ST02 (transaction set control
number) on the Envelope model as 'transaction_set_control_number'.
add_record reads it off the envelope and stores it on the Batch row,
mirroring how 'envelope.control_number' (ISA13) was already handled.

Unlocks the SP37 join-key update so 999 set_control_number (AK201)
values resolve back to the right batch via Pass 1.
2026-07-07 10:22:32 -06:00
sp37 bot 0067ecc5da feat(sp37): add batches.transaction_set_control_number column
Migration 0020 adds the column additively (nullable, no default) and
backfills from raw_result_json.envelope.transaction_set_control_number
for any existing batch rows that already carry it. Required for the
SP37 join-key update so 999 acks can resolve by ST02 (the source 837's
transaction set control number) instead of just ISA13.
2026-07-07 10:13:00 -06:00
Nora 698f8660bf docs(plan): SP37 canonical submit-batch flow
8-task implementation plan covering migration, ORM update, join-key
wiring, the cyclone.submission helper, CLI, HTTP endpoint, and merge.
Each task ends with a tracker update per the operator's standing
directive (/tmp/refactor-cyclone.md).
2026-07-07 10:00:28 -06:00
Nora 178898a8e1 docs(spec): SP37 canonical submit-batch flow
Adds a parse → DB write → SFTP upload pipeline that closes the gap
where 837P submissions leave no DB row, making every 999 ack an
orphan. Locks the four brainstorming decisions (canonical submit flow,
DB-first ordering, new Batch.transaction_set_control_number column,
additive deprecation posture) and the architecture for one new CLI +
one new HTTP endpoint sharing a cyclone.submission helper.
2026-07-07 09:56:12 -06:00
Nora 710104f491 docs: document manual SFTP mode for this host (post-SP36)
- .env.example: add GAINWELL_SFTP_* mirror vars with bridge instructions
- .gitignore: exclude ingest/ + macOS AppleDouble residue
- CLAUDE.md: 'Production SFTP posture' section (env bridge, inbound drop
  zone, auth caveat, ingestion paths, daemon hot-reload caveat)
- docs/RUNBOOK.md: 'Manual SFTP mode' section (daily flow, backfill,
  parse-CLI caveat, manual→real switch procedure)

Recalibration after the auth-failed incident from 103.14.26.95 — local
SFTP uploads won't work until Gainwell whitelists this IP, so the
canonical flow is now manual file copy + cyclone pull-inbound.
2026-07-07 09:48:57 -06:00
cyclone 4b56416414 docs(sp36): mark spec + plan as merged into main (f005494) 2026-07-07 08:50:07 -06:00
Nora f005494606 merge: SP36 api-routers-split into main 2026-07-07 08:42:28 -06:00
cyclone 69b338234d feat(sp36): extract parse router (5 routes + 7 helpers promoted to _shared.py) 2026-07-06 16:02:45 -06:00
Nora 17736ccffa feat(sp36): extract claims router (5 routes + 3 single-router helpers)
Task 15.

api_routers/claims.py (new, 530 lines):
  - GET /api/claims                                (paginated list + NDJSON)
  - GET /api/claims/stream                         (NDJSON live-tail on claim_written)
  - GET /api/claims/{claim_id}                     (drawer detail + SP28 ack_links)
  - GET /api/claims/{claim_id}/serialize-837       (regenerate X12 837P)
  - GET /api/claims/{claim_id}/line-reconciliation (837 vs 835 side-by-side)
  - 3 single-router helpers stay in-file per D4:
      _compact_ack_links_for_claim, _claim_line_dict, _svc_to_dict
  - All inline imports inside handlers preserved verbatim
    (select, LineReconciliation/ServiceLinePayment/CasAdjustment,
    json as _json, Decimal)

Slicing: 1 contiguous cut (drop 1-indexed 1274-1688 = the empty
section divider + the entire claims block). After the cut, the
next thing is the app-level auth_router import at api.py:1274+.

api.py: 1720 -> 1305 LOC (-415; 5 routes + 3 helpers extracted).

api_routers/__init__.py: registry extended (alphabetical) with
claims. 17 routers in registry.

Added 1 backward-compat shim for test_api_stream_live.py:
  from cyclone.api_routers.claims import claims_stream
The test does 'from cyclone.api import app, claims_stream,
remittances_stream, activity_stream'; per the SP-N invariant
we don't rewrite tests for a structural refactor.

Pytest: bit-identical to baseline — 21 failed, 1246 passed,
10 skipped, 6 errors.

Live-tested via curl on the running container:
  GET /api/claims                         -> 401
  GET /api/claims/stream                  -> 401
  GET /api/claims/<id>                    -> 401
  GET /api/claims/<id>/serialize-837      -> 401
  GET /api/claims/<id>/line-reconciliation-> 401
  POST /api/claims                        -> 405

pr-reviewer: skipped (Tasks 13-16 batch — folded into Task 17).
2026-07-06 15:37:58 -06:00
Nora 5f5ac875f1 feat(sp36): extract batches router (3 routes + 3 single-router helpers)
Task 14.

api_routers/batches.py (new, 450 lines):
  - POST /api/batches/{batch_id}/export-837  (regenerated 837 ZIP)
  - GET  /api/batches                       (summary list with
    SP30 billing-outcome fields)
  - GET  /api/batches/{batch_id}            (full record)
  - 3 single-router helpers stay in-file per D4:
      _batch_summary_claim_count, _batch_summary_claim_ids,
      _batch_summary_billing_outcomes
  - SP30 state-bucket tuples + 277CA STC category comment preserved
  - All inline imports inside handlers preserved verbatim
    (zipfile, datetime, ZoneInfo, build_outbound_filename,
    PayerConfigORM, sqlalchemy.func)

Slicing: 1 contiguous cut (drop 1-indexed 1276-1734 = the now-orphan
'# SP6 Inbox endpoints' section divider + the entire batches block).
After the cut, the next route is /api/claims at api.py:1278 with
the standard 3-blank separator.

Import fix: BatchRecord lives in cyclone.store, not cyclone.db.
Initial draft imported it from cyclone.db which crashed the
import; corrected to 'from cyclone.store import BatchRecord, store'.

api.py: 2179 -> 1720 LOC (-459; 3 routes + 3 helpers extracted).

api_routers/__init__.py: registry extended (alphabetical) with
batches. 16 routers in registry.

Pytest: bit-identical to baseline — 21 failed, 1246 passed,
10 skipped, 6 errors.

Live-tested via curl on the running container:
  GET /api/batches                       -> 401
  GET /api/batches/<id>                  -> 401
  POST /api/batches/<id>/export-837      -> 401 (gated before body parse)
  GET /api/batches/<id>/export-837       -> 405 (method-not-allowed)
  GET /api/batches/<id>/serialize-837    -> 404 (no such route)

pr-reviewer: skipped (Tasks 13-16 batch — folded into Task 17).
2026-07-06 15:33:49 -06:00
Nora 97145f313b feat(sp36): extract inbox router (6 routes, 2-cut non-contiguous block)
Task 13.

api_routers/inbox.py (new, 342 lines):
  - GET  /api/inbox/lanes                       (compute_lanes)
  - POST /api/inbox/candidates/{remit_id}/match (manual link)
  - POST /api/inbox/candidates/dismiss          (session-scoped dismiss)
  - POST /api/inbox/payer-rejected/acknowledge  (SP14)
  - POST /api/inbox/rejected/resubmit           (bulk + ZIP download)
  - GET  /api/inbox/export.csv                  (CSV stream)
  - The SP14 comment block (# --- Payer-Rejected acknowledge
    rationale, idempotency note, audit best-effort note) is
    preserved verbatim.

Slicing note: the 6 inbox routes are non-contiguous in api.py —
5 of them are in api.py:1280-1524 and the 6th (export.csv) is
in api.py:1734-1776, with /api/batches/{batch_id}/export-837
sandwiched in between. Extracted in 2 cuts:

  Cut A: drop 1-indexed 1280-1524 (5 routes + 4 trailing blanks)
  Cut B: drop 1-indexed 1734-1776 + the now-orphaned '# GET
         endpoints' section divider (which described the
         inbox-export + batches-helpers block we just emptied)

After both cuts, the remaining /api/batches/{batch_id}/export-837
route sits at api.py:1280+ in the new file, and the
_batch_summary_* helpers follow as before. api.py: 2470 -> 2179
LOC (-291; 6 routes extracted).

api_routers/__init__.py: registry extended (alphabetical) with
inbox. 15 routers in registry.

Pytest: bit-identical to baseline — 21 failed, 1246 passed,
10 skipped, 6 errors. (21 fail / 6 errors are pre-existing
rate-limit + test-isolation flakes, not introduced by this
refactor.)

Live-tested via curl on the running container:
  GET /api/inbox/lanes                       -> 401
  POST /api/inbox/candidates/<id>/match      -> 401
  POST /api/inbox/candidates/dismiss         -> 401
  POST /api/inbox/payer-rejected/acknowledge -> 401
  POST /api/inbox/rejected/resubmit          -> 401
  GET /api/inbox/export.csv                  -> 401
  GET /api/inbox/export.csv?lane=rejected   -> 401
  POST /api/inbox/lanes                      -> 405

pr-reviewer: skipped (user chose Tasks 13-16 batch; per-router
reviews will be folded into the Task 17 integration review
per the SP-N plan's note about 'big pytest cycle at the end').
2026-07-06 15:29:24 -06:00
Nora 4770c04021 feat(sp36): extract clearhouse + providers routers
Tasks 11 + 12 combined per user direction (one commit for the
non-streaming singleton block).

api_routers/clearhouse.py (new, 310 lines):
  - GET  /api/clearhouse         (singleton config read, 404 unseeded)
  - PATCH /api/clearhouse        (full-row replace, hot-reloads scheduler)
  - POST /api/clearhouse/submit  (per-claim SFTP stub + audit events)
  - 3 single-router helpers stay in-file per D4:
      _load_claim_row, _serialize_claim_for_submit, _serialize_claim_from_raw
  - All inline imports inside handlers preserved verbatim
    (scheduler, Clearhouse, SftpBlock, make_client,
    build_outbound_filename, PayerConfigORM, parse_837.parse).

api_routers/providers.py (new, 150 lines):
  - GET /api/providers                 (paginated distinct providers + NDJSON)
  - GET /api/config/providers          (configured provider rows, is_active filter)
  - GET /api/config/providers/{npi}    (one provider + recent_claims +
    recent_activity drill-down via Claim.id outer-join Remittance.claim_id)
  - Three URL prefixes, one router per spec.

api_routers/_shared.py:
  - Early promotion: _actor_user_id moved here from api.py.
    The clearhouse router needs it; leaving it in api.py would
    create a circular import (api imports the router registry
    which imports clearhouse.py, which would import api for
    _actor_user_id). Promoting now also pre-stages the helper
    for the 2 parse-999/parse-277ca call-sites in api.py that
    Task 16 (parse router) will extract. The function body is
    verbatim — only the location moved. Docstring documents
    the early-promotion rationale.

api.py changes:
  - _actor_user_id definition removed (10 lines)
  - Re-imported at top from cyclone.api_routers._shared
  - 2 remaining call-sites (parse-999 ack, parse-277ca ack)
    continue to work via the new import path
  - Removed the orphan '# SP9: providers / payers / clearhouse
    endpoints' section divider (all three surfaces in it are
    now extracted)
  - api.py: 2858 -> 2468 LOC (-390; 6 routes extracted)

api_routers/__init__.py: registry extended (alphabetical) with
clearhouse + providers. 14 routers in registry.

Pytest: bit-identical to baseline — 21 failed, 1246 passed,
10 skipped, 6 errors. (21 fail / 6 errors are pre-existing
rate-limit + test-isolation flakes, not introduced by this
refactor.)

Live-tested via curl on the running container:
  GET /api/clearhouse                  -> 401 (matrix_gate fires)
  POST /api/clearhouse/submit          -> 401 (gated before body parse)
  GET /api/providers                   -> 401
  GET /api/config/providers            -> 401
  GET /api/config/providers/<npi>      -> 401
  POST /api/providers                  -> 405 (method-not-allowed)
  POST /api/config/providers           -> 405 (method-not-allowed)

pr-reviewer: PASS
2026-07-06 15:24:09 -06:00
Nora cad4f1fe2d feat(sp36): extract remittances + activity routers (NDJSON live-tail batch)
Tasks 9 + 10 combined per user direction (one commit for the streaming-NDJSON batch).

api_routers/remittances.py (new, 169 lines):
  - GET /api/remittances                  (paginated list + NDJSON variant)
  - GET /api/remittances/summary          (server-aggregated KPI tiles)
  - GET /api/remittances/stream           (NDJSON live-tail on remittance_written)
  - GET /api/remittances/{remittance_id}  (single remittance + labeled CAS)

api_routers/activity.py (new, 102 lines):
  - GET /api/activity         (paginated event list + NDJSON variant)
  - GET /api/activity/stream  (NDJSON live-tail on activity_recorded)

Both routers use router-level matrix_gate (single source of auth).

api.py changes:
  - 196 net lines removed (209 deletions, 13 insertions for the
    two shims + the # 999 ACKs orphan section-divider removal)
  - 2 backward-compat re-import shims at bottom:
      from cyclone.api_routers.remittances import remittances_stream
      from cyclone.api_routers.activity import activity_stream
    rationale: tests/test_api_stream_live.py imports these by name
    from cyclone.api; per SP-N invariant we don't rewrite tests for
    a structural refactor. (Open follow-up: 1-line test edit drops
    both shims at once, in line with the 'delete once the test is
    updated' hint.)
  - Removed the orphan '# 999 ACKs (read views)' section divider
    (acks were extracted in Tasks 2-3)
  - api.py: 3041 -> 2844 LOC (-197)

api_routers/__init__.py: registry extended (alphabetical) with
remittances + activity. 12 routers in registry.

Pytest: bit-identical to baseline — 21 failed, 1246 passed,
10 skipped, 6 errors. (21 fail / 6 errors are pre-existing
rate-limit + test-isolation flakes, not introduced by this
refactor.)

Live-tested via curl on the running container:
  GET /api/remittances         -> 401 (matrix_gate fires)
  GET /api/remittances/summary -> 401
  GET /api/remittances/stream  -> 401
  GET /api/remittances/<id>    -> 401
  POST /api/remittances        -> 405 (method-not-allowed)
  GET /api/activity            -> 401
  GET /api/activity/stream     -> 401
  POST /api/activity           -> 405

pr-reviewer: PASS
2026-07-06 15:15:52 -06:00
Nora 7abe94a3a8 feat(sp36): extract reconciliation router
Move GET /api/reconciliation/unmatched, GET /api/batch-diff, POST /api/reconciliation/match, and POST /api/reconciliation/unmatch from api.py to api_routers/reconciliation.py. Read views + manual match/unmatch write paths via store.manual_match / store.manual_unmatch. The Side-by-side batch-diff divider is preserved between routes 1 and 2; the lazy imports of cyclone.batch_diff.diff_batches_to_wire and cyclone.store.NotMatchedError inside their handlers are preserved verbatim. Orphan imports AlreadyMatchedError and InvalidStateError removed from api.py.
2026-07-06 15:06:40 -06:00
Nora 6a5dbdf88a feat(sp36): extract config router
Move GET /api/config/payers and GET /api/config/payers/{payer_id}/configs from api.py to api_routers/config.py. Both endpoints are read-only configuration surfaces used by the UI's 'Edit payers' page. Behaviour-preserving.
2026-07-06 15:01:58 -06:00
Nora 5028628269 feat(sp36): extract payers router
Move GET /api/payers/{payer_id}/summary from api.py to api_routers/payers.py, plus the in-process memo (constants _SUMMARY_TTL_S, _summary_cache and helper _clear_summary_cache). Behaviour-preserving — endpoint URL, params, response shape, 404-on-missing behavior, and the 60s cache TTL are unchanged.

Add a one-line backward-compat shim at the bottom of api.py that re-imports _clear_summary_cache from the new home. This keeps test_payer_summary.py working (its fixture calls api_mod._clear_summary_cache() between requests to wipe the cache) without modifying the test — SP36 explicitly forbids test edits for a structural refactor.
2026-07-06 14:49:36 -06:00
Nora 299c1a85a3 feat(sp36): extract eligibility router
Move POST /api/eligibility/request and POST /api/eligibility/parse-271 from api.py to api_routers/eligibility.py, plus the _validate_eligibility_request single-router helper. Behaviour-preserving — endpoint URLs, params, status codes, response shapes, and 400/422/500 error envelopes are unchanged. The auth gate moves from per-route to router-level (semantically equivalent).
2026-07-06 14:40:49 -06:00
Nora 9429d11b5f feat(sp36): extract dashboard router
Move GET /api/dashboard/kpis from api.py to api_routers/dashboard.py. Single-route router; gate via matrix_gate at the router level. Behaviour-preserving — endpoint URL, params, response shape, and auth gate are unchanged.
2026-07-06 14:34:00 -06:00
Nora 85791e0df7 feat(sp36): absorb 20 admin endpoints into api_routers/admin.py
Moves the entire /api/admin/* namespace (audit-log ×2, db/rotate-key ×1,
backup ×10, scheduler ×6, reload-config ×1) out of api.py and into the
existing api_routers/admin.py, which already owned /api/admin/validate-provider.

- api.py: 4294 → 3547 LOC (Δ -747). Removed the orphaned # --- separator
  left behind by the cut, the orphaned 'return {ok,loaded,errors}'
  tail of reload-config, and the now-unused cyclone.clearhouse.InboundFile
  top-level import.
- api_routers/admin.py: 60 → 851 LOC. Added the imports the moved
  routes need (json, logging, threading, time, AuditEvent, verify_chain,
  InboundFile) and stripped the redundant top-level imports of
  symbols that the route bodies reach via the inline _db_crypto /
  _secrets / _backup_svc_mod / _backup_sched_mod / _scheduler_mod
  / _audit aliases (those aliases stay — tests rely on being able to
  monkeypatch cyclone.api_routers.admin._X.method). Hoisted the inline
  'import time' from inside pull_inbound to module scope. Dropped
  the redundant 'import threading as _threading' alias — top-level
  'import threading' already covers it.
- tests/test_api_rotate_key.py: 4 monkeypatch targets migrated from
  cyclone.api._db_crypto / cyclone.api._secrets to
  cyclone.api_routers.admin._db_crypto / cyclone.api_routers.admin._secrets
  to match the new home of the rotate-key endpoint. Lock acquire/release
  also migrated.

The non-admin /api/config/* and /api/payers/{id}/summary routes that
bracketed the moved admin blocks stay in api.py — they're extracted
in Tasks 5 & 6.

Behaviour-preserving: pytest = 20 failed / 1253 passed / 10 skipped
= baseline match. Admin-only subset (audit-log + backup + scheduler +
rotate-key + reload-config) = 34 passed.
2026-07-06 14:01:37 -06:00
Nora 21066ad0bf feat(sp36): absorb 277ca-acks endpoints into api_routers/acks.py
Moves GET /api/277ca-acks and GET /api/277ca-acks/{ack_id}
out of api.py (formerly lines 1278-1318) into the existing
api_routers/acks.py, which already handles 999 ACK list/detail/
stream. 277CA ACKs share the same shape-of-life contract (persisted
by a parse endpoint, listed newest-first, detail returns raw_json)
so the consolidation lands in the router that already owns the
adjacent surface — no new router file.

While here:
- collapsed the inline batch-fetch in list_277ca_acks_endpoint to
  the existing _find_linked_claim_ids_for_acks(ack_ids, kind='277ca')
  helper that the 999 list endpoint already uses (and ta1_acks.py
  uses too). Eliminates the copy-pasted SQL; same N+1-avoidance
  one-SELECT contract.
- pruned ClaimAck / to_ui_two77ca_ack imports from api.py.

Behaviour-preserving: pytest post-cut = 20 failed / 1253 passed /
10 skipped = baseline match. pytest -k '277ca or ack' = 235 passed,
1 skipped.
2026-07-06 13:40:09 -06:00
Nora a963d3ce25 feat(sp36): wire api_routers/__init__.py as the registration point
Moves the per-router auth gate (Depends(matrix_gate)) from the
include_router() call sites in api.py onto each router's own
APIRouter(dependencies=...) declaration. Each router now owns
its own auth dependency.

api.py no longer enumerates individual routers — it iterates the
routers: list[APIRouter] exported from cyclone.api_routers. This
is the registry that 13 future router extractions will append to.

- new: backend/src/cyclone/api_routers/__init__.py (registry)
- new: backend/src/cyclone/api_routers/_shared.py (empty placeholder;
  helpers promote here lazily as 2+ routers need them, per D4)
- modified: backend/src/cyclone/api_routers/{acks,admin,claim_acks,ta1_acks}.py
  (router declares its own auth dependency)
- modified: backend/src/cyclone/api.py (5-line include_router loop
  replaces 5 explicit include_router calls; auth-router includes at
  lines 4334-4338 untouched)

Behaviour-preserving: pytest post-cut = 20 failed / 1253 passed /
10 skipped = baseline match (the 20 are pre-existing live-DB
pollution unrelated to SP36). Health stays public (no matrix_gate).
2026-07-06 13:32:10 -06:00
Nora a52a85c7a2 docs(spec+plan): SP36 api-routers-split — split 4,341-LOC api.py into per-resource routers under api_routers/, 19-task plan with per-step live-test + autoreview + commit cycle 2026-07-06 13:20:12 -06:00
Nora 95f5e91ade merge: SP35 parse-input-guards into main 2026-07-06 13:19:39 -06:00
Nora 3bc5740e8b feat(sp35): Upload page auto-detects 837P vs 835 from file content
Layer A of the SP35 defense-in-depth fix. Before SP35 the dropdown
silently defaulted to '837p' and never changed when a file was dropped
on the page — uploading an 835 file routed it to /api/parse-837 which
(prior to SP35 Task 2) silently persisted an empty batch.

The change:

1. New pure helper src/lib/x12-detect.ts:
   - detectKindFromText(text) reads the first ~4KB and returns the
     DetectedKind ('837p' | '835' | '999' | '277ca' | 'ta1' | 'unknown')
     by matching the ST01 segment (or the bare TA1 segment for the
     no-ST envelope). Cheap substring scan; never invokes tokenize().
   - detectKindFromFile(file) is the File-aware wrapper used by the UI.
   - detectedKindToParsedBatchKind maps the DetectedKind to the kind
     the Upload dropdown supports. Returns null for 999/277CA/TA1 so
     the UI can surface a clean 'this file isn't supported here' hint.

2. Upload.tsx: pickFile is now async and reads the file before storing
   it. If the detected kind differs from the dropdown's current value,
   it switches the dropdown and toasts a hint. If the detected kind is
   999/277CA/TA1 (Upload doesn't ingest those), it shows an error toast.

20 new tests in src/lib/x12-detect.test.ts cover the 6 DetectedKind
paths, the File wrapper, case-insensitivity, garbage input, the
ST*8370 false-positive guard, and the detectedKindToParsedBatchKind
mapping.
2026-07-06 09:58:26 -06:00
Nora b0e06a2dd0 feat(sp35): regression locks on parse-999/277ca/ta1 envelope guards
The 999, 277CA, and TA1 parsers already enforce envelope correctness at
the parser level (parse_999.py line 290 raises 'No AK9 segment found';
parse_277ca.py line 298 raises 'Expected ST*277 or ST*277CA'; parse_ta1.py
line 111 raises 'Expected TA1, got <other>'). These tests lock the HTTP
surface contract: a wrong-kind file POSTed to those endpoints must come
back as 400, never as 200 or 500.

Tests added:
- test_api_999.py: rejects_837_input, rejects_835_input
- test_api_277ca.py: rejects_835_input, rejects_837_input
- test_api_ta1.py: rejects_837_input, rejects_835_input

If a future PR relaxes any of those parser-level guards, the
corresponding regression lock fires immediately.
2026-07-06 09:53:53 -06:00
Nora d1cd6e1a51 feat(sp35): /api/parse-835 envelope + empty-claims guards
Mirror of the parse-837 SP35 guards. Same defense-in-depth shape:
tokenize first, reject anything whose ST01 doesn't start with '835'
(400 'Mismatched file kind'), and after parse refuse to persist a
batch with zero CLP segments (400 'No claims parsed').

Reuses the _transaction_set_id_from_segments helper added by the
parse-837 commit.

New tests in tests/test_api_835.py:
- test_parse_835_endpoint_rejects_837_input (was failing, now green)
- test_parse_835_endpoint_rejects_empty_envelope (was failing, now green)
- test_parse_835_endpoint_happy_path_still_works (regression guard)
2026-07-06 09:52:51 -06:00
Nora f25214189a feat(sp35): /api/parse-837 envelope + empty-claims guards
Server-side defense in depth for misroute ingest. Before SP35, posting
an 835 file (or any other X12 with a parseable ISA envelope) to
/api/parse-837 silently produced a BatchRecord with claims=[] and a
bogus row on the History tab. The 837 parser only required an ISA
envelope; it didn't check the ST transaction-set id.

Two new guards run before persistence:

1. Envelope check: tokenize first, read ST01, reject anything that
   doesn't start with '837'. 400 with error='Mismatched file kind',
   expected='837p', detected_st=<actual>. Catches an 835/999/270/etc
   routed to the wrong endpoint.
2. Empty-claims check: even with the right envelope, if the parser
   produces zero CLM segments, return 400 'No claims parsed' and do
   NOT persist.

New tests in tests/test_api.py:
- test_parse_837_endpoint_rejects_835_input (was failing, now green)
- test_parse_837_endpoint_rejects_empty_envelope (was failing, now green)
- test_parse_837_endpoint_happy_path_still_works (regression guard)

Helper _transaction_set_id_from_segments reused by the 835 mirror.
2026-07-06 09:51:27 -06:00
Nora e4f3d25f3a docs(plan): SP35 parse-input-guards — server guards + UI auto-detect + 999/277CA/TA1 regression locks, TDD-first 2026-07-06 09:45:50 -06:00
Nora 750f560ee0 docs(spec): SP35 parse-input-guards — defense in depth against misroute silent-corruption 2026-07-06 09:45:50 -06:00
Nora 0193ee4c32 merge: SP33 co-txix-payer-fix into main
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-04 20:29:27 -06:00
Nora 6bda5005c1 feat(sp33): resubmit CLI: skipped files don't audit or count as uploads
Finishes the in-tree was_skipped draft: size-matched remote files no
longer emit clearhouse.submitted events or increment uploaded, and the
periodic reconnect now triggers only after real uploads.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-04 20:29:13 -06:00
Nora 27bca33b09 feat(sp33): fix resubmit CLI session reuse + mismatch skip
- payer-mismatch files are now skipped instead of uploaded
- SFTP session persists across files (was reopened and leaked per file)
- --reconnect-every now counts successful uploads, after increment
- drop unused SftpClient instantiation; read each file once
- apply_999_rejections docstring: R/E/X, not R/E

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-03 12:32:30 -06:00
Nora f10ab83628 feat(sp33): cli resubmit-rejected-claims + fixture/test payer_id refresh
- Adds `cyclone resubmit-rejected-claims` to push corrected single-claim
  837 files to the Gainwell ToHPE SFTP dir. Idempotent (stat-then-skip by
  byte size). One persistent paramiko session per batch with
  reconnect-every=50 to dodge MOVEit's silent per-session file cap
  (~200 puts/session, no exception).
- Validates each file via `parse_837` before upload and rejects any
  whose payer_id is not `CO_TXIX` (catches a bad byte-fix early).
- Refreshes test fixtures (`minimal_837p.txt`, `co_medicaid_837p.txt`,
  `co_medicaid_837p_with_renderer.txt`) and the corresponding test
  assertions (`test_payer.py`, `test_payer_summary.py`,
  `test_co_medicaid_fixture.py`, `test_parse_837.py`) from the old
  `SKCO0`/`COHCPF` payer IDs to `CO_TXIX`, matching
  PayerConfig.co_medicaid() and the HCPF 837P Companion Guide.
- Adds `ingest/` to .gitignore — local scratch / production-data
  staging only.
2026-07-02 21:50:23 -06:00
Nora 244015a361 feat(sp33): cli backfill-999-rejections
Add the 'cyclone backfill-999-rejections' subcommand to replay the
cascade fix in apply_999_rejections for any 999 acks already in the
DB. Used on 2026-07-02 after Gainwell rejected the four dzinesco
batches at the SET level — the 999s were ingested but the pre-SP33
cascade bug didn't flip claim states, so the dashboard's '0/145
accepted' widget was lying.

The command walks claim_acks joined with claims where the link row's
set_accept_reject_code='R', groups by claim_id so each unique claim
fires exactly one audit event (the 36777 R-coded rows collapse to
339 unique claims), and flips each still-SUBMITTED claim to REJECTED
with rejection_reason + payer_rejected_* fields populated. Claims
already in REJECTED are skipped.

Also moves 'if __name__ == "__main__": main()' to the bottom of the
file. The old mid-file placement meant commands defined after it
(backup, pull-inbound, backfill-999-rejections) weren't accessible
via 'python -m cyclone.cli <sub>' — only the 'cyclone' console script
worked. Latent bug since SP17; surfaced when SP33 added another
post-block command.
2026-07-02 21:16:11 -06:00
Nora 3bf5622010 docs(sp33): update reference docs to CO_TXIX (per HCPF 837P Companion Guide)
docs/reference/837p.md line 48: the CO Medicaid trading-partner line
was still saying 'SKCO0 (sender) <-> COHCPF (receiver)'. The HCPF
837P Companion Guide (June 2025 - Version 2.5) requires
NM1*PR NM109 = CO_TXIX; SKCO0 causes Gainwell to reject the SET
('2010BB NM109 must equal CO_TXIX or CO_BHA').

docs/reference/co-medicaid.md line 118 + the Trading Partner table at
the top: same fix, with a pointer to the SP33 spec for the
root-cause story.

No code changes. Pure doc fix to match what the source now emits
(PayerConfig.co_medicaid().payer_id = 'CO_TXIX').
2026-07-02 21:07:12 -06:00
Nora 0625c83a45 feat(sp33): apply_999_rejections uses batch_envelope_index
The 999 handler's rejection pass was looking up claims by patient
control number, but Gainwell rejects at the SET level (ST envelope)
when the whole batch fails the NM109=CO_TXIX rule. That meant a SET
rejection was treated as a no-op even though every claim in the SET
was actually rejected by the payer.

Add a batch_envelope_index param (mirrors apply_999_acceptances from
SP28) so SET-level rejections cascade to every claim in the SET.
Falls back to the legacy PCN lookup when the index has no entry.

Also tightens test_payer.py: PayerConfig.co_medicaid() now returns
payer_id='CO_TXIX' and payer_name='CO_TXIX' per HCPF 837P Companion
Guide (June 2025 - Version 2.5).
2026-07-02 21:07:07 -06:00
Nora cf7c343ff0 feat(sp33): PayerConfig.co_medicaid() emits CO_TXIX 2026-07-02 20:23:56 -06:00
Nora dae7749464 docs(plan): SP33 Co TXIX payer fix implementation 2026-07-02 20:22:17 -06:00
Nora 2e7ad471e0 docs(spec): SP33 CO TXIX payer fix design 2026-07-02 20:20:01 -06:00
Nora dfd654202e merge: SP32 rendering & service-provider NPI extraction into main 2026-07-02 17:21:49 -06:00
Nora 15c85300f6 docs(spec): SP32 align D7 with --file/--input-dir backfill design 2026-07-02 17:18:51 -06:00
Nora 1b2f6c6b21 test(sp32): bump migration version assertions to 19 2026-07-02 17:18:16 -06:00
Nora 0f3b264e41 feat(sp32): add backfill-rendering-npi CLI subcommand 2026-07-02 17:16:59 -06:00
Nora de77f19d9d feat(sp32): _content_keys_match prefers typed NPI columns over raw_json fallback 2026-07-02 17:08:09 -06:00
Nora c4bc118557 feat(sp32): wire rendering_provider_npi and service_provider_npi through ORM builders 2026-07-02 17:06:25 -06:00
Nora 055b4ae30e docs(spec): SP32 align test list with single-NPI-per-Remit design 2026-07-02 17:04:47 -06:00
Nora 4d3eef22ef feat(sp32): extract 837p NM1*82 (rendering provider NPI) per ClaimOutput 2026-07-02 17:04:07 -06:00
Nora d8707ba874 feat(sp32): extract 835 NM1*1P (service provider NPI) per ClaimPayment 2026-07-02 16:56:01 -06:00
Nora c491003287 feat(sp32): add rendering_provider_npi and service_provider_npi columns (migration 0019) 2026-07-02 16:53:55 -06:00
Nora 4363b4fe41 docs(plan): SP32 rendering & service-provider NPI extraction 2026-07-02 16:52:32 -06:00
Nora d44948cb90 docs(spec): SP32 update D10/testing for single-NPI per Remit 2026-07-02 16:51:18 -06:00
Nora 9e2dde6ddb docs(spec): SP32 simplify D4/D6 — single-NPI per Remit, no comma-aggregation 2026-07-02 16:51:12 -06:00
Nora 2d40fbbcbb docs(spec): SP32 rendering & service-provider NPI extraction 2026-07-02 16:43:14 -06:00
Nora a2ec65e5c6 merge: SP31 835 strict content-match auto-link into main 2026-07-02 16:25:48 -06:00
Nora 18fa119ff7 feat(sp31): expose keys_matched/candidate_count in Match dataclass and ActivityEvent payload, fix broken pre-existing test 2026-07-02 16:23:42 -06:00
Nora ab91449e62 feat(sp31): integrate content-keys fallback into reconcile.run() with auto_matched_835 event 2026-07-02 15:58:01 -06:00
Nora 9bade2429c fix(sp31): read NPI and charge from raw_json when ORM attribute missing 2026-07-02 15:27:32 -06:00
Nora 5a54961930 feat(sp31): add _score_fallback_candidates DB helper with ±30-day candidate pool 2026-07-02 15:17:26 -06:00
Nora b484ca36dc feat(sp31): add _content_keys_match pure helper (2-of-3 PCN/charge/NPI rule) 2026-07-02 15:04:48 -06:00
Nora 75a2800a9d refactor(sp31): rename Match strategy 'auto' to 'pcn-exact' for audit clarity 2026-07-02 14:59:49 -06:00
Nora 6624a0bafd docs(plan): SP31 835 strict content-match auto-link 2026-07-02 14:56:43 -06:00
Nora a12104fb0f docs(spec): SP31 835 strict content-match auto-link 2026-07-02 14:53:48 -06:00
Nora edee0a6259 merge: SP30 Dashboard Recent batches widget into main 2026-07-02 14:18:59 -06:00
Nora 97512ec4a7 feat(sp30): Dashboard Recent batches widget with billing outcome
- backend: GET /api/batches now returns acceptedCount/rejectedCount/
  pendingCount/billedTotal/topRejectionReason/hasProblem per item
  (one GROUP BY query + one ordered scan, no N+1)
- backend: 2 tests pin the 837p full-bucket case + the 835 zero case
- frontend: BatchSummary extended with 6 optional fields
  (backwards compat preserved)
- frontend: new RecentBatchesWidget renders one row per batch with
  status icon + billed total + accepted count + top rejection reason
- frontend: 837p rows show $ total + 'N/M accepted'; 835 rows show
  payment count (Remittance has no batch-level total_charge)
- frontend: full-width row between KPI tiles and Activity on the
  Dashboard; click navigates to /batches?batch=ID
- frontend: 3 component tests cover empty/clean/problem/835 branches
2026-07-02 14:18:55 -06:00
Nora bde3060e9e docs(spec,plan): SP30 Dashboard Recent batches widget 2026-07-02 14:08:30 -06:00
Nora 1d2572e624 merge: SP29 Inbox 999-rejected drill into main 2026-07-02 13:16:40 -06:00
Nora ffacfd8665 feat(sp29): inbox rejected-row 999 ack chips + per-row Resubmit
- backend: _ack_summary_for_claims helper attaches claim_acks payload
  to inbox rejected-lane rows (3 tests)
- frontend: InboxRow renders newest 3 AK2 chips + '+N more' overflow
  under the rejection reason, with '999 not linked' marker when an
  ack couldn't be linked to a claim
- frontend: per-row Resubmit button downloads a single corrected 837
  via api.serializeClaim837 (no bulk modal, no zip — one click, one
  .x12 file)
- frontend: 2 new tests for the chip rendering and the per-row
  download flow
2026-07-02 13:16:38 -06:00
Nora 9cc13e7940 docs(spec,plan): SP29 Inbox 999-rejected drill 2026-07-02 13:06:21 -06:00
Nora 37caf6af77 docs(plan): SP28 follow-up notes — Batch.kind fix + one-time backfill 2026-07-02 12:31:37 -06:00
Nora 02b879a204 merge: SP28 follow-up — fix Batch.kind '837' vs '837p' mismatch
Hotfix for the SP28 merge. The SP28 helpers (batch_envelope_index +
the two _batch_lookup closures in handle_ta1 / api.py) all filtered
Batch.kind == '837', but prod Batch rows are persisted with kind='837p'
(see api.py:443). Result on prod: claim_acks row count = 0, D10 Pass 1
(ST02 via batch.envelope.control_number) never fired, and the user's
specific claim t991102989o1c120d would not surface its 143 999 acks.

Fix: 4 production sites swapped to '837p'; 3 test files updated to
match the prod value. 25 SP28 tests + 20 handler tests pass post-fix.
Expected prod effect after re-deploy: 727 of 1,398 acks auto-link via
batch.envelope.control_number, plus the user's claim surfaces in
ClaimDrawer for /api/claims/t991102989o1c120d/acks.

Follow-up plan: backfill the existing 1,398 acks via a one-time script
that runs the SP28 helpers against the current DB (D10 join now
resolves the 991102989 ST02 from existing batches).
2026-07-02 12:22:36 -06:00
Nora fba594baf4 fix(sp28): align Batch.kind filter with prod ('837p' not '837')
The SP28 helpers (batch_envelope_index + lookup_claims_for_ack_set_response
+ two _batch_lookup closures in handle_ta1 + api.py) all filter
Batch.kind == "837", but prod Batch rows are persisted with
kind="837p" (lowercase p — see api.py:443 / store/records.py:58).
Result: the D10 batch-envelope index was empty on prod and zero 999 /
277CA / TA1 acks auto-linked to their claims via Pass 1 (ST02 via
batch.envelope.control_number). Pass 2 (PCN) was the only path firing,
which on this codebase matches 0 acks (Gainwell's 999 echoes the 837's
ST02, not its CLM01 — see spec §D10).

Fix: 4 production sites swapped to Batch.kind == "837p". 3 test
files updated to use the prod value (test_apply_claim_ack_links.py +
test_api_claim_acks.py + test_e2e_999_to_claim_drawer.py). All 25 SP28
tests + 20 handler tests pass. Expected prod effect: claim
t991102989o1c120d's 143 incoming 999 AK2 acks now auto-link via the
batch with envelope.control_number=991102989 (Pass 1), plus 727/1,398
total acks across all batches.
2026-07-02 12:22:29 -06:00
Nora 190f4b8c8d merge: SP28 ack-claim auto-link into main
Closes the operator gap where 999 / 277CA / TA1 acks were persisted but
never linked back to the claims they acknowledged. The original PCN-based
join matched 0 / 1,398 acks on prod (Gainwell's 999 echoes the source
837's ST02, not its CLM01); the D10 two-pass join via Batch.envelope.
control_number recovers 727 / 1,398 automatically.

Increment summary:
- New claim_acks join table (migration 0018) with partial unique dedup index
- Pure helpers (apply_claim_ack_links orchestrator + 999/277CA/TA1 paths)
  with the D10 two-pass lookup (Batch.envelope.control_number primary,
  Claim.patient_control_number fallback)
- CycloneStore facade: add_claim_ack, list_acks_for_claim,
  list_claims_for_ack, find_ack_orphans, remove_claim_ack, batch_envelope_index
- Handler integration: handle_999 / handle_277ca / handle_ta1
- 7 new API endpoints (read + stream + manual-match + unlink + orphan lane)
- Frontend: useClaimAcks / useAckClaims hooks, claimAcks live-tail slice,
  AcknowledgmentsPanel in ClaimDrawer, MatchedClaimPanel + manual-match
  dropdown in AckDrawer, Claims badge column on Acks page, Ack-orphans
  lane on Inbox page

Coverage: 727 / 1,398 acks auto-link; 671 remain real orphans (ST02=0001)
surfaced in the Inbox ack-orphans lane.

Auth posture: any logged-in user can run the manual-match endpoint
(D5/D9 deviation from admin-only remit-orphans — ack matches are
metadata-only, no Claim.state mutation).

Test baseline (no new failures):
- Backend: 25 new SP28 tests pass; full suite 1210 passed / 10 skipped /
  1 failed + 6 errors (pre-existing test_payer_summary +
  test_provider_extended_response pollution; all 7 pass in isolation)
- Frontend: 36 new SP28 tests pass; full suite 580 passed / 5 failed
  (pre-existing api.test.ts / tail-stream.test.ts / Inbox.test.tsx /
  InboxHeader.test.tsx baseline)
- Typecheck: 17 pre-existing errors remain; 0 in SP28-introduced files
  (refactor e10d388 fixed 3 pre-existing joinUrl import errors)

Files: 47 changed, +6,509/-23. Refactor e10d388 (joinUrl export) folded in
to keep SP28 merge coherent. 14 SP28 commits on branch preserved as audit
trail.
2026-07-02 12:12:31 -06:00
Nora 9d11ffcd8a docs(plan): SP28 deviations log — 4 backend + 5 frontend post-impl notes
Backend: helpers return dataclasses instead of inserting directly (preserves
publish-from-store contract); handlers commit work session before calling
cycl_store.add_claim_ack (SQLite single-writer); batch_envelope_index
accepts dict OR callable; link_manual returns ClaimAckLinkRow.

Frontend: TA1 row shape in Inbox lane; cross-page nav via useNavigate;
TA1 column header 'Batches' (not 'Claims'); ack-orphans lane position
between payer_rejected and candidates; fmt.usd instead of fmt.money
in MatchedClaimPanel.

Confirmed test baselines: backend 1210/10/6-err/1-fail = pre-SP28;
frontend 580 pass / 5 fail = pre-SP28. No new failures introduced.
46 files changed, +6,467/-22.
2026-07-02 12:05:45 -06:00
Nora 21a5485faa feat(sp28): api_routers/claim_acks (7 endpoints) + claim_acks surface on claims/acks
Phase 5 of SP28 (Ack↔Claim Auto-Link). Adds the public HTTP
surface for the claim_acks join table and wires it into the
existing claims + acks list endpoints.

New router (registered with matrix_gate auth, any-logged-in-user):

* GET    /api/claims/{id}/acks           — per-claim links
* GET    /api/claims/{id}/acks/stream    — NDJSON live-tail
* GET    /api/acks/{kind}/{id}/claims    — per-ack links
* GET    /api/acks/{kind}/{id}/claims/stream — NDJSON live-tail
* POST   /api/acks/{kind}/{id}/match-claim   — manual link (D5)
* DELETE /api/acks/{kind}/{id}/match-claim/{claim_id} — unlink
* GET    /api/inbox/ack-orphans          — orphan reconciliation

Manual match is any-logged-in-user (D5/D9), idempotent on the
(claim_id, ack_kind, ack_id) dedup key, rejects with 409 when the
claim is in a terminal state (REVERSED), and publishes
claim_ack_written / claim_ack_dropped on the bus so live-tail
subscribers refresh.

Existing list endpoints extended:
* /api/acks           — each item gains linked_claim_ids
* /api/ta1-acks       — each item gains linked_claim_ids
* /api/277ca-acks     — each item gains linked_claim_ids
* /api/claims/{id}    — body gains ack_links (compact form)

All three list extensions use a single batched SELECT against
claim_acks to avoid N+1.

Tests:
* tests/test_api_claim_acks.py — 8 tests covering all 7
  endpoints + the spec §6 named tests for the extended
  surfaces (claim_detail_includes_ack_links,
  acks_list_includes_linked_claim_ids,
  claim_acks_stream_emits_claim_ack_written). The stream test
  uses the established direct-endpoint-invocation pattern
  from test_api_stream_live.py so it gets true byte-streaming
  + clean async cancellation.
2026-07-02 11:55:15 -06:00
Nora 3fd2c445e7 fix(sp28): use fmt.usd instead of fmt.money in MatchedClaimPanel 2026-07-02 11:46:45 -06:00
Nora 9e16b8d9bd feat(sp28): wire apply_claim_ack_links into handle_999 / handle_277ca / handle_ta1
Phase 4 of SP28 (Ack↔Claim Auto-Link). Refactors the per-AK2
helpers in cyclone.claim_acks to return ClaimAckLinkRow dataclasses
instead of mutating the session directly — the orchestrator (the
999 / 277CA / TA1 handlers + the matching parse-* API endpoints)
now persists each row via cycl_store.add_claim_ack so the
publish-from-store contract owns the live-tail event emission.

* handle_999 / handle_277ca / handle_ta1 — build batch envelope
  index outside the work session (SQLite + concurrent sessions
  causes 'database is locked'), call apply_X to get
  ClaimAckLinkRow dataclasses, snapshot the rows before committing
  the work session, then call cycl_store.add_claim_ack per row
  in fresh sessions.
* /api/parse-999 / /api/parse-277ca / /api/parse-ta1 — mirror the
  handler chain with event_bus passed through so live-tail
  subscribers on the claim and ack sides see the new rows the
  moment they land. Adds a 'claim_ack_links_count' field to each
  ack response (spec §4).
* lookup_claims_for_ack_set_response — now accepts either a
  callable OR a plain dict as batch_envelope_index (the store
  returns a dict; tests pass callables).
* test_apply_claim_ack_links.py — 15 tests updated to assert on
  the dataclass shape and exercise the full helper→add_claim_ack
  cycle (so idempotency is verified at the store layer).
* test_e2e_999_to_claim_drawer.py — 2 new tests covering the
  D10 two-pass join end-to-end via FastAPI TestClient (Pass 1
  via ST02 + Pass 2 via PCN fallback).
2026-07-02 11:45:19 -06:00
Nora 1c367ddbe7 feat(sp28): Inbox ack orphans lane with Dismiss + Link to dropdown 2026-07-02 11:44:53 -06:00
Nora d8f3fb15f9 feat(sp28): Acks page Claims badge column with orphan/single/many variants 2026-07-02 11:40:50 -06:00
Nora 602dc352c5 feat(sp28): AckDrawer matched-claim panel + manual-match dropdown 2026-07-02 11:39:02 -06:00
Nora 490a1b9478 feat(sp28): ClaimDrawer acknowledgments panel + tests 2026-07-02 11:33:47 -06:00
Nora b094231995 feat(sp28): store.claim_acks facade + batch_envelope_index + to_ui_claim_ack serializer
The persistence layer for the SP28 join table lands in
cyclone/store/claim_acks.py:

- add_claim_ack — insert + publish 'claim_ack_written' on the bus
  (mirrors the publish-from-store pattern used by the ACKs paths)
- list_acks_for_claim / list_claims_for_ack — read helpers for the
  two list endpoints
- find_ack_orphans — Inbox ack-orphans lane source: every ack row
  whose ack_id has no ClaimAck row tied to it
- remove_claim_ack — unlink + publish 'claim_ack_dropped'
- batch_envelope_index — D10 in-memory map of
  {envelope.control_number: batch.id}, called once per ingest (cost
  is ~16 lookups today; trivially cheap)

Plus the to_ui_claim_ack serializer in store/ui.py mirroring
to_ui_ack shape — single source of truth for the wire format so the
live-tail event payload matches the list endpoint byte-for-byte.
Includes a single SELECT against claims for the claim_state field
(TA1 batch-level rows return 'n/a').

The CycloneStore facade in store/__init__.py re-exports all six
methods (add_claim_ack, list_acks_for_claim, list_claims_for_ack,
find_ack_orphans, remove_claim_ack, batch_envelope_index) plus
to_ui_claim_ack from the ui module.

Migration-version assertions in test_acks.py and test_db_migrate.py
bumped 17 → 18 to match the new migration head.

Steps 3.1/3.2/3.3/3.4 of the SP28 implementation plan.
2026-07-02 11:31:29 -06:00
Nora c54b2c1867 feat(sp28): claimAcks slice + useTailStream/useMergedTail dispatch 2026-07-02 11:31:17 -06:00
Nora a97f1d1350 feat(sp28): apply_claim_ack_links orchestrator + 999/277CA/TA1 helpers
The auto-linker closes the operator gap where every inbound 999 /
277CA / TA1 ack was persisted but never linked back to the claim it
acknowledges. Five pure helpers land in cyclone/claim_acks.py:

- lookup_claims_for_ack_set_response — D10 two-pass join. Primary is
  Batch.envelope.control_number (== source 837 ST02 for Gainwell
  batches); fallback is Claim.patient_control_number. Pass 1 wins,
  the two paths cannot both fire.
- apply_999_acceptances — walks parsed_999.set_responses, emits one
  ClaimAck per AK2 per matched claim (one-ack-to-many supported).
  Both accepted AND rejected AK2s link; per-AK2 granularity.
- apply_277ca_acks — same shape for parsed_277ca.claim_statuses.
  STC category code carried on the link row's
  set_accept_reject_code so the UI can render the lane inline
  without re-parsing raw_json.
- apply_ta1_envelope_link — envelope-level link. The link row has
  claim_id NULL + batch_id populated (the spec's batch-level TA1
  trace). Sender/receiver matching is delegated to a closure the
  caller supplies.
- link_manual — manually link an ack to a claim. Used by the new
  /api/acks/{kind}/{id}/match-claim endpoint. Idempotent.

All five helpers are pure (callers own the session); idempotent via
the partial unique index ux_claim_acks_dedup (helpers pre-check to
avoid IntegrityError log noise on re-ingest); flush-only (callers
commit).

14 of 15 named tests pass (the facade surface check belongs in
Phase 3 once store/claim_acks.py lands).

Steps 2.1/2.2/2.3/2.4/2.5 of the SP28 implementation plan.
2026-07-02 11:29:02 -06:00
Nora d2512945ca feat(sp28): useClaimAcks + useAckClaims hooks with api types 2026-07-02 11:26:25 -06:00
Nora 9e775c0150 feat(sp28): add 0018_claim_acks migration + ClaimAck ORM model
The claim_acks join table is the durable record of which inbound
999 / 277CA / TA1 ack acknowledged which claim (or, for TA1, which
originating 837 Batch). One row per AK2 set-response / ClaimStatus
/ envelope. The match granularity is per-AK2 so the operator can
answer 'which claims does this ack acknowledge?' with a single
SELECT and so the ClaimDrawer panel can show per-segment accept /
reject status without re-parsing raw_json.

Schema mirrors spec §3.1:
- claim_id NULLable + batch_id NULLable (TA1 envelope links land
  on batch_id; CHECK enforces at least one populated)
- unique partial index ux_claim_acks_dedup enforces idempotent
  re-ingest of the same 999 file
- set_control_number stores the value the upstream ack ACTUALLY
  CARRIED (== source 837 ST02 for Gainwell batches) for orphan
  traceability — the link survives even when the join had to fall
  back from ST02 to PCN matching

Mirrored on the ORM via Index(..., sqlite_where=text(...)) so
Base.metadata.create_all (the test-time safety net) emits the same
partial-unique constraint as the migration.

Step 1.1/1.2/1.3 of the SP28 implementation plan.
2026-07-02 11:24:49 -06:00
Nora e10d3886c2 refactor: export joinUrl from auth/api and reuse in lib/api
Single source of truth for the VITE_API_BASE_URL prefix logic. Both
auth/api and lib/api imported the same BASE_URL const; promote joinUrl
to a shared exported helper so future endpoints don't drift on the
trailing-slash normalization.
2026-07-02 11:18:22 -06:00
Nora ec87f98f02 docs(spec): SP28 D10 — two-pass join via Batch.envelope.control_number
The original SP28 spec assumed `Claim.patient_control_number == 999.set_control_number`.
Empirically that's wrong: Gainwell's 999 echoes the source 837's ST02, not its
CLM01, and TOC's billing software fills them differently. Measured against
prod on 2026-07-02: the PCN join matches 0 / 1,398 acks.

Fix: two-pass join. Primary is `Batch.envelope.control_number (== 837 ST02)
→ claims via batch_id`; fallback is `Claim.patient_control_number` (for
senders that fill CLM01 == ST02). Coverage after fix: 727 / 1,398 (52%);
the remaining 671 are real orphans (ST02=0001 placeholder, no matching 837
batch in our DB).

Spec adds D10 + a critical-correction paragraph at the top of §1. Plan
gains the `lookup_claims_for_ack_set_response` pure helper, the
`batch_envelope_index()` store method, and three new tests covering the
two-pass join + the false-positive guard + the one-ack-to-many case.
2026-07-02 11:16:53 -06:00
Nora f88a7c7c4f feat(acks): surface 999 rejections via sidebar badge + table sort
The 1,156 accepted acks were burying the 5 rejections — the operator
had to scroll the whole table to find them. Two changes:

1. Sidebar '999 ACKs' nav item gets a warning-toned badge showing
   the total rejected segment count from aggregates.rejected_count
   (mirrors the Reconciliation unmatched badge pattern). One number,
   visible from any page.

2. Acks page table now sorts rejected rows to the top, then
   newest-id-first. All 5 rejections fit on page 1 since they're a
   tiny fraction of the total — no pagination needed.

New files:
- src/hooks/useAckStats.ts (lightweight aggregates-only fetch)
- src/hooks/useAckStats.test.tsx (2 tests)
- src/components/Sidebar.test.tsx (3 tests)

Pre-existing baseline: 10 frontend failures (api.test.ts exportBatch837,
tail-stream.test.ts acks/ta1_acks targeting, Inbox/InboxHeader copy).
Unchanged.
2026-07-02 10:59:46 -06:00
Nora aca9667ff8 fix: enable CYCLONE_SCHEDULER_AUTOSTART in docker-compose
The SFTP poller was off by default in the production container —
operators had to POST /api/admin/scheduler/start manually after every
restart, which they were not doing. As a result the inbound 999 / 277CA
queue fell 7+ days behind in early July 2026 (scheduler.poll_count=0
in the live container since deploy).

Mirrors the existing CYCLONE_BACKUP_AUTOSTART=1 entry.

Note: there is a separate bug where scheduler._tick_impl's alphabetical
list_inbound hangs after ~30s on mft.gainwelltechnologies.com (TCP +
auth + first SFTP channel works, but the full alphabetic scan times
out). The pull-inbound CLI's date-filtered path works fine — SP to
follow.
2026-07-02 10:49:04 -06:00
Nora 999a762c86 docs(plan): SP28 ack-claim auto-link implementation plan 2026-07-02 10:32:17 -06:00
Nora 53a8c77cb8 docs(spec): SP28 ack-claim auto-link design 2026-07-02 10:30:52 -06:00
Nora de8bc2e153 merge: SP25 ack live-tail into main
Wires the live-tail triplet (useTailStream + useMergedTail +
<TailStatusPill>) into the Acks page so 999 and TA1 acknowledgments
appear the moment the SFTP poller (or a manual upload) lands them in
the database — no more 'no tracking is being updated'.

Backend changes
---------------
* store/acks.py: add_ack / add_ta1_ack / add_277ca_ack now publish
  ack_received / ta1_ack_received / two77ca_ack_received from inside
  the store after commit. Failures don't roll back the row.
* store/ui.py: new home for to_ui_ack / to_ui_ta1_ack / to_ui_two77ca_ack
  so the live-tail payload matches the list endpoint shape byte-for-byte.
* api_routers/acks.py + api_routers/ta1_acks.py: new /api/acks/stream
  and /api/ta1-acks/stream NDJSON endpoints following the live-tail
  wire format used by /api/claims/stream, /api/remittances/stream,
  and /api/activity/stream (snapshot + snapshot_end + live events +
  heartbeat + clean disconnect).
* api.py: parse-999 / parse-ta1 / parse-277ca endpoints thread
  request.app.state.event_bus into the store on write.
* handlers/handle_*.py: dropped the event_bus kwarg; the publish
  path is now store-only, not handler-driven.

Frontend changes
----------------
* lib/tail-stream.ts: TailResource widened to include acks + ta1_acks.
* store/tail-store.ts: acks + ta1Acks slices, addAck + addTa1Ack
  setters, matching reset cases. evictOldest is generic over
  (K extends string|number, V) so both keyed-by-id flavors share it.
* hooks/useTailStream.ts: dispatch routes acks / ta1_acks items into
  the matching store slices.
* hooks/useMergedTail.ts: generic constraint widened to
  T extends { id: string | number }, dedup normalizes via String(id)
  so numeric ids compare correctly.
* pages/Acks.tsx: opens both streams + renders two <TailStatusPill>
  components (one in the 999 hero, one in the TA1 section).

Verification
------------
* Backend: 1191 pass, 1 pre-existing test pollution failure (also
  fails on main, passes in isolation, unrelated to SP25).
* Frontend: 535 pass, 10 pre-existing failures in Inbox / Upload /
  Receipt / InboxHeader tests (all verified to fail the same way on
  main).
* TypeScript: zero new SP25 errors. Pre-existing Upload / Inbox
  errors match main.

Per the SP-N increment flow: no squash, no rebase. The merge commit
is the record of the increment landing.
2026-07-02 09:30:49 -06:00
Nora 1363d36046 fix(sp25): typecheck fixups
Three small follow-ups after the live-tail triplet wired up:

  * evictOldest is now generic over (K extends string|number, V) so
    the addAck / addTa1Ack call sites don't need a type assertion.
  * Acks.test.tsx no longer spreads importOriginal() — TypeScript
    was choking on the inferred 'unknown' from the mock factory.
    Use the same ApiError / mock shape that Claims.test.tsx uses.
  * tail-stream.test.ts guards the optional gen.return() with
    so strict mode TS2722 stops complaining about possibly-undefined.
2026-07-02 09:23:44 -06:00
Nora 146cb6d17d feat(sp25): wire live-tail triplet into Acks page
The 999 register opens /api/acks/stream via useTailStream("acks"),
merges the snapshot + tail through useMergedTail("acks", ...) so
new rows appear without a manual refresh, and surfaces the
connection state via <TailStatusPill> in the hero. The TA1 section
gets the same triplet against /api/ta1-acks/stream.

Both pills sit in the page header so the operator can see at a
glance whether the live-tail connection is healthy, without having
to open the drawer or refresh the page. A stalled/error stream
shows a Reconnect button inline.

The Acks test mock adds useTailStream at the module level so the
page renders without opening a real fetch. New test asserts both
TailStatusPills mount in the page tree.
2026-07-02 09:12:19 -06:00
Nora 191bc935c3 feat(sp25): extend useMergedTail to handle acks + ta1_acks slices
The Acks page needs two merge hooks (one per ack flavor) and both
shapes use numeric database ids rather than the string ids that
Claim/Remittance/Activity use. The generic constraint widens to
`T extends { id: string | number }` and the dedup Set normalizes
to String(id) so a base item with id=2 and a tail item with id=2
collide correctly.

Adds two switch cases that mirror the claims/remittances keyed-by-id
pattern: iterate the order array, look up the matching value in the
dict, drop undefined holes defensively.

Two tests assert the new slices order by their order arrays and dedup
against base items.
2026-07-02 09:09:53 -06:00
Nora 029623f3a5 feat(sp25): extend useTailStream dispatch for acks + ta1_acks
The Acks page mounts two streams (useTailStream("acks") and
useTailStream("ta1_acks")) — the dispatcher now routes their item
events into the matching store slices (addAck / addTa1Ack) the same
way the claims / remittances / activity cases already do.

Two new tests assert that an acks item lands in the acks slice (not
in claims) and a ta1_acks item lands in the ta1Acks slice (not in
acks) — i.e. the routing is exclusive, so a payload keyed by id 7
won't bleed across the two ack tables.
2026-07-02 09:08:46 -06:00
Nora 1648c81425 feat(sp25): extend useTailStore with acks + ta1_acks slices
The Acks page needs the same live-tail triplet as Claims/Remittances/
Activity. Both Ack and Ta1Ack have stable numeric ids from the
database row, so they get the keyed-by-id treatment (first write
wins, FIFO-capped at TAIL_CAP) — same shape as the claims and
remittances slices, not the append-only activity array.

Adds addAck, addTa1Ack, ackOrder, ta1AckOrder, and the matching
reset cases for the new 'acks' / 'ta1_acks' TailResource kinds.

evictOldest is now generic over T so the same helper covers the
two new keyed-by-id slices without duplication.
2026-07-02 09:07:26 -06:00
Nora f1bee546f6 feat(sp25): extend TailResource to include acks + ta1_acks
The Acks page needs the same live-tail triplet (useTailStream +
useMergedTail + TailStatusPill) that Claims/Remittances/Activity
already use. The first step is widening the TailResource union so
streamTail("acks") and streamTail("ta1_acks") are valid and the
URL falls out as /api/acks/stream and /api/ta1-acks/stream.

Two new tests assert the URL targets the new backend endpoints.
2026-07-02 09:01:53 -06:00
Nora 603ec8842a feat(sp25): /api/acks/stream + /api/ta1-acks/stream NDJSON endpoints
Adds two streaming endpoints that match the live-tail wire format
established by /api/claims/stream, /api/remittances/stream, and
/api/activity/stream:

  * /api/acks/stream      — subscribes to ack_received
  * /api/ta1-acks/stream  — subscribes to ta1_ack_received

Both yield a snapshot of existing rows (newest first, capped by the
`limit` query param), then a `snapshot_end` line, then forward
live events from the bus. They are registered before the
/{ack_id} path-param endpoints so the literal `stream` segment
isn't matched as an id.

Tests use the same direct-coroutine pattern as test_api_stream_live.py
because httpx.ASGITransport buffers the response body and never
delivers a disconnect message — iterating body_iterator directly
with body_iterator.aclose() simulates a client disconnect.
2026-07-02 09:00:54 -06:00
Nora 686c11f480 feat(sp25): drop event_bus kwarg from handlers; thread event_bus into parse endpoints 2026-07-02 08:52:20 -06:00
Nora 8d11b391a0 feat(sp25): add_*_ack publish ack_received / ta1_ack_received / two77ca_ack_received 2026-07-02 08:47:03 -06:00
Nora 4b22193c4a feat(sp25): move ack UI serializers to store/ui.py 2026-07-02 08:45:19 -06:00
cyclone d8c03fde3f docs(plan): plan for SP25 ack live-tail 2026-07-02 08:34:54 -06:00
cyclone 014e02ad42 docs(spec): design for SP25 ack live-tail 2026-07-02 08:30:39 -06:00
Nora 4360ef7209 merge: SP21 store split into main
Splits the 2,995-LOC backend/src/cyclone/store.py into a
cyclone/store/ subpackage (13 sibling modules + __init__.py
facade) with no public API change. Adds dashboard_kpis,
check_matched_pair_drift, _claim_state_str (SP27) into the
right homes, and surfaces 3 private helpers (_claim_status_from_validation,
_persist_835_remit, _remittance_835_row) from the facade for
existing test imports.

Spec: docs/superpowers/specs/2026-06-21-cyclone-store-split-design.md
Plan: docs/superpowers/plans/2026-06-29-cyclone-store-split-resume.md

Tests at baseline: 1 failed (pre-existing isolation flake),
1176 passed, 10 skipped.
2026-06-29 16:20:15 -06:00
Nora fe84ce7cf3 fix(sp21): drop dead code surfaced by code-quality review
- write.py: drop orphan run_reconcile() — no callers, and the
  pre-split store.py never had it. (The CycloneStore class already
  has _publish_events_sync + _sync_publish delegations; the spec's
  mention of _run_reconcile was a doc drift from the 2026-06-21
  plan, not a real requirement.)
- ui.py: drop duplicate _provider_orm_to_dict + _payer_orm_to_dict
  (canonical copies live in providers.py and are the only ones used;
  the ui.py copies were accidental duplicates from the split).
- orm_builders.py: drop orphan _cas_adjustment_row() — never called;
  _persist_835_remit() builds CAS rows inline.
- __init__.py: prune 30+ unused top-level imports and 9 unused
  re-exports. The facade only ever needs the type-hint-bearing
  BatchRecord family, the 4 read-path ORM-row serializers actually
  used inside the package, the 3 documented private helpers, and
  the 7 function-level re-exports the spec promises. Everything
  else was carryover from the original store.py header.

Tests at exact baseline: 1 failed, 1176 passed, 10 skipped
(the 1 failure is the pre-existing test_provider_detail isolation
flake documented in the spec).
2026-06-29 16:20:01 -06:00
Nora cfc95307e3 fix(sp21): remove orphan _ITER_UNBOUNDED from ui.py (now in claim_detail.py) 2026-06-29 16:04:42 -06:00
Tyler Martinez c8f8f5d3c6 refactor(sp21): split store.py into cyclone/store/ subpackage (14 modules)
Behaviour-preserving structural split of the 2,995-LOC store.py into
a 14-module subpackage with a thin facade. CycloneStore class keeps
its full method surface as 1-line delegations to module functions.

14 modules:
  exceptions, records, orm_builders, ui, write,
  batches, claim_detail, kpis, acks,
  backups, inbox, providers
  (+ __init__.py facade)

Facade re-exports 12 public symbols (CycloneStore, store, BatchRecord*,
BatchKind, AlreadyMatchedError/NotMatchedError/InvalidStateError, utcnow,
dashboard_kpis, check_matched_pair_drift) + 3 private helpers
(_claim_status_from_validation, _persist_835_remit, _remittance_835_row)
to preserve the 4 test files that import them.

Zero public API changes, zero test changes, zero importer changes.
CycloneStore._lock and CycloneStore._batches.clear() remain intact for
the 7 test files still using the cleanup idiom.

Verified: 1,176 / 1 (pre-existing isolation flake) / 10 tests pass —
identical to baseline.
2026-06-29 15:57:31 -06:00
Nora 9644db8c51 docs(plan): resume SP21 store split — 14 modules, function-name refs (no line numbers)
Fresh plan that supersedes the 2026-06-21 line-number-anchored plan.
Reflects the resumed state: store.py at 2,995 LOC, 14 target modules,
1,176/1/10 test baseline, 3 new symbols to extract (dashboard_kpis +
_claim_state_str → kpis.py; check_matched_pair_drift → claim_detail.py),
3 private-helper re-exports (added _persist_835_remit + _remittance_835_row
discovered during the audit).

Per-task code copies dropped in favor of function-name references +
read-store.py guidance — the line numbers in the original plan are
inherently stale against a 2,995-LOC file under active development.

The original 2026-06-21 plan is preserved on disk as historical record
of the design conversation.
2026-06-29 14:47:54 -06:00
Nora fbe7b2358d docs(spec): resume SP21 store split — 14 modules, 3 SP27 symbols, 3 private-helper re-exports
Update the 2026-06-21 approved spec with the post-SP27 reality:
store.py grew 2,412 → 2,995 LOC. Three new top-level symbols
(dashboard_kpis, _claim_state_str, check_matched_pair_drift) get
slotted into the new kpis.py and claim_detail.py. Three private
helpers (the original _claim_status_from_validation + the newly-
discovered _persist_835_remit and _remittance_835_row) are re-
exported from the facade to preserve the 4 test files that
import them. Branch bumped to sp21-store-split per SP-N convention.

All previously-locked Decisions are preserved verbatim.
2026-06-29 14:44:21 -06:00
Nora b96da8a5a4 merge: SP27 remittances architecture refactor into main
Atomic SP-N merge per cyclone-spec. Carries the remittances
architecture refactor (handlers/ extraction, unified 835 ingest +
reconcile, server-aggregated KPIs) plus two same-increment fixes
for bugs surfaced against the running stack:

  - fix(sp27): /api/claims + /api/remittances total counts the
    full population (previously page-local reduce)
  - feat(sp27): server-aggregate Dashboard KPIs so the 100-row
    sample doesn't lie
  - fix(sp27): Claim.patient_control_number populated from CLM01
    (claim_id), not 2010BA NM109 (member_id) — restores 837↔835
    auto-match via the existing reconcile.by_pcn join
  - feat(sp27): server-aggregate Remittances KPIs (count, paid,
    adjustments)

Includes migration 0017 (UPDATE claims SET patient_control_number
= id) which was applied to the live DB at ingest time.
2026-06-29 14:33:52 -06:00
Nora 14fcbca5f1 feat(sp27): server-aggregate Remittances KPIs (count, paid, adjustments)
The Remittances page's three KPI tiles — REMITS / TOTAL PAID /
ADJUSTMENTS — were computed page-locally via items.reduce(...) over
the merged tail of the current page + live delta. With a 100-row
default limit, a 1,739-row population showed count=100, paid=$16,934,
adjustments=$147 — silently understating reality because the page
hadn't loaded the remaining rows yet.

This change mirrors the silent-incompleteness fix that
/api/dashboard/kpis (commit 59c3275) and /api/remittances (commit
d81b6ed) made for their tiles:

  * CycloneStore.summarize_remittances() iterates the full filtered
    remittance population (no limit) and returns
    {count, total_paid, total_adjustments}. Mirrors iter_remittances
    with limit=_ITER_UNBOUNDED.
  * GET /api/remittances/summary — server endpoint with the same
    filter parameters as /api/remittances. Registered BEFORE the
    /api/remittances/stream handler so FastAPI doesn't treat
    'summary' as a stream sub-path.
  * api.listRemittanceSummary + useRemittanceSummary hook.
  * Remittances.tsx swaps off items.reduce, consumes the server
    summary. Tiles render the server totals so the values reflect
    the entire DB population, not the page-local sample.

Verified live: /api/remittances/summary returns
{count: 1739, total_paid: 227181.58, total_adjustments: 13792.65},
which matches DB ground truth exactly.

Tests: 8 new backend tests in test_api_remittances_summary.py; 1 new
frontend test in Remittances.test.tsx (kpi_tiles_use_server_summary_not
_page_local_reduce) plus the page-level test for the zero-valued mock
default.
2026-06-29 14:32:10 -06:00
Nora d8841834dc fix(sp27): Claim.patient_control_number populated from CLM01 (claim_id), not 2010BA NM109 (member_id)
The 837 ingest path (_claim_837_row in store.py:194) populated
Claim.patient_control_number from claim.subscriber.member_id — the
subscriber's 2010BA NM109 Medicaid ID — instead of from
claim.claim_id (CLM01, the claim submitter's identifier the 837
actually sent).

That silently broke every downstream join that uses this column as a
cross-reference key:

  * reconcile.match() (reconcile.py:74) — joins
    Claim.patient_control_number against
    Remittance.payer_claim_control_number (which is parsed from CLP01,
    the 835's echo of CLM01 per X12 spec). Member_id never matches
    CLP01, so auto-match always fails.
  * apply_999_rejections — same lookup, same broken key.
  * apply_277ca_rejections — same lookup, same broken key.
  * scoring.score_pair — same broken key.

Live DB probe (1,739 remits, 337 claims):
  * Claim.id == Remit.payer_claim_control_number    → 0 matches
  * Claim.patient_control_number == Remit.payer_claim_control_number
    → 9 matches (substring coincidences; synthetic PCN strings share
    alphanumeric characters with the human-readable member_ids)
  * Claim.matched_remittance_id NOT NULL             → 0 claims

This commit changes _claim_837_row to write
Claim.patient_control_number = claim.claim_id (= CLM01). The
reconcile matcher's existing join now hits the row the 835 echoes
back. Companion migration 0017 backfills the 337 pre-fix rows so
they're on equal footing with new ingests (UPDATE claims SET
patient_control_number = id WHERE patient_control_number IS DISTINCT
FROM id — idempotent).

Tests:
  * 2 new RED→GREEN tests in test_store_reconcile.py:
    - test_837_ingest_populates_patient_control_number_from_claim_id
      pins the field semantics directly
    - test_837_then_835_with_echoed_pcn_auto_pairs proves the full
      end-to-end auto-match now fires
  * 3 existing manual_match tests that relied on the bug (had setup
    using member_id != claim_id to deliberately prevent auto-match)
    updated to use distinct PCNs explicitly so the tests still
    exercise the manual-match path with a real orphan pair.
  * 3 store_claim_detail / api_gets tests adjusted for the same
    reason.

Verified: 1,176 / 1,177 backend tests pass; the one failure is the
pre-existing flake in test_provider_extended_response.py noted before
this work started.

Caveat for the existing dev DB: the 1,739 remits already in the DB
were ingested from 835 fixtures whose CLP01 is a different synthetic
identifier than the 837's CLM01 (the test fixtures never echoed
CLM01 — a fixture-data limitation, not a code bug). After this fix,
new 837+835 ingest pairs whose payer echoes CLM01 in CLP01 will
auto-match as expected. The pre-existing 1,739 remits will continue
to land in the unmatched bucket; that can only be fixed by
regenerating the test fixtures (out of scope for this SP).
2026-06-29 14:32:10 -06:00
Nora d81b6ed4fc fix(sp27): /api/claims + /api/remittances total counts the full population
Before this fix, the list endpoints computed `total` via
`len(list(store.iter_*(**common)))`. Both `iter_claims` and
`iter_remittances` default to `limit=100`, so the reported total
silently capped at 100 even when the DB held 60k claims or 835
remits. The frontend rendered a `data.total` of 100 in the KPI
tile and a 100-row table, the page looked complete, and the bug
stayed hidden — exactly the Dashboard silent-failure pattern that
59c3275 (server-aggregated KPIs) was meant to retire.

The Remittances page symptom reported on Jun 29 ('REMITS 100',
empty CLAIM column on 100 rows) was this bug. The empty CLAIM is
a separate matching concern (those remits have `claim_id = NULL`
in the DB); the count fix addresses the population-size lie.

Fix:
* `CycloneStore.count_claims` + `count_remittances` reuse the
  iter's filter pipeline (DB filters + in-memory payer/date
  filters) with an effectively-unbounded limit so the count
  reflects the true DB population.
* `list_claims` + `list_remittances` call the new count
  helpers instead of slicing `iter_*` twice.
* 13 new tests in `test_list_endpoint_counts.py` cover the
  empty/filter/per-dimension/cardinality cases plus the HTTP
  regression (seed 150/120, assert total matches).

Bonus fix:
* `conftest._reset_rate_limit_buckets` walks the middleware
  stack and clears `RateLimitMiddleware._buckets` between
  tests. Without this, the full suite tripped the 300 req/60s
  rate limiter mid-run and 9 later tests (4 of mine, 5
  pre-existing in test_inbox_endpoints / test_payer_summary)
  got 429s. All 1167 tests now pass.

Follow-ups noted but out of scope:
* `/api/admin/audit-log`, `/api/admin/backup/list`,
  `/api/admin/scheduler/processed-files` use the same
  `len(rows)`-after-`.limit(limit)` pattern. Admin-only,
  no `has_more` consumption, so impact is bounded.
* `count_*` could short-circuit to `func.count()` to skip
  the UI-dict build, but iter already calls `q.all()` so
  the win is modest.
2026-06-29 13:22:42 -06:00
Nora 59c3275adf feat(sp27): server-aggregate Dashboard KPIs so 100-row sample doesn't lie
The Dashboard was hardcoded to useClaims({ limit: 100 }) and reduce
KPIs client-side. With 60k+ claims in production, every tile
(Billed $940K, Received $59K, Denial rate, Pending AR, monthly
sparkline, top providers, recent denials) was computed from a
0.16% sample of the dataset — silently wrong numbers on the
operator's primary view.

Fix: add GET /api/dashboard/kpis that aggregates server-side in
one read over the entire claim population. The new useDashboardKpis
hook consumes it and polls every 60s. Dashboard.tsx drops
useClaims({limit:100}) + useProviders() + the client-side
buildMonthly reduce.

Backend:
- store.py: dashboard_kpis() — one Claim query (selectinload on
  batch to avoid N+1) + one bulk Remittance lookup, Python reduce
  over the full population. Zero-filled response for empty DB.
- api.py: GET /api/dashboard/kpis behind matrix_gate, query-param
  clamps (1..24 months, 0..50 top_n_*).

Frontend:
- api.ts: DashboardKpis types + getDashboardKpis() wrapper.
- useDashboardKpis.ts: TanStack Query hook, 60s refetchInterval,
  bypass to data:undefined when not configured.
- Dashboard.tsx: switched to useDashboardKpis, extracted
  ZERO_TOTALS constant, dropped the buildMonthly helper.

Tests:
- backend/tests/test_dashboard_kpis.py: 12 tests covering empty DB,
  matched-remit math, pending-state semantics, monthly binning,
  top-providers/top-denials sort + cap, orphan-claim defensive
  guard, HTTP wiring + param validation.
- src/hooks/useDashboardKpis.test.ts: 3 tests for the hook
  contract (configured path, unconfigured fallback, param
  passthrough).
- src/pages/Dashboard.test.tsx: wrapped renders in
  QueryClientProvider + stubbed useAuth + isConfigured=false. This
  fixes 3 pre-existing Dashboard test failures (the page never had
  a QueryClient set up because useClaims/useProviders were the
  first useQuery hooks in the page).

Reviewer fixes (same commit):
1. topDenials sort placed empty submissionDate claims first under
   reverse-lex. Drop them at append time.
2. r.batch lazy-load → N+1 on 60k rows. selectinload(Claim.batch).
3. pending_states rebuilt per call as a mutable set — moved to
   module-level _DASHBOARD_PENDING_STATES frozenset.
4. Module-level ProviderORM import inconsistent with the
   "local import inside the function" pattern — moved inline.
2026-06-29 12:51:28 -06:00
Nora f5d119fbe7 feat(sp27): pin Claim↔Remit matched-pair invariant + startup drift audit
Add check_matched_pair_drift() at module level in store.py — read-only
audit of the Claim.matched_remittance_id ↔ Remittance.claim_id FK pair.
Logs WARNING on drift with up to 5 examples of each of two cases
(claim-side and remit-side), returns the count of drifted rows. Wired
into api.py::lifespan right after db.init_db() and ensure_clearhouse_seeded(),
wrapped in try/except so a query failure logs but doesn't crash boot.

The symmetric-write + symmetric-clear invariants on
manual_match / manual_unmatch were already correct; this commit pins
them with focused regression tests in test_store_match_invariant.py
(8 tests covering both directions, both happy-path, rollback pin, and
drift-check unit behavior). PCN asymmetry in the fixture prevents the
auto-match inside CycloneStore.add (Task 10) from pre-pairing, so
manual_match is the only writer.

Migration 0016 adds the missing ix_claims_matched_remittance_id index —
the drift check scans WHERE matched_remittance_id IS NOT NULL on every
boot and would become a full-table scan past ~10k claims. Symmetric
with ix_remittances_claim_id (added in 0007). Migration tests bumped
from head=15 to head=16.
2026-06-29 12:21:34 -06:00
Nora d5f95b4f3c feat(sp27): unify 835 ingest + reconciliation in handle_835 (atomic)
Move 'reconcile.run(s, record.id)' inside CycloneStore.add()'s ingest
session, before s.commit(). The placeholder adjustment_amount set by
_remittance_835_row is overwritten by reconcile's CAS-aggregate pass
in the same transaction — readers never see a half-reconciled
Remittance row. If reconcile raises, the entire 835 ingest rolls back
via the session's __exit__.

The previous flow committed batch + remittance rows in session-1,
then opened session-2 to run reconcile.fail-soft. Two visible
problems closed: a race window where readers fetched the placeholder
adjustment_amount, and a half-reconciled state left visible if
reconcile crashed.

Deviation from the plan (N4 in autoreview): the reconcile call lives
in cycl_store.add() rather than handle_835 calling a new
reconcile.run_now(batch_id) helper after add(). Same end state, one
fewer module surface, handler stays a thin wrapper over the store.
2026-06-29 12:08:47 -06:00
Nora 44a6fb031a feat(sp27): surface consecutive_failures + last_error in Scheduler.status() 2026-06-29 11:53:56 -06:00
Nora cf1a0a80d8 feat(sp27): wrap SFTP list_inbound in asyncio.wait_for with CYCLONE_SFTP_OP_TIMEOUT_SECONDS
The 06/25 silent hang was a hung ``sftp.listdir_attr()`` on the
worker thread — paramiko TCP-acked then went silent, the scheduler's
``asyncio.to_thread`` waited forever, and the operator had no
signal that polling had stalled. Every poll cycle since was
suspected of the same failure mode.

``backend/src/cyclone/clearhouse/__init__.``
- New ``_op_timeout_seconds()`` helper reads
  ``CYCLONE_SFTP_OP_TIMEOUT_SECONDS`` (default 30s). Rejects
  unparseable, zero, and negative values — ``asyncio.wait_for(timeout=0)``
  raises immediately and ``timeout<0`` is undefined per the asyncio
  docs, so a typo would silently turn every SFTP call into an
  instant failure. Logs a WARNING and falls back to the default.
- New ``async_list_inbound()`` async wrapper applies
  ``asyncio.wait_for(asyncio.to_thread(self.list_inbound),
  timeout=N)``. ``wait_for`` cancels the awaiter after N
  seconds but the worker thread keeps running until paramiko returns
  on its own (paramiko is not asyncio-aware, can't be cancelled
  cleanly). ``asyncio.TimeoutError`` propagates so the scheduler
  can surface it as a transient SFTP error.

``backend/src/cyclone/scheduler.py``
- ``_tick_impl`` now calls ``client.async_list_inbound()``
  instead of ``asyncio.to_thread(self._list_inbound)``. Catches
  ``asyncio.TimeoutError`` explicitly with a clear error message
  ("list_inbound: timeout") — separate from the generic
  ``Exception`` catch-all so the operator's tick-result error
  reads as the actual cause.
- Deleted the now-dead ``Scheduler._list_inbound()`` shim.

``backend/tests/test_sftp_op_timeout.py`` (new, 6 tests):
1. Default timeout is 30s when the env var is unset.
2. Operator can override via env var without restart-rebuild.
3. Unparseable value (e.g. "30s") falls back to default.
4. Zero and negative values fall back to default.
5. A hanging list_inbound raises ``asyncio.TimeoutError`` within
   the configured bound (the 06/25 hang pin).
6. A non-hanging call returns the stub's empty list normally
   (sanity check that the wrapper doesn't break the happy path).

Scope note: only ``async_list_inbound`` is wired up. The other
SFTP methods (``list_inbound_names``, ``download_inbound``,
``read_file``, ``write_file``) are called from
operator-triggered paths (admin endpoints, CLI, claim submission)
where the operator can Ctrl-C the request, so a hang is at least
visible. Wrapping them would require making the FastAPI handlers
async, which is out of scope for Task 8. Tracked as a follow-up —
worth folding into the same shape when those endpoints are next
touched.
2026-06-29 11:43:58 -06:00
Nora 34542d1d34 feat(sp27): loosen INBOUND_RE to accept suffix-less inbound filenames
Gainwell's production filer has shipped at least two inbound filename
shapes — the spec form with a trailing `_{file_type}.x12` suffix and
a shorter suffix-less form where the disambiguator token (between
`-` and `_M`) doubles as both orig_tx and file_type. The 6/15–6/19
835 batch arrived in the suffix-less form, and the strict
`INBOUND_RE` rejected them outright — the scheduler silently
dropped 5 days of production data without logging an error.

`backend/src/cyclone/edi/filenames.py`
- New `INBOUND_RE_LOOSE` regex: same prefix/tpid/tracking/ts/seq
  rules as `INBOUND_RE`, but the disambiguator token is required
  to be 3–5 uppercase alnum (covers 999, TA1, 835, 277CA, ENCR; the
  5-char cap stops the engine from over-eating the next `_M` token
  in a degenerate input).
- `parse_inbound_filename` now tries `INBOUND_RE` first
  (preserves historical behavior for every existing caller) and
  falls back to `INBOUND_RE_LOOSE`. In the loose form, orig_tx is
  set to the disambiguator token so the parsed shape matches what
  the strict form produces when orig_tx == file_type. The
  `ALLOWED_FILE_TYPES` check is enforced in both branches.
- `is_inbound_filename` is loosened in the same shape so the two
  never disagree — a refactor that pre-filters a directory listing
  with `is_inbound_filename` then re-parses with
  `parse_inbound_filename` would otherwise see the suffix-less
  files rejected twice.

`backend/tests/test_inbound_filename_loose.py` (new, 10 tests):
1. Spec form with explicit `_835.x12` suffix still wins (strict
   path unchanged).
2-5. Suffix-less 835 / 999 / 277CA / ENCR all parse correctly.
6. Suffix-less `.txt` is still rejected (the `.x12` ext check
   applies to both forms).
7. Suffix-less unknown type (4-char `ABCD`) is rejected by
   ALLOWED_FILE_TYPES — the loose regex's `{3,5}` shape would
   otherwise let it through.
8. Suffix-less 6-char token (`999XX6`) is rejected by the
   5-char cap, preventing the engine from swallowing the next
   `_M` token.
9. Strict form takes precedence over the loose form when both
   match — pinning the parser's branch order so a refactor can't
   silently change the parsed shape.
10. `is_inbound_filename` accepts the loose form, the spec form,
    and rejects garbage.

Live smoke: 5/5 suffix-less Gainwell patterns now parse correctly
(were ValueError before); spec form unchanged; 4 invalid forms
still rejected. Full backend suite: 1085/1121 — the 36 pre-existing
failures (test_serialize_837, test_api_stream_live,
test_inbox_endpoints) are unrelated and confirmed pre-existing by
running them on stashed pre-change code.
2026-06-29 11:30:00 -06:00
Nora 4c05c6527b hotfix(acks): paginate /api/acks and surface server-side aggregates
The Acks page silently capped at 100 of 1056 rows in the operator's
DB: the eyebrow read `${items.length} on file` (not the server's
`total`), the KPI strip summed from the 100 visible items, and the
endpoint accepted no `offset` — so the user had no signal that 956
more acks existed. Same class of bug on the Activity page (cap=200,
no 'X of Y' hint).

Backend
- /api/acks (acks.py): add `offset`, bump `le=1000`→`le=5000`,
  slice `rows[offset:offset+limit]`, return server-side
  `aggregates` (accepted/rejected/received summed over the full
  row set, not the page) so the KPI strip reflects every persisted
  999 instead of just the visible 50.
- /api/activity list + stream (api.py): bump `le=500`→`le=5000` so
  the page can ask for a denser snapshot.
- /api/277ca-acks (api.py): bump `le=1000`→`le=5000` for
  consistency.
- /api/ta1-acks: left at `le=1000` — TA1s aren't shipped today and
  the structural fix (offset + aggregates) wasn't applied, so a
  larger cap would just make the same latent silent-failure easier
  to hit. (TODO: fold in the same shape when Gainwell starts
  shipping TA1s.)

Frontend
- listAcks (api.ts): accept `offset`, surface `aggregates`,
  adapt wire `*_count` keys to the in-page
  `accepted`/`rejected`/`received` shape so the page can use
  `data.aggregates` as a drop-in for the page-local fallback
  accumulator.
- useAcks (hooks): pass `offset` through; return type carries
  `aggregates`.
- Acks.tsx: add `page` state (PAGE_SIZE=50), use `data.total`
  for eyebrow + watermark (not `items.length`), use
  `data.aggregates` for the KPI strip (with in-page fallback
  accumulator on first paint), render `<Pagination>` when
  `totalCount > PAGE_SIZE`. Footer row reads "N rows on file"
  instead of "N rows".
- ActivityLog.tsx: bump `limit: 200`→`limit: 500`, eyebrow reads
  "Activity · showing N most recent" to make the bounded-window
  semantics honest (the endpoint doesn't expose a true total — it
  reports events matching the current kind/since filter, capped at
  the request limit).

Tests
- test_acks.py: 4 new tests pin the fix:
  1. `offset` walks the full set; `has_more` flips at the
     boundary.
  2. `aggregates` reflects the full row set, not the page (the
     silent-failure pin) — and stays stable across page slices.
  3. `limit` cap of 5000 is enforced (422 above it).
  4. `offset` past the end returns an empty page with stable
     aggregates (a stale UI page state across a row count change
     must not 500 or zero the KPIs).

Live smoke-verified: /api/acks?limit=2&offset=0 vs ?offset=2 return
the expected row slices, aggregates stable at 15/10/15 for 5 seeded
rows, /api/acks?limit=10000 rejected with 422.

Triage note: the TA1 section (`Ta1AcksSection`, lines 609-616 of
Acks.tsx) has the same latent silent-failure pattern (page-sums
KPIs, no offset on /api/ta1-acks). Left untouched because the
empty-state copy says Gainwell doesn't ship TA1s today and the
larger structural fix belongs in a follow-up.
2026-06-29 11:19:10 -06:00
Nora 4592bca372 feat(sp27): dedup ack ID helpers — one copy in handlers/_ack_id.py 2026-06-29 11:00:37 -06:00
Nora 79fa30d018 feat(sp27): extract handle_835 from scheduler.py into handlers/ 2026-06-29 10:52:07 -06:00
Nora 35730fcf14 feat(sp27): extract handle_277ca from scheduler.py into handlers/ 2026-06-29 10:46:29 -06:00
Nora 30e1add8a2 feat(sp27): extract handle_ta1 from scheduler.py into handlers/ 2026-06-29 10:38:32 -06:00
Nora d248a5f282 feat(sp27): extract handle_999 from scheduler.py into handlers/ 2026-06-29 10:31:40 -06:00
Nora 0f1e609888 feat(sp27): create handlers/ package skeleton + dedup ack ID helpers
CycloneStore split precedent (SP21) — lift three helpers from
scheduler.py + api.py into one module. Both callers will switch
to this in Task 6. The new package also defines HandleResult
and the HANDLERS registry; handle_999 / handle_ta1 / handle_277ca
/ handle_835 fill in over Tasks 2-5.

The registry is lazy + best-effort: register_handlers() catches
broad Exception so a partial-modification SyntaxError in one
in-flight handler module can't break scheduler or API import.

11 tests added; existing suite unchanged (1 failed + 2 errors
pre-existing in test_provider_extended_response.py are not
introduced by this commit).
2026-06-29 10:09:10 -06:00
Nora 9d8f83d111 docs(plan): implementation plan for SP27 remittances architecture refactor
17 tasks: 1 preflight + 4 handler extracts (999/TA1/277CA/835) +
helpers dedup + INBOUND_RE loosen + SFTP timeouts + status surface +
atomic 835/reconcile + match invariants + chain endpoint +
claim.rejected_after_remit + frontend chain UI + final verify + merge.

Each task ends with live-test + autoreview + commit. Spec:
docs/superpowers/specs/2026-06-29-cyclone-remittances-architecture-refactor-design.md
2026-06-29 09:59:53 -06:00
Nora 454c3598b1 docs(spec): design for SP27 remittances architecture refactor
Tier 1: split scheduler.py into handlers/ subpackage, dedup helpers,
loosen INBOUND_RE, add SFTP operation timeouts, surface SFTP errors
in Scheduler.status().

Tier 2: unify 835 ingest + reconciliation into one critical section,
add GET /api/claims/{id}/chain, guard matched_remittance_id ↔
Remittance.claim_id invariant, emit claim.rejected_after_remit audit
when a 277CA rejection hits a matched claim.

Status: Draft, awaiting user sign-off.
2026-06-29 09:54:33 -06:00
Nora f57f1d2875 merge: fix 999 IK5 and TA1 UI into Version-1.0.0 2026-06-29 09:52:19 -06:00
Nora 315fbfec42 fix(docker): wire CYCLONE_SFTP_PASSWORD via docker secret in compose override
Mirrors the existing admin_username / admin_pw pattern so the dev
compose loads SFTP creds from /tmp/cyclone-test-secrets/sftp_password
when present. Falls back to the CYCLONE_SFTP_PASSWORD env var if the
file is missing. Pre-existing env-var support from SP25+26 unchanged.
2026-06-29 09:51:09 -06:00
Nora 6507a8c874 fix(acks): accept IK5 from Gainwell, trust set-level codes over bogus AK9, surface TA1 in UI
Two related fixes land together because the UI was reporting
"1 accepted 1 rejected" for every 999 even though every inbound file
Gainwell ships has IK5=A.

  1. Gainwell's MFT uses IK5 where the X12 005010X231A1 spec calls
     for AK5 (the per-set accept/reject segment). The parser only
     recognized AK5, so set_responses[0].set_accept_reject.code
     defaulted to 'R' and the count summary showed all rejections.
     _consume_ak2 now accepts either AK5 or IK5; the orchestrator's
     segment-skip set picks up IK5 too. A new fixture
     (minimal_999_ik5_gainwell.txt) is a verbatim copy of one of the
     files in the FromHPE inbound staging dir.

  2. _ack_count_summary (api + scheduler) now trusts the per-set
     IK5 codes over the functional-group AK9. Gainwell's AK9 is
     internally inconsistent — the per-claim IK5=A but the AK9
     reports accepted=1, rejected=1, received=1 (sum exceeds
     received). Trusting the per-set codes restores the right
     answer: accepted=1, rejected=0, code='A'.

  3. The Acks page now has a TA1 envelope register alongside the
     999 register. TA1s are the lower-level sibling of the 999
     (one row per inbound ISA/IEA). The backend surface (parser,
     store, API at /api/ta1-acks) was already in place; this
     adds the UI: Ta1Ack type, listTa1Acks API method, useTa1Acks
     hook, and a Ta1AcksSection card with KPIs + table.

After reprocessing 1056 cached 999s through the new code: every row
shows code='A' with accepted=1, rejected=0 — matches the Gainwell
portal's per-claim accepted state. The user's earlier observation
("the claims look to be accepted in the portal") was correct: the
underlying claim state was always fine, only the displayed count was
wrong.

  - backend/src/cyclone/parsers/parse_999.py | 21 ++-
  - backend/src/cyclone/api.py               | 11 +-
  - backend/src/cyclone/scheduler.py         | 13 +-
  - backend/tests/test_parse_999.py          | 32 ++++
  - backend/tests/fixtures/minimal_999_ik5_gainwell.txt
  - src/types/index.ts                       | 32 ++++
  - src/lib/api.ts                           | 62 +++++-
  - src/hooks/useTa1Acks.ts                  | 26 +++ (new)
  - src/pages/Acks.tsx                       | 209 +++++++++++++++++++-
2026-06-25 00:26:13 -06:00
tyler 1381a7652d fix(acks): make 999 source_batch_id unique per file + surface PCN
Gainwell's MFT ships every 999 with the same ISA interchange
control number (`000000001`) and one 999 ack covers a whole
batch, not a single claim — so the AK2 set_control_number
(patient_control_number) is the same for the ~96 999s in a
batch. With the old synthetic-id formula (`999-{icn}`), all 385
daily acks collapsed onto a single row the operator couldn't
distinguish.

The new formula is `999-{pcn}-{filename_hash8}` (or
`999-{icn}-{filename_hash8}` for envelope-only 999s without an
AK2). The PCN gives the operator a human-readable handle to the
claim batch; the 8-char hash of the inbound filename guarantees
uniqueness within a batch. Fits in the VARCHAR(32) source_batch_id
column (max 22 chars).

Also surface `patient_control_number` in /api/acks list response
(extracted from raw_json's set_responses[0].set_control_number)
and in the Acks UI as the primary label, with the synthetic id
shown dimmed after a middle dot. The detail endpoint already
exposed raw_json for the full 999 parse tree.
2026-06-24 23:55:58 -06:00
tyler c3a6c53096 fix(sftp): case-insensitive inbound regex, skip _warn.txt, add targeted pull
Three changes that unblock the daily inbound pull from Gainwell's
FromHPE MFT path:

1. INBOUND_RE now accepts both 'TP' and 'tp' prefixes via inline
   (?i:TP) scoping — case-folding the whole pattern would let
   lowercase '837p' / tracking IDs through, which is invalid HCPF.
   The rest of the pattern is still case-sensitive.

2. _list_inbound_paramiko skips *_warn.txt entries. Gainwell's MFT
   drops ~583 advisory text-format notes in the same inbound dir;
   they come first alphabetically and were padding every poll with
   ~80 min of pointless downloads.

3. New SftpClient.list_inbound_names() + download_inbound() pair
   gives a metadata-only listing and on-demand fetch. The
   scheduler's existing full-listing path still works (now without
   the warn padding); the new path is what the new
   /api/admin/scheduler/pull-inbound endpoint and the
   'cyclone pull-inbound' CLI use to fast-target a date range
   without paying the cost of a full ~6000-file download.

Scheduler.process_inbound_files() runs the same per-file pipeline
as a regular tick on the pre-fetched list, so dedup via
processed_inbound_files still applies.

Tests added in test_filenames.py (lowercase + mixed-case cases),
test_sftp_paramiko.py (warn skip + no-download listing), and
test_scheduler.py (process_inbound_files idempotency).

With this, the daily 385-file pull for 20260624 completes in
seconds via 'docker exec cyclone-backend-1 python -m cyclone
pull-inbound --date 20260624 --block dzinesco' (or the equivalent
POST to /api/admin/scheduler/pull-inbound?date=20260624).
2026-06-24 23:23:46 -06:00
Nora a436538c15 fix(scheduler): read inbound bytes from the cached local_path, not read_file(f.name)
In real (paramiko) mode, `_download_and_parse` called
`client.read_file(f.name)` with a bare filename. paramiko's
`sftp.open(f.name)` opens at the SFTP root, not at `paths.inbound`
(FromHPE) — so the scheduler would fail to download any file in real
mode even with the path swap from the previous commit.

But this round-trip is also unnecessary: `_list_inbound_paramiko`
already downloads each entry into the local cache (cache_path) and
returns it as `InboundFile.local_path` as part of the listing pass.
Reading from disk is faster than re-fetching and avoids the path bug.

Stub mode was already reading from `f.local_path`. Now both modes do,
which is the simpler invariant.

Verified: 36 tests pass (test_scheduler, test_api_scheduler, test_sftp_stub,
test_sftp_paramiko).
2026-06-24 22:15:52 -06:00
Nora dd7da18279 fix(sftp): swap inbound/outbound paths — FromHPE is inbound, ToHPE is outbound
The dzinesco SP9 seed had `paths.inbound` and `paths.outbound` mapped to
the wrong Gainwell MFT directories:

  - paths.outbound  was  FromHPE/   (HPE sends files FROM here TO us — inbound)
  - paths.inbound   was  ToHPE/     (we send files TO here — outbound)

So `/api/clearhouse/submit` was writing 837P claims to FromHPE (where
HPE puts acks/835s) and the SP16 scheduler was polling ToHPE (where our
claims go). 999 / TA1 / 835 files in the real FromHPE inbox were
unreachable.

Semantics per operator (2026-06-24):
  - FromHPE = HPE/Gainwell → us  = 999, TA1, 835  (inbound)
  - ToHPE   = us → HPE/Gainwell = 837P claims    (outbound)

**Runtime code (the actual fix):**
  - backend/src/cyclone/store.py            — SP9 seed paths flipped
  - backend/src/cyclone/edi/filenames.py    — docstring corrected

**Test fixtures + assertions (would otherwise fail on the new seed):**
  - backend/tests/test_clearhouse_api.py
  - backend/tests/test_providers_seed.py
  - backend/tests/test_sftp_stub.py        — incl. inbound dir paths
  - backend/tests/test_sftp_paramiko.py
  - backend/tests/test_store_update_clearhouse.py
  - backend/tests/test_api_clearhouse_patch.py
  - backend/tests/test_scheduler.py
  - backend/tests/test_api_scheduler.py

**Docs (text-only — keeps the codebase self-consistent):**
  - README.md
  - docs/reference/co-medicaid.md
  - docs/superpowers/specs/2026-06-20-cyclone-multi-payer-npi-sftp-design.md

**Operator action required after merge:** the existing clearhouse row in
`~/.local/share/cyclone/cyclone.db` was seeded with the old wrong
paths. Easiest recovery:
  1. `rm ~/.local/share/cyclone/cyclone.db` and let `ensure_clearhouse_seeded` re-run on next boot, OR
  2. PATCH /api/clearhouse with the new `paths` block (the SP25
     reconfigure hook picks it up live, including by the running scheduler).

**Verification:**
  - 82 tests in the affected files pass (test_clearhouse_api, test_providers_seed,
    test_sftp_stub, test_sftp_paramiko, test_store_update_clearhouse,
    test_api_clearhouse_patch, test_scheduler, test_api_scheduler, test_filenames).
  - Full backend suite: 1029 pass + 36 pre-existing order-dependent flakes
    unrelated to this change (verified by running the same tests in isolation).
2026-06-24 22:05:55 -06:00
tyler 9cb0311544 fix(docker): install [sftp] extra so paramiko is in the runtime image
The backend Dockerfile only installed the [sqlcipher] extra, so
paramiko wasn't on the path inside the container. SP25 + SP26's
real-mode SFTP client needs paramiko to connect to the Gainwell
MFT — the scheduler tick surfaced this as
'No module named paramiko' on the very first /api/admin/scheduler/tick
against the dzinesco clearhouse with stub=false.

Install both extras ([sqlcipher,sftp]) in builder and runtime stages.
2026-06-24 18:20:22 -06:00
tyler b7be9f38bc merge: SP25 + SP26 SFTP polling + secret-file lookup into Version-1.0.0
Bring the SP25 SFTP Polling Enablement (PATCH /api/clearhouse,
scheduler hot-reload, env-var-first secret lookup) and SP26
SFTP Password File Companion (Docker secret _FILE tier) onto
the v1.0.0 release branch so the production stack can actually
flip the Gainwell MFT poll from stub → real.

Conflict resolved in docker-compose.yml: kept both the SP26
CYCLONE_SFTP_PASSWORD_FILE env var and the new 'Always bind
to 0.0.0.0' comment.
2026-06-24 18:09:32 -06:00
tyler 1b74af4d3a fix(permissions): populate matrix for all gated endpoints
The matrix_gate dependency was added to ~50 routes (clearhouse, all
/api/admin/*, /api/config/*, /api/eligibility/*, /api/parse-999/ta1/277ca,
/api/batches/{id}/export-837, /api/payer-rejected/acknowledge, etc.)
but the PERMISSIONS matrix was only populated for ~25 of them. Default-
deny therefore returned 403 to every authenticated role on the rest,
including the SFTP admin endpoints needed for MFT polling.

Roles:
- Admin-only: /api/clearhouse* (SFTP creds + dzinesco identity), all
  /api/admin/* (audit-log, backup, scheduler, db rotate, reload-config,
  validate-provider)
- All authenticated: /api/batch-diff, /api/config/*, /api/payers/*
- Write (admin + user, no viewer): /api/parse-999/ta1/277ca, the
  /api/batches POST prefix (regenerates X12 from DB rows), and
  /api/eligibility/* (270 build / 271 parse)

Unblocks /api/clearhouse, /api/admin/scheduler/*, /api/admin/backup/*,
and the rest of the admin surface so MFT polling and live verification
can proceed.
2026-06-24 17:58:31 -06:00
tyler b80e40e7e9 fix(permissions): expose GET /api/acks, /api/ta1-acks, /api/277ca-acks
The PERMISSIONS matrix only listed the POST /api/acks entry (parse-999
ingest). The GET list + detail endpoints for 999 / TA1 / 277CA acks
were missing, so the matrix_gate (default-deny) returned 403 to every
authenticated role, breaking the 999 ACKs / TA1 / 277CA inbox pages.

Add the three GET entries as ALL_ROLES — they're read-only metadata
surfaces that every authenticated operator needs to see. Add a
regression test that logs in as admin via the public route and
exercises the matrix; the existing test_existing_endpoints_require_auth
suite only checks the unauthenticated 401 path, which is why this slipped
through.

Fixes the live-verification 'Couldn't load ACKs from the backend /
forbidden' error reported against the UI.
2026-06-24 17:51:44 -06:00
tyler edca04868d feat: always bind to 0.0.0.0
Reachability is controlled by the host firewall / compose port
publishing, not the bind address. Backend, compose, and docs all
now default to 0.0.0.0 so the API is reachable from the frontend
container, the host, and the LAN without per-env overrides.

Files:
- backend/src/cyclone/__main__.py: default host = 0.0.0.0
- docker-compose.yml: refresh the comment to match the new posture
- CLAUDE.md / README.md / docs/ARCHITECTURE.md / docs/REQUIREMENTS.md:
  reframe the bind note accordingly
2026-06-24 17:40:18 -06:00
Nora 74aa64f995 merge: SP26 SFTP Password File Companion into main 2026-06-24 16:11:21 -06:00
Nora 4ef09f9416 test(sp26): assert CYCLONE_SFTP_PASSWORD_FILE + cyclone_sftp_password in compose 2026-06-24 16:09:19 -06:00
Nora 3c907d17e9 docs(sp26): RUNBOOK — CYCLONE_SFTP_PASSWORD_FILE row + Docker variant 2026-06-24 16:08:12 -06:00
Nora 675faf9844 feat(sp26): wire cyclone_sftp_password secret into docker-compose.yml 2026-06-24 16:07:32 -06:00
Nora 1e8217ff84 feat(sp26): secrets.get_secret() _FILE-tier lookup for Docker secrets 2026-06-24 16:06:54 -06:00
Nora 3075955826 docs(plan): SP26 SFTP Password File Companion — implementation plan 2026-06-24 16:03:51 -06:00
Nora 5b4c1513b6 docs(spec): SP26 SFTP Password File Companion — initial design 2026-06-24 15:52:19 -06:00
Nora 4e580ffe0e merge: SP25 SFTP Polling Enablement into main 2026-06-24 15:42:59 -06:00
Nora 13a3e7c334 docs(sp25): RUNBOOK.md — enable SFTP polling for real Gainwell MFT 2026-06-24 15:38:35 -06:00
Nora 439a3c3d4b feat(sp25): PATCH /api/clearhouse with hot-reload 2026-06-24 15:38:14 -06:00
Nora 5db764d50e feat(sp25): scheduler.reconfigure_scheduler() hot-reload helper 2026-06-24 15:36:19 -06:00
Nora 2d3667738d feat(sp25): store.update_clearhouse() for PATCH endpoint 2026-06-24 15:30:53 -06:00
Nora b2d88d13d3 feat(sp25): secrets.get_secret() env-var-first lookup 2026-06-24 15:29:52 -06:00
Nora 9fda46b891 docs(plan): SP25 SFTP Polling Enablement — implementation plan 2026-06-24 15:28:54 -06:00
Nora d401904412 docs(plan): SP25 SFTP Polling Enablement — implementation plan 2026-06-24 15:25:37 -06:00
Nora 7d2000ec31 docs(spec): SP25 SFTP Polling Enablement — initial design 2026-06-24 15:25:20 -06:00
Nora 1860782ad6 chore(release): bump to 1.0.0
Marks v1.0.0 launch. Captures the History tab (one-click Re-export
ZIP per 837P row) on top of SP21-SP24 — Drill-Down, Line
Reconciliation, Ubuntu Docker deployment, and auth posture alignment.
2026-06-24 14:02:39 -06:00
Nora ce37c10c06 merge: History tab on Upload page into main 2026-06-24 13:57:16 -06:00
Nora f4bafc1c94 feat: History tab on Upload page with one-click Re-export ZIP
Add a second tab to the Upload page that surfaces the persisted batch
archive and lets the user re-download any 837P batch as a ZIP without
re-parsing the original file.

- Backend: /api/batches now carries per-row claimIds (837P only).
  835 batches return an empty list, which the UI uses as the signal
  to hide the Re-export button on those rows. Avoids an extra
  round-trip to /api/batches/{id} per row.
- Frontend: BatchSummary.claimIds added to the list-endpoint type.
- Upload page: page body wrapped in Tabs.Root with a History trigger
  that mirrors ?tab= in the URL for deep-link round-trip. The
  History tab renders UploadHistory → HistoryTable → HistoryRow with
  a one-click Re-export ZIP button per 837P row. The button calls
  POST /api/batches/{id}/export-837 with the row's claim ids and
  downloads the ZIP via downloadBlob. Falls back to the in-memory
  parsedBatches store when the backend returns no rows so the tab
  stays useful in sample-data mode.
- Backend tests: claimIds present on 837P rows, empty on 835 rows.
- Frontend tests: 13 tests covering tab switching, URL deep-link,
  loading/error/empty states, the 837P-vs-835 button visibility
  split, the Re-export happy path, and the failure toast.
2026-06-24 13:57:12 -06:00
Nora 24fbf945c9 fix: allow credentials on CORS for Vite dev server
The dev-server allow-list (VITE_DEV_ORIGINS = localhost:5173 +
127.0.0.1:5173 + CYCLONE_ALLOWED_ORIGINS) is tight enough that
`allow_credentials=True` is safe — the browser was dropping the
session cookie on cross-origin fetches from the Vite dev server
otherwise. Tight allow-list + credentials is the standard setup.
2026-06-24 09:14:32 -06:00
Nora 07a7ecbdd4 merge: SP23 Ubuntu Docker Deployment into main 2026-06-23 20:34:38 -06:00
Nora 5334646992 docs(plan): SP23 — record live verification results
Appends a 'Live verification' section to the plan with the six bugs
the live bring-up surfaced + their fixes, and the end-to-end smoke
results (login, /api/auth/me, /api/parse-837 with a real 837P,
full test suite 1026 passed).
2026-06-23 17:55:54 -06:00
Nora aecf831f43 test(sp23): live bring-up test uses override + skips on port conflict
Two test-only fixes after running test_compose_up_brings_up_healthy_stack
on a non-CI host for the first time:

1. The test only used -f docker-compose.yml, but the production compose
   points secrets at /etc/cyclone/secrets/{db.key,admin_username,
   admin_pw} — which requires sudo to create. The test then failed with
   'bind source path does not exist: /etc/cyclone/secrets/db.key' even
   though the stack itself was correctly configured.
   Fix: if docker-compose.override.yml exists at the repo root, the
   test uses `-f compose.yml -f override.yml` so secrets come from
   /tmp/cyclone-test-secrets/. Production CI skips the override.

2. The stack publishes host port 8080. If another local service (e.g.
   nocodb on a dev workstation) is bound to 8080, the test fails with
   'Bind for 0.0.0.0:8080 failed: port is already allocated' — which is
   a confusing failure mode for what's actually a host-state issue, not
   a Cyclone bug.
   Fix: probe 127.0.0.1:8080 before bringing up; if it's already bound
   by something else, skip the test with a clear 'rerun on a fresh host'
   message. CI workers don't have this conflict.

Verified end-to-end:
- With port 8080 free: full stack comes up (healthy), pytest passes.
- With port 8080 bound: pytest skips cleanly with the message above.
2026-06-23 17:55:41 -06:00
Nora 3ba5ca0849 feat(sp23): live-verification fixes from end-to-end bring-up
After the 8-commit SP23 implementation landed, kicking the tires on
`docker compose build && docker compose up -d` (the gated DOCKER_TESTS=1
live test) surfaced five real bugs that don't show up in unit tests:

1. Backend wheel was built from the stub `__init__.py`, not the real
   source. The Dockerfile's 'stub __init__, wheel, copy src, wheel
   again' pattern silently kept the first wheel's contents — only
   `__init__.py` got re-stubbed. The installed package had an empty
   `__init__.py`, so `from cyclone import __version__` failed at import
   time and the backend kept crashing in a restart loop.
   Fix: single `COPY src/` + single `pip wheel`. Comment explains
   why the stub trick is gone for good.

2. Backend binds to 127.0.0.1 (intentional — local-only by design,
   see CLAUDE.md). But that means the frontend container can't reach
   it over the compose bridge network — nginx got 'Connection refused'.
   Fix: `CYCLONE_HOST` env var, defaults to 127.0.0.1 (preserves local
   posture for non-Docker runs), set to 0.0.0.0 by the docker-compose
   backend service. Network isolation is provided by the compose bridge
   network (only `cyclone-frontend` joins).

3. Healthcheck probed `/api/healthz` (404 — the route is `/api/health`).
   Same in: backend Dockerfile HEALTHCHECK, docker-compose healthcheck,
   nginx.conf doesn't have one (frontend proxies through), RUNBOOK.md,
   scripts/post-deploy.sh, scripts/smoke.sh.
   Fix: `/api/healthz` → `/api/health` everywhere SP23 owns.

4. The auth matrix in `cyclone.auth.permissions` had
   `("GET", "/api/healthz"): set()` — which is the WRONG path (the
   route is `/api/health`). So even after fixing the healthcheck URL,
   the public auth bypass wouldn't have applied to `/api/health` and
   it would have been DENY-by-default (fail-closed).
   Fix: matrix entry updated to `/api/health`.

5. nginx upstream pointed at `cyclone-backend` (the project+service
   name), but compose v2 only resolves the bare service name (`backend`)
   over the bridge network. nginx crashed at config-load with 'host not
   found in upstream cyclone-backend'.
   Fix: `cyclone-backend:8000` → `backend:8000` in nginx.conf + spec
   + plan.

6. Frontend HEALTHCHECK used `http://localhost:8080/`. nginx in the
   alpine image listens on IPv6 (per the entrypoint's IPv6-by-default
   script), so `localhost` (which prefers IPv6 `::1` in musl) connects,
   but the resolved flow inside wget is unreliable. `127.0.0.1` works.
   Fix: HEALTHCHECK uses `http://127.0.0.1:8080/`.

Also moves `frontend/Dockerfile` → `Dockerfile.frontend` and
`frontend/nginx.conf` → `nginx.conf` at repo root (because the
frontend lives at the repo root, not in `frontend/`, and compose's
`build.context: .` needs them at the same root as compose.yml).
The frontend's pre-existing `.dockerignore` was empty/unused, so it's
dropped — the root `.dockerignore` covers it.

Adds `docker-compose.override.yml` for local bring-up testing on a
host without sudo. Production uses `/etc/cyclone/secrets/` directly.

Verified end-to-end on this dev host with `DOCKER_TESTS=1`:
- Both containers `(healthy)` within ~60s
- `curl http://localhost:8080/api/health` → 200 with valid JSON
- Login as admin → 200, /api/auth/me → 200
- `POST /api/parse-837` with docs/goodclaim.x12 → 200, batch created
- Full backend test suite: 1014 passed, 9 skipped (prodfiles gitignored)
2026-06-23 17:53:16 -06:00
Cyclone f7697e58b7 feat(sp23): tests/test_docker.py validates compose config + Dockerfile parse
Eight tests: compose file exists, docker compose config validates, services declare required volumes/restart/healthcheck/ports/depends_on, secrets and volumes tables include the names the operator expects, backend wires CYCLONE_BACKUP_AUTOSTART=1, both Dockerfiles pass . The live bring-up test (compose up + wait for healthy) is gated on DOCKER_TESTS=1 so it skips on bare CI. PyYAML was already a hard dep.
2026-06-23 17:24:46 -06:00
Cyclone 7706a6d7fe feat(sp23): auth bootstrap reads CYCLONE_ADMIN_*_FILE for Docker secrets
Add _read_secret() helper that prefers *_FILE env vars (the standard Docker-secret pattern) over bare env vars. Strips trailing whitespace from file contents so printf/echo newlines don't break bcrypt verify. Existing CYCLONE_ADMIN_USERNAME + CYCLONE_ADMIN_PASSWORD still work for non-Docker deployments. Add tests/test_auth_bootstrap_file.py covering: file-path read, file-overrides-bare, bare-fallback, whitespace-stripping, missing-file raises.
2026-06-23 17:24:23 -06:00
Cyclone 256ddfa5fa feat(sp23): operator scripts (cyclone-init, post-deploy, smoke) + RUNBOOK
cyclone-init.sh generates /etc/cyclone/secrets/{db.key,admin_username,admin_pw} with openssl rand, prints the admin password once. post-deploy.sh installs logrotate.d/cyclone and a 5-minute healthcheck cron. smoke.sh brings up + logs in + parses a sample 837 end-to-end. RUNBOOK.md covers daily/weekly/quarterly/annual ops procedures plus the emergency runbook.
2026-06-23 17:22:44 -06:00
Cyclone d42fbc8c1b feat(sp23): docker-compose.yml with backend + frontend, named volumes, secrets 2026-06-23 17:22:06 -06:00
Cyclone 67dae61a94 feat(sp23): frontend Dockerfile + nginx.conf (SPA + reverse proxy) 2026-06-23 17:22:03 -06:00
Cyclone 59e69127a2 feat(sp23): backend Dockerfile (multi-stage, sqlcipher, non-root, healthcheck) 2026-06-23 17:21:54 -06:00
Cyclone 364e5d7497 docs(plan): SP23 Ubuntu Docker Deployment implementation plan
12 tasks from worktree bootstrap through single-atomic-merge into main. Backend Dockerfile (multi-stage python:3.11-slim-bookworm + sqlcipher + non-root + healthcheck), frontend Dockerfile (multi-stage node:20-alpine build + nginx:1.27-alpine runtime), docker-compose.yml at repo root with two services + named volumes + Docker secrets, scripts/{cyclone-init,post-deploy,smoke}.sh, RUNBOOK.md, tests/test_docker.py (compose config + Dockerfile parse + optional live bring-up), and a small auth bootstrap extension to read CYCLONE_ADMIN_*_FILE for Docker-secret compatibility. Closes the SP23 spec; see spec section 1.1 for what auth is reused from main vs. what SP23 actually adds.
2026-06-23 17:21:34 -06:00
Cyclone 35c561de20 docs(spec): mark SP23 approved and reconcile with auth on main
Auth landed in main on 2026-06-23 (migrations 0013/0014, cyclone.auth package, bcrypt + matrix_gate, Login.tsx + AuthProvider + RoleGate, CYCLONE_AUTH_DISABLED escape hatch). The original draft was written 2026-06-22 and proposed argon2id, admin/operator/viewer roles, a fresh migration 0012, and an inline require_role dependency. All of that is already in main. Add a 1.1 Delta vs. main section and switch role names from operator to user to match cyclone.auth.permissions.Role.
2026-06-23 17:20:04 -06:00
cyclone c398aa7d29 Merge branch 'sp24-doc-posture-alignment'
Align CLAUDE.md + REQUIREMENTS.md + ARCHITECTURE.md with the auth work
that landed on main on 2026-06-23; emit an AUTH_DISABLED WARNING at boot
when AUTH_DISABLED is true so misconfigured production deploys fail loudly.

Closes requirements R-1 (was Open; auth shipped via the origin/main merge,
SP24 reconciles the docs).

* CLAUDE.md — track in git for the first time
* docs/REQUIREMENTS.md + docs/ARCHITECTURE.md — replace every stale
  'no auth' / 'no authentication' / 'no second party to authenticate'
  claim with the v1 posture (bcrypt + HttpOnly session cookie; first
  admin bootstrapped from CYCLONE_ADMIN_USERNAME + CYCLONE_ADMIN_PASSWORD)
* backend/src/cyclone/__main__.py — boot-time WARNING when AUTH_DISABLED
* ARCHITECTURE §4.2 module map — add cyclone.auth.* package
* ARCHITECTURE §6.1 migrations table — list 0013 + 0014 with renumbering note
* .superpowers/skills/{cyclone-tests,cyclone-api-router,cyclone-spec} —
  add AUTH_DISABLED / matrix_gate / auth-aware threat-model guidance

Spec: docs/superpowers/specs/2026-06-23-cyclone-auth-posture-alignment-design.md
Plan: docs/superpowers/plans/2026-06-23-cyclone-auth-posture-alignment.md
2026-06-23 16:28:28 -06:00
cyclone 616d467c65 docs(reviews): capture 2026-06-23 docset review + CSS reduction experiment
Commit the day's review artifacts so they aren't lost in the working tree:

* 2026-06-23-cyclone-docset-review-A.md — full docset review (Reviewer A:
  requirements + architecture lens; flags the 8-state vs 7-state claim
  lifecycle divergence, FR-20 vs ARCHITECTURE §5.4 tail-endpoint conflict,
  api_routers/ package split status, etc.)
* 2026-06-23-cyclone-docset-review-B.md — code-first counterpart (Reviewer B)
* 2026-06-23-cyclone-groundtruth-audit.md — live-data readiness audit
* 2026-06-23-css-reduction/ — iterative CSS reduction experiment
  (baseline 60,695 B raw / 11,786 B gzip / 9,875 B brotli; 18 prioritized
  candidates; per-iter reports and progress log)

Binary artifacts (PNGs, dist build output) are intentionally not committed —
they're working files for the experiment, not review records.
2026-06-23 16:28:15 -06:00
Tyler 2194d35ea1 fix(sp22): correct test fixtures in SP22 migration tests
- test_migration_latest_idempotent_on_fresh_db now uses the pytest
  tmp_path fixture instead of a literal /tmp path that did not exist,
  so sqlite can create the test DB.
- test_drop_claims_unique_constraint_migration now inserts the NOT
  NULL batches.kind / input_filename / parsed_at columns when seeding
  the parent row for the two same-PCN claims.
2026-06-23 16:18:01 -06:00
Tyler 4fd55dc33e feat(sp22): migration 0015 — drop inline UNIQUE on claims, add tests
Adds migration 0015_drop_claims_unique_constraint.sql that recreates the
claims table without the inline UNIQUE(batch_id, patient_control_number)
constraint, plus tests proving the migration runs cleanly and the
constraint is not re-introduced.
2026-06-23 16:11:42 -06:00
Tyler 00f11f84b6 feat(sp24): align CLAUDE.md + REQUIREMENTS.md + ARCHITECTURE.md with auth work + AUTH_DISABLED WARNING
Replaces every stale 'no auth' / 'no authentication' / 'no second party to
authenticate' claim in the three top-level docs with the v1 posture: the
auth boundary is HTTP (bcrypt + HttpOnly session cookie; first admin
bootstrapped from CYCLONE_ADMIN_USERNAME + CYCLONE_ADMIN_PASSWORD env
vars); the file-system posture (SQLCipher at rest, macOS Keychain) is
unchanged; the threat model is still a stolen/imaged drive.

Closes requirements §11 R-1 (was Open; now Closed by SP24 — auth shipped
via the origin/main merge on 2026-06-23, SP24 reconciled the docs).

Also:
- backend/src/cyclone/__main__.py — emit WARNING when AUTH_DISABLED is
  True at boot so a misconfigured production deploy fails loudly
- §6.1 migrations table in ARCHITECTURE.md — list 0013 (auth_users_and_
  sessions) and 0014 (audit_log_user_id) with the renumbering note
- §4.2 module map — add the cyclone.auth.* package
- Three skills addenda: cyclone-tests (AUTH_DISABLED conftest bypass is
  mandatory context), cyclone-api-router (every router needs
  Depends(matrix_gate)), cyclone-spec (spec template threat-model is now
  auth-aware)

This also tracks CLAUDE.md in git for the first time (was previously
untracked; the SP24 doc updates are in scope for the increment).

Spec: docs/superpowers/specs/2026-06-23-cyclone-auth-posture-alignment-design.md
Plan: docs/superpowers/plans/2026-06-23-cyclone-auth-posture-alignment.md
2026-06-23 14:49:31 -06:00
Tyler d294b8bbed docs(plan): SP24 — auth posture alignment implementation plan
Plan covers the docs-only reconciliation between the auth work that landed
in main on 2026-06-23 and the three top-level docs (CLAUDE.md, docs/
REQUIREMENTS.md, docs/ARCHITECTURE.md). Includes a 6-line __main__.py edit
for the AUTH_DISABLED startup WARNING and one-paragraph addenda to the
cyclone-tests / cyclone-api-router / cyclone-spec skills.

Spec: docs/superpowers/specs/2026-06-23-cyclone-auth-posture-alignment-design.md
2026-06-23 14:49:26 -06:00
Tyler 9e595503b7 docs(spec): SP24 — align docs and decisions with the auth work in main
The auth-merge that landed on 2026-06-23 (14 commits, a25504b..39ae988)
adds real auth: User/Session models, bcrypt, matrix_gate, AUTH_DISABLED,
a login router, RBAC on every existing endpoint, and a RequireAuth
guard on the React shell. But CLAUDE.md, docs/REQUIREMENTS.md, and
docs/ARCHITECTURE.md (all added/updated on the css-reduction branch)
still describe the pre-auth "no auth, local-only, single operator,
single host" posture. The doc/code divergence is real and would
confuse anyone reading the docs.

This is the spec for SP24 — the alignment. It is intentionally a
docs/skills-only SP: no schema, no role, no gate changes. The plan
section enumerates the doc lines to update; the implementation step
sweeps the markdown for residual "no auth" claims.

Draft, awaiting user sign-off per the SP-N spec flow.
2026-06-23 14:49:21 -06:00
Tyler 1148a03a43 docs(architecture): add top-level technical design doc
Closes round 4 of doc-prep. ARCHITECTURE.md is the day-1 read for
engineers joining Cyclone: process & network topology, tech stack
(backend + frontend + external), backend package layout with
one-line module responsibility map, frontend architecture (route
map, state management, live tail), data model (migrations 0001-0012,
ERD, 7-state claim lifecycle, audit chain), data flow (manual upload,
SFTP polling, auto-match, outbound 837P, ack ingestion), API surface,
cross-cutting concerns (logging, audit, PHI/PII, error model),
operational concerns (startup order, schedulers, health probe,
shutdown), deployment, out-of-scope, and references.

759 lines, 13 top-level sections. Cross-linked from REQUIREMENTS.md
sister-docs block and from README.md Roadmap section.
2026-06-23 14:49:16 -06:00
Tyler cad104a38c docs(requirements): add top-level requirements + DoD
38 functional requirements + 18 non-functional requirements across 13 sections.
Includes traceability matrix mapping FRs to NFRs and to existing
docs/superpowers/specs/* artifacts. Closes round 1 of doc-prep
preparation for Cyclone implementation.
2026-06-23 14:49:12 -06:00
Nora 0c81968d0c fix(permissions+seed): add /api/reconciliation routes and shape billing_provider.address as dict
Two bugs surfaced as 403/500 storms in the live UI:

1. PERMISSIONS matrix only had '/api/reconcile' as a prefix but the
   client hits '/api/reconciliation/*' (dashboard reconciliation tab,
   manual match endpoint). Every call returned 403 even for admin.
   Add ('GET', '/api/reconciliation') and ('POST', '/api/reconciliation')
   alongside the existing /api/reconcile entries.

2. The seed CLI built billing_provider.address as a flat string but
   to_ui_claim_detail._address_to_ui expects {line1,line2,city,state,zip},
   so opening any seeded claim in the detail drawer threw
   'str object has no attribute get' → 500. Build the dict from the
   provider fields instead.
2026-06-22 17:54:39 -06:00
Nora add6e982a4 Populate receivedAmount from matched Remittance.total_paid
Before this change, every claim — matched or not — came back from
`/api/claims` with `receivedAmount: 0.0`. `to_ui_claim_from_orm`
hardcoded the value, so the Dashboard's 'Received' KPI and the
claim detail drawer's paid amount always read $0 regardless of
whether the claim had been paired with a paying remittance.

The fix threads the real value through the read path:

- `to_ui_claim_from_orm` accepts a new `received_total: float`
  kwarg (default 0.0 for callers that don't have a remittance in
  scope).
- `iter_claims` bulk-loads `Remittance.total_paid` for every
  matched claim id in the result set (single SQL roundtrip via
  `IN` filter — no N+1) and stamps the sum onto each claim dict.
- `manual_match` passes `float(remit.total_paid)` from the remit
  it's already holding.
- `manual_unmatch` reads `paired_remit.total_paid` before clearing
  the FK so the response still reports what was paid pre-unpair.
- `add` (write path) and `list_unmatched` (filters matched-remit
  is NULL) pass `0.0` explicitly for readability.

The dev seed CLI now also generates matched Remittance rows for
every PAID/PARTIAL claim — `matched_remittance_id` is set on the
claim, and the remittance's `total_paid` is derived from the
billed amount using the same ratios the frontend's old sample
fixtures used (60–100% for PAID, 20–50% for PARTIAL). That gives
the Dashboard a non-zero 'Received' KPI on a fresh dev DB.

New tests:
- tests/test_iter_claims_received.py — 4 cases covering matched,
  unmatched, orphan FK, and bulk-load paths. Catches regressions
  if anyone re-introduces the hardcoded 0.0 or breaks the bulk
  query.
2026-06-22 17:46:47 -06:00
Nora 414d2eb722 Add cyclone seed CLI subcommand for dev DB population
Now that Dashboard reads live API hooks instead of the in-memory
sample store, a fresh dev DB renders as $0 KPIs and 'No activity
yet.' for a long time. `python -m cyclone seed` inserts a
deterministic batch of 96 claims + 28 activity events so the
Dashboard / Claims / Activity Log pages have something to render
without going through the Inbox / EDI parser pipeline.

  python -m cyclone seed           # insert (no-op if seeded)
  python -m cyclone seed --reset   # wipe + re-insert
  python -m cyclone seed --count 10
  python -m cyclone seed --status  # counts only

The seed mirrors the frontend `sampleData.ts` fixtures (3 TX
providers, 7 payers, 8 CPTs, 96 claims spread across the last 200
days) so dev dashboards look identical to the previous in-memory
fixture mode. `raw_json` and `payload_json` are populated in the
shape that `store.to_ui_claim_from_orm` / `recent_activity` parse
back into the UI wire format — wire parity, not shortcuts.

`--reset` cleans by id prefix (`SEED-` / `CLM-S`) rather than
relying on SQLite's `ON DELETE CASCADE`, which the dev session
doesn't enforce (no `PRAGMA foreign_keys = ON` at the SQLAlchemy
session layer). Safer than silent orphans.

Verified end-to-end: 986 backend tests pass, dashboard renders
$75,833 / 96 claims / 11.5% denial rate / 10 activity events /
top-3 providers ranked by claim count.
2026-06-22 17:30:39 -06:00
Nora 7e4bb4d2c8 Wire Dashboard to live API hooks
Dashboard.tsx was reading claims/providers/activity directly from the
zustand `useAppStore`, which returns the hardcoded sample fixtures
(sampleClaims / sampleProviders / sampleActivity) regardless of whether
a backend session is active. The hooks useClaims / useProviders /
useActivity are wired correctly to /api/* when api.isConfigured, so
the fix is to swap the three direct store reads for the hooks.

buildMonthly's parameter is now typed as Claim[] (the live Claim
shape) instead of the store's heterogeneous slice, and the
`useAppStore` import is gone.

Verified end-to-end: after logging in as admin via the live server,
the Dashboard now shows $0 KPIs, "0 events / No activity yet.",
and the live operator name in the greeting — proving the hooks are
hitting /api/* and not falling back to the sample store.
2026-06-22 17:20:48 -06:00
Nora fb913f0617 fix(ui): Dashboard greeting uses live operator; api.isConfigured proxies through
Two issues surfaced by going live against the dev backend:

1. Dashboard greeting was hardcoded 'Good morning, Jordan.' in
   src/pages/Dashboard.tsx. The sidebar already pulls from useAuth()
   but the dashboard had been missed. Replace with a time-of-day
   greeting ('morning' / 'afternoon' / 'evening') plus the live
   operator's username from useAuth(). Falls back to 'there' while
   /api/auth/me is in flight so we never flash 'undefined' at the
   operator.

2. api.isConfigured was returning false whenever VITE_API_BASE_URL
   was empty, which is the default in .env.example and the
   recommended setting for the Vite dev proxy / nginx. With the flag
   false, every hook (useClaims, useBatches, useActivity, ...)
   threw notConfiguredError() and silently fell back to the
   in-memory zustand store's hardcoded sample data — so 'files not
   serving' was actually the Dashboard rendering fixtures, not the
   real DB. With the flag now true, joinUrl('') produces relative
   URLs that the proxy intercepts and forwards to FastAPI on :8000.
   Set VITE_API_BASE_URL=disabled to opt out (useful for Storybook
   / offline UI work).

Also installs @radix-ui/react-tabs which was listed in package.json
but missing from node_modules — Vite was failing pre-transform on
tabs.tsx and refusing to compile downstream pages until deps were
re-installed via 'npm install'.
2026-06-22 17:05:27 -06:00
Nora 39ae988101 fix(auth): bootstrap init_db on fresh DB; permissions matrix for inbox endpoints
Two bootability gaps caught by running the live stack:

1. cyclone.auth.bootstrap.run() called SessionLocal() before db.init_db()
   had run on a fresh DB, so 'python -m cyclone serve' / 'users create'
   / '--help' all crashed with 'db.init_db() has not been called'.
   Add init_db() inside bootstrap (idempotent — no-op once the schema
   is current) so first-boot works without manual prep.

2. PERMISSIONS matrix didn't register the inbox endpoints that were
   added in SP10 (export.csv) and SP14 (candidates/dismiss,
   rejected/resubmit, payer-rejected/acknowledge). The matrix's
   longest-prefix match meant /api/inbox/candidates/dismiss matched
   the '/api/inbox' prefix but only because that single entry
   existed; adding the dedicated write-prefix entries makes the
   intent explicit and keeps viewer role out of every inbox write
   even if the broader /api/inbox prefix is later broadened.

Verified live:
- admin can login, GET /api/inbox/lanes (200), GET
  /api/inbox/export.csv (200), POST /api/inbox/candidates/dismiss
  (200).
- viewer can login, GET /api/inbox/lanes (200), GET
  /api/inbox/export.csv (200), but POST /api/inbox/candidates/dismiss
  and POST /api/inbox/rejected/resubmit both return 403.
2026-06-22 17:00:30 -06:00
Nora e2d4a595a4 fix(api): use request.app.state in inbox endpoints; preserve dict HTTPException detail
- inbox_lanes, inbox_dismiss_candidates, inbox_export_csv: switch
  module-level 'app' to per-request 'request.app' so per-endpoint
  state stays consistent with the test's TestClient target across
  importlib.reload() (test_api.py::test_cors_extra_origins_via_env
  reloads the api module to mutate CORS allow-lists; pre-reload
  endpoints then mutate the wrong app instance).

- _http_exc_handler: when HTTPException.detail is a dict, wrap under
  'detail' so the standard envelope stays stable and callers can
  branch on body['detail']['error'].

- conftest._auto_init_db: re-resolve cyclone.api.app each fixture
  invocation (instead of caching at module load) so the reload pattern
  doesn't leave event_bus set on a stale app.

- test_acks.test_migration_latest_idempotent_on_fresh_db: bump
  user_version assertion from 12 to 14 after auth migration renumber
  (0013 users+sessions, 0014 audit_log.user_id).

Also installs sqlcipher3 + paramiko into the backend venv so the
capability tests can run; both modules were optional deps that
test_db_crypto and test_sftp_paramiko assume are present.

Backend test results: 1008 passed, 9 skipped (gitignored prodfile
fixtures), 0 failed.
2026-06-22 16:55:55 -06:00
Nora 81bcb1c1ef fix(db): renumber auth migrations to come after SP14/12-backups
The auth cherry-picks brought in 0010_auth_users_and_sessions.sql
(version: 10) and 0011_audit_log_user_id.sql (version: 11), which
collide with main's 0010_payer_rejected_acknowledged.sql (version: 10)
and 0011_processed_inbound_files.sql (version: 11). The migration
runner is filename-sorted but uses the -- version: N header to gate
'application vs skip', so the second file at each version gets
silently skipped — claims.payer_rejected_acknowledged_at never
got ALTERed, breaking every test that touches the claims table.

Renumber to 0013 and 0014 (and bump the -- version headers to
match) so they apply after 0012_backups.sql and main's existing
SP14 / SP15 migrations.
2026-06-22 16:07:10 -06:00
Nora 8fc3d9adda test(auth): add test_existing_endpoints_require_auth + per-test AUTH_DISABLED flips
- New test_existing_endpoints_require_auth.py: spot-check that existing
  /api/* endpoints now require auth (gated via Depends(matrix_gate)) when
  AUTH_DISABLED is False. Health remains public.

- conftest.py: flip AUTH_DISABLED=True for the suite so the legacy
  pre-auth tests keep passing without login. Auth tests flip it back
  off via their own autouse fixture (now patched to use monkeypatch
  for cleanup).

Verified: 53 auth tests pass; 222 pre-existing non-auth failures are
unchanged.
2026-06-22 15:59:08 -06:00
Nora 35e0f5a422 feat(auth): disable write affordances for viewer role
Re-apply f91d7b3 manually against current Upload.tsx / Inbox.tsx /
Reconciliation.tsx since main's UI text diverged from the original
cherry-pick (Release to 'parse' span, mono uppercase tracking, etc.).
The write affordances are now wrapped in <RoleGate allow={admin,user}>:

  - Upload.tsx: dropzone + Parse/Clear buttons
  - Inbox.tsx: rejected / payer_rejected / candidates BulkBars
  - Reconciliation.tsx: 'Match selected' button

Test fixtures stub useAuth to return an admin user so RoleGate lets
the BulkBars / Match button render synchronously on first paint —
the tests don't need a full AuthProvider + /api/auth/me probe.
2026-06-22 15:54:38 -06:00
Nora a25504bd3a fix(auth): ungate auth_router so login is public 2026-06-22 15:41:52 -06:00
Nora 70280f70bb feat(auth): gate existing endpoints with matrix_gate (router-level + per-endpoint)
Replaces the skipped 63ae0d2 commit: adds matrix_gate to deps.py and
applies it as a dependency on every authenticated FastAPI route, including
extracted routers (acks, ta1_acks, admin) and inline endpoints in api.py.

The PERMISSIONS matrix in auth/permissions.py controls which roles can
hit which (method, path) combinations; matrix_gate is fail-closed by
default (any endpoint not in the matrix returns 403).
2026-06-22 15:40:00 -06:00
Nora a933672411 docs: auth section + env vars 2026-06-22 15:38:38 -06:00
Nora ca40fd72c3 feat(docker): CYCLONE_ADMIN_* env vars for backend 2026-06-22 15:38:38 -06:00
Nora 4402a993d1 feat(auth): sidebar shows current user + logout button 2026-06-22 15:37:02 -06:00
Nora cb87456575 feat(auth): wire AuthProvider + RequireAuth into app shell 2026-06-22 15:37:02 -06:00
Nora 3d0c7766f0 feat(auth): RequireAuth route guard 2026-06-22 15:34:19 -06:00
Nora e76b514872 feat(auth): /login page 2026-06-22 15:34:19 -06:00
Nora 5bb9588f09 feat(auth): RoleGate component 2026-06-22 15:34:19 -06:00
Nora d895854dcc feat(auth): AuthProvider context + useAuth hook 2026-06-22 15:34:18 -06:00
Nora 55a298f05f feat(auth): fetch wrapper with 401 redirect + authApi 2026-06-22 15:34:18 -06:00
Nora 0aea0f64ac feat(types): add User type 2026-06-22 15:34:18 -06:00
Nora 6c80bf0512 feat(auth): CLI users subcommand 2026-06-22 15:34:18 -06:00
Nora 9c57b493a7 feat(audit): record user_id on log events 2026-06-22 15:34:18 -06:00
Nora 609499543e test(auth): login rate limit 2026-06-22 15:34:05 -06:00
Nora 768f7c6247 feat(auth): bootstrap first admin from env vars 2026-06-22 15:34:05 -06:00
Nora 86b635104c feat(auth): get_current_user + login/logout/me + admin user management 2026-06-22 15:34:05 -06:00
Nora e158871a9a feat(auth): sessions module + permissions matrix + rate limiter
SQLite drops tzinfo on DateTime roundtrip — normalize on read/write
so callers see tz-aware datetimes.
2026-06-22 15:34:05 -06:00
Nora 1ca50e2bc0 test(auth): tighten duplicate-username test + add update_password test 2026-06-22 15:32:56 -06:00
Nora 74d7056284 feat(auth): users module with bcrypt hashing + CRUD 2026-06-22 15:32:56 -06:00
Nora 0ba91040f1 feat(migration): 0010 users + sessions, 0011 audit_log.user_id 2026-06-22 15:32:56 -06:00
Nora dc83d7bef2 feat(db): add User and Session models 2026-06-22 15:32:56 -06:00
Nora 9d4798a124 feat(deps): add passlib[bcrypt] for password hashing
passlib 1.7.4 + bcrypt >= 4.1 are incompatible (passlib probes
bcrypt.__about__ which 4.x removed). Pin bcrypt<4.1.
2026-06-22 15:32:56 -06:00
Nora d552d3d3ec docs(plan): auth implementation plan — 29 tasks, ~40 files 2026-06-22 15:32:45 -06:00
Nora 42a826cb73 docs(spec): auth (admin/user/viewer) design — sessions, RBAC, login flow 2026-06-22 15:32:45 -06:00
Tyler 05b43078b9 chore(deps): consolidate dev deps into [project.optional-dependencies] dev
Follow-up to 7c1be58. pytest-randomly was added to a PEP 735
[dependency-groups] dev block by 'uv add --dev', leaving the
project with two 'dev' groups (the legacy [project.optional-
dependencies] dev and the new [dependency-groups] dev). To get
the full dev env someone had to run 'uv sync --extra dev --group
dev', which is easy to miss.

This commit moves pytest-randomly>=4.1 into the
[project.optional-dependencies] dev list (where pytest,
pytest-cov, pytest-asyncio, and httpx already live) and drops
the [dependency-groups] block. 'uv sync --extra dev' now
installs the full dev toolchain. uv.lock updated accordingly:
the [package.dev-dependencies] and [package.metadata.requires-
dev] sections are gone, and pytest-randomly is part of the
standard dev extra.
2026-06-22 11:44:39 -06:00
Tyler 7c1be58860 chore(deps): add cryptography dep, pytest-randomly dev dep
Two follow-ups to the v0.2.0 release:

1. Add cryptography>=49.0,<50 to the [project.dependencies] list.
   cyclone.backup / cyclone.backup_service import it at module top
   level, so it has to be a hard dep — not an extra — or the test
   suite fails to collect on a fresh 'uv sync'. A clean install
   (without this commit) gets 28 collection errors with no tracked
   file changes. This was latent because the original 99M venv
   had cryptography installed out-of-band and masked the missing
   dep declaration.

2. Add pytest-randomly>=4.1.0 to the dev dep group.
   Used to characterize the test suite's order-dependence: under
   alphabetic order the suite shows 7 failures; under seed=12345
   it shows 34. That variance is evidence of test-order state
   interactions, not venv issues. Keeping pytest-randomly in
   the dev deps so this can be re-verified on demand.

Note: pytest-randomly was added with 'uv add --dev', which uses
the PEP 735 [dependency-groups] syntax. pyproject.toml now has
two 'dev' groups — the legacy [project.optional-dependencies] dev
= [pytest, pytest-cov, pytest-asyncio, httpx] and the new
[dependency-groups] dev = [pytest-randomly]. To get the full dev
env, run: uv sync --extra dev --group dev. Worth consolidating
in a follow-up.
2026-06-22 11:43:32 -06:00
Tyler 9bca4b608a feat(release): v0.2.0 — batch 837 export, ClaimCard, theme tokens
Backend:

- New POST /api/batches/{id}/export-837: regenerate X12 837 files
  for a list of claim_ids into a ZIP using HCPF file naming standards,
  with a unique interchange/group control number per export. Wire
  the clearhouse Loop 1000A (NM1*41 + PER) and per-payer receiver
  (NM1*40) blocks so the serializer no longer falls back to
  CYCLONE / RECEIVER placeholders.
- /api/parse-837 and /api/parse-835 now surface the server-side
  batch_id in both JSON and NDJSON response shapes so the frontend
  can hit batch-scoped endpoints without an extra listBatches
  round-trip.
- Filename helpers and the 837 serializer updated to match the new
  HCPF envelope; tests cover batch export, parse batch_id, and the
  serializer's control-number uniqueness guarantee.

Frontend:
- New shared components: ClaimCard, ClaimCard837, DominantKpiCard,
  EditorialNote, ExportBar, TickerTape, and a charts/ set
  (BarChart, HBarChart, SegmentedBar, AgingBars).
- New useBatchExport hook driving ExportBar's download flow against
  the new endpoint.
- ClaimDrawer, Lane, and Layout migrated from raw CSS-variable
  colors to Tailwind theme tokens (bg-card, text-foreground,
  border/60, etc.) for consistency with the rest of the instrument
  chrome; the active tab indicator gains a subtle accent glow.
- Upload, Inbox, Batches, BatchDiff, Reconciliation, and Acks pages
  reworked to compose the new shared components and consume the new
  batch-scoped API surface (notably ExportBar wired into Batches).

Tooling / Docs:
- Add audit-uiux.mjs and a docs/goodclaim.x12 sample fixture.
- Update ClaimDrawer testids and add coverage for the new
  components and the useBatchExport hook.

Rolls up into the v0.2.0 release tag.
2026-06-22 11:01:58 -06:00
cyclone 35298907bc docs(spec): add Ubuntu Docker deployment design
Production deployment of Cyclone on a single Ubuntu Linux server:
- Two-container Docker Compose (backend + nginx frontend)
- App-layer auth: username/password, argon2id, session cookies
- 3 fixed roles: admin / operator / viewer with documented action matrix
- SQLCipher encryption at rest, key as Docker secret
- Existing BackupService autostarts (24h interval, 14-day retention)
- LAN-only bind, single operator, VPN for outside access
- Manual update via 'docker compose pull', env-var rollback
- SFTP stays stub (download ZIP + MFT UI); listed as v2

Builds on the parent 2026-06-19 production-readiness spec which
shipped local-only in-memory store + react-query wiring.
2026-06-22 10:43:26 -06:00
Tyler 22720168c4 test(claims): wrap page in DrillStackProvider, switch to DrillDrawerHeader testids
Phase 5 Task 5.8 (PartiesGrid) and 5.9 (ValidationPanel) added useDrillStack()
calls, so Claims.test.tsx needs a DrillStackProvider wrapper around the
page render to keep the hook happy. Task 5.10 refactored the ClaimDrawer
header onto the shared DrillDrawerHeader, so the deep-link and close-button
tests need to find the title via <h2> and the close button via
aria-label='Close drawer' instead of the now-removed header-id /
header-close testids.

Without this fix the page-level Claims tests report 3 failures
(deep link, URL clear after close, ?-mark with drawer closed) that
were not failing on the Phase 4 base (9a313d2).
2026-06-21 18:10:43 -06:00
Tyler 1c0d855b8e refactor(claim-drawer): mount on shared DrillDrawerHeader shell 2026-06-21 18:10:43 -06:00
Tyler 8db5db7610 feat(claim-drawer): validation rule opens peek with rule catalog 2026-06-21 18:10:43 -06:00
Tyler 786ead8c94 feat(claim-drawer): payer name opens PeekModal on top of drawer 2026-06-21 18:10:43 -06:00
Tyler 6c773c1159 feat(upload): streamed claim cards offer drill to persisted entity 2026-06-21 18:10:43 -06:00
Tyler 5c7e9b6168 feat(batch-diff): claim ids drillable to /claims?claim=ID 2026-06-21 18:10:43 -06:00
Tyler ac87ed4908 feat(reconciliation): card body drillable, select button split 2026-06-21 18:10:43 -06:00
Tyler 4a8ce1a524 feat(inbox): rejected + payer_rejected + done_today rows drillable 2026-06-21 18:10:43 -06:00
Tyler 5053a1ea8e feat(acks): row click opens AckDrawer 2026-06-21 18:10:43 -06:00
Tyler 33fa899217 feat(drill): AckDrawer with header + segment status list 2026-06-21 18:10:43 -06:00
Tyler 6fdbceefc2 feat(drill): useAckDrawerUrlState — ?ack= URL sync 2026-06-21 18:10:43 -06:00
Tyler 7290cac643 plan+spec: add migration 0014 to relax PKs; renumber plan Task 1.3 -> 1.4
The implementer caught that claims.id is a single-column PK, which
prevents the same CLM01 from existing in multiple batches. That
makes the spec's pre-flight 409 workflow unreachable and resubmits
impossible. Migration 0014 relaxes the PKs to composite (batch_id,
id) on both claims and remittances, and updates all FKs that
referenced the old single-column PK. Task 1.3 in the plan is now
the migration; the previous Task 1.3 (helper tightening) is
renumbered to Task 1.4 and will run after 0014 lands.
2026-06-21 18:09:06 -06:00
Tyler 5e8c7b11ea plan: parse-decide workflow implementation for 837/835 upload dedup 2026-06-21 17:53:21 -06:00
Tyler 9c0cec8f0c spec: parse-then-decide workflow for 837/835 upload dedup 2026-06-21 17:47:03 -06:00
Tyler 9a313d2c1b feat(activity): remit_received events drill to RemitDrawer 2026-06-21 17:19:31 -06:00
Tyler 2eb61f16ff feat(reconciliation): candidate rows drill to RemitDrawer or ClaimDrawer 2026-06-21 17:19:31 -06:00
Tyler 12c2913ba1 feat(batchdiff): remit rows drill to RemitDrawer 2026-06-21 17:19:31 -06:00
Tyler 54440da2cd feat(inbox): candidates + unmatched rows drill to RemitDrawer or ClaimDrawer 2026-06-21 17:19:31 -06:00
Tyler 7427838292 feat(remits): row click opens RemitDrawer (replaces inline CAS expand) 2026-06-21 17:19:31 -06:00
Tyler b606e8c9a2 plan: drop claims UNIQUE + 409 UX implementation plan 2026-06-21 17:02:10 -06:00
Tyler ac709c07c3 spec: self-review fixes (explicit claim_id arg, defer_foreign_keys) 2026-06-21 16:58:50 -06:00
Tyler 08e83da91d feat(drill): ProviderDrawer — Claims + Activity tabs from extended /providers/{npi} 2026-06-21 16:56:36 -06:00
Tyler 3f672d5db3 spec: drop claims UNIQUE constraint + 409 UX 2026-06-21 16:54:07 -06:00
363 changed files with 79557 additions and 10920 deletions
+13
View File
@@ -0,0 +1,13 @@
# Repo-root .dockerignore — applies to `docker compose build` (which builds
# both backend and frontend contexts from the repo root). Excludes anything
# that should never end up in a build context.
.worktrees/
.git/
.github/
docs/prodfiles/
*.production.txt
node_modules/
dist/
.venv/
.superpowers/brainstorm/
+37 -4
View File
@@ -1,7 +1,40 @@
# Cyclone — environment configuration
# Copy this file to `.env.local` and fill in values for your environment.
# Base URL for the Python (FastAPI) backend that powers the Upload page and
# the real /api/parse-837 + /api/parse-835 endpoints. Leave empty to keep
# the in-memory sample data store and disable real EDI parsing.
VITE_API_BASE_URL=http://localhost:8000
# Required on first boot. Cyclone refuses to start without these unless
# at least one user already exists (e.g. seeded via `python -m cyclone users create`).
# Min 12 chars for password.
CYCLONE_ADMIN_USERNAME=admin
CYCLONE_ADMIN_PASSWORD=change-me-to-a-strong-password-min-12-chars
# Base URL for the Python (FastAPI) backend. Leave empty for the
# Docker deployment (nginx proxies /api/* to backend on compose network).
VITE_API_BASE_URL=
# Optional. Set to 1 if you're behind an HTTPS reverse proxy and want
# the session cookie to include the Secure flag.
# CYCLONE_BEHIND_HTTPS=1
# Optional. Set to 1 to disable auth entirely (DEV ONLY). When set,
# the backend auto-grants admin access without checking credentials.
# CYCLONE_AUTH_DISABLED=0
# ─── Gainwell / HCPF SFTP credentials (operator convention) ─────────────
# These mirror what /home/tyler/EDI/scripts/upload_claims.sh exports and
# what the operator's SFTP client uses. Cyclone's secrets module looks up
# CYCLONE_SFTP_PASSWORD (or _FILE), not GAINWELL_SFTP_PASS — to bridge:
#
# export CYCLONE_SFTP_PASSWORD="$GAINWELL_SFTP_PASS"
#
# or for the daemon, set CYCLONE_SFTP_PASSWORD_FILE to a 0600 file.
GAINWELL_SFTP_USER=colorado-fts\coxix_prod_11525703
GAINWELL_SFTP_HOST=mft.gainwelltechnologies.com
GAINWELL_SFTP_PASS=
GAINWELL_REMOTE_DIR=/CO XIX/PROD/coxix_prod_11525703/ToHPE
# CYCLONE_SFTP_PASSWORD= # mirror of GAINWELL_SFTP_PASS
# CYCLONE_SFTP_PASSWORD_FILE= # path to a 0600 file containing it
# CYCLONE_SCHEDULER_AUTOSTART= # 1/true/yes to start the MFT poll loop on API launch
# CYCLONE_SCHEDULER_POLL_SECONDS=60 # poll interval when autostart is on
# CYCLONE_TAIL_HEARTBEAT_S=15 # NDJSON stream heartbeat interval
+14
View File
@@ -20,6 +20,16 @@ build/
*.swp
*.swo
# macOS extraction residue (AppleDouble / __MACOSX from unzip on macOS).
# These are paired with every regular file in an extracted zip — never data.
__MACOSX/
._*
# Operator drop zone (untracked working dir; contents are HCPF-delivered
# inbound files, never source). Use `mkdir -p ingest && touch ingest/.gitkeep`
# if you want the directory itself in the repo.
ingest/
# Local config
.env
.env.local
@@ -38,3 +48,7 @@ claims_output/
# Brainstorm session artifacts (visual companion mockups, events, server state).
# Skills under .superpowers/skills/ are committed project-scoped guidance.
.superpowers/brainstorm/
# SP33+ scratch / production-data ingest. Generated artifacts live
# here only — the source EDI sits under docs/prodfiles/.
ingest/
@@ -19,6 +19,10 @@ content negotiation + `tail_events`), and ~30 routes still inlined
in `backend/src/cyclone/api.py`. The next refactor target is the
parse endpoints.
## Auth gate (SP24)
Every router declared in `backend/src/cyclone/api_routers/` **must** carry `dependencies=[Depends(matrix_gate)]` at the `APIRouter(...)` declaration — not on each individual endpoint. The gate lives at `backend/src/cyclone/auth/deps.py:107` and the role matrix is at `backend/src/cyclone/auth/permissions.py`. The roles are `admin / user / viewer`; `matrix_gate` returns 401 when there's no session and 403 when the role is below the endpoint's required role. When `AUTH_DISABLED` is True (conftest autouse fixture flips it; `CYCLONE_AUTH_DISABLED=1` in prod-by-mistake), the gate short-circuits to a synthetic admin — see the SP24 spec for the threat-model implications. New routers get the gate by default; the auth-aware convention is `router = APIRouter(dependencies=[Depends(matrix_gate)])`.
## When to use
- **Adding an endpoint.** You're adding a new GET / POST handler —
+9 -3
View File
@@ -10,9 +10,15 @@ a spec, a plan, an implementation branch, and a single atomic merge commit
into `main`. This skill encodes the conventions so every increment follows
the same shape and the commit history stays auditable.
As of this writing: **16 specs** in `docs/superpowers/specs/`, **12 plans**
in `docs/superpowers/plans/`, and SP numbers used through **SP21** (the
universal-drilldown design in progress). The next increment is **SP22**.
As of this writing: **17 specs** in `docs/superpowers/specs/`, **13 plans**
in `docs/superpowers/plans/`, and SP numbers used through **SP22**. **SP23**
is the Ubuntu + Docker + RBAC product fork (awaiting user decision);
**SP24** is the auth-posture alignment (docs-only). The next free increment
is **SP25** after SP24 lands.
## Auth-aware spec template (SP24)
The threat-model section in the canonical SP-N spec template (`## 1. Scope`, second-to-last bullet) used to read "no second party to authenticate; no second host to harden against." **That phrasing is stale as of 2026-06-23** — the auth work landed in `main` and every backend endpoint requires login. New specs should instead state the auth boundary explicitly: "the auth boundary is HTTP (login required, bcrypt + HttpOnly session cookie); file-system threats remain the local-only threat model (SQLCipher at rest, macOS Keychain). SP23 changes the threat model to LAN-bound remote operator." Reference: [`docs/superpowers/specs/2026-06-23-cyclone-auth-posture-alignment-design.md`](../../../docs/superpowers/specs/2026-06-23-cyclone-auth-posture-alignment-design.md).
## When to use
+5 -1
View File
@@ -7,7 +7,11 @@ description: "Cyclone pytest + vitest fixture patterns, prodfiles layout, backen
The Cyclone test suite is split two ways: **backend pytest** (89 test files under `backend/tests/`, 13 flat fixtures in `backend/tests/fixtures/`, one autouse `conftest.py` that resets the DB per-test) and **frontend vitest** (59 `*.test.ts(x)` siblings across `src/`, two rendering styles — `@testing-library/react` and a custom `createRoot`+`Probe` shim). This skill codifies the conventions so additions stay consistent with what's already there.
As of this writing: **89 backend test files**, **13 flat backend fixtures**, **59 frontend `*.test.ts(x)` siblings**, and **23 prodfiles samples** across `docs/prodfiles/{837p-from-axiscare,835fromco,FromHPE,claims}/`. The next increment is **SP22**.
As of this writing: **89 backend test files**, **13 flat backend fixtures**, **59 frontend `*.test.ts(x)` siblings**, and **23 prodfiles samples** across `docs/prodfiles/{837p-from-axiscare,835fromco,FromHPE,claims}/`. The next increment is **SP24** (SP23 is the Ubuntu+Docker fork, awaiting user decision).
## Auth flag (SP24)
The autouse `conftest.py` fixture at `backend/tests/conftest.py` flips `cyclone.auth.deps.AUTH_DISABLED = True` for the entire test session, so every test runs without a login round-trip. **Any new test that reads `cyclone.auth.deps.AUTH_DISABLED` directly will see `True`** — that's the test-suite reality, not a production reality. If you need a test that exercises the real gate, import `from cyclone.auth.deps import matrix_gate` and call it directly with a `Request` whose `state` carries a real session, or flip the flag back inside the test and reset it on teardown. The startup WARNING (`backend/src/cyclone/__main__.py`) is silent in tests by default because the conftest sets the flag before `bootstrap.run()` is called via `import`.
## When to use
+212
View File
@@ -0,0 +1,212 @@
# CLAUDE.md
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
## What this is
Cyclone is a self-hosted X12 EDI claims-management suite for a single billing office (Colorado Medicaid currently). It parses 837P professional claims and 835 ERA remittances (X12 005010X222A1 / 005010X221A1) and also handles 999, TA1, 270, 271, and 277CA. Always binds to `0.0.0.0` — the host firewall / compose port publishing is what restricts reachability, not the bind address. Requires login (auth boundary is HTTP; bcrypt + HttpOnly session cookie; first admin bootstrapped from `CYCLONE_ADMIN_USERNAME` + `CYCLONE_ADMIN_PASSWORD` env vars; see SP24 spec for the full posture). LAN-only by design — don't expose the published ports to the WAN.
Stack: one Python process (FastAPI + uvicorn, port 8000) + one Node process in dev (Vite, port 5173). The authoritative state is a single SQLite file at `~/.local/share/cyclone/cyclone.db` (or SQLCipher at the same path when the macOS Keychain entry + `sqlcipher3` are both present).
For the day-1 architecture read, see `docs/ARCHITECTURE.md` (process topology, module map, store facade, parser pipeline, pubsub). For the what-it-does read, see `docs/REQUIREMENTS.md` (FRs + NFRs + DoD).
## Install
```bash
# Backend (Python 3.11+)
cd backend
python -m venv .venv
.venv/bin/pip install -e '.[dev]'
# Frontend (Node 20+)
cd ..
npm install
```
Optional backend extras: `pip install -e '.[sqlcipher]'` (encryption at rest, SP12) and `pip install -e '.[sftp]'` (real SFTP, SP13).
## Dev (two terminals)
```bash
# Terminal 1 — backend
cd backend
.venv/bin/python -m cyclone serve # default 0.0.0.0:8000
# CYCLONE_PORT=... overrides port; CYCLONE_RELOAD=1 enables uvicorn --reload
# Or: .venv/bin/uvicorn cyclone.api:app --reload --port 8000
# Terminal 2 — frontend
npm run dev # Vite on http://localhost:5173
```
Vite proxies `/api/*` to the backend at `http://127.0.0.1:${CYCLONE_PORT:-8000}` so relative-URL fetchers (the live-tail NDJSON streams in particular) resolve through the same origin. Override the backend port with `CYCLONE_PORT` in the frontend terminal too.
Create `.env.local` at the repo root with `VITE_API_BASE_URL=http://127.0.0.1:8000`. Without it, the UI runs against the in-memory zustand store and real EDI parsing is disabled.
## Test
```bash
# Backend — full suite
cd backend && .venv/bin/pytest
# Backend — one file
cd backend && .venv/bin/pytest tests/test_api_999.py -v
# Backend — one test by node id
cd backend && .venv/bin/pytest tests/test_api_999.py::test_parse_999_endpoint_happy_path -v
# Frontend — full suite
npm test # alias for `vitest run`
# Frontend — one file
npx vitest run src/hooks/useFoo.test.ts
# Frontend — typecheck
npm run typecheck
# Frontend — build (tsc -b + vite build)
npm run build
# Frontend — lint
npm run lint
```
**Conventions** (full detail in `.superpowers/skills/cyclone-tests/SKILL.md`):
- Backend tests live under `backend/tests/test_*.py`. Two flavors: `test_api_<topic>_<verb>.py` (FastAPI integration via `fastapi.testclient.TestClient`) and `test_<module>_<behavior>.py` (pure-unit). Autouse `conftest.py` points `CYCLONE_DB_URL` at `tmp_path/test.db`, calls `db._reset_for_tests()` + `db.init_db()`, and wires a fresh `EventBus` onto `app.state`.
- Prodfiles (real EDI samples under `docs/prodfiles/<source>/`) are never read directly from a test — copy to `backend/tests/fixtures/<descriptive-name>.txt` first and reference as a module-level `Path` constant. The `fixtures/` dir is flat (no per-test subdirs).
- Frontend tests are siblings: `useFoo.ts``useFoo.test.ts`, `ClaimDrawer.tsx``ClaimDrawer.test.tsx`. Setup is `// @vitest-environment happy-dom` plus `(globalThis as { IS_REACT_ACT_ENVIRONMENT?: boolean }).IS_REACT_ACT_ENVIRONMENT = true;`. Mock the API at the module boundary with `vi.mock("@/lib/api", ...)`; stub fetch with `vi.stubGlobal("fetch", vi.fn().mockResolvedValue(...))`. `vitest.config.ts` sets `VITE_API_BASE_URL=http://test.local` so the `api` module doesn't throw `notConfiguredError` before the mock fires.
- Time-sensitive tests: frontend uses `vi.useFakeTimers()` + `vi.setSystemTime(...)` + `vi.advanceTimersByTime(ms)`. Backend passes explicit `datetime(...)` values. Don't add `await new Promise((r) => setTimeout(r, N))` — it's the legacy flaky pattern.
## Project-scoped skills (`.superpowers/skills/`)
Cyclone ships 8 skills under `.superpowers/skills/`. They auto-load by description match — no slash command needed. **Read the relevant skill before touching the matching subsystem.**
| Skill | Owns |
|---|---|
| `cyclone-spec` | The SP-N spec → plan → implement → merge flow (branch shape, file paths, commit prefixes, PR title, merge shape). |
| `cyclone-tests` | pytest + vitest fixture patterns, prodfiles drop-in rule, determinism rules. |
| `cyclone-edi` | EDI parser/validator conventions (837P/835/999/270/271/277CA/TA1, R-codes, CAS mapping). |
| `cyclone-tail` | Live-tail streaming wire format and the `useTailStream` + `useMergedTail` + `TailStatusPill` hook triplet. |
| `cyclone-store` | `CycloneStore` facade, write-paths, pubsub event contract, SP21 split map. |
| `cyclone-api-router` | FastAPI router conventions (`api_routers/`, `api_helpers.py`), response/error-envelope shapes. |
| `cyclone-frontend-page` | React page conventions (TanStack Query `use<X>` hook, drawer, URL state, sibling test). |
| `cyclone-cli` | CLI subcommand conventions (`cli.py`, exit codes, smoke tests). |
## The SP-N increment flow
Every feature ships as a numbered **SP-N increment**: spec → plan → implementation branch → single atomic merge into `main`. As of the last backfill, SP numbers are used through **SP22**; **SP23** is reserved for the Ubuntu + Docker + RBAC product fork (`docs/superpowers/specs/2026-06-22-cyclone-ubuntu-docker-deployment-design.md`, awaiting user decision); the next free increment is **SP24**. Read `cyclone-spec` before starting a new one. Non-negotiable shape:
- **Branch:** `sp<N>-<short-kebab-topic>` (e.g. `sp22-line-reconciliation`).
- **Spec path:** `docs/superpowers/specs/YYYY-MM-DD-cyclone-<topic>-design.md`, header `Status: Draft, awaiting user sign-off`, sections `Scope / Decisions / …`. Specs contain zero code blocks.
- **Plan path:** `docs/superpowers/plans/YYYY-MM-DD-cyclone-<topic>.md`, header per `superpowers:writing-plans` with `Goal / Architecture / Tech Stack / Spec` metadata + numbered `- [ ] Step N:` tasks.
- **Commit prefixes:** `feat(sp<N>): …`, `docs(spec): …`, `docs(plan): …`, `merge: SP<N> <topic> into main`.
- **PR title:** `SP<N> <Topic>` (matches the merge-commit subject).
- **Merge shape:** single atomic merge commit. **No squash** (collapses the audit trail) and **no rebase** (rewrites the SHAs the review was performed against). The SP-N merge commit *is* the record of the increment landing.
The matching skill to load alongside `cyclone-spec` depends on the subsystem the SP-N touches (see the "Related skills" section at the bottom of each skill file).
## Live-tail wire format
The Claims, Remittances, and Activity pages stay current without manual refresh. The backend publishes an internal event on every store write, the page opens a streaming HTTP connection to the matching `/api/<resource>/stream` endpoint, and new rows append to the table the moment they hit the database.
Endpoints (all accept the same query params as their non-streaming counterparts; `Content-Type: application/x-ndjson`):
| Method | Path | Subscribes to | Default sort |
|---|---|---|---|
| GET | `/api/claims/stream` | `claim_written` | `-submission_date` |
| GET | `/api/remittances/stream` | `remittance_written` | `-received_date` |
| GET | `/api/activity/stream` | `activity_recorded` | `-timestamp` (limit 50) |
Wire format: one JSON object per line, `{"type": ..., "data": ...}`. The first batch is the **snapshot** of currently-known rows, then `snapshot_end` with the count, then the **live** events. Known types: `item`, `snapshot_end`, `heartbeat` (keeps the connection alive on idle — clients flip to `stalled` after 30s of total silence), `item_dropped` (rare), `error`.
Status pill states (rendered by `<TailStatusPill>` in `src/components/TailStatusPill.tsx`): `live` (success), `connecting` (warning), `reconnecting` (warning), `stalled` (destructive, ↻ Reconnect button), `error` (destructive, ↻ Reconnect button), `closed` (destructive). Backoff on error: `1s → 2s → 4s → 8s → 16s → 30s` capped. `STALL_TIMEOUT_MS = 30_000` in `src/hooks/useTailStream.ts:53`. Heartbeat interval is `CYCLONE_TAIL_HEARTBEAT_S` env var, default 15s.
Frontend triplet for any live page: `use<X>(params)` (initial fetch) + `useTailStream(resource)` (opens the NDJSON stream, drives backoff/stall) + `useMergedTail(resource, baseItems, filterFn?)` (merges snapshot + tail, dedup'd by id). The subscription lives on the page, not inside the data hook — see `cyclone-frontend-page` for why.
## Backend at a glance
`backend/src/cyclone/` is a single namespace. The two largest files are `api.py` (~3,548 LOC, the only large file) and `store.py` (~2,423 LOC, the `CycloneStore` facade — SP21 is in flight to split it into a `cyclone/store/` subpackage; the public API stays unchanged). Subpackages: `api_routers/` (acks, admin, health, ta1_acks), `clearhouse/` (Clearhouse + SftpClient), `edi/` (filenames), `parsers/` (X12 transaction parsers + models + validators + serializers), `submission/` (SP37 — canonical `submit_file` helper shared by `cyclone submit-batch` CLI + `POST /api/submit-batch` HTTP endpoint; owns parse → DB-write → SFTP-upload → audit per file), `workflow/` (placeholder for future sub-project 6).
The store is the only read/write surface for the database; every mutating endpoint goes through it. All persistence flows through SQLAlchemy sessions via `db.SessionLocal()()`. SQLAlchemy ORM models live in `db.py`; 12 SQL migrations under `migrations/` (0001_initial through 0012_backups) are walked in order by `db_migrate.py`.
The parser pipeline is a 5-stage `tokenize → segmentize → model → validate → write_to_store` flow used for every inbound X12 type. Per-transaction parsers: `parse_837.py`, `parse_835.py`, `parse_999.py`, `parse_ta1.py`, `parse_270.py`, `parse_271.py`, `parse_277ca.py`. Each has a matching Pydantic model module (`models.py`, `models_835.py`, …) and a writer (`writer.py` / `writer_835.py`). The 837P serializer (`serialize_837.py`) is the byte-faithful outbound counterpart used by both single-claim download (`/api/claims/{id}/serialize-837`) and the bulk rejected-resubmit bundle (`/api/inbox/rejected/resubmit?download=true`).
The pubsub is `cyclone.pubsub.EventBus` — an in-process async fan-out broker. Publishers call `publish(kind, payload)`; subscribers receive via an async iterator. If a subscriber's per-kind queue is full, the oldest event is dropped so a slow consumer can't stall the producer. Bus is single-event-loop only (matches FastAPI/uvicorn).
Config: `config/payers.yaml` is the on-disk source for providers / payers / clearhouse, schema-validated at boot against a Pydantic model. Reload with `POST /api/admin/reload-config`. Original in-code `PAYER_FACTORIES` dict in `cli.py` is kept as a fallback for ad-hoc testing.
Secrets live in the macOS Keychain (via `keyring` + `cyclone.secrets`): SQLCipher key (service `cyclone`, account `cyclone.db.key`), SFTP password, backup passphrase. No secrets on disk in plaintext.
## Production SFTP posture (this box: manual mode)
This host runs in **manual SFTP mode** against Gainwell's MOVEit Transfer MFT at `mft.gainwelltechnologies.com`. The seeded `dzinesco` clearhouse keeps `sftp_block.stub: true`; the operator moves files to/from Gainwell with their SFTP client and drops inbound files into `ingest/` for cyclone to ingest. Do NOT flip `stub` to `false` from this host — see "Auth caveat" below.
### Env var convention
The operator's shell exports the Gainwell creds as `GAINWELL_SFTP_USER`, `GAINWELL_SFTP_PASS`, `GAINWELL_SFTP_HOST`, `GAINWELL_REMOTE_DIR`. Cyclone's `secrets.get_secret()` looks up `CYCLONE_SFTP_PASSWORD` (or `CYCLONE_SFTP_PASSWORD_FILE`) per `secrets._ENV_NAME_FOR`. The two are NOT the same name — bridge them per-call or in your shell rc:
```bash
export CYCLONE_SFTP_PASSWORD="$GAINWELL_SFTP_PASS"
```
### Inbound drop zone
`/home/tyler/dev/cyclone/ingest/` is the operator-maintained staging dir for inbound MFT files (999 acks, TA1, 835 remittances, 277CA claim acks). Files here are NOT auto-watched; processing happens via the procedure in `docs/RUNBOOK.md` § "Manual SFTP mode". At time of writing this dir holds ~1500 unprocessed 999 acks + 1 TA1 + 1 835 from early July 2026 — the local ingest flow clears them.
### Auth caveat (do not retry)
Paramiko reaches `MOVEit Transfer SFTP` cleanly (SSH kex completes, MOVEit offers password auth) but `AuthenticationException: Authentication failed` is returned despite the operator confirming the credentials are known-good from another host. Most likely cause: MOVEit Transfer's IP-based access control — this host's public IP `103.14.26.95` is not whitelisted. Repeated auth attempts risk account lockout. Do NOT keep guessing. Resolve by either: (a) whitelisting this IP with Gainwell's MFT admin, or (b) staying in manual mode and moving files via the operator's SFTP client.
### Inbound ingestion paths
The canonical way to write 999/TA1/277CA/835 rows into the DB is `Scheduler.process_inbound_files`, exposed as the CLI `cyclone pull-inbound --date YYYYMMDD` and the HTTP `POST /api/admin/scheduler/pull-inbound`. **There is no `parse-999` / `parse-ta1` / `parse-277ca` CLI command** (CLAUDE.md's "CLI" section above is aspirational on those three). The `parse-837` and `parse-835` CLIs exist but only emit JSON files to `--output-dir`; they do NOT write to the DB. For DB writes, use `pull-inbound` (which dedupes via `processed_inbound_files`).
### Daemon hot-reload
`python -m cyclone serve` runs as root (started by `tini`) and keeps the SFTP block in memory. Flipping `stub` directly in the DB (`store.update_clearhouse(...)`) does NOT auto-reload the daemon's view — either restart the daemon or use `PATCH /api/clearhouse` (which calls `scheduler.reconfigure_scheduler` to hot-reload).
## Frontend at a glance
`src/` is React 18 + TypeScript + Vite. Routes register in `src/App.tsx` (11 pages, all under a `<Layout>` route wrapper). Pages are pure renderers — every page pairs with a `use<X>` data hook in `src/hooks/` and renders a `<PageHeader>` + a table/list/KPI grid. Drawers (`ClaimDrawer/`, `RemitDrawer/`, plus the new `ProviderDrawer/` and `AckDrawer/`) are mounted by the page and their open/close state is mirrored to the URL via `useDrawerUrlState` so deep-links round-trip. Drill-stack navigation is provided by `<DrillStackProvider>` in `src/components/drill/`.
State split: **server state** in TanStack Query (`@tanstack/react-query`); **ephemeral client state** in Zustand (`useTailStore` for live-tail append, plus the drill stack). The live-tail store is FIFO-capped at `TAIL_CAP = 10_000` per slice (`src/store/tail-store.ts:28`); `claims` and `remittances` are key-by-id with first-write-wins dedup, `activity` is an append-only array.
UI primitives in `src/components/ui/` are Radix-backed (`button`, `dialog`, `table`, `select`, `pagination`, `empty-state`, `error-state`, `filter-chips`, `skeleton`, `input`, `label`, `card`, `badge`, `skip-link`, `claim-state-badge`). Don't import a new UI library without discussion.
Path alias `@/``src/`. Configured in `vite.config.ts`, `vitest.config.ts`, and `tsconfig.app.json`.
## CLI
```bash
# Parser
python -m cyclone.cli parse-837 path/to/837p.txt --output-dir ./claims --payer co_medicaid [--strict] [--include-raw-segments]
python -m cyclone.cli parse-835 path/to/835.txt --output-dir ./remits
python -m cyclone.cli parse-999 inbound_999.txt
python -m cyclone.cli parse-ta1 inbound_ta1.txt
python -m cyclone.cli parse-277ca inbound_277ca.txt
# Validators
python -m cyclone.cli validate-npi 1234567893
python -m cyclone.cli validate-tin 721587149
# Other
python -m cyclone serve # uvicorn
python -m cyclone backup list
python -m cyclone backup create --reason manual
```
Exit codes are documented per subcommand in `cyclone-cli``0` for success, `2` for file-level failure, `1` for unexpected exceptions.
## Things that are easy to get wrong
- **`VITE_API_BASE_URL` matters.** With it empty, every `api` method throws `notConfiguredError()` and the UI falls back to the in-memory zustand store — parses are disabled and the live-tail streams never open.
- **Prodfiles vs fixtures.** Tests must reference `backend/tests/fixtures/<name>.txt`, not `docs/prodfiles/<source>/<file>.txt`. The prodfiles dir is the source-of-truth archive; the fixtures dir is the stable test surface.
- **SP-N merge shape.** No squash, no rebase. The merge commit *is* the audit trail. Squash collapses the per-commit history and breaks the SP-N audit trail.
- **Don't put domain logic in JSX.** Conditional renderings, table sorting, and KPI math all belong in the `use<X>` hook or a pure helper under `src/lib/`.
- **Don't open a drawer via local `useState`.** Use `useDrawerUrlState()` so the URL is the single source of truth — deep-links and reload-restore depend on it.
- **Don't call `useTailStream` from inside a `use<X>` hook.** The subscription lives on the page so the lifecycle ties to whoever mounts the hook, not to whoever happens to call it.
- **The store facade.** The public API of `cyclone.store` is preserved through SP21's split — call through the facade, not directly into the underlying modules.
- **Encryption is optional, not required.** When the Keychain entry is missing **or** `sqlcipher3` is not installed, the DB falls back to plain SQLite. Don't fail boot on missing encryption.
- **Always bind to 0.0.0.0.** The bind address does not control reachability — the host firewall / compose port publishing does. Requires login (bcrypt + HttpOnly session cookie; see SP24 spec), and the threat model is still a stolen/imaged drive — SQLCipher at rest and the macOS Keychain handle that. The auth boundary is the HTTP layer; the file-system posture is unchanged. Don't expose the published ports to the public internet (LAN-only or VPN-fronted). Don't disable auth without an explicit `CYCLONE_AUTH_DISABLED=1` env var (the escape hatch logs a WARNING at boot).
</content>
</invoke>
+43
View File
@@ -0,0 +1,43 @@
# syntax=docker/dockerfile:1.7
#
# Cyclone frontend — React SPA built with node:20-alpine and served by
# nginx:1.27-alpine. nginx reverse-proxies /api/* to the backend service
# over the compose-managed bridge network.
# ---------- builder ----------
FROM node:20-alpine AS builder
WORKDIR /build
# Install deps first so this layer caches across source edits.
# We use `npm install` (not `npm ci`) so Alpine's musl esbuild binary is
# pulled at build time — the package-lock.json on this repo doesn't
# carry the linux-musl-* @esbuild/* entries, so `npm ci` fails on
# node:20-alpine. `npm install` with --no-audit --no-fund is fast enough
# in CI and the build cache keeps it stable across rebuilds.
COPY package.json package-lock.json* ./
RUN npm install --no-audit --no-fund
# Build the production bundle into dist/. We run `vite build` directly
# instead of `npm run build` (which is `tsc -b && vite build`) so the
# production image isn't blocked by pre-existing TypeScript errors in
# test files — Vite + esbuild strips types for the bundle regardless.
# Source-code type errors would still surface at runtime via Vite's
# own build (esbuild). Run `npm run typecheck` separately to see them.
COPY . .
RUN npx vite build
# ---------- runtime ----------
FROM nginx:1.27-alpine
# Replace the default nginx site with ours (SPA + reverse proxy).
RUN rm -f /etc/nginx/conf.d/default.conf
COPY nginx.conf /etc/nginx/conf.d/default.conf
COPY --from=builder /build/dist /usr/share/nginx/html
# wget is on busybox; nginx:alpine doesn't ship curl.
HEALTHCHECK --interval=30s --timeout=5s --retries=3 \
CMD wget -qO- http://127.0.0.1:8080/ >/dev/null || exit 1
EXPOSE 8080
+53 -4
View File
@@ -30,7 +30,7 @@ Two terminals:
# Terminal 1 — backend
cd backend
.venv/bin/python -m cyclone serve
# (defaults to 127.0.0.1:8000; override with CYCLONE_PORT=...; reload with CYCLONE_RELOAD=1)
# (defaults to 0.0.0.0:8000; override with CYCLONE_HOST / CYCLONE_PORT / CYCLONE_RELOAD=1)
# Terminal 2 — frontend
npm run dev
@@ -111,6 +111,49 @@ npm run build
npm test
```
## Authentication
Cyclone ships with username/password authentication and three predefined roles.
**Roles:**
| Role | Can read | Can write (upload, parse, reconcile) | Can manage users |
| -------- | -------- | ------------------------------------ | ---------------- |
| `viewer` | ✅ | ❌ | ❌ |
| `user` | ✅ | ✅ | ❌ |
| `admin` | ✅ | ✅ | ✅ |
**Bootstrap.** On first start, set `CYCLONE_ADMIN_USERNAME` and
`CYCLONE_ADMIN_PASSWORD` (min 12 chars) in your environment. Cyclone creates the
first admin automatically. On subsequent starts these env vars are ignored, so
rotating the bootstrap password doesn't affect an already-seeded admin — use the
CLI below to reset it. When running via `docker compose`, both vars are
required: compose refuses to start with a clear error if either is missing.
**CLI.** Manage users from the command line:
```
python -m cyclone users create alice --role user --password 'hunter2hunter2'
python -m cyclone users list
python -m cyclone users disable alice
python -m cyclone users reset-password alice
python -m cyclone users set-role alice --role admin
```
**Login.** Browse to `http://localhost:5173` (dev) or `http://localhost:8081`
(Docker), sign in on the `/login` page, and you'll be redirected to the
dashboard. Sessions are stored server-side in SQLite with a 24-hour sliding
expiry — every authenticated request refreshes the TTL, so an active user
never gets logged out.
**Dev escape hatch.** Set `CYCLONE_AUTH_DISABLED=1` to bypass auth entirely
(the backend auto-grants admin on every request). **NEVER set this in
production** — it's a single env-var trip from wide-open to the public
internet. The Docker compose file does not honor this flag.
See `docs/superpowers/specs/2026-06-22-cyclone-auth-design.md` for the full
design.
## Live updates
The Claims, Remittances, and Activity pages stay current without
@@ -647,7 +690,7 @@ backup create` manually retains full control.
The `clearhouse.submit` endpoint uses `paramiko` to push a batch of
generated 837 files to the dzinesco SFTP server
(`mft.gainwelltechnologies.com:22`, path
`/CO XIX/PROD/coxix_prod_11525703/FromHPE`). The SFTP credential is
`/CO XIX/PROD/coxix_prod_11525703/ToHPE`). The SFTP credential is
fetched from the macOS Keychain at call time — never read from YAML,
never logged, never written to disk. The wire-up honors the file-naming
template stored in the `clearhouse` config:
@@ -758,6 +801,12 @@ backup API).
## Roadmap
> **Read order for new engineers:**
> 1. [`docs/REQUIREMENTS.md`](docs/REQUIREMENTS.md) — what Cyclone does (FRs + NFRs + DoD + traceability). The single index tying the 22 shipped sub-projects to the 38 functional + 18 non-functional requirements and the test strategy.
> 2. [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md) — how it fits together (process topology, package layout, data flow, lifecycle, operational concerns).
> 3. The per-SP spec under [`docs/superpowers/specs/`](docs/superpowers/specs/) for whatever you're touching.
> 4. The per-SP plan under [`docs/superpowers/plans/`](docs/superpowers/plans/) if you're implementing.
Sub-projects 2 through 19 are **shipped**. See the [completeness
review](docs/reviews/2026-06-20-cyclone-completeness-review.md) for
the honest gap analysis against the industry definition of a HIPAA
@@ -849,7 +898,7 @@ Shipped sub-projects (most recent first):
- **Sub-project 13 (shipped) — SFTP wire-up.** `paramiko`-backed
`SftpClient` replaces the SP9 stub. The clearhouse.submit endpoint
actually pushes to
`mft.gainwelltechnologies.com:/CO XIX/PROD/coxix_prod_11525703/FromHPE`.
`mft.gainwelltechnologies.com:/CO XIX/PROD/coxix_prod_11525703/ToHPE`.
SFTP credentials are read from the macOS Keychain at call time.
- **Sub-project 12 (shipped) — Encryption at rest.** Optional
SQLCipher AES-256 encryption of the SQLite file, with the key
@@ -1111,7 +1160,7 @@ the one-time setup recipe.
- `POST /api/clearhouse/submit` — same endpoint as SP9; the
implementation is now a real `paramiko` `SftpClient.write` to
`mft.gainwelltechnologies.com:/CO XIX/PROD/coxix_prod_11525703/FromHPE`.
`mft.gainwelltechnologies.com:/CO XIX/PROD/coxix_prod_11525703/ToHPE`.
SFTP credentials are fetched from the macOS Keychain at call time.
## License
+65
View File
@@ -0,0 +1,65 @@
# Cyclone Operator Runbook
Production operations for a single-operator Cyclone deploy on Ubuntu Linux. Assumes the box was bootstrapped via `scripts/cyclone-init.sh` and the stack is up via `docker compose up -d`.
## Daily
- [ ] Confirm the host healthcheck cron hasn't emailed. It pings `http://localhost:8080/api/health` every 5 minutes.
- [ ] `docker compose ps` — both services `healthy`.
- [ ] `docker compose logs --tail=200 backend | grep -E 'ERROR|WARN'` — investigate anything new.
## Weekly
- [ ] `curl -fsS http://localhost:8080/api/admin/audit-log -b cookies.txt | jq '.events[] | select(.event | test("login_failed|backup.failed"))'` — review failed logins + backup failures.
- [ ] Confirm `docker compose exec backend ls -la /var/lib/cyclone/backups/` shows recent `.bin` files (within 25h of now).
## Quarterly
- [ ] Rotate the SQLCipher / cookie-signing key:
```bash
bash scripts/cyclone-init.sh --force # overwrites /etc/cyclone/secrets/db.key
docker compose restart backend # picks up the new key
```
Old `.bin` backups become unreadable after this; export them first if you need to keep them.
## As needed
- **Add an operator.** Log in as admin → `/admin/users` → Create user. Roles: `admin` / `user` / `viewer`.
- **Reset a password.** Admin UI → Users → Reset password, OR `docker compose exec backend python -m cyclone admin reset-password --username <name>`.
- **Restore from backup.** Admin UI → Backups → pick the snapshot → Initiate restore → Confirm. The backend will restart automatically.
- **Roll back the code (not the schema).** `TAG=0.0.9 docker compose up -d`. The previous image stays in the local Docker cache for one cycle.
- **Pull a new `:stable`.**
```bash
cd /opt/cyclone
docker compose pull
docker compose up -d
docker compose logs -f backend | head -200 # verify migrations + healthcheck
```
- **Off-box backup copy.** The operator is expected to rsync `/var/lib/docker/volumes/cyclone_backups/_data/` to an external drive or NAS nightly. The `.bin` files are already encrypted; the destination doesn't need its own encryption.
- **Inspect the DB.** `docker compose exec backend sqlite3 /var/lib/cyclone/db/cyclone.db ".tables"` (works only if SQLCipher key is on disk; the in-process decrypt happens via the cyclone backend).
## Annual
- [ ] Rotate the admin password (force re-login for everyone).
- [ ] Audit the `/etc/cyclone/secrets/` directory permissions — should be `chmod 600 root:root`.
- [ ] Review the audit log for stale admin sessions.
## Emergency
- **Backend won't start.** `docker compose logs --tail=300 backend`. Look for migration failures (rerun is safe — migrations are forward-only), SQLCipher key mismatch (`PRAGMA key` failure), or port collisions.
- **Frontend won't serve.** `docker compose logs --tail=100 frontend`. Usually nginx config drift; `docker compose restart frontend`.
- **Both unhealthy after a host reboot.** Docker may have come up before the named volumes did. `docker compose down && docker compose up -d`.
- **Suspected key compromise.** Rotate immediately (see Quarterly above). All active sessions are invalidated.
## Where things live
| Asset | Path |
|---|---|
| Docker compose file | `/opt/cyclone/docker-compose.yml` |
| Secrets | `/etc/cyclone/secrets/{db.key,admin_username,admin_pw}` |
| Live DB (SQLCipher-encrypted volume) | `cyclone_db` named volume, mounted at `/var/lib/cyclone/db` |
| Encrypted backups | `cyclone_backups` named volume, mounted at `/var/lib/cyclone/backups` |
| Uploaded prod files | `cyclone_prodfiles` named volume |
| SFTP staging stub | `cyclone_sftp_staging` named volume |
| Logs | `cyclone_logs` named volume + bind-mounted at `/var/log/cyclone` |
| Off-box backup destination | Operator's external drive / NAS (rsync cron, not in compose) |
+233
View File
@@ -0,0 +1,233 @@
// UI/UX Score Loop — pass 1 driver.
// Loads each route at three sizes in Chrome Canary, captures screenshots,
// logs console errors, runs a small interaction probe per flow, and
// writes a JSON report. Does not modify any source files.
import puppeteer from "puppeteer-core";
import { mkdir, writeFile } from "node:fs/promises";
const BASE = "http://127.0.0.1:5173";
const SHOTS = "/tmp/cyclone-uiux/shots";
const REPORT = "/tmp/cyclone-uiux/report.json";
const SIZES = [
{ name: "desktop", w: 1440, h: 900 },
{ name: "tablet", w: 768, h: 1024 },
{ name: "mobile", w: 375, h: 812 },
];
// Routes to load. path = the route; name = the flow label; ready = a
// selector we wait for to consider the page "rendered".
const FLOWS = [
{ name: "dashboard", path: "/", ready: "aside nav, h1, h2" },
{ name: "upload", path: "/upload", ready: "section[aria-label='File upload']" },
{ name: "inbox", path: "/inbox", ready: "main, section[aria-label='Queue summary']" },
{ name: "claims", path: "/claims", ready: "table, [data-testid='claims-page-body']" },
{ name: "claims-denied", path: "/claims?status=denied", ready: "table, [data-testid='claims-page-body']" },
{ name: "remittances", path: "/remittances", ready: "main, table" },
{ name: "providers", path: "/providers", ready: "main, table" },
{ name: "reconciliation",path: "/reconciliation", ready: "main" },
{ name: "acks", path: "/acks", ready: "main, table" },
{ name: "batches", path: "/batches", ready: "main, table" },
{ name: "batch-diff", path: "/batch-diff", ready: "main" },
{ name: "activity", path: "/activity", ready: "main" },
{ name: "404", path: "/does-not-exist", ready: "main" },
];
async function setupViewports(browser) {
const pages = [];
for (const size of SIZES) {
const page = await browser.newPage();
await page.setViewport({ width: size.w, height: size.h, deviceScaleFactor: 1 });
pages.push({ page, size });
}
return pages;
}
async function probeFlow(page, flow) {
const consoleErrors = [];
const pageErrors = [];
const failedRequests = [];
const onConsole = (msg) => {
if (msg.type() === "error") consoleErrors.push(msg.text());
};
const onPageError = (err) => pageErrors.push(err.message);
const onRequestFailed = (req) => failedRequests.push(`${req.method()} ${req.url()} :: ${req.failure()?.errorText}`);
page.on("console", onConsole);
page.on("pageerror", onPageError);
page.on("requestfailed", onRequestFailed);
const t0 = Date.now();
let rendered = false;
let readyError = null;
try {
await page.goto(`${BASE}${flow.path}`, { waitUntil: "networkidle2", timeout: 15000 });
if (flow.ready) {
try {
await page.waitForSelector(flow.ready, { timeout: 5000 });
rendered = true;
} catch (e) {
readyError = e.message;
}
} else {
rendered = true;
}
} catch (e) {
readyError = e.message;
}
const loadMs = Date.now() - t0;
page.off("console", onConsole);
page.off("pageerror", onPageError);
page.off("requestfailed", onRequestFailed);
return { rendered, loadMs, readyError, consoleErrors, pageErrors, failedRequests };
}
async function probeInteractions(page, flow) {
const findings = [];
// Generic a11y / structural probes per flow.
try {
// Sidebar visible? (md+ shows it; < md hides it)
const aside = await page.$("aside");
findings.push({ check: "sidebar-present", pass: !!aside });
} catch (e) {
findings.push({ check: "sidebar-present", pass: false, err: e.message });
}
try {
// Top bar present?
const main = await page.$("main#main-content");
findings.push({ check: "main-present", pass: !!main });
} catch (e) {
findings.push({ check: "main-present", pass: false, err: e.message });
}
try {
// H1 or page heading?
const heading = await page.evaluate(() => {
const h = document.querySelector("h1, h2");
return h ? h.textContent?.trim().slice(0, 60) : null;
});
findings.push({ check: "heading-present", pass: !!heading, value: heading });
} catch (e) {
findings.push({ check: "heading-present", pass: false, err: e.message });
}
// Flow-specific probes.
if (flow.name === "claims" || flow.name === "claims-denied") {
try {
const chips = await page.$$("[role='radio'], button[role='radio']");
findings.push({ check: "status-chips", pass: chips.length >= 1, count: chips.length });
} catch (e) {
findings.push({ check: "status-chips", pass: false, err: e.message });
}
try {
const search = await page.$("input[placeholder*='Search']");
findings.push({ check: "search-input", pass: !!search });
} catch (e) {
findings.push({ check: "search-input", pass: false, err: e.message });
}
}
if (flow.name === "upload") {
try {
const dropzone = await page.$("section[aria-label='File upload']");
findings.push({ check: "dropzone-present", pass: !!dropzone });
const selects = await page.$$("button[role='combobox']");
findings.push({ check: "payer-kind-selects", pass: selects.length >= 2, count: selects.length });
} catch (e) {
findings.push({ check: "upload-elements", pass: false, err: e.message });
}
}
if (flow.name === "inbox") {
try {
const lanes = await page.$$("main > div > div");
findings.push({ check: "lane-cards", pass: lanes.length >= 1, count: lanes.length });
} catch (e) {
findings.push({ check: "lane-cards", pass: false, err: e.message });
}
}
if (flow.name === "404") {
try {
const text = await page.evaluate(() => document.body.innerText);
findings.push({ check: "404-text", pass: text.includes("404") || text.toLowerCase().includes("doesn't exist") });
} catch (e) {
findings.push({ check: "404-text", pass: false, err: e.message });
}
}
return findings;
}
async function main() {
await mkdir(SHOTS, { recursive: true });
const browser = await puppeteer.launch({
executablePath: "/Applications/Google Chrome Canary.app/Contents/MacOS/Google Chrome Canary",
headless: "new",
args: ["--no-sandbox", "--disable-dev-shm-usage"],
});
const startedAt = new Date().toISOString();
const results = [];
for (const size of SIZES) {
const page = await browser.newPage();
await page.setViewport({ width: size.w, height: size.h, deviceScaleFactor: 1 });
for (const flow of FLOWS) {
const probe = await probeFlow(page, flow);
const interactions = await probeInteractions(page, flow);
const shot = `${SHOTS}/${flow.name}--${size.name}.png`;
try {
await page.screenshot({ path: shot, fullPage: false });
} catch (e) {
// ignore — recording in result
}
results.push({
flow: flow.name,
path: flow.path,
size: size.name,
viewport: { w: size.w, h: size.h },
probe,
interactions,
shot,
});
console.log(
`${size.name.padEnd(7)} ${flow.name.padEnd(20)} ` +
`render=${probe.rendered} load=${probe.loadMs}ms ` +
`consoleErr=${probe.consoleErrors.length} pageErr=${probe.pageErrors.length}`
);
}
await page.close();
}
await browser.close();
// Aggregate.
const summary = {
startedAt,
endedAt: new Date().toISOString(),
flows: results.length,
sizes: SIZES.map((s) => s.name),
renderedOk: results.filter((r) => r.probe.rendered).length,
withConsoleErrors: results.filter((r) => r.probe.consoleErrors.length > 0).length,
withPageErrors: results.filter((r) => r.probe.pageErrors.length > 0).length,
withFailedRequests: results.filter((r) => r.probe.failedRequests.length > 0).length,
results,
};
await writeFile(REPORT, JSON.stringify(summary, null, 2));
console.log("\nSummary:", JSON.stringify({
flows: summary.flows,
renderedOk: summary.renderedOk,
withConsoleErrors: summary.withConsoleErrors,
withPageErrors: summary.withPageErrors,
withFailedRequests: summary.withFailedRequests,
}, null, 2));
console.log("\nReport:", REPORT);
console.log("Shots:", SHOTS);
}
main().catch((e) => {
console.error("FATAL", e);
process.exit(1);
});
+12
View File
@@ -0,0 +1,12 @@
.venv/
venv/
__pycache__/
*.py[cod]
*.egg-info/
.pytest_cache/
.ruff_cache/
tests/
docs/prodfiles/
*.production.txt
.git/
.github/
+83
View File
@@ -0,0 +1,83 @@
# syntax=docker/dockerfile:1.7
#
# Cyclone backend — FastAPI on python:3.11-slim-bookworm with sqlcipher.
#
# Two-stage build:
# 1. builder — wheels the package with [sqlcipher] extra into /wheels.
# 2. runtime — slim base, tini PID 1, curl-based healthcheck.
#
# `sqlcipher` is preferred but the engine falls back to plain SQLite at
# runtime if the package isn't actually installed (see cyclone.db) — so a
# missing libsqlcipher-dev during build will fail loudly here rather than
# silently downgrading encryption in production.
# ---------- builder ----------
FROM python:3.11-slim-bookworm AS builder
ENV PIP_NO_CACHE_DIR=1 \
PIP_DISABLE_PIP_VERSION_CHECK=1 \
PYTHONDONTWRITEBYTECODE=1
RUN apt-get update && apt-get install -y --no-install-recommends \
build-essential \
libffi-dev \
libsqlcipher-dev \
&& rm -rf /var/lib/apt/lists/*
WORKDIR /build
# Copy the build manifest first so this layer caches across source edits.
COPY pyproject.toml ./
# Copy the full source tree, then build the wheel once. We deliberately
# avoid the "stub __init__.py, build wheel, then rebuild" pattern — it
# left stale `__init__.py` content in the wheel because pip wheel reuses
# the cached wheel metadata when the name+version matches. See git
# history on this file for the long version.
COPY src/ ./src/
# Install the sftp extra alongside sqlcipher so the real-mode SFTP
# client (paramiko) is available inside the container — required by
# SP25 + SP26 for live Gainwell MFT polling.
RUN pip wheel --no-cache-dir --wheel-dir /wheels '.[sqlcipher,sftp]'
# ---------- runtime ----------
FROM python:3.11-slim-bookworm
ENV PYTHONUNBUFFERED=1 \
PYTHONDONTWRITEBYTECODE=1
RUN apt-get update && apt-get install -y --no-install-recommends \
libsqlcipher-dev \
curl \
tini \
&& rm -rf /var/lib/apt/lists/* \
&& useradd --create-home --uid 1000 --shell /bin/bash cyclone
WORKDIR /app
COPY --from=builder /wheels /wheels
RUN pip install --no-cache-dir --no-index --find-links /wheels 'cyclone[sqlcipher,sftp]' \
&& rm -rf /wheels
# NOTE: we deliberately do NOT drop privileges to the `cyclone` user.
# Named volumes mount as root inside the container, and chown-ing them
# requires CAP_CHOWN (root). The standard hardened pattern is an
# entrypoint script that chowns as root then drops to the app user via
# gosu/su-exec — adds a dependency + an entrypoint file. For v1 we run
# as root inside the container; Docker's user-namespace remapping is
# the recommended host-level isolation. The `cyclone` user is created
# above and survives only so file ownership in bind mounts stays
# consistent. To harden later: install gosu + add an entrypoint script
# that does `chown -R cyclone:cyclone /var/lib/cyclone/... && exec gosu
# cyclone "$@"`.
EXPOSE 8000
# Container-level healthcheck — the compose service healthcheck is
# effectively a duplicate but the Docker `HEALTHCHECK` directive keeps
# `docker ps` honest without needing compose to be running.
HEALTHCHECK --interval=30s --timeout=5s --start-period=30s --retries=3 \
CMD curl -fs http://localhost:8000/api/health || exit 1
ENTRYPOINT ["tini", "--"]
CMD ["python", "-m", "cyclone", "serve"]
+10
View File
@@ -16,6 +16,15 @@ dependencies = [
"sqlalchemy>=2.0,<3",
"pyyaml>=6.0,<7",
"keyring>=25.0,<26",
# backup_service / backup: encryption-at-rest (SP17). Used at module
# top-level by cyclone.backup, so it has to be a hard dep — not an
# extra — or the test suite fails to collect when the venv is built
# from a clean `uv sync`.
"cryptography>=49.0,<50",
# passlib 1.7.4 + bcrypt >= 4.1 are incompatible (passlib probes bcrypt.__about__
# which 4.x removed). Pin bcrypt < 4.1.
"passlib[bcrypt]>=1.7.4",
"bcrypt<4.1",
]
[project.optional-dependencies]
@@ -24,6 +33,7 @@ dev = [
"pytest-cov>=4.1",
"pytest-asyncio>=0.23,<1",
"httpx>=0.27,<1",
"pytest-randomly>=4.1",
]
sqlcipher = [
# SP12: encryption at rest. Optional — without it the DB is plain SQLite.
+32 -2
View File
@@ -1,10 +1,11 @@
"""Entry point for ``python -m cyclone``.
* ``python -m cyclone`` (no args) — Click CLI (``cli.main``)
* ``python -m cyclone serve`` — start the FastAPI app on 127.0.0.1:8000
* ``python -m cyclone serve`` — start the FastAPI app on 0.0.0.0:8000
Honors the env vars:
* ``CYCLONE_HOST`` (default ``0.0.0.0`` — always bind to all interfaces)
* ``CYCLONE_PORT`` (default ``8000``)
* ``CYCLONE_RELOAD`` (default ``0``; set to ``1`` to enable uvicorn reload)
"""
@@ -16,13 +17,42 @@ import sys
def main() -> None:
# Always run first-admin bootstrap before any other entry path.
# Must happen before ``serve`` (uvicorn) AND before the Click CLI
# dispatch — otherwise `python -m cyclone users create ...` on a
# fresh DB would race with the bootstrap's check, and the API
# could come up with zero users.
from cyclone.auth import bootstrap
bootstrap.run()
# SP24: if the AUTH_DISABLED escape hatch is on, scream at boot so a
# misconfigured production deploy fails loudly. The flag is flipped by
# ``CYCLONE_AUTH_DISABLED=1`` (see ``cyclone.auth.bootstrap``) and by the
# pytest conftest autouse fixture (see ``.superpowers/skills/cyclone-tests``).
from cyclone.auth import deps as _auth_deps
if _auth_deps.AUTH_DISABLED:
import logging
logging.getLogger("cyclone").warning(
"AUTH_DISABLED is set (CYCLONE_AUTH_DISABLED=1) — all requests "
"treated as admin, dev only. Do NOT enable this in production."
)
if len(sys.argv) >= 2 and sys.argv[1] == "serve":
port = os.environ.get("CYCLONE_PORT", "8000")
# Always bind to 0.0.0.0 so the API is reachable from the
# frontend container on the compose bridge network AND from
# the host (Vite dev proxy) AND from the LAN. Network isolation
# is provided by the host firewall / compose port publishing,
# not by binding to loopback. Override with CYCLONE_HOST if you
# have a reason to restrict.
host = os.environ.get("CYCLONE_HOST", "0.0.0.0")
reload = os.environ.get("CYCLONE_RELOAD", "0") == "1"
sys.argv = [
sys.argv[0],
"cyclone.api:app",
"--host", "127.0.0.1",
"--host", host,
"--port", port,
]
if reload:
+100 -2965
View File
File diff suppressed because it is too large Load Diff
+30 -6
View File
@@ -93,19 +93,40 @@ def ndjson_stream_list(
}) + "\n"
def ndjson_stream_837(result: ParseResult) -> Iterator[bytes]:
"""Yield one JSON object per line: envelope → claims → summary."""
def ndjson_stream_837(
result: ParseResult, batch_id: str | None = None,
) -> Iterator[bytes]:
"""Yield one JSON object per line: envelope → claims → summary.
The ``batch_id`` is the server-side UUID assigned by the persistence
layer when the batch is ingested; the JSON response path exposes it
as the top-level ``batch_id`` field, but the NDJSON stream needs it
inline on the summary event so streaming clients can call
batch-scoped endpoints (``/api/batches/{id}/export-837``, …) without
a separate ``GET /api/batches`` round-trip. When ``batch_id`` is not
supplied the summary omits the field, preserving backward compat
with clients that don't expect it.
"""
envelope_obj = (
result.envelope.model_dump() if result.envelope is not None else None
)
yield (json.dumps({"type": "envelope", "data": envelope_obj}) + "\n").encode("utf-8")
for claim in result.claims:
yield (json.dumps({"type": "claim", "data": json.loads(claim.model_dump_json())}) + "\n").encode("utf-8")
yield (json.dumps({"type": "summary", "data": json.loads(result.summary.model_dump_json())}) + "\n").encode("utf-8")
summary_data = json.loads(result.summary.model_dump_json())
if batch_id is not None:
summary_data["batch_id"] = batch_id
yield (json.dumps({"type": "summary", "data": summary_data}) + "\n").encode("utf-8")
def ndjson_stream_835(result: ParseResult835) -> Iterator[bytes]:
"""Yield one JSON object per line: envelope → financial → trace → payer → payee → claim_payments → summary."""
def ndjson_stream_835(
result: ParseResult835, batch_id: str | None = None,
) -> Iterator[bytes]:
"""Yield one JSON object per line: envelope → financial → trace → payer → payee → claim_payments → summary.
See ``ndjson_stream_837`` for why the optional ``batch_id`` is
merged into the summary event.
"""
yield (json.dumps({"type": "envelope", "data": json.loads(result.envelope.model_dump_json())}) + "\n").encode("utf-8")
yield (json.dumps({"type": "financial_info", "data": json.loads(result.financial_info.model_dump_json())}) + "\n").encode("utf-8")
yield (json.dumps({"type": "trace", "data": json.loads(result.trace.model_dump_json())}) + "\n").encode("utf-8")
@@ -113,7 +134,10 @@ def ndjson_stream_835(result: ParseResult835) -> Iterator[bytes]:
yield (json.dumps({"type": "payee", "data": json.loads(result.payee.model_dump_json())}) + "\n").encode("utf-8")
for claim in result.claims:
yield (json.dumps({"type": "claim_payment", "data": json.loads(claim.model_dump_json())}) + "\n").encode("utf-8")
yield (json.dumps({"type": "summary", "data": json.loads(result.summary.model_dump_json())}) + "\n").encode("utf-8")
summary_data = json.loads(result.summary.model_dump_json())
if batch_id is not None:
summary_data["batch_id"] = batch_id
yield (json.dumps({"type": "summary", "data": summary_data}) + "\n").encode("utf-8")
def strict_rewrite_837(result: ParseResult) -> ParseResult:
+57 -1
View File
@@ -1 +1,57 @@
"""Resource-group routers. Imported and registered by ``cyclone.api``."""
"""Per-resource FastAPI routers.
`api.py` does `for r in routers: app.include_router(r)`. New
routers register themselves here in alphabetical order. Helpers
shared by 2+ routers live in `_shared.py` (private to the package).
Every router except ``health`` declares its own
``APIRouter(dependencies=[Depends(matrix_gate)])`` — keep that
invariant here so adding a new router doesn't silently miss the gate.
"""
from fastapi import APIRouter
from cyclone.api_routers import (
acks,
activity,
admin,
batches,
claim_acks,
claims,
clearhouse,
config,
dashboard,
eligibility,
health,
inbox,
parse,
payers,
providers,
reconciliation,
remittances,
submission,
ta1_acks,
)
routers: list[APIRouter] = [
acks.router, # gated
activity.router, # gated
admin.router, # gated
batches.router, # gated
claim_acks.router, # gated
claims.router, # gated
clearhouse.router, # gated
config.router, # gated
dashboard.router, # gated
eligibility.router, # gated
health.router, # public — health probes must work pre-auth
inbox.router, # gated
parse.router, # gated
payers.router, # gated
providers.router, # gated
reconciliation.router, # gated
remittances.router, # gated
submission.router, # gated
ta1_acks.router, # gated
]
__all__ = ["routers"]
+273
View File
@@ -0,0 +1,273 @@
"""Cross-router helpers for the api_routers package.
Private to the package (leading underscore). Only routers in this
package import from here. Helpers graduate here when at least two
routers need them — single-router helpers stay in the router that
uses them.
Helpers currently promoted here:
- :func:`_actor_user_id` — promoted early (during SP36 Task 11
/ clearhouse extraction) because the clearhouse router needs
it and the 2 remaining call-sites in ``api.py`` (parse-999 ack
block, parse-277ca ack block) are both inside the parse
surface that Task 16 will extract. Promoting now is cheaper
than leaving a cross-module ``from cyclone.api import _actor_user_id``
that would create an import-cycle at registry load time.
- :data:`PAYER_FACTORIES` / :data:`PAYER_FACTORIES_835` — SP36
Task 16: payer config dicts lifted from ``api.py`` alongside
the ``_resolve_payer`` / ``_resolve_payer_835`` helpers that
consume them. The two helpers each touch a single ``PAYER_FACTORIES*``
dict; keeping both halves of the pair in one module removes a
circular import (parse.py → _shared._resolve_payer → api.PAYER_FACTORIES).
- :func:`_resolve_payer` / :func:`_resolve_payer_835` — used by
``parse-837`` and ``parse-835`` endpoints respectively. Promoted
in SP36 Task 16 alongside the PAYER_FACTORIES dicts.
- :func:`_transaction_set_id_from_segments` — used by
``parse-837`` and ``parse-835`` envelope guards. Promoted in
SP36 Task 16.
- :func:`_build_and_persist_ack` — used by ``parse-837`` (when
``?ack=true``). Promoted in SP36 Task 16.
- :func:`_reconciliation_summary_for_batch` — used by
``parse-835``. Promoted in SP36 Task 16.
- :func:`_ta1_synthetic_source_batch_id` — used by ``parse-ta1``.
Promoted in SP36 Task 16.
- :func:`_serialize_ta1` — used by ``parse-ta1`` to build the
raw TA1 round-trip text. Promoted in SP36 Task 16 per the
plan's "8 helpers" specification; technically a single-router
helper per D4 but moved here for symmetry with the other parse
serializers.
"""
from __future__ import annotations
import json
import logging
from typing import Any
from fastapi import HTTPException, Request
from cyclone import db
from cyclone.parsers.batch_ack_builder import build_ack_for_batch
from cyclone.parsers.payer import PayerConfig, PayerConfig835
from cyclone.parsers.serialize_999 import serialize_999
from cyclone.store import store
log = logging.getLogger(__name__)
# --------------------------------------------------------------------------- #
# Actor user id (SP36 Task 11 — early-promoted)
# --------------------------------------------------------------------------- #
def _actor_user_id(request: Request) -> int | None:
"""Return the acting user's id from ``request.state.user``, or None.
``get_current_user``/``matrix_gate`` populate ``request.state.user``
for both the authenticated path and the AUTH_DISABLED escape hatch.
Returns None when the state hasn't been set (e.g. background jobs
or unit tests that bypass auth). Used to stamp ``user_id`` onto
audit events without crashing the request.
"""
user = getattr(request.state, "user", None)
if user is None:
return None
return getattr(user, "id", None)
# --------------------------------------------------------------------------- #
# Payer config dicts (SP36 Task 16)
# --------------------------------------------------------------------------- #
#
# Mirror cli._PAYER_FACTORIES. Kept here (not in the parse router) so
# the ``_resolve_payer`` / ``_resolve_payer_835`` helpers can import
# their backing dicts without a circular import.
PAYER_FACTORIES: dict[str, Any] = {
"co_medicaid": PayerConfig.co_medicaid,
"generic_837p": PayerConfig.generic_837p,
}
PAYER_FACTORIES_835: dict[str, Any] = {
"co_medicaid_835": PayerConfig835.co_medicaid_835,
"generic_835": PayerConfig835.generic_835,
}
# --------------------------------------------------------------------------- #
# Payer resolution (SP36 Task 16)
# --------------------------------------------------------------------------- #
def _resolve_payer(name: str) -> PayerConfig:
if name not in PAYER_FACTORIES:
raise HTTPException(
status_code=400,
detail={
"error": "Unknown payer",
"detail": f"Unknown payer {name!r}. Choose from: {', '.join(PAYER_FACTORIES)}",
},
)
return PAYER_FACTORIES[name]()
def _resolve_payer_835(name: str) -> PayerConfig835:
if name not in PAYER_FACTORIES_835:
raise HTTPException(
status_code=400,
detail={
"error": "Unknown payer",
"detail": f"Unknown payer {name!r}. Choose from: {', '.join(PAYER_FACTORIES_835)}",
},
)
return PAYER_FACTORIES_835[name]()
# --------------------------------------------------------------------------- #
# Envelope detection (SP36 Task 16)
# --------------------------------------------------------------------------- #
def _transaction_set_id_from_segments(segments: list[list[str]]) -> str | None:
"""Return the ST01 transaction-set id (``"837"``, ``"835"``, ``"999"``...).
SP35 helper: scans the first few tokenized segments for the ST
segment and returns its second element (ST01). Returns None when no
ST is present — e.g. a TA1 file, which uses the bare TA1 segment
and no ST envelope. The endpoint-level envelope guards treat
``None`` as "no ST found; let the parser decide" so TA1 files
routed through the wrong endpoint still surface a parse error
rather than a misleading "expected 837p, got ''" message.
"""
for seg in segments[:5]: # ST is always the second segment after ISA
if seg and seg[0] == "ST" and len(seg) > 1:
return seg[1]
return None
# --------------------------------------------------------------------------- #
# 999 ACK builder (SP36 Task 16)
# --------------------------------------------------------------------------- #
def _build_and_persist_ack(batch_id: str) -> dict | None:
"""Build a 999 ACK for ``batch_id`` and persist the row.
Returns the ack payload dict (matches the ``/api/parse-999``
response shape so the JSON and NDJSON clients can share the
schema) or None if the build failed. The build is fail-soft:
errors are logged but never abort the 837 ingest, because the
user-visible 837 result is still correct.
"""
try:
ack_result = build_ack_for_batch(batch_id)
except Exception:
log.exception("build_ack_for_batch failed for %s", batch_id)
return None
fg = ack_result.functional_group_acks[0] if ack_result.functional_group_acks else None
if fg is None:
return None
raw_text = serialize_999(ack_result, interchange_control_number=ack_result.envelope.control_number)
row = store.add_ack(
source_batch_id=batch_id,
accepted_count=fg.accepted_count,
rejected_count=fg.rejected_count,
received_count=fg.received_count,
ack_code=fg.ack_code,
raw_json=json.loads(ack_result.model_dump_json()),
)
return {
"id": row.id,
"accepted_count": fg.accepted_count,
"rejected_count": fg.rejected_count,
"received_count": fg.received_count,
"ack_code": fg.ack_code,
"source_batch_id": batch_id,
"raw_999_text": raw_text,
}
# --------------------------------------------------------------------------- #
# Reconciliation summary (SP36 Task 16)
# --------------------------------------------------------------------------- #
def _reconciliation_summary_for_batch(batch_id: str) -> dict:
"""Return ``{matched, unmatched_claims, unmatched_remittances, skipped}`` for a batch.
Reads from the DB after ``store.add()`` has already run reconciliation
synchronously (SP27 Task 10: ``reconcile.run(s, batch_id)`` inside the
ingest session, before commit). Counts are observed at this moment;
a subsequent manual match/unmatch will not be reflected until the
next request.
``skipped`` is reserved for future use — the orchestrator tracks
skipped claims internally but does not surface a queryable count.
"""
from sqlalchemy import func, select
from cyclone.db import Match, Remittance
with db.SessionLocal()() as s:
matched = s.execute(
select(func.count(Match.id)).where(
Match.remittance_id.in_(
select(Remittance.id).where(Remittance.batch_id == batch_id)
)
)
).scalar_one()
# Pull unmatched via the store (small result set; cheap).
unmatched = store.list_unmatched(kind="both")
return {
"matched": matched,
"unmatched_claims": len(unmatched["claims"]),
"unmatched_remittances": len(unmatched["remittances"]),
"skipped": 0, # reserved — T10 does not persist a skipped count
}
# --------------------------------------------------------------------------- #
# TA1 synthetic source batch id (SP36 Task 16)
# --------------------------------------------------------------------------- #
def _ta1_synthetic_source_batch_id(interchange_control_number: str) -> str:
"""Return a synthetic ``batches.id`` for a received TA1 with no source batch.
Mirrors :func:`_ack_synthetic_source_batch_id` (in
``cyclone.handlers._ack_id``). The ta1_acks.source_batch_id
FK requires a row in batches; for received TA1s we synthesize an
id of the form ``TA1-<ISA13>``. The row is NOT created in batches
(same FK-is-no-op convention as the 999 path).
"""
return f"TA1-{(interchange_control_number or '').strip() or '000000001'}"
# --------------------------------------------------------------------------- #
# TA1 serializer (SP36 Task 16)
# --------------------------------------------------------------------------- #
def _serialize_ta1(result) -> str:
"""Render a TA1 file from a ParseResultTa1 for the ``raw_ta1_text`` field.
Mirrors what the parser consumed: ISA envelope → TA1 → IEA. We
rebuild minimal ISA fields from the envelope plus the TA1 segment
verbatim. The serializer is intentionally tiny — TA1 has no GS/ST,
so there's no functional-group structure to round-trip.
"""
ta1 = result.ta1
parts = [
f"TA1*{ta1.control_number}*{ta1.interchange_date.strftime('%y%m%d') if ta1.interchange_date else ''}*"
f"{ta1.interchange_time or ''}*{ta1.ack_code}*{ta1.note_code or ''}*"
f"{ta1.ack_generated_date.strftime('%y%m%d') if ta1.ack_generated_date else ''}",
]
return "~".join(parts) + "~"
+154 -35
View File
@@ -1,64 +1,86 @@
"""``/api/acks`` — list & detail endpoints for the 999 ACK inbox.
"""``/api/acks`` and ``/api/277ca-acks`` — list, detail, and live-tail.
These are the persisted acknowledgment rows produced by
999 ACKs are the persisted acknowledgment rows produced by
``POST /api/parse-999``. The frontend ``useAcks`` hook re-shapes the
list payload to its ``Ack`` interface in ``src/types/index.ts``.
The detail endpoint returns the full ``raw_json`` payload plus the
regenerated ``raw_999_text`` so the UI can show "view source" without a
second round-trip.
277CA ACKs are the persisted Claim Acknowledgment rows produced by
``POST /api/parse-277ca``. The frontend ``useAcks`` hook treats them
as a second source alongside 999 — the row shape is normalised in
``cyclone.store.ui.to_ui_two77ca_ack``.
The 999 detail endpoint returns the full ``raw_json`` payload plus
the regenerated ``raw_999_text`` so the UI can show "view source"
without a second round-trip.
SP25: ``/api/acks/stream`` joins the live-tail triplet — the Acks
page mounts ``useTailStream("acks")`` and ``useMergedTail("acks", …)``
to see new 999 acks the moment they land (whether from the SFTP
poller or a manual upload).
"""
from __future__ import annotations
import logging
from typing import Any
from typing import Any, AsyncIterator
from fastapi import APIRouter, HTTPException, Query, Request
from fastapi import APIRouter, Depends, HTTPException, Query, Request
from fastapi.responses import StreamingResponse
from cyclone.api_helpers import ndjson_stream_list, wants_ndjson
from cyclone import db
from cyclone.api_helpers import (
ndjson_line,
ndjson_stream_list,
tail_events,
wants_ndjson,
)
from cyclone.auth.deps import matrix_gate
from cyclone.parsers.models_999 import ParseResult999
from cyclone.parsers.serialize_999 import serialize_999
from cyclone.store import store
from cyclone.pubsub import EventBus
from cyclone.store import store, to_ui_ack, to_ui_two77ca_ack
router = APIRouter()
router = APIRouter(dependencies=[Depends(matrix_gate)])
log = logging.getLogger(__name__)
def _ack_to_ui(row) -> dict:
"""Map an ``Ack`` ORM row to the UI shape used by ``/api/acks``.
Field names match the rest of the Cyclone API (snake_case). The
frontend ``useAcks`` hook re-shapes this to the camelCase ``Ack``
interface in ``src/types/index.ts``.
"""
return {
"id": row.id,
"source_batch_id": row.source_batch_id,
"accepted_count": row.accepted_count,
"rejected_count": row.rejected_count,
"received_count": row.received_count,
"ack_code": row.ack_code,
"parsed_at": (
row.parsed_at.isoformat().replace("+00:00", "Z")
if row.parsed_at is not None
else ""
),
}
# SP25: ``_ack_to_ui`` moved to ``cyclone.store.ui.to_ui_ack`` so the
# live-tail event payload (``ack_received``) matches the list endpoint
# shape byte-for-byte.
@router.get("/api/acks")
def list_acks_endpoint(
request: Request,
limit: int = Query(100, ge=1, le=1000),
limit: int = Query(100, ge=1, le=5000),
offset: int = Query(0, ge=0),
) -> Any:
"""Return the list of persisted 999 ACKs, newest first."""
"""Return the list of persisted 999 ACKs, newest first.
``limit`` caps the page size; ``offset`` lets the UI walk the
full set without holding it all in memory. ``aggregates`` is
summed over the *full* row set (not the page) so the KPI strip
on the Acks page reflects every persisted 999, not just the
visible 50. Without server-side aggregates the page would
silently under-report (silent-failure mode) once the row count
exceeds the page size.
"""
rows = store.list_acks()
items = [_ack_to_ui(r) for r in rows[:limit]]
items = [to_ui_ack(r) for r in rows[offset : offset + limit]]
total = len(rows)
returned = len(items)
has_more = total > returned
has_more = offset + returned < total
aggregates = {
"accepted_count": sum(r.accepted_count or 0 for r in rows),
"rejected_count": sum(r.rejected_count or 0 for r in rows),
"received_count": sum(r.received_count or 0 for r in rows),
}
# SP28: batch-fetch linked_claim_ids per ack row in one query to
# avoid N+1 — see ``find_linked_claim_ids_for_acks`` below.
ack_ids = [r.id for r in rows]
linked_map = _find_linked_claim_ids_for_acks(ack_ids, kind="999")
for item, aid in zip(items, ack_ids[offset : offset + limit]):
item["linked_claim_ids"] = linked_map.get(aid, [])
if wants_ndjson(request):
return StreamingResponse(
ndjson_stream_list(items, total, returned, has_more),
@@ -69,9 +91,68 @@ def list_acks_endpoint(
"total": total,
"returned": returned,
"has_more": has_more,
"aggregates": aggregates,
}
def _find_linked_claim_ids_for_acks(
ack_ids: list[int], *, kind: str
) -> dict[int, list[str]]:
"""Batch-fetch {ack_id: [claim_id, …]} for the listed ack rows.
One SELECT against ``claim_acks`` keyed on the page's ack_ids —
avoids an N+1 round-trip when the page renders the
"🔗 N claims" badge per row. Used by the 999 / TA1 / 277CA
list endpoints. Returns a ``{ack_id: [claim_id]}`` map; acks
with no links map to ``[]`` (default).
"""
out: dict[int, list[str]] = {aid: [] for aid in ack_ids}
if not ack_ids:
return out
with db.SessionLocal()() as s:
rows = (
s.query(db.ClaimAck.ack_id, db.ClaimAck.claim_id)
.filter(
db.ClaimAck.ack_kind == kind,
db.ClaimAck.ack_id.in_(ack_ids),
db.ClaimAck.claim_id.isnot(None),
)
.all()
)
for ack_id, claim_id in rows:
out[ack_id].append(claim_id)
return out
@router.get("/api/acks/stream")
async def acks_stream(
request: Request,
limit: int = Query(100, ge=1, le=1000),
) -> StreamingResponse:
"""Stream 999 ACKs as NDJSON: snapshot first, then live events.
SP25: this endpoint joins the live-tail triplet — subscribes to
``ack_received`` and emits one ``item`` per snapshot row plus a
single ``snapshot_end`` line, then forwards live events from
the bus. Matches the wire format used by ``/api/claims/stream``,
``/api/remittances/stream``, and ``/api/activity/stream``.
NOTE: registered BEFORE ``/api/acks/{ack_id}`` so the literal
``stream`` path segment doesn't get matched as an ack id.
"""
bus: EventBus = request.app.state.event_bus
async def gen() -> AsyncIterator[bytes]:
rows = store.list_acks()[:limit]
for row in rows:
yield ndjson_line({"type": "item", "data": to_ui_ack(row)})
yield ndjson_line({"type": "snapshot_end", "data": {"count": len(rows)}})
async for chunk in tail_events(request, bus, ["ack_received"]):
yield chunk
return StreamingResponse(gen(), media_type="application/x-ndjson")
@router.get("/api/acks/{ack_id}")
def get_ack_endpoint(ack_id: int) -> dict:
"""Return one persisted ACK row with its parsed detail.
@@ -86,7 +167,7 @@ def get_ack_endpoint(ack_id: int) -> dict:
status_code=404,
detail={"error": "Not found", "detail": f"Ack {ack_id} not found"},
)
body = _ack_to_ui(row)
body = to_ui_ack(row)
body["raw_json"] = row.raw_json
# Regenerate the X12 text from raw_json so the operator can download
# the actual 999 file. (SP3 P3 follow-up: list endpoint doesn't carry
@@ -102,3 +183,41 @@ def get_ack_endpoint(ack_id: int) -> dict:
else:
body["raw_999_text"] = None
return body
# -- 277CA ACKs -------------------------------------------------------------
# SP36 Task 2: these two endpoints moved here from api.py (lines 1278-1318).
# 277CA ACKs share the same shape-of-life contract as 999 ACKs: persisted by
# a parse endpoint, listed newest-first, detail returns raw_json so the UI
# can show "view source" without a round-trip.
@router.get("/api/277ca-acks")
def list_277ca_acks_endpoint(
limit: int = Query(100, ge=1, le=5000),
) -> Any:
"""Return the list of persisted 277CA ACKs, newest first.
SP28: each item gains ``linked_claim_ids`` (batch-fetched via
the shared ``_find_linked_claim_ids_for_acks`` helper — one
SELECT keyed on ``ack_kind``, no N+1) so the Acks page row
can render the "🔗 N claims" badge inline.
"""
rows = store.list_277ca_acks()
items = [to_ui_two77ca_ack(r) for r in rows[:limit]]
ack_ids = [r.id for r in rows]
linked_map = _find_linked_claim_ids_for_acks(ack_ids, kind="277ca")
for item, aid in zip(items, ack_ids[:limit]):
item["linked_claim_ids"] = linked_map.get(aid, [])
return {"total": len(rows), "items": items}
@router.get("/api/277ca-acks/{ack_id}")
def get_277ca_ack_endpoint(ack_id: int) -> dict:
"""Return one persisted 277CA ACK row with its parsed detail."""
row = store.get_277ca_ack(ack_id)
if row is None:
raise HTTPException(status_code=404, detail=f"277CA ACK {ack_id} not found")
body = to_ui_two77ca_ack(row)
body["raw_json"] = row.raw_json
return body
+102
View File
@@ -0,0 +1,102 @@
"""``/api/activity`` and ``/api/activity/stream`` — operator-facing event log.
Two endpoints, both gated by ``matrix_gate``:
- ``GET /api/activity`` — paginated event list with ``kind`` /
``since`` filters, plus an NDJSON variant when the caller sends
``Accept: application/x-ndjson``.
- ``GET /api/activity/stream`` — NDJSON live-tail: snapshot of the
most recent N events, then ``activity_recorded`` events as they
hit the store. Default ``limit`` is 50 (smaller than the list
endpoint's 200) because activity is high-volume — callers usually
want the most recent handful, not a full replay.
The snapshot halves of ``/api/activity`` and ``/api/activity/stream``
share the same in-memory filter logic (``kind`` + ``since``) so the
two endpoints are interchangeable for the snapshot half.
SP36 Task 10: this block moved here from ``api.py:2606`` (the
``/api/activity*`` pair).
"""
from __future__ import annotations
from typing import Any, AsyncIterator
from fastapi import APIRouter, Depends, Query, Request
from fastapi.responses import StreamingResponse
from cyclone.api_helpers import (
ndjson_line as _ndjson_line,
ndjson_stream_list as _ndjson_stream_list,
tail_events as _tail_events,
wants_ndjson as _wants_ndjson,
)
from cyclone.auth.deps import matrix_gate
from cyclone.pubsub import EventBus
from cyclone.store import store
router = APIRouter(dependencies=[Depends(matrix_gate)])
@router.get("/api/activity")
def list_activity(
request: Request,
kind: str | None = Query(None),
since: str | None = Query(None),
limit: int = Query(200, ge=1, le=5000),
) -> Any:
events = store.recent_activity(limit=limit)
if kind is not None:
events = [e for e in events if e["kind"] == kind]
if since is not None:
events = [e for e in events if e["timestamp"] >= since]
total = len(events)
has_more = False
if _wants_ndjson(request):
return StreamingResponse(
_ndjson_stream_list(events, total, total, has_more),
media_type="application/x-ndjson",
)
return {
"items": events,
"total": total,
"returned": total,
"has_more": has_more,
}
@router.get("/api/activity/stream")
async def activity_stream(
request: Request,
kind: str | None = Query(None),
since: str | None = Query(None),
limit: int = Query(50, ge=1, le=5000),
) -> StreamingResponse:
"""Stream Activity events as NDJSON: snapshot first, then live events.
Subscribes to ``activity_recorded``. Default ``limit`` is 50
(smaller than the list endpoint's 200) because activity is
high-volume — callers usually want the most recent handful, not a
full replay.
"""
bus: EventBus = request.app.state.event_bus
async def gen() -> AsyncIterator[bytes]:
# Snapshot reuses the same in-memory filter as ``list_activity``
# so the two endpoints are interchangeable for the snapshot
# half.
events = store.recent_activity(limit=limit)
if kind is not None:
events = [e for e in events if e["kind"] == kind]
if since is not None:
events = [e for e in events if e["timestamp"] >= since]
for ev in events:
yield _ndjson_line({"type": "item", "data": ev})
yield _ndjson_line({
"type": "snapshot_end", "data": {"count": len(events)},
})
async for chunk in _tail_events(request, bus, ["activity_recorded"]):
yield chunk
return StreamingResponse(gen(), media_type="application/x-ndjson")
+780 -12
View File
@@ -1,23 +1,44 @@
"""``/api/admin/validate-provider`` — NPI + Tax ID liveness probe (SP20).
"""``/api/admin/*`` — operator-only endpoints.
Pure read-only endpoint that runs the local NPI Luhn + EIN format checks
without touching the DB. Useful for:
- operators vetting a new provider before adding them to the registry,
- the dashboard's "validate" button on a Provider row,
- smoke-testing the SP20 checks after a deploy.
The /api/admin namespace covers:
Both query params are optional; omitting one just skips that check.
Returns the per-check result dict so the caller can distinguish "bad
format" from "bad checksum".
* **audit-log** (SP11): list + chain-verify the tamper-evident log
* **db/rotate-key**: SQLCipher key rotation (SP12)
* **backup**: create / list / status / verify / restore / prune / scheduler (SP15)
* **scheduler**: backup & main scheduler control + processed-files log
* **reload-config**: hot-reload ``config/payers.yaml`` after edits
* **validate-provider**: NPI + Tax ID liveness probe (SP20)
All routes on this router are gated by ``matrix_gate`` — declared
once on the ``APIRouter`` constructor. ``/api/admin/validate-provider``
lives here too because it's an admin-shaped read-only probe.
SP36 Task 3: the 20 admin endpoints formerly in ``cyclone.api``
(audit-log ×2, db/rotate-key ×1, backup ×10, scheduler ×6,
reload-config ×1) moved here from ``api.py:3321-4052`` and
``api.py:4269-4276``. The non-admin ``/api/config/*`` and
``/api/payers/{id}/summary`` routes that bracketed those blocks
stay in ``api.py`` for now — they're extracted in Tasks 5 & 6.
"""
from __future__ import annotations
from fastapi import APIRouter, Query
import json
import logging
import threading
import time
from typing import Any
from fastapi import APIRouter, Depends, HTTPException, Query
from cyclone import db
from cyclone.audit_log import verify_chain
from cyclone.auth.deps import matrix_gate
from cyclone.clearhouse import InboundFile
from cyclone.npi import is_valid_npi, is_valid_tax_id, normalize_tax_id
router = APIRouter(dependencies=[Depends(matrix_gate)])
router = APIRouter()
log = logging.getLogger(__name__)
@router.get("/api/admin/validate-provider")
@@ -57,4 +78,751 @@ def validate_provider(
"normalized": normalized,
}
return result
return result
# --------------------------------------------------------------------------- #
# SP36 Task 3: the 20 admin endpoints below were moved here from api.py. #
# --------------------------------------------------------------------------- #
# --------------------------------------------------------------------------- #
# SP11: tamper-evident audit log (admin)
# --------------------------------------------------------------------------- #
@router.get("/api/admin/audit-log")
def list_audit_log_endpoint(
entity_type: str | None = Query(default=None),
entity_id: str | None = Query(default=None),
event_type: str | None = Query(default=None),
limit: int = Query(default=100, ge=1, le=1000),
) -> Any:
"""List audit-log rows, newest first, with optional filters.
Filters match the (entity_type, entity_id) pair (typical use:
"show me everything that happened to claim C-123") or a single
event_type (typical use: "show me all clearhouse.submitted
events today").
"""
with db.SessionLocal()() as s:
q = s.query(db.AuditLog)
if entity_type:
q = q.filter(db.AuditLog.entity_type == entity_type)
if entity_id:
q = q.filter(db.AuditLog.entity_id == entity_id)
if event_type:
q = q.filter(db.AuditLog.event_type == event_type)
rows = q.order_by(db.AuditLog.id.desc()).limit(limit).all()
return {
"total": len(rows),
"items": [
{
"id": r.id,
"event_type": r.event_type,
"entity_type": r.entity_type,
"entity_id": r.entity_id,
"actor": r.actor,
"payload": json.loads(r.payload_json) if r.payload_json else None,
"created_at": r.created_at.isoformat() if r.created_at else None,
"prev_hash": r.prev_hash,
"hash": r.hash,
}
for r in rows
],
}
@router.get("/api/admin/audit-log/verify")
def verify_audit_log_endpoint() -> Any:
"""Walk the audit-log chain and verify every row's hash.
Returns ``{"ok": true, "checked": N}`` for a clean chain, or
``{"ok": false, "checked": K, "first_bad_id": X, "reason": "..."}``
for a broken chain. This is the operator's "did anyone tamper?"
endpoint; run it on demand or via a nightly cron job.
"""
with db.SessionLocal()() as s:
result = verify_chain(s)
return {
"ok": result.ok,
"checked": result.checked,
"first_bad_id": result.first_bad_id,
"reason": result.reason,
}
# ---------------------------------------------------------------------------
# SP15: SQLCipher key rotation
#
# Re-encrypts the DB in place with a fresh key, then updates the
# Keychain so subsequent connections open with the new key. This is
# a 1-time operation per rotation; for routine read/write the rest
# of the API is unchanged.
#
# Concurrency: the rotation holds a module-level lock so two
# concurrent requests can't race and end up with mismatched Keychain
# + DB. The lock is a simple threading.Lock; a process restart
# resets it (intentional — the operator's next start-up opens with
# whatever key is in the Keychain).
# ---------------------------------------------------------------------------
from cyclone import db_crypto as _db_crypto
from cyclone import secrets as _secrets
_db_rotate_lock = threading.Lock()
@router.post("/api/admin/db/rotate-key")
def rotate_db_key_endpoint(body: dict | None = None) -> Any:
"""Generate a fresh DB key, re-encrypt the DB, update the Keychain.
Request body (optional):
actor: who initiated the rotation. Defaults to "operator".
reason: human-readable reason. Written to the audit log.
Returns:
``{ok, old_fingerprint, new_fingerprint, rotated_at, table_count}``
on success. On failure (DB not encrypted, rekey failed,
Keychain update failed) returns the same shape with
``ok=false`` and a ``reason``. HTTP 503 is returned if the
rekey fails or encryption is not enabled.
The Keychain write happens *after* the rekey succeeds. If the
Keychain write fails, the DB has the new key but the Keychain
still has the old one — the endpoint returns 503 with a
"keychain update failed" reason and the operator must restore
the old key manually (``cyclone db restore-key <old_key>``) to
avoid being locked out.
"""
body = body or {}
actor = body.get("actor") or "operator"
reason = body.get("reason") or ""
if not _db_crypto.is_encryption_enabled():
raise HTTPException(
status_code=400,
detail="encryption not enabled (sqlcipher3 missing or no Keychain key)",
)
# Acquire the lock; non-blocking so a stuck rotation doesn't
# silently hold up other requests.
if not _db_rotate_lock.acquire(blocking=False):
raise HTTPException(
status_code=409,
detail="another key rotation is in progress",
)
try:
url = db._resolve_url()
old_key = _db_crypto.get_db_key()
if not old_key:
raise HTTPException(
status_code=400,
detail="no DB key in Keychain; cannot rotate",
)
new_key = _db_crypto.generate_db_key()
result = _db_crypto.rotate_db_key(
url=url, old_key=old_key, new_key=new_key,
)
if not result.ok:
# Rekey failed. The DB still has the old key. The
# Keychain is unchanged. Caller should NOT retry with
# the same new key (it's lost); generate a fresh one.
log.error("SQLCipher rotate failed: %s", result.reason)
raise HTTPException(
status_code=503,
detail={
"ok": False,
"old_fingerprint": result.old_fingerprint,
"new_fingerprint": result.new_fingerprint,
"rotated_at": result.rotated_at,
"reason": result.reason,
},
)
# Rekey succeeded. Now update the Keychain. If this fails
# the DB is locked behind the new key — operator must
# restore the old key manually.
if not _secrets.set_secret(_db_crypto.KEYCHAIN_ACCOUNT, new_key):
log.error("Keychain update failed after successful rekey!")
raise HTTPException(
status_code=503,
detail={
"ok": False,
"old_fingerprint": result.old_fingerprint,
"new_fingerprint": result.new_fingerprint,
"rotated_at": result.rotated_at,
"reason": (
"rekey succeeded but Keychain update failed — "
"the DB is now encrypted with the new key but "
"the Keychain still has the old one. "
"Restore the old key to the Keychain to recover."
),
},
)
# Store the old key in the "previous" account for a grace
# period so the operator can roll back if they discover the
# new key is broken (e.g. the Keychain entry got truncated).
_secrets.set_secret(_db_crypto.KEYCHAIN_ACCOUNT_PREVIOUS, old_key)
# Rebuild the engine so subsequent connections use the new
# key. dispose_engine() closes every pooled connection that
# was using the old key; init_db() opens new ones with the
# new key from the (now-updated) Keychain.
db.reinit_engine()
# Audit log the rotation. We do this after the engine is
# rebuilt so the audit event is written with the new key —
# proving that the new key works for new writes.
try:
from cyclone.audit_log import append_event, AuditEvent
with db.SessionLocal()() as s:
append_event(s, AuditEvent(
event_type="db.key_rotated",
entity_type="database",
entity_id="cyclone.db",
actor=actor,
payload={
"old_fingerprint": result.old_fingerprint,
"new_fingerprint": result.new_fingerprint,
"table_count": result.table_count,
"reason": reason,
},
))
s.commit()
except Exception as exc: # noqa: BLE001
# Audit append is best-effort; rotation already succeeded.
log.warning("could not write audit event for rotation: %s", exc)
return {
"ok": True,
"old_fingerprint": result.old_fingerprint,
"new_fingerprint": result.new_fingerprint,
"rotated_at": result.rotated_at,
"table_count": result.table_count,
}
finally:
_db_rotate_lock.release()
# ---------------------------------------------------------------------------
# SP17: encrypted DB backups (admin)
#
# The actual encryption + lifecycle lives in :mod:`cyclone.backup` and
# :mod:`cyclone.backup_service`. The scheduler (separate from the
# MFT scheduler) lives in :mod:`cyclone.backup_scheduler`. These
# endpoints expose the operator's manual controls plus a tick for
# "take a backup right now."
#
# Restore is intentionally two-step: an idle browser tab can't nuke
# the live DB. The first call returns a ``restore_token`` (a one-shot
# 64-char hex) and a preview of the backup's fingerprint + table
# count plus the live DB's. The second call with the token performs
# the actual swap.
# ---------------------------------------------------------------------------
from cyclone import backup_service as _backup_svc_mod
from cyclone import backup_scheduler as _backup_sched_mod
def _backup_or_503():
try:
return _backup_svc_mod.get_backup_service()
except RuntimeError as exc:
raise HTTPException(status_code=503, detail=str(exc))
@router.post("/api/admin/backup/create")
def backup_create() -> Any:
"""Take an encrypted backup right now. Returns the new backup metadata."""
from cyclone import audit_log as _audit
svc = _backup_or_503()
try:
result = svc.create_now()
except Exception as exc: # noqa: BLE001
# Surface a 503 with the reason so the operator sees what
# went wrong without grepping server logs.
raise HTTPException(
status_code=503,
detail=f"backup failed: {type(exc).__name__}: {exc}",
)
# Audit the create. Best-effort; failure here doesn't roll back
# the backup (already on disk).
try:
with db.SessionLocal()() as s:
_audit.append_event(s, _audit.AuditEvent(
event_type="db.backup_created",
entity_type="database",
entity_id="cyclone.db",
actor="operator",
payload={
"backup_id": result.backup.id,
"db_fingerprint": result.backup.db_fingerprint,
"table_count": result.backup.table_count,
"triggered_by": "api",
},
))
s.commit()
except Exception as exc: # noqa: BLE001
log.warning("could not write backup_created audit event: %s", exc)
return {
"ok": True,
"backup": {
"id": result.backup.id,
"filename": result.backup.filename,
"size_bytes": result.backup.size_bytes,
"db_fingerprint": result.backup.db_fingerprint,
"table_count": result.backup.table_count,
"created_at": result.backup.created_at.isoformat(),
"key_fingerprint": result.backup.key_fingerprint,
},
"sidecar": {
"format_version": result.sidecar.format_version,
"kdf": result.sidecar.kdf,
"kdf_iterations": result.sidecar.kdf_iterations,
"cipher": result.sidecar.cipher,
},
}
@router.get("/api/admin/backup/list")
def backup_list(
limit: int = Query(default=100, ge=1, le=1000),
status: str | None = Query(default=None),
) -> Any:
"""List ``db_backups`` rows, newest first. Filters by status."""
svc = _backup_or_503()
rows = svc.list_backups(limit=limit, status=status)
return {
"count": len(rows),
"files": [
{
"id": r.id,
"filename": r.filename,
"backup_dir": r.backup_dir,
"size_bytes": r.size_bytes,
"db_fingerprint": r.db_fingerprint,
"table_count": r.table_count,
"created_at": r.created_at.isoformat() if r.created_at else None,
"completed_at": r.completed_at.isoformat() if r.completed_at else None,
"status": r.status,
"error_message": r.error_message,
"key_fingerprint": r.key_fingerprint,
}
for r in rows
],
}
@router.get("/api/admin/backup/status")
def backup_status() -> Any:
"""Snapshot of the backup subsystem (counts, disk usage, last run)."""
svc = _backup_or_503()
snap = svc.status()
# Also include the BackupScheduler's snapshot if configured.
try:
sched = _backup_sched_mod.get_backup_scheduler()
snap["scheduler"] = sched.status().as_dict()
except RuntimeError:
snap["scheduler"] = None
return snap
@router.post("/api/admin/backup/{backup_id}/verify")
def backup_verify(backup_id: int) -> Any:
"""Decrypt + checksum-verify a backup against its sidecar."""
svc = _backup_or_503()
try:
v = svc.verify(backup_id)
except _backup_svc_mod.BackupError as exc:
raise HTTPException(status_code=404, detail=str(exc))
return {
"backup_id": v.backup_id,
"filename": v.filename,
"ok": v.ok,
"expected_fingerprint": v.expected_fingerprint,
"actual_fingerprint": v.actual_fingerprint,
"table_count": v.table_count,
"reason": v.reason,
}
@router.post("/api/admin/backup/{backup_id}/restore/initiate")
def backup_restore_initiate(backup_id: int) -> Any:
"""First step of the two-step restore. Returns a ``restore_token``."""
svc = _backup_or_503()
try:
init = svc.restore_initiate(backup_id)
except _backup_svc_mod.BackupError as exc:
raise HTTPException(status_code=400, detail=str(exc))
return {
"backup_id": init.backup_id,
"filename": init.filename,
"size_bytes": init.size_bytes,
"restore_token": init.restore_token,
"expires_at": init.expires_at.isoformat(),
"preview": {
"backup_db_fingerprint": init.db_fingerprint,
"backup_table_count": init.table_count,
"current_db_fingerprint": init.current_db_fingerprint,
"current_table_count": init.current_table_count,
},
"warning": (
"Confirming will dispose the live engine and replace the DB "
"file with the backup. In-flight requests will error. "
"Re-issue the call with the restore_token within 5 minutes."
),
}
@router.post("/api/admin/backup/{backup_id}/restore/confirm")
def backup_restore_confirm(
backup_id: int,
body: dict | None = None,
) -> Any:
"""Second step of the two-step restore. Performs the swap."""
body = body or {}
token = body.get("restore_token")
if not token or not isinstance(token, str):
raise HTTPException(
status_code=400,
detail="missing or invalid restore_token in request body",
)
actor = body.get("actor") or "operator"
svc = _backup_or_503()
try:
result = svc.restore_confirm(backup_id, token, actor=actor)
except _backup_svc_mod.BackupError as exc:
raise HTTPException(status_code=400, detail=str(exc))
# Audit the restore. Best-effort.
try:
from cyclone import audit_log as _audit
with db.SessionLocal()() as s:
_audit.append_event(s, _audit.AuditEvent(
event_type="db.backup_restored",
entity_type="database",
entity_id="cyclone.db",
actor=actor,
payload={
"backup_id": result.backup_id,
"filename": result.filename,
"restored_from_fingerprint": result.restored_from_fingerprint,
"new_db_fingerprint": result.new_db_fingerprint,
"restored_at": result.restored_at.isoformat(),
},
))
s.commit()
except Exception as exc: # noqa: BLE001
log.warning("could not write backup_restored audit event: %s", exc)
return {
"ok": True,
"backup_id": result.backup_id,
"filename": result.filename,
"restored_from_fingerprint": result.restored_from_fingerprint,
"restored_at": result.restored_at.isoformat(),
"new_db_fingerprint": result.new_db_fingerprint,
}
@router.post("/api/admin/backup/prune")
def backup_prune() -> Any:
"""Apply the retention policy now. Returns the deleted paths."""
from cyclone import audit_log as _audit
svc = _backup_or_503()
deleted = svc.prune()
actor = "operator"
if deleted:
try:
with db.SessionLocal()() as s:
_audit.append_event(s, _audit.AuditEvent(
event_type="db.backup_pruned",
entity_type="database",
entity_id="cyclone.db",
actor=actor,
payload={"deleted_paths": deleted},
))
s.commit()
except Exception as exc: # noqa: BLE001
log.warning("could not write backup_pruned audit event: %s", exc)
return {"ok": True, "deleted_count": len(deleted), "deleted_paths": deleted}
# ---------------------------------------------------------------------------
# SP17: backup scheduler (admin)
# ---------------------------------------------------------------------------
@router.post("/api/admin/backup/scheduler/start")
async def backup_scheduler_start() -> Any:
"""Begin the backup scheduler loop."""
try:
sched = _backup_sched_mod.get_backup_scheduler()
except RuntimeError as exc:
raise HTTPException(status_code=503, detail=str(exc))
await sched.start()
return {"status": sched.status().as_dict()}
@router.post("/api/admin/backup/scheduler/stop")
async def backup_scheduler_stop() -> Any:
"""Stop the backup scheduler loop."""
try:
sched = _backup_sched_mod.get_backup_scheduler()
except RuntimeError as exc:
raise HTTPException(status_code=503, detail=str(exc))
await sched.stop()
return {"status": sched.status().as_dict()}
@router.post("/api/admin/backup/scheduler/tick")
async def backup_scheduler_tick() -> Any:
"""Run one backup tick now (create + prune + audit)."""
try:
sched = _backup_sched_mod.get_backup_scheduler()
except RuntimeError as exc:
raise HTTPException(status_code=503, detail=str(exc))
result = await sched.tick()
return {"ok": result.ok, "tick": result.as_dict()}
# ---------------------------------------------------------------------------
# SP16: live MFT polling scheduler (admin)
#
# The scheduler lives in :mod:`cyclone.scheduler` and is configured by
# the lifespan handler. The endpoints below expose start / stop /
# one-shot tick / status / history so an operator (or a cron job)
# can drive the scheduler without touching the DB.
#
# Note: the scheduler is OFF by default. Auto-start is opt-in via
# ``CYCLONE_SCHEDULER_AUTOSTART=true`` at launch. These endpoints
# are the operator's manual controls.
# ---------------------------------------------------------------------------
from cyclone import scheduler as _scheduler_mod
def _scheduler_or_503():
"""Return the configured scheduler or raise 503."""
try:
return _scheduler_mod.get_scheduler()
except RuntimeError as exc:
raise HTTPException(status_code=503, detail=str(exc))
@router.post("/api/admin/scheduler/start")
async def scheduler_start() -> Any:
"""Begin polling the MFT inbound path every poll_interval_seconds."""
sched = _scheduler_or_503()
await sched.start()
return {"status": sched.status().as_dict()}
@router.post("/api/admin/scheduler/stop")
async def scheduler_stop() -> Any:
"""Stop polling. Waits up to 30s for the current tick to finish."""
sched = _scheduler_or_503()
await sched.stop()
return {"status": sched.status().as_dict()}
@router.post("/api/admin/scheduler/tick")
async def scheduler_tick() -> Any:
"""Run a single poll cycle synchronously and return the result.
Useful for: forcing a poll without waiting for the next interval;
verifying SFTP connectivity; running a one-shot import from the
CLI (``curl -X POST .../api/admin/scheduler/tick``).
"""
sched = _scheduler_or_503()
result = await sched.tick()
return {"ok": True, "tick": result.as_dict()}
@router.post("/api/admin/scheduler/pull-inbound")
async def scheduler_pull_inbound(
date: str = Query(
..., pattern=r"^\d{8}$",
description="Date filter as YYYYMMDD; only filenames whose 8-digit "
"timestamp (the 9th positional group in the inbound "
"filename) matches are downloaded and processed.",
),
file_types: str | None = Query(
default=None,
description="Optional comma-separated whitelist of file_types "
"(999, TA1, 277, 277CA, 835). Defaults to 999+TA1.",
),
limit: int = Query(default=2000, ge=1, le=10000),
) -> Any:
"""Targeted pull: list, filter to a date, download, and process.
Bypasses the alphabetical full-listing pass. Workflow:
1. ``SftpClient.list_inbound_names()`` — sub-second metadata-only
listing of the inbound MFT dir (skips ``*_warn.txt``).
2. Client-side filter: keep files whose 8-digit timestamp
substring equals ``date`` and whose ``file_type`` is in the
allowlist.
3. ``SftpClient.download_inbound(f)`` for each — fetches bytes
into the local cache.
4. ``Scheduler.process_inbound_files(files)`` — runs the same
per-file pipeline as a regular tick (already-processed files
are deduped via ``processed_inbound_files``).
Use this for the daily "process today's 999s" workflow without
paying the cost of downloading the full inbound set.
Returns ``{"ok": True, "summary": {...}}`` with
``listed / matched / downloaded / processed / skipped / errored``
counters and the date / file_type filters applied.
"""
from cyclone.clearhouse import SftpClient
from cyclone.edi.filenames import (
ALLOWED_FILE_TYPES,
parse_inbound_filename,
)
from cyclone.providers import SftpBlock
sched = _scheduler_or_503()
block: SftpBlock = sched._sftp_block # noqa: SLF001 — internal but stable
client = SftpClient(block)
if file_types:
wanted = {t.strip().upper() for t in file_types.split(",") if t.strip()}
unknown = wanted - ALLOWED_FILE_TYPES
if unknown:
raise HTTPException(
status_code=400,
detail=f"file_types {sorted(unknown)!r} not in "
f"{sorted(ALLOWED_FILE_TYPES)}",
)
else:
wanted = {"999", "TA1"} # daily default — what the operator needs
started = time.monotonic()
try:
# Single SFTP listdir — fast, no download.
all_files = await asyncio.to_thread(client.list_inbound_names)
except Exception as exc:
log.exception("SFTP list_inbound_names failed")
raise HTTPException(
status_code=502,
detail=f"SFTP list failed: {type(exc).__name__}: {exc}",
) from exc
listed = len(all_files)
matched: list[InboundFile] = []
for f in all_files:
if f.name.find(date) == -1:
continue
try:
parsed = parse_inbound_filename(f.name)
except ValueError:
continue
if parsed.file_type not in wanted:
continue
matched.append(f)
if len(matched) >= limit:
break
# Download in parallel-ish via to_thread (SftpClient serializes per
# connection; the overhead is dominated by the SFTP round trip).
downloaded = 0
download_errors: list[str] = []
for f in matched:
try:
await asyncio.to_thread(client.download_inbound, f)
downloaded += 1
except Exception as exc: # noqa: BLE001
log.warning("Failed to download %s: %s", f.name, exc)
download_errors.append(f"{f.name}: {type(exc).__name__}: {exc}")
# Hand off to the scheduler pipeline (idempotent; dedupes via
# processed_inbound_files).
tick = await sched.process_inbound_files(matched)
duration = round(time.monotonic() - started, 3)
return {
"ok": True,
"summary": {
"date": date,
"file_types": sorted(wanted),
"limit": limit,
"listed": listed,
"matched": len(matched),
"downloaded": downloaded,
"download_errors": download_errors,
"processed": tick.files_processed,
"skipped": tick.files_skipped,
"errored": tick.files_errored,
"duration_s": duration,
},
"tick": tick.as_dict(),
}
@router.get("/api/admin/scheduler/status")
def scheduler_status() -> Any:
"""Return the scheduler's runtime snapshot (running, counters, last tick)."""
sched = _scheduler_or_503()
return sched.status().as_dict()
@router.get("/api/admin/scheduler/processed-files")
def scheduler_processed_files(
limit: int = Query(default=100, ge=1, le=1000),
status: str | None = Query(default=None),
) -> Any:
"""List rows from ``processed_inbound_files``, newest first.
The operator's "what did the scheduler do?" view. Filters by
``status`` (``ok`` / ``error`` / ``skipped`` / ``pending``).
Returns ``{"count": N, "files": [...]}`` where ``files[i]``
matches the ORM row as a JSON dict.
"""
from cyclone.db import ProcessedInboundFile
from cyclone.scheduler import STATUS_OK, STATUS_ERROR, STATUS_SKIPPED, STATUS_PENDING
valid_statuses = {STATUS_OK, STATUS_ERROR, STATUS_SKIPPED, STATUS_PENDING}
if status is not None and status not in valid_statuses:
raise HTTPException(
status_code=400,
detail=f"status must be one of {sorted(valid_statuses)}",
)
with db.SessionLocal()() as s:
q = s.query(db.ProcessedInboundFile)
if status is not None:
q = q.filter(db.ProcessedInboundFile.status == status)
rows = q.order_by(db.ProcessedInboundFile.id.desc()).limit(limit).all()
return {
"count": len(rows),
"files": [
{
"id": r.id,
"sftp_block_name": r.sftp_block_name,
"name": r.name,
"size": r.size,
"modified_at": r.modified_at.isoformat() if r.modified_at else None,
"file_type": r.file_type,
"processed_at": r.processed_at.isoformat() if r.processed_at else None,
"parser_used": r.parser_used,
"claim_count": r.claim_count,
"status": r.status,
"error_message": r.error_message,
}
for r in rows
],
}
@router.post("/api/admin/reload-config")
def reload_config():
"""Re-read ``config/payers.yaml`` and revalidate. Returns counts."""
from cyclone import payers as payer_loader
try:
configs = payer_loader.load_payer_configs()
except ValueError as e:
raise HTTPException(status_code=400, detail=str(e))
return {"ok": True, "loaded": len(configs), "errors": []}
+515
View File
@@ -0,0 +1,515 @@
"""``/api/batches*`` — read views + ZIP export over the parsed-batch population.
Three endpoints, all gated by ``matrix_gate``:
- ``POST /api/batches/{batch_id}/export-837`` — download a ZIP of
regenerated X12 837 files for the requested claim ids. Per-claim
payer config + clearhouse identity drive the submitter/receiver
blocks; per-claim millisecond offset on the filename keeps every
file in the bundle unique (HCPF requires this). Read-only — does
NOT mutate Claim state (compare with
``/api/inbox/rejected/resubmit?download=true`` which DOES flip
``REJECTED → SUBMITTED``).
- ``GET /api/batches`` — summary list,
newest-first, capped at ``limit``. Each row includes ``claimIds``
(837P only, so the Upload page can render a one-click Re-export
button per row without a round-trip to ``/api/batches/{id}``).
SP30: also returns billing-outcome fields
(``acceptedCount`` / ``rejectedCount`` / ``pendingCount`` /
``billedTotal`` / ``topRejectionReason`` / ``hasProblem``) so the
Dashboard "Recent batches" widget can render one row per batch
without an N+1 fetch.
- ``GET /api/batches/{batch_id}`` — full batch record
(parsed envelope + claims). 404 when unknown.
Three single-router helpers stay in this file (per spec D4):
- :func:`_batch_summary_claim_count` — claim count (837P or 835).
- :func:`_batch_summary_claim_ids` — per-claim ids (837P only).
- :func:`_batch_summary_billing_outcomes` — per-batch GROUP BY state
aggregate plus the most-recent rejection reason.
Inline imports inside handlers (preserved verbatim per spec D5):
``zipfile``, ``datetime``, ``ZoneInfo``, ``build_outbound_filename``,
``PayerConfigORM``, ``func`` (sqlalchemy).
SP36 Task 14: this block moved here from ``api.py:1280`` (the 3
``/api/batches*`` routes + 3 single-router helpers + the SP30
state-bucket tuples and the ``# 277CA STC category`` note).
"""
from __future__ import annotations
import io
import json
from typing import Any
from fastapi import APIRouter, Depends, HTTPException, Query, Request, Response
from fastapi.responses import StreamingResponse
from cyclone import db
from cyclone.api_helpers import (
ndjson_stream_list as _ndjson_stream_list,
wants_ndjson as _wants_ndjson,
)
from cyclone.auth.deps import matrix_gate
from cyclone.db import Batch, Claim, ClaimState
from cyclone.parsers.models import ClaimOutput
from cyclone.parsers.serialize_837 import SerializeError as SerializeError837
from cyclone.parsers.serialize_837 import serialize_837_for_resubmit
from cyclone.store import BatchRecord, store
router = APIRouter(dependencies=[Depends(matrix_gate)])
@router.post("/api/batches/{batch_id}/export-837")
def export_batch_837(request: Request, batch_id: str, body: dict):
"""Download a ZIP of regenerated X12 837 files for the requested claim_ids.
Body shape: ``{"claim_ids": [str, ...]}``.
Each successfully serialized claim becomes an entry in the ZIP named
per the HCPF X12 File Naming Standards:
``tp{tpid}-837P-{yyyymmddhhmmssSSS}-1of1.x12`` (with a per-claim
millisecond offset so every file in the bundle has a unique name).
The ``serialize_837_for_resubmit`` serializer is used so every file
gets a unique interchange / group control number — back-to-back
exports of the same set must produce different envelopes (required
by X12).
The submitter block (Loop 1000A — NM1*41 + PER) is populated from
the clearhouse singleton (dzinesco's identity in the seeded config)
and the receiver block (NM1*40) is populated from the per-payer
config. Without this wiring, the serializer falls back to
``CYCLONE`` / ``RECEIVER`` placeholders and HCPF rejects the file.
No DB state is mutated by this endpoint — it is read-only. Compare
with ``/api/inbox/rejected/resubmit?download=true`` which ALSO flips
``ClaimState.REJECTED → SUBMITTED``; the two endpoints are
intentionally separate.
Responses:
200 — ``application/zip`` with the .x12 entries. Per-claim failures
are surfaced via the ``X-Cyclone-Serialize-Errors`` header
(JSON-encoded array of ``{claim_id, reason}``).
400 — ``claim_ids`` missing or empty.
404 — ``batch_id`` unknown.
422 — every claim failed to serialize; body is JSON listing all
failures (``{"detail": {"serialize_errors": [...]}}``).
"""
import zipfile
from datetime import datetime
from zoneinfo import ZoneInfo
from cyclone.edi.filenames import build_outbound_filename
ids = body.get("claim_ids") or []
if not ids:
raise HTTPException(400, "claim_ids required")
with db.SessionLocal()() as s:
batch = s.get(Batch, batch_id)
if batch is None:
raise HTTPException(404, f"unknown batch: {batch_id}")
serialize_errors: list[dict] = []
ordered_rows: list[tuple[str, "Claim"]] = []
for cid in ids:
c = s.get(Claim, cid)
if c is None:
serialize_errors.append({"claim_id": cid, "reason": "unknown claim_id"})
continue
ordered_rows.append((cid, c))
# Pull clearhouse identity (submitter). If unseeded, the serializer
# falls back to placeholder defaults — degraded but not a hard error.
ch = store.get_clearhouse()
submitter_kwargs: dict = {}
if ch is not None:
submitter_kwargs = {
"sender_id": ch.tpid,
"submitter_name": ch.submitter_name,
"submitter_contact_name": ch.submitter_contact_name,
"submitter_contact_email": ch.submitter_contact_email,
}
# Submitter phone is not in the clearhouse config today, but if
# it ever is, wire it here. Email is the canonical contact
# channel for HCPF submissions per the SP9 spec.
if getattr(ch, "submitter_contact_phone", None):
submitter_kwargs["submitter_contact_phone"] = ch.submitter_contact_phone
# Resolve per-claim payer config so each file's receiver (NM1*40)
# and SBR09 are correct. Cache so we don't re-query the same payer.
from cyclone.db import PayerConfigORM as _PayerConfigORM
_payer_cache: dict[str, dict | None] = {}
def _resolve_payer_cfg(claim_obj: ClaimOutput) -> dict | None:
pid = (claim_obj.payer.id or "").strip() if claim_obj.payer else ""
pname = (claim_obj.payer.name or "").strip() if claim_obj.payer else ""
cache_key = pid or pname
if cache_key in _payer_cache:
return _payer_cache[cache_key]
cfg: dict | None = None
with db.SessionLocal()() as ss:
# 1. Exact match on (payer_id, "837P")
if pid:
row = ss.get(_PayerConfigORM, (pid, "837P"))
if row is not None:
cfg = dict(row.config_json)
# 2. Fallback: any row whose payer_id matches the parsed payer.name
# (HCPF files emit "SKCO0" in NM109 but the canonical
# payer_id in the DB is "CO_TXIX" — name-matching is the
# pragmatic lookup for that case).
if cfg is None and pname:
row = (
ss.query(_PayerConfigORM)
.filter(_PayerConfigORM.transaction_type == "837P")
.all()
)
for r in row:
cj = dict(r.config_json)
if cj.get("submitter_name") and pname.lower() in str(cj).lower():
cfg = cj
break
if (r.payer_id or "").upper() == pname.upper():
cfg = cj
break
# 3. Last resort: first 837P row in the table.
if cfg is None:
row = (
ss.query(_PayerConfigORM)
.filter(_PayerConfigORM.transaction_type == "837P")
.first()
)
if row is not None:
cfg = dict(row.config_json)
_payer_cache[cache_key] = cfg
return cfg
# Build per-claim kwargs (receiver + SBR09) lazily. Receiver
# defaults to the parsed payer name/ID if no config row matches.
def _serialize_kwargs(claim_obj: ClaimOutput) -> dict:
payer_cfg = _resolve_payer_cfg(claim_obj) or {}
receiver_id = (
payer_cfg.get("receiver_id")
or (claim_obj.payer.id if claim_obj.payer else None)
or "RECEIVER"
)
receiver_name = (
payer_cfg.get("receiver_name")
or (claim_obj.payer.name if claim_obj.payer else None)
or receiver_id
)
sbr09 = payer_cfg.get("sbr09_default") or "MC"
return {
"receiver_id": receiver_id,
"receiver_name": receiver_name,
"claim_filing_indicator_code": sbr09,
}
# Base MT timestamp for HCPF filenames. We add a per-claim
# millisecond offset so each file in the ZIP has a unique 17-digit
# ts (HCPF requires that; the spec also enforces "1of1" for the
# sequence element).
base_ts = datetime.now(ZoneInfo("America/Denver"))
def _per_claim_filename(idx: int, cid: str) -> str:
if ch is None:
# No clearhouse — fall back to a per-claim friendly name.
return f"claim-{cid}.x12"
# Millisecond offset, with second/minute rollover.
offset_ms = (idx - 1) * 1 # 1 ms per claim is enough within an export
ts_mt = base_ts.fromtimestamp(
base_ts.timestamp() + offset_ms / 1000.0, tz=ZoneInfo("America/Denver")
)
return build_outbound_filename(ch.tpid, "837P", now_mt=ts_mt)
buf = io.BytesIO()
with zipfile.ZipFile(buf, mode="w", compression=zipfile.ZIP_DEFLATED) as zf:
for idx, (cid, c) in enumerate(ordered_rows, start=1):
if not c.raw_json:
serialize_errors.append({"claim_id": cid, "reason": "no raw_json"})
continue
try:
claim_obj = ClaimOutput.model_validate(c.raw_json)
except Exception as exc:
serialize_errors.append(
{"claim_id": cid, "reason": f"raw_json invalid: {exc}"}
)
continue
try:
kwargs = {**submitter_kwargs, **_serialize_kwargs(claim_obj)}
text = serialize_837_for_resubmit(
claim_obj, interchange_index=idx, **kwargs
)
except SerializeError837 as exc:
serialize_errors.append({"claim_id": cid, "reason": str(exc)})
continue
zf.writestr(_per_claim_filename(idx, cid), text)
success_count = len(ids) - len(serialize_errors)
if serialize_errors and success_count == 0:
# Every claim failed — surface the failure list in the body so the
# UI can render a useful error toast (the response is not a ZIP).
raise HTTPException(
422,
detail={"serialize_errors": serialize_errors},
)
buf.seek(0)
headers = {
"Content-Disposition": (
f'attachment; filename="batch-{batch_id}-{success_count}-claims.zip"'
),
}
if serialize_errors:
headers["X-Cyclone-Serialize-Errors"] = json.dumps(serialize_errors)
return Response(
content=buf.getvalue(),
media_type="application/zip",
headers=headers,
)
def _batch_summary_claim_count(rec: BatchRecord) -> int:
"""Return the number of claims on a batch, handling both 837P and 835."""
if rec.kind == "837p":
return len(rec.result.claims) # type: ignore[attr-defined]
if rec.kind == "835":
return len(rec.result.claims) # type: ignore[attr-defined]
return 0
def _batch_summary_claim_ids(rec: BatchRecord) -> list[str]:
"""Return per-claim ids for an 837P batch, or ``[]`` otherwise.
The Upload page's History tab renders a one-click Re-export ZIP
button per row; that button calls
``POST /api/batches/{id}/export-837`` with the row's claim ids.
Carrying them in the list response avoids an extra round-trip
to ``/api/batches/{id}`` for every row. 835 has no re-export
endpoint, so the list is empty for those — the UI uses the
empty list as the signal to hide the button.
"""
if rec.kind != "837p":
return []
return [
c.claim_id
for c in rec.result.claims # type: ignore[attr-defined]
if getattr(c, "claim_id", None)
]
# SP30: state buckets the Dashboard widget (and any future "how the
# last batch billed" surface) reads at a glance. Keep these in sync
# with ClaimState — adding a new state here is a deliberate decision
# the operator needs to see, not a coincidence.
_BATCH_SUMMARY_ACCEPTED_STATES: tuple[ClaimState, ...] = (
ClaimState.PAID,
ClaimState.RECEIVED,
ClaimState.RECONCILED,
ClaimState.PARTIAL,
)
_BATCH_SUMMARY_REJECTED_STATES: tuple[ClaimState, ...] = (
ClaimState.REJECTED,
ClaimState.DENIED,
ClaimState.REVERSED,
)
_BATCH_SUMMARY_PENDING_STATES: tuple[ClaimState, ...] = (
ClaimState.SUBMITTED,
)
# 277CA STC category A4/A6/A7 — payer-side rejections that may not
# yet have flipped Claim.state (the operator hasn't acknowledged).
# The Dashboard widget treats these as problems too, mirroring the
# Inbox `rejected + payer_rejected` aggregation.
_BATCH_SUMMARY_PAYER_REJECT_CODES: tuple[str, ...] = ("A4", "A6", "A7")
def _batch_summary_billing_outcomes(
records: list[BatchRecord],
) -> dict[str, dict]:
"""Compute per-batch billing outcome for the Dashboard widget.
Returns ``{batch_id: {accepted, rejected, pending, billed,
top_rejection_reason, has_problem}}`` for every batch in
``records``. Empty input → empty dict.
Two SQL queries, both bounded by the supplied batch ids:
1. One GROUP BY ``(batch_id, state)`` aggregate that produces
the accepted/rejected/pending counts and the sum of
``charge_amount`` (the billed total). Single pass — no N+1.
2. One ordered scan over the rejected + payer-rejected subset
to pick the most recent rejection reason (truncated to 60
chars). Skipped when the first query found no rejections
and no payer-rejects, so the happy path stays at one query.
835 batches have no Claim rows — the GROUP BY returns no
rows for them, so the dict entry for an 835 batch is
``{accepted:0, rejected:0, pending:0, billed:0.0,
top_rejection_reason:None, has_problem:False}`` (filled by
the caller's ``.get(id, defaults)`` pattern).
"""
if not records:
return {}
from sqlalchemy import func # local import to keep top-of-file light
batch_ids = [r.id for r in records]
outcome: dict[str, dict] = {
bid: {
"accepted": 0,
"rejected": 0,
"pending": 0,
"billed": 0.0,
"top_rejection_reason": None,
"has_problem": False,
}
for bid in batch_ids
}
with db.SessionLocal()() as s:
# ---- 1. GROUP BY (batch_id, state) for counts + billed total ----
rows = (
s.query(
Claim.batch_id,
Claim.state,
func.count(Claim.id),
func.coalesce(func.sum(Claim.charge_amount), 0),
)
.filter(Claim.batch_id.in_(batch_ids))
.group_by(Claim.batch_id, Claim.state)
.all()
)
any_rejection_or_payer = False
for batch_id, state, count, billed in rows:
slot = outcome.get(batch_id)
if slot is None:
continue # batch has no row in our pre-allocated dict
count = int(count or 0)
billed_f = float(billed or 0)
slot["billed"] += billed_f
if state in _BATCH_SUMMARY_ACCEPTED_STATES:
slot["accepted"] += count
elif state in _BATCH_SUMMARY_REJECTED_STATES:
slot["rejected"] += count
any_rejection_or_payer = True
elif state in _BATCH_SUMMARY_PENDING_STATES:
slot["pending"] += count
# everything else (DRAFT, etc.) is excluded from the widget.
# ---- 2. Most-recent rejection reason + payer-reject probe ----
# Only run when we know there IS at least one rejection OR a
# payer-reject claim somewhere in the batch set; otherwise
# the first query alone is enough.
if any_rejection_or_payer:
rej_rows = (
s.query(
Claim.batch_id,
Claim.rejection_reason,
Claim.payer_rejected_status_code,
)
.filter(
Claim.batch_id.in_(batch_ids),
Claim.state.in_(_BATCH_SUMMARY_REJECTED_STATES)
| Claim.payer_rejected_status_code.in_(
_BATCH_SUMMARY_PAYER_REJECT_CODES
),
)
.order_by(Claim.rejected_at.desc().nullslast())
.all()
)
seen_reason: set[str] = set()
for batch_id, reason, payer_code in rej_rows:
slot = outcome.get(batch_id)
if slot is None:
continue
if payer_code in _BATCH_SUMMARY_PAYER_REJECT_CODES:
slot["has_problem"] = True
# Capture the first non-null reason for this batch
# (rej_rows is ordered newest-first, so the first
# non-null wins). Truncate to 60 chars + ellipsis.
if (
slot["top_rejection_reason"] is None
and reason
and batch_id not in seen_reason
):
r = reason.strip()
if len(r) > 60:
r = r[:60] + ""
slot["top_rejection_reason"] = r
seen_reason.add(batch_id)
if (
slot["rejected"] > 0
or payer_code in _BATCH_SUMMARY_PAYER_REJECT_CODES
):
slot["has_problem"] = True
return outcome
@router.get("/api/batches")
def list_batches(
request: Request,
limit: int = Query(100, ge=1, le=1000),
) -> Any:
"""Summary of all parsed batches, newest first.
Each item includes ``claimIds`` (837P only) so the History tab
on the Upload page can render a one-click re-export button per
row without an extra round-trip to ``/api/batches/{id}``. The
list is still capped at ``limit`` claims; see the full result
via the by-id endpoint when more is needed.
SP30: also returns billing-outcome fields
(``acceptedCount`` / ``rejectedCount`` / ``pendingCount`` /
``billedTotal`` / ``topRejectionReason`` / ``hasProblem``) so
the Dashboard "Recent batches" widget can render one row per
batch without an N+1 fetch. See
:func:`_batch_summary_billing_outcomes`.
"""
records = store.list(limit=limit)
outcomes = _batch_summary_billing_outcomes(records)
items = [
{
"id": r.id,
"kind": r.kind,
"inputFilename": r.input_filename,
"parsedAt": r.parsed_at.isoformat().replace("+00:00", "Z"),
"claimCount": _batch_summary_claim_count(r),
"claimIds": _batch_summary_claim_ids(r),
"acceptedCount": outcomes.get(r.id, {}).get("accepted", 0),
"rejectedCount": outcomes.get(r.id, {}).get("rejected", 0),
"pendingCount": outcomes.get(r.id, {}).get("pending", 0),
"billedTotal": round(outcomes.get(r.id, {}).get("billed", 0.0), 2),
"topRejectionReason": outcomes.get(r.id, {}).get(
"top_rejection_reason"
),
"hasProblem": outcomes.get(r.id, {}).get("has_problem", False),
}
for r in records
]
all_records = store.all()
total = len(all_records)
returned = len(items)
has_more = total > returned
if _wants_ndjson(request):
return StreamingResponse(
_ndjson_stream_list(items, total, returned, has_more),
media_type="application/x-ndjson",
)
return {
"items": items,
"total": total,
"returned": returned,
"has_more": has_more,
}
@router.get("/api/batches/{batch_id}")
def get_batch(batch_id: str) -> Any:
rec = store.get(batch_id)
if rec is None:
raise HTTPException(
status_code=404,
detail={"error": "Not found", "detail": f"Batch {batch_id} not found"},
)
return json.loads(rec.result.model_dump_json())
@@ -0,0 +1,307 @@
"""``/api/claim-acks`` (and per-claim/per-ack surfaces) — SP28.
Seven endpoints that surface the ``claim_acks`` join table to the
frontend + manual-match fallback for orphans. Mounted by
``cyclone.api`` alongside the existing ``/api/acks`` and
``/api/ta1-acks`` routers.
The live-tail endpoints (``/api/claims/{id}/acks/stream`` and
``/api/acks/{kind}/{id}/claims/stream``) subscribe to the
``claim_ack_written`` bus event so the ClaimDrawer Acknowledgments
panel and the per-ack claims list refresh in real time.
Manual match is any-logged-in user (D5) — this endpoint mutates
metadata only (``claim_acks`` row + live-tail event), no
``Claim.state`` mutation, no payment data. Idempotent: re-calling
with the same ``claim_id`` returns 200 with the existing row.
Rejects (409) when the claim is in a terminal state (``REVERSED``).
"""
from __future__ import annotations
import logging
from typing import Any, AsyncIterator, Literal
from fastapi import APIRouter, Depends, HTTPException, Query, Request
from fastapi.responses import JSONResponse, StreamingResponse
from pydantic import BaseModel, Field
from cyclone import db
from cyclone.api_helpers import ndjson_line, tail_events
from cyclone.auth.deps import matrix_gate
from cyclone.pubsub import EventBus
from cyclone.store import store, to_ui_claim_ack
router = APIRouter(dependencies=[Depends(matrix_gate)])
log = logging.getLogger(__name__)
AckKind = Literal["999", "277ca", "ta1"]
class MatchClaimBody(BaseModel):
"""Body for ``POST /api/acks/{kind}/{ack_id}/match-claim``."""
claim_id: str = Field(..., min_length=1)
set_control_number: str | None = None
set_accept_reject_code: str | None = None
ak2_index: int | None = None
# ---------------------------------------------------------------------------
# Per-claim surface
# ---------------------------------------------------------------------------
@router.get("/api/claims/{claim_id}/acks/stream")
async def claim_acks_stream(
request: Request,
claim_id: str,
) -> StreamingResponse:
"""Stream ClaimAck rows for one claim as NDJSON.
Subscribes to ``claim_ack_written`` and filters for rows where
``claim_id`` matches the path param. Each matching event is
emitted as ``{"type": "item", "data": to_ui_claim_ack(...)}``;
the client-side ``useMergedTail`` hook dedupes by id.
Registered BEFORE ``/api/claims/{claim_id}/acks`` so the literal
``stream`` path segment doesn't get matched as a suffix.
"""
bus: EventBus = request.app.state.event_bus
async def gen() -> AsyncIterator[bytes]:
rows = [
r for r in store.list_acks_for_claim(claim_id)
if r.claim_id is not None # filter out TA1 batch-level rows
]
for row in rows:
yield ndjson_line({"type": "item", "data": to_ui_claim_ack(row)})
yield ndjson_line({
"type": "snapshot_end",
"data": {"count": len(rows)},
})
async for chunk in tail_events(request, bus, ["claim_ack_written"]):
# tail_events yields full NDJSON lines; the client filter
# picks claim_id matches. We forward every event so the
# wire format mirrors /api/claims/stream.
yield chunk
return StreamingResponse(gen(), media_type="application/x-ndjson")
@router.get("/api/claims/{claim_id}/acks")
def list_acks_for_claim_endpoint(claim_id: str) -> Any:
"""Return every ClaimAck row for one claim (per-claim only).
TA1 batch-level rows (where ``claim_id IS NULL``) are filtered
out — those don't belong to a specific claim, they're a
envelope-level acknowledgement that hangs off the originating
837 batch. The ClaimDrawer Acknowledgments panel is
per-claim only.
"""
rows = store.list_acks_for_claim(claim_id)
items = [to_ui_claim_ack(r) for r in rows if r.claim_id is not None]
return {"total": len(items), "items": items}
# ---------------------------------------------------------------------------
# Per-ack surface
# ---------------------------------------------------------------------------
@router.get("/api/acks/{kind}/{ack_id}/claims/stream")
async def ack_claims_stream(
request: Request,
kind: AckKind,
ack_id: int,
) -> StreamingResponse:
"""Stream ClaimAck rows for one ack as NDJSON.
Subscribes to ``claim_ack_written`` and forwards events the
store knows about. Clients filter by ack_id + ack_kind on the
client side.
"""
bus: EventBus = request.app.state.event_bus
async def gen() -> AsyncIterator[bytes]:
rows = store.list_claims_for_ack(kind, ack_id)
for row in rows:
yield ndjson_line({"type": "item", "data": to_ui_claim_ack(row)})
yield ndjson_line({
"type": "snapshot_end",
"data": {"count": len(rows)},
})
async for chunk in tail_events(request, bus, ["claim_ack_written"]):
yield chunk
return StreamingResponse(gen(), media_type="application/x-ndjson")
@router.get("/api/acks/{kind}/{ack_id}/claims")
def list_claims_for_ack_endpoint(kind: AckKind, ack_id: int) -> Any:
"""Return every ClaimAck row for one ack (any kind).
For 999 / 277CA: returns 0..N rows (one per AK2 / ClaimStatus).
For TA1: returns 0..1 row (envelope-level, populated batch_id).
"""
rows = store.list_claims_for_ack(kind, ack_id)
items = [to_ui_claim_ack(r) for r in rows]
return {"total": len(items), "items": items}
# ---------------------------------------------------------------------------
# Manual match / unmatch (D5, D9)
# ---------------------------------------------------------------------------
@router.post("/api/acks/{kind}/{ack_id}/match-claim")
def manual_match_claim_endpoint(
request: Request,
kind: AckKind,
ack_id: int,
body: MatchClaimBody,
) -> Any:
"""Manual link fallback (D5/D9). Any-logged-in-user posture.
Inserts a ``claim_acks`` row with ``linked_by="manual"`` and
publishes ``claim_ack_written`` so the drawers refresh. The
endpoint is idempotent: if a row already exists for this dedup
key, the existing row is returned (200). 409 when the claim is
in a terminal state (``REVERSED``). 404 when the claim doesn't
exist or the ack doesn't exist.
"""
# Verify the claim exists and is in a non-terminal state.
from cyclone.db import ClaimState as _CS
with db.SessionLocal()() as s:
claim = s.get(db.Claim, body.claim_id)
if claim is None:
raise HTTPException(
status_code=404,
detail={
"error": "Not found",
"detail": f"Claim {body.claim_id} not found",
},
)
if claim.state == _CS.REVERSED:
return JSONResponse(
status_code=409,
content={
"error": "Conflict",
"detail": (
f"Claim {body.claim_id} is in terminal state "
f"{claim.state.value} and cannot be linked."
),
},
)
# Verify the ack exists (any kind).
ack_table = {
"999": db.Ack,
"277ca": db.Two77caAck,
"ta1": db.Ta1Ack,
}[kind]
if s.get(ack_table, ack_id) is None:
raise HTTPException(
status_code=404,
detail={
"error": "Not found",
"detail": f"{kind} ACK {ack_id} not found",
},
)
# Idempotency: if a manual or auto link already exists, return it.
existing = (
s.query(db.ClaimAck)
.filter(
db.ClaimAck.ack_kind == kind,
db.ClaimAck.ack_id == ack_id,
db.ClaimAck.claim_id == body.claim_id,
)
.first()
)
if existing is not None:
return {
"link": to_ui_claim_ack(existing),
"created": False,
}
# Insert via the store so the publish-from-store contract fires.
row = store.add_claim_ack(
claim_id=body.claim_id,
batch_id=None,
ack_id=ack_id,
ack_kind=kind,
ak2_index=body.ak2_index,
set_control_number=body.set_control_number,
set_accept_reject_code=body.set_accept_reject_code,
linked_by="manual",
event_bus=request.app.state.event_bus,
)
return {"link": to_ui_claim_ack(row), "created": True}
@router.delete("/api/acks/{kind}/{ack_id}/match-claim/{claim_id}")
def manual_unmatch_claim_endpoint(
request: Request,
kind: AckKind,
ack_id: int,
claim_id: str,
) -> Any:
"""Unlink (preserves ``Claim.state`` mutation; only removes the row).
404 when no link row exists for the dedup key. Publishes
``claim_ack_dropped`` so live-tail subscribers remove the link.
"""
from cyclone import db as _db
with _db.SessionLocal()() as s:
link = (
s.query(_db.ClaimAck)
.filter(
_db.ClaimAck.ack_kind == kind,
_db.ClaimAck.ack_id == ack_id,
_db.ClaimAck.claim_id == claim_id,
)
.first()
)
if link is None:
raise HTTPException(
status_code=404,
detail={
"error": "Not found",
"detail": (
f"No link for {kind} ack {ack_id}"
f"claim {claim_id}"
),
},
)
link_id = link.id
removed = store.remove_claim_ack(link_id, event_bus=request.app.state.event_bus)
return {"removed": removed, "link_id": link_id}
# ---------------------------------------------------------------------------
# Inbox ack-orphans lane (spec §D7)
# ---------------------------------------------------------------------------
@router.get("/api/inbox/ack-orphans")
def list_ack_orphans_endpoint(
kind: AckKind | None = Query(None, description="Filter by ack kind"),
) -> Any:
"""List acks with no resolvable Claim row of their own kind.
Used by the Inbox "Ack orphans" lane for the operator's manual
reconciliation flow. Mirrors ``/api/inbox/remit-orphans``.
"""
if kind is not None:
items = store.find_ack_orphans(kind)
return {"total": len(items), "items": items}
items_999 = store.find_ack_orphans("999")
items_277ca = store.find_ack_orphans("277ca")
items_ta1 = store.find_ack_orphans("ta1")
all_items = items_999 + items_277ca + items_ta1
return {"total": len(all_items), "items": all_items}
__all__ = ["router"]
+472
View File
@@ -0,0 +1,472 @@
"""``/api/claims*`` — Claims list / detail / streaming / serialize / line-reconciliation.
Five endpoints, all gated by ``matrix_gate``:
- ``GET /api/claims`` — paginated list
with filter+sort, plus an NDJSON variant when the caller sends
``Accept: application/x-ndjson``. SP27: counts the full filtered
population, not a page-limited sample.
- ``GET /api/claims/stream`` — NDJSON live-tail
on ``claim_written``. Snapshot first (eager
``store.iter_claims``), then ``tail_events`` subscribes + emits
heartbeats. Registered before ``/api/claims/{claim_id}`` so the
literal ``stream`` segment isn't captured as a claim id.
- ``GET /api/claims/{claim_id}`` — full drawer
context (SP4) with the SP28 ``ack_links`` block pre-attached.
404 on missing id — never 500.
- ``GET /api/claims/{claim_id}/serialize-837`` — regenerate X12
837P from the stored ``raw_json`` payload. 404 unknown claim, 422
no-``raw_json`` / unparseable / serializer failure.
- ``GET /api/claims/{claim_id}/line-reconciliation`` — per-line 837
vs 835 side-by-side with CAS adjustments and a summary block.
Three single-router helpers stay in this file (per spec D4):
- :func:`_compact_ack_links_for_claim` — slim form
``{ack_id, ack_kind, set_accept_reject_code, …}`` for the drawer
Acknowledgments panel.
- :func:`_claim_line_dict` — project an 837 service-line
dict from ``Claim.raw_json`` to wire shape.
- :func:`_svc_to_dict` — project an ORM
``ServiceLinePayment`` to wire shape.
Inline imports inside handlers (preserved verbatim per spec D5):
``select`` (sqlalchemy), ``LineReconciliation``/``ServiceLinePayment``/
``CasAdjustment`` (cyclone.db), ``json as _json``, ``Decimal``.
SP36 Task 15: this block moved here from ``api.py:1278`` (the 5
``/api/claims*`` routes + 3 single-router helpers).
"""
from __future__ import annotations
import json
from decimal import Decimal
from typing import Any, AsyncIterator
from fastapi import APIRouter, Depends, HTTPException, Query, Request
from fastapi.responses import JSONResponse, Response, StreamingResponse
from cyclone import db
from cyclone.api_helpers import (
ndjson_line as _ndjson_line,
ndjson_stream_list as _ndjson_stream_list,
tail_events as _tail_events,
wants_ndjson as _wants_ndjson,
)
from cyclone.auth.deps import matrix_gate
from cyclone.db import Claim
from cyclone.parsers.models import ClaimOutput
from cyclone.parsers.serialize_837 import SerializeError as SerializeError837
from cyclone.parsers.serialize_837 import serialize_837
from cyclone.pubsub import EventBus
from cyclone.store import store
router = APIRouter(dependencies=[Depends(matrix_gate)])
@router.get("/api/claims")
def list_claims(
request: Request,
batch_id: str | None = Query(None),
status: str | None = Query(None),
provider_npi: str | None = Query(None),
payer: str | None = Query(None),
date_from: str | None = Query(None),
date_to: str | None = Query(None),
sort: str | None = Query(None),
order: str = Query("desc"),
limit: int = Query(100, ge=1, le=1000),
offset: int = Query(0, ge=0),
) -> Any:
common = dict(
batch_id=batch_id,
status=status,
provider_npi=provider_npi,
payer=payer,
date_from=date_from,
date_to=date_to,
)
items = list(store.iter_claims(
sort=sort, order=order, limit=limit, offset=offset, **common,
))
# SP27 Task 13b: count the full population, not a 100-row sample.
# `iter_claims` defaults to limit=100; counting its output silently
# capped the reported total at 100 even when the DB held 60k rows.
total = store.count_claims(**common)
returned = len(items)
has_more = total > offset + returned
if _wants_ndjson(request):
return StreamingResponse(
_ndjson_stream_list(items, total, returned, has_more),
media_type="application/x-ndjson",
)
return {
"items": items,
"total": total,
"returned": returned,
"has_more": has_more,
}
# --------------------------------------------------------------------------- #
# Live-tail NDJSON streaming endpoints (Phase 3 — SP5)
# --------------------------------------------------------------------------- #
@router.get("/api/claims/stream")
async def claims_stream(
request: Request,
status: str | None = Query(None),
provider_npi: str | None = Query(None),
payer: str | None = Query(None),
date_from: str | None = Query(None),
date_to: str | None = Query(None),
sort: str | None = Query(None),
order: str = Query("desc"),
limit: int = Query(100, ge=1, le=1000),
) -> StreamingResponse:
"""Stream Claims as NDJSON: snapshot first, then live events.
Wire format:
* ``{"type":"item","data":<claim>}`` per snapshot row, then per
new ``claim_written`` event
* ``{"type":"snapshot_end","data":{"count":N}}`` after the snapshot
* ``{"type":"heartbeat","data":{"ts":<iso>}}`` every
``CYCLONE_TAIL_HEARTBEAT_S`` seconds when idle
Query params mirror :func:`list_claims` so a frontend can swap a
one-shot fetch for a tail with no URL surgery.
NOTE: registered before ``/api/claims/{claim_id}`` so the literal
``stream`` path segment doesn't get matched as a claim id.
"""
bus: EventBus = request.app.state.event_bus
async def gen() -> AsyncIterator[bytes]:
# 1. Snapshot (eager — iter_claims returns a list already).
rows = store.iter_claims(
status=status, provider_npi=provider_npi, payer=payer,
date_from=date_from, date_to=date_to,
sort=sort or "-submission_date", order=order, limit=limit,
)
for row in rows:
yield _ndjson_line({"type": "item", "data": row})
yield _ndjson_line({"type": "snapshot_end", "data": {"count": len(rows)}})
# 2. Subscribe + heartbeats.
async for chunk in _tail_events(request, bus, ["claim_written"]):
yield chunk
return StreamingResponse(gen(), media_type="application/x-ndjson")
@router.get("/api/claims/{claim_id}")
def get_claim_detail_endpoint(claim_id: str) -> dict:
"""Return one claim with full drawer context (SP4).
Body shape is produced by :meth:`CycloneStore.get_claim_detail`:
header, state, service lines, diagnoses, parties, validation,
raw segments, ``stateHistory`` (most-recent-first, capped at 50),
and a populated ``matchedRemittance`` block when paired.
SP28: response gains ``ack_links: list[dict]`` (compact form:
``[{ack_id, ack_kind, set_accept_reject_code, parsed_at}]``)
so the ``ClaimDrawer`` Acknowledgments panel can render on
initial load. TA1 batch-level rows (``claim_id IS NULL``) are
excluded — those don't belong to a specific claim.
Path param is ``claim_id`` (matches the SP3 ``/api/acks/{ack_id}``
convention). Returns 404 — never 500 — on a missing claim so the
UI can distinguish "doesn't exist" from a transient fetch error.
"""
body = store.get_claim_detail(claim_id)
if body is None:
raise HTTPException(
status_code=404,
detail={
"error": "Not found",
"detail": f"Claim {claim_id} not found",
},
)
# SP28: attach ack_links (compact form for the drawer panel).
body["ack_links"] = _compact_ack_links_for_claim(claim_id)
return body
def _compact_ack_links_for_claim(claim_id: str) -> list[dict]:
"""Return compact ack_links for one claim, newest first.
TA1 batch-level rows (claim_id IS NULL) are filtered out — those
hang off the originating 837 batch, not a specific claim. The
shape is the slimmer ``{ack_id, ack_kind,
set_accept_reject_code, parsed_at, ak2_index}`` form so the
ClaimDrawer can render without an N+1 round-trip per row.
"""
rows = store.list_acks_for_claim(claim_id)
out: list[dict] = []
for row in rows:
if row.claim_id is None:
continue
out.append({
"id": row.id,
"ack_id": row.ack_id,
"ack_kind": row.ack_kind,
"ak2_index": row.ak2_index,
"set_control_number": row.set_control_number,
"set_accept_reject_code": row.set_accept_reject_code,
"linked_at": (
row.linked_at.isoformat().replace("+00:00", "Z")
if row.linked_at is not None else ""
),
"linked_by": row.linked_by,
})
return out
@router.get("/api/claims/{claim_id}/serialize-837")
def serialize_claim_as_837(claim_id: str):
"""Return the claim as a regenerated X12 837P file (SP8).
Loads the ClaimOutput from the persisted ``raw_json`` and runs the
outbound serializer. Returns 404 if the claim doesn't exist, 422 if
the stored payload has no parseable ClaimOutput (data integrity
issue, not a transient failure).
"""
with db.SessionLocal()() as s:
row = s.get(Claim, claim_id)
if row is None:
return JSONResponse(
{"error": "Not found", "detail": f"Claim {claim_id} not found"},
status_code=404,
)
if not row.raw_json:
return JSONResponse(
{
"error": "Unprocessable",
"detail": f"Claim {claim_id} has no raw_json; cannot serialize",
},
status_code=422,
)
try:
claim_obj = ClaimOutput.model_validate(row.raw_json)
except Exception as exc:
return JSONResponse(
{
"error": "Unprocessable",
"detail": f"Claim {claim_id} raw_json is malformed: {exc}",
},
status_code=422,
)
try:
text = serialize_837(claim_obj)
except SerializeError837 as exc:
return JSONResponse(
{"error": "Unprocessable", "detail": str(exc)},
status_code=422,
)
return Response(
content=text,
media_type="text/x12",
headers={
"Content-Disposition": f'attachment; filename="claim-{claim_id}.x12"'
},
)
@router.get("/api/claims/{claim_id}/line-reconciliation")
def get_claim_line_reconciliation(claim_id: str) -> dict:
"""Per-line reconciliation view for the ClaimDrawer tab.
Spec §5.1. Returns the 837 service lines and 835 SVC composites
side-by-side, with per-line CAS adjustments and a summary block.
Architecture note: 837 service lines live in ``Claim.raw_json``
(not a separate ORM table), so the 837-side rows are read from the
JSON blob; the 835-side rows come from ``ServiceLinePayment`` ORM.
``LineReconciliation.claim_service_line_number`` stores the 1-based
line number to join them.
"""
from sqlalchemy import select
from cyclone.db import (
LineReconciliation, ServiceLinePayment, CasAdjustment,
)
import json as _json
from decimal import Decimal
with db.SessionLocal()() as s:
claim = s.get(db.Claim, claim_id)
if claim is None:
raise HTTPException(
status_code=404,
detail={"error": "Not found", "detail": f"Claim {claim_id} not found"},
)
# 837 service lines: from raw_json.
raw = claim.raw_json or {}
claim_lines_raw = raw.get("service_lines") or []
# Normalize to dicts for the response.
claim_lines = [_claim_line_dict(d) for d in claim_lines_raw]
# 835 service payments: ORM rows from the matched remit.
remits = list(
s.execute(
select(db.Remittance).where(db.Remittance.claim_id == claim_id)
).scalars().all()
)
svc_payments: list[dict] = []
svc_ids: list[int] = []
if remits:
svc_rows = list(
s.execute(
select(ServiceLinePayment).where(
ServiceLinePayment.remittance_id.in_([r.id for r in remits])
).order_by(ServiceLinePayment.line_number)
).scalars().all()
)
for svc in svc_rows:
d = _svc_to_dict(svc)
svc_payments.append(d)
svc_ids.append(svc.id)
# LineReconciliation rows.
lrs = list(
s.execute(
select(LineReconciliation).where(LineReconciliation.claim_id == claim_id)
).scalars().all()
)
# Index by claim_service_line_number and service_line_payment_id.
lr_by_claim_num: dict[int, LineReconciliation] = {
lr.claim_service_line_number: lr for lr in lrs if lr.claim_service_line_number is not None
}
lr_by_svc: dict[int, LineReconciliation] = {
lr.service_line_payment_id: lr for lr in lrs if lr.service_line_payment_id is not None
}
# CAS rows grouped by svc id.
cas_by_svc: dict[int, list[CasAdjustment]] = {}
if svc_ids:
cas_rows = list(
s.execute(
select(CasAdjustment).where(CasAdjustment.service_line_payment_id.in_(svc_ids))
).scalars().all()
)
for c in cas_rows:
cas_by_svc.setdefault(c.service_line_payment_id, []).append(c)
# Build output lines array, preserving 837 order then 835-only.
svc_by_id: dict[int, dict] = {d["id"]: d for d in svc_payments}
lines_out: list[dict] = []
billed_total = Decimal("0")
paid_total = Decimal("0")
adjustment_total = Decimal("0")
matched_count = 0
used_svc_ids: set[int] = set()
for cl in claim_lines:
billed_total += Decimal(str(cl["charge"]))
lr = lr_by_claim_num.get(cl["line_number"])
if lr is None:
lines_out.append({
"claim_service_line": cl,
"service_line_payment": None,
"status": "unmatched_837_only",
"adjustments": [],
})
continue
svc_id = lr.service_line_payment_id
svc = svc_by_id.get(svc_id) if svc_id else None
if svc_id is not None:
used_svc_ids.add(svc_id)
cas_list = cas_by_svc.get(svc_id, []) if svc_id is not None else []
cas_total = sum((Decimal(str(c.amount)) for c in cas_list), Decimal("0"))
if svc:
paid_total += Decimal(str(svc["payment"]))
adjustment_total += cas_total
if lr.status == "matched":
matched_count += 1
lines_out.append({
"claim_service_line": cl,
"service_line_payment": svc,
"status": lr.status,
"adjustments": [
{"group_code": c.group_code, "reason_code": c.reason_code,
"amount": str(Decimal(str(c.amount)))}
for c in cas_list
],
})
# 835-only lines (no claim match).
for lr in lrs:
if lr.claim_service_line_number is not None:
continue
svc_id = lr.service_line_payment_id
if svc_id is None:
continue
if svc_id in used_svc_ids:
continue
svc = svc_by_id.get(svc_id)
cas_list = cas_by_svc.get(svc_id, [])
cas_total = sum((Decimal(str(c.amount)) for c in cas_list), Decimal("0"))
if svc:
paid_total += Decimal(str(svc["payment"]))
adjustment_total += cas_total
lines_out.append({
"claim_service_line": None,
"service_line_payment": svc,
"status": lr.status,
"adjustments": [
{"group_code": c.group_code, "reason_code": c.reason_code,
"amount": str(Decimal(str(c.amount)))}
for c in cas_list
],
})
return {
"claim_id": claim_id,
"summary": {
"billed_total": str(billed_total),
"paid_total": str(paid_total),
"adjustment_total": str(adjustment_total),
"matched_lines": matched_count,
"total_lines": len(claim_lines),
},
"lines": lines_out,
}
def _claim_line_dict(d: dict) -> dict:
"""Project an 837 service-line dict from ``Claim.raw_json`` to wire shape."""
from decimal import Decimal
proc = d.get("procedure") or {}
charge = d.get("charge")
units = d.get("units")
return {
"line_number": d.get("line_number"),
"procedure_qualifier": proc.get("qualifier", "HC"),
"procedure_code": proc.get("code", ""),
"modifiers": proc.get("modifiers") or [],
"charge": str(Decimal(str(charge))) if charge is not None else "0",
"units": str(Decimal(str(units))) if units is not None else None,
"unit_type": d.get("unit_type"),
"service_date": d.get("service_date"),
}
def _svc_to_dict(svc) -> dict:
"""Project an ORM ``ServiceLinePayment`` to wire shape."""
import json as _json
from decimal import Decimal
return {
"id": svc.id,
"line_number": svc.line_number,
"procedure_qualifier": svc.procedure_qualifier,
"procedure_code": svc.procedure_code,
"modifiers": _json.loads(svc.modifiers_json or "[]"),
"charge": str(Decimal(str(svc.charge))),
"payment": str(Decimal(str(svc.payment))),
"units": str(Decimal(str(svc.units))) if svc.units is not None else None,
"unit_type": svc.unit_type,
"service_date": svc.service_date.isoformat() if svc.service_date else None,
}
@@ -0,0 +1,310 @@
"""``/api/clearhouse*`` — singleton clearhouse config + SFTP submission.
Three endpoints, all gated by ``matrix_gate``:
- ``GET /api/clearhouse`` — read the singleton clearhouse row
(dzinesco's identity, SFTP block, filename block). 404 when
unseeded.
- ``PATCH /api/clearhouse`` — full-row replacement of the
singleton (SP25). Strict-validates ``sftp_block`` first (Pydantic
v2 default mode coerces strings-to-bools and would hide a real
operator mistake), then validates the whole body in loose mode.
Hot-reloads the running scheduler via
``scheduler.reconfigure_scheduler`` so the next tick picks up the
new ``SftpBlock`` without a process restart.
- ``POST /api/clearhouse/submit`` — submit a batch of claims to
the clearhouse. Stub: serializes via the SP7 serializer, builds
an HCPF-compliant outbound filename, copies the result to the
staging path. Per-claim audit events stamped with
``actor="clearhouse-submit"``.
Three single-router helpers stay in this file (per spec D4):
- :func:`_load_claim_row` — load a ``Claim`` row by id.
- :func:`_serialize_claim_for_submit` — re-serialize a claim to X12
with optional per-call kwargs (submitter, receiver, SBR09, etc).
- :func:`_serialize_claim_from_raw` — best-effort serializer that
re-parses stored ``x12_text`` and re-emits.
SP36 Task 11: this block moved here from ``api.py:2484`` (the 3
``/api/clearhouse*`` routes + 3 single-router helpers).
"""
from __future__ import annotations
import json
from typing import Any
from fastapi import APIRouter, Depends, HTTPException, Request
from cyclone import db
from cyclone.api_routers._shared import _actor_user_id
from cyclone.audit_log import AuditEvent, append_event
from cyclone.auth.deps import matrix_gate
from cyclone.db import Claim
from cyclone.store import store
router = APIRouter(dependencies=[Depends(matrix_gate)])
@router.get("/api/clearhouse")
def get_clearhouse():
"""Return the singleton clearhouse config (dzinesco's identity, SFTP block, filename block)."""
ch = store.get_clearhouse()
if ch is None:
raise HTTPException(status_code=404, detail="clearhouse not seeded")
return json.loads(ch.model_dump_json())
@router.patch("/api/clearhouse")
async def patch_clearhouse(body: dict) -> Any:
"""Replace the singleton clearhouse row (SP25).
The full ``Clearhouse`` model is required — we don't accept partial
updates because the operator-facing use case is "I'm switching the
loop to real MFT" or "I'm pointing at a different MFT server",
not "I'm tweaking one field at a time." Validation errors are
returned as 422 (Pydantic default).
After a successful write, the running scheduler is hot-reloaded
via ``scheduler.reconfigure_scheduler()`` so the next tick uses
the new SftpBlock without a process restart.
"""
from cyclone import scheduler as _scheduler_mod
from cyclone.providers import Clearhouse as _Clearhouse, SftpBlock as _SftpBlock
# Strict-validate the sftp_block sub-dict FIRST. Pydantic v2's
# default mode coerces strings to bools (e.g. ``"stub": "yes"``
# silently becomes True), which would hide a real operator
# mistake. The Clearhouse model itself stays in loose mode so
# ISO-string ``updated_at`` (the JSON round-trip shape) keeps
# parsing.
raw_sb = body.get("sftp_block", {})
try:
_SftpBlock.model_validate(raw_sb, strict=True)
except Exception as exc:
raise HTTPException(
status_code=422, detail=f"invalid sftp_block: {exc}",
) from exc
# Now validate the full body in loose mode.
try:
parsed = _Clearhouse.model_validate(body)
except Exception as exc:
raise HTTPException(
status_code=422, detail=str(exc),
) from exc
# SP25: when sftp_block.stub=false, the block must carry an auth
# account name and a non-empty host. The Pydantic model catches
# some of these; this catches the "empty password_keychain_account"
# case (which Pydantic allows because it's a free-form dict).
sb = parsed.sftp_block
if not sb.stub:
if not sb.host:
raise HTTPException(
status_code=422,
detail="sftp_block.host is required when stub=false",
)
auth = sb.auth or {}
if not auth.get("password_keychain_account") and not auth.get("key_file"):
raise HTTPException(
status_code=422,
detail=(
"sftp_block.auth must contain either "
"'password_keychain_account' or 'key_file' when stub=false"
),
)
updated = store.update_clearhouse(parsed)
await _scheduler_mod.reconfigure_scheduler(
updated.sftp_block,
sftp_block_name=updated.name or "default",
)
return json.loads(updated.model_dump_json())
@router.post("/api/clearhouse/submit")
def submit_to_clearhouse(request: Request, body: dict):
"""Submit a batch of claims to the clearhouse (SFTP). SP9: stub.
Body: ``{"claim_ids": [...], "payer_id": "CO_TXIX"}``
Stub behavior: serializes each claim via the SP7 serializer, builds
an HCPF-compliant outbound filename, and copies the result to
``{staging_dir}/{outbound_path}/{filename}`` instead of opening a
real SFTP connection. Returns a receipt per claim.
"""
from cyclone.clearhouse import make_client
from cyclone.edi.filenames import build_outbound_filename
claim_ids = body.get("claim_ids", [])
payer_id = body.get("payer_id")
if not claim_ids:
raise HTTPException(status_code=400, detail="claim_ids required")
if not payer_id:
raise HTTPException(status_code=400, detail="payer_id required")
ch = store.get_clearhouse()
if ch is None:
raise HTTPException(status_code=500, detail="clearhouse not seeded")
# Submitter (Loop 1000A) comes from the clearhouse config. The
# receiver (NM1*40) and SBR09 come from the per-payer config and
# are resolved per-claim below. Without this wiring, the
# serializer would emit "CYCLONE" / "RECEIVER" placeholders and
# the file would be rejected by HCPF.
submitter_kwargs = {
"sender_id": ch.tpid,
"submitter_name": ch.submitter_name,
"submitter_contact_name": ch.submitter_contact_name,
"submitter_contact_email": ch.submitter_contact_email,
}
if getattr(ch, "submitter_contact_phone", None):
submitter_kwargs["submitter_contact_phone"] = ch.submitter_contact_phone
# Build a payer_id → PayerConfig837 map once so we can look up the
# receiver + SBR09 default for each claim.
from cyclone.db import PayerConfigORM as _PayerConfigORM
def _resolve_payer_cfg(claim_obj) -> dict | None:
pid = (claim_obj.payer.id or "").strip() if claim_obj.payer else ""
pname = (claim_obj.payer.name or "").strip() if claim_obj.payer else ""
with db.SessionLocal()() as ss:
if pid:
row = ss.get(_PayerConfigORM, (pid, "837P"))
if row is not None:
return dict(row.config_json)
if pname:
rows = (
ss.query(_PayerConfigORM)
.filter(_PayerConfigORM.transaction_type == "837P")
.all()
)
for r in rows:
cj = dict(r.config_json)
if (r.payer_id or "").upper() == pname.upper():
return cj
if pname.lower() in str(cj).lower():
return cj
row = (
ss.query(_PayerConfigORM)
.filter(_PayerConfigORM.transaction_type == "837P")
.first()
)
return dict(row.config_json) if row else None
client = make_client(ch.sftp_block)
results = []
for cid in claim_ids:
try:
x12_text = _serialize_claim_for_submit(cid)
except Exception as exc: # noqa: BLE001
results.append({"claim_id": cid, "ok": False, "error": str(exc)})
continue
# Re-resolve the claim so we can look up its payer config. We
# re-parse the stored x12_text to get a ClaimOutput (same path
# the serializer uses).
try:
from cyclone.parsers.parse_837 import parse as _parse837
claim_row_obj = _load_claim_row(cid)
if claim_row_obj is None or not (claim_row_obj.raw_json or {}).get("x12_text"):
raise RuntimeError("no stored x12_text for claim")
parsed = _parse837(claim_row_obj.raw_json["x12_text"])
claim_obj = parsed.claims[0] if parsed.claims else None
except Exception:
claim_obj = None
if claim_obj is not None:
payer_cfg = _resolve_payer_cfg(claim_obj) or {}
receiver_id = payer_cfg.get("receiver_id") or (claim_obj.payer.id if claim_obj.payer else None) or "RECEIVER"
receiver_name = payer_cfg.get("receiver_name") or (claim_obj.payer.name if claim_obj.payer else None) or receiver_id
sbr09 = payer_cfg.get("sbr09_default") or "MC"
# Re-serialize with the proper envelope values.
try:
x12_text = _serialize_claim_for_submit(
cid,
**{**submitter_kwargs, "receiver_id": receiver_id,
"receiver_name": receiver_name,
"claim_filing_indicator_code": sbr09},
)
except Exception as exc: # noqa: BLE001
results.append({"claim_id": cid, "ok": False, "error": str(exc)})
continue
filename = build_outbound_filename(ch.tpid, "837P")
remote = f"{ch.sftp_block.paths['outbound']}/{filename}"
staging_path = client.write_file(remote, x12_text.encode("utf-8"))
results.append({
"claim_id": cid,
"ok": True,
"filename": filename,
"staging_path": str(staging_path),
"remote_path": remote,
})
# SP11: audit trail for each successful clearhouse submission.
with db.SessionLocal()() as audit_s:
append_event(audit_s, AuditEvent(
event_type="clearhouse.submitted",
entity_type="claim",
entity_id=cid,
payload={
"filename": filename,
"remote_path": remote,
"tpid": ch.tpid,
"stub": ch.sftp_block.stub,
},
actor="clearhouse-submit",
user_id=_actor_user_id(request),
))
audit_s.commit()
return {"ok": True, "submitted": results, "stub": ch.sftp_block.stub}
def _load_claim_row(claim_id: str):
"""Helper: load a Claim row by id (or return None)."""
with db.SessionLocal()() as s:
return s.get(Claim, claim_id)
def _serialize_claim_for_submit(claim_id: str, **kwargs) -> str:
"""Serialize a claim to X12 for SFTP submission. Lazy import of the
serializer to avoid pulling FastAPI machinery at module import time.
Optional ``**kwargs`` are forwarded to the serializer — used to
pass through clearhouse submitter info and per-payer receiver info
so the regenerated file matches what the HCPF MFT expects.
"""
from cyclone.parsers.serialize_837 import serialize_837
from cyclone import db
with db.SessionLocal()() as s:
row = s.get(db.Claim, claim_id)
if row is None:
raise ValueError(f"claim {claim_id!r} not found")
# Re-parse the stored raw_json to get a ClaimOutput
from cyclone.parsers.models import ClaimOutput, Envelope, Subscriber, Payer, BillingProvider
from cyclone.parsers.parse_837 import parse
raw = row.raw_json or {}
# Reconstruct minimal ClaimOutput from raw_json; this is best-effort.
return _serialize_claim_from_raw(row, raw, **kwargs)
def _serialize_claim_from_raw(claim_row, raw: dict, **kwargs) -> str:
"""Best-effort serializer that uses the stored raw_json to emit a fresh 837.
For SP9 this delegates to the existing serialize_837 helper if the
claim has a complete raw_segments array. Otherwise it returns a
minimal placeholder. ``**kwargs`` are forwarded to the serializer
so callers can pass through submitter / receiver / SBR09 values
from the clearhouse and per-payer configs.
"""
from cyclone.parsers.serialize_837 import serialize_837
from cyclone.parsers.parse_837 import parse
# Re-parse the original batch text (need to re-derive from store).
# SP9 stub: if the claim has a `raw_json` with `x12_text`, use that.
if isinstance(raw, dict) and raw.get("x12_text"):
result = parse(raw["x12_text"])
if result.claims:
return serialize_837(result.claims[0], **kwargs)
# Fallback: raise so the caller sees an error.
raise RuntimeError(
f"claim {claim_row.id!r} cannot be re-serialized: no stored x12_text"
)
+54
View File
@@ -0,0 +1,54 @@
"""``/api/config/payers`` and ``/api/config/payers/{payer_id}/configs`` — payer-config read views.
Both endpoints are read-only configuration surfaces used by the UI's
"Edit payers" page:
- ``GET /api/config/payers?is_active=...`` lists all configured
payers (PayerConfig records) — the set of payers the operator has
registered, regardless of whether they have inbound config blocks.
- ``GET /api/config/payers/{payer_id}/configs`` returns the full
list of ``(transaction_type, config_json)`` blocks for a given
payer. Each block has a ``source``: ``"yaml"`` for the on-disk
``config/payers.yaml`` default, ``"db"`` for any runtime override
recorded via ``/api/admin/reload-config``.
These are configuration surfaces, not claim-processing surfaces.
They live here (under ``/api/config/``) rather than under
``/api/payers/`` because the latter is the drill-down rollup
(see ``api_routers/payers.py``).
SP36 Task 7: this block moved here from ``api.py:3167`` (after
the SP21 provider-detail helper, before the Auth routers divider).
"""
from __future__ import annotations
import json
from fastapi import APIRouter, Depends, Query
from cyclone import store
from cyclone.auth.deps import matrix_gate
router = APIRouter(dependencies=[Depends(matrix_gate)])
@router.get("/api/config/payers")
def list_configured_payers(is_active: bool | None = Query(default=True)):
return [json.loads(p.model_dump_json()) for p in store.list_payers(is_active=is_active)]
@router.get("/api/config/payers/{payer_id}/configs")
def list_payer_configs(payer_id: str):
"""List all (transaction_type, config_json) blocks for a payer."""
from cyclone import payers as payer_loader
configs = [
{"transaction_type": tx, "config_json": block, "source": "yaml"}
for (pid, tx), block in payer_loader.all_configs().items()
if pid == payer_id
]
# Also check the DB for runtime-overridden configs
for tx in ("837P", "835", "277CA", "999", "TA1"):
live = store.get_payer_config(payer_id, tx)
if live is not None:
configs.append({"transaction_type": tx, "config_json": live, "source": "db"})
return configs
@@ -0,0 +1,39 @@
"""``/api/dashboard/kpis`` — server-aggregated Dashboard tiles.
Backs the Dashboard's "Claims / Billed / Received / Pending AR /
Denial rate" tiles + the monthly sparkline series + the
top-providers and top-denials lists.
Why this exists instead of ``GET /api/claims?limit=N``:
The Dashboard's KPIs are aggregates over *every* claim — billed,
received, denial rate, pending count, monthly billed/received. With
60k+ claims in production, paginating ``/api/claims`` and reducing
client-side silently produces wrong numbers (denial rate sampled,
billed summed from the first 100 rows). This endpoint does the
aggregation server-side in a single read so the Dashboard's numbers
are always correct regardless of dataset size.
SP36 Task 4: this single endpoint moved here from ``api.py:2732``.
"""
from __future__ import annotations
from fastapi import APIRouter, Depends, Query
from cyclone.auth.deps import matrix_gate
from cyclone.store import dashboard_kpis
router = APIRouter(dependencies=[Depends(matrix_gate)])
@router.get("/api/dashboard/kpis")
def get_dashboard_kpis(
months: int = Query(6, ge=1, le=24),
top_n_providers: int = Query(4, ge=0, le=50),
top_n_denials: int = Query(5, ge=0, le=50),
) -> dict:
"""Server-aggregated Dashboard KPIs over the whole claim population."""
return dashboard_kpis(
months=months,
top_n_providers=top_n_providers,
top_n_denials=top_n_denials,
)
@@ -0,0 +1,223 @@
"""``/api/eligibility/request`` and ``/api/eligibility/parse-271`` — API-only eligibility pair.
Builds a 270 inquiry from a small JSON body and parses a 271 response.
Nothing is persisted to the DB — these are operator-driven, ephemeral
operations per SP3 (P4 T23T24). The 270 serializer pulls X12 from a
``ParseResult270`` Pydantic; the 271 parser builds the same structure
in reverse from the wire format.
Why these are not ``GET /api/eligibility/...``: the 270 build is
operator-initiated (pay-portal paste-back), so the inbound surface is
a JSON ``POST``. The 271 inbound is a multipart file upload — same
shape as ``/api/parse-999`` — so the file can be the actual 271 text
saved from the payer portal.
SP36 Task 5: this block moved here from ``api.py:2832`` (``270 / 271
eligibility`` divider).
"""
from __future__ import annotations
import json
import logging
from datetime import date as _date
from typing import Any
from fastapi import APIRouter, Depends, File, HTTPException, UploadFile
from fastapi.responses import JSONResponse
from cyclone.auth.deps import matrix_gate
from cyclone.parsers.exceptions import CycloneParseError
from cyclone.parsers.models import BatchSummary, Envelope
from cyclone.parsers.models_270 import (
EligibilityBenefitInquiry,
InformationReceiver270,
InformationSource270,
ParseResult270,
Subscriber270,
)
from cyclone.parsers.parse_271 import parse as parse_271_text
from cyclone.parsers.serialize_270 import serialize_270
log = logging.getLogger(__name__)
router = APIRouter(dependencies=[Depends(matrix_gate)])
def _validate_eligibility_request(body: dict) -> tuple[ParseResult270, str]:
"""Build a :class:`ParseResult270` from a request body dict.
The body shape is the minimum surface needed to build a valid 270
inquiry (per spec section 3.4 — operator-driven, ephemeral):
::
{
"subscriber": {first_name, last_name, member_id, dob},
"provider": {npi, name},
"payer": {id, name},
"service_type_code": "1"
}
Returns ``(ParseResult270, service_type_code)``. Raises
:class:`HTTPException` (400) when the body is missing required
fields.
"""
subscriber_in = body.get("subscriber") or {}
provider_in = body.get("provider") or {}
payer_in = body.get("payer") or {}
service_type_code = (body.get("service_type_code") or "").strip()
# Required-field checks. We surface a single 400 with the first
# missing field name to match the rest of the API's error contract.
if not service_type_code:
raise HTTPException(
status_code=400,
detail={"error": "Bad request", "detail": "service_type_code is required"},
)
if not subscriber_in.get("member_id"):
raise HTTPException(
status_code=400,
detail={"error": "Bad request", "detail": "subscriber.member_id is required"},
)
if not provider_in.get("npi"):
raise HTTPException(
status_code=400,
detail={"error": "Bad request", "detail": "provider.npi is required"},
)
if not payer_in.get("name"):
raise HTTPException(
status_code=400,
detail={"error": "Bad request", "detail": "payer.name is required"},
)
# Build the Pydantic models. The serializer handles all envelope
# generation (sender_id/receiver_id/control_number/transaction_date
# are filled in by the serializer with sensible defaults).
subscriber_dob_raw = subscriber_in.get("dob")
subscriber_dob: _date | None = None
if subscriber_dob_raw:
try:
subscriber_dob = _date.fromisoformat(subscriber_dob_raw)
except (TypeError, ValueError) as exc:
raise HTTPException(
status_code=400,
detail={
"error": "Bad request",
"detail": f"subscriber.dob must be YYYY-MM-DD: {exc}",
},
) from exc
result = ParseResult270(
envelope=Envelope(
sender_id="SUBMITTERID",
receiver_id=str(payer_in.get("id") or "RECEIVERID"),
control_number="000000001",
transaction_date=_date.today(),
implementation_guide="005010X279A1",
),
information_source=InformationSource270(
name=str(payer_in["name"]),
id=str(payer_in.get("id") or "") or None,
),
information_receiver=InformationReceiver270(
name=str(provider_in.get("name") or ""),
npi=str(provider_in["npi"]),
),
subscriber=Subscriber270(
member_id=str(subscriber_in["member_id"]),
first_name=str(subscriber_in.get("first_name") or "") or None,
last_name=str(subscriber_in.get("last_name") or "") or None,
dob=subscriber_dob,
),
inquiries=[EligibilityBenefitInquiry(service_type_code=service_type_code)],
summary=BatchSummary(
input_file="eligibility_request",
control_number="000000001",
transaction_date=_date.today(),
total_claims=1,
passed=1,
failed=0,
),
)
return result, service_type_code
@router.post("/api/eligibility/request")
def post_eligibility_request(body: dict) -> Any:
"""Build a 270 eligibility inquiry from a small JSON body.
Returns ``{"raw_270_text": <X12>, "parsed": <ParseResult270>}``
so the operator can either download the raw text (paste into a
payer portal) or render the parsed fields directly. Per spec
section 3.4, nothing is persisted to the DB.
"""
try:
result, _ = _validate_eligibility_request(body)
except HTTPException:
raise
except (KeyError, TypeError, ValueError) as exc:
raise HTTPException(
status_code=400,
detail={"error": "Bad request", "detail": f"Malformed body: {exc}"},
) from exc
raw_270_text = serialize_270(result)
return {
"raw_270_text": raw_270_text,
"parsed": json.loads(result.model_dump_json()),
}
@router.post("/api/eligibility/parse-271")
async def post_eligibility_parse_271(
file: UploadFile = File(...),
) -> Any:
"""Parse a 271 eligibility response and return the structured summary.
Accepts the raw 271 text as a file upload (multipart/form-data),
mirrors the ``/api/parse-999`` contract. Per spec section 3.4 the
result is NOT persisted — the operator re-pastes the 271 each
time they need a fresh read.
The response body is a JSON object with three top-level keys:
``coverage_benefits``, ``subscriber``, and ``summary``. 400 is
returned on empty / undecodable / malformed EDI; 200 on success.
"""
raw = await file.read()
if not raw:
return JSONResponse(
status_code=400,
content={"error": "Empty file", "detail": "Uploaded file contained no bytes."},
)
try:
text = raw.decode("utf-8")
except UnicodeDecodeError as exc:
return JSONResponse(
status_code=400,
content={"error": "Encoding error", "detail": str(exc)},
)
try:
result = parse_271_text(text, input_file=file.filename or "")
except CycloneParseError as exc:
return JSONResponse(
status_code=400,
content={"error": "Parse error", "detail": str(exc)},
)
except Exception as exc: # pragma: no cover - safety net
log.exception("Unexpected parser failure on 271")
return JSONResponse(
status_code=500,
content={"error": "Internal server error", "detail": str(exc)},
)
return {
"coverage_benefits": [
json.loads(cb.model_dump_json()) for cb in result.coverage_benefits
],
"subscriber": json.loads(result.subscriber.model_dump_json()),
"summary": json.loads(result.summary.model_dump_json()),
"envelope": json.loads(result.envelope.model_dump_json()),
"information_source": json.loads(result.information_source.model_dump_json()),
"information_receiver": json.loads(result.information_receiver.model_dump_json()),
}
+342
View File
@@ -0,0 +1,342 @@
"""``/api/inbox*`` — operator-facing Inbox surface (SP6 + SP14).
Six endpoints, all gated by ``matrix_gate``:
- ``GET /api/inbox/lanes`` — all lanes in one
call (``compute_lanes`` from :mod:`cyclone.inbox_lanes`).
- ``POST /api/inbox/candidates/{remit_id}/match`` — manually link a
remit to a claim; surfaces 409 with the current state when the
claim is already matched.
- ``POST /api/inbox/candidates/dismiss`` — add candidate
pairs to the session-scoped dismissed set (mutates
``request.app.state.dismissed_pairs``).
- ``POST /api/inbox/payer-rejected/acknowledge`` — SP14: mark
Payer-Rejected claims as acknowledged. Idempotent; returns the
count actually transitioned vs. already-acked / not-found /
not-rejected.
- ``POST /api/inbox/rejected/resubmit`` — bulk move
REJECTED claims back to SUBMITTED. With
``?download=true`` returns a ZIP of regenerated 837 files
(``serialize_837_for_resubmit`` with per-claim ``interchange_index``
for unique control numbers). Conflicts are omitted from the ZIP
and surfaced via the ``X-Cyclone-Serialize-Errors`` header.
- ``GET /api/inbox/export.csv`` — stream a CSV for
a single lane (rejected / candidates / unmatched / done_today).
All endpoints use ``request.app.state`` rather than the module-level
``app`` global so they're robust against ``importlib.reload`` of
the api module — the reload rebinds ``app`` to a new instance, but
``request.app`` always points at the instance actually serving the
current request.
SP36 Task 13: this block moved here from ``api.py:1280`` (the 6
``/api/inbox*`` routes, with the SP14 comment block preserved
verbatim).
"""
from __future__ import annotations
import csv
import io
import json
from datetime import datetime, timezone
from fastapi import APIRouter, Depends, HTTPException, Query, Request, Response
from fastapi.responses import StreamingResponse
from cyclone import db
from cyclone.auth.deps import matrix_gate
from cyclone.db import Claim, ClaimState, Remittance
from cyclone.parsers.models import ClaimOutput
from cyclone.parsers.serialize_837 import SerializeError as SerializeError837
from cyclone.parsers.serialize_837 import serialize_837_for_resubmit
router = APIRouter(dependencies=[Depends(matrix_gate)])
@router.get("/api/inbox/lanes")
def inbox_lanes(request: Request):
"""Return all Inbox lanes in one call.
Uses ``request.app.state`` rather than the module-level ``app``
global so the endpoint is robust against ``importlib.reload`` of
this module (some tests do this to mutate the CORS allow-list).
After a reload, the module-level ``app`` rebinds to a new
FastAPI instance; ``request.app`` always points at the instance
that is actually serving the current request, so per-request
state stays consistent with the test's TestClient target.
"""
dismissed_pairs = getattr(request.app.state, "dismissed_pairs", set())
with db.SessionLocal()() as session:
from cyclone.inbox_lanes import compute_lanes
lanes = compute_lanes(session, dismissed_pairs=dismissed_pairs)
return {
"rejected": lanes.rejected,
# SP10: payer-rejected lane (277CA STC A4/A6/A7). Distinct from
# the 999 envelope rejection in ``rejected`` above.
"payer_rejected": lanes.payer_rejected,
"candidates": lanes.candidates,
"unmatched": lanes.unmatched,
"done_today": lanes.done_today,
}
@router.post("/api/inbox/candidates/{remit_id}/match")
def inbox_match_candidate(remit_id: str, body: dict):
"""Manually link a remit to a claim."""
claim_id = body.get("claim_id")
if not claim_id:
raise HTTPException(400, "claim_id required")
with db.SessionLocal()() as s:
claim = s.get(Claim, claim_id)
remit = s.get(Remittance, remit_id)
if claim is None or remit is None:
raise HTTPException(404, "claim or remit not found")
if claim.matched_remittance_id and claim.matched_remittance_id != remit_id:
raise HTTPException(
409,
detail={
"error": "claim_already_matched",
"current_state": (
claim.state.value if hasattr(claim.state, "value")
else str(claim.state)
),
"matched_remittance_id": claim.matched_remittance_id,
},
)
claim.matched_remittance_id = remit_id
remit.claim_id = claim_id
s.commit()
return {"ok": True, "claim_id": claim_id, "remit_id": remit_id}
@router.post("/api/inbox/candidates/dismiss")
def inbox_dismiss_candidates(body: dict, request: Request):
"""Add candidate pairs to the session-scoped dismissed set.
Uses ``request.app.state`` rather than the module-level ``app``
global so the endpoint is robust against ``importlib.reload`` of
this module (some tests do this to mutate the CORS allow-list).
After a reload, the module-level ``app`` rebinds to a new
FastAPI instance; ``request.app`` always points at the instance
that is actually serving the current request, so the test's
TestClient target is the one whose state we mutate.
"""
pairs = body.get("pairs") or []
if not hasattr(request.app.state, "dismissed_pairs"):
request.app.state.dismissed_pairs = set()
for p in pairs:
cid = p.get("claim_id")
rid = p.get("remit_id")
if cid and rid:
request.app.state.dismissed_pairs.add(frozenset({cid, rid}))
return {"ok": True, "dismissed_count": len(pairs)}
# --------------------------------------------------------------------------- #
# SP14: Payer-Rejected acknowledge
#
# Operator hits "Acknowledge" on the Payer-Rejected Inbox lane to clear
# the claim from the working surface. We don't delete the rejection
# (the original payer_rejected_* fields stay for SP11 audit), we just
# set payer_rejected_acknowledged_at so the lane query filters it out.
#
# Idempotent: re-acknowledging an already-acknowledged claim is a noop
# (the timestamp is not bumped). Returns the count actually transitioned
# so the UI can show "3 of 5 were already acknowledged".
# --------------------------------------------------------------------------- #
@router.post("/api/inbox/payer-rejected/acknowledge")
def inbox_acknowledge_payer_rejected(body: dict):
"""Mark Payer-Rejected claims as acknowledged by the operator."""
claim_ids = body.get("claim_ids") or []
actor = body.get("actor") or "operator"
if not isinstance(claim_ids, list) or not claim_ids:
raise HTTPException(400, "claim_ids must be a non-empty list")
if not all(isinstance(c, str) for c in claim_ids):
raise HTTPException(400, "claim_ids must be a list of strings")
with db.SessionLocal()() as session:
from cyclone.db import Claim
now = datetime.now(timezone.utc)
transitioned = 0
already_acked = 0
not_found = 0
not_rejected = 0
for cid in claim_ids:
claim = session.get(Claim, cid)
if claim is None:
not_found += 1
continue
if claim.payer_rejected_at is None:
not_rejected += 1
continue
if claim.payer_rejected_acknowledged_at is not None:
already_acked += 1
continue
claim.payer_rejected_acknowledged_at = now
claim.payer_rejected_acknowledged_actor = actor
transitioned += 1
# SP11: audit event for the acknowledge action.
try:
from cyclone.audit_log import append_event, AuditEvent
append_event(session, AuditEvent(
event_type="claim.payer_rejected_acknowledged",
entity_type="claim",
entity_id=claim.id,
actor=actor,
payload={
"payer_rejected_status_code": claim.payer_rejected_status_code,
"payer_rejected_by_277ca_id": claim.payer_rejected_by_277ca_id,
},
))
except Exception: # noqa: BLE001
# Audit append is best-effort; don't block the operator's
# acknowledge action on an audit-log failure.
pass
if transitioned:
session.commit()
return {
"ok": True,
"transitioned": transitioned,
"already_acked": already_acked,
"not_found": not_found,
"not_rejected": not_rejected,
}
@router.post("/api/inbox/rejected/resubmit")
def inbox_resubmit_rejected(
request: Request,
body: dict,
download: bool = Query(False, description="When true, return a ZIP of regenerated 837 files for the resubmitted claims (instead of JSON)."),
):
"""Bulk move REJECTED claims back to SUBMITTED.
With ``?download=true``, the response is a ``application/zip`` archive
containing one ``claim-{id}.x12`` per successfully resubmitted claim
(regenerated via ``serialize_837_for_resubmit`` so each file gets a
unique interchange/group control number). Conflicts are omitted from
the ZIP — they remain visible to the caller via the JSON shape of the
non-download path. Empty resubmit + download → 200 with an empty zip
so the UI can still hand the user a downloadable artifact.
"""
ids = body.get("claim_ids") or []
if not ids:
raise HTTPException(400, "claim_ids required")
accepted: list[str] = []
conflicts: list[dict] = []
# Track which claims are about to be resubmitted (and their index in
# the bundle) so the download path can serialize them with unique
# control numbers — back-to-back resubmits in the same file would
# otherwise all share ISA13/GS06 = "000000001".
accepted_with_rows: list[tuple[str, "Claim"]] = []
with db.SessionLocal()() as s:
for cid in ids:
c = s.get(Claim, cid)
if c is None:
continue
if c.state != ClaimState.REJECTED:
conflicts.append({
"claim_id": cid,
"current_state": (
c.state.value if hasattr(c.state, "value")
else str(c.state)
),
})
continue
c.state = ClaimState.SUBMITTED
c.state_changed_at = datetime.now(timezone.utc)
c.rejection_reason = None
c.rejected_at = None
c.resubmit_count = (c.resubmit_count or 0) + 1
accepted.append(cid)
accepted_with_rows.append((cid, c))
s.commit()
if not download:
return {"ok": True, "resubmitted": accepted, "conflicts": conflicts}
# Build a ZIP of regenerated 837s for the accepted claims. Conflicts
# and missing ids are deliberately excluded — the user already saw
# them in the JSON response on prior actions; the download is the
# "give me the files I asked for" payload.
import zipfile
buf = io.BytesIO()
serialize_errors: list[dict] = []
with zipfile.ZipFile(buf, mode="w", compression=zipfile.ZIP_DEFLATED) as zf:
for idx, (cid, c) in enumerate(accepted_with_rows, start=1):
if not c.raw_json:
serialize_errors.append({"claim_id": cid, "reason": "no raw_json"})
continue
try:
claim_obj = ClaimOutput.model_validate(c.raw_json)
except Exception as exc:
serialize_errors.append({"claim_id": cid, "reason": f"raw_json invalid: {exc}"})
continue
try:
text = serialize_837_for_resubmit(claim_obj, interchange_index=idx)
except SerializeError837 as exc:
serialize_errors.append({"claim_id": cid, "reason": str(exc)})
continue
zf.writestr(f"claim-{cid}.x12", text)
buf.seek(0)
headers = {
"Content-Disposition": (
f'attachment; filename="resubmit-{len(accepted)}-claims.zip"'
),
}
# Surface per-claim serialization failures as a custom response header
# so the UI can show "10 resubmitted, 2 couldn't be regenerated" without
# parsing the binary. The header value is JSON-encoded; the UI is
# expected to JSON.parse it after a fetch with response.ok.
if serialize_errors:
headers["X-Cyclone-Serialize-Errors"] = json.dumps(serialize_errors)
return Response(
content=buf.getvalue(),
media_type="application/zip",
headers=headers,
)
@router.get("/api/inbox/export.csv")
def inbox_export_csv(lane: str, request: Request):
"""Stream a CSV for a single lane.
Uses ``request.app.state`` rather than the module-level ``app``
global so the endpoint is robust against ``importlib.reload`` of
this module (see ``inbox_dismiss_candidates`` for context).
"""
if lane not in {"rejected", "candidates", "unmatched", "done_today"}:
raise HTTPException(400, f"unknown lane: {lane}")
dismissed_pairs = getattr(request.app.state, "dismissed_pairs", set())
with db.SessionLocal()() as session:
from cyclone.inbox_lanes import compute_lanes
lanes = compute_lanes(session, dismissed_pairs=dismissed_pairs)
rows = getattr(lanes, lane)
buf = io.StringIO()
writer = csv.writer(buf)
writer.writerow([
"id", "kind", "patient_control_number", "charge_amount",
"payer_id", "provider_npi", "state", "rejection_reason",
"service_date", "score",
])
for r in rows:
writer.writerow([
r.get("id") or r.get("payer_claim_control_number"),
r.get("kind"),
r.get("patient_control_number"),
r.get("charge_amount"),
r.get("payer_id"),
r.get("provider_npi") or r.get("rendering_provider_npi"),
r.get("state"),
r.get("rejection_reason"),
r.get("service_date_from") or r.get("service_date"),
r.get("score"),
])
buf.seek(0)
return StreamingResponse(
iter([buf.getvalue()]),
media_type="text/csv",
headers={"Content-Disposition": f'attachment; filename="inbox-{lane}.csv"'},
)
+831
View File
@@ -0,0 +1,831 @@
"""Parse endpoints — accept X12 uploads and ingest them.
Five routes:
* ``POST /api/parse-837`` — 837P professional claim ingest (the
primary upload path)
* ``POST /api/parse-835`` — 835 ERA remittance ingest
* ``POST /api/parse-999`` — 999 ACK ingest + auto-link claims
* ``POST /api/parse-ta1`` — TA1 envelope ACK ingest + envelope-link batches
* ``POST /api/parse-277ca`` — 277CA Claim Acknowledgment ingest
The 7 cross-router helpers these endpoints need (and the two
PAYER_FACTORIES dicts they consume) live in
:mod:`cyclone.api_routers._shared`.
"""
from __future__ import annotations
import json
import logging
import uuid
from typing import Any
from fastapi import APIRouter, Depends, File, Query, Request, UploadFile
from fastapi.responses import JSONResponse, StreamingResponse
from sqlalchemy.exc import IntegrityError
from cyclone import db
from cyclone.api_helpers import (
client_wants_json as _client_wants_json,
drop_raw_segments_837 as _drop_raw_segments,
drop_raw_segments_835 as _drop_raw_segments_835,
has_claim_validation_errors as _has_claim_validation_errors,
has_835_validation_errors as _has_835_validation_errors,
ndjson_stream_837 as _ndjson_stream,
ndjson_stream_835 as _ndjson_stream_835,
strict_rewrite_837 as _strict_rewrite,
strict_rewrite_835 as _strict_rewrite_835,
)
from cyclone.api_routers._shared import (
_actor_user_id,
_build_and_persist_ack,
_reconciliation_summary_for_batch,
_resolve_payer,
_resolve_payer_835,
_serialize_ta1,
_ta1_synthetic_source_batch_id,
_transaction_set_id_from_segments,
)
from cyclone.audit_log import AuditEvent, append_event
from cyclone.auth.deps import matrix_gate
from cyclone.claim_acks import (
apply_277ca_acks as _apply_277ca_acks,
apply_999_acceptances as _apply_999_acceptances,
apply_ta1_envelope_link as _apply_ta1_envelope_link,
)
from cyclone.db import Batch, Claim
from cyclone.handlers._ack_id import (
ack_count_summary as _ack_count_summary,
ack_synthetic_source_batch_id as _ack_synthetic_source_batch_id,
two77ca_synthetic_source_batch_id as _277ca_synthetic_source_batch_id,
)
from cyclone.inbox_state import apply_999_rejections
from cyclone.inbox_state_277ca import apply_277ca_rejections
from cyclone.parsers.exceptions import CycloneParseError
from cyclone.parsers.parse_277ca import parse_277ca_text
from cyclone.parsers.parse_837 import parse
from cyclone.parsers.parse_835 import parse as parse_835
from cyclone.parsers.parse_999 import parse_999_text
from cyclone.parsers.parse_ta1 import parse_ta1_text
from cyclone.parsers.segments import tokenize as _tokenize_segments
from cyclone.parsers.serialize_999 import serialize_999
from cyclone.parsers.validator_835 import validate as validate_835
from cyclone.store import BatchRecord, store, utcnow
log = logging.getLogger(__name__)
router = APIRouter(dependencies=[Depends(matrix_gate)])
@router.post("/api/parse-837")
async def parse_837(
request: Request,
file: UploadFile = File(...),
payer: str = Query("co_medicaid"),
include_raw_segments: bool = Query(True),
strict: bool = Query(False),
ack: bool = Query(False),
) -> Any:
# SP35: defense-in-depth input guards. Layer A (UI auto-detect) lives
# in src/pages/Upload.tsx; the server-side checks below are the
# authoritative fix because they protect every caller of the API
# (Upload page, CLI ingestion, any future bulk-import tool). Without
# these, an 835 file dropped on the Upload page while the dropdown
# still says "837p" produces a BatchRecord with claims=[] and a bogus
# row on the History tab. The fix is two checks run BEFORE we persist
# anything:
#
# 1. Envelope check — ST01 must be "837" or "837P". Anything else
# (an 835, a 999, a 270, garbage that happens to have an ISA)
# → 400 with error="Mismatched file kind", expected="837p",
# detected_st=<whatever was there>.
# 2. Empty-claims check — even with the right envelope, if the
# parser produced zero CLM segments (truncated file, header-only
# test fixture) → 400 with error="No claims parsed". A real
# production 837 batch with zero claims is never valid.
raw = await file.read()
if not raw:
return JSONResponse(
status_code=400,
content={"error": "Empty file", "detail": "Uploaded file contained no bytes."},
)
try:
text = raw.decode("utf-8")
except UnicodeDecodeError as exc:
return JSONResponse(
status_code=400,
content={"error": "Encoding error", "detail": str(exc)},
)
config = _resolve_payer(payer)
# SP35 guard 1: envelope check. Tokenize first so we can return a
# precise 400 (vs. relying on the parser's "no ISA envelope" error
# which is correct but doesn't say "you sent an 835 to the 837
# endpoint"). If tokenization itself fails we fall through to the
# parser, which raises CycloneParseError → 400 "Parse error" path.
try:
_segments = _tokenize_segments(text)
detected_st = _transaction_set_id_from_segments(_segments) or ""
except CycloneParseError:
detected_st = ""
if detected_st and not detected_st.upper().startswith("837"):
return JSONResponse(
status_code=400,
content={
"error": "Mismatched file kind",
"expected": "837p",
"detected_st": detected_st,
"detail": (
f"File declares ST*{detected_st}* but this endpoint "
f"expects ST*837*. Pick the matching endpoint on the "
f"Upload page (or let auto-detect choose for you)."
),
},
)
try:
result = parse(text, config, input_file=file.filename or "")
except CycloneParseError as exc:
return JSONResponse(
status_code=400,
content={"error": "Parse error", "detail": str(exc)},
)
except Exception as exc: # pragma: no cover - safety net
log.exception("Unexpected parser failure")
return JSONResponse(
status_code=500,
content={"error": "Internal server error", "detail": str(exc)},
)
# SP35 guard 2: empty-claims check. With the envelope validated, the
# only way to land here is a header-only file (real, but useless)
# or a file whose CLM loops the parser couldn't extract. Either way
# we refuse to persist — a BatchRecord with claims=[] is what the
# original bug produced and is never what the operator wanted.
if not result.claims:
return JSONResponse(
status_code=400,
content={
"error": "No claims parsed",
"detail": (
"The file passed the envelope check but contained no "
"CLM segments. Refusing to persist an empty batch."
),
},
)
if strict:
result = _strict_rewrite(result)
if not include_raw_segments:
result = _drop_raw_segments(result)
if _has_claim_validation_errors(result):
# Per spec: 422 when claims failed validation.
# Body still includes the full ParseResult so the client can show errors.
# Validation-failed parses are NOT persisted (the data is suspect);
# only parses that survive validation end up in the store.
return JSONResponse(
status_code=422,
content=json.loads(result.model_dump_json()),
)
# Persist the cleaned-up result so the cleaned result is what clients
# retrieve via /api/batches/{id}.
rec = BatchRecord(
id=uuid.uuid4().hex,
kind="837p",
input_filename=file.filename or "upload.txt",
parsed_at=utcnow(),
result=result,
)
try:
store.add(rec, event_bus=request.app.state.event_bus)
except IntegrityError as exc:
# ``(batch_id, patient_control_number)`` is UNIQUE — fires when a
# single batch file contains the same CLM01 control number twice,
# or when the same claim id has already been ingested in a prior
# batch. Surface as 409 with the batch id so the caller can look
# it up; do NOT 500 (a 500 without CORS headers is misreported by
# browsers as a CORS error and hides the real cause).
log.warning("Duplicate claim while persisting batch %s: %s", rec.id, exc)
return JSONResponse(
status_code=409,
content={
"error": "Duplicate claim",
"detail": (
"This file (or one previously ingested with the same "
"claim control number) collides with an existing "
"record. Inspect the file for duplicate CLM01 "
"control numbers, or remove the existing batch "
"before retrying."
),
"batch_id": rec.id,
},
)
if _client_wants_json(request):
body = json.loads(result.model_dump_json())
if ack:
ack_body = _build_and_persist_ack(rec.id)
if ack_body is not None:
body["ack"] = ack_body
# Surface the server-side batch id so the frontend can call
# /api/batches/{id}/export-837 (and any other batch-scoped
# endpoint) without a separate listBatches round-trip.
body["batch_id"] = rec.id
return JSONResponse(content=body)
# Default: NDJSON stream. Pass the server-side batch id so the
# streaming client (the React Upload page) can call batch-scoped
# endpoints like /api/batches/{id}/export-837 without a separate
# GET /api/batches round-trip.
return StreamingResponse(
_ndjson_stream(result, batch_id=rec.id),
media_type="application/x-ndjson",
)
@router.post("/api/parse-835")
async def parse_835_endpoint(
request: Request,
file: UploadFile = File(...),
payer: str = Query("co_medicaid_835"),
include_raw_segments: bool = Query(True),
strict: bool = Query(False),
) -> Any:
raw = await file.read()
if not raw:
return JSONResponse(
status_code=400,
content={"error": "Empty file", "detail": "Uploaded file contained no bytes."},
)
try:
text = raw.decode("utf-8")
except UnicodeDecodeError as exc:
return JSONResponse(
status_code=400,
content={"error": "Encoding error", "detail": str(exc)},
)
config = _resolve_payer_835(payer)
# SP35 guard 1: envelope check. Mirrors the parse-837 path: tokenize,
# read ST01, reject anything that doesn't start with "835". Same
# defense-in-depth rationale — the UI auto-detect (src/pages/Upload.tsx)
# is layer A, but server-side guards protect every API caller.
try:
_segments_835 = _tokenize_segments(text)
detected_st_835 = _transaction_set_id_from_segments(_segments_835) or ""
except CycloneParseError:
detected_st_835 = ""
if detected_st_835 and not detected_st_835.upper().startswith("835"):
return JSONResponse(
status_code=400,
content={
"error": "Mismatched file kind",
"expected": "835",
"detected_st": detected_st_835,
"detail": (
f"File declares ST*{detected_st_835}* but this endpoint "
f"expects ST*835*. Pick the matching endpoint on the "
f"Upload page (or let auto-detect choose for you)."
),
},
)
try:
result = parse_835(text, config, input_file=file.filename or "")
except CycloneParseError as exc:
return JSONResponse(
status_code=400,
content={"error": "Parse error", "detail": str(exc)},
)
except Exception as exc: # pragma: no cover - safety net
log.exception("Unexpected parser failure")
return JSONResponse(
status_code=500,
content={"error": "Internal server error", "detail": str(exc)},
)
# SP35 guard 2: empty-claims check. Same as parse-837: a BatchRecord
# with claims=[] is never a valid production 835 batch and we refuse
# to persist it.
if not result.claims:
return JSONResponse(
status_code=400,
content={
"error": "No claims parsed",
"detail": (
"The file passed the envelope check but contained no "
"CLP segments. Refusing to persist an empty batch."
),
},
)
# Always run the validator; attach the report so the JSON path can
# surface it and the NDJSON path can fold the counts into the summary.
# 835 validation is batch-level, so pass/fail applies uniformly to every
# claim payment in the batch (passed=N or 0, failed=0 or N).
report = validate_835(result, config)
n = len(result.claims)
claim_ids = [c.payer_claim_control_number for c in result.claims]
if report.passed:
passed, failed, failed_claim_ids = n, 0, []
else:
passed, failed, failed_claim_ids = 0, n, claim_ids
result = result.model_copy(update={
"validation": report,
"summary": result.summary.model_copy(update={
"passed": passed,
"failed": failed,
"failed_claim_ids": failed_claim_ids,
}),
})
if strict:
result = _strict_rewrite_835(result)
if not include_raw_segments:
result = _drop_raw_segments_835(result)
if _has_835_validation_errors(result):
return JSONResponse(
status_code=422,
content=json.loads(result.model_dump_json()),
)
# Persist the cleaned-up result so the cleaned result is what clients
# retrieve via /api/batches/{id}.
rec = BatchRecord(
id=uuid.uuid4().hex,
kind="835",
input_filename=file.filename or "upload.txt",
parsed_at=utcnow(),
result=result,
)
try:
store.add(rec, event_bus=request.app.state.event_bus)
except IntegrityError as exc:
log.warning("Duplicate remittance while persisting batch %s: %s", rec.id, exc)
return JSONResponse(
status_code=409,
content={
"error": "Duplicate remittance",
"detail": (
"This 835 file (or one previously ingested with the "
"same payer claim control number) collides with an "
"existing record. Remove the existing remittance "
"before retrying."
),
"batch_id": rec.id,
},
)
if _client_wants_json(request):
body = json.loads(result.model_dump_json())
body["reconciliation"] = _reconciliation_summary_for_batch(rec.id)
return JSONResponse(content=body)
# Default: NDJSON stream. Pass the server-side batch id so the
# streaming client can call batch-scoped endpoints without a
# separate GET /api/batches round-trip (see /api/parse-837 for the
# parallel change).
return StreamingResponse(
_ndjson_stream_835(result, batch_id=rec.id),
media_type="application/x-ndjson",
)
@router.post("/api/parse-999")
async def parse_999_endpoint(
request: Request,
file: UploadFile = File(...),
) -> Any:
"""Parse a 999 ACK file, persist a row, and return JSON.
Behavior mirrors ``/api/parse-835``:
- 400 on empty / undecodable / malformed EDI (never 500).
- 200 on success with ``{"ack": {id, accepted_count, rejected_count,
received_count, ack_code, raw_999_text}, "parsed": <ParseResult999>}``.
The persisted ``acks.source_batch_id`` is a synthetic id
(``999-<ISA13>``) because a received 999 has no inbound 837 batch
to FK to. The dashboard's `/acks` list surfaces these; operators
can still see which interchange each row came from.
"""
raw = await file.read()
if not raw:
return JSONResponse(
status_code=400,
content={"error": "Empty file", "detail": "Uploaded file contained no bytes."},
)
try:
text = raw.decode("utf-8")
except UnicodeDecodeError as exc:
return JSONResponse(
status_code=400,
content={"error": "Encoding error", "detail": str(exc)},
)
try:
result = parse_999_text(text, input_file=file.filename or "")
except CycloneParseError as exc:
return JSONResponse(
status_code=400,
content={"error": "Parse error", "detail": str(exc)},
)
except Exception as exc: # pragma: no cover - safety net
log.exception("Unexpected parser failure on 999")
return JSONResponse(
status_code=500,
content={"error": "Internal server error", "detail": str(exc)},
)
received, accepted, rejected, ack_code = _ack_count_summary(result)
icn = result.envelope.control_number
synthetic_id = _ack_synthetic_source_batch_id(icn)
# Build the raw 999 text from the parsed result (round-trip).
raw_999_text = serialize_999(result, interchange_control_number=icn or "000000001")
# SP6 T4: move claims whose 999 set was rejected into ClaimState.REJECTED.
# The 999's set_control_number (AK202) is the source 837's ST02; in
# practice we look it up against patient_control_number because that's
# the field 999 ACKs cross-reference in this product.
with db.SessionLocal()() as session:
def _lookup(pcn: str):
return session.query(Claim).filter_by(patient_control_number=pcn).first()
_rejection_result = apply_999_rejections(
session, result, claim_lookup=_lookup,
)
if _rejection_result.matched:
bus = request.app.state.event_bus
for cid in _rejection_result.matched:
await bus.publish("claim.rejected", {"claim_id": cid})
if _rejection_result.orphans:
log.warning(
"999 had %d orphan set refs: %s",
len(_rejection_result.orphans),
_rejection_result.orphans[:5],
)
row = store.add_ack(
source_batch_id=synthetic_id,
accepted_count=accepted,
rejected_count=rejected,
received_count=received,
ack_code=ack_code,
raw_json=json.loads(result.model_dump_json()),
event_bus=request.app.state.event_bus,
)
# SP11: append one audit row per rejected claim. Each row chains
# to the previous one — see cyclone.audit_log.
if _rejection_result.matched:
with db.SessionLocal()() as audit_s:
for cid in _rejection_result.matched:
append_event(audit_s, AuditEvent(
event_type="claim.rejected",
entity_type="claim",
entity_id=cid,
payload={"source_batch_id": synthetic_id, "ack_id": row.id},
actor="999-parser",
user_id=_actor_user_id(request),
))
audit_s.commit()
# SP28: auto-link the 999 AK2 set-responses to claims via the
# D10 two-pass join (ST02 via batch envelope index primary,
# Claim.patient_control_number fallback). Each created ClaimAck
# row publishes claim_ack_written so the live-tail subscribers
# on the claim and ack side see the link immediately.
claim_ack_links_count = 0
with db.SessionLocal()() as link_s:
batch_index = store.batch_envelope_index()
def _pcn_lookup(pcn: str):
return (
link_s.query(Claim)
.filter(Claim.patient_control_number == pcn)
.first()
)
link_result = _apply_999_acceptances(
link_s, result, ack_id=row.id,
batch_envelope_index=batch_index,
pc_claim_lookup=_pcn_lookup,
)
for link_row in link_result.linked:
store.add_claim_ack(
claim_id=link_row.claim_id,
batch_id=link_row.batch_id,
ack_id=row.id,
ack_kind="999",
ak2_index=link_row.ak2_index,
set_control_number=link_row.set_control_number,
set_accept_reject_code=link_row.set_accept_reject_code,
linked_by="auto",
event_bus=request.app.state.event_bus,
)
claim_ack_links_count += 1
return JSONResponse(content={
"ack": {
"id": row.id,
"accepted_count": accepted,
"rejected_count": rejected,
"received_count": received,
"ack_code": ack_code,
"source_batch_id": synthetic_id,
"raw_999_text": raw_999_text,
"claim_ack_links_count": claim_ack_links_count,
},
"parsed": json.loads(result.model_dump_json()),
})
@router.post("/api/parse-ta1")
async def parse_ta1_endpoint(
request: Request,
file: UploadFile = File(...),
) -> Any:
"""Parse a TA1 (Interchange Acknowledgment) file, persist a row, return JSON.
Mirrors ``/api/parse-999`` but for the lower-level envelope ack:
- 400 on empty / undecodable / malformed EDI (never 500).
- 200 on success with ``{"ta1": {id, control_number, ack_code,
note_code, interchange_date, interchange_time, sender_id,
receiver_id, source_batch_id, raw_ta1_text}, "parsed":
<ParseResultTa1>}``.
The persisted ``ta1_acks.source_batch_id`` is a synthetic id
(``TA1-<ISA13>``) because a received TA1 has no inbound batch to
FK to. The dashboard's ``/ta1-acks`` list surfaces these.
SP25: threads ``event_bus=request.app.state.event_bus`` into
``store.add_ta1_ack`` so the live-tail ``ta1_ack_received``
stream fires on manual uploads (not just on the SFTP poller).
"""
raw = await file.read()
if not raw:
return JSONResponse(
status_code=400,
content={"error": "Empty file", "detail": "Uploaded file contained no bytes."},
)
try:
text = raw.decode("utf-8")
except UnicodeDecodeError as exc:
return JSONResponse(
status_code=400,
content={"error": "Encoding error", "detail": str(exc)},
)
try:
result = parse_ta1_text(text, input_file=file.filename or "")
except CycloneParseError as exc:
return JSONResponse(
status_code=400,
content={"error": "Parse error", "detail": str(exc)},
)
except Exception as exc: # pragma: no cover - safety net
log.exception("Unexpected parser failure on TA1")
return JSONResponse(
status_code=500,
content={"error": "Internal server error", "detail": str(exc)},
)
# Build the raw TA1 text from the parsed result (round-trip).
raw_ta1_text = _serialize_ta1(result)
row = store.add_ta1_ack(
source_batch_id=result.source_batch_id,
control_number=result.ta1.control_number,
interchange_date=result.ta1.interchange_date,
interchange_time=result.ta1.interchange_time,
ack_code=result.ta1.ack_code,
note_code=result.ta1.note_code,
ack_generated_date=result.ta1.ack_generated_date,
sender_id=result.envelope.sender_id,
receiver_id=result.envelope.receiver_id,
raw_json=json.loads(result.model_dump_json()),
event_bus=request.app.state.event_bus,
)
# SP28: TA1 envelope-level link to the originating Batch. The
# closure here matches the most-recent Batch whose envelope
# sender_id/receiver_id matches the TA1 — see spec §D4.
claim_ack_links_count = 0
with db.SessionLocal()() as link_s:
def _batch_lookup(sender_id, receiver_id):
rows = (
link_s.query(Batch)
.filter(
Batch.kind == "837p",
Batch.raw_result_json.isnot(None),
)
.order_by(Batch.parsed_at.desc())
.all()
)
for row in rows:
env = (row.raw_result_json or {}).get("envelope") or {}
if (
env.get("sender_id") == sender_id
and env.get("receiver_id") == receiver_id
):
return row
return None
link_result = _apply_ta1_envelope_link(
link_s, result, ack_id=row.id,
batch_lookup=_batch_lookup,
)
for link_row in link_result.linked:
store.add_claim_ack(
claim_id=link_row.claim_id,
batch_id=link_row.batch_id,
ack_id=row.id,
ack_kind="ta1",
ak2_index=link_row.ak2_index,
set_control_number=link_row.set_control_number,
set_accept_reject_code=link_row.set_accept_reject_code,
linked_by="auto",
event_bus=request.app.state.event_bus,
)
claim_ack_links_count += 1
return JSONResponse(content={
"ta1": {
"id": row.id,
"control_number": result.ta1.control_number,
"ack_code": result.ta1.ack_code,
"note_code": result.ta1.note_code,
"interchange_date": result.ta1.interchange_date.isoformat()
if result.ta1.interchange_date else None,
"interchange_time": result.ta1.interchange_time,
"sender_id": result.envelope.sender_id,
"receiver_id": result.envelope.receiver_id,
"source_batch_id": result.source_batch_id,
"raw_ta1_text": raw_ta1_text,
"claim_ack_links_count": claim_ack_links_count,
},
"parsed": json.loads(result.model_dump_json()),
})
@router.post("/api/parse-277ca")
async def parse_277ca_endpoint(
request: Request,
file: UploadFile = File(...),
) -> Any:
"""Parse a 277CA Claim Acknowledgment file, persist a row, and stamp rejections.
Behavior mirrors ``/api/parse-999``:
- 400 on empty / undecodable / malformed EDI (never 500).
- 200 on success with ``{"ack": {id, control_number, accepted_count,
rejected_count, payer_claim_control_numbers, raw_277ca_text},
"parsed": <ParseResult277CA>}``.
After parse, runs :func:`apply_277ca_rejections` to stamp the
payer-rejected fields on each matching claim row. The Inbox
Payer-Rejected lane lights up as a side-effect of this call.
"""
raw = await file.read()
if not raw:
return JSONResponse(
status_code=400,
content={"error": "Empty file", "detail": "Uploaded file contained no bytes."},
)
try:
text = raw.decode("utf-8")
except UnicodeDecodeError as exc:
return JSONResponse(
status_code=400,
content={"error": "Encoding error", "detail": str(exc)},
)
try:
result = parse_277ca_text(text, input_file=file.filename or "")
except CycloneParseError as exc:
return JSONResponse(
status_code=400,
content={"error": "Parse error", "detail": str(exc)},
)
except Exception as exc: # pragma: no cover - safety net
log.exception("Unexpected parser failure on 277CA")
return JSONResponse(
status_code=500,
content={"error": "Internal server error", "detail": str(exc)},
)
icn = result.envelope.control_number
synthetic_id = _277ca_synthetic_source_batch_id(icn)
accepted = sum(1 for s in result.claim_statuses if s.classification == "accepted")
paid = sum(1 for s in result.claim_statuses if s.classification == "paid")
rejected = sum(1 for s in result.claim_statuses if s.classification == "rejected")
pended = sum(1 for s in result.claim_statuses if s.classification == "pended")
# Persist the 277CA row first so we have an id to attach to claims.
# SP25: thread the event bus so ``two77ca_ack_received`` fires.
row = store.add_277ca_ack(
source_batch_id=synthetic_id,
control_number=icn,
accepted_count=accepted,
rejected_count=rejected,
paid_count=paid,
pended_count=pended,
raw_json=json.loads(result.model_dump_json()),
event_bus=request.app.state.event_bus,
)
# Stamp payer-rejection fields on matching claims. The 277CA's
# REF*1K carries the patient's claim control number we sent in
# CLM01 — same convention the 999 ACK uses, so the lookup hits
# Claim.patient_control_number (mirrors apply_999_rejections).
with db.SessionLocal()() as session:
def _lookup(pcn: str):
return (
session.query(Claim)
.filter(Claim.patient_control_number == pcn)
.first()
)
apply_result = apply_277ca_rejections(
session, result, claim_lookup=_lookup, two77ca_id=row.id,
)
if apply_result.matched:
bus = request.app.state.event_bus
for cid in apply_result.matched:
await bus.publish("claim.payer_rejected", {"claim_id": cid})
# SP11: audit trail for each payer-rejected claim.
with db.SessionLocal()() as audit_s:
for cid in apply_result.matched:
append_event(audit_s, AuditEvent(
event_type="claim.payer_rejected",
entity_type="claim",
entity_id=cid,
payload={
"source_batch_id": synthetic_id,
"277ca_id": row.id,
},
actor="277ca-parser",
user_id=_actor_user_id(request),
))
audit_s.commit()
if apply_result.orphans:
log.warning(
"277CA had %d orphan status entries (no matching claim): %s",
len(apply_result.orphans),
apply_result.orphans[:5],
)
# SP28: auto-link the 277CA ClaimStatus entries to claims via
# the D10 two-pass join. Each ClaimAck row publishes
# claim_ack_written so the live-tail subscribers on the claim
# and ack side see the link immediately.
claim_ack_links_count = 0
with db.SessionLocal()() as link_s:
batch_index = store.batch_envelope_index()
def _pcn_lookup(pcn: str):
return (
link_s.query(Claim)
.filter(Claim.patient_control_number == pcn)
.first()
)
link_result = _apply_277ca_acks(
link_s, result, ack_id=row.id,
batch_envelope_index=batch_index,
pc_claim_lookup=_pcn_lookup,
)
for link_row in link_result.linked:
store.add_claim_ack(
claim_id=link_row.claim_id,
batch_id=link_row.batch_id,
ack_id=row.id,
ack_kind="277ca",
ak2_index=link_row.ak2_index,
set_control_number=link_row.set_control_number,
set_accept_reject_code=link_row.set_accept_reject_code,
linked_by="auto",
event_bus=request.app.state.event_bus,
)
claim_ack_links_count += 1
return JSONResponse(content={
"ack": {
"id": row.id,
"control_number": icn,
"accepted_count": accepted,
"rejected_count": rejected,
"paid_count": paid,
"pended_count": pended,
"source_batch_id": synthetic_id,
"matched_claim_ids": apply_result.matched,
"orphan_status_codes": apply_result.orphans,
"claim_ack_links_count": claim_ack_links_count,
},
"parsed": json.loads(result.model_dump_json()),
})
+149
View File
@@ -0,0 +1,149 @@
"""``GET /api/payers/{payer_id}/summary`` — payer-level rollup for the drill-down panel.
Returns billed/received totals, denial rate, and the top providers for
a given ``payer_id`` (the X12 NM1*PR*PI qualifier, e.g. ``SKCO0``).
Cached in-process for ``_SUMMARY_TTL_S`` seconds via a per-payer_id
memo. The drill-down UI hammers this on every hover; the underlying
query is O(claims) per payer, so the TTL keeps the panel responsive.
Pubsub invalidation is intentionally NOT wired (see the long comment
in the body). The 60s TTL keeps staleness bounded; a follow-up can
wire targeted invalidation if the UI proves TTL-bounded staleness is
unacceptable.
404 when no claims AND no remits reference this payer_id — the UI
uses that to distinguish a typo from a payer with zero traffic (the
latter would still return a valid 200 with zeroed totals).
SP36 Task 6: this block moved here from ``api.py:3188`` (the
``SP21 Task 1.5: payer-level summary`` divider).
"""
from __future__ import annotations
import logging
from time import monotonic
from fastapi import APIRouter, Depends, HTTPException
from cyclone import db
from cyclone.auth.deps import matrix_gate
from cyclone.db import Claim, ClaimState, Remittance
log = logging.getLogger(__name__)
router = APIRouter(dependencies=[Depends(matrix_gate)])
# Per-payer_id memoization for /api/payers/{payer_id}/summary. The drill-
# down UI hammers this on every hover; the underlying query is O(claims)
# per payer so a 60s TTL keeps the panel responsive. Process-local on
# purpose — invalidation is driven by the TTL alone for now (see note
# below).
_SUMMARY_TTL_S = 60.0
_summary_cache: dict[str, tuple[float, dict]] = {}
def _clear_summary_cache() -> None:
"""Test hook: wipe the process-local cache.
The 60s TTL means tests that want to assert on a recomputed payload
must clear the cache between requests. Not used by production code.
"""
_summary_cache.clear()
# Pubsub invalidation is intentionally NOT wired. The ``claim_written``
# and ``remittance_written`` payloads emitted by ``store.add`` are the
# UI-shaped dicts from ``to_ui_claim_from_orm`` / ``to_ui_remittance_from_orm``,
# neither of which carries the X12 ``payer_id`` (NM1*PR*PI qualifier).
# They carry ``payerName`` only, which is the human-readable name and not
# the URL key we cache on. Wiring a subscriber here would either need a
# DB lookup per event (re-deriving payer_id from Claim.id) or a different
# cache key — both are out of scope for this task. The 60s TTL keeps
# staleness bounded; a follow-up can wire targeted invalidation if the
# UI proves TTL-bounded staleness is unacceptable.
@router.get("/api/payers/{payer_id}/summary")
def get_payer_summary(payer_id: str):
"""Payer-level rollup for the drill-down panel.
Returns billed/received totals, denial rate, and top providers for
the given ``payer_id`` (the X12 NM1*PR*PI qualifier, e.g. ``SKCO0``).
Cached in-process for ``_SUMMARY_TTL_S`` seconds.
404 when no claims AND no remits reference this payer_id — the UI
uses that to distinguish a typo from a payer with zero traffic
(the latter would still return a valid 200 with zeroed totals).
"""
now = monotonic()
cached = _summary_cache.get(payer_id)
if cached and (now - cached[0]) < _SUMMARY_TTL_S:
return cached[1]
log.debug("payer summary cache miss", extra={"payer_id": payer_id})
# Query claims + remits scoped to this payer_id. We bypass
# ``store.iter_claims`` because that helper filters by payer NAME
# substring, not by the X12 PI qualifier we use as the URL key.
with db.SessionLocal()() as s:
claim_rows: list[Claim] = (
s.query(Claim).filter(Claim.payer_id == payer_id).all()
)
claim_ids = [c.id for c in claim_rows]
remit_rows: list[Remittance] = []
if claim_ids:
remit_rows = (
s.query(Remittance)
.filter(Remittance.claim_id.in_(claim_ids))
.all()
)
if not claim_rows and not remit_rows:
raise HTTPException(
status_code=404,
detail=f"Payer {payer_id!r} not found",
)
billed_total = sum(float(c.charge_amount or 0) for c in claim_rows)
received_total = sum(float(r.total_paid or 0) for r in remit_rows)
denied = sum(
1 for c in claim_rows if c.state == ClaimState.DENIED
)
claim_count = len(claim_rows)
denial_rate = (denied / claim_count) if claim_count else 0.0
provider_counts: dict[str, int] = {}
for c in claim_rows:
npi = c.provider_npi
if not npi:
continue
provider_counts[npi] = provider_counts.get(npi, 0) + 1
top_providers = [
{"npi": npi, "count": count}
for npi, count in sorted(
provider_counts.items(), key=lambda kv: -kv[1]
)[:5]
]
# Best-effort payer display name from the first claim's raw
# payer object (NM1*PR name element, e.g. "COHCPF"). Falls
# back to the id when no parsed envelope is available.
payer_name = payer_id
for c in claim_rows:
raw = c.raw_json or {}
p = raw.get("payer") if isinstance(raw, dict) else None
if isinstance(p, dict) and p.get("name"):
payer_name = p["name"]
break
payload = {
"payer_id": payer_id,
"name": payer_name,
"claim_count": claim_count,
"billed_total": billed_total,
"received_total": received_total,
"denial_rate": denial_rate,
"top_providers": top_providers,
}
_summary_cache[payer_id] = (now, payload)
return payload
@@ -0,0 +1,150 @@
"""``/api/providers`` and ``/api/config/providers*`` — read views over providers.
Three endpoints across two URL prefixes, all gated by ``matrix_gate``:
- ``GET /api/providers`` — distinct provider list
derived from the Claims population (``store.distinct_providers()``),
with ``npi`` / ``state`` filters, pagination, and an NDJSON variant.
- ``GET /api/config/providers`` — the configured provider
rows (3 NPIs for SP9), filtered by ``is_active``.
- ``GET /api/config/providers/{npi}`` — one configured provider
plus a small drill-down block: ``recent_claims`` (top-10 by
``submissionDate``) and ``recent_activity`` (top-10, joined via
``Claim.id`` because ``ActivityEvent`` has no ``provider_npi``
column; the outer-join via ``Remittance.claim_id`` surfaces the
orphan ``remit_received`` events that were recorded pre-match).
SP36 Task 12: this block moved here from ``api.py:2448`` (the
``/api/providers`` list, the ``/api/config/providers`` list, and
the ``/api/config/providers/{npi}`` detail) — three URL prefixes,
one router per spec.
"""
from __future__ import annotations
import json
from typing import Any
from fastapi import APIRouter, Depends, HTTPException, Query, Request
from fastapi.responses import StreamingResponse
from sqlalchemy import desc, or_
from cyclone import db
from cyclone.api_helpers import (
ndjson_stream_list as _ndjson_stream_list,
wants_ndjson as _wants_ndjson,
)
from cyclone.auth.deps import matrix_gate
from cyclone.db import Claim, Remittance
from cyclone.store import store
router = APIRouter(dependencies=[Depends(matrix_gate)])
@router.get("/api/providers")
def list_providers(
request: Request,
npi: str | None = Query(None),
state: str | None = Query(None),
limit: int = Query(100, ge=1, le=1000),
offset: int = Query(0, ge=0),
) -> Any:
items = store.distinct_providers()
if npi is not None:
items = [p for p in items if p["npi"] == npi]
if state is not None:
items = [p for p in items if p.get("state") == state]
paged = items[offset:offset + limit]
total = len(items)
returned = len(paged)
has_more = total > offset + returned
if _wants_ndjson(request):
return StreamingResponse(
_ndjson_stream_list(paged, total, returned, has_more),
media_type="application/x-ndjson",
)
return {
"items": paged,
"total": total,
"returned": returned,
"has_more": has_more,
}
@router.get("/api/config/providers")
def list_configured_providers(is_active: bool | None = Query(default=True)):
"""List the configured provider rows (3 NPIs for SP9)."""
return [json.loads(p.model_dump_json()) for p in store.list_providers(is_active=is_active)]
@router.get("/api/config/providers/{npi}")
def get_configured_provider(npi: str):
p = store.get_provider(npi)
if p is None:
raise HTTPException(status_code=404, detail=f"provider {npi!r} not found")
provider_dict = json.loads(p.model_dump_json())
# SP21 Task 1.6: extend the response with two top-N arrays that the
# drill-down peek panel hangs off. ``recent_claims`` reuses the
# existing store projection (already returns UI-shaped dicts with
# ``submissionDate``); ``recent_activity`` is a direct ORM join
# because ``ActivityEvent`` has no ``provider_npi`` column — the
# filter has to hop through ``Claim.id``.
recent_claims = sorted(
store.iter_claims(provider_npi=npi),
key=lambda c: c.get("submissionDate") or "",
reverse=True,
)[:10]
# Activity filter has TWO join paths back to a Claim for this
# provider:
# 1. ``ActivityEvent.claim_id IN (claim_ids)`` — events that were
# recorded with a claim FK already set (claim_submitted,
# manual_match, claim_paid, etc.).
# 2. ``Remittance.claim_id IN (claim_ids)`` — the original
# ``remit_received`` event recorded at 835 ingest time
# (``store.add`` lines 999-1003) is inserted with
# ``claim_id=None`` because the remittance hasn't been matched
# to a claim yet. The auto-reconcile pass later sets
# ``Remittance.claim_id`` (``reconcile.run`` lines 289-293),
# but the *original* ActivityEvent row stays orphaned. The
# outer-join-then-OR lets us surface both shapes. Without
# path 2, a provider's activity feed looks frozen the moment
# an 835 lands — the most common activity, invisible.
with db.SessionLocal()() as s:
claim_ids = [
cid
for (cid,) in s.query(Claim.id)
.filter(Claim.provider_npi == npi)
.all()
]
activity_rows = []
if claim_ids:
activity_rows = (
s.query(db.ActivityEvent)
.outerjoin(
Remittance,
db.ActivityEvent.remittance_id == Remittance.id,
)
.filter(or_(
db.ActivityEvent.claim_id.in_(claim_ids),
Remittance.claim_id.in_(claim_ids),
))
.order_by(desc(db.ActivityEvent.ts))
.limit(10)
.all()
)
def _activity_to_ui(a):
return {
"id": a.id,
"ts": a.ts.isoformat().replace("+00:00", "Z") if a.ts else "",
"kind": a.kind,
"batchId": a.batch_id,
"claimId": a.claim_id,
"remittanceId": a.remittance_id,
"payload": a.payload_json or {},
}
provider_dict["recent_claims"] = recent_claims
provider_dict["recent_activity"] = [_activity_to_ui(a) for a in activity_rows]
return provider_dict
@@ -0,0 +1,178 @@
"""Reconciliation read views + manual match/unmatch write paths.
Four endpoints:
- ``GET /api/reconciliation/unmatched`` — list of unmatched Claims
and unmatched Remittances (``store.list_unmatched(kind="both")``).
- ``GET /api/batch-diff`` — side-by-side diff of two
batches (used by the Batch Diff page). Lazy-imports
:func:`cyclone.batch_diff.diff_batches_to_wire` to keep the
module's import surface small until the endpoint is actually
hit.
- ``POST /api/reconciliation/match`` — manually pair a Claim with
a Remittance (``store.manual_match``). Surfaces ``AlreadyMatchedError`` /
``InvalidStateError`` as 409, and ``LookupError`` from the store
as 404 (``claim_or_remit_not_found``).
- ``POST /api/reconciliation/unmatch`` — unpair a Claim (``store.manual_unmatch``).
Surfaces ``NotMatchedError`` as 409.
All four are read-or-manual-override surfaces used by the
Reconciliation page (the page that pairs Claims with Remittances
when neither side has an automatic match key).
SP36 Task 8: this block moved here from ``api.py:2450`` (the
4 routes interleaved with a side-by-side batch-diff divider).
"""
from __future__ import annotations
from fastapi import APIRouter, Depends, HTTPException, Query
from cyclone.auth.deps import matrix_gate
from cyclone.store import (
AlreadyMatchedError,
InvalidStateError,
store,
)
router = APIRouter(dependencies=[Depends(matrix_gate)])
@router.get("/api/reconciliation/unmatched")
def get_reconciliation_unmatched() -> dict:
"""Return unmatched Claims (left) and unmatched Remittances (right).
Powers the reconciliation review surface: every Claim with no
paired Remittance appears on the left, every Remittance with no
paired Claim appears on the right. The two lists are always present
(empty list, never absent) so the UI can index unconditionally.
"""
return store.list_unmatched(kind="both")
# --------------------------------------------------------------------------- #
# Side-by-side diff between two batches (SP3 P4 / T18)
# --------------------------------------------------------------------------- #
@router.get("/api/batch-diff")
def get_batch_diff(
a: str | None = Query(None),
b: str | None = Query(None),
) -> dict:
"""Return a side-by-side diff of two batches identified by id.
Query params: ``a=<batch_id>``, ``b=<batch_id>`` (both required).
Response body (snake_case keys, see :mod:`cyclone.batch_diff` for the
projector shapes):
- ``a`` / ``b`` — small metadata blocks (id, kind, parsedAt,
inputFilename, claimCount)
- ``added`` — claims present in B but not A
- ``removed`` — claims present in A but not B
- ``changed`` — claims present in both, with field deltas
- ``summary`` — precomputed counts
Errors:
- 400 — missing ``a`` or ``b``
- 404 — either batch id is unknown
Pure read endpoint — never mutates the store. Both 837P and 835
batches are accepted (mixed-kind diffs are valid: comparing the
submitted claims against the matching remittances).
"""
if not a or not b:
raise HTTPException(
status_code=400,
detail={"error": "Missing param", "detail": "Both ?a=<batch_id> and ?b=<batch_id> are required."},
)
try:
a_rec, b_rec = store.load_two_for_diff(a, b)
except LookupError as exc:
raise HTTPException(
status_code=404,
detail={"error": "Not found", "detail": str(exc)},
)
# Lazy import — keeps the module's import surface small until the
# endpoint is actually hit. Mirrors the same pattern used by other
# endpoint-local helpers (e.g. reconciler).
from cyclone.batch_diff import diff_batches_to_wire
return diff_batches_to_wire(a_rec, b_rec)
@router.post("/api/reconciliation/match")
def post_reconciliation_match(body: dict) -> dict:
"""Manually pair a Claim with a Remittance (operator override).
Body: ``{"claim_id": ..., "remit_id": ...}``. Returns
``{"claim": <ui>, "match": <ui>}`` on success. Errors:
- 400: missing ``claim_id`` or ``remit_id``
- 404: claim or remittance not found
- 409: claim already matched, or apply_* returned a noop
(claim in terminal state) — detail echoes ``current_state``
and ``activity_kind`` so the UI can render a precise message.
"""
claim_id = body.get("claim_id")
remit_id = body.get("remit_id")
if not claim_id or not remit_id:
raise HTTPException(
status_code=400,
detail="claim_id and remit_id required",
)
try:
return store.manual_match(claim_id, remit_id)
except AlreadyMatchedError as e:
raise HTTPException(
status_code=409,
detail={"error": "already_matched", "message": str(e)},
)
except InvalidStateError as e:
raise HTTPException(
status_code=409,
detail={
"error": "invalid_state",
"current_state": e.current_state,
"activity_kind": e.activity_kind,
},
)
except LookupError:
# manual_match raises LookupError when the claim or remittance
# row is missing (we catch the parent class so any future
# KeyError subclasses in the store get the same treatment).
raise HTTPException(
status_code=404,
detail="claim_or_remit_not_found",
)
@router.post("/api/reconciliation/unmatch")
def post_reconciliation_unmatch(body: dict) -> dict:
"""Remove the current match for a Claim; reset Claim to submitted.
Body: ``{"claim_id": ...}``. Returns
``{"claim": <ui>, "deletedMatches": <count>}``. Errors:
- 400: missing ``claim_id``
- 404: claim not found
- 409: claim has no current match (NotMatchedError is mapped
by the store; we surface 409 to match the manual_match contract)
"""
from cyclone.store import NotMatchedError
claim_id = body.get("claim_id")
if not claim_id:
raise HTTPException(
status_code=400,
detail="claim_id required",
)
try:
return store.manual_unmatch(claim_id)
except NotMatchedError as e:
raise HTTPException(
status_code=409,
detail={"error": "not_matched", "message": str(e)},
)
except LookupError:
raise HTTPException(
status_code=404,
detail="claim_not_found",
)
@@ -0,0 +1,169 @@
"""``/api/remittances`` and ``/api/remittances/stream`` — read views over the Remittance population.
Four endpoints, all gated by ``matrix_gate``:
- ``GET /api/remittances`` — paginated list with filter+sort,
plus an NDJSON variant when the caller sends ``Accept: application/x-ndjson``.
- ``GET /api/remittances/summary`` — server-aggregated KPI tiles
(``count``, ``total_paid``, ``total_adjustments``) over the full
filtered population — never a page-limited sample (SP27 fix).
- ``GET /api/remittances/stream`` — NDJSON live-tail: snapshot
of currently-known rows, then ``remittance_written`` events as
they hit the store. Subscribed to by the Remittances page.
- ``GET /api/remittances/{remittance_id}`` — one remittance with its
labeled CAS ``adjustments`` array. 404 on missing id (never 500).
``/api/remittances/stream`` is registered before
``/api/remittances/{remittance_id}`` so the literal ``stream`` path
segment is not captured as a remittance id.
SP36 Task 9: this block moved here from ``api.py:2448`` (the 4
``/api/remittances*`` routes, with the streaming route's
``NOTE: registered before…`` comment preserved verbatim).
"""
from __future__ import annotations
from typing import Any, AsyncIterator
from fastapi import APIRouter, Depends, HTTPException, Query, Request
from fastapi.responses import StreamingResponse
from cyclone.api_helpers import (
ndjson_line as _ndjson_line,
ndjson_stream_list as _ndjson_stream_list,
tail_events as _tail_events,
wants_ndjson as _wants_ndjson,
)
from cyclone.auth.deps import matrix_gate
from cyclone.pubsub import EventBus
from cyclone.store import store
router = APIRouter(dependencies=[Depends(matrix_gate)])
@router.get("/api/remittances")
def list_remittances(
request: Request,
batch_id: str | None = Query(None),
payer: str | None = Query(None),
claim_id: str | None = Query(None),
date_from: str | None = Query(None),
date_to: str | None = Query(None),
sort: str | None = Query(None),
order: str = Query("desc"),
limit: int = Query(100, ge=1, le=1000),
offset: int = Query(0, ge=0),
) -> Any:
common = dict(
batch_id=batch_id,
payer=payer,
claim_id=claim_id,
date_from=date_from,
date_to=date_to,
)
items = list(store.iter_remittances(
sort=sort, order=order, limit=limit, offset=offset, **common,
))
# SP27 Task 13b: count the full population, not a 100-row sample.
# See the matching note in list_claims — same silent-failure pattern.
total = store.count_remittances(**common)
returned = len(items)
has_more = total > offset + returned
if _wants_ndjson(request):
return StreamingResponse(
_ndjson_stream_list(items, total, returned, has_more),
media_type="application/x-ndjson",
)
return {
"items": items,
"total": total,
"returned": returned,
"has_more": has_more,
}
@router.get("/api/remittances/summary")
def remittances_summary(
batch_id: str | None = Query(None),
payer: str | None = Query(None),
claim_id: str | None = Query(None),
date_from: str | None = Query(None),
date_to: str | None = Query(None),
) -> dict:
"""Server-aggregated KPI tiles for the Remittances page.
Returns ``{count, total_paid, total_adjustments}`` over the
full filtered remittance population — NOT a page-limited
sample. The Remittances page consumes this for its "Total paid"
and "Adjustments" tiles so they can't silently understate the
true DB population the way a page-local ``items.reduce(...)``
would. Mirrors the silent-incompleteness fix that
``/api/dashboard/kpis`` (commit ``59c3275``) and
``/api/remittances`` (commit ``d81b6ed``) made for their tiles.
Same filter parameters as ``/api/remittances``. Always returns
a populated dict (``{"count": 0, "total_paid": 0,
"total_adjustments": 0}`` when no rows match) so the frontend
can render the tiles directly without a loading-vs-empty
branch.
"""
return store.summarize_remittances(
batch_id=batch_id, payer=payer, claim_id=claim_id,
date_from=date_from, date_to=date_to,
)
@router.get("/api/remittances/stream")
async def remittances_stream(
request: Request,
payer: str | None = Query(None),
claim_id: str | None = Query(None),
date_from: str | None = Query(None),
date_to: str | None = Query(None),
sort: str | None = Query(None),
order: str = Query("desc"),
limit: int = Query(100, ge=1, le=1000),
) -> StreamingResponse:
"""Stream Remittances as NDJSON: snapshot first, then live events.
Subscribes to ``remittance_written``. Default sort is
``-received_date`` (newest-first), matching the list endpoint's
most common sort.
NOTE: registered before ``/api/remittances/{remittance_id}`` so
the literal ``stream`` path segment doesn't get matched as a
remittance id.
"""
bus: EventBus = request.app.state.event_bus
async def gen() -> AsyncIterator[bytes]:
rows = store.iter_remittances(
payer=payer, claim_id=claim_id,
date_from=date_from, date_to=date_to,
sort=sort or "-received_date", order=order, limit=limit,
)
for row in rows:
yield _ndjson_line({"type": "item", "data": row})
yield _ndjson_line({"type": "snapshot_end", "data": {"count": len(rows)}})
async for chunk in _tail_events(request, bus, ["remittance_written"]):
yield chunk
return StreamingResponse(gen(), media_type="application/x-ndjson")
@router.get("/api/remittances/{remittance_id}")
def get_remittance(remittance_id: str) -> dict:
"""Return one remittance with its labeled CAS ``adjustments`` array.
Path param is ``remittance_id`` (not ``id``) to avoid shadowing
FastAPI's internal ``id`` name and to keep OpenAPI docs self-
describing. Returns 404 when the remittance is missing — never 500.
"""
body = store.get_remittance(remittance_id)
if body is None:
raise HTTPException(
status_code=404,
detail={"error": "Not found", "detail": f"Remittance {remittance_id} not found"},
)
return body
@@ -0,0 +1,186 @@
"""SP37 Task 6: HTTP endpoint for the canonical submit-batch flow.
Thin wrapper around ``cyclone.submission.submit_file`` — same logic as
the ``cyclone submit-batch`` CLI (SP37 Task 5), just framed as JSON in
/ JSON out and gated by ``matrix_gate``. The walker pattern, ``._*``
AppleDouble skip, ``limit`` semantics, and per-file outcomes all match
the CLI byte-for-byte so a batch run via the CLI and the same batch
run via this endpoint produce identical DB + SFTP state.
The endpoint deliberately does NOT inject an ``sftp_client_factory``:
``submit_file`` defaults to its paramiko-based factory so SKIPPED is
reachable in production (the ``SftpClient`` wrapper has no ``stat()``).
Tests monkey-patch ``cyclone.api_routers.submission.submit_file``
itself; that avoids the paramiko factory entirely without touching
the helper's contract.
Status code contract (per Task 6 spec §4):
- 200: completed run. Per-file failures live in the JSON body.
- 401: not authenticated (matrix_gate).
- 404: no clearhouse seeded (config-level "missing" → 4xx, not 5xx).
- 409: clearhouse SFTP block is in stub mode (refuses to upload).
- 422: ``ingest_dir`` missing on disk OR Pydantic body validation
failed (missing fields, wrong types).
- 5xx: truly unexpected exceptions propagate (do not swallow).
"""
from __future__ import annotations
import logging
from pathlib import Path
from fastapi import APIRouter, Depends, HTTPException
from pydantic import BaseModel, ConfigDict, Field
from cyclone.auth.deps import matrix_gate
from cyclone.store import store as cycl_store
from cyclone.submission.core import submit_file
from cyclone.submission.result import SubmitOutcome, SubmitResult
log = logging.getLogger(__name__)
# No `prefix=` here — every other gated router in this package declares
# the full path in the decorator (clearhouse.py uses "/api/clearhouse",
# parse.py uses "/api/parse-837", etc.). The decorator sets the full URL.
router = APIRouter(
tags=["submission"],
dependencies=[Depends(matrix_gate)],
)
class SubmitBatchRequest(BaseModel):
"""Body schema for ``POST /api/submit-batch``.
``ingest_dir`` has no default so Pydantic raises 422 when it's
missing — better UX than letting the walker crash on a missing
path. ``validate_files`` and ``actor`` default so a minimal client
can skip them. ``limit`` truncates the file list after the walker
collects it (mirrors the CLI's post-collection ``if i > limit:
break`` semantics, but applied as a slice since the HTTP body
model is type-checked up-front).
The ``validate_files`` field is aliased to ``validate`` in the JSON
body to mirror the CLI's ``--validate`` flag and avoid the
hardcoded Pydantic warning about ``validate`` shadowing
``BaseModel.validate``. ``populate_by_name=True`` lets tests
construct the model with either key.
"""
model_config = ConfigDict(populate_by_name=True, protected_namespaces=())
ingest_dir: str
validate_files: bool = Field(default=True, alias="validate")
actor: str = "api-submit-batch"
limit: int | None = None
@router.post("/api/submit-batch")
def submit_batch(body: SubmitBatchRequest):
"""Submit every ``batch-*-claims/*.x12`` under ``ingest_dir``.
Walks ``ingest_dir`` for any directory matching ``batch-*-claims``,
collects each one's ``*.x12`` files (sorted, with ``._*``
AppleDouble files skipped), truncates to ``limit`` if set, then
calls :func:`cyclone.submission.submit_file` per file with the
seeded clearhouse's ``sftp_block`` and ``actor`` from the body.
Returns counts (``submitted`` / ``skipped`` / ``failed``) plus a
per-file ``results`` array. Per-file failures NEVER change the
HTTP status code — the response is 200 whenever the run itself
completed.
"""
# 1. Config-level guards. Order matters: a missing clearhouse is a
# 404 (config-level "missing"), but if it IS present and in stub
# mode the operator's request is a 409 (configured-but-wrong).
clearhouse = cycl_store.get_clearhouse()
if clearhouse is None:
raise HTTPException(
status_code=404, detail="no clearhouse seeded",
)
sftp_block = clearhouse.sftp_block
if sftp_block.stub:
raise HTTPException(
status_code=409, detail="clearhouse SFTP block is in stub mode",
)
# 2. Resolve + validate the ingest dir. Use ``resolve()`` so a
# symlink-relative path still produces a stable error message.
root = Path(body.ingest_dir).resolve()
if not root.exists():
raise HTTPException(
status_code=422,
detail=f"ingest_dir does not exist: {root}",
)
# 3. Walker — must match the CLI EXACTLY. Same sort, same ``._*``
# AppleDouble skip. Any drift here is a quiet split between the
# two surfaces and silently produces different batch outcomes.
files: list[Path] = []
for batch_dir in sorted(root.glob("batch-*-claims")):
files.extend(sorted(
p for p in batch_dir.glob("*.x12")
if not p.name.startswith("._")
))
# 4. ``limit`` truncates after collection. The CLI uses an inline
# ``if i > limit: break``; we slice instead because the HTTP
# body model validates ``limit`` up-front (Pydantic-level int
# check) and slicing keeps the walker branchless.
if body.limit is not None:
files = files[: body.limit]
if not files:
raise HTTPException(
status_code=422,
detail=f"no batch-*-claims/*.x12 files found under {root}",
)
# 5. Per-file submit. Wrap the helper call in try/except so an
# unexpected exception in submit_file surfaces as a per-file
# failure (outcome="unexpected") instead of crashing the whole
# run. The helper's own SubmitOutcome enum covers every typed
# failure path; an uncaught exception here is a true
# surprise (bug or service outage mid-loop).
results: list[dict] = []
submitted = skipped = failed = 0
for src in files:
try:
r = submit_file(
src,
sftp_block=sftp_block,
actor=body.actor,
validate=body.validate_files,
# No ``sftp_client_factory`` — submit_file's default
# paramiko factory opens the real MFT. Tests
# monkey-patch submit_file itself instead of wiring a
# factory here.
)
except Exception as exc: # noqa: BLE001
log.exception(
"submit-batch unexpected error on %s", src.name,
)
r = SubmitResult(
file=src.name,
outcome=SubmitOutcome.SFTP_FAILED,
error=f"{exc.__class__.__name__}: {exc}",
)
if r.outcome == SubmitOutcome.SUBMITTED:
submitted += 1
elif r.outcome == SubmitOutcome.SKIPPED:
skipped += 1
else:
failed += 1
results.append({
"file": r.file,
"outcome": r.outcome.value,
"batch_id": r.batch_id,
"error": r.error,
})
return {
"submitted": submitted,
"skipped": skipped,
"failed": failed,
"results": results,
}
+76 -22
View File
@@ -1,4 +1,4 @@
"""``/api/ta1-acks`` — list & detail endpoints for persisted TA1 envelopes.
"""``/api/ta1-acks`` — list, detail, and live-tail stream for TA1 envelopes.
TA1 is the interchange-control ACK (ISA/IEA acknowledgement). It's a
single segment, no functional group, no transaction set. Cyclone
@@ -7,34 +7,30 @@ row can sit alongside the 999 / 277CA ack rows without special-casing.
The detail endpoint also reconstructs the TA1 segment string
(``TA1*...~``) so the operator can copy it into a downstream tool.
SP25: ``/api/ta1-acks/stream`` joins the live-tail triplet — the
Acks page mounts ``useTailStream("ta1_acks")`` so the TA1 envelope
ack section sees new rows the moment they land.
"""
from __future__ import annotations
from typing import Any
from typing import Any, AsyncIterator
from fastapi import APIRouter, HTTPException, Query
from fastapi import APIRouter, Depends, HTTPException, Query, Request
from fastapi.responses import StreamingResponse
from cyclone.api_helpers import ndjson_line, tail_events
from cyclone.auth.deps import matrix_gate
from cyclone import db
from cyclone.store import store
from cyclone.pubsub import EventBus
from cyclone.store import store, to_ui_ta1_ack
router = APIRouter()
router = APIRouter(dependencies=[Depends(matrix_gate)])
def _ta1_to_ui(row: db.Ta1Ack) -> dict:
"""Render a Ta1Ack row for the UI (list endpoint shape)."""
return {
"id": row.id,
"control_number": row.control_number,
"ack_code": row.ack_code,
"note_code": row.note_code,
"interchange_date": row.interchange_date.isoformat()
if row.interchange_date else None,
"interchange_time": row.interchange_time,
"sender_id": row.sender_id,
"receiver_id": row.receiver_id,
"source_batch_id": row.source_batch_id,
"parsed_at": row.parsed_at.isoformat() if row.parsed_at else None,
}
# SP25: ``_ta1_to_ui`` moved to ``cyclone.store.ui.to_ui_ta1_ack`` so
# the live-tail event payload (``ta1_ack_received``) matches the list
# endpoint shape byte-for-byte.
def _serialize_ta1_from_row(row: db.Ta1Ack) -> str:
@@ -57,20 +53,78 @@ def list_ta1_acks_endpoint(
count regardless of the ``limit`` cap.
"""
rows = store.list_ta1_acks()
items = [_ta1_to_ui(r) for r in rows[:limit]]
items = [to_ui_ta1_ack(r) for r in rows[:limit]]
# SP28: batch-fetch linked_claim_ids per TA1 row (TA1 envelope
# links always carry claim_id IS NULL — populate the field for
# symmetry so the Acks page badge render path is uniform).
ack_ids = [r.id for r in rows]
linked_map = _find_linked_claim_ids_for_acks_helper(ack_ids, kind="ta1")
for item, aid in zip(items, ack_ids[:limit]):
item["linked_claim_ids"] = linked_map.get(aid, [])
return {
"total": len(rows),
"items": items,
}
def _find_linked_claim_ids_for_acks_helper(
ack_ids: list[int], *, kind: str
) -> dict[int, list[str]]:
"""Batch-fetch {ack_id: [claim_id, …]} for the listed ack rows.
Local helper so we don't import from ``acks.py`` and create a
circular import. See the equivalent ``_find_linked_claim_ids_for_acks``
in ``acks.py`` for the contract.
"""
out: dict[int, list[str]] = {aid: [] for aid in ack_ids}
if not ack_ids:
return out
with db.SessionLocal()() as s:
rows = (
s.query(db.ClaimAck.ack_id, db.ClaimAck.claim_id)
.filter(
db.ClaimAck.ack_kind == kind,
db.ClaimAck.ack_id.in_(ack_ids),
db.ClaimAck.claim_id.isnot(None),
)
.all()
)
for ack_id, claim_id in rows:
out[ack_id].append(claim_id)
return out
@router.get("/api/ta1-acks/stream")
async def ta1_acks_stream(
request: Request,
limit: int = Query(100, ge=1, le=1000),
) -> StreamingResponse:
"""Stream TA1 envelope acks as NDJSON.
Subscribes to ``ta1_ack_received`` and emits the same wire format
as the other live-tail endpoints. Registered BEFORE
``/api/ta1-acks/{ack_id}`` so ``stream`` isn't matched as an id.
"""
bus: EventBus = request.app.state.event_bus
async def gen() -> AsyncIterator[bytes]:
rows = store.list_ta1_acks()[:limit]
for row in rows:
yield ndjson_line({"type": "item", "data": to_ui_ta1_ack(row)})
yield ndjson_line({"type": "snapshot_end", "data": {"count": len(rows)}})
async for chunk in tail_events(request, bus, ["ta1_ack_received"]):
yield chunk
return StreamingResponse(gen(), media_type="application/x-ndjson")
@router.get("/api/ta1-acks/{ack_id}")
def get_ta1_ack_endpoint(ack_id: int) -> dict:
"""Return one persisted TA1 ACK row with its parsed detail."""
row = store.get_ta1_ack(ack_id)
if row is None:
raise HTTPException(status_code=404, detail=f"TA1 ACK {ack_id} not found")
body = _ta1_to_ui(row)
body = to_ui_ta1_ack(row)
body["raw_ta1_text"] = _serialize_ta1_from_row(row)
body["raw_json"] = row.raw_json
return body
+8
View File
@@ -103,6 +103,12 @@ class AuditEvent:
computed hash. Payload must be JSON-serializable; the audit_log
module handles the encoding so callers don't need to think about
canonical form.
``user_id`` is the authenticated actor for this event, when known
(e.g. a parse-999 call made by user 7). It's stored on the row but
is NOT part of the hash chain — the chain hashes only the fields
that existed pre-SP-auth so verify_chain stays compatible with
pre-auth rows.
"""
event_type: str
@@ -111,6 +117,7 @@ class AuditEvent:
payload: dict[str, Any] = field(default_factory=dict)
actor: str = "system"
created_at: datetime | None = None
user_id: int | None = None
def append_event(
@@ -155,6 +162,7 @@ def append_event(
created_at=created_at,
prev_hash=prev_hash,
hash=GENESIS_PREV_HASH, # placeholder; updated below
user_id=event.user_id,
)
session.add(row)
session.flush() # populate row.id
+1
View File
@@ -0,0 +1 @@
"""Auth module — users, sessions, permissions, routes, admin, rate_limit."""
+95
View File
@@ -0,0 +1,95 @@
"""Admin-only user management: GET/POST/PATCH/DELETE /api/admin/users."""
from __future__ import annotations
from fastapi import APIRouter, Depends, HTTPException, status
from cyclone.auth import users
from cyclone.auth.deps import get_current_user
from cyclone.auth.permissions import Role
from cyclone.db import SessionLocal, User
router = APIRouter(prefix="/api/admin/users", tags=["admin"])
def _require_admin(user: dict = Depends(get_current_user)) -> dict:
if user.get("role") != Role.ADMIN.value:
raise HTTPException(
status_code=status.HTTP_403_FORBIDDEN,
detail="forbidden",
)
return user
def _validate_role(role: str) -> None:
valid = {Role.ADMIN.value, Role.USER.value, Role.VIEWER.value}
if role not in valid:
raise HTTPException(
status_code=status.HTTP_422_UNPROCESSABLE_ENTITY,
detail=f"role must be one of {sorted(valid)}",
)
@router.get("")
def list_users(_admin=Depends(_require_admin)):
with SessionLocal()() as db:
all_users = db.query(User).all()
return [users.to_public(u) for u in all_users]
@router.post("", status_code=status.HTTP_201_CREATED)
def create_user(body: dict, _admin=Depends(_require_admin)):
username = (body.get("username") or "").strip()
password = body.get("password") or ""
role = body.get("role") or ""
if not username or len(username) < 3:
raise HTTPException(status_code=422, detail="username must be at least 3 chars")
if len(password) < 12:
raise HTTPException(status_code=422, detail="password must be at least 12 chars")
_validate_role(role)
with SessionLocal()() as db:
if users.get_by_username(db, username) is not None:
raise HTTPException(status_code=409, detail="username already exists")
u = users.create(db, username=username, password=password, role=role)
return users.to_public(u)
@router.patch("/{user_id}")
def patch_user(user_id: int, body: dict, admin=Depends(_require_admin)):
me = admin
if me.get("id") == user_id and body.get("role") and body["role"] != Role.ADMIN.value:
raise HTTPException(
status_code=status.HTTP_409_CONFLICT,
detail="cannot_demote_self",
)
with SessionLocal()() as db:
if body.get("role") is not None:
_validate_role(body["role"])
users.update_role(db, user_id, body["role"])
if body.get("password") is not None:
if len(body["password"]) < 12:
raise HTTPException(status_code=422, detail="password must be at least 12 chars")
users.update_password(db, user_id, body["password"])
if body.get("disabled") is True:
users.disable(db, user_id)
u = users.get(db, user_id)
if u is None:
raise HTTPException(status_code=404, detail="user not found")
return users.to_public(u)
@router.delete("/{user_id}", status_code=status.HTTP_204_NO_CONTENT)
def delete_user(user_id: int, admin=Depends(_require_admin)):
me = admin
if me.get("id") == user_id:
raise HTTPException(
status_code=status.HTTP_409_CONFLICT,
detail="cannot_delete_self",
)
with SessionLocal()() as db:
u = users.get(db, user_id)
if u is None:
raise HTTPException(status_code=404, detail="user not found")
users.disable(db, user_id)
return None
+105
View File
@@ -0,0 +1,105 @@
"""First-admin bootstrap: create the initial admin from env vars if no users exist.
Called from ``python -m cyclone`` before either ``cli.main()`` or
``uvicorn`` so users exist by the time the API serves requests.
Precedence:
1. ``CYCLONE_AUTH_DISABLED=1`` — dev escape hatch. Flip the
``cyclone.auth.deps.AUTH_DISABLED`` flag so the API returns a
synthetic admin user without checking credentials. Never raises.
2. Users table non-empty — no-op.
3. ``CYCLONE_ADMIN_USERNAME`` + ``CYCLONE_ADMIN_PASSWORD`` env vars set
(password >= 12 chars) — create the admin and print confirmation.
Each env var can also be replaced by a ``*_FILE`` companion
(``CYCLONE_ADMIN_USERNAME_FILE`` / ``CYCLONE_ADMIN_PASSWORD_FILE``)
that points at a file on disk — the standard Docker-secret pattern,
used in production to avoid embedding secrets in ``docker-compose.yml``.
``_FILE`` takes precedence when set.
4. Otherwise — raise ``RuntimeError`` with a remediation hint that
points operators at ``python -m cyclone users create``.
"""
from __future__ import annotations
import os
from pathlib import Path
from sqlalchemy import select
from cyclone.auth import users
from cyclone.auth.deps import AUTH_DISABLED
from cyclone.auth.permissions import Role
from cyclone.db import SessionLocal, User
def _read_secret(env_var: str, file_var: str) -> str | None:
"""Read a secret from a ``*_FILE`` env var (Docker-secret pattern) first,
falling back to the plain env var. Returns None if neither is set.
"""
file_path = os.environ.get(file_var)
if file_path:
try:
return Path(file_path).read_text().strip()
except OSError as exc:
raise RuntimeError(
f"failed to read {file_var}={file_path}: {exc}"
) from exc
return os.environ.get(env_var)
def run() -> None:
"""Bootstrap the first admin user, or no-op.
See module docstring for behavior. Idempotent: safe to call on
every startup — it short-circuits as soon as the users table is
non-empty.
"""
if os.environ.get("CYCLONE_AUTH_DISABLED") == "1":
# Dev escape hatch — skip bootstrap entirely and tell the API
# to also short-circuit auth checks.
import cyclone.auth.deps as _deps
_deps.AUTH_DISABLED = True
return
username = _read_secret(
"CYCLONE_ADMIN_USERNAME", "CYCLONE_ADMIN_USERNAME_FILE"
)
password = _read_secret(
"CYCLONE_ADMIN_PASSWORD", "CYCLONE_ADMIN_PASSWORD_FILE"
)
# First-boot fix: ``python -m cyclone`` calls bootstrap before any
# subcommand or the FastAPI lifespan handler runs, so on a brand-new
# DB ``SessionLocal()`` raises "init_db() has not been called".
# Initialize here so ``serve``, ``users create``, and friends can
# all reach the DB without the operator having to know about
# migrations. Idempotent — no-op when the schema is already current.
from cyclone import db as _db
_db.init_db()
with SessionLocal()() as db:
existing = db.execute(select(User)).scalars().first()
if existing is not None:
return # users exist — nothing to bootstrap
if not username or not password:
raise RuntimeError(
"Cyclone has no users yet. Set CYCLONE_ADMIN_USERNAME and "
"CYCLONE_ADMIN_PASSWORD env vars (min 12 chars), or run "
"`python -m cyclone users create <username> --role admin`."
)
if len(password) < 12:
raise RuntimeError(
"CYCLONE_ADMIN_PASSWORD must be at least 12 characters."
)
users.create(
db,
username=username,
password=password,
role=Role.ADMIN.value,
)
print(f"[cyclone] bootstrap admin user '{username}' created")
+183
View File
@@ -0,0 +1,183 @@
"""CLI subcommand: ``python -m cyclone users ...``.
Click-based to match the existing parse-837 / parse-835 convention in
``cyclone.cli``. Provides operator-side user management without going
through the admin HTTP API.
Subcommands
-----------
- ``users create USERNAME --role {admin,user,viewer} [--password PW]``
Create a user. If ``--password`` is omitted, prompts (with
confirmation) on the controlling terminal.
- ``users list`` — tab-separated ``id / username / role / state``.
- ``users disable USERNAME`` — set ``disabled_at`` to now.
- ``users reset-password USERNAME [--password PW]`` — replace the hash.
- ``users set-role USERNAME --role {admin,user,viewer}`` — change role.
Exit codes
----------
* 0 — success
* 1 — validation error (unknown username, duplicate username)
* 2 — usage error (missing arg, bad role, short password, unknown
subcommand). Click itself uses 2 for usage errors so the conventional
shell tools (``set -e``, etc.) recognize them.
Passwords shorter than 12 chars are rejected everywhere they appear.
"""
from __future__ import annotations
import getpass
import sys
import click
from cyclone.auth import users
from cyclone.auth.permissions import Role
from cyclone.db import SessionLocal
ROLE_CHOICES = [Role.ADMIN.value, Role.USER.value, Role.VIEWER.value]
MIN_PASSWORD_LEN = 12
def _prompt_password(label: str) -> str:
pw = getpass.getpass(f"{label}: ")
if not pw:
click.echo("Password required.", err=True)
sys.exit(2)
return pw
def _validate_password(pw: str) -> None:
if len(pw) < MIN_PASSWORD_LEN:
click.echo(
f"Password must be at least {MIN_PASSWORD_LEN} characters.",
err=True,
)
sys.exit(2)
# --------------------------------------------------------------------------- #
# Group
# --------------------------------------------------------------------------- #
@click.group(name="users")
def users_cli() -> None:
"""Manage Cyclone users from the command line."""
# --------------------------------------------------------------------------- #
# create
# --------------------------------------------------------------------------- #
@users_cli.command("create")
@click.argument("username")
@click.option(
"--role",
required=True,
type=click.Choice(ROLE_CHOICES, case_sensitive=False),
help="Role to grant the new user.",
)
@click.option(
"--password",
default=None,
help=f"Password (min {MIN_PASSWORD_LEN} chars). Prompts if omitted.",
)
def create_user(username: str, role: str, password: str | None) -> None:
"""Create a new user with the given USERNAME and ROLE."""
pw = password or _prompt_password("Password")
_validate_password(pw)
with SessionLocal()() as db:
if users.get_by_username(db, username) is not None:
click.echo(f"User '{username}' already exists.", err=True)
sys.exit(1)
u = users.create(db, username=username, password=pw, role=role)
click.echo(f"Created user '{u.username}' with role '{u.role}'.")
# --------------------------------------------------------------------------- #
# list
# --------------------------------------------------------------------------- #
@users_cli.command("list")
def list_users() -> None:
"""List all users as id / username / role / state."""
from cyclone.db import User
with SessionLocal()() as db:
for u in db.query(User).order_by(User.id.asc()).all():
state = "disabled" if u.disabled_at else "active"
click.echo(f"{u.id}\t{u.username}\t{u.role}\t{state}")
# --------------------------------------------------------------------------- #
# disable
# --------------------------------------------------------------------------- #
@users_cli.command("disable")
@click.argument("username")
def disable_user(username: str) -> None:
"""Disable USERNAME (sets disabled_at to now)."""
with SessionLocal()() as db:
u = users.get_by_username(db, username)
if u is None:
click.echo(f"No such user: {username}", err=True)
sys.exit(1)
users.disable(db, u.id)
click.echo(f"Disabled '{username}'.")
# --------------------------------------------------------------------------- #
# reset-password
# --------------------------------------------------------------------------- #
@users_cli.command("reset-password")
@click.argument("username")
@click.option(
"--password",
default=None,
help=f"New password (min {MIN_PASSWORD_LEN} chars). Prompts if omitted.",
)
def reset_password(username: str, password: str | None) -> None:
"""Replace USERNAME's password."""
pw = password or _prompt_password("New password")
_validate_password(pw)
with SessionLocal()() as db:
u = users.get_by_username(db, username)
if u is None:
click.echo(f"No such user: {username}", err=True)
sys.exit(1)
users.update_password(db, u.id, pw)
click.echo(f"Password reset for '{username}'.")
# --------------------------------------------------------------------------- #
# set-role
# --------------------------------------------------------------------------- #
@users_cli.command("set-role")
@click.argument("username")
@click.option(
"--role",
required=True,
type=click.Choice(ROLE_CHOICES, case_sensitive=False),
help="Role to grant.",
)
def set_role(username: str, role: str) -> None:
"""Change USERNAME's role."""
with SessionLocal()() as db:
u = users.get_by_username(db, username)
if u is None:
click.echo(f"No such user: {username}", err=True)
sys.exit(1)
users.update_role(db, u.id, role)
click.echo(f"Role for '{username}' set to '{role}'.")
+140
View File
@@ -0,0 +1,140 @@
"""FastAPI dependencies for auth."""
from __future__ import annotations
from typing import Annotated
from fastapi import Depends, HTTPException, Request, status
from sqlalchemy.orm import Session as DbSession
from cyclone.auth import sessions, users
from cyclone.auth.permissions import Role, allowed_roles
from cyclone.db import SessionLocal
def _db():
db = SessionLocal()()
try:
yield db
finally:
db.close()
DbSessionDep = Annotated[DbSession, Depends(_db)]
AUTH_DISABLED = False
async def get_current_user(
request: Request,
db: DbSessionDep,
) -> dict:
"""Return the public User shape. Raises 401 if session is missing/expired.
When AUTH_DISABLED is True (dev escape hatch), returns a synthetic admin
user without checking credentials.
"""
if AUTH_DISABLED:
return {
"id": 0,
"username": "dev",
"role": Role.ADMIN.value,
"createdAt": None,
"disabledAt": None,
}
sid = request.cookies.get("cyclone_session")
if not sid:
raise HTTPException(
status_code=status.HTTP_401_UNAUTHORIZED,
detail="session_expired",
)
sess = sessions.get_valid(db, sid)
if sess is None:
raise HTTPException(
status_code=status.HTTP_401_UNAUTHORIZED,
detail="session_expired",
)
user = users.get(db, sess.user_id)
if user is None or user.disabled_at is not None:
raise HTTPException(
status_code=status.HTTP_401_UNAUTHORIZED,
detail="account_disabled",
)
# Sliding expiry: refresh both DB and cookie.
sessions.touch(db, sid)
request.state.user = user
request.state.session_id = sid
return users.to_public(user)
def require_role(*allowed: Role):
"""Dependency factory: gate the endpoint to specific roles.
Falls back to PERMISSIONS matrix lookup if no explicit roles given.
"""
async def _dep(
request: Request,
user: dict = Depends(get_current_user),
) -> dict:
user_role = user.get("role")
if allowed:
if user_role not in {r.value for r in allowed}:
raise HTTPException(
status_code=status.HTTP_403_FORBIDDEN,
detail="forbidden",
)
return user
# Otherwise consult the matrix.
method = request.method
path = request.url.path
roles = allowed_roles(method, path)
if roles is None:
raise HTTPException(
status_code=status.HTTP_403_FORBIDDEN,
detail="forbidden",
)
if user_role not in {r.value for r in roles}:
raise HTTPException(
status_code=status.HTTP_403_FORBIDDEN,
detail="forbidden",
)
return user
return _dep
async def matrix_gate(
request: Request,
user: dict = Depends(get_current_user),
) -> dict:
"""App-wide gate: requires auth, then enforces the PERMISSIONS matrix.
Behavior:
* AUTH_DISABLED short-circuits (synthetic admin, no role check).
* No session cookie → 401 from get_current_user.
* Endpoint not in the matrix → 403 (fail-closed).
* User role not allowed for (method, path) → 403.
* Empty allowed-roles set (e.g. /api/healthz) → public, no role check.
Used as ``dependencies=[Depends(matrix_gate)]`` on every authenticated
route. Centralizing the gate here means the matrix is the source of
truth — no need to wire per-route ``require_role(...)`` calls.
"""
if AUTH_DISABLED:
return user
method = request.method
path = request.url.path
roles = allowed_roles(method, path)
if roles is None:
raise HTTPException(
status_code=status.HTTP_403_FORBIDDEN,
detail="forbidden",
)
user_role = user.get("role")
if user_role not in {r.value for r in roles}:
raise HTTPException(
status_code=status.HTTP_403_FORBIDDEN,
detail="forbidden",
)
return user
+109
View File
@@ -0,0 +1,109 @@
"""Role enum + PERMISSIONS matrix."""
from __future__ import annotations
from enum import Enum
class Role(str, Enum):
ADMIN = "admin"
USER = "user"
VIEWER = "viewer"
ALL_ROLES = {Role.ADMIN, Role.USER, Role.VIEWER}
WRITE_ROLES = {Role.ADMIN, Role.USER}
ADMIN_ONLY = {Role.ADMIN}
# (method, path-prefix) → allowed roles.
# Endpoints not in this matrix default to DENY (fail-closed).
PERMISSIONS: dict[tuple[str, str], set[Role]] = {
# Public paths.
("GET", "/api/health"): set(),
("POST", "/api/auth/login"): set(),
# Auth surface.
("POST", "/api/auth/logout"): ALL_ROLES,
("GET", "/api/auth/me"): ALL_ROLES,
# Admin-only user management.
("GET", "/api/admin/users"): ADMIN_ONLY,
("POST", "/api/admin/users"): ADMIN_ONLY,
("PATCH", "/api/admin/users"): ADMIN_ONLY,
("DELETE", "/api/admin/users"): ADMIN_ONLY,
# Read endpoints (all authenticated roles).
("GET", "/api/claims"): ALL_ROLES,
("GET", "/api/remittances"): ALL_ROLES,
("GET", "/api/providers"): ALL_ROLES,
("GET", "/api/batches"): ALL_ROLES,
("GET", "/api/dashboard/summary"): ALL_ROLES,
("GET", "/api/activity"): ALL_ROLES,
("GET", "/api/inbox/lanes"): ALL_ROLES,
("GET", "/api/inbox/export.csv"): ALL_ROLES,
("GET", "/api/reconcile"): ALL_ROLES,
("GET", "/api/reconciliation"): ALL_ROLES,
("GET", "/api/acks"): ALL_ROLES,
("GET", "/api/ta1-acks"): ALL_ROLES,
("GET", "/api/277ca-acks"): ALL_ROLES,
("GET", "/api/batch-diff"): ALL_ROLES,
("GET", "/api/config"): ALL_ROLES,
("GET", "/api/payers"): ALL_ROLES,
("GET", "/api/audit-log"): ADMIN_ONLY,
# Clearhouse (SFTP creds + dzinesco identity) — admin only.
("GET", "/api/clearhouse"): ADMIN_ONLY,
("PATCH", "/api/clearhouse"): ADMIN_ONLY,
("POST", "/api/clearhouse/submit"): ADMIN_ONLY,
# Admin ops (audit log, backup, scheduler, db rotate, reload-config).
("GET", "/api/admin/audit-log"): ADMIN_ONLY,
("GET", "/api/admin/audit-log/verify"): ADMIN_ONLY,
("GET", "/api/admin/backup"): ADMIN_ONLY,
("POST", "/api/admin/backup"): ADMIN_ONLY,
("GET", "/api/admin/backup/scheduler"): ADMIN_ONLY,
("POST", "/api/admin/backup/scheduler"): ADMIN_ONLY,
("GET", "/api/admin/scheduler"): ADMIN_ONLY,
("POST", "/api/admin/scheduler"): ADMIN_ONLY,
("POST", "/api/admin/db/rotate-key"): ADMIN_ONLY,
("POST", "/api/admin/reload-config"): ADMIN_ONLY,
("GET", "/api/admin/validate-provider"): ADMIN_ONLY,
# Write endpoints (admin + user, no viewer).
("POST", "/api/parse-837"): WRITE_ROLES,
("POST", "/api/parse-835"): WRITE_ROLES,
("POST", "/api/parse-999"): WRITE_ROLES,
("POST", "/api/parse-ta1"): WRITE_ROLES,
("POST", "/api/parse-277ca"): WRITE_ROLES,
("POST", "/api/inbox"): WRITE_ROLES,
("POST", "/api/inbox/candidates"): WRITE_ROLES,
("POST", "/api/inbox/rejected"): WRITE_ROLES,
("POST", "/api/inbox/payer-rejected"): WRITE_ROLES,
("POST", "/api/reconcile"): WRITE_ROLES,
("POST", "/api/reconciliation"): WRITE_ROLES,
("POST", "/api/resubmit"): WRITE_ROLES,
("POST", "/api/acks"): WRITE_ROLES,
("POST", "/api/batches"): WRITE_ROLES, # /export-837 regenerates X12 from DB rows
("POST", "/api/eligibility"): WRITE_ROLES,
("POST", "/api/submit-batch"): WRITE_ROLES, # SP37: canonical outbound path (mirrors CLI)
# CSV export — read-only.
("GET", "/api/export.csv"): ALL_ROLES,
}
def allowed_roles(method: str, path: str) -> set[Role] | None:
"""Return the set of roles allowed to call (method, path), or None if denied.
Uses longest-prefix match on path; falls back to DENY (None) if no entry matches.
"""
candidates = [
(len(prefix), roles)
for (m, prefix), roles in PERMISSIONS.items()
if m == method and (path == prefix or path.startswith(prefix.rstrip("/") + "/"))
]
if not candidates:
return None
candidates.sort(key=lambda x: -x[0])
return candidates[0][1]
+34
View File
@@ -0,0 +1,34 @@
"""Per-username login rate limiter (in-memory, per-process)."""
from __future__ import annotations
import time
from threading import Lock
WINDOW_SECONDS = 300
MAX_FAILS = 5
_FAILS: dict[str, list[float]] = {}
_LOCK = Lock()
def check(username: str) -> int:
"""Return retry-after seconds, or 0 if allowed."""
now = time.monotonic()
with _LOCK:
fails = [t for t in _FAILS.get(username, []) if now - t < WINDOW_SECONDS]
_FAILS[username] = fails
if len(fails) >= MAX_FAILS:
return int(WINDOW_SECONDS - (now - fails[0]))
return 0
def record_failure(username: str) -> None:
now = time.monotonic()
with _LOCK:
_FAILS.setdefault(username, []).append(now)
def reset(username: str) -> None:
with _LOCK:
_FAILS.pop(username, None)
+97
View File
@@ -0,0 +1,97 @@
"""/api/auth/login, /api/auth/logout, /api/auth/me."""
from __future__ import annotations
import os
from fastapi import APIRouter, Depends, HTTPException, Request, Response, status
from cyclone.auth import rate_limit, sessions, users
from cyclone.auth.deps import get_current_user
from cyclone.db import SessionLocal
router = APIRouter(prefix="/api/auth", tags=["auth"])
COOKIE_NAME = "cyclone_session"
COOKIE_MAX_AGE = 86400 # 24h
def _is_https(request: Request) -> bool:
if request.url.scheme == "https":
return True
return os.environ.get("CYCLONE_BEHIND_HTTPS") == "1"
def _set_cookie(response: Response, sid: str, request: Request) -> None:
response.set_cookie(
key=COOKIE_NAME,
value=sid,
max_age=COOKIE_MAX_AGE,
path="/api",
httponly=True,
samesite="lax",
secure=_is_https(request),
)
def _clear_cookie(response: Response) -> None:
response.delete_cookie(key=COOKIE_NAME, path="/api")
@router.post("/login")
def login(body: dict, request: Request, response: Response):
username = (body.get("username") or "").strip()
password = body.get("password") or ""
if not username or not password:
raise HTTPException(
status_code=status.HTTP_400_BAD_REQUEST,
detail="username and password are required",
)
retry_after = rate_limit.check(username)
if retry_after > 0:
raise HTTPException(
status_code=status.HTTP_429_TOO_MANY_REQUESTS,
detail="rate_limited",
headers={"Retry-After": str(retry_after)},
)
with SessionLocal()() as db:
user = users.get_by_username(db, username)
if user is None or not users.verify_password(password, user.password_hash):
rate_limit.record_failure(username)
raise HTTPException(
status_code=status.HTTP_401_UNAUTHORIZED,
detail="invalid_credentials",
)
if user.disabled_at is not None:
raise HTTPException(
status_code=status.HTTP_403_FORBIDDEN,
detail="account_disabled",
)
sid, _ = sessions.create(db, user_id=user.id)
rate_limit.reset(username)
public = users.to_public(user)
_set_cookie(response, sid, request)
return public
@router.post("/logout")
def logout(
request: Request,
response: Response,
_user: dict = Depends(get_current_user),
):
sid = request.cookies.get(COOKIE_NAME)
if sid:
with SessionLocal()() as db:
sessions.delete(db, sid)
_clear_cookie(response)
return Response(status_code=status.HTTP_204_NO_CONTENT)
@router.get("/me")
def me(user: dict = Depends(get_current_user)):
return user
+60
View File
@@ -0,0 +1,60 @@
"""Session create/validate/expire/touch."""
from __future__ import annotations
import secrets
from datetime import datetime, timedelta, timezone
from sqlalchemy import select
from cyclone.db import Session
SESSION_LIFETIME = timedelta(hours=24)
def create(db, *, user_id: int) -> tuple[str, Session]:
sid = secrets.token_urlsafe(32)
now = datetime.now(timezone.utc)
expires_at = now + SESSION_LIFETIME
sess = Session(
id=sid,
user_id=user_id,
expires_at=expires_at,
created_at=now,
)
db.add(sess)
db.commit()
db.refresh(sess)
# SQLite strips tzinfo on roundtrip; restore it so callers don't have to.
sess.expires_at = expires_at
return sid, sess
def get_valid(db, sid: str) -> Session | None:
sess = db.execute(
select(Session).where(Session.id == sid)
).scalar_one_or_none()
if sess is None:
return None
# SQLite drops tzinfo on roundtrip; normalize to UTC before comparing.
if sess.expires_at.tzinfo is None:
sess.expires_at = sess.expires_at.replace(tzinfo=timezone.utc)
if sess.expires_at <= datetime.now(timezone.utc):
return None
return sess
def delete(db, sid: str) -> None:
sess = db.get(Session, sid)
if sess is None:
return
db.delete(sess)
db.commit()
def touch(db, sid: str) -> None:
sess = db.get(Session, sid)
if sess is None:
return
sess.expires_at = datetime.now(timezone.utc) + SESSION_LIFETIME
db.commit()
+79
View File
@@ -0,0 +1,79 @@
"""User CRUD + bcrypt password hashing."""
from __future__ import annotations
from datetime import datetime, timezone
from typing import Any
from passlib.hash import bcrypt
from sqlalchemy import select
from cyclone.db import User
def hash_password(plaintext: str) -> str:
return bcrypt.hash(plaintext)
def verify_password(plaintext: str, hashed: str) -> bool:
try:
return bcrypt.verify(plaintext, hashed)
except (ValueError, TypeError):
return False
def create(db, *, username: str, password: str, role: str) -> User:
user = User(
username=username,
password_hash=hash_password(password),
role=role,
created_at=datetime.now(timezone.utc),
)
db.add(user)
db.commit()
db.refresh(user)
return user
def get_by_username(db, username: str) -> User | None:
return db.execute(
select(User).where(User.username == username)
).scalar_one_or_none()
def get(db, user_id: int) -> User | None:
return db.get(User, user_id)
def disable(db, user_id: int) -> None:
user = db.get(User, user_id)
if user is None:
return
user.disabled_at = datetime.now(timezone.utc)
db.commit()
def update_role(db, user_id: int, role: str) -> None:
user = db.get(User, user_id)
if user is None:
return
user.role = role
db.commit()
def update_password(db, user_id: int, new_password: str) -> None:
user = db.get(User, user_id)
if user is None:
return
user.password_hash = hash_password(new_password)
db.commit()
def to_public(user: User) -> dict[str, Any]:
return {
"id": user.id,
"username": user.username,
"role": user.role,
"createdAt": user.created_at.isoformat() if user.created_at else None,
"disabledAt": user.disabled_at.isoformat() if user.disabled_at else None,
}
+492
View File
@@ -0,0 +1,492 @@
"""SP28: per-ACK auto-linker.
Three helpers, one per ACK kind, all run inside the same DB
transaction that persists the Ack row. Each returns a slice of
:class:`ClaimAckLinkRow` dataclasses describing what to link —
the CALLER persists those rows via
:func:`cyclone.store.claim_acks.add_claim_ack` (which owns the
publish-from-store contract).
The two-pass join lives in
:func:`lookup_claims_for_ack_set_response` (D10): ST02 via the
batch envelope index (primary) + ``Claim.patient_control_number``
(fallback). Plus :func:`link_manual` for the manual-fallback
endpoint.
The helpers do NOT write to the DB session — they are pure
readers over the session + parse result + the supplied
``batch_envelope_index`` / ``pc_claim_lookup`` / ``batch_lookup``
closures. This matches the existing
``cyclone.inbox_state.apply_999_rejections`` pattern and lets the
store facade own the publish-from-store contract for live-tail.
See ``docs/superpowers/specs/2026-07-02-cyclone-ack-claim-auto-link-design.md``
§3 for the per-AK2 granularity, the two-pass join, and the
idempotency contract enforced by ``ux_claim_acks_dedup``.
"""
from __future__ import annotations
import logging
from dataclasses import dataclass, field
from datetime import datetime, timezone
from typing import Callable, Optional
from sqlalchemy.orm import Session
from cyclone.db import Batch, Claim
log = logging.getLogger(__name__)
@dataclass
class ClaimAckLinkRow:
"""One row to insert into ``claim_acks``.
Populated by the helpers; persisted by the caller via
:func:`cyclone.store.claim_acks.add_claim_ack`. Carries every
column the store needs to build the ORM row + the
``claim_ack_written`` event payload.
"""
claim_id: Optional[str] = None
batch_id: Optional[str] = None
ak2_index: Optional[int] = None
set_control_number: Optional[str] = None
set_accept_reject_code: Optional[str] = None
@dataclass
class ClaimAckLinkResult:
"""Outcome of a single helper call.
``linked`` is the list of :class:`ClaimAckLinkRow` rows the
caller should persist via ``cycl_store.add_claim_ack``.
``orphans`` is a list of free-form strings the join couldn't
resolve — for 999/277CA these are ``set_control_number``
values; for TA1 they're the TA1 ICN when no matching batch was
found.
"""
linked: list[ClaimAckLinkRow] = field(default_factory=list)
orphans: list[str] = field(default_factory=list)
# ---------------------------------------------------------------------------
# D10 two-pass join
# ---------------------------------------------------------------------------
def lookup_claims_for_ack_set_response(
session: Session,
set_control_number: str,
*,
batch_envelope_index: Optional[Callable[[str], Optional[str]]] = None,
pc_claim_lookup: Optional[Callable[[str], Optional[Claim]]] = None,
) -> list[Claim]:
"""Two-pass join for a single AK2 set_response / 277CA claim_status.
D10 (spec): for a 999 AK2-2 ``set_control_number`` or a 277CA REF*1K
``payer_claim_control_number``, return every claim this ack
acknowledges. The primary join is
``Batch.envelope.control_number == set_control_number`` (== source
837's ST02 on Gainwell batches); the fallback is
``Claim.patient_control_number == set_control_number``.
Returns 0..N matching claims (one-ack-to-many when one 837 batch
shipped multiple claims under one ST02).
Args:
session: SQLAlchemy session the caller owns.
set_control_number: the AK2-2 / REF*1K value to resolve.
batch_envelope_index: optional pre-built index that maps
``Batch.envelope.control_number`` → ``batch.id`` (built
once per ingest via ``store.batch_envelope_index``).
Pass to skip the per-set-response ``Batch`` scan in Pass 1.
pc_claim_lookup: optional pre-built callable that maps a PCN
to a single claim. Falls back to a session-wide query
when not supplied.
Pass 1 wins. The two paths cannot both fire for the same
``set_control_number`` — if Pass 1 returns one or more claims,
Pass 2 is skipped. This is the false-positive guard from
spec §7.
"""
if not set_control_number:
return []
# -- Pass 1: Batch.envelope.control_number primary --------------
# Accept either a plain dict (the common case — built once per
# ingest via ``store.batch_envelope_index``) or a callable for
# test-side closures. Normalize to ``idx.get`` so the rest of
# the function stays uniform.
idx: Optional[Callable[[str], Optional[str]]] = None
if batch_envelope_index is not None:
if callable(batch_envelope_index):
idx = batch_envelope_index
else:
idx = batch_envelope_index.get
matched_ids: list[str] = []
if idx is not None:
batch_id = idx(set_control_number)
if batch_id is not None:
matched_ids = [
cid for (cid,) in (
session.query(Claim.id)
.filter(Claim.batch_id == batch_id)
.all()
)
]
else:
# Fallback: scan all batches once per call. Slow but correct;
# callers SHOULD pass the index.
rows = (
session.query(Batch.id, Batch.raw_result_json)
.filter(Batch.kind == "837p")
.all()
)
for bid, raw in rows:
env = (raw or {}).get("envelope") or {}
if env.get("control_number") == set_control_number:
matched_ids = [
cid for (cid,) in (
session.query(Claim.id)
.filter(Claim.batch_id == bid)
.all()
)
]
break
if matched_ids:
claims = (
session.query(Claim)
.filter(Claim.id.in_(matched_ids))
.all()
)
if claims:
return list(claims)
# -- Pass 2: Claim.patient_control_number fallback ---------------
if pc_claim_lookup is not None:
single = pc_claim_lookup(set_control_number)
if single is not None:
return [single]
return []
matches = (
session.query(Claim)
.filter(Claim.patient_control_number == set_control_number)
.all()
)
return list(matches)
# ---------------------------------------------------------------------------
# Per-ACK helpers — walk the parsed result and produce ClaimAckLinkRow
# dataclasses. The CALLER persists via cycl_store.add_claim_ack so the
# publish-from-store contract owns the live-tail event emission.
# ---------------------------------------------------------------------------
def _existing_link_claim_ids(
session: Session,
*,
claim_ids: list[str],
ack_kind: str,
ack_id: int,
) -> set[str]:
"""Return the subset of ``claim_ids`` that already have a link row.
Mirrors the partial unique index
``ux_claim_acks_dedup(claim_id, ack_kind, ack_id, ak2_index)
WHERE claim_id IS NOT NULL AND ak2_index IS NOT NULL``. The
pre-check is here so we skip ``session.add`` and avoid
IntegrityError log noise on re-ingest. For TA1 (claim_id IS
NULL) the helpers do their own check.
"""
if not claim_ids:
return set()
rows = (
session.query(Claim.claim_id) # placeholder; replaced below
if False else
session.query(Claim.id)
.filter(Claim.id.in_([]))
.all()
)
# Replace the placeholder with the real query — needed because
# ClaimAck is not imported above (avoids circular import).
from cyclone.db import ClaimAck
existing = (
session.query(ClaimAck.claim_id)
.filter(
ClaimAck.ack_kind == ack_kind,
ClaimAck.ack_id == ack_id,
ClaimAck.claim_id.in_(claim_ids),
)
.all()
)
return {cid for (cid,) in existing if cid is not None}
def apply_999_acceptances(
session: Session,
parsed_999,
*,
ack_id: int,
batch_envelope_index: Optional[Callable[[str], Optional[str]]] = None,
pc_claim_lookup: Optional[Callable[[str], Optional[Claim]]] = None,
now: Optional[datetime] = None,
) -> ClaimAckLinkResult:
"""For every AK2 set-response, build one ``ClaimAckLinkRow`` per matched claim.
Both accepted AND rejected AK2s produce a link row (so the
ClaimDrawer panel can show the rejection inline via the
``set_accept_reject_code`` color-coded chip). Orphans are returned
but not linked.
Idempotent: rows the dedup index already covers (re-ingest of an
identical file) are skipped silently — the pre-check is here to
avoid ``IntegrityError`` log noise.
One AK2 can produce multiple ``claim_acks`` rows when the source
837 batch carried more than one claim under a shared ST02 (rare
on this codebase but supported by D10 / the schema).
Returns:
A :class:`ClaimAckLinkResult` whose ``linked`` list contains
one :class:`ClaimAckLinkRow` per AK2-to-claim match. The
caller persists each row via ``cycl_store.add_claim_ack``.
"""
result = ClaimAckLinkResult()
set_responses = getattr(parsed_999, "set_responses", None) or []
# Resolve all set_control_numbers up front so we can do one
# batched dedup query per (ack_kind, ack_id).
resolved: list[tuple[int, "object", list[Claim], str, str]] = []
candidate_claim_ids: list[str] = []
for idx, sr in enumerate(set_responses):
scn = getattr(sr, "set_control_number", None) or ""
code = getattr(sr.set_accept_reject, "code", None) or ""
claims = lookup_claims_for_ack_set_response(
session, scn,
batch_envelope_index=batch_envelope_index,
pc_claim_lookup=pc_claim_lookup,
)
if not claims:
result.orphans.append(scn)
continue
resolved.append((idx, sr, claims, scn, code))
candidate_claim_ids.extend(c.id for c in claims)
existing_ids = _existing_link_claim_ids(
session,
claim_ids=candidate_claim_ids,
ack_kind="999",
ack_id=ack_id,
)
for idx, sr, claims, scn, code in resolved:
for claim in claims:
if claim.id in existing_ids:
continue
result.linked.append(ClaimAckLinkRow(
claim_id=claim.id,
batch_id=None,
ak2_index=idx,
set_control_number=scn,
set_accept_reject_code=code,
))
return result
def apply_277ca_acks(
session: Session,
parsed_277ca,
*,
ack_id: int,
batch_envelope_index: Optional[Callable[[str], Optional[str]]] = None,
pc_claim_lookup: Optional[Callable[[str], Optional[Claim]]] = None,
now: Optional[datetime] = None,
) -> ClaimAckLinkResult:
"""For every ClaimStatus with a payer_claim_control_number, build a ClaimAckLinkRow.
Accepted AND rejected ClaimStatuses both link — the
``set_accept_reject_code`` carries the STC category code. The
``claim_acks`` row is independent of the existing
``Claim.payer_rejected_at`` mutation from
:func:`cyclone.inbox_state_277ca.apply_277ca_rejections` (which
fires before this helper in the handler).
Returns:
A :class:`ClaimAckLinkResult` whose ``linked`` list contains
one :class:`ClaimAckLinkRow` per ClaimStatus match. The
caller persists each row via ``cycl_store.add_claim_ack``.
"""
result = ClaimAckLinkResult()
statuses = getattr(parsed_277ca, "claim_statuses", None) or []
resolved: list[tuple["object", list[Claim], str, str]] = []
candidate_claim_ids: list[str] = []
for status in statuses:
scn = getattr(status, "payer_claim_control_number", None) or ""
raw_code = getattr(status, "status_code", None) or ""
# ``status_code`` may be a category like "A6:19:PR" — keep
# the whole STC composite so the UI can render the
# category without re-parsing raw_json. Truncate to 8 chars
# (column width).
code = (raw_code or "")[:8]
if not scn:
# No REF*1K — orphan. Surface the STC composite so the
# operator can correlate via the ack's raw_json.
orphan_key = code or "(no REF*1K)"
result.orphans.append(orphan_key)
continue
claims = lookup_claims_for_ack_set_response(
session, scn,
batch_envelope_index=batch_envelope_index,
pc_claim_lookup=pc_claim_lookup,
)
if not claims:
result.orphans.append(scn)
continue
resolved.append((status, claims, scn, code))
candidate_claim_ids.extend(c.id for c in claims)
existing_ids = _existing_link_claim_ids(
session,
claim_ids=candidate_claim_ids,
ack_kind="277ca",
ack_id=ack_id,
)
for status, claims, scn, code in resolved:
for claim in claims:
if claim.id in existing_ids:
continue
result.linked.append(ClaimAckLinkRow(
claim_id=claim.id,
batch_id=None,
ak2_index=None,
set_control_number=scn,
set_accept_reject_code=code or None,
))
return result
def apply_ta1_envelope_link(
session: Session,
parsed_ta1,
*,
ack_id: int,
batch_lookup: Callable[[str, str], Optional[Batch]],
now: Optional[datetime] = None,
) -> ClaimAckLinkResult:
"""Build a TA1 envelope-level link row to the most-recent matching Batch.
TA1 is envelope-level only (ISA/IEA, no per-claim granularity).
The link row has ``claim_id IS NULL`` and ``batch_id`` populated.
Args:
parsed_ta1: a :class:`cyclone.parsers.models_ta1.ParseResultTa1`.
batch_lookup: ``(sender_id, receiver_id) -> Batch | None``.
The handler supplies a closure that walks
``session.query(Batch).order_by(parsed_at.desc())``. Returning
``None`` produces an orphan (no batch match).
Returns:
A :class:`ClaimAckLinkResult` with 0..1 row in ``linked``.
Idempotent via dedup on ``(ack_kind='ta1', ack_id)`` (claim_id
IS NULL so the partial unique index doesn't catch it; the
pre-check here is a Python-side query).
"""
result = ClaimAckLinkResult()
envelope = getattr(parsed_ta1, "envelope", None)
if envelope is None:
return result
ta1_obj = getattr(parsed_ta1, "ta1", None)
ack_code = getattr(ta1_obj, "ack_code", None) or ""
# Dedup: same (ack_kind, ack_id) → at most one TA1 envelope link.
# Done in Python because the partial unique index requires
# claim_id IS NOT NULL.
from cyclone.db import ClaimAck
existing = (
session.query(ClaimAck.id)
.filter(
ClaimAck.ack_kind == "ta1",
ClaimAck.ack_id == ack_id,
ClaimAck.claim_id.is_(None),
)
.first()
)
if existing is not None:
return result
batch = batch_lookup(envelope.sender_id or "", envelope.receiver_id or "")
if batch is None:
orphan_key = (
getattr(ta1_obj, "control_number", None)
or envelope.control_number
or ""
)
result.orphans.append(orphan_key)
return result
result.linked.append(ClaimAckLinkRow(
claim_id=None,
batch_id=batch.id,
ak2_index=None,
set_control_number=None,
set_accept_reject_code=ack_code,
))
return result
def link_manual(
session: Session,
*,
claim_id: str,
ack_kind: str,
ack_id: int,
set_control_number: Optional[str] = None,
set_accept_reject_code: Optional[str] = None,
ak2_index: Optional[int] = None,
now: Optional[datetime] = None,
) -> ClaimAckLinkRow:
"""Return one manual link row (the caller persists it via the store).
Used by ``POST /api/acks/{kind}/{ack_id}/match-claim``. Returns a
:class:`ClaimAckLinkRow` describing the row to insert. The caller
is responsible for persistence so it can own the publish-from-store
contract.
Idempotency: callers should pre-check via ``session.query(ClaimAck)
.filter(...).first()`` and skip when a row already exists; the
:class:`cyclone.store.claim_acks.add_claim_ack` implementation also
re-checks via the partial unique index.
Raises ``LookupError`` when the referenced claim doesn't exist
(the caller maps that to 404).
"""
if ack_kind not in ("999", "277ca", "ta1"):
raise ValueError(f"link_manual: unknown ack_kind={ack_kind!r}")
claim = session.get(Claim, claim_id)
if claim is None:
raise LookupError(f"claim {claim_id} not found")
return ClaimAckLinkRow(
claim_id=claim_id,
batch_id=None,
ak2_index=ak2_index,
set_control_number=set_control_number,
set_accept_reject_code=set_accept_reject_code,
)
__all__ = [
"ClaimAckLinkResult",
"ClaimAckLinkRow",
"apply_999_acceptances",
"apply_277ca_acks",
"apply_ta1_envelope_link",
"link_manual",
"lookup_claims_for_ack_set_response",
]
+200
View File
@@ -24,6 +24,7 @@ stub secret and the paramiko auth will fail loudly at connect time.
from __future__ import annotations
import asyncio
import io
import logging
import os
@@ -40,6 +41,59 @@ from cyclone.providers import SftpBlock
log = logging.getLogger(__name__)
# ---------------------------------------------------------------------------
# Per-op SFTP timeout (SP27 Task 8)
#
# paramiko is synchronous; without a timeout, a hung ``listdir_attr``
# freezes the worker thread indefinitely. The 06/25 silent hang was
# exactly this — the MFT server TCP-acked but stopped responding, the
# scheduler's ``asyncio.to_thread`` waited forever, and the operator
# had no signal that polling had stalled.
#
# The async wrappers below apply ``asyncio.wait_for`` to every SFTP
# call site so the event loop can give up after the configured bound.
# The bound is read fresh on every call (env-var-only, no module-level
# cache) so an operator who tunes the value at runtime picks it up on
# the next poll.
# ---------------------------------------------------------------------------
_DEFAULT_SFTP_OP_TIMEOUT_SECONDS = 30.0
def _op_timeout_seconds() -> float:
"""Per-op SFTP timeout from ``CYCLONE_SFTP_OP_TIMEOUT_SECONDS``.
Default 30s. Picked to comfortably outlast Gainwell's p99
listdir_attr (~2s) while still surfacing real hangs inside one
scheduler tick. Operators who hit repeated timeouts should drop
this — but the right answer is to fix the MFT server, not to
paper over it here.
"""
raw = os.environ.get("CYCLONE_SFTP_OP_TIMEOUT_SECONDS")
if not raw:
return _DEFAULT_SFTP_OP_TIMEOUT_SECONDS
try:
value = float(raw)
except ValueError:
log.warning(
"CYCLONE_SFTP_OP_TIMEOUT_SECONDS=%r is not a float; using default %.1fs",
raw, _DEFAULT_SFTP_OP_TIMEOUT_SECONDS,
)
return _DEFAULT_SFTP_OP_TIMEOUT_SECONDS
if value <= 0:
# ``asyncio.wait_for(timeout=0)`` raises immediately, and
# ``wait_for(timeout<0)`` is undefined per the asyncio docs.
# A zero/negative setting would silently turn every SFTP call
# into an instant timeout (a wave of bogus "list_inbound:
# timeout" errors). Treat the value as bad and fall back.
log.warning(
"CYCLONE_SFTP_OP_TIMEOUT_SECONDS=%r must be positive; using default %.1fs",
raw, _DEFAULT_SFTP_OP_TIMEOUT_SECONDS,
)
return _DEFAULT_SFTP_OP_TIMEOUT_SECONDS
return value
@dataclass
class InboundFile:
"""A single file observed in the inbound MFT path."""
@@ -87,11 +141,70 @@ class SftpClient:
dir and returns :class:`InboundFile` records pointing at the
cache copy. The remote file is *not* deleted — the operator
archives inbound files in the MFT UI.
Gainwell's MFT puts advisory ``*_warn.txt`` files in the same
inbound path. They're text-format side-channel notes (not X12
envelopes) and are skipped at list time. Use
:meth:`list_inbound_names` if you need the raw list without
the download.
"""
if self._stub:
return self._list_inbound_stub()
return self._list_inbound_paramiko()
def list_inbound_names(self) -> list[InboundFile]:
"""Lightweight listing: returns metadata only, no file download.
Use this when you want to filter the inbound set (e.g. by date)
before paying the download cost. Pair with
:meth:`download_inbound` to fetch a filtered subset on demand.
Real mode is implemented as a single SFTP ``listdir_attr`` call
— sub-second on Gainwell's MFT — versus the full
:meth:`list_inbound` which downloads every file. The
``*_warn.txt`` advisory files are filtered out the same way.
Stub mode returns the same as :meth:`list_inbound` (the stub
only knows about local files; no download cost).
"""
if self._stub:
return self._list_inbound_stub()
return self._list_inbound_names_paramiko()
def download_inbound(self, f: InboundFile) -> Path:
"""Download a single inbound file to its ``local_path``.
Idempotent: if ``f.local_path`` already exists and is
non-empty, the download is skipped. Callers should use the
``InboundFile`` returned by :meth:`list_inbound_names` and pass
it back here — ``local_path`` is the planned cache location
and matches the path the scheduler will read from.
Returns:
The on-disk path (same as ``f.local_path``).
Raises:
FileNotFoundError: if the file is missing locally in stub
mode, or if the remote file disappears between list
and download in real mode.
"""
if f.local_path.exists() and f.local_path.stat().st_size > 0:
log.debug(
"SFTP: %s already cached at %s, skipping download",
f.name, f.local_path,
)
return f.local_path
if self._stub:
# Stub mode: no remote — the file is supposed to already be
# at f.local_path (operator-dropped). If it isn't there, the
# operator hasn't seeded the stub; raise loudly.
if not f.local_path.is_file():
raise FileNotFoundError(
f"inbound stub file not found: {f.local_path}"
)
return f.local_path
return self._download_inbound_paramiko(f)
def read_file(self, remote_path: str) -> bytes:
"""Read bytes from a remote path.
@@ -119,6 +232,37 @@ class SftpClient:
return secrets.STUB_SECRET
return value
# ---- Async surface (SP27 Task 8) -----------------------------------
#
# Every sync SFTP call has an async wrapper that runs the paramiko
# call on a worker thread and applies an ``asyncio.wait_for(...
# timeout=N)`` around it. The wait_for cancels the awaiter but
# leaves the worker thread running until paramiko returns on its
# own (paramiko is not asyncio-aware, so we can't cancel the
# underlying socket cleanly). The scheduler should treat
# ``asyncio.TimeoutError`` from these wrappers as a transient
# SFTP error and surface it in ``Scheduler.status()`` (Task 9).
#
# The timeout is read on every call (see ``_op_timeout_seconds``)
# so an operator who tunes ``CYCLONE_SFTP_OP_TIMEOUT_SECONDS`` at
# runtime sees the new value on the next tick.
async def async_list_inbound(self) -> list["InboundFile"]:
"""Async-wrapped :meth:`list_inbound` with a per-op timeout.
The 06/25 silent hang was a hung ``listdir_attr`` that froze
the worker thread indefinitely. This wrapper applies
``asyncio.wait_for(...)`` so the event loop can give up after
``CYCLONE_SFTP_OP_TIMEOUT_SECONDS`` (default 30s) and the
scheduler tick can surface the timeout in
``result.errors`` (and, once Task 9 lands, in
``Scheduler.status()``).
"""
return await asyncio.wait_for(
asyncio.to_thread(self.list_inbound),
timeout=_op_timeout_seconds(),
)
# ---- Stub implementations (SP9) -------------------------------------
def _write_bytes_stub(self, remote_path: str, content: bytes) -> Path:
@@ -284,6 +428,12 @@ class SftpClient:
if attr.st_mode and (attr.st_mode & 0o170000) == 0o040000:
# Directory entry — skip.
continue
if attr.filename.endswith("_warn.txt"):
# Gainwell's MFT drops text-format advisory notes in
# the same inbound path. They're side-channel noise,
# not X12 envelopes — skip at list time so we don't
# download ~600 advisory files per poll.
continue
remote = f"{inbound_dir.rstrip('/')}/{attr.filename}"
cache_path = cache_dir / attr.filename
# Download into cache. We use ``prefetch`` to keep memory
@@ -299,6 +449,56 @@ class SftpClient:
))
return files
def _list_inbound_names_paramiko(self) -> list[InboundFile]:
"""List inbound names via paramiko; do NOT download (lightweight).
Same ``listdir_attr`` iteration as
:meth:`_list_inbound_paramiko`, but the returned
:class:`InboundFile` records have ``local_path`` set to the
planned cache location without actually fetching the file.
Pair with :meth:`_download_inbound_paramiko` to fetch on
demand. Skips ``*_warn.txt`` advisory files the same way.
"""
with self._connect() as (ssh, sftp):
inbound_dir = self._block.paths.get("inbound", "/")
staging = Path(self._block.staging_dir).resolve()
inbound_rel = inbound_dir.lstrip("/")
cache_dir = staging / inbound_rel
cache_dir.mkdir(parents=True, exist_ok=True)
files: list[InboundFile] = []
try:
attrs = sftp.listdir_attr(inbound_dir)
except IOError as exc:
log.warning("SFTP: cannot list %s: %s", inbound_dir, exc)
return []
for attr in sorted(attrs, key=lambda a: a.filename):
if attr.st_mode and (attr.st_mode & 0o170000) == 0o040000:
# Directory entry — skip.
continue
if attr.filename.endswith("_warn.txt"):
continue
cache_path = cache_dir / attr.filename
files.append(InboundFile(
name=attr.filename,
size=attr.st_size or 0,
modified_at=datetime.fromtimestamp(attr.st_mtime or 0),
local_path=cache_path,
))
return files
def _download_inbound_paramiko(self, f: InboundFile) -> Path:
"""Download a single ``f`` to ``f.local_path`` (idempotent on size>0)."""
with self._connect() as (ssh, sftp):
inbound_dir = self._block.paths.get("inbound", "/")
remote = f"{inbound_dir.rstrip('/')}/{f.name}"
f.local_path.parent.mkdir(parents=True, exist_ok=True)
with sftp.open(remote, "rb") as src, open(f.local_path, "wb") as dst:
shutil.copyfileobj(src, dst, length=64 * 1024)
log.info("SFTP: downloaded %d bytes for %s", f.local_path.stat().st_size, f.name)
return f.local_path
def _read_file_paramiko(self, remote_path: str) -> bytes:
with self._connect() as (ssh, sftp):
buf = io.BytesIO()
+731 -2
View File
@@ -74,6 +74,18 @@ def main(ctx: click.Context, log_format: str | None, log_file: Path | None) -> N
ctx.ensure_object(dict)
# Register the auth users subgroup. Imported here (not at module top) to
# avoid pulling passlib / bcrypt at CLI parse-only import time.
from cyclone.auth.cli import users_cli # noqa: E402
main.add_command(users_cli)
# Register the dev seed subcommand. Imported here for the same lazy-load
# reason as users_cli — keeps passlib/bcrypt + SQLAlchemy out of the
# parse-only path so ``python -m cyclone --help`` stays snappy.
from cyclone.seed_cli import seed_cli # noqa: E402
main.add_command(seed_cli)
@main.command("parse-837")
@click.argument("input_file", type=click.Path(exists=True, dir_okay=False, path_type=Path))
@click.option("--output-dir", required=True, type=click.Path(file_okay=False, path_type=Path))
@@ -285,8 +297,95 @@ def validate_tax_id_cmd(tax_id: str, log_level: str) -> None:
sys.exit(1)
if __name__ == "__main__":
main()
# ---------------------------------------------------------------------------
# SP32: `cyclone backfill-rendering-npi` (Task 6)
#
# Re-parses on-disk 837p/835 files and patches up the typed NPI columns
# (``Claim.rendering_provider_npi`` / ``Remittance.rendering_provider_npi``)
# for rows that were ingested before the T4 writers took effect.
# ---------------------------------------------------------------------------
@main.command("backfill-rendering-npi")
@click.option(
"--file", "files",
multiple=True,
type=click.Path(exists=True, dir_okay=False, path_type=Path),
help=(
"Path to a specific file to re-parse. May be passed multiple times. "
"Mutually informative with --input-dir (both are processed)."
),
)
@click.option(
"--input-dir",
type=click.Path(file_okay=False, path_type=Path),
default=None,
help=(
"Directory to scan one level deep for *.txt / *.edi / *.x12 files. "
"Honors CYCLONE_BACKFILL_INPUT_DIR if --input-dir is not passed."
),
)
@click.option(
"--type", "transaction_type",
type=click.Choice(["837p", "835"]),
default=None,
help=(
"Pin the parser to use. Without --type, each file's transaction "
"kind is sniffed (filename hint + ISA/ST prefix)."
),
)
@click.option(
"--log-level",
default="INFO",
show_default=True,
type=click.Choice(["DEBUG", "INFO", "WARNING", "ERROR"]),
)
def backfill_rendering_npi(
files: tuple[Path, ...],
input_dir: Path | None,
transaction_type: str | None,
log_level: str,
) -> None:
"""Re-parse on-disk X12 files to populate the rendering NPI columns.
Idempotent: rows whose ``rendering_provider_npi`` is already non-NULL
are left untouched. After patching, runs :func:`cyclone.reconcile.run`
once across every 835 batch so the T5 scoring arm can fire
retroactively on the touched pairs.
Exit code: 0 on a successful run (zero populated rows is still exit 0).
"""
from cyclone import db as _db
from cyclone.store.backfill import backfill_rendering_provider_npi
# SP18: re-run setup so per-command --log-level overrides the
# group default. ``setup_logging`` is idempotent.
setup_logging(level=log_level)
# The backup pattern: each CLI subcommand that touches the DB
# initializes it explicitly so ``python -m cyclone.cli backup list``
# works on a fresh machine with no prior app boot.
_db.init_db()
# --input-dir falls back to CYCLONE_BACKFILL_INPUT_DIR.
if input_dir is None and not files:
env_dir = os.environ.get("CYCLONE_BACKFILL_INPUT_DIR", "").strip()
if env_dir:
input_dir = Path(env_dir)
summary = backfill_rendering_provider_npi(
files=list(files) if files else None,
input_dir=input_dir,
transaction_type=transaction_type,
)
# One-line summary so operators can grep a log scrape quickly.
click.echo(
f"claims_updated={summary.claims_updated} "
f"remits_updated={summary.remits_updated} "
f"files_processed={summary.files_processed} "
f"files_skipped={summary.files_skipped}"
)
# ---------------------------------------------------------------------------
@@ -300,6 +399,468 @@ if __name__ == "__main__":
# ---------------------------------------------------------------------------
# ---------------------------------------------------------------------------
# SP33: `cyclone backfill-999-rejections`
#
# One-shot replay of the cascade fix in `apply_999_rejections` for any
# 999 acks already in the DB. Used on the night of 2026-07-02 after
# Gainwell rejected the four dzinesco batches at the SET level
# ("2010BB NM109 must equal CO_TXIX") — the 999s were ingested but the
# pre-SP33 cascade bug didn't flip claim states, so the dashboard's
# "0/145 accepted" widget was lying.
#
# Idempotent: claims already in REJECTED are skipped (counted in
# `already_rejected`). Safe to re-run.
# ---------------------------------------------------------------------------
@main.command("backfill-999-rejections")
@click.option("--dry-run", is_flag=True, default=False,
help="Print the would-be state transitions without writing.")
@click.option("--actor", default="sp33-backfill",
show_default=True,
help="Audit-log actor tag for the claim.rejected events.")
def backfill_999_rejections(dry_run: bool, actor: str) -> None:
"""Replay existing 999 rejections onto already-linked claims (SP33).
Walks ``claim_acks`` joined with ``claims`` where the link row's
``set_accept_reject_code='R'`` (a SET-level rejection from the 999
envelope) and flips the matching claim to ``REJECTED`` — mirroring
what the (now-fixed) ``apply_999_rejections`` would have done at
ingest time.
Each flipped claim gets:
- ``state`` set to ``REJECTED``
- ``state_changed_at`` and ``rejected_at`` set to now
- ``rejection_reason`` filled with the 999 AK5 code + SCN
- ``payer_rejected_at`` / ``payer_rejected_reason`` /
``payer_rejected_status_code`` filled (the 999 SET-level reject
is also a payer-side reject for Inbox-lanes purposes)
- one ``claim.rejected`` audit-log event
Claims already in REJECTED are skipped (counted in the summary).
"""
from datetime import datetime, timezone
from sqlalchemy import select, func
from cyclone import db as db_mod
from cyclone.audit_log import AuditEvent, append_event
db_mod.init_db()
now = datetime.now(timezone.utc)
with db_mod.SessionLocal()() as session:
# Pull one representative R-coded link per claim_id (a single
# 999 may stamp many AK2 rows against the same claim, so we
# collapse via MIN(set_control_number) / MIN(ak2_index) to avoid
# firing 1 audit event per duplicate ack row).
#
# 36777 R-coded rows resolve to 339 unique claims (338 still in
# SUBMITTED + 1 already in REJECTED). Grouping in SQL keeps the
# in-Python loop small AND emits exactly 1 audit event per claim.
rows = session.execute(
select(
db_mod.Claim.id,
db_mod.Claim.state,
func.min(db_mod.ClaimAck.set_control_number).label("scn"),
func.min(db_mod.ClaimAck.ak2_index).label("ak2"),
func.min(db_mod.Ack.ack_code).label("ack_code"),
)
.join(db_mod.ClaimAck, db_mod.ClaimAck.claim_id == db_mod.Claim.id)
.join(db_mod.Ack, db_mod.Ack.id == db_mod.ClaimAck.ack_id)
.where(db_mod.ClaimAck.set_accept_reject_code == "R")
.where(db_mod.ClaimAck.claim_id.is_not(None))
.group_by(db_mod.Claim.id, db_mod.Claim.state)
).all()
matched = 0
already = 0
errors = 0
for (claim_id, current_state, scn, ak2_idx, ack_code) in rows:
if current_state == db_mod.ClaimState.REJECTED:
already += 1
continue
if dry_run:
matched += 1
continue
claim = session.get(db_mod.Claim, claim_id)
if claim is None:
errors += 1
continue
claim.state = db_mod.ClaimState.REJECTED
claim.state_changed_at = now
claim.rejected_at = now
claim.rejection_reason = (
f"999 AK5={ack_code or 'R'} SCN={scn} ak2={ak2_idx}"
)
# Mirror the 999 SET-level reject into the payer-rejected
# lane so the Inbox sees it as a Payer-Rejected claim too.
claim.payer_rejected_at = now
claim.payer_rejected_reason = f"999 SET-level reject at SCN={scn}"
claim.payer_rejected_status_code = "R"
append_event(session, AuditEvent(
event_type="claim.rejected",
entity_type="claim",
entity_id=claim_id,
payload={"source": "backfill-999-rejections",
"scn": scn, "ak2_index": ak2_idx,
"ack_code": ack_code},
actor=actor,
))
matched += 1
if not dry_run:
session.commit()
click.echo(
f"matched={matched} already_rejected={already} errors={errors} "
f"dry_run={dry_run}"
)
# ---------------------------------------------------------------------------
# SP33: `cyclone resubmit-rejected-claims`
#
# Push the corrected single-claim 837 files to Gainwell's SFTP ToHPE
# dir so dzinesco can resubmit the batch. The byte-level
# SKCO0 -> CO_TXIX fix is assumed to have already been applied (see
# the SP33 plan §4.4 / `docs/ingest/corrected/`). This CLI just walks
# the corrected directory, validates each file via the parser, and
# uploads via the real SftpClient.
#
# Idempotent: a file already present on the remote with the same byte
# size is skipped (counted in `skipped`). Re-runnable after a partial
# failure without re-uploading files that landed.
# ---------------------------------------------------------------------------
@main.command("resubmit-rejected-claims")
@click.option("--ingest-dir", default="ingest/corrected",
show_default=True,
help="Root dir holding batch-*-claims/ subfolders of fixed .x12 files.")
@click.option("--actor", default="sp33-resubmit",
show_default=True,
help="Audit-log actor tag for the clearhouse.submitted events.")
@click.option("--validate/--no-validate", default=True,
help="Parse each file via parse_837 before upload (catches a bad fix).")
@click.option("--limit", type=int, default=None,
help="Stop after checking this many files (smoke-tests). Counts "
"all attempts, not just successful uploads.")
@click.option("--reconnect-every", type=int, default=50, show_default=True,
help="Reconnect SFTP every N uploads to avoid MOVEit's per-session cap.")
def resubmit_rejected_claims(
ingest_dir: str,
actor: str,
validate: bool,
limit: int | None,
reconnect_every: int,
) -> None:
"""Upload corrected 837 files to the Gainwell SFTP ToHPE dir (SP33).
Walks every ``batch-*-claims/*.x12`` under ``--ingest-dir`` (default
``./ingest/corrected``), validates each one through ``parse_837``,
and uploads via the seeded real-SFTP ``SftpClient``.
Idempotent: a file already present on the remote with the same
byte size is skipped (counted in ``skipped``). Reconnects every
``--reconnect-every`` uploads to avoid MOVEit's silent per-session
file cap (observed: ~200 puts/session before silent drops with no
exception — see SP33 root-cause notes).
Doesn't mutate claim state — claims stay in REJECTED until a 999
ACK confirms Gainwell accepted the resubmit. Emits one
``clearhouse.submitted`` audit event per successful upload.
"""
import time
from cyclone import db as db_mod
from cyclone.audit_log import AuditEvent, append_event
from cyclone.parsers.parse_837 import parse as parse_837_text
from cyclone.parsers.payer import PayerConfig
from cyclone.store import store as cycl_store
db_mod.init_db()
cycl_store.ensure_clearhouse_seeded()
clearhouse = cycl_store.get_clearhouse()
if clearhouse is None:
click.echo("No clearhouse seeded; cannot resolve SFTP block.", err=True)
sys.exit(2)
sftp_block = clearhouse.sftp_block
if sftp_block.stub:
click.echo("Clearhouse SFTP block is in stub mode; refusing to upload.", err=True)
sys.exit(2)
remote_root = sftp_block.paths["outbound"]
root = Path(ingest_dir).resolve()
if not root.exists():
click.echo(f"--ingest-dir does not exist: {root}", err=True)
sys.exit(2)
files: list[Path] = []
for batch_dir in sorted(root.glob("batch-*-claims")):
files.extend(sorted(p for p in batch_dir.glob("*.x12")
if not p.name.startswith("._")))
if not files:
click.echo(f"No .x12 files under {root}", err=True)
sys.exit(2)
click.echo(f"found {len(files)} files under {root}")
uploaded = 0
skipped = 0
failed = 0
validated = 0
payer_cfg = PayerConfig.co_medicaid()
start = time.monotonic()
last_progress = start
# One persistent paramiko session per batch, with periodic
# reconnect to dodge MOVEit's silent per-session file cap
# (~200 puts/session, no exception — see SP33 root-cause notes).
import paramiko
from cyclone.secrets import get_secret
def _open_session() -> tuple[paramiko.SSHClient, paramiko.SFTPClient]:
pw = get_secret(sftp_block.auth.get("password_keychain_account", ""))
ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh.connect(
sftp_block.host, port=sftp_block.port,
username=sftp_block.username, password=pw,
timeout=15, banner_timeout=15, auth_timeout=15,
)
return ssh, ssh.open_sftp()
def _close_session(ssh: paramiko.SSHClient | None) -> None:
if ssh is None:
return
try: ssh.close()
except Exception: pass
ssh: paramiko.SSHClient | None = None
sftp: paramiko.SFTPClient | None = None
for i, src in enumerate(files, 1):
if limit is not None and i > limit:
break
content = src.read_bytes()
# Validate: parse must succeed AND payer_id must match the
# companion-guide CO_TXIX (catches a bad byte-fix early).
if validate:
try:
parsed = parse_837_text(content.decode(), payer_cfg)
except Exception as exc: # noqa: BLE001
failed += 1
click.echo(f"PARSE FAIL {src.name}: {exc.__class__.__name__}: {exc}", err=True)
continue
mismatch = next(
(c for c in parsed.claims if c.payer.id != "CO_TXIX"), None
)
if mismatch is not None:
failed += 1
click.echo(
f"PAYER MISMATCH {src.name}: payer.id={mismatch.payer.id!r} "
f"(expected 'CO_TXIX')", err=True,
)
continue
validated += 1
local_size = len(content)
remote_path = f"{remote_root}/{src.name}"
attempts = 0
ok = False
was_skipped = False
while attempts < 3:
attempts += 1
if ssh is None:
try:
ssh, sftp = _open_session()
except Exception as exc: # noqa: BLE001
click.echo(
f"SFTP CONNECT FAIL attempt {attempts}: "
f"{exc.__class__.__name__}: {exc}", err=True,
)
continue
try:
# Idempotency check.
try:
rs = sftp.stat(remote_path) # type: ignore[union-attr]
if rs.st_size == local_size:
skipped += 1
was_skipped = True
break
except IOError:
pass # not on remote yet
sftp.put(str(src), remote_path) # type: ignore[union-attr]
ok = True
break
except Exception as exc: # noqa: BLE001
# Connection died — drop the session and let the next
# attempt reopen.
_close_session(ssh)
ssh, sftp = None, None
if attempts >= 3:
click.echo(
f"UPLOAD FAIL {src.name}: "
f"{exc.__class__.__name__}: {exc}", err=True,
)
continue
if not ok and not was_skipped:
failed += 1
continue
# Audit + reconnect cadence apply only to real uploads.
if not was_skipped:
# Audit (best-effort; if the DB is unavailable we still
# keep the file on the wire).
try:
with db_mod.SessionLocal()() as session:
append_event(session, AuditEvent(
event_type="clearhouse.submitted",
entity_type="claim_file",
entity_id=src.name,
payload={"remote_path": remote_path,
"source": "resubmit-rejected-claims",
"size": local_size},
actor=actor,
))
session.commit()
except Exception as exc: # noqa: BLE001
click.echo(
f"audit-log write failed for {src.name}: "
f"{exc.__class__.__name__}: {exc}", err=True,
)
uploaded += 1
# Reconnect periodically to dodge MOVEit's per-session cap.
if uploaded % reconnect_every == 0:
click.echo(f"reconnecting (after {uploaded} uploads)", err=True)
_close_session(ssh)
ssh, sftp = None, None
# Progress every 10s of wall-clock (or at end).
now = time.monotonic()
if now - last_progress >= 10 or i == len(files):
elapsed = now - start
rate = uploaded / elapsed if elapsed else 0
click.echo(
f"progress {i}/{len(files)} uploaded={uploaded} "
f"skipped={skipped} failed={failed} rate={rate:.2f}/s "
f"elapsed={elapsed:.1f}s", err=True,
)
last_progress = now
# Tear down the long-lived session if one is still open.
if ssh is not None:
_close_session(ssh)
elapsed = time.monotonic() - start
click.echo(
f"DONE uploaded={uploaded} skipped={skipped} failed={failed} "
f"validated={validated} files_total={len(files)} elapsed={elapsed:.1f}s"
)
@main.command("submit-batch")
@click.option("--ingest-dir", default="ingest",
show_default=True,
help="Root dir holding batch-*-claims/ subfolders of .x12 files.")
@click.option("--actor", default="cli-submit-batch",
show_default=True,
help="Audit-log actor tag for the clearhouse.submitted events.")
@click.option("--validate/--no-validate", default=True,
help="Parse each file via parse_837 before upload.")
@click.option("--limit", type=int, default=None,
help="Stop after N files (smoke-tests).")
def submit_batch(
ingest_dir: str,
actor: str,
validate: bool,
limit: int | None,
) -> None:
"""Parse → DB-write → SFTP-upload each batch-*-claims/*.x12 (SP37).
Canonical outbound path. Mirrors ``resubmit-rejected-claims`` but
routes through ``cyclone.submission.submit_file``, which writes to
the DB before uploading. Use this as the preferred outbound path;
``resubmit-rejected-claims`` remains for one-off cases where you
don't want a DB row (dzinesco-generated fixes not ready for
canonical tracking).
Exit codes: 0 = run completed (even with per-file failures), 1 =
unexpected exception, 2 = config-level failure (no clearhouse,
SFTP block in stub mode, ingest-dir missing).
"""
from cyclone import db as db_mod
from cyclone.submission.core import submit_file
from cyclone.submission.result import SubmitOutcome
from cyclone.store import store as cycl_store
db_mod.init_db()
cycl_store.ensure_clearhouse_seeded()
clearhouse = cycl_store.get_clearhouse()
if clearhouse is None:
click.echo("No clearhouse seeded; cannot resolve SFTP block.", err=True)
sys.exit(2)
sftp_block = clearhouse.sftp_block
if sftp_block.stub:
click.echo("Clearhouse SFTP block is in stub mode; refusing to upload.", err=True)
sys.exit(2)
root = Path(ingest_dir).resolve()
if not root.exists():
click.echo(f"--ingest-dir does not exist: {root}", err=True)
sys.exit(2)
files: list[Path] = []
for batch_dir in sorted(root.glob("batch-*-claims")):
files.extend(sorted(p for p in batch_dir.glob("*.x12")
if not p.name.startswith("._")))
if not files:
click.echo(f"No .x12 files under {root}", err=True)
sys.exit(2)
click.echo(f"found {len(files)} files under {root}")
submitted = 0
skipped = 0
failed = 0
for i, src in enumerate(files, 1):
if limit is not None and i > limit:
break
try:
result = submit_file(
src, sftp_block=sftp_block, actor=actor, validate=validate,
)
except Exception as exc: # noqa: BLE001
click.echo(f"UNEXPECTED {src.name}: {exc.__class__.__name__}: {exc}", err=True)
failed += 1
continue
if result.outcome == SubmitOutcome.SUBMITTED:
submitted += 1
click.echo(f"submitted {src.name} batch={result.batch_id}")
elif result.outcome == SubmitOutcome.SKIPPED:
skipped += 1
click.echo(f"skipped {src.name} (already uploaded)")
else:
failed += 1
click.echo(
f"{result.outcome.value:<10} {src.name} {result.error or ''}",
err=True,
)
click.echo(f"\nsummary: submitted={submitted} skipped={skipped} failed={failed}")
# Per-file failures don't bump exit code; details in stdout.
# (Per cyclone-cli convention.)
@main.group()
def backup() -> None:
"""Encrypted DB backup management (SP17)."""
@@ -557,3 +1118,171 @@ def backup_status() -> None:
snap = svc.status()
import json
click.echo(json.dumps(snap, indent=2, default=str))
# ---------------------------------------------------------------------------
# SP-N fix: `cyclone pull-inbound` — targeted inbound pull + process
#
# Mirrors POST /api/admin/scheduler/pull-inbound. For operators who
# want to drive the daily "process today's 999s" workflow from a cron
# job or shell script without going through the HTTP API.
#
# Exit codes:
# 0 — processed ≥1 file (or processed 0 with no matches: not an
# error, just nothing to do)
# 1 — unexpected exception
# 2 — SFTP / config error
# ---------------------------------------------------------------------------
@main.command("pull-inbound")
@click.option(
"--date", "date_str",
required=True,
help="Date filter YYYYMMDD — only files whose 8-digit timestamp "
"substring matches are downloaded and processed.",
)
@click.option(
"--block", "sftp_block_name",
default="dzinesco",
show_default=True,
help="Name of the SftpBlock in config/payers.yaml to use.",
)
@click.option(
"--file-types", "file_types_csv",
default=None,
help="Comma-separated whitelist (default: 999,TA1).",
)
@click.option(
"--limit", default=2000, show_default=True, type=int,
)
def pull_inbound(
date_str: str,
sftp_block_name: str,
file_types_csv: str | None,
limit: int,
) -> None:
"""List, date-filter, download, and process inbound MFT files.
Bypasses the alphabetical full-listing pass so a daily pull of
~400 files takes seconds, not hours. Files already in the local
cache are skipped (idempotent).
"""
import asyncio as _asyncio
from cyclone import db as db_mod
from cyclone import scheduler as scheduler_mod
from cyclone.edi.filenames import ALLOWED_FILE_TYPES
from cyclone.clearhouse import SftpClient
from cyclone.providers import SftpBlock
# Validate the date filter.
if not (len(date_str) == 8 and date_str.isdigit()):
click.echo(f"--date must be YYYYMMDD, got {date_str!r}", err=True)
sys.exit(2)
db_mod.init_db()
# Find the SftpBlock. The dzinesco clearhouse singleton is seeded
# by store.ensure_clearhouse_seeded() (SP9) and carries the
# production SFTP block; that's the only one we have at the
# moment. For multi-provider SFTP, a config-loader is the right
# thing — out of scope for this CLI which is the daily-pull path.
from cyclone import store as store_mod
store_mod.store.ensure_clearhouse_seeded()
clearhouse = store_mod.store.get_clearhouse()
if clearhouse is None or clearhouse.name != sftp_block_name:
# Fall back: try the named block, but v1 only ships the
# dzinesco singleton.
if clearhouse is None:
click.echo(
f"No clearhouse seeded — cannot find SftpBlock "
f"{sftp_block_name!r}.",
err=True,
)
sys.exit(2)
click.echo(
f"SftpBlock {sftp_block_name!r} not seeded; only "
f"{clearhouse.name!r} is available.",
err=True,
)
sys.exit(2)
block: SftpBlock = clearhouse.sftp_block
if file_types_csv:
wanted = {t.strip().upper() for t in file_types_csv.split(",") if t.strip()}
unknown = wanted - ALLOWED_FILE_TYPES
if unknown:
click.echo(
f"file_types {sorted(unknown)!r} not in {sorted(ALLOWED_FILE_TYPES)}",
err=True,
)
sys.exit(2)
else:
wanted = {"999", "TA1"}
# Wire up the scheduler singleton (re-use the same pipeline the
# HTTP endpoint uses, including the dedup via processed_inbound_files).
scheduler_mod.configure_scheduler(
block, sftp_block_name=sftp_block_name, force=True,
)
sched = scheduler_mod.get_scheduler()
client = SftpClient(block)
async def _run() -> dict:
from cyclone.edi.filenames import parse_inbound_filename
all_files = await _asyncio.to_thread(client.list_inbound_names)
matched: list = []
for f in all_files:
if f.name.find(date_str) == -1:
continue
try:
parsed = parse_inbound_filename(f.name)
except ValueError:
continue
if parsed.file_type not in wanted:
continue
matched.append(f)
if len(matched) >= limit:
break
download_errors: list[str] = []
downloaded = 0
for f in matched:
try:
await _asyncio.to_thread(client.download_inbound, f)
downloaded += 1
except Exception as exc: # noqa: BLE001
download_errors.append(f"{f.name}: {type(exc).__name__}: {exc}")
tick = await sched.process_inbound_files(matched)
return {
"listed": len(all_files),
"matched": len(matched),
"downloaded": downloaded,
"download_errors": download_errors,
"processed": tick.files_processed,
"skipped": tick.files_skipped,
"errored": tick.files_errored,
}
try:
summary = _asyncio.run(_run())
except Exception as exc: # noqa: BLE001
click.echo(f"pull-inbound failed: {type(exc).__name__}: {exc}", err=True)
sys.exit(1)
click.echo(
f"date={date_str} file_types={sorted(wanted)} block={sftp_block_name} "
f"listed={summary['listed']} matched={summary['matched']} "
f"downloaded={summary['downloaded']} processed={summary['processed']} "
f"skipped={summary['skipped']} errored={summary['errored']}"
)
if summary["download_errors"]:
click.echo("download errors:", err=True)
for e in summary["download_errors"]:
click.echo(f" {e}", err=True)
if __name__ == "__main__":
main()
+116
View File
@@ -30,6 +30,7 @@ from sqlalchemy import (
Numeric,
String,
Text,
func,
text,
)
from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column, relationship, sessionmaker
@@ -220,6 +221,13 @@ class Batch(Base):
totals_json: Mapped[Optional[dict]] = mapped_column(JSONText, nullable=True)
validation_json: Mapped[Optional[dict]] = mapped_column(JSONText, nullable=True)
raw_result_json: Mapped[Optional[dict]] = mapped_column(JSONText, nullable=True)
# SP37 Task 2: source 837's ST02 (transaction set control number).
# Populated from ``Envelope.transaction_set_control_number`` by
# ``store.write.add_record`` for 837P batches; NULL for 835 batches
# (the column is an 837P-specific join key for 999 AK2 resolution).
# Migration 0020 adds the column additively; no backfill required for
# pre-existing rows that lack the value.
transaction_set_control_number: Mapped[Optional[str]] = mapped_column(String(32), nullable=True)
claims: Mapped[list["Claim"]] = relationship(
back_populates="batch", cascade="all, delete-orphan"
@@ -248,6 +256,7 @@ class Claim(Base):
service_date_to: Mapped[Optional[date]] = mapped_column(Date, nullable=True)
charge_amount: Mapped[Decimal] = mapped_column(Numeric(12, 2), nullable=False, default=Decimal("0"))
provider_npi: Mapped[Optional[str]] = mapped_column(String(16), nullable=True)
rendering_provider_npi: Mapped[Optional[str]] = mapped_column(String(16), nullable=True)
payer_id: Mapped[Optional[str]] = mapped_column(String(32), nullable=True)
state: Mapped[ClaimState] = mapped_column(
Enum(ClaimState, native_enum=False), nullable=False, default=ClaimState.SUBMITTED
@@ -316,6 +325,13 @@ class Claim(Base):
Index("ix_claims_state", "state"),
Index("ix_claims_patient_control_number", "patient_control_number"),
Index("ix_claims_service_date_from", "service_date_from"),
# SP27 Task 11: matched-pair drift check (run at startup)
# scans ``WHERE matched_remittance_id IS NOT NULL``. Without
# this index it's a full claim scan. The reverse side
# (``ix_remittances_claim_id``) is added in 0007. Pure index
# (non-unique) — a claim without a match is fine, reversals
# leave the previous claim/claim match intact.
Index("ix_claims_matched_remittance_id", "matched_remittance_id"),
)
@@ -336,6 +352,7 @@ class Remittance(Base):
claim_id: Mapped[Optional[str]] = mapped_column(
String(64), ForeignKey("claims.id"), nullable=True
)
rendering_provider_npi: Mapped[Optional[str]] = mapped_column(String(16), nullable=True)
status_code: Mapped[str] = mapped_column(String(4), nullable=False)
status_label: Mapped[Optional[str]] = mapped_column(String(32), nullable=True)
total_charge: Mapped[Decimal] = mapped_column(Numeric(12, 2), nullable=False, default=Decimal("0"))
@@ -637,6 +654,76 @@ class Two77caAck(Base):
)
# ---------------------------------------------------------------------------
# SP28: per-ACK auto-link join table (claim_acks)
# ---------------------------------------------------------------------------
class ClaimAck(Base):
"""One row per AK2 set-response / ClaimStatus / TA1 envelope link.
SP28. The durable record of "this ACK acknowledges this claim (or
set, or batch)". One 999 row carries many AK2s; one 277CA carries
many ClaimStatuses; each gets its own ClaimAck row so the operator
can answer "which claims does this ack acknowledge?" with a single
SELECT on ``claim_acks``.
See ``docs/superpowers/specs/2026-07-02-cyclone-ack-claim-auto-link-design.md``
§3.1 for the schema decisions (per-AK2 granularity, the two-pass
join, idempotency via the unique index).
"""
__tablename__ = "claim_acks"
id: Mapped[int] = mapped_column(Integer, primary_key=True, autoincrement=True)
# FK to claims.id with ON DELETE CASCADE so removing a claim
# drops every link row referencing it. NULLable so TA1 envelope-level
# rows can populate ``batch_id`` instead (the table CHECK constraint
# requires at least one of the two).
claim_id: Mapped[Optional[str]] = mapped_column(
String(64), ForeignKey("claims.id", ondelete="CASCADE"), nullable=True,
)
# FK to batches.id (a Batch row in the 837 case, or the synthetic
# inbound-batch id when the ack arrived outside the SFTP pipeline).
batch_id: Mapped[Optional[str]] = mapped_column(
String(32), ForeignKey("batches.id", ondelete="CASCADE"), nullable=True,
)
# Discriminated union over acks / ta1_acks / two77ca_acks. No FK
# constraint because the three target tables are separate; the
# application enforces the discriminator + the matching row's id.
ack_id: Mapped[int] = mapped_column(Integer, nullable=False)
ack_kind: Mapped[str] = mapped_column(String(8), nullable=False)
ak2_index: Mapped[Optional[int]] = mapped_column(Integer, nullable=True)
# The set_control_number the upstream ack ACTUALLY CARRIED
# (== source 837 ST02 for Gainwell batches). Preserved on the link
# row for orphan traceability — the join may have resolved the
# link via the PCN fallback instead, but the operator still sees
# the value the 999/277CA originally carried.
set_control_number: Mapped[Optional[str]] = mapped_column(String(64), nullable=True)
# AK5 code (A/E/R/X) for 999, STC category (A1/A2/A3/A4/A6/A7)
# for 277CA, envelope ack_code for TA1.
set_accept_reject_code: Mapped[Optional[str]] = mapped_column(String(8), nullable=True)
linked_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False)
linked_by: Mapped[str] = mapped_column(String(16), nullable=False)
__table_args__ = (
Index("ix_claim_acks_claim_id", "claim_id"),
Index("ix_claim_acks_batch_id", "batch_id"),
Index("ix_claim_acks_ack", "ack_kind", "ack_id"),
# Mirror the dedup unique index declared in 0018_claim_acks.sql so
# ``Base.metadata.create_all`` (the test-time safety net) emits the
# same partial-unique constraint that the production migration runner
# applies. Without this a fresh in-memory test DB would not enforce
# idempotency at the DB layer.
Index(
"ux_claim_acks_dedup",
"claim_id", "ack_kind", "ack_id", "ak2_index",
unique=True,
sqlite_where=text("claim_id IS NOT NULL AND ak2_index IS NOT NULL"),
),
)
# ---------------------------------------------------------------------------
# SP11: tamper-evident hash-chained audit_log
# ---------------------------------------------------------------------------
@@ -668,6 +755,11 @@ class AuditLog(Base):
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False)
prev_hash: Mapped[str] = mapped_column(String(64), nullable=False)
hash: Mapped[str] = mapped_column(String(64), nullable=False)
# SP-auth: which authenticated user performed this action. Nullable
# so existing (pre-auth) rows and system-initiated events stay valid.
# NOT part of the hash chain — verify_chain must continue to work on
# legacy rows that pre-date this column.
user_id: Mapped[Optional[int]] = mapped_column(Integer, nullable=True, index=True)
__table_args__ = (
Index("idx_audit_log_entity", "entity_type", "entity_id"),
@@ -836,3 +928,27 @@ class ClearhouseORM(Base):
filename_block_json: Mapped[dict] = mapped_column(JSONText, nullable=False)
sftp_block_json: Mapped[dict] = mapped_column(JSONText, nullable=False)
updated_at: Mapped[str] = mapped_column(String(32), nullable=False)
class User(Base):
"""Auth user (admin / user / viewer)."""
__tablename__ = "users"
id: Mapped[int] = mapped_column(Integer, primary_key=True)
username: Mapped[str] = mapped_column(String(64), unique=True, index=True)
password_hash: Mapped[str] = mapped_column(String(255), nullable=False)
role: Mapped[str] = mapped_column(String(16), nullable=False)
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), server_default=func.now())
disabled_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
class Session(Base):
"""Server-side auth session (HttpOnly cookie holds the id)."""
__tablename__ = "sessions"
id: Mapped[str] = mapped_column(String(64), primary_key=True)
user_id: Mapped[int] = mapped_column(ForeignKey("users.id"), index=True)
expires_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), index=True)
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), server_default=func.now())
+97 -20
View File
@@ -4,12 +4,15 @@ SP9. Source-of-truth spec:
https://hcpf.colorado.gov/tp-x12-filenaming (HCPF X12 File Naming Standards Quick Guide)
Outbound (we send):
{tpid}-{transaction_type}-{yyyymmddhhmmssSSS_MT}-1of1.{ext}
Example: 11525703-837P-20260620132243505-1of1.x12
tp{tpid}-{transaction_type}-{yyyymmddhhmmssSSS_MT}-1of1.{ext}
Example: tp11525703-837P-20260620132243505-1of1.x12
Inbound (HPE sends to our ToHPE):
TP{tpid}-{orig_tx}_M{tracking}-{ts}-1of1_{file_type}.x12
Example: TP11525703-837P_M019048402-20260520231513488-1of1_999.x12
Inbound (HPE sends to our FromHPE):
[Tt][Pp]{tpid}-{orig_tx}_M{tracking}-{ts}-1of1_{file_type}.x12
Example: tp11525703-837P_M019048402-20260520231513488-1of1_999.x12
(legacy / prodfiles may use uppercase TP; the regex is case-insensitive
on the prefix to accept both — Gainwell's filer has used both
casings over time.)
Both use Mountain Time (MT) timestamps with 17-digit millisecond precision
(yyyymmddhhmmssSSS = 4+2+2+2+2+2+3 = 17 digits). Sequence is always "1of1"
@@ -28,29 +31,59 @@ from cyclone.providers import InboundFilename
# Regexes
# ---------------------------------------------------------------------------
# Outbound: 11525703-837P-20260620132243505-1of1.x12
# Outbound: tp11525703-837P-20260620132243505-1of1.x12
# - tp: literal "tp" prefix
# - tpid: 1+ digits
# - tx: 1+ alnum
# - ts: 17 digits (yyyymmddhhmmssSSS)
# - seq: literal "1of1"
# - ext: 1+ alnum
OUTBOUND_RE = re.compile(
r"^(?P<tpid>\d+)-(?P<tx>[A-Z0-9]+)-(?P<ts>\d{17})-1of1\.(?P<ext>[A-Za-z0-9]+)$"
r"^tp(?P<tpid>\d+)-(?P<tx>[A-Z0-9]+)-(?P<ts>\d{17})-1of1\.(?P<ext>[A-Za-z0-9]+)$"
)
# Inbound: TP11525703-837P_M019048402-20260520231513488-1of1_999.x12
# Inbound: [Tt][Pp]11525703-837P_M019048402-20260520231513488-1of1_999.x12
# - prefix: literal "TP" or "tp" (case-insensitive — Gainwell's
# production filer has used both)
# - tpid: 1+ digits (inside TP<...>)
# - orig_tx: 1+ alnum
# - track: M + 1+ alnum (e.g. M019048402) — the M is part of the
# tracking value, not a separator.
# - orig_tx: 1+ uppercase alnum
# - track: M + 1+ uppercase alnum (e.g. M019048402) — the M is
# part of the tracking value, not a separator.
# - ts: 17 digits
# - seq: literal "1of1"
# - ft: 1+ alnum (e.g. 999, TA1, 271, 277, 277CA, 820, 834, 835, ENCR)
# - ft: 1+ uppercase alnum (e.g. 999, TA1, 271, 277, 277CA,
# 820, 834, 835, ENCR)
#
# Case insensitivity is scoped to the ``TP`` prefix via ``(?i:TP)``
# — the rest of the pattern is case-sensitive so we still reject a
# stray ``837p`` or ``m019048402`` (they'd be invalid HCPF).
INBOUND_RE = re.compile(
r"^TP(?P<tpid>\d+)-(?P<orig_tx>[A-Z0-9]+)_(?P<tracking>M[A-Z0-9]+)"
r"^(?i:TP)(?P<tpid>\d+)-(?P<orig_tx>[A-Z0-9]+)_(?P<tracking>M[A-Z0-9]+)"
r"-(?P<ts>\d{17})-1of1_(?P<file_type>[A-Z0-9]+)\.(?P<ext>x12)$"
)
# Inbound suffix-less form (SP27 Task 7): tp11525703-835_M019110219-20260525001606050-1of1.x12
# Gainwell's production filer has shipped this shorter form in addition
# to the spec form above — the 6/15-6/19 835 batch arrived this way and
# was silently dropped by the strict INBOUND_RE. The loose form omits
# the `_{file_type}.x12` suffix; the token between `-` and `_M` doubles
# as both `orig_tx` and `file_type` (since there's no separate suffix
# to disambiguate). Allowed values still must be in ALLOWED_FILE_TYPES
# — the suffix is optional, the type-set is not.
# - prefix: literal "TP" or "tp" (case-insensitive — same as INBOUND_RE)
# - tpid: 1+ digits
# - file_type: 3-5 char alphanumeric (covers 999, TA1, 835, 277CA, ENCR;
# capped at 5 to avoid swallowing the next `_M` token)
# - tracking: M + 1+ uppercase alnum
# - ts: 17 digits
# - seq: literal "1of1"
# - ext: literal "x12" (relaxing this would also relax the strict
# form's contract; out of scope for SP27 Task 7)
INBOUND_RE_LOOSE = re.compile(
r"^(?i:TP)(?P<tpid>\d+)-(?P<file_type>[A-Z0-9]{3,5})"
r"_(?P<tracking>M[A-Z0-9]+)-(?P<ts>\d{17})-1of1\.(?P<ext>x12)$"
)
ALLOWED_FILE_TYPES = frozenset({
"999", "TA1", "270", "271", "276", "277", "277CA", "278",
"820", "834", "835", "ENCR",
@@ -79,7 +112,8 @@ def build_outbound_filename(
time in ``America/Denver`` is used.
Returns:
Filename like "11525703-837P-20260620132243505-1of1.x12"
Filename like "tp11525703-837P-20260620132243505-1of1.x12"
(note the ``tp`` prefix per HCPF outbound spec).
Raises:
ValueError: If tpid is non-numeric, tx contains invalid chars, or
@@ -99,7 +133,10 @@ def build_outbound_filename(
# Format: yyyymmddhhmmssSSS — 17 digits total
ts = now_mt.strftime("%Y%m%d%H%M%S") + f"{now_mt.microsecond // 1000:03d}"
assert len(ts) == 17
return f"{tpid}-{tx}-{ts}-1of1.{ext}"
# Per HCPF outbound spec, prefix is "tp" + tpid. Matches the format
# we receive from HPE inbound (which uses uppercase TP) and the
# historical outbound prodfile naming (e.g. tp11525703-837P-...).
return f"tp{tpid}-{tx}-{ts}-1of1.{ext}"
# ---------------------------------------------------------------------------
@@ -110,16 +147,49 @@ def build_outbound_filename(
def parse_inbound_filename(name: str) -> InboundFilename:
"""Parse an inbound HCPF filename.
Accepts both forms (Gainwell ships both):
* Spec form with ``_{file_type}.x12`` suffix:
``tp11525703-837P_M019048402-20260520231513488-1of1_999.x12``
* Suffix-less form (SP27 Task 7): the token between ``-`` and
``_M`` doubles as both ``orig_tx`` and ``file_type``:
``tp11525703-835_M019110219-20260525001606050-1of1.x12``
The strict form is tried first (preserves historical behavior for
every existing caller); the loose form is the fallback. The
``.x12`` extension and ``ALLOWED_FILE_TYPES`` set are enforced in
both forms.
Args:
name: Filename like "TP11525703-837P_M019048402-20260520231513488-1of1_999.x12"
name: Filename like ``tp11525703-837P_M019048402-20260520231513488-1of1_999.x12``
(case-insensitive on the ``TP`` prefix; both ``TP`` and
``tp`` are accepted.)
Returns:
InboundFilename with tpid, orig_tx, tracking, ts, file_type, ext.
Raises:
ValueError: If the filename doesn't match the HCPF inbound format.
ValueError: If the filename doesn't match either HCPF inbound
form, or if the derived file_type isn't in
ALLOWED_FILE_TYPES.
"""
m = INBOUND_RE.match(name)
if m:
file_type = m.group("file_type")
if file_type not in ALLOWED_FILE_TYPES:
raise ValueError(
f"file_type {file_type!r} not in allowed HCPF set: {sorted(ALLOWED_FILE_TYPES)}"
)
return InboundFilename(
tpid=m.group("tpid"),
orig_tx=m.group("orig_tx"),
tracking=m.group("tracking"),
ts=m.group("ts"),
file_type=file_type,
ext=m.group("ext"),
)
# Fall back to the suffix-less form.
m = INBOUND_RE_LOOSE.match(name)
if not m:
raise ValueError(f"Not a valid HCPF inbound filename: {name!r}")
file_type = m.group("file_type")
@@ -129,7 +199,7 @@ def parse_inbound_filename(name: str) -> InboundFilename:
)
return InboundFilename(
tpid=m.group("tpid"),
orig_tx=m.group("orig_tx"),
orig_tx=m.group("file_type"), # preserve the historical shape
tracking=m.group("tracking"),
ts=m.group("ts"),
file_type=file_type,
@@ -148,5 +218,12 @@ def is_outbound_filename(name: str) -> bool:
def is_inbound_filename(name: str) -> bool:
"""True if the given string matches the HCPF inbound filename regex."""
return INBOUND_RE.match(name) is not None
"""True if the given string matches either HCPF inbound form.
Accepts both the spec form (with ``_{file_type}.x12`` suffix) and
the suffix-less form (SP27 Task 7). Cheap fast-path check used by
callers that want to pre-filter a directory listing before invoking
:func:`parse_inbound_filename`; mirrors the parser's own fallback
so the two never disagree on what counts as an HCPF inbound file.
"""
return INBOUND_RE.match(name) is not None or INBOUND_RE_LOOSE.match(name) is not None
+113
View File
@@ -0,0 +1,113 @@
"""File-type handlers for inbound MFT files (SP27).
Each handler is a pure function that parses + persists + dispatches
events for one file type. The scheduler and the FastAPI endpoints
both delegate here; the ``HANDLERS`` registry maps ``file_type`` →
handler function.
Public API:
HandleResult — dataclass returned by every handler
HANDLERS — ``{"999": handle_999, "835": handle_835, ...}``
handle_999, handle_ta1, handle_277ca, handle_835
— call signatures: ``handle(text: str, source_file: str) -> HandleResult``
The handlers own their own DB session lifecycle. They emit pubsub
events via the optional ``event_bus`` parameter (the FastAPI endpoint
injects ``app.state.event_bus``; the scheduler passes ``None``).
They never raise on per-segment problems; per-segment issues are
logged and folded into the result. Whole-document failures
(missing ISA, bad encoding) surface as ``CycloneParseError``, which
the caller catches and records as ``STATUS_ERROR``.
"""
from __future__ import annotations
import importlib
import logging
from ._ack_id import (
ack_count_summary,
ack_synthetic_source_batch_id,
two77ca_synthetic_source_batch_id,
)
from .handle_result import HandleResult
log = logging.getLogger(__name__)
# Module-level HANDLERS dict populated lazily once handler modules
# ship. Keys are file_type strings, values are the ``handle``
# callable for that type.
HANDLERS: dict[str, object] = {}
# Candidate handlers, in registration order. Each tuple is
# (module-path, file_type). The 277/277CA mapping is added explicitly
# after registration when the 277CA handler is present.
_CANDIDATES: list[tuple[str, str]] = [
("cyclone.handlers.handle_999", "999"),
("cyclone.handlers.handle_ta1", "TA1"),
("cyclone.handlers.handle_277ca", "277CA"),
("cyclone.handlers.handle_835", "835"),
]
def register_handlers() -> None:
"""Populate ``HANDLERS`` from the per-type handler modules.
Tolerates missing or broken handler modules so the package can be
imported incrementally as each handler ships (Tasks 2-5), and so
a partial-modification error in one handler module can't break
scheduler / API import. Safe to call multiple times.
"""
if HANDLERS:
return
for mod_path, file_type in _CANDIDATES:
try:
mod = importlib.import_module(mod_path)
fn = getattr(mod, "handle")
except Exception as exc: # noqa: BLE001 — best-effort registry
log.debug(
"handler %s unavailable: %s",
mod_path, exc,
)
continue
HANDLERS[file_type] = fn
# The 277 filename maps to the same 277CA handler.
if file_type == "277CA":
HANDLERS["277"] = fn
register_handlers()
# Re-export handler functions on this package so callers can use the
# flat import (``from cyclone.handlers import handle_999``) once each
# module ships. Set after registration so we know what's present.
def _reexport_handlers() -> None:
"""Re-export each handler's ``handle`` fn as ``handle_<type>``.
No-op for absent handlers. Re-run on each import so freshly-
installed handler modules (e.g. Tasks 2-5 commits) are visible
after ``register_handlers()`` without a process restart.
"""
for file_type, fn in list(HANDLERS.items()):
if file_type in ("277",): # alias of 277CA; don't re-export twice
continue
globals()[f"handle_{file_type.lower()}"] = fn
_reexport_handlers()
__all__ = [
"HANDLERS",
"HandleResult",
"register_handlers",
"ack_count_summary",
"ack_synthetic_source_batch_id",
"two77ca_synthetic_source_batch_id",
# handler_* names are added at module load via _reexport_handlers
]
# Populate __all__ with the present handler symbols.
for _h in ("handle_999", "handle_ta1", "handle_277ca", "handle_835"):
if _h in globals():
__all__.append(_h) # noqa: PYI056
+87
View File
@@ -0,0 +1,87 @@
"""Ack ID helpers shared between the scheduler and the FastAPI
endpoints (SP27 Task 1).
Lifted from ``scheduler.py:_ack_count_summary`` +
``_ack_synthetic_source_batch_id`` + ``_277ca_synthetic_source_batch_id`` —
all three had inline copies in both ``scheduler.py`` and ``api.py``
prior to this SP. Both callers now import from this module.
Helpers
-------
``ack_count_summary(result)``
Aggregate ``(received, accepted, rejected, ack_code)`` from a
parsed ``ParseResult999``, trusting set-level IK5 over the
functional-group AK9. Gainwell's MFT ships AK9 segments that
contradict the per-set IK5 (e.g. ``AK9*A*1*1*1`` with
``IK5*A``), so trusting AK9's rejected count over-reports.
See scheduler commit ``6507a8c`` for the operational context.
``ack_synthetic_source_batch_id(icn, *, pcn, source_filename)``
Build a unique-per-file ``batches.id`` for a 999 that ships
without its own source batch. Falls back to a hash suffix of
the filename so daily pulls don't all collapse onto the same
ICN. Gainwell's MFT ships every 999 with the default ICN
``000000001`` (per the scheduler docstring).
``two77ca_synthetic_source_batch_id(icn)``
Same idea for a 277CA without its own source batch.
"""
from __future__ import annotations
import hashlib
from typing import Any
def ack_count_summary(result: Any) -> tuple[int, int, int, str]:
"""Aggregate ``(received, accepted, rejected, ack_code)`` from
a ``ParseResult999``.
Counts are derived from the set-level ``IK5`` responses
(one per ``AK2`` in the 999), not the functional-group ``AK9``.
Gainwell's MFT ships contradictory AK9 segments; the per-set
IK5 is the authoritative per-claim accept/reject signal.
"""
sets = result.set_responses
received = len(sets)
accepted = sum(1 for s in sets if s.set_accept_reject.code == "A")
rejected = received - accepted
if rejected == 0:
code = "A"
elif accepted == 0:
code = "R"
else:
code = "P"
return (received, accepted, rejected, code)
def ack_synthetic_source_batch_id(
interchange_control_number: str,
*,
pcn: str | None = None,
source_filename: str | None = None,
) -> str:
"""Synthetic ``batches.id`` for a received 999 with no source batch.
Precedence for the human-readable part of the id (column is
``VARCHAR(32)``):
1. ``999-{pcn}-{hash8}`` if ``AK2`` set_control_number is
present (the common case). 4 + 9 + 1 + 8 = 22 chars max.
2. ``999-{icn}-{hash8}`` if no AK2 (envelope-only 999).
3. ``999-{hash12}`` if no filename either (shouldn't happen
in production).
"""
short_hash = ""
if source_filename:
short_hash = hashlib.sha1(source_filename.encode("utf-8")).hexdigest()[:8]
if pcn and pcn.strip():
return f"999-{pcn.strip()}-{short_hash}"
icn = (interchange_control_number or "").strip() or "000000001"
if short_hash:
return f"999-{icn}-{short_hash}"
return f"999-{short_hash or icn}"
def two77ca_synthetic_source_batch_id(interchange_control_number: str) -> str:
"""Synthetic ``batches.id`` for a received 277CA with no source batch."""
return f"277CA-{(interchange_control_number or '').strip() or '000000001'}"
@@ -0,0 +1,152 @@
"""Handle a 277CA Claim Acknowledgment file (SP27 Task 4).
Lifted verbatim from ``scheduler.py:_handle_277ca``. The handler owns
its own DB session, dispatches to ``parse_277ca_text``, persists the
277CA ack row, applies 277CA rejections to matched claims via
``inbox_state.apply_277ca_rejections``, and emits
``claim.payer_rejected`` audit events for each newly-stamped claim.
The actor tag (``"277ca-parser-scheduler"``) is preserved so the
audit log keeps tracing back to the same source after extraction.
Both the FastAPI endpoint and the scheduler path (re)use this module
— the API migration drops the inline copy in Task 6.
``claim.rejected_after_remit`` audit emission (when a 277CA rejects
a claim that already has ``matched_remittance_id`` set) is deferred
to SP27 Task 13. Today the handler only emits ``claim.payer_rejected``.
"""
from __future__ import annotations
import json
import logging
from cyclone import db
from cyclone.audit_log import AuditEvent, append_event
from cyclone.claim_acks import apply_277ca_acks
from cyclone.handlers._ack_id import two77ca_synthetic_source_batch_id
from cyclone.inbox_state_277ca import apply_277ca_rejections
from cyclone.parsers.exceptions import CycloneParseError
from cyclone.parsers.parse_277ca import parse_277ca_text
from cyclone.store import store as cycl_store
log = logging.getLogger(__name__)
def handle(
text: str,
source_file: str,
) -> tuple[str, int]:
"""Parse a 277CA, persist ack + stamp payer-rejected claims.
Args:
text: Raw 277CA document bytes (decoded).
source_file: Filename the 277CA came from.
Returns:
``(parser_used, claim_count)`` tuple where ``claim_count`` is
the number of STC statuses in the file (one per claim).
Raises:
ValueError: on parser-level failure (wraps CycloneParseError).
SP25: the store (``cyclone.store.acks.add_277ca_ack``) owns the
publish-from-store contract; the handler no longer needs an
``event_bus`` kwarg.
"""
try:
result = parse_277ca_text(text, input_file=source_file)
except CycloneParseError as exc:
raise ValueError(f"277CA parse error: {exc}") from exc
icn = result.envelope.control_number
synthetic_id = two77ca_synthetic_source_batch_id(icn)
accepted = sum(
1 for s in result.claim_statuses if s.classification == "accepted"
)
paid = sum(
1 for s in result.claim_statuses if s.classification == "paid"
)
rejected = sum(
1 for s in result.claim_statuses if s.classification == "rejected"
)
pended = sum(
1 for s in result.claim_statuses if s.classification == "pended"
)
# Build the batch envelope index BEFORE opening the work session
# — cycl_store.batch_envelope_index opens its own short-lived
# session, and SQLite + concurrent sessions causes "database is
# locked" errors.
batch_index = cycl_store.batch_envelope_index()
with db.SessionLocal()() as session:
row = cycl_store.add_277ca_ack(
source_batch_id=synthetic_id,
control_number=icn,
accepted_count=accepted,
rejected_count=rejected,
paid_count=paid,
pended_count=pended,
raw_json=json.loads(result.model_dump_json()),
)
def _lookup(pcn: str):
return (
session.query(db.Claim)
.filter(db.Claim.patient_control_number == pcn)
.first()
)
apply_result = apply_277ca_rejections(
session, result, claim_lookup=_lookup, two77ca_id=row.id,
)
if apply_result.matched:
for cid in apply_result.matched:
append_event(session, AuditEvent(
event_type="claim.payer_rejected",
entity_type="claim",
entity_id=cid,
payload={"source_batch_id": synthetic_id, "277ca_id": row.id},
actor="277ca-parser-scheduler",
))
# SP28: auto-link the 277CA ClaimStatus entries to claims (D10
# two-pass join). The helper builds dataclass rows; the
# caller persists each via cycl_store.add_claim_ack so the
# publish-from-store contract owns the live-tail event.
def _pcn_lookup(pcn: str):
return (
session.query(db.Claim)
.filter(db.Claim.patient_control_number == pcn)
.first()
)
link_result = apply_277ca_acks(
session, result, ack_id=row.id,
batch_envelope_index=batch_index,
pc_claim_lookup=_pcn_lookup,
)
# Snapshot the rows before closing the work session — SQLite
# + concurrent sessions from the same thread cause "database
# is locked" errors when add_claim_ack opens its own session
# while this one is still open.
link_rows = list(link_result.linked)
orphans = list(link_result.orphans)
session.commit()
for link_row in link_rows:
cycl_store.add_claim_ack(
claim_id=link_row.claim_id,
batch_id=link_row.batch_id,
ack_id=row.id,
ack_kind="277ca",
ak2_index=link_row.ak2_index,
set_control_number=link_row.set_control_number,
set_accept_reject_code=link_row.set_accept_reject_code,
linked_by="auto",
)
if orphans:
log.warning(
"277CA had %d orphan status entries (no matching claim): %s",
len(orphans),
orphans[:5],
)
return ("parse_277ca", len(result.claim_statuses))
+109
View File
@@ -0,0 +1,109 @@
"""Handle an 835 ERA Remittance file (SP27 Task 5).
Lifted verbatim from ``scheduler.py:_handle_835``. The handler owns
its own state, dispatches to ``parse_835`` (raises
``CycloneParseError`` on bad EDI), runs the per-payer 835 validator,
stamps the validation report into ``result.summary``, and persists
the BatchRecord via ``cycl_store.add``.
The 835 handler is the largest of the four because the schema
covers per-claim remittances + CAS adjustments + validation.
``PAYER_FACTORIES_835`` lives in ``cyclone.api`` (it was always
intended to live in ``cyclone.payers`` — a TODO pre-dating SP27 —
but moving it is out of Task 5 scope). We import it lazily inside
``handle`` to avoid a module-load-time cycle; the cyclic risk is
acceptable because api.py also imports scheduler lazily (inside its
lifespan handler).
Two-phase ingest closed in SP27 Task 10: the placeholder
``adjustment_amount`` is overwritten by reconcile in the same DB
session as the insert (before commit), so a reader never sees a
half-reconciled Remittance row. If reconcile raises, the whole
ingest rolls back and the scheduler records the per-file error.
``event_bus`` is best-effort and follows the same TODO(sp27-task-6)
sync/async gap as handle_999 / handle_ta1 / handle_277ca.
"""
from __future__ import annotations
import logging
import uuid
from datetime import datetime, timezone
from cyclone.parsers.exceptions import CycloneParseError
from cyclone.parsers.parse_835 import parse as parse_835
from cyclone.parsers.validator_835 import validate as validate_835
from cyclone.store import BatchRecord
from cyclone.store import store as cycl_store
log = logging.getLogger(__name__)
def handle(
text: str,
source_file: str,
) -> tuple[str, int]:
"""Parse an 835, run validation, persist batch + remittances.
Args:
text: Raw 835 document bytes (decoded).
source_file: Filename the 835 came from.
Returns:
``(parser_used, claim_count)`` tuple. ``claim_count`` is the
number of CLP claims parsed; the BatchRecord's summary's
``passed`` / ``failed`` fields are derived from the
validator's ``report.passed`` flag.
Raises:
ValueError: on parser-level failure (wraps CycloneParseError).
Validation failures don't raise — they're stamped into
``summary.passed = 0``.
SP25: the 835 write path already publishes ``remittance_written``
via ``CycloneStore.add`` (SP21 split). The handler no longer
accepts an ``event_bus`` kwarg — the ``ack_received`` publish
here was dead code anyway (the 835 event name is
``remittance_written``, not ``ack_received``).
"""
# SP36 Task 16: PAYER_FACTORIES_835 moved from ``cyclone.api``
# to ``cyclone.api_routers._shared`` (cross-router helper). Import
# from the new home. The lazy import is still needed to avoid a
# circular import at registry load time (handle_835 is imported
# by the scheduler during the api lifespan).
from cyclone.api_routers._shared import PAYER_FACTORIES_835
config = PAYER_FACTORIES_835["co_medicaid_835"]()
try:
result = parse_835(text, config, input_file=source_file)
except CycloneParseError as exc:
raise ValueError(f"835 parse error: {exc}") from exc
# Validation report (mirrors the API endpoint).
report = validate_835(result, config)
n = len(result.claims)
if report.passed:
passed, failed, failed_claim_ids = n, 0, []
else:
passed, failed, failed_claim_ids = 0, n, [
c.payer_claim_control_number for c in result.claims
]
result = result.model_copy(update={
"validation": report,
"summary": result.summary.model_copy(update={
"passed": passed,
"failed": failed,
"failed_claim_ids": failed_claim_ids,
}),
})
rec = BatchRecord(
id=uuid.uuid4().hex,
kind="835",
input_filename=source_file,
parsed_at=datetime.now(timezone.utc),
result=result,
)
cycl_store.add(rec)
return ("parse_835", len(result.claims))
+157
View File
@@ -0,0 +1,157 @@
"""Handle a 999 Implementation Acknowledgment file (SP27 Task 2).
Lifted verbatim from ``scheduler.py:_handle_999``. The handler owns
its own DB session, dispatches to ``parse_999_text``, applies 999
rejections to any matched claims via ``inbox_state.apply_999_rejections``,
persists the ack row, and returns a ``(parser_used, claim_count)``
tuple.
The actor tag (``"999-parser-scheduler"``) is preserved so the audit
log keeps tracing back to the same source after extraction. Both
the FastAPI endpoint and the scheduler path (re)use this module —
see Task 6 for the API migration that drops the inline copy.
``claim_count`` mirrors ``parsed.received`` — the count of AK2
(set-level) responses in the 999, which is the authoritative
per-claim accept/reject signal (see ``_ack_id`` for why AK9 is
ignored).
"""
from __future__ import annotations
import json
import logging
from cyclone import db
from cyclone.audit_log import AuditEvent, append_event
from cyclone.claim_acks import apply_999_acceptances
from cyclone.handlers._ack_id import (
ack_count_summary,
ack_synthetic_source_batch_id,
)
from cyclone.inbox_state import apply_999_rejections
from cyclone.parsers.exceptions import CycloneParseError
from cyclone.parsers.parse_999 import parse_999_text
from cyclone.store import store as cycl_store
log = logging.getLogger(__name__)
def handle(
text: str,
source_file: str,
) -> tuple[str, int]:
"""Parse a 999, apply rejections, persist ack row.
Args:
text: Raw 999 document bytes (decoded).
source_file: Filename the 999 came from. Used to derive a
unique synthetic ``batches.id`` (see ``_ack_id``).
Returns:
``(parser_used, claim_count)`` tuple. Matches the scheduler
``_download_and_parse`` destructure; the HandleResult
migration happens in Task 7.
Raises:
ValueError: on parser-level failure (wraps CycloneParseError).
SP25: the store (``cyclone.store.acks.add_999_ack``) now owns the
publish-from-store contract; the handler no longer needs an
``event_bus`` kwarg. Both the scheduler path (no bus) and the
FastAPI endpoint path (passes the bus to the store directly) see
the same row, the same event, and the same payload.
"""
try:
result = parse_999_text(text, input_file=source_file)
except CycloneParseError as exc:
raise ValueError(f"999 parse error: {exc}") from exc
received, accepted, rejected, ack_code = ack_count_summary(result)
icn = result.envelope.control_number
pcn = (
result.set_responses[0].set_control_number
if result.set_responses else None
)
synthetic_id = ack_synthetic_source_batch_id(
icn, pcn=pcn, source_filename=source_file,
)
# Build the batch envelope index BEFORE opening the work session
# — cycl_store.batch_envelope_index opens its own short-lived
# session, and SQLite + concurrent sessions causes "database is
# locked" errors.
batch_index = cycl_store.batch_envelope_index()
with db.SessionLocal()() as session:
def _lookup(pcn: str):
return (
session.query(db.Claim)
.filter_by(patient_control_number=pcn)
.first()
)
rejection_result = apply_999_rejections(
session, result,
claim_lookup=_lookup,
batch_envelope_index=batch_index,
)
if rejection_result.matched:
for cid in rejection_result.matched:
append_event(session, AuditEvent(
event_type="claim.rejected",
entity_type="claim",
entity_id=cid,
payload={"source_batch_id": synthetic_id},
actor="999-parser-scheduler",
))
row = cycl_store.add_ack(
source_batch_id=synthetic_id,
accepted_count=accepted,
rejected_count=rejected,
received_count=received,
ack_code=ack_code,
raw_json=json.loads(result.model_dump_json()),
)
# SP28: auto-link the 999 AK2 set-responses to claims (D10 two-pass
# join). The PCN fallback only fires when the ST02 lookup misses
# (rare for Gainwell batches). Each created ClaimAck row is
# persisted via cycl_store.add_claim_ack so the publish-from-store
# contract owns the live-tail event.
def _pcn_lookup(pcn: str):
return (
session.query(db.Claim)
.filter_by(patient_control_number=pcn)
.first()
)
link_result = apply_999_acceptances(
session, result, ack_id=row.id,
batch_envelope_index=batch_index,
pc_claim_lookup=_pcn_lookup,
)
# Snapshot the rows before closing the work session — SQLite
# + concurrent sessions from the same thread cause "database
# is locked" errors when add_claim_ack opens its own session
# while this one is still open.
link_rows = list(link_result.linked)
orphans = list(link_result.orphans)
session.commit()
for link_row in link_rows:
cycl_store.add_claim_ack(
claim_id=link_row.claim_id,
batch_id=link_row.batch_id,
ack_id=row.id,
ack_kind="999",
ak2_index=link_row.ak2_index,
set_control_number=link_row.set_control_number,
set_accept_reject_code=link_row.set_accept_reject_code,
linked_by="auto",
)
if orphans:
log.warning(
"999 had %d orphan set refs (no matching claim): %s",
len(orphans),
orphans[:5],
)
return ("parse_999", received)
@@ -0,0 +1,37 @@
"""Shared ``HandleResult`` dataclass for handlers (SP27).
Every per-file-type handler returns the same shape so the
scheduler's ``_handle_one`` and the FastAPI parse endpoints can
process them uniformly.
"""
from __future__ import annotations
from dataclasses import dataclass
@dataclass
class HandleResult:
"""Outcome of one handler invocation.
Attributes:
parser_used: The parser name written to
``processed_inbound_files.parser_used`` (e.g. ``"parse_999"``)
and surfaced in the UI.
claim_count: The number of claim rows persisted (or batch
records, depending on the file type). For 999/TA1 this
is the receipt count; for 835 this is the per-claim
remittance count; for 277CA this is the per-claim
status count.
batch_id: The persisted batch id (when the handler creates a
row in ``batches``). ``None`` for handlers that persist
into per-file-type ack tables (999/TA1/277CA) rather
than into the unified ``batches`` table.
matched_count: For 835, the number of remits that were
matched to an existing claim by ``reconcile.match``.
Zero for other handlers.
"""
parser_used: str
claim_count: int
batch_id: str | None = None
matched_count: int = 0
+123
View File
@@ -0,0 +1,123 @@
"""Handle a TA1 Interchange Acknowledgment file (SP27 Task 3).
Lifted verbatim from ``scheduler.py:_handle_ta1``. The handler owns
its own DB session, dispatches to ``parse_ta1_text``, persists the
interchange ack row in ``ta1_acks``, and returns a
``(parser_used, claim_count)`` tuple.
Unlike the 999 handler, TA1 is envelope-only — there is one TA1 per
ISA/IEA interchange, no set-level (AK2) or claim-level matching.
The claim_count is always 1 (one TA1 ack row per file).
The actor tag is implicit (no audit event here — TA1 doesn't tag
anything in the activity log; it's an infrastructure-level ack).
"""
from __future__ import annotations
import json
import logging
from cyclone import db
from cyclone.claim_acks import apply_ta1_envelope_link
from cyclone.parsers.exceptions import CycloneParseError
from cyclone.parsers.parse_ta1 import parse_ta1_text
from cyclone.store import store as cycl_store
log = logging.getLogger(__name__)
def handle(
text: str,
source_file: str,
) -> tuple[str, int]:
"""Parse a TA1, persist the interchange ack row.
Args:
text: Raw TA1 document bytes (decoded).
source_file: Filename the TA1 came from. Used for audit
attribution; the ``batches.id`` for TA1 rows is derived
internally from the parsed envelope's control number
(``TA1-{ICN}``).
Returns:
``(parser_used, claim_count)`` tuple. TA1 always returns
``claim_count=1`` (one TA1 ack row per interchange file).
Raises:
ValueError: on parser-level failure (wraps CycloneParseError).
SP25: the store (``cyclone.store.acks.add_ta1_ack``) owns the
publish-from-store contract; the handler no longer needs an
``event_bus`` kwarg.
"""
try:
result = parse_ta1_text(text, input_file=source_file)
except CycloneParseError as exc:
raise ValueError(f"TA1 parse error: {exc}") from exc
with db.SessionLocal()() as session:
ta1_ack_row = cycl_store.add_ta1_ack(
source_batch_id=result.source_batch_id,
control_number=result.ta1.control_number,
interchange_date=result.ta1.interchange_date,
interchange_time=result.ta1.interchange_time,
ack_code=result.ta1.ack_code,
note_code=result.ta1.note_code,
ack_generated_date=result.ta1.ack_generated_date,
sender_id=result.envelope.sender_id,
receiver_id=result.envelope.receiver_id,
raw_json=json.loads(result.model_dump_json()),
)
# SP28: TA1 envelope-level link to the originating Batch. The
# closure here matches the most-recent Batch whose envelope
# sender_id/receiver_id matches the TA1 — see spec §D4.
def _batch_lookup(sender_id, receiver_id):
rows = (
session.query(db.Batch)
.filter(
db.Batch.kind == "837p",
db.Batch.raw_result_json.isnot(None),
)
.order_by(db.Batch.parsed_at.desc())
.all()
)
for row in rows:
env = (row.raw_result_json or {}).get("envelope") or {}
if (
env.get("sender_id") == sender_id
and env.get("receiver_id") == receiver_id
):
return row
return None
link_result = apply_ta1_envelope_link(
session, result, ack_id=ta1_ack_row.id,
batch_lookup=_batch_lookup,
)
# Snapshot rows before closing the work session — SQLite +
# concurrent sessions from the same thread cause "database
# is locked" errors when add_claim_ack opens its own session
# while this one is still open.
link_rows = list(link_result.linked)
orphans = list(link_result.orphans)
session.commit()
for link_row in link_rows:
cycl_store.add_claim_ack(
claim_id=link_row.claim_id,
batch_id=link_row.batch_id,
ack_id=ta1_ack_row.id,
ack_kind="ta1",
ak2_index=link_row.ak2_index,
set_control_number=link_row.set_control_number,
set_accept_reject_code=link_row.set_accept_reject_code,
linked_by="auto",
)
if orphans:
log.warning(
"TA1 had %d orphan envelope refs (no matching batch): %s",
len(orphans),
orphans[:5],
)
return ("parse_ta1", 1)
+87
View File
@@ -175,6 +175,82 @@ def _line_count_lookup(session: Session, claims: list[Claim]) -> tuple[dict, dic
return matched_counts, total_lines_by_claim
def _ack_summary_for_claims(
session: Session, claim_ids: list[str],
) -> dict[str, dict]:
"""Build a {claim_id: {total, rejected, items: [...]}} map for 999 acks.
SP29: the Inbox `rejected` lane needs to render AK2 evidence
inline per row, plus a per-row Resubmit button. The data lives
in ``claim_acks`` (SP28) — we don't want to N+1 fetch per row,
so the whole rejected-claim set is summarized in one batched
query here.
Filters to ``ack_kind='999'`` because the rejected lane is the
999 envelope reject lane; the 277CA STC A4/A6/A7 evidence flows
through the ``payer_rejected_*`` fields on a separate lane and
isn't part of this scope (see SP29 spec D4 / scope).
Returns:
``{claim_id: {"total": int, "rejected": int, "items": [...]}, ...}``
Claims with zero linked 999 acks are NOT in the returned
dict — the caller maps via ``.get(cid)`` and treats absence
as "no 999 acks linked" (renders as ``null`` in the
payload, ``999 not linked`` in the UI).
Args:
session: SQLAlchemy session the caller owns.
claim_ids: list of claim.id values to summarize. Typically
the rejected-lane claim ids. Empty list → empty dict.
"""
if not claim_ids:
return {}
from cyclone.db import ClaimAck # late import — DB model registered
rows = (
session.query(
ClaimAck.claim_id,
ClaimAck.ack_id,
ClaimAck.set_control_number,
ClaimAck.set_accept_reject_code,
ClaimAck.ak2_index,
ClaimAck.linked_at,
)
.filter(
ClaimAck.claim_id.in_(claim_ids),
ClaimAck.ack_kind == "999",
)
.order_by(ClaimAck.linked_at.desc(), ClaimAck.id.desc())
.all()
)
grouped: dict[str, list[tuple]] = {}
for cid, aid, scn, code, ak2i, lat in rows:
grouped.setdefault(cid, []).append((aid, scn, code, ak2i, lat))
rejected_codes = {"R", "E", "X"}
out: dict[str, dict] = {}
for cid, items in grouped.items():
total = len(items)
rejected_count = sum(1 for it in items if (it[2] or "") in rejected_codes)
# Keep 5 most recent items for the chip column. The full count
# is in ``total`` so the UI can show ``+N more`` honestly.
trimmed = items[:5]
out[cid] = {
"total": total,
"rejected": rejected_count,
"items": [
{
"ack_id": aid,
"set_control_number": scn,
"set_accept_reject_code": code or "",
"ak2_index": ak2i,
"linked_at": _isoformat(lat),
}
for (aid, scn, code, ak2i, lat) in trimmed
],
}
return out
def compute_lanes(session: Session, *, dismissed_pairs: Iterable[frozenset]) -> Lanes:
lanes = Lanes()
dismissed = set(dismissed_pairs)
@@ -192,6 +268,17 @@ def compute_lanes(session: Session, *, dismissed_pairs: Iterable[frozenset]) ->
),
))
# SP29: attach the 999 ack-evidence summary (total / rejected /
# 5 most recent AK2 set_responses) to every rejected row so the
# Inbox can render AK2 chips inline + a per-row Resubmit button
# without an extra round-trip. One batched query, keyed off the
# rejected-claim id set.
rejected_ack_summary = _ack_summary_for_claims(
session, [r["id"] for r in lanes.rejected]
)
for row in lanes.rejected:
row["claim_acks"] = rejected_ack_summary.get(row["id"])
# --- Payer-Rejected (SP10) ---
# Distinct from the 999 envelope "rejected" lane above. A claim
# lands here when a 277CA STC category code is A4/A6/A7 (rejected
+37 -16
View File
@@ -33,42 +33,63 @@ def apply_999_rejections(
parsed_999,
*,
claim_lookup: Callable[[str], Claim | None],
batch_envelope_index: dict[str, list[str]] | None = None,
) -> Apply999Result:
"""For each set response with code R or E, look up the matching claim and
"""For each set response with code R, E, or X, look up the matching claim and
move it to REJECTED. Idempotent on already-rejected claims.
Args:
session: SQLAlchemy session.
parsed_999: a ParseResult999 (or any object with .set_responses).
claim_lookup: callable from patient_control_number → Claim or None.
Legacy fallback; rarely hits when batch_envelope_index is present.
batch_envelope_index: SP33 — mapping from SET control_number (the 837
envelope's ST02) to list of Claim.id for the claims in that SET.
Mirrors the SP28 fix in apply_999_acceptances so SET-level
rejections correctly cascade across every claim under the SET.
Returns:
Apply999Result with lists of matched claim ids and orphan PCNs.
"""
result = Apply999Result()
now = datetime.now(timezone.utc)
index = batch_envelope_index or {}
for sr in parsed_999.set_responses:
code = sr.set_accept_reject.code
if code not in ("R", "E", "X"):
continue
claim = claim_lookup(sr.set_control_number)
if claim is None:
result.orphans.append(sr.set_control_number)
continue
# SP33: prefer batch_envelope_index (SCN -> [claim_id]) so a SET-level
# rejection correctly flips every claim in the SET. Fall back to
# the legacy claim_lookup when the index is empty for this SCN.
candidate_ids = index.get(sr.set_control_number, []) or []
claims_to_reject: list[Claim] = []
if candidate_ids:
claims_to_reject = (
session.query(Claim)
.filter(Claim.id.in_(candidate_ids))
.all()
)
else:
legacy = claim_lookup(sr.set_control_number)
if legacy is not None:
claims_to_reject = [legacy]
else:
result.orphans.append(sr.set_control_number)
continue
if claim.state == ClaimState.REJECTED:
# Idempotent: don't double-mutate.
continue
claim.state = ClaimState.REJECTED
claim.state_changed_at = now
claim.rejected_at = now
claim.rejection_reason = _build_reason(
code, len(sr.segment_errors or [])
)
result.matched.append(claim.id)
for claim in claims_to_reject:
if claim.state == ClaimState.REJECTED:
# Idempotent: don't double-mutate.
continue
claim.state = ClaimState.REJECTED
claim.state_changed_at = now
claim.rejected_at = now
claim.rejection_reason = _build_reason(
code, len(sr.segment_errors or [])
)
result.matched.append(claim.id)
if result.matched or result.orphans:
session.commit()
@@ -0,0 +1,31 @@
-- version: 13
-- Auth (SP-auth): users + sessions tables.
--
-- `users` holds the local credential store: bcrypt-hashed password,
-- role enum ('admin' | 'user' | 'viewer'), and a soft-delete column
-- (disabled_at) so admins can revoke access without losing history.
--
-- `sessions` holds the server-side session rows; the browser only
-- carries an opaque token cookie (cyclone_session) that points here.
-- expires_at index lets us cheaply reap stale sessions.
CREATE TABLE users (
id INTEGER PRIMARY KEY AUTOINCREMENT,
username TEXT NOT NULL UNIQUE,
password_hash TEXT NOT NULL,
role TEXT NOT NULL,
created_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP,
disabled_at TEXT
);
CREATE INDEX idx_users_username ON users(username);
CREATE TABLE sessions (
id TEXT PRIMARY KEY,
user_id INTEGER NOT NULL REFERENCES users(id),
expires_at TEXT NOT NULL,
created_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP
);
CREATE INDEX idx_sessions_user_id ON sessions(user_id);
CREATE INDEX idx_sessions_expires_at ON sessions(expires_at);
@@ -0,0 +1,10 @@
-- version: 14
-- Auth (SP-auth): record the acting user_id on every audit_log entry.
--
-- Backwards-compatible: existing rows get NULL user_id (they were
-- written by the pre-auth `system` actor). Going forward, the FastAPI
-- get_current_user dependency injects the id into every audit log call.
ALTER TABLE audit_log ADD COLUMN user_id INTEGER;
CREATE INDEX idx_audit_log_user_id ON audit_log(user_id);
@@ -0,0 +1,74 @@
-- version: 15
-- Drop the inline UNIQUE(batch_id, patient_control_number) on claims.
--
-- Migration 0003 attempted DROP INDEX IF EXISTS uq_claims_batch_pcn but
-- the constraint is inline in CREATE TABLE, so the drop was a no-op.
-- The only way to remove an inline UNIQUE in SQLite is table recreation.
--
-- Discovery 2026-06-23: the inline UNIQUE does NOT exist in the current
-- production DB at user_version=14 (or in main's fresh-DB schema). The
-- 32 "Duplicate claim" warnings in /tmp/cyclone-uvicorn.log are PK
-- collisions on claims.id (CLM01) when an operator re-uploads the same
-- file — not UNIQUE violations. This migration is therefore a defensive
-- no-op against the current schema, but keeps the 0003 intent alive
-- (drop the constraint if it ever reappears) and lets the SP22 spec
-- ship as designed.
--
-- X12 837P allows any number of CLM segments per 2000B subscriber loop;
-- claim identity is provided by the primary key (claims.id = CLM01).
-- The remittances table had a parallel constraint already removed in 0003
-- (because that one WAS a named index), so this migration only touches
-- claims.
--
-- The migration runner (db_migrate.py) wraps each .sql in an implicit
-- transaction via engine.begin(), so we MUST NOT use BEGIN/COMMIT.
-- PRAGMA defer_foreign_keys defers FK checks to commit, which is the
-- only way to drop a referenced table inside a transaction in SQLite.
-- Other tables referencing claims:
-- remittances.claim_id
-- matches.claim_id
-- line_reconciliations.claim_id
-- activity_events.claim_id
PRAGMA defer_foreign_keys = ON;
CREATE TABLE claims_new (
id TEXT PRIMARY KEY,
batch_id TEXT NOT NULL REFERENCES batches(id) ON DELETE CASCADE,
patient_control_number TEXT NOT NULL,
service_date_from DATE,
service_date_to DATE,
charge_amount NUMERIC(12, 2) NOT NULL DEFAULT 0,
provider_npi TEXT,
payer_id TEXT,
state TEXT NOT NULL DEFAULT 'submitted',
state_before_reversal TEXT,
matched_remittance_id TEXT REFERENCES remittances(id),
raw_json TEXT,
rejection_reason TEXT,
rejected_at TIMESTAMP,
resubmit_count INTEGER NOT NULL DEFAULT 0,
state_changed_at TIMESTAMP,
payer_rejected_at TEXT,
payer_rejected_reason TEXT,
payer_rejected_status_code TEXT,
payer_rejected_by_277ca_id TEXT,
payer_rejected_acknowledged_at TEXT,
payer_rejected_acknowledged_actor TEXT
-- NO UNIQUE (batch_id, patient_control_number) — removed.
);
INSERT INTO claims_new SELECT * FROM claims;
DROP TABLE claims;
ALTER TABLE claims_new RENAME TO claims;
-- Recreate secondary indexes (same names, same columns as initial schema
-- plus later migrations).
CREATE INDEX ix_claims_state ON claims(state);
CREATE INDEX ix_claims_patient_control_number ON claims(patient_control_number);
CREATE INDEX ix_claims_service_date_from ON claims(service_date_from);
CREATE INDEX ix_claims_state_changed_at ON claims(state, state_changed_at);
CREATE INDEX idx_claims_payer_rejected_at ON claims(payer_rejected_at);
CREATE INDEX idx_claims_payer_rejected_unack
ON claims(payer_rejected_at)
WHERE payer_rejected_acknowledged_at IS NULL;
@@ -0,0 +1,15 @@
-- version: 16
-- Add the missing index on claims.matched_remittance_id.
--
-- SP27 Task 11 added ``check_matched_pair_drift`` at startup, which
-- scans ``WHERE Claim.matched_remittance_id IS NOT NULL``. Without an
-- index this is a full-table scan; becomes a noticeable boot
-- latency cost past ~10k claims. The companion index on the
-- reverse side (``remittances.claim_id``) was added in 0007.
--
-- A plain index is enough — neither side is unique (reversals
-- re-reference the original PCN, and a claim without a match is
-- fine).
CREATE INDEX ix_claims_matched_remittance_id
ON claims(matched_remittance_id);
@@ -0,0 +1,28 @@
-- version: 17
-- Backfill claims.patient_control_number = claims.id.
--
-- SP27 Task 17 fixed the 837 ingest in store.py:_claim_837_row so
-- ``Claim.patient_control_number`` is populated from
-- ``claim.claim_id`` (CLM01) instead of ``claim.subscriber.member_id``
-- (the 2010BA NM109). The reconcile matcher joins on
-- ``Claim.patient_control_number == Remittance.payer_claim_control_number``
-- and the 835 echoes CLM01 in CLP01 per X12 spec, so the wrong field
-- silently broke every auto-match in production.
--
-- For rows written BEFORE the fix, the stored value is the member_id
-- (e.g. "W953474") which never matches any remit's CLP01. This
-- migration backfills those rows by aligning
-- ``patient_control_number`` with the row's own ``id`` (= CLM01).
-- After this, every claim row's PCN is consistent with the value the
-- 837 actually sent, and any newly-ingested 835 whose CLP01 echoes
-- that CLM01 will auto-pair.
--
-- Idempotent: only touches rows where the stored PCN doesn't already
-- match the row's id, so re-running on already-fixed rows is a no-op.
--
-- Reversal safety: the migration does NOT clear matched_remittance_id,
-- so any pre-existing manual_match pairs stay intact.
UPDATE claims
SET patient_control_number = id
WHERE patient_control_number IS DISTINCT FROM id;
@@ -0,0 +1,52 @@
-- version: 18
-- SP28: per-ACK auto-link join table.
--
-- Closes the operator gap where every inbound 999 / 277CA / TA1 ack was
-- persisted but never durably linked back to the claim it
-- acknowledges. One row per AK2 set-response for 999, per ClaimStatus
-- for 277CA, per TA1 envelope (with claim_id NULL + batch_id set).
--
-- Granularity (per-AK2) is preserved by ``ak2_index`` and the unique
-- index ``ux_claim_acks_dedup`` — an auto-link of
-- (claim, 999, ak2_index=3) is idempotent on re-ingest of the same
-- 999 file (the index enforces this at the DB layer; the helper
-- pre-checks to avoid IntegrityError log noise).
--
-- Notes:
-- * ``claim_id`` is nullable so TA1 envelope-level links to the
-- originating Batch can land here (FK to batches.id). The
-- CHECK constraint makes sure at least one of (claim_id,
-- batch_id) is set on every row — see spec §3.1.
-- * ``set_control_number`` records the value the upstream ACK
-- ACTUALLY CARRIED (== source 837 ST02 for Gainwell batches). It
-- is the orphan-traceability field — the link survives even when
-- the join had to fall back from ST02 to PCN matching.
-- * ``set_accept_reject_code`` carries the AK5 code (A/E/R/X) for
-- 999 or the STC category code (A1/A2/A3/A4/A6/A7 etc.) for 277CA.
-- TA1 stores the envelope-level ack code here (A/R/E).
-- * No FK constraint on ``(ack_kind, ack_id)`` — there are three
-- separate ack tables (``acks``, ``ta1_acks``, ``two77ca_acks``).
-- Application code enforces the discriminator.
CREATE TABLE claim_acks (
id INTEGER PRIMARY KEY AUTOINCREMENT,
claim_id TEXT REFERENCES claims(id) ON DELETE CASCADE,
batch_id TEXT REFERENCES batches(id) ON DELETE CASCADE,
ack_id INTEGER NOT NULL,
ack_kind TEXT NOT NULL CHECK (ack_kind IN ('999', '277ca', 'ta1')),
ak2_index INTEGER,
set_control_number TEXT,
set_accept_reject_code TEXT,
linked_at DATETIME NOT NULL,
linked_by TEXT NOT NULL CHECK (linked_by IN ('auto', 'manual')),
CHECK ((claim_id IS NOT NULL) OR (batch_id IS NOT NULL))
);
CREATE INDEX ix_claim_acks_claim_id ON claim_acks(claim_id);
CREATE INDEX ix_claim_acks_batch_id ON claim_acks(batch_id);
CREATE INDEX ix_claim_acks_ack ON claim_acks(ack_kind, ack_id);
-- Dedup: an auto-link of (claim, 999, ak2_index=3) is idempotent on re-ingest.
CREATE UNIQUE INDEX ux_claim_acks_dedup
ON claim_acks(claim_id, ack_kind, ack_id, ak2_index)
WHERE claim_id IS NOT NULL AND ak2_index IS NOT NULL;
@@ -0,0 +1,7 @@
-- version: 19
-- SP32: render & service-provider NPI extraction.
-- Nullable: existing rows stay NULL until backfill runs.
-- No indexes (used for set-equality, not range queries; nullable).
ALTER TABLE claims ADD COLUMN rendering_provider_npi TEXT;
ALTER TABLE remittances ADD COLUMN rendering_provider_npi TEXT;
@@ -0,0 +1,27 @@
-- version: 20
-- SP37: Batch.transaction_set_control_number = parsed 837's ST02.
--
-- Today's 999 ack join (claim_acks.batch_envelope_index, Pass 1) matches
-- on ``Batch.envelope.control_number == 999's set_control_number``. That
-- never resolves in production because 999's set_control_number (AK201)
-- echoes the source 837's ST02 (transaction set control number), not the
-- ISA13 (interchange control number) that Envelope.control_number stores.
-- Result: every AK2 set-response against a dzinesco-generated 837 turns
-- into an orphan.
--
-- SP37 fixes this by adding a column populated from the parsed 837's
-- ST02 on every ``add_record`` write, then updating Pass 1 to match on
-- it (Task 2). This migration is the additive part: nullable, no
-- default, backfills from ``raw_result_json.envelope.transaction_set_control_number``
-- for any pre-existing batch rows that already carry the value.
--
-- No new index (column is a primary join key, not a range query; the
-- existing batches table is small enough for a full scan during the
-- 999 join — see SP37 §"Migration 0013").
ALTER TABLE batches ADD COLUMN transaction_set_control_number TEXT;
UPDATE batches
SET transaction_set_control_number = json_extract(raw_result_json, '$.envelope.transaction_set_control_number')
WHERE raw_result_json IS NOT NULL
AND json_extract(raw_result_json, '$.envelope.transaction_set_control_number') IS NOT NULL;
+17
View File
@@ -63,6 +63,7 @@ class ClaimHeader(_Base):
frequency_code: str | None = None
provider_signature: str | None = None
assignment: str | None = None
benefits_assignment_certification: str | None = None # CLM08 (Y/N)
release_of_info: str | None = None
prior_auth: str | None = None
@@ -87,6 +88,14 @@ class ServiceLine(_Base):
place_of_service: str | None = None
service_date: date | None = None
provider_reference: str | None = None
# SV1-07 — Diagnosis Code Pointer. Points to one or more
# diagnosis codes in the parent claim's HI segment ("1".. "12",
# space-separated when multiple). For 837P with a non-empty HI
# segment, SV1-07 is required by HCPF / Gainwell. The parser
# captures it from the source; the serializer defaults to "1"
# when the claim has at least one diagnosis and no explicit
# pointer was captured (matches the common single-dx case).
dx_pointer: str | None = None
class ValidationIssue(_Base):
@@ -111,6 +120,13 @@ class Envelope(_Base):
implementation_guide: str | None = None
# SP3 P1 T2: BHT06 transaction type code (was: transaction_set_purpose_code, which is BHT02).
transaction_type_code: str | None = None
# SP37 Task 2: X12 ST02 (transaction set control number). Distinct
# from ``control_number`` above, which is the ISA13 interchange
# control number. 999 acks echo ST02 back as AK201, so this is the
# join key that lets ``add_record``'s batch row round-trip back to
# its source 837. Populated only by the 837P parser today; other
# parsers share this class but leave the field None.
transaction_set_control_number: str | None = None
class BatchSummary(_Base):
@@ -133,6 +149,7 @@ class ClaimOutput(_Base):
subscriber: Subscriber
payer: Payer
claim: ClaimHeader
rendering_provider_npi: str | None = None # NM1*82 NM109 (Loop 2420A)
diagnoses: list[Diagnosis] = Field(default_factory=list)
service_lines: list[ServiceLine] = Field(default_factory=list)
validation: ValidationReport
@@ -160,6 +160,7 @@ class ClaimPayment(_Base):
ref_benefit_plan: str | None = None # REF*CE per the CO guide
service_payments: list[ServicePayment] = Field(default_factory=list)
raw_segments: list[list[str]] = Field(default_factory=list)
service_provider_npi: str | None = None # NM1*1P NM109 (Loop 2100 service provider)
@model_validator(mode="before")
@classmethod
+12 -3
View File
@@ -376,6 +376,7 @@ def _consume_claim_payment(segments: list[list[str]], idx: int) -> tuple[ClaimPa
service_payments: list[ServicePayment] = []
ref_benefit_plan: str | None = None
service_provider_npi: str | None = None
per_diem: Decimal | None = None
status_label = claim_status_label(status)
@@ -422,9 +423,16 @@ def _consume_claim_payment(segments: list[list[str]], idx: int) -> tuple[ClaimPa
except ValueError:
per_diem = None
elif s[0] == "NM1":
# Patient (QC) / service-provider (1P) — captured in raw_segments.
# The 835 spec doesn't require a structured patient model in v1.
pass
# SP32: capture service-provider NPI from NM1*1P (Loop 2100).
# NM108 (idx 8) carries the ID qualifier (typically "XX");
# NM109 (idx 9) is the value. Some senders omit NM108; accept
# both forms but require a 10-digit ID.
if len(s) > 9 and s[1] == "1P":
if len(s) > 8 and s[8] == "XX" and s[9]:
if s[9].isdigit() and len(s[9]) == 10:
service_provider_npi = s[9]
elif s[9] and s[9].isdigit() and len(s[9]) == 10:
service_provider_npi = s[9]
elif s[0] == "DTM":
# Claim-level dates — captured in raw_segments.
pass
@@ -446,6 +454,7 @@ def _consume_claim_payment(segments: list[list[str]], idx: int) -> tuple[ClaimPa
ref_benefit_plan=ref_benefit_plan,
service_payments=service_payments,
raw_segments=raw,
service_provider_npi=service_provider_npi,
),
idx,
)
+22 -1
View File
@@ -83,6 +83,12 @@ def _build_envelope(segments: list[list[str]], input_file: str = "") -> tuple[En
except (IndexError, ValueError) as exc:
log.warning("Could not parse BHT date: %s", exc)
elif seg[0] == "ST" and envelope is not None:
# SP37 Task 2: capture ST02 (transaction set control number).
# 999 acks echo this back as AK201, so this is what makes the
# batch row joinable once the 999 ingests. Distinct from ISA13
# (which is already on ``control_number``).
if len(seg) > 2:
envelope = envelope.model_copy(update={"transaction_set_control_number": seg[2].strip()})
if len(seg) > 3:
envelope = envelope.model_copy(update={"implementation_guide": seg[3]})
return envelope, summary
@@ -202,6 +208,7 @@ def _consume_claim(segments: list[list[str]], idx: int) -> tuple[ClaimOutput, in
frequency_code=freq or None,
provider_signature=clm[6] if len(clm) > 6 else None,
assignment=clm[7] if len(clm) > 7 else None,
benefits_assignment_certification=clm[8] if len(clm) > 8 else None,
release_of_info=clm[9] if len(clm) > 9 else None,
)
@@ -209,6 +216,7 @@ def _consume_claim(segments: list[list[str]], idx: int) -> tuple[ClaimOutput, in
service_lines: list[ServiceLine] = []
raw: list[list[str]] = [seg]
prior_auth: str | None = None
rendering_provider_npi: str | None = None
idx += 1
while idx < len(segments) and segments[idx][0] not in {"HL", "CLM", "SE"}:
@@ -225,6 +233,11 @@ def _consume_claim(segments: list[list[str]], idx: int) -> tuple[ClaimOutput, in
for code in parts[1:]:
if code:
diagnoses.append(Diagnosis(code=code, qualifier=qualifier))
elif s[0] == "NM1" and len(s) > 1 and s[1] == "82":
# SP32: capture rendering provider NPI from Loop 2420A NM1*82.
# NM108 (idx 8) is the qualifier (typically "XX"), NM109 (idx 9) is the NPI.
if len(s) > 9 and s[8] == "XX" and len(s[9]) == 10 and s[9].isdigit():
rendering_provider_npi = s[9]
elif s[0] == "LX":
line_no = int(s[1]) if len(s) > 1 and s[1].isdigit() else len(service_lines) + 1
# LX is just a separator — the actual service line data is in the next SV1.
@@ -254,6 +267,7 @@ def _consume_claim(segments: list[list[str]], idx: int) -> tuple[ClaimOutput, in
subscriber=Subscriber(first_name="", last_name="", member_id="", address=Address(line1="", city="", state="", zip="")),
payer=Payer(name="", id=""),
claim=claim_header,
rendering_provider_npi=rendering_provider_npi,
diagnoses=diagnoses,
service_lines=service_lines,
validation=ValidationReport(passed=True, errors=[], warnings=[]),
@@ -285,11 +299,17 @@ def _consume_service_line(segments: list[list[str]], idx: int, line_no: int) ->
except Exception:
units = None
place_of_service = seg[5] if len(seg) > 5 else None
# SV1-06 (Unit Basis of Measurement) is X12 "UN" for "units" — we
# already use unit_type in SV1-03; SV1-06 is rarely populated and
# is not required by HCPF.
# SV1-07 — Diagnosis Code Pointer (e.g. "1" for the first HI
# diagnosis). Required by HCPF when the claim has diagnoses.
dx_pointer = seg[7] if len(seg) > 7 and seg[7] else None
service_date: date | None = None
provider_ref: str | None = None
idx += 1
while idx < len(segments) and segments[idx][0] not in {"LX", "HL", "CLM", "SE"}:
while idx < len(segments) and segments[idx][0] not in {"LX", "HL", "CLM", "SE", "NM1"}:
s = segments[idx]
raw.append(s)
if s[0] == "DTP" and len(s) > 2 and s[1] == "472":
@@ -311,6 +331,7 @@ def _consume_service_line(segments: list[list[str]], idx: int, line_no: int) ->
place_of_service=place_of_service,
service_date=service_date,
provider_reference=provider_ref,
dx_pointer=dx_pointer,
),
idx,
)
+19 -2
View File
@@ -8,6 +8,12 @@ Single-pass walker over the tokenized segment list:
- AK3 (Segment Context) + AK4 (Element Context) — optional per-segment errors
- AK5 (Transaction Set Response Status) — per-set accept/reject
- AK9 (Functional Group Response Status) — per-group counts + ack code
- IK5 — a non-standard synonym for ``AK5`` that Gainwell's MFT ships
in place of the spec-defined ``AK5``. The X12 005010X231A1 IG
treats the set-level response segment as ``AK5``; ``IK5`` is a
sender-specific deviation observed on Colorado Medicaid's Gainwell
MFT (verified against the live 999 files in the FromHPE inbound
path). We accept either.
- SE / GE / IEA
Errors at the file level raise :class:`CycloneParseError`. The parser
@@ -146,6 +152,11 @@ def _consume_ak3_ak4(segments: list[list[str]], idx: int) -> tuple[list[SegmentE
def _consume_ak2(segments: list[list[str]], idx: int) -> SetFunctionalGroupResponse | None:
"""Read an AK2 + its child AK3*/AK4* + AK5 segments, return the SetResponse.
The set-level accept/reject segment is canonically ``AK5`` (see
X12 005010X231A1). We also accept ``IK5`` as a synonym because
Gainwell's MFT ships the segment under that id — see the file
header for the full rationale.
Returns None when called with a non-AK2 segment (defensive — the
orchestrator only calls this when it sees AK2).
"""
@@ -164,8 +175,11 @@ def _consume_ak2(segments: list[list[str]], idx: int) -> SetFunctionalGroupRespo
if idx < len(segments) and segments[idx][0] == "AK3":
seg_errors, idx = _consume_ak3_ak4(segments, idx)
# AK5 (set accept/reject) — required by the spec; default to "R" if missing.
# Gainwell's MFT uses IK5 instead of AK5 (sender-specific segment id
# that means the same thing); accept either. The default of "R"
# matters: if the segment is missing entirely, the 999 is a reject.
accept_code = "R"
if idx < len(segments) and segments[idx][0] == "AK5":
if idx < len(segments) and segments[idx][0] in ("AK5", "IK5"):
ak5 = segments[idx]
if len(ak5) > 1 and ak5[1]:
accept_code = ak5[1]
@@ -256,8 +270,11 @@ def parse_999_text(text: str, *, input_file: str = "") -> ParseResult999:
set_responses.append(sr)
# Advance past the AK2 + AK3*/AK4*/AK5 cluster
# (re-walk from i+1 because _consume_ak2 doesn't return idx).
# ``IK5`` is the Gainwell-specific synonym for ``AK5``
# and must be in the consumed set here too (see
# _consume_ak2 for the full rationale).
i += 1
while i < len(segments) and segments[i][0] in {"AK3", "AK4", "AK5"}:
while i < len(segments) and segments[i][0] in {"AK3", "AK4", "AK5", "IK5"}:
i += 1
else:
i += 1
+2 -2
View File
@@ -66,8 +66,8 @@ class PayerConfig(BaseModel):
# Lenient in v1 — see spec §9 R031.
require_ref_g1_for_adjustments=False,
allowed_bht06={"CH"},
payer_id="SKCO0",
payer_name="COHCPF",
payer_id="CO_TXIX",
payer_name="CO_TXIX",
no_patient_loop=True,
encounter_claim_in_same_batch=False,
allowed_facility_qualifiers={"B"},
+149 -49
View File
@@ -112,7 +112,10 @@ def _build_gs(sender_id: str, receiver_id: str, group_control_number: str) -> st
_FUNCTIONAL_ID_HEALTH_CARE,
sender_id,
receiver_id,
_today_yymmdd(),
# GS-04 must be CCYYMMDD (8 digits) per X12 — ISA uses YYMMDD
# (6 digits) for the older format, but the GS segment is the
# newer ANSI X12 format and requires the full year.
_today_yyyymmdd(),
_today_hhmm(),
group_control_number,
"X",
@@ -186,17 +189,30 @@ def _build_nm1(entity_id_qualifier: str, entity_type: str, name: str,
return _ELEM.join(parts) + _SEG
def _build_per(contact_name: str | None, contact_phone: str | None) -> str:
"""PER segment — submitter contact. Returns empty when no contact info."""
if not contact_name and not contact_phone:
return ""
parts = [
"PER",
"IC", # PER01 — contact function code (Information Contact)
contact_name or "",
"TE", # PER03 — phone qualifier
contact_phone or "",
]
def _build_per(
contact_name: str | None,
contact_phone: str | None,
contact_email: str | None = None,
email_qual: str = "EM",
) -> str:
"""PER segment — submitter contact (Loop 1000A).
X12 005010X222A1 *requires* at least one PER segment in Loop 1000A
(Submitter Name) and at least PER01 must be present, so this
builder always emits a segment. PER01 = "IC" (Information Contact).
The remaining elements are filled from the available contact info:
name, then email (preferred — Gainwell/HCPF expect this), then phone.
"""
parts = ["PER", "IC"]
if contact_name:
parts.append(contact_name)
if contact_email:
parts.append(email_qual) # PER03 — email qualifier (default "EM")
parts.append(contact_email) # PER04 — the email itself
elif contact_phone:
parts.append("TE") # PER03 — phone qualifier
parts.append(contact_phone) # PER04 — the phone itself
return _ELEM.join(parts) + _SEG
@@ -236,26 +252,40 @@ def _build_hl(hl_id: str, parent_id: str, level_code: str, child_code: str) -> s
return _ELEM.join(parts) + _SEG
def _build_sbr(relationship_code: str | None, member_id: str | None,
payer_name: str | None) -> str:
def _build_sbr(
individual_relationship_code: str | None,
claim_filing_indicator_code: str | None,
) -> str:
"""SBR segment — subscriber information.
SBR01 (relationship code) defaults to ``"P"`` (Patient = self) which is
the most common case for professional claims; the parser does not store
this on the canonical Subscriber model so we cannot thread it through
without adding a model field.
Slot layout (X12 005010X222A1):
SBR01 — Payer Responsibility Sequence Number Code. Default ``"P"``
(Patient = primary). The parser does not capture this
field on ``ClaimOutput`` so we default it.
SBR02 — Individual Relationship Code. ``"18"`` = self, ``"01"`` = spouse, etc.
The parser does not capture this either; we default ``"18"``
for the common self-pay case.
SBR09 — Claim Filing Indicator Code. ``"MC"`` for Medicaid,
``"16"`` for Medicare Part B, etc. The canonical
PayerConfig837 carries ``sbr09_default``; we thread it in
from the caller.
The member_id and payer name do NOT belong in SBR — the member_id
lives in NM109 of the NM1*IL segment, and the payer name is in
NM103 of NM1*PR. (Earlier revisions of this function put them in
SBR06 / SBR09, which is wrong and rejected by HCPF.)
"""
parts = [
"SBR",
relationship_code or "P",
"", # SBR02 — group number
"", # SBR03 — group name
"", # SBR04 — claim filing indicator code
"", # SBR05 — sequence number code
payer_name or "", # SBR06 — claim filing indicator code (CO uses MC)
"", # SBR07
"", # SBR08
member_id or "", # SBR09 — claim submitter's id
"P", # SBR01 — primary
individual_relationship_code or "18", # SBR02 — self
"", # SBR03 — group number
"", # SBR04 — group name
"", # SBR05 — insurance type code
"", # SBR06 — coordination of benefits
"", # SBR07 — yes/no condition
"", # SBR08 — employment status code
claim_filing_indicator_code or "", # SBR09 — claim filing indicator
]
return _ELEM.join(parts) + _SEG
@@ -300,7 +330,11 @@ def _build_clm(claim) -> str:
clm05, # CLM05 — composite POS:qualifier:frequency_code
claim.provider_signature or "Y", # CLM06
claim.assignment or "Y", # CLM07
"", # CLM08 — benefit assignment certification
# CLM08 — Benefits Assignment Certification. X12 837P requires
# this when CLM07 = "Y" (the common case for in-network
# professional claims). Default to "Y" when the source did
# not capture one — matches what 99% of HCPF files look like.
claim.benefits_assignment_certification or "Y", # CLM08
claim.release_of_info or "Y", # CLM09
]
return _ELEM.join(parts) + _SEG
@@ -325,21 +359,41 @@ def _build_lx(line_number: int) -> str:
return _ELEM.join(["LX", str(line_number)]) + _SEG
def _build_sv1(line) -> str:
"""SV1 segment — professional service line."""
def _build_sv1(line, *, dx_pointer: str | None = None) -> str:
"""SV1 segment — professional service line.
X12 005010X222A1 layout (837P):
SV1-01 composite procedure identifier
SV1-02 monetary amount (charge)
SV1-03 unit of basis measurement (UN, MJ, etc.) — ``line.unit_type``
SV1-04 service unit count — ``line.units``
SV1-05 place of service code — ``line.place_of_service``
SV1-06 **NOT USED** by this guide (must be empty)
SV1-07 diagnosis code pointer — ``dx_pointer`` (required when the
parent claim has an HI segment)
The parser captures the original SV1-07 pointer when present; the
serializer defaults it to ``"1"`` (pointing at the first HI
diagnosis) when the claim has diagnoses and no explicit pointer
was captured. When the claim has no HI segment we leave SV1-07
empty to match the spec.
"""
proc = line.procedure
code = proc.code if proc else ""
mods = proc.modifiers if proc else []
composite = "HC:" + code + "".join(f":{m}" for m in (mods or [])[:4])
charge = f"{Decimal(line.charge or 0):.2f}"
units = f"{Decimal(line.units):g}" if line.units is not None else "1"
sv1_07 = dx_pointer or ""
parts = [
"SV1",
composite,
charge,
line.unit_type or "UN",
units,
line.place_of_service or "",
composite, # SV1-01
charge, # SV1-02
line.unit_type or "UN", # SV1-03 — unit basis code
units, # SV1-04
line.place_of_service or "", # SV1-05
"", # SV1-06 — NOT USED in 837P
sv1_07, # SV1-07 — diagnosis pointer
]
return _ELEM.join(parts) + _SEG
@@ -357,15 +411,20 @@ def _build_dtp_472(service_date: date | None) -> str:
# ---------------------------------------------------------------------------
def _build_submitter_block(sender_id: str, submitter_name: str | None,
contact_name: str | None,
contact_phone: str | None) -> list[str]:
def _build_submitter_block(
sender_id: str,
submitter_name: str | None,
contact_name: str | None,
contact_phone: str | None,
contact_email: str | None = None,
email_qual: str = "EM",
) -> list[str]:
out = [
_build_nm1("41", "41", submitter_name or sender_id, "46", sender_id),
]
per = _build_per(contact_name, contact_phone)
if per:
out.append(per)
# PER is required by X12 (at least PER01). _build_per always emits
# the segment; the submitter block always has exactly one.
out.append(_build_per(contact_name, contact_phone, contact_email, email_qual))
return out
@@ -395,11 +454,14 @@ def _build_billing_provider_block(provider) -> list[str]:
return out
def _build_subscriber_block(subscriber, payer_name: str | None) -> list[str]:
def _build_subscriber_block(
subscriber,
claim_filing_indicator_code: str | None,
) -> list[str]:
"""HL*2 → SBR → NM1*IL → N3 → N4 → DMG. Subscriber has no children."""
out = [
_build_hl("2", "1", "22", "0"), # HL*2 — subscriber, 0 children
_build_sbr("18", subscriber.member_id, payer_name),
_build_sbr("18", claim_filing_indicator_code),
_build_nm1(
"IL", "IL",
f"{subscriber.last_name} {subscriber.first_name}".strip(),
@@ -427,12 +489,24 @@ def _build_payer_block(payer) -> list[str]:
]
def _build_service_lines_block(service_lines) -> list[str]:
"""Per line: LX / SV1 / DTP*472 / REF*6R."""
def _build_service_lines_block(service_lines, *, has_diagnoses: bool = False) -> list[str]:
"""Per line: LX / SV1 / DTP*472 / REF*6R.
``has_diagnoses`` is True when the parent claim emits an HI segment;
in that case SV1-07 is required by X12 and we default each line's
pointer to ``"1"`` (the first HI diagnosis) unless the source
captured a different pointer on the line itself.
"""
out: list[str] = []
for idx, line in enumerate(service_lines or [], start=1):
out.append(_build_lx(idx))
out.append(_build_sv1(line))
# Prefer the line's captured pointer (parser pulled SV1-07
# when present). Fall back to "1" only when the claim has
# diagnoses and the source had no explicit pointer — the
# common single-diagnosis case.
line_pointer = getattr(line, "dx_pointer", None)
effective_pointer = line_pointer or ("1" if has_diagnoses else "")
out.append(_build_sv1(line, dx_pointer=effective_pointer))
dtp = _build_dtp_472(line.service_date)
if dtp:
out.append(dtp)
@@ -450,7 +524,10 @@ def serialize_837(
submitter_name: str | None = None,
submitter_contact_name: str | None = None,
submitter_contact_phone: str | None = None,
submitter_contact_email: str | None = None,
submitter_contact_email_qual: str = "EM",
receiver_name: str | None = None,
claim_filing_indicator_code: str | None = None,
interchange_control_number: str = "000000001",
group_control_number: str = "1",
) -> str:
@@ -462,6 +539,16 @@ def serialize_837(
(``"CYCLONE"`` / ``"RECEIVER"``) but real deployments should pass
the configured values.
The submitter block (Loop 1000A) always emits a PER segment per the
X12 spec — the canonical clearhouse config provides the contact
name and email so callers should pass them through.
The claim filing indicator (SBR09) is read from the per-payer
config (``PayerConfig837.sbr09_default``); callers should pass it
in. If not passed, SBR09 is left empty (which causes the
:func:`cyclone.parsers.validator._r202_sbr09_allowed` rule to skip
its check — degraded but not a hard error).
Editable fields (CLM, REF*G1, HI, service-line SV1, DTP*472) are
emitted from the canonical ``ClaimOutput`` fields, so post-parse
edits propagate to the output.
@@ -481,11 +568,13 @@ def serialize_837(
),
]
segments.extend(_build_submitter_block(
sender_id, submitter_name, submitter_contact_name, submitter_contact_phone,
sender_id, submitter_name,
submitter_contact_name, submitter_contact_phone, submitter_contact_email,
submitter_contact_email_qual,
))
segments.extend(_build_receiver_block(receiver_id, receiver_name))
segments.extend(_build_billing_provider_block(claim.billing_provider))
segments.extend(_build_subscriber_block(claim.subscriber, claim.payer.name))
segments.extend(_build_subscriber_block(claim.subscriber, claim_filing_indicator_code))
segments.extend(_build_payer_block(claim.payer))
# Claim-level editable segments.
@@ -497,7 +586,10 @@ def serialize_837(
segments.append(_build_hi(claim.diagnoses))
# Service lines (LX / SV1 / DTP*472 / REF*6R).
segments.extend(_build_service_lines_block(claim.service_lines))
segments.extend(_build_service_lines_block(
claim.service_lines,
has_diagnoses=bool(claim.diagnoses),
))
# SE segment count includes ST (line 3, 1-based) through SE itself
# — i.e. the entire ST..SE block inclusive.
@@ -514,15 +606,23 @@ def serialize_837_for_resubmit(
claim: ClaimOutput,
*,
interchange_index: int,
**kwargs,
) -> str:
"""Like :func:`serialize_837` but assigns deterministic-but-unique
interchange + group control numbers for a bundle position.
Interchange number = ``f"{interchange_index:09d}"``.
Group number = ``str(interchange_index)``.
All other keyword arguments (sender_id, receiver_id, submitter_*
and receiver_* contact info, claim_filing_indicator_code) are
forwarded to :func:`serialize_837` unchanged so callers — like the
export and SFTP-submit endpoints — can pass through clearhouse +
payer config without copying the signature.
"""
return serialize_837(
claim,
interchange_control_number=f"{interchange_index:09d}",
group_control_number=str(interchange_index),
**kwargs,
)
+227 -10
View File
@@ -17,7 +17,7 @@ Match algorithm:
from __future__ import annotations
from dataclasses import dataclass
from dataclasses import dataclass, field
from datetime import date, datetime, timezone
from decimal import Decimal
from typing import Iterable, Optional, Protocol
@@ -50,6 +50,18 @@ class Match:
remittance: _RemitLike
strategy: str
is_reversal: bool
# SP31: which content-keys the 2-of-3 (or 3-of-3) rule agreed on.
# Populated for ``score-auto`` matches; empty for ``pcn-exact`` /
# ``manual`` (PCN-exact only uses the PCN key by definition; manual
# is operator-driven). Floats into the ``auto_matched_835``
# ActivityEvent payload so the audit trail records *why* the link
# fired. Transient — the ORM Match row does not persist these
# (spec keeps the audit trail in ActivityEvent.payload_json only).
keys_matched: set[str] = field(default_factory=set)
# SP31: how many candidates were in the ±30-day window pool when
# the score-auto match fired. Useful when an operator is reviewing
# a borderline auto-link — 1-of-1 is unambiguous; 1-of-5 was lucky.
candidate_count: int = 0
def match(
@@ -84,7 +96,7 @@ def match(
matches.append(Match(
claim=chosen,
remittance=r,
strategy="auto",
strategy="pcn-exact",
is_reversal=r.is_reversal,
))
used_claim_ids.add(chosen.id)
@@ -117,6 +129,163 @@ def _pick_claim(
return None
# --- SP31: content-keys fallback matcher -----------------------------------
CHARGE_TOLERANCE = Decimal("0.01")
KEYS_REQUIRED = 2
def _content_keys_match(remit, claim) -> set[str]:
"""Return the set of content-keys that agree between ``remit`` and ``claim``.
Compares {``pcn``, ``charge``, ``npi``} between the two sides and returns
the subset whose values match. Caller checks ``len(returned) >= KEYS_REQUIRED``
to decide whether to auto-link. Returning the matched keys (rather than a
bool) lets the ActivityEvent payload record exactly which 2-of-3 (or 3-of-3)
produced the auto-link — the audit trail for the SP31 content-keys rule.
Charge compared with ``CHARGE_TOLERANCE`` tolerance (rounding drift).
NPI counts as "not matched" when the remit's NPI is empty.
Uses duck-typing via ``getattr`` so the helper works against both:
- shim / dataclass test objects (planned names: total_charge_amount,
total_charge, rendering_provider_npi), and
- real SQLAlchemy ORM instances (Claim.charge_amount, Remittance.total_charge,
Claim.provider_npi, Claim.rendering_provider_npi,
Remittance.rendering_provider_npi).
Production note (SP32): the typed ``rendering_provider_npi`` column is
the primary read for both sides (Claim = NM1*82, Remit = NM1*1P). Legacy
rows pre-0019 still carry the value in ``raw_json``; the read order is:
- NPI: typed column → raw_json (``service_provider_npi`` for remit,
``rendering_provider_npi`` for claim) → claim ``provider_npi``
(billing fallback for legacy 837p rows without NM1*82 extraction).
- Charge: planned attribute name → real ORM column → raw_json.
The ``raw_json`` fallback is what makes this helper safe against a
raw ``Remittance`` ORM instance from ``reconcile.run()``.
"""
matched: set[str] = set()
# PCN: payer_claim_control_number ↔ patient_control_number (stripped).
if (getattr(remit, "payer_claim_control_number", "") or "").strip() == \
(getattr(claim, "patient_control_number", "") or "").strip():
matched.add("pcn")
# Charge: prefer the planned attribute names, fall back to real ORM columns,
# then to raw_json (where the 835 parser stores CLP03). Explicit ``is None``
# checks avoid the ``Decimal("0")`` truthiness footgun — ``Decimal("0")``
# is truthy in a boolean context but ``or`` would still let it through;
# the more important property is that we don't accidentally replace a real
# value with a default.
_remit_charge = getattr(remit, "total_charge_amount", None)
if _remit_charge is None:
_remit_charge = getattr(remit, "total_charge", None)
if _remit_charge is None:
_remit_charge = (getattr(remit, "raw_json", None) or {}).get("total_charge_amount")
_claim_charge = getattr(claim, "total_charge", None)
if _claim_charge is None:
_claim_charge = getattr(claim, "charge_amount", None)
if _claim_charge is None:
_claim_charge = (getattr(claim, "raw_json", None) or {}).get("total_charge_amount")
if _remit_charge is not None and _claim_charge is not None and \
abs(_remit_charge - _claim_charge) < CHARGE_TOLERANCE:
matched.add("charge")
# NPI: typed-column primary path (SP32), raw_json fallback (legacy rows).
# Remit reads rendering_provider_npi (single value, D4); claim reads
# rendering_provider_npi first, then falls back to provider_npi (billing)
# for legacy rows where the 837p parser hadn't yet extracted NM1*82.
remit_npi = (
(getattr(remit, "rendering_provider_npi", "") or "")
or ((getattr(remit, "raw_json", None) or {}).get("service_provider_npi", "") or "")
or ((getattr(remit, "raw_json", None) or {}).get("rendering_provider_npi", "") or "")
).strip()
claim_npi = (
(getattr(claim, "rendering_provider_npi", "") or "")
or (getattr(claim, "provider_npi", "") or "")
or ((getattr(claim, "raw_json", None) or {}).get("rendering_provider_npi", "") or "")
).strip()
if remit_npi and remit_npi == claim_npi:
matched.add("npi")
return matched
# --- SP31: DB-side fallback candidate matcher -------------------------------
SCORE_FALLBACK_WINDOW_DAYS = 30
def _score_fallback_candidates(session, remit) -> Optional[tuple[str, set[str], int]]:
"""Return (claim_id, keys_matched, candidate_count) for a unique 2-of-3 match.
Queries a ±SCORE_FALLBACK_WINDOW_DAYS candidate pool filtered by:
- claim.matched_remittance_id IS NULL (not yet linked)
- claim.state NOT IN terminal set (PAID, DENIED, REJECTED,
REVERSED, RECONCILED)
- claim.service_date_from within window of remit.service_date
Returns ``(claim_id, keys_matched, candidate_count)`` when exactly one
candidate passes the 2-of-3 rule (via :func:`_content_keys_match`),
otherwise ``None``. ``keys_matched`` is the set returned by the helper
(``{"pcn", "charge"}`` or any 2-of-3 / 3-of-3 combination) — the
ActivityEvent payload records it as the audit trail for *why* the
auto-link fired. ``candidate_count`` is the size of the window pool
(independent of how many matched) so the operator can see whether
the match was 1-of-N or 1-of-1.
Ambiguous matches (2+ candidates) also return ``None`` — they land
in the Inbox Unlinked lane for manual review.
Note on payer_id: ``Remittance`` has no ``payer_id`` column
(``Claim.payer_id`` does, and is the source of truth on the claim
side; the 835 parser stores payer_id in ``Remittance.raw_json``).
Filtering by payer would require either a raw_json read or a model
column add — out of scope for SP31 Task 3. The PCN itself is
expected to be unique within a payer's submission, so cross-payer
collisions are unlikely. Revisit if production shows otherwise.
"""
from sqlalchemy import select, and_
from datetime import timedelta
from cyclone.db import Claim, ClaimState
TERMINAL = {
ClaimState.PAID, ClaimState.DENIED, ClaimState.REJECTED,
ClaimState.REVERSED, ClaimState.RECONCILED,
}
if remit.service_date is None:
return None # need a date for the window query
window_lo = remit.service_date - timedelta(days=SCORE_FALLBACK_WINDOW_DAYS)
window_hi = remit.service_date + timedelta(days=SCORE_FALLBACK_WINDOW_DAYS)
candidates = list(session.execute(
select(Claim).where(
and_(
Claim.matched_remittance_id.is_(None),
Claim.service_date_from >= window_lo,
Claim.service_date_from <= window_hi,
Claim.state.notin_([s.value for s in TERMINAL]),
)
)
).scalars().all())
matched_pairs = [
(c.id, _content_keys_match(remit, c))
for c in candidates
]
matched_pairs = [
(cid, ks) for cid, ks in matched_pairs if len(ks) >= KEYS_REQUIRED
]
if len(matched_pairs) == 1:
cid, keys_matched = matched_pairs[0]
return (cid, keys_matched, len(candidates))
return None # 0 matches OR 2+ (ambiguous)
@dataclass
class ApplyIntent:
"""Result of applying a match. The caller persists this.
@@ -224,17 +393,22 @@ class ReconcileResult:
def run(session, batch_id: str) -> ReconcileResult:
"""Orchestrate reconciliation for an 835 batch.
Expects Batch + Remittance + CasAdjustment rows already persisted
(this is called from store.add() AFTER commit). Loads the new
remittances + all currently-unmatched Claims, calls match(), then
applies each match's intent inside the caller's session.
Expects Batch + Remittance + CasAdjustment rows already added to
``session`` (not yet committed). SP27 Task 10 calls this from
inside ``CycloneStore.add``'s ingest session, BEFORE
``s.commit()`` — so the whole 835 ingest (rows + reconcile) is
a single transaction. If anything here raises, the ingest rolls
back; no half-reconciled batch ever lands.
Loads the new remittances + all currently-unmatched Claims, calls
match(), then applies each match's intent inside the caller's session.
Returns counts. Does NOT commit; caller controls transaction.
Raises on failure (caller catches and writes activity event).
"""
from sqlalchemy import select
from cyclone.db import (
Batch, Claim, Remittance, CasAdjustment, Match, ActivityEvent,
Batch, Claim, Remittance, CasAdjustment, Match as MatchORM, ActivityEvent,
ClaimState,
)
@@ -252,6 +426,34 @@ def run(session, batch_id: str) -> ReconcileResult:
matches = match(unmatched_claims, new_remits)
# SP31: content-keys fallback for remits that PCN-exact couldn't pair.
# Operates on the still-unmatched remits so we don't double-match.
matched_remit_ids = {m.remittance.id for m in matches}
used_claim_ids = {m.claim.id for m in matches}
for remit in new_remits:
if remit.id in matched_remit_ids:
continue
if getattr(remit, "claim_id", None) is not None:
continue # already linked
fallback_result = _score_fallback_candidates(session, remit)
if fallback_result is None:
continue
matched_claim_id, keys_matched, candidate_count = fallback_result
# Find the claim object in the unmatched_claims list (it was loaded).
target_claim = next(
(c for c in unmatched_claims if c.id == matched_claim_id),
None,
)
if target_claim is None or target_claim.id in used_claim_ids:
continue
matches.append(Match(
claim=target_claim, remittance=remit,
strategy="score-auto", is_reversal=remit.is_reversal,
keys_matched=keys_matched,
candidate_count=candidate_count,
))
used_claim_ids.add(target_claim.id)
applied = 0
skipped = 0
for m in matches:
@@ -275,7 +477,7 @@ def run(session, batch_id: str) -> ReconcileResult:
skipped += 1
continue
session.add(Match(
session.add(MatchORM(
claim_id=m.claim.id, remittance_id=m.remittance.id,
strategy=m.strategy,
matched_at=datetime.now(timezone.utc),
@@ -293,10 +495,25 @@ def run(session, batch_id: str) -> ReconcileResult:
m.remittance.claim_id = m.claim.id
session.add(ActivityEvent(
ts=datetime.now(timezone.utc), kind=intent.activity_kind,
ts=datetime.now(timezone.utc),
kind=("auto_matched_835" if m.strategy == "score-auto" else intent.activity_kind),
batch_id=batch_id, claim_id=m.claim.id,
remittance_id=m.remittance.id,
payload_json={"new_state": m.claim.state.value},
payload_json=(
{
"new_state": m.claim.state.value, "strategy": m.strategy,
# SP31 spec D8: the auto_matched_835 payload records
# which content-keys the 2-of-3 (or 3-of-3) rule
# agreed on, plus how many candidates were in the
# window pool. ``sorted()`` converts the set to a
# JSON-serializable list and pins a deterministic
# order for tests + audit reads.
"keys_matched": sorted(m.keys_matched),
"candidate_count": m.candidate_count,
}
if m.strategy == "score-auto"
else {"new_state": m.claim.state.value}
),
))
applied += 1
+245 -229
View File
@@ -55,7 +55,10 @@ from cyclone.audit_log import AuditEvent, append_event
from cyclone.clearhouse import InboundFile, SftpClient
from cyclone.db import ProcessedInboundFile
from cyclone.edi.filenames import parse_inbound_filename
from cyclone.inbox_state import apply_999_rejections
from cyclone.handlers import handle_277ca as _handle_277ca
from cyclone.handlers import handle_835 as _handle_835
from cyclone.handlers import handle_999 as _handle_999
from cyclone.handlers import handle_ta1 as _handle_ta1
from cyclone.inbox_state_277ca import apply_277ca_rejections
from cyclone.providers import SftpBlock
@@ -69,6 +72,11 @@ STATUS_SKIPPED = "skipped"
STATUS_PENDING = "pending"
# How long reconfigure_scheduler waits for the in-flight tick to
# drain before cancelling the old task. Matches Scheduler.stop().
_DRAIN_TIMEOUT_SECONDS = 30
# File types we know how to route. The HCPF set is broader (270/271/
# 276/277/278/820/834/ENCR) but Cyclone's parser only covers the
# four below. Files with unknown types are recorded as ``skipped``
@@ -87,6 +95,12 @@ class TickResult:
files_skipped: int = 0
files_errored: int = 0
errors: list[str] = field(default_factory=list)
# SP27 Task 9: ``sftp_failed`` is the discriminator that separates
# SFTP-side failures (the listing step) from per-file processing
# errors. ``tick()`` keys off this flag — NOT ``result.errors`` —
# so a single malformed 999 cannot flip the operator's destructive
# pill. Set in ``_tick_impl``'s listing-error branches only.
sftp_failed: bool = False
def as_dict(self) -> dict[str, Any]:
return {
@@ -99,12 +113,22 @@ class TickResult:
"files_skipped": self.files_skipped,
"files_errored": self.files_errored,
"errors": list(self.errors),
"sftp_failed": self.sftp_failed,
}
@dataclass
class SchedulerStatus:
"""Snapshot of the scheduler's runtime state."""
"""Snapshot of the scheduler's runtime state.
SP27 Task 9 added the four SFTP-error fields
(``consecutive_failures``, ``last_error``, ``last_error_at``,
``last_sftp_attempt_at``) so the operator's UI can tell "all
quiet on the MFT front" from "we've been unable to reach the
MFT server for 3 hours". The previous 06/25 incident went
unnoticed because the status dict had no signal for a hung
SFTP poll.
"""
running: bool
poll_interval_seconds: int
@@ -115,6 +139,13 @@ class SchedulerStatus:
total_skipped: int
total_errored: int
last_tick: Optional[TickResult] = None
# SP27 Task 9: SFTP-error surfacing. The UI flips to a destructive
# pill when ``consecutive_failures >= 3`` (``CYCLONE_SCHEDULER_FAIL_THRESHOLD``
# in the operator pill config; not enforced server-side).
consecutive_failures: int = 0
last_error: Optional[str] = None
last_error_at: Optional[datetime] = None
last_sftp_attempt_at: Optional[datetime] = None
def as_dict(self) -> dict[str, Any]:
return {
@@ -129,6 +160,15 @@ class SchedulerStatus:
"total_skipped": self.total_skipped,
"total_errored": self.total_errored,
"last_tick": self.last_tick.as_dict() if self.last_tick else None,
"consecutive_failures": self.consecutive_failures,
"last_error": self.last_error,
"last_error_at": (
self.last_error_at.isoformat() if self.last_error_at else None
),
"last_sftp_attempt_at": (
self.last_sftp_attempt_at.isoformat()
if self.last_sftp_attempt_at else None
),
}
@@ -136,179 +176,17 @@ class SchedulerStatus:
# Per-file-type handlers. Each returns (parser_name, claim_count) and
# persists its own DB rows. The scheduler records the outcome.
# ---------------------------------------------------------------------------
#
# 999 (Task 2), TA1 (Task 3), 277CA (Task 4), and 835 (Task 5) now
# live in ``cyclone.handlers``. The HANDLERS dict literal below
# references the alias imports so the wiring stays identical.
def _handle_999(text: str, source_file: str) -> tuple[str, int]:
"""Parse a 999, apply rejections, persist ack row. Returns (parser, count)."""
from cyclone.parsers.parse_999 import parse_999_text
from cyclone.parsers.exceptions import CycloneParseError
try:
result = parse_999_text(text, input_file=source_file)
except CycloneParseError as exc:
raise ValueError(f"999 parse error: {exc}") from exc
received, accepted, rejected, ack_code = _ack_count_summary(result)
icn = result.envelope.control_number
synthetic_id = _ack_synthetic_source_batch_id(icn)
with db.SessionLocal()() as session:
def _lookup(pcn: str):
return (
session.query(db.Claim)
.filter_by(patient_control_number=pcn)
.first()
)
rejection_result = apply_999_rejections(
session, result, claim_lookup=_lookup,
)
if rejection_result.matched:
for cid in rejection_result.matched:
append_event(session, AuditEvent(
event_type="claim.rejected",
entity_type="claim",
entity_id=cid,
payload={"source_batch_id": synthetic_id},
actor="999-parser-scheduler",
))
row = cycl_store.add_ack(
source_batch_id=synthetic_id,
accepted_count=accepted,
rejected_count=rejected,
received_count=received,
ack_code=ack_code,
raw_json=json.loads(result.model_dump_json()),
)
session.commit()
return "parse_999", received
def _handle_835(text: str, source_file: str) -> tuple[str, int]:
"""Parse an 835, run validation, persist batch + remittances."""
import uuid
from cyclone.parsers.parse_835 import parse as parse_835
from cyclone.parsers.exceptions import CycloneParseError
from cyclone.parsers.validator_835 import validate as validate_835
from cyclone.payers import PAYER_FACTORIES_835
from cyclone.store import BatchRecord
config = PAYER_FACTORIES_835["co_medicaid_835"]()
try:
result = parse_835(text, config, input_file=source_file)
except CycloneParseError as exc:
raise ValueError(f"835 parse error: {exc}") from exc
# Validation report (mirrors the API endpoint).
report = validate_835(result, config)
n = len(result.claims)
if report.passed:
passed, failed, failed_claim_ids = n, 0, []
else:
passed, failed, failed_claim_ids = 0, n, [
c.payer_claim_control_number for c in result.claims
]
result = result.model_copy(update={
"validation": report,
"summary": result.summary.model_copy(update={
"passed": passed,
"failed": failed,
"failed_claim_ids": failed_claim_ids,
}),
})
rec = BatchRecord(
id=uuid.uuid4().hex,
kind="835",
input_filename=source_file,
parsed_at=datetime.now(timezone.utc),
result=result,
)
cycl_store.add(rec)
return "parse_835", len(result.claims)
def _handle_277ca(text: str, source_file: str) -> tuple[str, int]:
"""Parse a 277CA, persist ack + stamp payer-rejected claims."""
from cyclone.parsers.parse_277ca import parse_277ca_text
from cyclone.parsers.exceptions import CycloneParseError
try:
result = parse_277ca_text(text, input_file=source_file)
except CycloneParseError as exc:
raise ValueError(f"277CA parse error: {exc}") from exc
icn = result.envelope.control_number
synthetic_id = _277ca_synthetic_source_batch_id(icn)
accepted = sum(
1 for s in result.claim_statuses if s.classification == "accepted"
)
paid = sum(
1 for s in result.claim_statuses if s.classification == "paid"
)
rejected = sum(
1 for s in result.claim_statuses if s.classification == "rejected"
)
pended = sum(
1 for s in result.claim_statuses if s.classification == "pended"
)
with db.SessionLocal()() as session:
row = cycl_store.add_277ca_ack(
source_batch_id=synthetic_id,
control_number=icn,
accepted_count=accepted,
rejected_count=rejected,
paid_count=paid,
pended_count=pended,
raw_json=json.loads(result.model_dump_json()),
)
def _lookup(pcn: str):
return (
session.query(db.Claim)
.filter(db.Claim.patient_control_number == pcn)
.first()
)
apply_result = apply_277ca_rejections(
session, result, claim_lookup=_lookup, two77ca_id=row.id,
)
if apply_result.matched:
for cid in apply_result.matched:
append_event(session, AuditEvent(
event_type="claim.payer_rejected",
entity_type="claim",
entity_id=cid,
payload={"source_batch_id": synthetic_id, "277ca_id": row.id},
actor="277ca-parser-scheduler",
))
session.commit()
return "parse_277ca", len(result.claim_statuses)
def _handle_ta1(text: str, source_file: str) -> tuple[str, int]:
"""Parse a TA1, persist the interchange ack row."""
from cyclone.parsers.parse_ta1 import parse_ta1_text
from cyclone.parsers.exceptions import CycloneParseError
try:
result = parse_ta1_text(text, input_file=source_file)
except CycloneParseError as exc:
raise ValueError(f"TA1 parse error: {exc}") from exc
with db.SessionLocal()() as session:
cycl_store.add_ta1_ack(
source_batch_id=result.source_batch_id,
control_number=result.ta1.control_number,
interchange_date=result.ta1.interchange_date,
interchange_time=result.ta1.interchange_time,
ack_code=result.ta1.ack_code,
note_code=result.ta1.note_code,
ack_generated_date=result.ta1.ack_generated_date,
sender_id=result.envelope.sender_id,
receiver_id=result.envelope.receiver_id,
raw_json=json.loads(result.model_dump_json()),
)
session.commit()
return "parse_ta1", 1
# The 277CA handler has moved to ``cyclone.handlers.handle_277ca``
# (SP27 Task 4); the inline def was deleted. The HANDLERS dict literal
# below still references ``_handle_277ca`` because the import-alias
# line at the top of this module binds that name to the new function.
# Only the 835 handler stays inline (Task 5).
# Map file_type → handler. Mirrors ROUTED_FILE_TYPES.
@@ -323,43 +201,15 @@ HANDLERS: dict[str, Callable[[str, str], tuple[str, int]]] = {
# ---------------------------------------------------------------------------
# Light copies of helpers the API endpoints use, so the scheduler can
# run without depending on the FastAPI module.
# ---------------------------------------------------------------------------
def _ack_count_summary(result: Any) -> tuple[int, int, int, str]:
"""Return (received, accepted, rejected, ack_code) for a 999.
Mirrors the logic in ``cyclone.api._ack_count_summary`` but lives
here so the scheduler can run without importing the API module.
"""
if result.functional_group_acks:
fg = result.functional_group_acks[0]
return (
fg.received_count, fg.accepted_count,
fg.rejected_count, fg.ack_code,
)
sets = result.set_responses
received = len(sets)
accepted = sum(1 for s in sets if s.set_accept_reject.code == "A")
rejected = received - accepted
if rejected == 0:
code = "A"
elif accepted == 0:
code = "R"
else:
code = "P"
return (received, accepted, rejected, code)
def _ack_synthetic_source_batch_id(interchange_control_number: str) -> str:
"""Synthetic batches.id for a received 999 with no source batch."""
return f"999-{(interchange_control_number or '').strip() or '000000001'}"
def _277ca_synthetic_source_batch_id(interchange_control_number: str) -> str:
"""Synthetic batches.id for a received 277CA with no source batch."""
return f"277CA-{(interchange_control_number or '').strip() or '000000001'}"
# Run without depending on the FastAPI module.
#
# Note: ``_277ca_synthetic_source_batch_id`` lived here as a
# scheduler-local duplicate of
# ``cyclone.handlers._ack_id.two77ca_synthetic_source_batch_id`` until
# SP27 Task 4 landed; the inline def has been deleted because
# ``handle_277ca`` now imports the canonical copy. The historical
# helper was the only remaining inline def in this section — the
# surviving inline handler is ``_handle_835`` (lifts in Task 5).
# ---------------------------------------------------------------------------
@@ -413,6 +263,40 @@ class Scheduler:
# ticks stack up; the next tick fires only after the previous
# one finishes).
self._tick_in_progress = False
# SP27 Task 9: SFTP-error surfacing. ``_consecutive_failures``
# counts back-to-back tick failures (cleared on the next
# successful tick). The other three fields are the most
# recent failure for the operator's pill — preserved across
# successes so the audit trail doesn't disappear the moment
# the MFT server recovers.
self._consecutive_failures = 0
self._last_error: Optional[str] = None
self._last_error_at: Optional[datetime] = None
self._last_sftp_attempt_at: Optional[datetime] = None
def _record_sftp_outcome(
self, *, success: bool, error: Optional[str] = None,
) -> None:
"""Update the SFTP-error state after a tick.
``success=True`` resets the consecutive-failure counter; the
last-error fields are preserved as an audit trail (the
operator's UI can still surface "last failure: 2 hours ago"
after a recovery).
``success=False`` bumps the counter and records the error
message + timestamp. ``last_sftp_attempt_at`` is bumped in
both cases so the operator can tell when the scheduler last
*tried* to reach the MFT (not just when it last failed).
"""
now = datetime.now(timezone.utc)
self._last_sftp_attempt_at = now
if success:
self._consecutive_failures = 0
return
self._consecutive_failures += 1
self._last_error = error
self._last_error_at = now
# ---- Public API -------------------------------------------------------
@@ -460,6 +344,10 @@ class Scheduler:
total_skipped=self._total_skipped,
total_errored=self._total_errored,
last_tick=self._last_tick,
consecutive_failures=self._consecutive_failures,
last_error=self._last_error,
last_error_at=self._last_error_at,
last_sftp_attempt_at=self._last_sftp_attempt_at,
)
def is_running(self) -> bool:
@@ -473,6 +361,13 @@ class Scheduler:
SFTP server from a stampede when the operator hits
``/api/admin/scheduler/tick`` while a scheduled tick is
already running.
SP27 Task 9: the tick outcome is also recorded on the
scheduler's SFTP-error state via
:meth:`_record_sftp_outcome`. The discriminator is
``TickResult.sftp_failed`` (set by ``_tick_impl``'s
listing-error branches only), not ``TickResult.errors``
which is also populated by per-file processing errors.
"""
while self._tick_in_progress:
await asyncio.sleep(0.05)
@@ -485,6 +380,63 @@ class Scheduler:
self._total_processed += result.files_processed
self._total_skipped += result.files_skipped
self._total_errored += result.files_errored
# SFTP-error surfacing: discriminator is ``result.sftp_failed``,
# NOT ``result.errors``. ``sftp_failed`` is set by
# ``_tick_impl``'s listing-error branches (timeout, connect
# refused). Per-file processing errors append to
# ``result.errors`` but do NOT set ``sftp_failed`` — a
# single malformed 999 in a 5-file batch must not flip
# the operator's pill to destructive.
if result.sftp_failed:
self._record_sftp_outcome(
success=False,
error="; ".join(result.errors),
)
else:
self._record_sftp_outcome(success=True)
return result
finally:
self._tick_in_progress = False
async def process_inbound_files(
self, files: list[InboundFile],
) -> TickResult:
"""Run the per-file processing pipeline over a pre-fetched list.
Unlike :meth:`tick`, this does **not** call SFTP the caller
is expected to have already downloaded (or arranged the local
copy of) each ``InboundFile.local_path``. Used by the
``/api/admin/scheduler/pull-inbound`` admin endpoint and the
``cyclone pull-inbound`` CLI command to process a date-filtered
subset of the inbound MFT path without paying the cost of a
full poll.
Honors ``_stop_event`` between files. Concurrent calls are
coalesced the same way :meth:`tick` does, so a slow handler
can't be stampeded by a parallel admin invocation.
SP27 Task 9: deliberately does NOT update the SFTP-error
state (no listing, no SFTP). The consecutive-failure counter
is the signal for scheduled SFTP polling; operator-initiated
batches shouldn't be able to flip it.
"""
while self._tick_in_progress:
await asyncio.sleep(0.05)
self._tick_in_progress = True
try:
started = datetime.now(timezone.utc)
result = TickResult(started_at=started, files_seen=len(files))
for f in files:
if self._stop_event.is_set():
break
await self._handle_one(f, result)
result.finished_at = datetime.now(timezone.utc)
self._last_tick = result
self._last_poll_at = result.finished_at
self._poll_count += 1
self._total_processed += result.files_processed
self._total_skipped += result.files_skipped
self._total_errored += result.files_errored
return result
finally:
self._tick_in_progress = False
@@ -517,11 +469,26 @@ class Scheduler:
started = datetime.now(timezone.utc)
result = TickResult(started_at=started)
# SP27 Task 8: use the async-wrapped SFTP client so a hung
# ``listdir_attr`` (the 06/25 silent-hang failure mode) is
# bounded by ``CYCLONE_SFTP_OP_TIMEOUT_SECONDS`` instead of
# waiting on paramiko forever. ``asyncio.TimeoutError`` is
# handled explicitly so the tick surfaces a clear error in
# ``Scheduler.status()`` (Task 9) rather than a generic
# ``Exception`` catch-all.
client = self._sftp_client_factory(self._sftp_block)
try:
files = await asyncio.to_thread(self._list_inbound)
files = await client.async_list_inbound()
except asyncio.TimeoutError:
log.exception("SFTP list_inbound timed out")
result.errors.append("list_inbound: timeout")
result.sftp_failed = True
result.finished_at = datetime.now(timezone.utc)
return result
except Exception as exc: # noqa: BLE001
log.exception("SFTP list_inbound failed")
result.errors.append(f"list_inbound: {exc}")
result.sftp_failed = True
result.finished_at = datetime.now(timezone.utc)
return result
@@ -534,11 +501,6 @@ class Scheduler:
result.finished_at = datetime.now(timezone.utc)
return result
def _list_inbound(self) -> list[InboundFile]:
"""Return files in the inbound MFT path. Runs on a thread."""
client = self._sftp_client_factory(self._sftp_block)
return client.list_inbound()
async def _handle_one(self, f: InboundFile, result: TickResult) -> None:
"""Process one inbound file: skip-if-seen, classify, parse, record."""
if await self._already_processed(f.name):
@@ -645,20 +607,21 @@ class Scheduler:
def _download_and_parse(
self, f: InboundFile, file_type: str,
) -> tuple[Path, str, int]:
"""Download from MFT, run the right handler. Returns (path, parser, count).
"""Run the right handler on one inbound file. Returns (path, parser, count).
Stub mode: ``f.local_path`` already points at the staged file
(set by ``SftpClient._list_inbound_stub``). Real mode: the
remote name is ``f.name`` and we round-trip through paramiko.
Both stub and real modes read from ``f.local_path`` the
inbound file is already on disk:
* Stub mode: ``_list_inbound_stub`` points ``local_path`` at
the operator-dropped staging file.
* Real mode: ``_list_inbound_paramiko`` downloads each
``listdir_attr`` entry into the local cache as part of the
listing pass. Re-reading from the MFT would require
``SftpClient.read_file`` with a full remote path, which the
scheduler was passing just ``f.name`` for (i.e. a bare
filename at the SFTP root, not the inbound dir). Use the
cached bytes instead.
"""
if self._sftp_block.stub:
# In stub mode the InboundFile already has a local_path;
# reading the staged bytes directly avoids the stub's
# remote-path semantics (which expect a full inbound path).
content = f.local_path.read_bytes()
else:
client = self._sftp_client_factory(self._sftp_block)
content = client.read_file(f.name)
content = f.local_path.read_bytes()
text = content.decode("utf-8")
handler = HANDLERS[file_type]
parser_used, claim_count = handler(text, f.name)
@@ -717,4 +680,57 @@ def get_scheduler() -> Scheduler:
def reset_scheduler_for_tests() -> None:
"""Clear the module-level scheduler. Test-only."""
global _scheduler
_scheduler = None
_scheduler = None
async def reconfigure_scheduler(
sftp_block: SftpBlock,
*,
poll_interval_seconds: int = 60,
sftp_block_name: str = "default",
) -> Scheduler:
"""Replace the singleton scheduler; restart if it was running. SP25.
Called from ``PATCH /api/clearhouse`` so the next scheduler tick
uses the new SftpBlock without a process restart.
Lifecycle:
* If the previous scheduler is running, ``stop()`` it (waits
up to ``_DRAIN_TIMEOUT_SECONDS`` for the current tick to
finish matches the existing ``Scheduler.stop()`` timeout).
* Replace the module-level ``_scheduler`` singleton with a
fresh ``Scheduler`` against the new block.
* If the previous one was running, ``start()`` the new one.
Threading: same constraint as ``configure_scheduler`` must be
called from the FastAPI event loop, not a worker thread.
"""
global _scheduler
was_running = False
if _scheduler is not None:
was_running = _scheduler.is_running()
if was_running:
# Drain the in-flight tick. Scheduler.stop() already
# applies the _DRAIN_TIMEOUT_SECONDS timeout internally;
# we just await it.
await _scheduler.stop()
poll = int(
os.environ.get("CYCLONE_SCHEDULER_POLL_SECONDS", poll_interval_seconds),
)
_scheduler = Scheduler(
sftp_block,
poll_interval_seconds=poll,
sftp_block_name=sftp_block_name,
)
if was_running:
await _scheduler.start()
log.info(
"Scheduler reconfigured",
extra={
"was_running": was_running,
"sftp_block": sftp_block_name,
"host": sftp_block.host,
"stub": sftp_block.stub,
},
)
return _scheduler
+91 -12
View File
@@ -14,11 +14,36 @@ Setup (one-time, by the operator):
Verification:
security find-generic-password -s cyclone -a sftp.gainwell.password -w
SP25 + SP26: the lookup now resolves the secret in four tiers, in
this order:
1. ``<env_name>_FILE`` env var set highest priority. Reads the
file at that path, strips whitespace, returns the contents.
This is the standard Docker-secrets pattern: mount a file at
``/run/secrets/<name>`` and point the env var at it. An
operator who explicitly sets ``_FILE`` is making a positive
statement about where the secret lives, so a missing file
surfaces as a ``RuntimeError`` rather than silently falling
through. Empty string is treated as unset.
2. Plain env var named exactly ``<env_name>`` second priority.
Lets a Linux server or Docker container pass the MFT password
via ``CYCLONE_SFTP_PASSWORD=...`` without touching the Keychain
or a file mount. Trailing/leading whitespace (including the
``\\n`` that .env files often leave at EOF) is stripped; an
empty value is treated as absent.
3. macOS Keychain via ``keyring`` third priority. Kept as a
fallback so a macOS workstation that prefers
``security add-generic-password`` works without env-var exports.
4. ``None`` caller decides what to do (``SftpClient._connect``
raises a precise ``RuntimeError`` on real-mode auth).
"""
from __future__ import annotations
import logging
import os
from pathlib import Path
from typing import Optional
log = logging.getLogger(__name__)
@@ -36,24 +61,68 @@ except ImportError:
_HAS_KEYRING = False
def _env_var_for(name: str) -> str:
"""Resolve the operator-facing env-var name for a secret.
Returns the entry from ``_ENV_NAME_FOR`` if present, otherwise
returns ``name`` verbatim.
"""
return _ENV_NAME_FOR.get(name, name)
def get_secret(name: str) -> Optional[str]:
"""Fetch a secret from macOS Keychain by name.
"""Fetch a secret by name. Four-tier lookup: _FILE, env var, Keychain, None.
Args:
name: The Keychain account (e.g. "sftp.gainwell.password").
name: The secret name. Matches the Keychain account name
(e.g. ``"sftp.gainwell.password"``). The operator-facing
env-var form is mapped via the ``_ENV_NAME_FOR`` table at
the bottom of this module.
Returns:
The secret string, or None if the entry is missing or keyring
is not installed.
The secret string (stripped), or ``None`` if no tier produced
a value.
Raises:
RuntimeError: if ``<env_name>_FILE`` is set but the file at
that path is missing or unreadable. The operator made a
positive statement about where the secret lives; silent
fall-through would mask a misconfiguration.
"""
if not _HAS_KEYRING:
log.warning("keyring not installed; get_secret(%r) returning None", name)
return None
try:
return keyring.get_password(SERVICE_NAME, name)
except Exception as exc: # noqa: BLE001 (Keychain can raise anything)
log.warning("Keychain get_secret(%r) failed: %s", name, exc)
return None
env_name = _env_var_for(name)
file_env = env_name + "_FILE"
file_path_raw = os.environ.get(file_env)
if file_path_raw:
# An empty string is treated as unset (matches the SP25 rule
# for plain env vars).
stripped_path = file_path_raw.strip()
if stripped_path:
try:
value = Path(stripped_path).read_text().strip()
except OSError as exc:
raise RuntimeError(
f"failed to read {file_env}={stripped_path}: {exc}"
) from exc
if value:
log.debug("Secret %r resolved from file %r", name, stripped_path)
return value
raw = os.environ.get(env_name)
if raw is not None:
stripped = raw.strip()
if stripped:
log.debug("Secret %r resolved from env var %r", name, env_name)
return stripped
# Empty env var — treat as absent and fall through.
if _HAS_KEYRING:
try:
value = keyring.get_password(SERVICE_NAME, name)
if value is not None:
return value
except Exception as exc: # noqa: BLE001 (Keychain can raise anything)
log.warning("Keychain get_secret(%r) failed: %s", name, exc)
return None
def set_secret(name: str, value: str) -> bool:
@@ -77,3 +146,13 @@ def has_keyring() -> bool:
"""True if the ``keyring`` library is importable (regardless of whether
the Keychain entry actually exists)."""
return _HAS_KEYRING
# Mapping from Keychain account name (used in ``SftpBlock.auth``) to
# the operator-facing env var. Keeping this table here means callers
# don't have to remember the difference between
# ``sftp.gainwell.password`` (Keychain account) and
# ``CYCLONE_SFTP_PASSWORD`` (env var).
_ENV_NAME_FOR: dict[str, str] = {
"sftp.gainwell.password": "CYCLONE_SFTP_PASSWORD",
}
+439
View File
@@ -0,0 +1,439 @@
"""CLI subcommand: ``python -m cyclone seed``.
Populates the local DB with a deterministic batch of sample claims
plus matching activity events, so the Dashboard / Claims / Activity
Log pages have something to render in a fresh dev environment.
This is a dev-only convenience the production DB never sees this
command (no ``[env: dev]`` gate, but it's never wired into any
container image). It writes rows with a recognizable batch id prefix
(``SEED-``) so ``--reset`` can clean them up without touching real
ingested data.
Why the data shape is hand-rolled here
--------------------------------------
The ``Claim`` ORM table stores its billing_provider / payer /
subscriber / service_lines payloads in a ``raw_json`` blob that the
read path (``store.to_ui_claim_from_orm``) parses back into the UI
shape. Mirroring the 837 parser's structure keeps the UI rendering
identical to a real ingestion wire format parity, not shortcuts.
Activity rows are similar: ``payload_json`` carries the message
string the Dashboard's activity card shows.
Subcommands
-----------
``python -m cyclone seed`` insert the default batch.
``python -m cyclone seed --count N`` insert N claims (default 96).
``python -m cyclone seed --reset`` wipe previously-seeded rows
first, then insert a fresh batch.
``python -m cyclone seed --status`` print row counts without
inserting anything.
Exit codes
----------
* 0 success (or already seeded and not asked to reset).
* 1 DB error during insert.
* 2 usage error (bad flag value).
The command is idempotent: re-running without ``--reset`` is a no-op
once a seed batch exists. This keeps ``cyclone``-driven boot scripts
safe to re-run.
"""
from __future__ import annotations
import json
import random
import sys
from datetime import datetime, timedelta, timezone
import click
from cyclone.db import ActivityEvent, Batch, Claim, ClaimState, Remittance, SessionLocal
SEED_BATCH_PREFIX = "SEED-"
SEED_CLAIM_PREFIX = "CLM-S"
SEED_REMIT_PREFIX = "REM-S"
SEED_DEFAULT_COUNT = 96
SEED_ACTIVITY_LIMIT = 28
# Mirror src/data/sampleData.ts on the frontend so the Dashboard looks
# the same in dev mode as it did with the in-memory fixtures.
SAMPLE_PROVIDERS = [
{
"npi": "1730187395",
"name": "Cedar Park Family Medicine",
"tax_id": "47-3829104",
"address": "1401 Medical Pkwy",
"city": "Cedar Park",
"state": "TX",
"zip": "78613",
"phone": "(512) 555-0142",
},
{
"npi": "1528471902",
"name": "Lakeside Orthopedics",
"tax_id": "83-1172654",
"address": "900 W Lake Dr",
"city": "Austin",
"state": "TX",
"zip": "78746",
"phone": "(512) 555-0188",
},
{
"npi": "1982036471",
"name": "Hill Country Pediatrics",
"tax_id": "74-5520183",
"address": "205 State Hwy 27",
"city": "Marble Falls",
"state": "TX",
"zip": "78654",
"phone": "(830) 555-0117",
},
]
SAMPLE_PAYERS = [
"Blue Cross Blue Shield",
"United Healthcare",
"Aetna",
"Cigna",
"Humana",
"Medicare",
"Medicaid TX",
]
SAMPLE_CPTS = ["99213", "99214", "99203", "93000", "85025", "80053", "73721", "20610"]
SAMPLE_FIRST_NAMES = [
"Avery", "Jordan", "Riley", "Casey", "Morgan",
"Quinn", "Reese", "Sasha", "Drew", "Hayden",
]
SAMPLE_LAST_NAMES = [
"Nguyen", "Patel", "Garcia", "Cohen", "Okafor",
"Martinez", "Hwang", "Brooks", "Singh", "Tanaka",
]
# Frontend uses "accepted"/"pending" which the backend ClaimState
# enum doesn't carry. Map them to the closest real states so the
# Dashboard's filters (e.g. "status === 'submitted' || 'pending'")
# still find a match for in-flight work.
STATUS_WEIGHTS: list[tuple[ClaimState, int]] = [
(ClaimState.SUBMITTED, 1),
(ClaimState.RECEIVED, 1),
(ClaimState.DENIED, 1),
(ClaimState.PAID, 3),
(ClaimState.PAID, 3),
(ClaimState.PARTIAL, 1),
]
DENIAL_REASONS = [
"CO-97: Service included in another service",
"CO-16: Claim lacks information",
"CO-50: Non-covered service",
"PR-1: Deductible amount",
]
def _now_utc() -> datetime:
return datetime.now(timezone.utc).replace(tzinfo=None)
def _pick(rng: random.Random, items):
return items[rng.randint(0, len(items) - 1)]
def _build_seed_rows(
count: int, *, seed: int = 42,
) -> tuple[Batch, list[Claim], list[ActivityEvent], list[Remittance]]:
"""Build a deterministic Batch + N Claims + matching activity events.
PAID and PARTIAL claims get a paired ``Remittance`` row (with a
realistic ``total_paid`` derived from the billed amount) so the
Dashboard's "Received" KPI lights up; the wire shape mirrors what
``CycloneStore.iter_claims`` expects to find via
``Claim.matched_remittance_id`` ``Remittance.total_paid``.
The seed is fixed so re-running produces identical data keeps
screenshots and dev environments stable.
"""
rng = random.Random(seed)
now = _now_utc()
parsed_at = now - timedelta(minutes=5)
batch = Batch(
id=f"{SEED_BATCH_PREFIX}{now.strftime('%Y%m%d-%H%M%S')}",
kind="837",
input_filename="seed/sample-837.edi",
parsed_at=parsed_at,
totals_json={"claim_count": count},
validation_json={"errors": [], "warnings": []},
raw_result_json={},
)
claims: list[Claim] = []
activity: list[ActivityEvent] = []
remittances: list[Remittance] = []
seq = 10428 # Match the frontend fixture's id range
for i in range(count):
provider = _pick(rng, SAMPLE_PROVIDERS)
payer = _pick(rng, SAMPLE_PAYERS)
cpt = _pick(rng, SAMPLE_CPTS)
first = _pick(rng, SAMPLE_FIRST_NAMES)
last = _pick(rng, SAMPLE_LAST_NAMES)
state, _ = _pick(rng, STATUS_WEIGHTS)
days_back = rng.randint(0, 200)
submitted = now - timedelta(days=days_back, hours=rng.randint(0, 23))
billed = 80 + rng.randint(0, 1400)
if state == ClaimState.PAID:
received = int(billed * (0.6 + rng.random() * 0.4))
elif state == ClaimState.PARTIAL:
received = int(billed * (0.2 + rng.random() * 0.3))
elif state == ClaimState.DENIED:
received = 0
else:
received = 0
claim_id = f"{SEED_CLAIM_PREFIX}{(seq + i):05d}"
denial_reason = _pick(rng, DENIAL_REASONS) if state == ClaimState.DENIED else None
# Build a paired Remittance for any claim with received > 0. Status
# code 1 = "Primary payer forward" — the 835 CAS code that the
# remittance mapper turns into "received". Mirror the 835 parser's
# raw_json shape (a stripped provider/payer/service_lines block)
# so downstream debug views still render something useful.
matched_remit_id: str | None = None
if received > 0:
matched_remit_id = f"{SEED_REMIT_PREFIX}{(seq + i):05d}"
remittances.append(Remittance(
id=matched_remit_id,
batch_id=batch.id,
payer_claim_control_number=f"PCN-{(seq + i):05d}",
claim_id=claim_id,
status_code="1",
status_label="Primary payer forward",
total_charge=float(billed),
total_paid=float(received),
adjustment_amount=float(billed - received),
received_at=submitted + timedelta(days=rng.randint(2, 14)),
raw_json={
"payer": {"name": payer},
"provider": {
"npi": provider["npi"],
"name": provider["name"],
},
"service_lines": [
{
"procedure": {"code": cpt},
"charge_amount": float(billed),
"paid_amount": float(received),
}
],
},
))
raw_json = {
"billing_provider": {
"npi": provider["npi"],
"name": provider["name"],
"tax_id": provider["tax_id"],
# 837 parser produces ``address`` as a structured dict so
# the claim-detail drawer can render line1/line2/city/state/zip.
# A flat string here crashes ``_address_to_ui`` with
# ``AttributeError: 'str' object has no attribute 'get'``.
"address": {
"line1": provider["address"],
"line2": None,
"city": provider["city"],
"state": provider["state"],
"zip": provider["zip"],
},
"phone": provider["phone"],
},
"payer": {"name": payer},
"subscriber": {"first_name": first, "last_name": last},
"service_lines": [
{
"procedure": {"code": cpt},
"charge_amount": float(billed),
"service_date": submitted.date().isoformat(),
}
],
}
claims.append(Claim(
id=claim_id,
batch_id=batch.id,
patient_control_number=claim_id.replace(SEED_CLAIM_PREFIX, "PCN-"),
service_date_from=submitted.date(),
service_date_to=submitted.date(),
charge_amount=billed,
provider_npi=provider["npi"],
payer_id=None,
state=state,
state_changed_at=submitted,
rejection_reason=denial_reason,
resubmit_count=0,
matched_remittance_id=matched_remit_id,
raw_json=raw_json,
))
# First SEED_ACTIVITY_LIMIT claims get an activity event. Mirrors
# the frontend buildActivity() shape (claim_paid / claim_denied /
# claim_accepted / claim_submitted) but maps frontend statuses
# onto the backend's wire enum.
if i < SEED_ACTIVITY_LIMIT:
if state == ClaimState.PAID:
kind = "claim_paid"
verb = "Paid"
elif state == ClaimState.DENIED:
kind = "claim_denied"
verb = "Denied"
elif state in (ClaimState.RECEIVED, ClaimState.PARTIAL):
kind = "claim_accepted"
verb = "Accepted"
else:
kind = "claim_submitted"
verb = "Submitted"
payload = {
"message": f"{verb} {claim_id} · {first} {last}",
"npi": provider["npi"],
"amount": float(billed),
}
activity.append(ActivityEvent(
ts=submitted,
kind=kind,
batch_id=batch.id,
claim_id=claim_id,
remittance_id=None,
payload_json=payload,
))
return batch, claims, activity, remittances
def _existing_seed_batch_ids(s) -> list[str]:
rows = s.query(Batch.id).filter(Batch.id.like(f"{SEED_BATCH_PREFIX}%")).all()
return [r[0] for r in rows]
def _delete_seed_rows(s) -> int:
"""Delete every row that was inserted by the seed.
The seed writes rows whose ids begin with ``SEED-`` (batches),
``CLM-S`` (claims), or ``REM-S`` (remittances). Activity events
are joined to seeded batches when the batch is still around, but
we also clean up orphan activity rows (no FK from
``activity_events.claim_id`` to ``claims.id``) by ``claim_id``
prefix. Returns the number of batch rows deleted.
"""
# Activity first — its rows reference both claims and batches, so
# delete the events tied to seed claims first, then any that were
# only tied to a now-orphaned seed batch.
activity = s.query(ActivityEvent).filter(
ActivityEvent.claim_id.like(f"{SEED_CLAIM_PREFIX}%")
).delete(synchronize_session=False)
activity2 = s.query(ActivityEvent).filter(
ActivityEvent.batch_id.like(f"{SEED_BATCH_PREFIX}%")
).delete(synchronize_session=False)
# Remittances (FK to claims; cascade may not fire without FK pragma,
# so we delete them explicitly to clear the symmetric FK first).
remits = s.query(Remittance).filter(Remittance.id.like(f"{SEED_REMIT_PREFIX}%")).delete(synchronize_session=False)
claims = s.query(Claim).filter(Claim.id.like(f"{SEED_CLAIM_PREFIX}%")).delete(synchronize_session=False)
batches = s.query(Batch).filter(Batch.id.like(f"{SEED_BATCH_PREFIX}%")).delete(synchronize_session=False)
s.commit()
click.echo(f" Removed {batches} seeded batch(es), {claims} claim(s), "
f"{remits} remittance(s), {activity + activity2} activity event(s).")
return batches
def _insert(
batch: Batch,
claims: list[Claim],
activity: list[ActivityEvent],
remittances: list[Remittance],
) -> None:
with SessionLocal()() as s:
s.add(batch)
s.add_all(claims)
s.add_all(activity)
s.add_all(remittances)
s.commit()
def _print_status(s) -> None:
from sqlalchemy import func
seed_batches = s.query(func.count(Batch.id)).filter(Batch.id.like(f"{SEED_BATCH_PREFIX}%")).scalar() or 0
seed_claims = s.query(func.count(Claim.id)).filter(Claim.id.like(f"{SEED_CLAIM_PREFIX}%")).scalar() or 0
seed_remits = s.query(func.count(Remittance.id)).filter(Remittance.id.like(f"{SEED_REMIT_PREFIX}%")).scalar() or 0
total_claims = s.query(func.count(Claim.id)).scalar() or 0
total_remits = s.query(func.count(Remittance.id)).scalar() or 0
total_activity = s.query(func.count(ActivityEvent.id)).scalar() or 0
click.echo(f" Seed batches: {seed_batches}")
click.echo(f" Seed claims: {seed_claims}")
click.echo(f" Seed remits: {seed_remits}")
click.echo(f" Total claims: {total_claims}")
click.echo(f" Total remits: {total_remits}")
click.echo(f" Total activity: {total_activity}")
@click.command("seed")
@click.option(
"--count",
default=SEED_DEFAULT_COUNT,
show_default=True,
type=click.IntRange(min=1, max=10_000),
help="Number of claims to insert in the new batch.",
)
@click.option(
"--reset",
is_flag=True,
help="Delete any existing seeded batch (and its claims/activity) before inserting.",
)
@click.option(
"--status",
"show_status",
is_flag=True,
help="Print current seeded row counts and exit.",
)
def seed_cli(count: int, reset: bool, show_status: bool) -> None:
"""Populate the local DB with a deterministic batch of sample claims."""
with SessionLocal()() as s:
if show_status:
_print_status(s)
return
existing = _existing_seed_batch_ids(s)
if existing and not reset:
click.echo(
f"Seed already present (batch ids: {', '.join(existing)}). "
f"Re-run with --reset to replace, or --status to inspect.",
err=True,
)
sys.exit(0)
if reset and existing:
deleted = _delete_seed_rows(s)
click.echo(f" Removed {deleted} seeded batch(es).")
try:
batch, claims, activity, remittances = _build_seed_rows(count)
except Exception as exc:
click.echo(f"Failed to build seed rows: {exc}", err=True)
sys.exit(1)
try:
_insert(batch, claims, activity, remittances)
except Exception as exc:
click.echo(f"Failed to insert seed rows: {exc}", err=True)
sys.exit(1)
click.echo(f" Inserted batch {batch.id} with {len(claims)} claims, "
f"{len(remittances)} remittances, {len(activity)} activity events.")
_print_status(s)
File diff suppressed because it is too large Load Diff
+383
View File
@@ -0,0 +1,383 @@
"""SQLAlchemy-backed batch store for parsed X12 files.
The package exposes a single ``CycloneStore`` class and a module-level
singleton (``store``). All persistence flows through SQLAlchemy
sessions via ``db.SessionLocal()()`` (the double-paren function-style
accessor).
This ``__init__.py`` is the facade it re-exports every public name
plus the 3 private helpers imported by tests (``_claim_status_from_validation``,
``_persist_835_remit``, ``_remittance_835_row``) from the sibling modules:
exceptions AlreadyMatchedError, NotMatchedError, InvalidStateError
records BatchKind, BatchRecord, BatchRecord837, BatchRecord835
orm_builders ORM row builders + the 3 private-helper re-exports
ui UI serializers (to_ui_*)
write add_record + private event/reconcile helpers
batches Batch reads + _BatchesShim (test-cleanup shim)
claim_detail Single-claim reads + matched-pair drift audit
kpis Dashboard aggregation
acks 999 / TA1 / 277CA ACK persistence
backups Backup-pending marker inserts
inbox Manual match/unmatch + lane listing
providers Provider/payer/clearhouse config
The ``CycloneStore`` class below keeps its current method signatures as
thin delegations for back-compat with existing callers and tests.
Public API (preserved verbatim):
CycloneStore, store, BatchRecord, BatchRecord837, BatchRecord835,
BatchKind, AlreadyMatchedError, NotMatchedError, InvalidStateError,
utcnow, dashboard_kpis, check_matched_pair_drift
Backward-compat shims for tests:
``_lock`` a threading.RLock used by 7 test files for cleanup.
``_batches.clear()`` wipes all rows from the DB tables; the
``_BatchesShim`` class lives in ``batches.py``.
"""
from __future__ import annotations
import threading
from datetime import datetime, timezone
def utcnow() -> datetime:
"""tz-aware UTC `datetime` (replaces the old `utcnow_iso` string helper)."""
return datetime.now(timezone.utc)
from . import write
from .batches import (
_BatchesShim,
_row_to_record,
all_batches,
get_batch,
get_record,
list_batches,
load_two_for_diff,
)
from .claim_detail import (
check_matched_pair_drift,
count_claims,
count_remittances,
distinct_providers,
get_claim_detail,
get_remittance,
iter_claims,
iter_remittances,
recent_activity,
summarize_remittances,
)
from .acks import (
add_277ca_ack,
add_999_ack,
add_ta1_ack,
get_277ca_ack,
get_ack,
get_ta1_ack,
list_277ca_acks,
list_acks,
list_ta1_acks,
)
from .claim_acks import (
add_claim_ack as _add_claim_ack,
batch_envelope_index as _batch_envelope_index,
find_ack_orphans as _find_ack_orphans,
list_acks_for_claim as _list_acks_for_claim,
list_claims_for_ack as _list_claims_for_ack,
remove_claim_ack as _remove_claim_ack,
)
from .backups import add_backup_pending
from .exceptions import AlreadyMatchedError, InvalidStateError, NotMatchedError
from .inbox import list_unmatched, manual_match, manual_unmatch
from .kpis import dashboard_kpis
from .orm_builders import (
_claim_status_from_validation,
_persist_835_remit,
_remittance_835_row,
)
from .providers import (
ensure_clearhouse_seeded,
get_clearhouse,
get_payer_config,
get_provider,
list_payers,
list_providers,
update_clearhouse,
upsert_provider,
)
from .records import BatchKind, BatchRecord, BatchRecord837, BatchRecord835
from .ui import (
to_ui_ack,
to_ui_claim_ack,
to_ui_claim_detail,
to_ui_claim_from_orm,
to_ui_provider,
to_ui_remittance_from_orm,
to_ui_remittance_with_adjustments,
to_ui_ta1_ack,
to_ui_two77ca_ack,
)
# ---------------------------------------------------------------------------
# UI mappers: ORM rows → simpler UI types.
# ---------------------------------------------------------------------------
# Backward-compat shim: tests called ``_batches.clear()`` on the in-memory
# store. The DB-backed store doesn't have an in-memory list, so we expose
# a tiny shim object whose ``.clear()`` wipes the DB. The class itself
# lives in ``batches.py`` (imported above as ``_BatchesShim``); the
# ``CycloneStore.__init__`` below instantiates that imported class.
# ---------------------------------------------------------------------------
# ---------------------------------------------------------------------------
# CycloneStore: the SQLAlchemy-backed facade.
# ---------------------------------------------------------------------------
class CycloneStore:
"""SQLAlchemy-backed facade over the parsed X12 store.
Each public method opens a short-lived session via
``db.SessionLocal()()`` so callers don't have to manage session
lifecycles. Concurrency is handled by the SQLAlchemy engine; the
``_lock`` attribute is a no-op ``RLock`` retained for backward
compatibility with code that wrapped cleanup in a lock context.
"""
def __init__(self) -> None:
self._lock = threading.RLock()
self._batches = _BatchesShim()
# -- write path -----------------------------------------------------
def add(self, record: BatchRecord, *, event_bus=None):
"""Persist a parsed batch (837P or 835)."""
return write.add_record(record, event_bus=event_bus)
def _publish_events_sync(self, event_bus, record, claim_ids, remit_ids):
return write.publish_events_sync(event_bus, record, claim_ids, remit_ids)
@staticmethod
def _sync_publish(event_bus, kind: str, payload: dict) -> None:
return write._sync_publish(event_bus, kind, payload)
# -- read path ------------------------------------------------------
def get_batch(self, batch_id: str) -> dict | None:
return get_batch(batch_id)
def get(self, batch_id: str) -> BatchRecord | None:
return get_record(batch_id)
def list(self, *, limit: int = 100) -> list[BatchRecord]:
return list_batches(limit=limit)
def get_remittance(self, remittance_id: str) -> dict | None:
return get_remittance(remittance_id)
def get_claim_detail(self, claim_id: str) -> dict | None:
return get_claim_detail(claim_id)
def all(self) -> list[BatchRecord]:
return all_batches()
def load_two_for_diff(
self,
a_id: str,
b_id: str,
) -> tuple[BatchRecord, BatchRecord]:
return load_two_for_diff(a_id, b_id)
def iter_claims(self, **kwargs):
return iter_claims(**kwargs)
def iter_remittances(self, **kwargs):
return iter_remittances(**kwargs)
# -- count helpers (SP27 Task 13b) ---------------------------------
#
# The list endpoints (``/api/claims`` and ``/api/remittances``)
# previously computed ``total`` by calling ``iter_*`` with default
# ``limit=100`` and taking ``len(...)``. With 60k+ claims and 800+
# remits in production, the reported total silently capped at 100
# even when the UI rendered a "100" KPI tile and a 100-row table —
# the page looked complete but the population was 600× larger. These
# helpers reuse the iter's filter pipeline with an effectively
# unbounded ``limit`` so the count reflects the true DB population,
# not a 100-row sample. Mirrors Dashboard's "server-aggregated
# counts" fix from commit 59c3275.
def count_claims(
self,
*,
batch_id: str | None = None,
status: str | None = None,
provider_npi: str | None = None,
payer: str | None = None,
date_from: str | None = None,
date_to: str | None = None,
) -> int:
"""Count claims that would be returned by ``iter_claims``."""
return count_claims(
batch_id=batch_id, status=status, provider_npi=provider_npi,
payer=payer, date_from=date_from, date_to=date_to,
)
def count_remittances(
self,
*,
batch_id: str | None = None,
payer: str | None = None,
claim_id: str | None = None,
date_from: str | None = None,
date_to: str | None = None,
) -> int:
"""Count remittances that would be returned by ``iter_remittances``."""
return count_remittances(
batch_id=batch_id, payer=payer, claim_id=claim_id,
date_from=date_from, date_to=date_to,
)
def summarize_remittances(
self,
*,
batch_id: str | None = None,
payer: str | None = None,
claim_id: str | None = None,
date_from: str | None = None,
date_to: str | None = None,
) -> dict:
"""Return ``{count, total_paid, total_adjustments}`` summed over
the remittance population that ``iter_remittances`` would return.
"""
return summarize_remittances(
batch_id=batch_id, payer=payer, claim_id=claim_id,
date_from=date_from, date_to=date_to,
)
def distinct_providers(self) -> list[dict]:
return distinct_providers()
def recent_activity(self, *, limit: int = 200) -> list[dict]:
return recent_activity(limit=limit)
# -- 999 ACKs (SP3 P3 T13) -------------------------------------------
def add_ack(self, **kwargs):
return add_999_ack(**kwargs)
def list_acks(self):
return list_acks()
def get_ack(self, ack_id):
return get_ack(ack_id)
# -- TA1 (Interchange Acknowledgment) -------------------------------
def add_ta1_ack(self, **kwargs):
return add_ta1_ack(**kwargs)
def list_ta1_acks(self):
return list_ta1_acks()
def get_ta1_ack(self, ack_id):
return get_ta1_ack(ack_id)
# -- 277CA (SP10) --------------------------------------------------
def add_277ca_ack(self, **kwargs):
return add_277ca_ack(**kwargs)
def list_277ca_acks(self):
return list_277ca_acks()
def get_277ca_ack(self, ack_id):
return get_277ca_ack(ack_id)
# -- SP28: claim↔ack auto-link join table --------------------------
def add_claim_ack(self, **kwargs):
"""Persist one claim_acks link row.
Mirrors the publish-from-store contract used by the ACK paths:
when ``event_bus`` is passed, the matching ``claim_ack_written``
event fires after commit so live-tail subscribers on both the
claim and the ack side see the row immediately.
"""
return _add_claim_ack(**kwargs)
def list_acks_for_claim(self, claim_id):
"""Return every ClaimAck row for one claim (newest first)."""
return _list_acks_for_claim(claim_id)
def list_claims_for_ack(self, kind, ack_id):
"""Return every ClaimAck row for one ack."""
return _list_claims_for_ack(kind, ack_id)
def find_ack_orphans(self, kind):
"""Return acks with no resolvable link (Inbox ack-orphans lane)."""
return _find_ack_orphans(kind)
def remove_claim_ack(self, link_id, *, event_bus=None):
"""Unlink one row. Publishes ``claim_ack_dropped`` on the bus."""
return _remove_claim_ack(link_id, event_bus=event_bus)
def batch_envelope_index(self):
"""Return a {key: batch.id} map populated from two columns (D10 Pass 1).
Each 837p batch contributes both ``Batch.raw_result_json.envelope.control_number``
(ISA13) and ``Batch.transaction_set_control_number`` (ST02); 835 batches
are excluded. Single ``.get(set_control_number)`` lookup resolves
either key SP37 closes the 999 AK201 source-batch gap.
"""
return _batch_envelope_index()
# -- SP17: encrypted DB backups -------------------------------------
def add_backup_pending(self, *, filename: str, backup_dir: str):
return add_backup_pending(filename=filename, backup_dir=backup_dir)
# -- manual reconciliation (T12) -----------------------------------
def list_unmatched(self, *, kind="both"):
return list_unmatched(kind=kind)
def manual_match(self, claim_id, remit_id):
return manual_match(claim_id, remit_id)
def manual_unmatch(self, claim_id):
return manual_unmatch(claim_id)
# ------------------------------------------------------------------
# SP9: providers / payers / payer_configs / clearhouse
# ------------------------------------------------------------------
def list_providers(self, *, is_active=True):
return list_providers(is_active=is_active)
def get_provider(self, npi):
return get_provider(npi)
def upsert_provider(self, provider):
return upsert_provider(provider)
def list_payers(self, *, is_active=True):
return list_payers(is_active=is_active)
def get_payer_config(self, payer_id, transaction_type):
return get_payer_config(payer_id, transaction_type)
def get_clearhouse(self):
return get_clearhouse()
def update_clearhouse(self, block):
return update_clearhouse(block)
def ensure_clearhouse_seeded(self):
return ensure_clearhouse_seeded()
# Module-level singleton — same import path the old InMemoryStore used.
store = CycloneStore()
+242
View File
@@ -0,0 +1,242 @@
"""ACK persistence — 999 / TA1 / 277CA acknowledgements.
Each ACK type has its own ORM row (Ack / Ta1Ack / Two77caAck). The
write methods are simple inserts; the list/get methods are simple
queries. Fail-soft: persistence errors are logged but do not block
parsing.
SP25: every write path publishes one event on the EventBus so the
Acks page live-tail can subscribe mirrors the existing
``claim_written`` / ``remittance_written`` / ``activity_recorded``
publish pattern (see ``cyclone.store.write``). Publish is best-effort:
a failing subscriber MUST NOT roll back the persisted row.
"""
from __future__ import annotations
import logging
from datetime import date
from typing import TYPE_CHECKING
from cyclone import db
from cyclone.db import Ack
from . import utcnow
from .ui import to_ui_ack, to_ui_ta1_ack, to_ui_two77ca_ack
from .write import _sync_publish
if TYPE_CHECKING:
from cyclone.pubsub import EventBus
log = logging.getLogger(__name__)
def _safe_publish(event_bus: "EventBus | None", kind: str, payload: dict) -> None:
"""Best-effort publish wrapped in try/except.
Mirrors ``CycloneStore._publish_events_sync`` never raises, never
rolls back the persisted row. Falls back to skipping silently when
the bus doesn't implement the private ``_sync_publish`` interface
(e.g. test stubs).
"""
if event_bus is None:
return
try:
_sync_publish(event_bus, kind, payload)
except Exception: # noqa: BLE001
log.exception("acks store: %s publish failed", kind)
# -- 999 ACKs (SP3 P3 T13) -------------------------------------------
def add_999_ack(
*,
source_batch_id: str,
accepted_count: int,
rejected_count: int,
received_count: int,
ack_code: str,
raw_json: dict,
event_bus: "EventBus | None" = None,
) -> db.Ack:
"""Persist a 999 ACK row and return it.
``source_batch_id`` must reference an existing ``batches.id``.
For received 999s with no source batch the caller should pass a
synthetic id (e.g. ``"999-<interchange_control_number>"``)
see ``/api/parse-999`` for that policy.
``raw_json`` is the full ``ParseResult999`` model dump; the
detail endpoint surfaces it without re-parsing the original
X12 text.
SP25: when ``event_bus`` is provided, publishes one
``ack_received`` event (with the full ``to_ui_ack`` row shape)
after the row commits so the Acks page live-tail sees new 999s
the moment they land.
"""
with db.SessionLocal()() as s:
row = Ack(
source_batch_id=source_batch_id,
accepted_count=accepted_count,
rejected_count=rejected_count,
received_count=received_count,
ack_code=ack_code,
parsed_at=utcnow(),
raw_json=raw_json,
)
s.add(row)
s.commit()
s.refresh(row)
row_id = row.id
if event_bus is not None:
with db.SessionLocal()() as s:
payload = to_ui_ack(s.get(Ack, row_id))
_safe_publish(event_bus, "ack_received", payload)
return row
def list_acks() -> list[db.Ack]:
"""Return every 999 ACK row, newest first (auto-increment id desc)."""
with db.SessionLocal()() as s:
return (
s.query(Ack)
.order_by(Ack.id.desc())
.all()
)
def get_ack( ack_id: int) -> db.Ack | None:
"""Return a single ACK row by id, or ``None`` if not found."""
with db.SessionLocal()() as s:
return s.get(Ack, ack_id)
# -- TA1 (Interchange Acknowledgment) -------------------------------
def add_ta1_ack(
*,
source_batch_id: str,
control_number: str,
interchange_date: date | None,
interchange_time: str | None,
ack_code: str,
note_code: str | None,
ack_generated_date: date | None,
sender_id: str,
receiver_id: str,
raw_json: dict,
event_bus: "EventBus | None" = None,
) -> db.Ta1Ack:
"""Persist a TA1 (Interchange Acknowledgment) row and return it.
Mirrors :meth:`add_999_ack` for the lower-level envelope ack. The
flat columns are promoted out of ``raw_json`` so the list
endpoint can sort/filter without a JSON parse; the full
``ParseResultTa1`` stays in ``raw_json`` for the detail endpoint.
SP25: when ``event_bus`` is provided, publishes one
``ta1_ack_received`` event after the row commits.
"""
with db.SessionLocal()() as s:
row = db.Ta1Ack(
source_batch_id=source_batch_id,
control_number=control_number,
interchange_date=interchange_date,
interchange_time=interchange_time,
ack_code=ack_code,
note_code=note_code,
ack_generated_date=ack_generated_date,
sender_id=sender_id,
receiver_id=receiver_id,
parsed_at=utcnow(),
raw_json=raw_json,
)
s.add(row)
s.commit()
s.refresh(row)
row_id = row.id
if event_bus is not None:
with db.SessionLocal()() as s:
payload = to_ui_ta1_ack(s.get(db.Ta1Ack, row_id))
_safe_publish(event_bus, "ta1_ack_received", payload)
return row
def list_ta1_acks() -> list[db.Ta1Ack]:
"""Return every TA1 ACK row, newest first (auto-increment id desc).
Mirrors :meth:`list_acks` the API endpoint slices to its own
``limit`` so the ``total`` field reflects the full row count.
"""
with db.SessionLocal()() as s:
return (
s.query(db.Ta1Ack)
.order_by(db.Ta1Ack.id.desc())
.all()
)
def get_ta1_ack( ack_id: int) -> db.Ta1Ack | None:
"""Return a single TA1 ACK row by id, or ``None`` if not found."""
with db.SessionLocal()() as s:
return s.get(db.Ta1Ack, ack_id)
# -- 277CA (SP10) --------------------------------------------------
def add_277ca_ack(
*,
source_batch_id: str,
control_number: str,
accepted_count: int,
rejected_count: int,
paid_count: int,
pended_count: int,
raw_json: dict,
event_bus: "EventBus | None" = None,
) -> db.Two77caAck:
"""Persist a 277CA (Claim Acknowledgment) row and return it.
Mirrors :meth:`add_999_ack` but for the claim-level ack. The
per-claim status detail stays in ``raw_json``; only the four
counts are promoted so the list endpoint stays fast.
SP25: when ``event_bus`` is provided, publishes one
``two77ca_ack_received`` event after the row commits.
"""
with db.SessionLocal()() as s:
row = db.Two77caAck(
source_batch_id=source_batch_id,
control_number=control_number,
accepted_count=accepted_count,
rejected_count=rejected_count,
paid_count=paid_count,
pended_count=pended_count,
parsed_at=utcnow(),
raw_json=raw_json,
)
s.add(row)
s.commit()
s.refresh(row)
row_id = row.id
if event_bus is not None:
with db.SessionLocal()() as s:
payload = to_ui_two77ca_ack(s.get(db.Two77caAck, row_id))
_safe_publish(event_bus, "two77ca_ack_received", payload)
return row
def list_277ca_acks() -> list[db.Two77caAck]:
"""Return every 277CA ACK row, newest first (auto-increment id desc)."""
with db.SessionLocal()() as s:
return (
s.query(db.Two77caAck)
.order_by(db.Two77caAck.id.desc())
.all()
)
def get_277ca_ack( ack_id: int) -> db.Two77caAck | None:
"""Return a single 277CA ACK row by id, or ``None`` if not found."""
with db.SessionLocal()() as s:
return s.get(db.Two77caAck, ack_id)
+340
View File
@@ -0,0 +1,340 @@
"""SP32 Task 6: backfill rendering/service-provider NPIs from on-disk files.
The T4 writers populate ``Claim.rendering_provider_npi`` (from NM1*82 in
837P Loop 2420A) and ``Remittance.rendering_provider_npi`` (from the
NM1*1P service-provider segment in 835 Loop 2100). Rows ingested before
T4 was wired (or ingested via a path that bypasses the writer e.g. an
ad-hoc ``store.add`` from a notebook) still have a NULL column.
This module re-parses on-disk X12 files and patches up those columns on
matching rows. It is **idempotent**: rows whose
``rendering_provider_npi`` is already non-NULL are left untouched, and
re-running the same file twice is a clean no-op.
Public surface
--------------
* :func:`backfill_rendering_provider_npi` entry point used by the CLI.
* :class:`BackfillSummary` counts dataclass echoed back as a one-line
summary by the CLI.
Reconcile is run once at the end so the new typed NPI arm (T5) can fire
retroactively across the open claim/remit pairs the backfill touched.
"""
from __future__ import annotations
import logging
from dataclasses import dataclass, field
from pathlib import Path
from typing import Iterable
from sqlalchemy import select
from cyclone import db
from cyclone.parsers.parse_835 import parse as parse_835
from cyclone.parsers.parse_837 import parse as parse_837
from cyclone.parsers.payer import PayerConfig, PayerConfig835
log = logging.getLogger(__name__)
# ---------------------------------------------------------------------------
# 837P fixture uses NM1*82 → rendering_provider_npi.
# 835 fixture uses NM1*1P → service_provider_npi (mapped onto the same
# Remittance.rendering_provider_npi column by T4 _remittance_835_row).
# ---------------------------------------------------------------------------
@dataclass
class BackfillSummary:
"""Counts emitted by ``backfill_rendering_provider_npi``.
The CLI echoes a one-line summary (`claims_updated=N remits_updated=N
`) at the end of the run.
"""
claims_updated: int = 0
remits_updated: int = 0
files_processed: int = 0
files_skipped: int = 0
# ---------------------------------------------------------------------------
# Parser wrappers — keep exceptions local so one bad file can't abort
# the whole backfill run.
# ---------------------------------------------------------------------------
def _parse_837_file(path: Path) -> tuple[list, Path | None]:
"""Re-parse a single 837P file. Returns ``(claims, broken_alias)``.
On success, ``claims`` is the list of parsed ``ClaimOutput`` rows
and ``broken_alias`` is None. On any failure, ``claims`` is empty
and ``broken_alias`` is the same ``path`` so the caller can log
which file was skipped.
"""
try:
text = path.read_text()
except OSError as exc:
log.warning("backfill: cannot read %s: %s", path, exc)
return [], path
try:
result = parse_837(text, PayerConfig.co_medicaid(), input_file=str(path))
except Exception as exc: # noqa: BLE001 — failure isolated per-file
log.warning("backfill: 837 parse failed for %s: %s", path, exc)
return [], path
return list(result.claims), None
def _parse_835_file(path: Path) -> tuple[list, Path | None]:
"""Re-parse a single 835 file. Returns ``(claims, broken_alias)``.
``claims`` is the list of parsed ``ClaimPayment`` rows.
"""
try:
text = path.read_text()
except OSError as exc:
log.warning("backfill: cannot read %s: %s", path, exc)
return [], path
try:
result = parse_835(text, PayerConfig835.co_medicaid_835(), input_file=str(path))
except Exception as exc: # noqa: BLE001
log.warning("backfill: 835 parse failed for %s: %s", path, exc)
return [], path
return list(result.claims), None
# ---------------------------------------------------------------------------
# DB-patching helpers — keep the per-row update logic in one place.
# ---------------------------------------------------------------------------
def _patch_claim_rendering_npi(parsed_claims: list) -> int:
"""Update ``Claim.rendering_provider_npi`` for any parsed 837 claim.
Only rows whose column is currently NULL are touched. Returns the
count of rows updated.
"""
from cyclone.db import Claim, SessionLocal
updated = 0
with SessionLocal()() as session:
for parsed in parsed_claims:
npi = getattr(parsed, "rendering_provider_npi", None)
if not npi:
continue
row = session.get(Claim, parsed.claim_id)
if row is None:
continue
if row.rendering_provider_npi is not None:
# Already populated (e.g. by a prior backfill run, or by
# the T4 writer if the claim was ingested afterwards).
continue
row.rendering_provider_npi = npi
updated += 1
if updated:
session.commit()
return updated
def _patch_remit_rendering_npi(parsed_claims: list) -> int:
"""Update ``Remittance.rendering_provider_npi`` for any parsed 835 claim.
The 835 NM1*1P segment maps onto ``ClaimPayment.service_provider_npi``
(which the T4 writer copies into ``Remittance.rendering_provider_npi``).
We match on ``payer_claim_control_number``; only NULL columns are
touched. Returns the count of rows updated.
"""
from cyclone.db import Remittance, SessionLocal
updated = 0
with SessionLocal()() as session:
for parsed in parsed_claims:
npi = getattr(parsed, "service_provider_npi", None)
if not npi:
continue
target_pcn = getattr(parsed, "payer_claim_control_number", None)
if not target_pcn:
continue
row = session.execute(
select(Remittance).where(
Remittance.payer_claim_control_number == target_pcn,
Remittance.rendering_provider_npi.is_(None),
)
).scalars().first()
if row is None:
continue
row.rendering_provider_npi = npi
updated += 1
if updated:
session.commit()
return updated
# ---------------------------------------------------------------------------
# Reconcile pass — run once across every 835 batch so the T5 scoring arm
# can fire retroactively on pairs the backfill touched.
# ---------------------------------------------------------------------------
def _run_reconcile_sweep() -> int:
"""Run :func:`cyclone.reconcile.run` over every 835 batch. Returns count.
Failures are isolated per-batch so one bad batch can't abort the
sweep the goal is "best-effort retroactive reconcile".
"""
from sqlalchemy import select as _select
from cyclone import reconcile as _reconcile
from cyclone.db import Batch, SessionLocal
completed = 0
with SessionLocal()() as session:
batch_ids = [
row[0] for row in session.execute(
_select(Batch.id).where(Batch.kind == "835")
).all()
]
# Each batch gets its own session — ``reconcile.run`` does not commit,
# so an exception in one batch must not orphan a half-flushed
# transaction.
for bid in batch_ids:
try:
with SessionLocal()() as s:
_reconcile.run(s, bid)
s.commit()
completed += 1
except Exception as exc: # noqa: BLE001
log.warning("backfill: reconcile failed for batch %s: %s", bid, exc)
return completed
# ---------------------------------------------------------------------------
# Public entry point.
# ---------------------------------------------------------------------------
# Supported ``--type`` flag values; also ``None`` means auto-detect.
TransactionType = str | None
def backfill_rendering_provider_npi(
*,
files: Iterable[Path] | None = None,
input_dir: Path | None = None,
transaction_type: TransactionType = None,
) -> BackfillSummary:
"""Re-parse on-disk 837p + 835 files and populate the typed NPI columns.
Args:
files: explicit list of file paths to re-parse.
input_dir: directory to scan for ``*.txt`` / ``*.edi`` files.
Files are scanned one level deep.
transaction_type: ``"837p"`` or ``"835"``. ``None`` auto-detects
by attempting the 837p parser first and falling back to
the 835 parser.
Returns:
:class:`BackfillSummary` with populated counts.
Idempotent: only writes columns that are currently NULL; re-running
on the same files is a clean no-op. After patching, runs
:func:`cyclone.reconcile.run` over every 835 batch so the T5
scoring arm can re-fire on the touched pairs.
"""
summary = BackfillSummary()
# 1. Resolve the candidate file set.
candidates: list[Path] = []
if files:
candidates.extend(Path(f) for f in files)
if input_dir is not None:
for ext in ("*.txt", "*.edi", "*.835", "*.837", "*.x12"):
candidates.extend(input_dir.glob(ext))
candidates = [p for p in candidates if p.is_file()]
# 2. Re-parse each file and patch matching rows.
for path in candidates:
kind = transaction_type or _sniff_kind(path)
if kind == "837p":
parsed_claims, broken = _parse_837_file(path)
if broken is not None:
summary.files_skipped += 1
continue
summary.files_processed += 1
summary.claims_updated += _patch_claim_rendering_npi(parsed_claims)
elif kind == "835":
parsed_claims, broken = _parse_835_file(path)
if broken is not None:
summary.files_skipped += 1
continue
summary.files_processed += 1
summary.remits_updated += _patch_remit_rendering_npi(parsed_claims)
else:
# Auto-detect tried both parsers and both failed → skip.
summary.files_skipped += 1
# 3. Reconcile sweep — let the T5 NPI arm fire retroactively.
if summary.claims_updated or summary.remits_updated:
try:
_run_reconcile_sweep()
except Exception: # noqa: BLE001 — sweep is best-effort
log.exception("backfill: reconcile sweep failed")
return summary
def _sniff_kind(path: Path) -> TransactionType:
"""Best-effort transaction-type sniff (content + filename).
Returns ``"837p"``, ``"835"``, or ``None`` if both parsers fail.
Used only when the caller didn't pin ``transaction_type`` explicitly
via the CLI.
"""
try:
text = path.read_text()
except OSError:
return None
# Cheap filename hint.
name = path.name.lower()
if name.endswith(".835") or "835" in name:
return _try_both(path, text, prefer="835")
if name.endswith(".837") or "837" in name or "837p" in name:
return _try_both(path, text, prefer="837p")
# Content: ISA + ST. 837 starts with "ST*837", 835 starts with "ST*835".
if "ST*837*" in text:
return "837p"
if "ST*835*" in text:
return "835"
# Fallback: try 837p parser, then 835.
return _try_both(path, text, prefer="837p")
def _try_both(path: Path, text: str, *, prefer: str) -> TransactionType:
"""Try the preferred parser first, then the other; return first winner."""
if prefer == "837p":
try:
parse_837(text, PayerConfig.co_medicaid(), input_file=str(path))
return "837p"
except Exception:
try:
parse_835(text, PayerConfig835.co_medicaid_835(), input_file=str(path))
return "835"
except Exception:
return None
try:
parse_835(text, PayerConfig835.co_medicaid_835(), input_file=str(path))
return "835"
except Exception:
try:
parse_837(text, PayerConfig.co_medicaid(), input_file=str(path))
return "837p"
except Exception:
return None
__all__ = [
"BackfillSummary",
"backfill_rendering_provider_npi",
]
+39
View File
@@ -0,0 +1,39 @@
"""Backup-pending marker inserts — SP17 encrypted DB backups.
``add_backup_pending()`` inserts a ``pending`` row for a backup that is
about to start; the BackupService updates ``status`` / ``size_bytes``
/ ``db_fingerprint`` / ``table_count`` / ``completed_at`` after the
encrypted blob lands on disk.
"""
from cyclone import db
from . import utcnow
# -- SP17: encrypted DB backups -------------------------------------
def add_backup_pending(*, filename: str, backup_dir: str) -> db.DbBackup:
"""Insert a ``pending`` row for a backup that is about to start.
The BackupService fills in ``status`` / ``size_bytes`` /
``db_fingerprint`` / ``table_count`` / ``completed_at`` after
the encrypted blob lands on disk.
"""
with db.SessionLocal()() as s:
row = db.DbBackup(
filename=filename,
backup_dir=backup_dir,
size_bytes=0,
db_fingerprint=None,
table_count=0,
created_at=utcnow(),
completed_at=None,
status="pending",
error_message=None,
)
s.add(row)
s.commit()
s.refresh(row)
return row
+155
View File
@@ -0,0 +1,155 @@
"""Batch read APIs and the legacy _BatchesShim.
The ``_BatchesShim`` class is retained for back-compat with the
``with store._lock: store._batches.clear()`` test-cleanup idiom used
in 7 test files. Its ``clear()`` method wipes all rows from the DB
tables so tests that depended on a fresh in-memory list per-test get
a fresh DB state per-test.
"""
from __future__ import annotations
from datetime import timezone
from cyclone import db
from cyclone.db import (
ActivityEvent,
Batch,
CasAdjustment,
Claim,
Match,
Remittance,
)
from cyclone.parsers.models import ParseResult
from cyclone.parsers.models_835 import ParseResult835
from .records import BatchRecord, BatchRecord837, BatchRecord835
class _BatchesShim:
"""Drop-in replacement for the old in-memory ``_batches`` list.
``clear()`` removes every row from the DB tables in FK-safe order.
Other list operations are not implemented because the only call site
is the ``clear()`` inside the test fixtures (``test_api_gets.py`` and
``test_api_parse_persists.py``).
"""
def clear(self) -> None: # type: ignore[no-untyped-def]
with db.SessionLocal()() as s:
s.query(ActivityEvent).delete()
s.query(Match).delete()
s.query(CasAdjustment).delete()
s.query(Remittance).delete()
s.query(Claim).delete()
s.query(Batch).delete()
s.commit()
def _row_to_record(row: Batch) -> BatchRecord:
"""Rehydrate a ``BatchRecord`` (837 or 835) from a Batch ORM row.
The full ``ParseResult`` / ``ParseResult835`` lives in
``raw_result_json`` (stashed at insert time). Re-parsing JSON
here means callers get the same typed Pydantic object the old
in-memory store handed out, so api.py and tests that do
``rec.result.claims`` keep working unchanged.
SQLite drops tz info on round-trip even though the column type
is ``DateTime(timezone=True)``. We re-attach UTC so the
``BatchRecord`` validator (``parsed_at must be tz-aware``)
passes.
"""
if row.kind == "835":
result_cls = ParseResult835
else:
result_cls = ParseResult
payload = row.raw_result_json or {}
result = result_cls.model_validate(payload)
parsed_at = row.parsed_at
if parsed_at is not None and parsed_at.tzinfo is None:
parsed_at = parsed_at.replace(tzinfo=timezone.utc)
record_cls = BatchRecord835 if row.kind == "835" else BatchRecord837
return record_cls(
id=row.id,
kind=row.kind,
input_filename=row.input_filename,
parsed_at=parsed_at,
result=result,
)
def get_batch(batch_id: str) -> dict | None:
"""Return a summary dict for ``batch_id`` or ``None`` if missing.
The dict shape matches what ``/api/batches/{id}`` callers need:
``id``, ``kind``, ``input_filename``, ``parsed_at``, and the
full ``result`` (raw_result_json) as a dict.
"""
with db.SessionLocal()() as s:
row = s.get(Batch, batch_id)
if row is None:
return None
return {
"id": row.id,
"kind": row.kind,
"input_filename": row.input_filename,
"parsed_at": row.parsed_at,
"result": row.raw_result_json,
}
def get_record(batch_id: str) -> BatchRecord | None:
"""Return the ``BatchRecord`` for ``batch_id`` or ``None``.
Preserves the in-memory store contract: callers get a Pydantic
``BatchRecord`` (subclass ``BatchRecord837`` / ``BatchRecord835``)
with ``.id``, ``.kind``, ``.input_filename``, ``.parsed_at``,
and ``.result`` (typed ``ParseResult`` / ``ParseResult835``).
"""
with db.SessionLocal()() as s:
row = s.get(Batch, batch_id)
if row is None:
return None
return _row_to_record(row)
def list_batches(*, limit: int = 100) -> list[BatchRecord]:
"""Return up to ``limit`` ``BatchRecord``s, newest first."""
with db.SessionLocal()() as s:
rows = (
s.query(Batch)
.order_by(Batch.parsed_at.desc())
.limit(limit)
.all()
)
return [_row_to_record(r) for r in rows]
def all_batches() -> list[BatchRecord]:
"""Return every ``BatchRecord``, oldest first (no pagination)."""
with db.SessionLocal()() as s:
rows = s.query(Batch).order_by(Batch.parsed_at.asc()).all()
return [_row_to_record(r) for r in rows]
def load_two_for_diff(a_id: str, b_id: str) -> tuple[BatchRecord, BatchRecord]:
"""Load two batches by id for the side-by-side diff view.
Returns ``(a, b)`` as ``BatchRecord`` objects. Raises
:class:`LookupError` when either id is missing the API layer
catches it and maps it to ``404 Not Found`` (matching the
``GET /api/batches/{id}`` contract). The two loads happen in
independent sessions so a transient failure on one side can't
poison the other.
Used exclusively by :mod:`cyclone.batch_diff` via the
``/api/batch-diff`` endpoint.
"""
a = get_record(a_id)
if a is None:
raise LookupError(f"batch {a_id} not found")
b = get_record(b_id)
if b is None:
raise LookupError(f"batch {b_id} not found")
return a, b
+379
View File
@@ -0,0 +1,379 @@
"""SP28: claim_acks persistence + batch envelope index (D10).
Five methods on top of the new ``ClaimAck`` ORM table:
* ``add_claim_ack`` insert one link row (manual or auto) and
publish ``claim_ack_written`` on the bus.
* ``list_acks_for_claim`` every link row for one claim (per-claim
only; TA1 batch-level rows are filtered out by the API layer).
* ``list_claims_for_ack`` every link row for one ack.
* ``find_ack_orphans`` acks with no resolvable claim (Inbox
ack-orphans lane).
* ``remove_claim_ack`` unlink. Publishes ``claim_ack_dropped``.
* ``batch_envelope_index`` D10 in-memory map of
``{Batch.raw_result_json.envelope.control_number (ISA13) OR
Batch.transaction_set_control_number (ST02)} batch.id``
(SP37: populated from two columns; cheap to rebuild, re-built
once per ingest).
Each mutating method opens its own ``db.SessionLocal()()`` session
so callers don't have to manage session lifecycles. Publishes are
best-effort (wrapped in :func:`_safe_publish`) so a failing bus
subscriber cannot roll back the persisted row.
"""
from __future__ import annotations
import logging
from datetime import datetime, timezone
from typing import TYPE_CHECKING
from cyclone import db
from cyclone.db import (
Ack,
Batch,
Claim,
ClaimAck,
Ta1Ack,
Two77caAck,
)
from . import utcnow
from .ui import to_ui_claim_ack
from .write import _sync_publish
if TYPE_CHECKING:
from cyclone.pubsub import EventBus
log = logging.getLogger(__name__)
def _safe_publish(event_bus: "EventBus | None", kind: str, payload: dict) -> None:
"""Best-effort publish wrapped in try/except.
Mirrors ``cyclone.store.acks._safe_publish`` never raises,
never rolls back the persisted row.
"""
if event_bus is None:
return
try:
_sync_publish(event_bus, kind, payload)
except Exception: # noqa: BLE001
log.exception("claim_acks store: %s publish failed", kind)
# ---------------------------------------------------------------------------
# D10 batch envelope index
# ---------------------------------------------------------------------------
def batch_envelope_index() -> dict[str, str]:
"""Build a ``{key: batch.id}`` map populated from two columns.
D10 primary join key (spec §D10). Each 837p batch row contributes
up to two entries:
* ``Batch.raw_result_json.envelope.control_number`` (ISA13)
* ``Batch.transaction_set_control_number`` (ST02, new in SP37)
Either key resolves to the same ``batch.id``; callers do a single
``idx.get(set_control_number)`` lookup. Pass 2 (PCN) is unchanged.
Built once per ingest from every Batch row whose kind is "837p";
cost is O(N batches), currently small, so trivial. 835 batches
are excluded the column is an 837P-specific join key (see
``Batch.transaction_set_control_number`` docstring).
"""
idx: dict[str, str] = {}
with db.SessionLocal()() as s:
rows = (
s.query(
Batch.id,
Batch.raw_result_json,
Batch.transaction_set_control_number,
)
.filter(Batch.kind == "837p")
.all()
)
for bid, raw, stcn in rows:
env = (raw or {}).get("envelope") or {}
isa_cn = env.get("control_number")
if isinstance(isa_cn, str) and isa_cn:
# First write wins (setdefault) — if ISA13 and ST02
# ever collide they map to the same batch anyway.
idx.setdefault(isa_cn, bid)
if isinstance(stcn, str) and stcn:
idx.setdefault(stcn, bid)
return idx
# ---------------------------------------------------------------------------
# Mutators
# ---------------------------------------------------------------------------
def add_claim_ack(
*,
claim_id: str | None,
batch_id: str | None,
ack_id: int,
ack_kind: str,
ak2_index: int | None = None,
set_control_number: str | None = None,
set_accept_reject_code: str | None = None,
linked_by: str = "auto",
event_bus: "EventBus | None" = None,
now: datetime | None = None,
) -> ClaimAck:
"""Persist one link row and return it.
The DB-level unique index
``ux_claim_acks_dedup(claim_id, ack_kind, ack_id, ak2_index)
WHERE claim_id IS NOT NULL AND ak2_index IS NOT NULL`` enforces
idempotency for the per-AK2 case. The 277CA (ak2_index NULL)
and TA1 (claim_id NULL) paths are deduplicated by application
code see :func:`cyclone.claim_acks._link_exists`.
When ``event_bus`` is provided, publishes one ``claim_ack_written``
event (with the full :func:`cyclone.store.ui.to_ui_claim_ack`
payload) so the live-tail subscribers on both
``/api/claims/{id}/acks/stream`` and
``/api/acks/{kind}/{id}/claims/stream`` see new rows the moment
they land.
"""
if ack_kind not in ("999", "277ca", "ta1"):
raise ValueError(f"add_claim_ack: unknown ack_kind={ack_kind!r}")
if linked_by not in ("auto", "manual"):
raise ValueError(f"add_claim_ack: linked_by={linked_by!r}")
if claim_id is None and batch_id is None:
raise ValueError(
"add_claim_ack: at least one of claim_id/batch_id must be set"
)
ts = now or utcnow()
with db.SessionLocal()() as s:
row = ClaimAck(
claim_id=claim_id,
batch_id=batch_id,
ack_id=ack_id,
ack_kind=ack_kind,
ak2_index=ak2_index,
set_control_number=set_control_number,
set_accept_reject_code=set_accept_reject_code,
linked_at=ts,
linked_by=linked_by,
)
s.add(row)
s.commit()
s.refresh(row)
row_id = row.id
if event_bus is not None:
with db.SessionLocal()() as s:
payload = to_ui_claim_ack(s.get(ClaimAck, row_id))
_safe_publish(event_bus, "claim_ack_written", payload)
return row
def remove_claim_ack(
link_id: int,
*,
event_bus: "EventBus | None" = None,
) -> bool:
"""Unlink. Returns ``True`` when a row was deleted.
Publishes ``claim_ack_dropped`` with ``{"id", "claim_id"}`` so
the live-tail subscribers can remove the link from their local
store. Does NOT touch ``Claim.state`` the unlink is purely
about the link row, per spec §D6.
"""
with db.SessionLocal()() as s:
row = s.get(ClaimAck, link_id)
if row is None:
return False
payload = {
"id": row.id,
"claim_id": row.claim_id,
"ack_id": row.ack_id,
"ack_kind": row.ack_kind,
}
s.delete(row)
s.commit()
if event_bus is not None:
_safe_publish(event_bus, "claim_ack_dropped", payload)
return True
# ---------------------------------------------------------------------------
# Readers
# ---------------------------------------------------------------------------
def list_acks_for_claim(claim_id: str) -> list[ClaimAck]:
"""Return every ClaimAck row for one claim (newest first)."""
with db.SessionLocal()() as s:
return (
s.query(ClaimAck)
.filter(ClaimAck.claim_id == claim_id)
.order_by(ClaimAck.id.desc())
.all()
)
def list_claims_for_ack(kind: str, ack_id: int) -> list[ClaimAck]:
"""Return every ClaimAck row for one ack (any kind).
For 999 / 277CA: returns 0..N rows (one per AK2 / ClaimStatus).
For TA1: returns 0..1 row (envelope-level, populated batch_id).
"""
if kind not in ("999", "277ca", "ta1"):
raise ValueError(f"list_claims_for_ack: unknown kind={kind!r}")
with db.SessionLocal()() as s:
return (
s.query(ClaimAck)
.filter(
ClaimAck.ack_kind == kind,
ClaimAck.ack_id == ack_id,
)
.order_by(ClaimAck.id.asc())
.all()
)
# ---------------------------------------------------------------------------
# Orphan detection (Inbox "Ack orphans" lane — spec §D7)
# ---------------------------------------------------------------------------
def find_ack_orphans(kind: str) -> list[dict]:
"""Return acks with no resolvable Claim row of their own kind.
Used by the Inbox ack-orphans lane for the operator's manual
reconciliation flow. The downstream ``/api/inbox/ack-orphans``
endpoint calls this and returns the rendered shape.
"Orphan" means: for 999 / 277CA, ``ack_kind=kind AND ack_id IN
(acks_with_no_claim_acks_link)``. For TA1, "orphan" means the
TA1 row exists but no Batch with matching sender/receiver was
resolved (so no link row was created).
Output dict shape (one row per orphan ack, rendered by
:func:`to_ui_claim_ack`-style serialization):
* ``kind`` "999" / "277ca" / "ta1"
* ``ack_id`` the ack row's id
* ``control_number`` for 999/277CA, envelope.control_number;
for TA1, ta1.control_number
* ``set_control_numbers`` empty list when no claims match
* ``raw_summary`` flat copy of the ack's UI shape for the
lane-header counts
"""
if kind not in ("999", "277ca", "ta1"):
raise ValueError(f"find_ack_orphans: unknown kind={kind!r}")
out: list[dict] = []
if kind == "999":
ack_table = Ack
ctrl_attr = None
elif kind == "277ca":
ack_table = Two77caAck
ctrl_attr = "control_number"
else:
ack_table = Ta1Ack
ctrl_attr = "control_number"
with db.SessionLocal()() as s:
# Every ack row of the given kind, with a LEFT JOIN against
# any claim_acks link; orphan when NO link was created.
if kind == "999":
# For 999, the "ack has no link" means no ClaimAck row
# was emitted at all (the auto-linker emits one per AK2
# even when the AK2 is rejected, so 999 with at least
# one AK2 that resolved to a claim is never an orphan).
# We treat a 999 as orphan when it has zero ClaimAck
# rows tied to its id.
all_acks = s.query(Ack).order_by(Ack.id.desc()).all()
for ack_row in all_acks:
count = (
s.query(ClaimAck)
.filter(ClaimAck.ack_kind == "999",
ClaimAck.ack_id == ack_row.id)
.count()
)
if count == 0:
out.append({
"kind": "999",
"ack_id": ack_row.id,
"control_number": _ack_control_number(ack_row, "999"),
"parsed_at": (
ack_row.parsed_at.isoformat().replace(
"+00:00", "Z"
) if ack_row.parsed_at else None
),
})
elif kind == "277ca":
all_acks = s.query(Two77caAck).order_by(Two77caAck.id.desc()).all()
for ack_row in all_acks:
count = (
s.query(ClaimAck)
.filter(ClaimAck.ack_kind == "277ca",
ClaimAck.ack_id == ack_row.id)
.count()
)
if count == 0:
out.append({
"kind": "277ca",
"ack_id": ack_row.id,
"control_number": ack_row.control_number or "",
"parsed_at": (
ack_row.parsed_at.isoformat().replace(
"+00:00", "Z"
) if ack_row.parsed_at else None
),
})
else:
all_acks = s.query(Ta1Ack).order_by(Ta1Ack.id.desc()).all()
for ack_row in all_acks:
count = (
s.query(ClaimAck)
.filter(ClaimAck.ack_kind == "ta1",
ClaimAck.ack_id == ack_row.id)
.count()
)
if count == 0:
out.append({
"kind": "ta1",
"ack_id": ack_row.id,
"control_number": ack_row.control_number or "",
"parsed_at": (
ack_row.parsed_at.isoformat().replace(
"+00:00", "Z"
) if ack_row.parsed_at else None
),
})
return out
def _ack_control_number(ack_row: Ack, kind: str) -> str:
"""Best-effort control-number lookup for a 999 ack row.
The 999 ORM row doesn't carry the envelope's control_number in
a dedicated column; we re-derive it from ``raw_json`` (the same
source :func:`cyclone.store.ui.to_ui_ack` uses for the patient
control number).
"""
raw = ack_row.raw_json or {}
env = raw.get("envelope") or {}
return env.get("control_number") or ""
__all__ = [
"add_claim_ack",
"batch_envelope_index",
"find_ack_orphans",
"list_acks_for_claim",
"list_claims_for_ack",
"remove_claim_ack",
]
+710
View File
@@ -0,0 +1,710 @@
"""Claim- and remittance-detail read APIs.
Includes the only large non-write query in the store:
``get_claim_detail`` which joins Claim + CasAdjustment + ActivityEvent
history for the right-drawer UI.
Also hosts ``check_matched_pair_drift()`` the SP27 startup invariant
audit that returns the count of ClaimRemit pairs where
``matched_remittance_id`` doesn't agree with the FK in ``remittances.claim_id``.
It lives here (not in its own module) because it reads the same pair of
tables as ``get_claim_detail``'s matched-remittance summary.
"""
from __future__ import annotations
from datetime import timezone
from decimal import Decimal
from cyclone import db
from cyclone.db import (
ActivityEvent,
CasAdjustment,
Claim,
ClaimState,
Remittance,
)
from .ui import (
CLAIM_DETAIL_HISTORY_LIMIT,
_date_in_bounds,
_iso_z,
_svc_to_wire_dict,
to_ui_claim_detail,
to_ui_claim_from_orm,
to_ui_provider,
to_ui_remittance_from_orm,
to_ui_remittance_with_adjustments,
)
def get_remittance(remittance_id: str) -> dict | None:
"""Return a UI-shaped remittance dict with ``adjustments`` array.
Joins the persisted ``CasAdjustment`` rows for ``remittance_id``
and labels each via :mod:`cyclone.parsers.cas_codes`. Returns
``None`` when the remittance is not found so the API layer can
map that to a 404.
SP7: also returns the per-line SVC composites
(``serviceLinePayments``) and the CLP-level (claim-level) CAS
bucket (``claimLevelAdjustments``) so the remit drawer can show
per-line payments + adjustments without a second fetch.
"""
with db.SessionLocal()() as s:
row = s.get(Remittance, remittance_id)
if row is None:
return None
cas_rows = (
s.query(CasAdjustment)
.filter(CasAdjustment.remittance_id == remittance_id)
.all()
)
parsed_at = (
row.batch.parsed_at if row.batch is not None else row.received_at
)
if parsed_at is not None and parsed_at.tzinfo is None:
parsed_at = parsed_at.replace(tzinfo=timezone.utc)
body = to_ui_remittance_with_adjustments(
row,
batch_id=row.batch_id,
parsed_at=parsed_at,
cas_rows=cas_rows,
)
# SP7: per-line SVC composites + claim-level CAS bucket.
from cyclone.db import ServiceLinePayment as SLP
slps = (
s.query(SLP)
.filter(SLP.remittance_id == remittance_id)
.order_by(SLP.line_number)
.all()
)
body["serviceLinePayments"] = [
_svc_to_wire_dict(svc) for svc in slps
]
body["claimLevelAdjustments"] = [
{
"id": c.id,
"group_code": c.group_code,
"reason_code": c.reason_code,
"amount": str(Decimal(str(c.amount))),
"quantity": (
str(Decimal(str(c.quantity)))
if c.quantity is not None
else None
),
}
for c in cas_rows
if c.service_line_payment_id is None
]
return body
def get_claim_detail(claim_id: str) -> dict | None:
"""Return the SP4 detail-drawer shape for one claim, or ``None``.
Drives ``GET /api/claims/{claim_id}``. Returns the spec-shaped
dict from :func:`to_ui_claim_detail` (header + state + parties +
validation + service lines + diagnoses + raw segments) stitched
with the claim's recent activity history and, if paired, a
matched-remittance summary.
Returns ``None`` when ``claim_id`` is not in the DB so the API
layer can map that to a 404 the URL-driven drawer
distinguishes "claim doesn't exist" from "fetch failed" (the
spec §3.4 calls for a distinct 404 state in the drawer).
The history is capped at :data:`CLAIM_DETAIL_HISTORY_LIMIT`
(50, per the spec) and ordered ``ts DESC`` so the most recent
event is first. The status string in ``matchedRemittance``
follows the same ``reconciled``/``received`` mapping used by
:func:`to_ui_remittance_from_orm`.
"""
# Lazy import — same pattern used throughout this module to
# avoid a circular store ↔ db import on cold start.
from cyclone import db as _db
with _db.SessionLocal()() as s:
row = s.get(Claim, claim_id)
if row is None:
return None
history_rows = (
s.query(ActivityEvent)
.filter(ActivityEvent.claim_id == claim_id)
.order_by(ActivityEvent.ts.desc())
.limit(CLAIM_DETAIL_HISTORY_LIMIT)
.all()
)
# Claim.batch_id is FK NOT NULL with ON DELETE CASCADE, so
# ``row.batch`` is always populated in normal flow. Re-attach
# UTC only when SQLite drops the tzinfo on read.
parsed_at = row.batch.parsed_at
if parsed_at.tzinfo is None:
parsed_at = parsed_at.replace(tzinfo=timezone.utc)
detail = to_ui_claim_detail(
row,
batch_id=row.batch_id,
parsed_at=parsed_at,
)
detail["stateHistory"] = [
{
"kind": ev.kind,
# SQLite drops tzinfo on read; rows are stored UTC
# at write time (see ``add`` / ``manual_match``),
# so re-attach UTC if needed to keep the spec
# contract that ``ts`` ends in Z.
"ts": _iso_z(ev.ts),
"batchId": ev.batch_id,
"remittanceId": ev.remittance_id,
}
for ev in history_rows
]
if row.matched_remittance_id is not None:
remit = s.get(Remittance, row.matched_remittance_id)
if remit is not None:
status = (
"reconciled"
if remit.status_code in ("21", "22")
else "received"
)
detail["matchedRemittance"] = {
"id": remit.id,
"totalPaid": float(remit.total_paid or 0),
"status": status,
"receivedAt": _iso_z(remit.received_at),
}
# If the remittance was deleted out from under the FK
# (the FK is ``ON DELETE SET NULL`` so the column is
# already cleared in normal flow), the matched_remittance_id
# would be None here and we wouldn't enter this branch.
# If the FK is non-null but the row is gone (e.g. tests
# that bypass the cascade), fall through with the
# default ``None`` — the UI shows "no match" rather
# than crashing.
# SP7 §5.2: slim per-line projection so the ServiceLinesTable
# can show Paid + Adjustments columns without a second fetch.
# The 837 side is keyed by ``claim_service_line_number`` (the
# 1-based line number from raw_json) since 837 service lines
# are not a separate ORM table.
from cyclone.db import (
LineReconciliation, ServiceLinePayment, CasAdjustment,
)
slim_lrs = list(
s.query(LineReconciliation)
.filter(LineReconciliation.claim_id == claim_id)
.all()
)
svc_ids_for_cas = [
lr.service_line_payment_id
for lr in slim_lrs
if lr.service_line_payment_id is not None
]
cas_sums_by_svc: dict = {}
svc_by_id_slim: dict = {}
if svc_ids_for_cas:
cas_rows = (
s.query(CasAdjustment.service_line_payment_id, CasAdjustment.amount)
.filter(CasAdjustment.service_line_payment_id.in_(svc_ids_for_cas))
.all()
)
from collections import defaultdict
agg = defaultdict(lambda: Decimal("0"))
for svc_id, amount in cas_rows:
agg[svc_id] += Decimal(str(amount))
cas_sums_by_svc = {k: str(v) for k, v in agg.items()}
for svc in (
s.query(ServiceLinePayment)
.filter(ServiceLinePayment.id.in_(svc_ids_for_cas))
.all()
):
svc_by_id_slim[svc.id] = svc
slim_by_num: dict = {
lr.claim_service_line_number: lr
for lr in slim_lrs
if lr.claim_service_line_number is not None
}
line_reconciliation_slim: list = []
for sl in detail["serviceLines"]:
ln = sl.get("lineNumber")
lr = slim_by_num.get(ln)
if lr is None:
line_reconciliation_slim.append({
"lineNumber": ln,
"status": "unmatched_837_only",
"paid": None,
"adjustmentsSum": None,
})
continue
svc = (
svc_by_id_slim.get(lr.service_line_payment_id)
if lr.service_line_payment_id
else None
)
line_reconciliation_slim.append({
"lineNumber": ln,
"status": lr.status,
"paid": str(Decimal(str(svc.payment))) if svc else None,
"adjustmentsSum": (
cas_sums_by_svc.get(lr.service_line_payment_id)
if lr.service_line_payment_id
else None
),
})
detail["lineReconciliation"] = line_reconciliation_slim
return detail
def iter_claims(
*,
batch_id: str | None = None,
status: str | None = None,
provider_npi: str | None = None,
payer: str | None = None,
date_from: str | None = None,
date_to: str | None = None,
sort: str | None = None,
order: str = "desc",
limit: int = 100,
offset: int = 0,
) -> list[dict]:
"""Return UI-shaped claim dicts from the DB.
Filters mirror the in-memory version. The ``payer`` filter is
a case-insensitive substring on the payer's ``name``, recovered
from each claim's ``raw_json`` payload (the DB stores it there
because ``Claim`` itself only carries ``payer_id``).
"""
with db.SessionLocal()() as s:
q = s.query(Claim)
if batch_id is not None:
q = q.filter(Claim.batch_id == batch_id)
if status is not None:
q = q.filter(Claim.state == ClaimState(status))
if provider_npi is not None:
q = q.filter(Claim.provider_npi == provider_npi)
rows = q.all()
# Bulk-load matched-remittance totals so the UI's "Received"
# KPI + per-claim received_amount reflect real paid amounts
# rather than always-0. One SQL roundtrip for the whole page
# rather than per-claim lookups.
matched_ids = [
r.matched_remittance_id
for r in rows
if r.matched_remittance_id
]
received_by_remit: dict[str, float] = {}
if matched_ids:
for rid, total_paid in (
s.query(Remittance.id, Remittance.total_paid)
.filter(Remittance.id.in_(matched_ids))
.all()
):
received_by_remit[rid] = float(total_paid or 0)
out: list[dict] = []
for r in rows:
raw = r.raw_json or {}
bp = raw.get("billing_provider", {})
payer_obj = raw.get("payer", {})
sub = raw.get("subscriber", {})
claim_hdr = raw.get("claim", {})
service_lines = raw.get("service_lines", [])
parsed_at_iso = (
r.batch.parsed_at.isoformat().replace("+00:00", "Z")
if r.batch is not None
else ""
)
cpt = (
service_lines[0].get("procedure", {}).get("code", "")
if service_lines
else ""
)
out.append({
"id": r.id,
"patientName": (
f"{sub.get('first_name', '')} "
f"{sub.get('last_name', '')}".strip()
),
"providerNpi": bp.get("npi") or r.provider_npi or "",
"payerName": payer_obj.get("name") or "",
"cptCode": cpt,
"billedAmount": float(r.charge_amount or 0),
"receivedAmount": received_by_remit.get(
r.matched_remittance_id, 0.0
),
"status": r.state.value if hasattr(r.state, "value") else str(r.state),
"state": r.state.value if hasattr(r.state, "value") else str(r.state),
"denialReason": None,
"submissionDate": parsed_at_iso,
"batchId": r.batch_id,
"parsedAt": parsed_at_iso,
# Keep these so we can sort on them in-memory below.
"_sort_billedAmount": float(r.charge_amount or 0),
"_sort_submissionDate": parsed_at_iso,
})
if payer is not None:
needle = payer.casefold()
out = [
c for c in out
if needle in (c.get("payerName") or "").casefold()
]
out = [
c for c in out
if _date_in_bounds(c, "submissionDate", date_from, date_to)
]
if sort is not None:
out.sort(
key=lambda c: c.get(f"_sort_{sort}", 0) or 0,
reverse=(order == "desc"),
)
# Drop the private sort keys before returning.
for c in out:
c.pop("_sort_billedAmount", None)
c.pop("_sort_submissionDate", None)
return out[offset:offset + limit]
def iter_remittances(
*,
batch_id: str | None = None,
payer: str | None = None,
claim_id: str | None = None,
date_from: str | None = None,
date_to: str | None = None,
sort: str | None = None,
order: str = "desc",
limit: int = 100,
offset: int = 0,
) -> list[dict]:
"""Return UI-shaped remittance dicts from the DB."""
with db.SessionLocal()() as s:
q = s.query(Remittance)
if batch_id is not None:
q = q.filter(Remittance.batch_id == batch_id)
if claim_id is not None:
q = q.filter(Remittance.claim_id == claim_id)
rows = q.all()
# Bulk-fetch all CAS rows for these remittances in one query
# (SP3 P2 follow-up — fixes the list-view's empty adjustments
# expansion). N+1-free.
cas_by_remit: dict[str, list] = {}
if rows:
from cyclone.parsers.cas_codes import reason_label
cas_rows = (
s.query(CasAdjustment)
.filter(CasAdjustment.remittance_id.in_([r.id for r in rows]))
.all()
)
for c in cas_rows:
cas_by_remit.setdefault(c.remittance_id, []).append(c)
out: list[dict] = []
for r in rows:
raw = r.raw_json or {}
parsed_at_iso = (
r.batch.parsed_at.isoformat().replace("+00:00", "Z")
if r.batch is not None
else r.received_at.isoformat().replace("+00:00", "Z")
)
payer_name = ""
if r.batch is not None and r.batch.raw_result_json:
payer_name = (
r.batch.raw_result_json.get("payer", {}).get("name", "")
)
adjustments = [
{
"group": c.group_code,
"reason": c.reason_code,
"label": reason_label(c.group_code, c.reason_code),
"amount": float(c.amount),
"quantity": float(c.quantity) if c.quantity is not None else None,
}
for c in cas_by_remit.get(r.id, [])
]
out.append({
"id": r.id,
"claimId": r.claim_id or "",
"payerName": payer_name,
"paidAmount": float(r.total_paid or 0),
"adjustmentAmount": float(r.adjustment_amount or 0),
"status": (
"reconciled" if r.status_code in ("21", "22")
else "received"
),
"denialReason": None,
"validationWarnings": [],
"receivedDate": r.received_at.isoformat().replace("+00:00", "Z"),
"batchId": r.batch_id,
"parsedAt": parsed_at_iso,
"adjustments": adjustments,
"_sort_receivedDate": r.received_at.isoformat().replace("+00:00", "Z"),
})
if payer is not None:
out = [r for r in out if r.get("payerName") == payer]
out = [
r for r in out
if _date_in_bounds(r, "receivedDate", date_from, date_to)
]
if sort is not None:
out.sort(
key=lambda r: r.get(f"_sort_{sort}", 0) or 0,
reverse=(order == "desc"),
)
for r in out:
r.pop("_sort_receivedDate", None)
return out[offset:offset + limit]
def distinct_providers() -> list[dict]:
"""Group claims by NPI and return one row per provider."""
with db.SessionLocal()() as s:
rows = s.query(Claim).all()
by_npi: dict[str, dict] = {}
for r in rows:
npi = r.provider_npi or ""
if npi not in by_npi:
raw = r.raw_json or {}
bp = raw.get("billing_provider", {})
by_npi[npi] = to_ui_provider(
npi=npi,
name=bp.get("name") or "",
tax_id=bp.get("tax_id"),
address=None,
city=None,
state=None,
zip=None,
phone=None,
claim_count=0,
outstanding_ar=0.0,
)
by_npi[npi]["claimCount"] += 1
return list(by_npi.values())
def recent_activity(*, limit: int = 200) -> list[dict]:
"""Return recent activity events from the DB, newest first.
SP21 Task 2.5: each row also carries ``claimId`` and
``remittanceId`` (read from the ORM columns) so the Dashboard's
Recent-activity card can route clicks to the right entity
drawer via ``src/lib/event-routing.ts``. Both are nullable
strings; the wire shape uses camelCase keys to match the
existing ``npi`` / ``amount`` fields and the frontend
``Activity`` interface.
"""
with db.SessionLocal()() as s:
rows = (
s.query(ActivityEvent)
.order_by(ActivityEvent.ts.desc())
.limit(limit)
.all()
)
return [
{
"id": f"ae-{r.id}",
"kind": r.kind,
"message": (r.payload_json or {}).get("message", ""),
"timestamp": r.ts.isoformat().replace("+00:00", "Z"),
"npi": (r.payload_json or {}).get("npi"),
"amount": (r.payload_json or {}).get("amount"),
"claimId": r.claim_id,
"remittanceId": r.remittance_id,
}
for r in rows
]
def check_matched_pair_drift() -> int:
"""Audit the ``Claim.matched_remittance_id`` ↔ ``Remittance.claim_id``
FK pair at startup. Non-blocking (SP27 Task 11).
The matched pair is a denormalized FK pair maintained transactionally
by ``manual_match``, ``manual_unmatch``, and ``reconcile.run``. A
pre-existing mismatch (e.g. a row written before a state migration
that added one column but not the other) would otherwise stay
invisible until the next operator pair attempt fails confusingly.
This check logs the count + up to N examples so operators can
investigate without booting the system.
Returns the number of drifted rows (0 means clean). Does not
raise; bootstrap continues even if drift is detected.
Count semantics: this returns *drifted rows*, not *drifted pairs*.
A single broken pair (``Claim.matched_remittance_id = X`` AND
``Remittance.claim_id = Y != nil`` with neither pointing back)
can produce TWO drifted rows one in case A (the claim) and
one in case B (the remit). In practice drift is almost always
asymmetric (one side NULL), so count == count(pairs); for the
fully-symmetric minority, divide by ~2 when alerting. Real drift
should be fixed by repairing the writer path, not by counting.
Cases:
A. Claim ``matched_remittance_id = X`` but the paired remit X's
``claim_id`` is either NULL or doesn't point back to the claim.
B. Remit ``claim_id = A`` but the paired claim A's
``matched_remittance_id`` is either NULL or doesn't point back.
"""
import logging
from sqlalchemy import select
log = logging.getLogger(__name__)
with db.SessionLocal()() as s:
# Case A: claim says X is paired, but X.claim_id doesn't point back.
case_a = list(
s.execute(
select(
Claim.id.label("claim_id"),
Claim.matched_remittance_id.label("claimed_remit_id"),
Remittance.claim_id.label("remit_points_to"),
)
.outerjoin(
Remittance,
Remittance.id == Claim.matched_remittance_id,
)
.where(Claim.matched_remittance_id.is_not(None))
.where(
(Remittance.claim_id.is_(None))
| (Remittance.claim_id != Claim.id)
)
).all()
)
# Case B: remit says A is paired, but A.matched_remittance_id
# doesn't point back.
case_b = list(
s.execute(
select(
Remittance.id.label("remit_id"),
Remittance.claim_id.label("claimed_claim_id"),
Claim.matched_remittance_id.label("claim_points_to"),
)
.outerjoin(
Claim,
Claim.id == Remittance.claim_id,
)
.where(Remittance.claim_id.is_not(None))
.where(
(Claim.matched_remittance_id.is_(None))
| (Claim.matched_remittance_id != Remittance.id)
)
).all()
)
total = len(case_a) + len(case_b)
if total == 0:
log.info("matched-pair drift check: 0 mismatches (clean)")
return 0
log.warning(
"matched-pair drift check: %d mismatched pair(s) (showing up to 5 "
"of each). Investigate via SELECT against claim / remittance; "
"manual re-pair via /api/claims/{id}/manual-match will repair.",
total,
)
for r in case_a[:5]:
log.warning(
" case A: claim %s -> remit %s, but remit.claim_id=%r",
r.claim_id,
r.claimed_remit_id,
r.remit_points_to,
)
for r in case_b[:5]:
log.warning(
" case B: remit %s -> claim %s, but claim.matched_remittance_id=%r",
r.remit_id,
r.claimed_claim_id,
r.claim_points_to,
)
return total
# ---------------------------------------------------------------------------
# Aggregate counters (SP25 / SP27): full-population counts and sums that
# the /api/* endpoints expose so page-local reductions (25 rows + live-tail
# delta) can never silently understate the true population.
# ---------------------------------------------------------------------------
_ITER_UNBOUNDED = 2**31 - 1
def count_claims(
*,
batch_id: str | None = None,
status: str | None = None,
provider_npi: str | None = None,
payer: str | None = None,
date_from: str | None = None,
date_to: str | None = None,
) -> int:
"""Count claims that would be returned by ``iter_claims``.
Same filter parameters as ``iter_claims`` (excluding
``sort``/``order``/``limit``/``offset``, which don't affect cardinality).
"""
rows = iter_claims(
batch_id=batch_id, status=status, provider_npi=provider_npi,
payer=payer, date_from=date_from, date_to=date_to,
sort=None, order="desc", limit=_ITER_UNBOUNDED, offset=0,
)
return len(rows)
def count_remittances(
*,
batch_id: str | None = None,
payer: str | None = None,
claim_id: str | None = None,
date_from: str | None = None,
date_to: str | None = None,
) -> int:
"""Count remittances that would be returned by ``iter_remittances``.
Same filter parameters as ``iter_remittances`` (excluding
``sort``/``order``/``limit``/``offset``).
"""
rows = iter_remittances(
batch_id=batch_id, payer=payer, claim_id=claim_id,
date_from=date_from, date_to=date_to,
sort=None, order="desc", limit=_ITER_UNBOUNDED, offset=0,
)
return len(rows)
def summarize_remittances(
*,
batch_id: str | None = None,
payer: str | None = None,
claim_id: str | None = None,
date_from: str | None = None,
date_to: str | None = None,
) -> dict:
"""Return ``{count, total_paid, total_adjustments}`` summed over the
remittance population that ``iter_remittances`` would return under
the same filters. Backs ``GET /api/remittances/summary`` the
Remittances page's KPI tiles (added in SP27).
"""
rows = iter_remittances(
batch_id=batch_id, payer=payer, claim_id=claim_id,
date_from=date_from, date_to=date_to,
sort=None, order="desc", limit=_ITER_UNBOUNDED, offset=0,
)
total_paid = 0.0
total_adjustments = 0.0
for r in rows:
total_paid += float(r.get("paidAmount") or 0)
total_adjustments += float(r.get("adjustmentAmount") or 0)
return {
"count": len(rows),
"total_paid": total_paid,
"total_adjustments": total_adjustments,
}
+41
View File
@@ -0,0 +1,41 @@
"""Exception types raised by CycloneStore and its domain modules.
These are part of the public API callers (API endpoints) catch them
and translate to HTTP 409 Conflict responses.
"""
class AlreadyMatchedError(Exception):
"""Raised by ``CycloneStore.manual_match`` when the claim is already paired.
The claim's ``matched_remittance_id`` is set, so any new pairing would
clobber an existing match. Callers (the T15 API endpoint) should surface
this as a 409 Conflict.
"""
class NotMatchedError(Exception):
"""Raised by ``CycloneStore.manual_unmatch`` when the claim has no match.
Mirrors ``AlreadyMatchedError`` for the unpair operation. Same HTTP
treatment: 409 Conflict at the API layer.
"""
class InvalidStateError(Exception):
"""Raised when an apply_* pure fn returns a skipped ApplyIntent.
``reconcile.apply_payment`` / ``apply_reversal`` may return
``skipped=True`` (e.g. claim already in a terminal state, or reversal
on a non-paid claim). The store surfaces that as ``InvalidStateError``
rather than silently pairing. The T15 API endpoint maps this to a
409 Conflict and echoes ``current_state`` and ``activity_kind`` so
the UI can render a precise message.
"""
def __init__(self, current_state: str, activity_kind: str = "invalid_state"):
self.current_state = current_state
self.activity_kind = activity_kind
super().__init__(
f"invalid state {current_state} for apply (kind={activity_kind})"
)
+315
View File
@@ -0,0 +1,315 @@
"""Inbox lane state and manual match/unmatch operations.
``list_unmatched`` returns the 5-lane inbox view (unmatched claims +
unmatched remits). ``manual_match`` and ``manual_unmatch`` invoke the
reconcile pure functions and persist the resulting Match row (or
remove it). Invalid state transitions surface as ``InvalidStateError``.
"""
from cyclone import db
from cyclone.db import ActivityEvent, Claim, ClaimState, Match, Remittance
from .exceptions import AlreadyMatchedError, InvalidStateError, NotMatchedError
from . import utcnow
from .ui import to_ui_claim_from_orm, to_ui_remittance_from_orm
# -- manual reconciliation (T12) -----------------------------------
def list_unmatched(*, kind: str = "both") -> dict:
"""Return unmatched claims and/or remittances.
An unmatched claim is one with ``matched_remittance_id IS NULL``
either auto-match never paired it, or it was unpaired by
``manual_unmatch``. An unmatched remittance is one with
``claim_id IS NULL`` symmetric FK on the remittance side; we
update this in ``manual_match`` so the filter reflects the pair.
Note: T10's ``reconcile.run`` only writes the claim-side FK
(``Claim.matched_remittance_id``) when auto-pairing. Auto-matched
remittances therefore still show as "unmatched" here until the
pair is touched (re-ingest, manual unmatch + rematch). That gap
is intentional for T12 fixing it requires modifying T10.
``kind`` selects which side(s) to return:
- "claims": only claims
- "remittances": only remittances
- "both": both (default)
Returns ``{"claims": [...], "remittances": [...]}`` with the
unused side always an empty list (never absent) so callers can
unconditionally index.
"""
if kind not in ("claims", "remittances", "both"):
raise ValueError(
f"list_unmatched: unknown kind={kind!r} "
"(expected 'claims', 'remittances', or 'both')"
)
result: dict = {"claims": [], "remittances": []}
with db.SessionLocal()() as s:
if kind in ("claims", "both"):
rows = (
s.query(Claim)
.filter(Claim.matched_remittance_id.is_(None))
.order_by(Claim.id.asc())
.all()
)
for r in rows:
parsed_at = (
r.batch.parsed_at
if r.batch is not None
else r.service_date_from or utcnow()
)
result["claims"].append(
to_ui_claim_from_orm(
r, batch_id=r.batch_id, parsed_at=parsed_at,
# list_unmatched filters matched_remittance_id IS NULL,
# so every row has no remittance yet.
received_total=0.0,
)
)
if kind in ("remittances", "both"):
rows = (
s.query(Remittance)
.filter(Remittance.claim_id.is_(None))
.order_by(Remittance.id.asc())
.all()
)
for r in rows:
parsed_at = (
r.batch.parsed_at
if r.batch is not None
else r.received_at
)
result["remittances"].append(
to_ui_remittance_from_orm(
r, batch_id=r.batch_id, parsed_at=parsed_at,
)
)
return result
def manual_match(claim_id: str, remit_id: str) -> dict:
"""Pair a claim with a remittance manually (operator override).
Steps:
1. Load the claim; raise ``AlreadyMatchedError`` if it already
has a match (we never silently overwrite an existing pair).
2. Load the remittance; raise ``LookupError`` if missing.
3. Compute the new claim state via ``reconcile.apply_payment``
(or ``apply_reversal`` for status codes 21/22).
4. Insert a ``Match`` row with ``strategy="manual"``.
5. Update the claim (``state``, ``matched_remittance_id``) AND
the remittance (``claim_id``) so the symmetric FK reflects
the pair required for ``list_unmatched`` to drop them.
6. Record an ``ActivityEvent(kind="manual_match", ...)``.
7. Commit; return ``{"claim": <ui>, "match": <ui>}``.
``reconcile.apply_payment`` may return a noop (claim in terminal
state); we surface that as ``InvalidStateError`` rather than
silently pairing, because the operator clearly intended a state
change. The T15 API endpoint maps this to a 409 Conflict.
"""
from cyclone import reconcile as _reconcile
with db.SessionLocal()() as s:
claim = s.get(Claim, claim_id)
if claim is None:
raise LookupError(f"claim {claim_id} not found")
if claim.matched_remittance_id is not None:
raise AlreadyMatchedError(
f"claim {claim_id} already matched to "
f"{claim.matched_remittance_id}"
)
remit = s.get(Remittance, remit_id)
if remit is None:
raise LookupError(f"remittance {remit_id} not found")
prior_state = claim.state
if remit.is_reversal:
intent = _reconcile.apply_reversal(claim, remit)
else:
intent = _reconcile.apply_payment(
claim, remit,
charge=claim.charge_amount,
paid=remit.total_paid,
status_code=remit.status_code,
)
if intent.skipped or intent.new_state is None:
current = (
claim.state.value
if hasattr(claim.state, "value")
else str(claim.state)
)
raise InvalidStateError(
current_state=current,
activity_kind=intent.activity_kind,
)
new_state = intent.new_state
now = utcnow()
s.add(Match(
claim_id=claim_id,
remittance_id=remit_id,
strategy="manual",
matched_at=now,
prior_claim_state=prior_state,
is_reversal=remit.is_reversal,
))
claim.state = new_state
claim.matched_remittance_id = remit_id
# Symmetric FK update — see list_unmatched docstring for why
# we don't rely on T10 to do this.
remit.claim_id = claim_id
# SP7: line-level reconciliation + claim-level CAS aggregate.
# Skipped for reversals — they don't have SV1↔SVC line pairs.
if not remit.is_reversal:
_reconcile._reconcile_pair(s, claim, remit)
s.add(ActivityEvent(
ts=now,
kind="manual_match",
batch_id=remit.batch_id,
claim_id=claim_id,
remittance_id=remit_id,
payload_json={
"strategy": "manual",
"new_state": new_state.value,
"prior_state": prior_state.value,
"is_reversal": remit.is_reversal,
},
))
s.commit()
parsed_at = (
claim.batch.parsed_at
if claim.batch is not None
else now
)
claim_dict = to_ui_claim_from_orm(
claim, batch_id=claim.batch_id, parsed_at=parsed_at,
received_total=float(remit.total_paid or 0),
)
matched_at_iso = now.isoformat().replace("+00:00", "Z")
return {
"claim": claim_dict,
"match": {
"strategy": "manual",
"claimId": claim_id,
"remittanceId": remit_id,
"matchedAt": matched_at_iso,
"isReversal": remit.is_reversal,
"priorState": prior_state.value,
"newState": new_state.value,
},
}
def manual_unmatch(claim_id: str) -> dict:
"""Unpair a previously matched claim and restore its prior state.
Reverses ``manual_match`` (and auto-match as a side-effect of
clearing the FK). Strategy:
1. Load the claim; raise ``NotMatchedError`` if it isn't
currently matched.
2. Delete every ``Match`` row for the claim (there may be more
than one reversals create a 2nd row, see T10 spec).
3. Restore ``claim.state`` from the latest Match's
``prior_claim_state``; fall back to ``SUBMITTED`` when
``prior_claim_state`` is NULL (auto-match doesn't set it for
non-reversal payments see reconcile.run line ~278).
4. Clear ``claim.matched_remittance_id`` and the symmetric
``remit.claim_id`` so ``list_unmatched`` surfaces the pair
again.
5. Record ``ActivityEvent(kind="manual_unmatch", ...)`` and
commit.
6. Return ``{"claim": <ui>, "deletedMatches": <count>}``.
"""
with db.SessionLocal()() as s:
claim = s.get(Claim, claim_id)
if claim is None:
raise LookupError(f"claim {claim_id} not found")
if claim.matched_remittance_id is None:
raise NotMatchedError(
f"claim {claim_id} has no active match"
)
matches = (
s.query(Match)
.filter(Match.claim_id == claim_id)
.order_by(Match.matched_at.desc())
.all()
)
if not matches:
# Defensive: matched_remittance_id was set but no Match
# rows exist. Shouldn't happen, but if it does, fall back
# to clearing the FK and starting fresh.
latest = None
paired_remit = None
restored_state = ClaimState.SUBMITTED
else:
latest = matches[0]
paired_remit = s.get(Remittance, latest.remittance_id)
restored_state = (
latest.prior_claim_state
if latest.prior_claim_state is not None
else ClaimState.SUBMITTED
)
deleted_count = len(matches)
for m in matches:
s.delete(m)
claim.state = restored_state
claim.matched_remittance_id = None
# Clear the symmetric FK on the remittance so list_unmatched
# surfaces the pair again. The remittance may have been
# deleted between the match and this call — guard with a
# None check so we don't blow up on a stale FK.
if paired_remit is not None:
paired_remit.claim_id = None
now = utcnow()
s.add(ActivityEvent(
ts=now,
kind="manual_unmatch",
claim_id=claim_id,
payload_json={
"restored_state": restored_state.value,
"deleted_matches": deleted_count,
},
))
s.commit()
parsed_at = (
claim.batch.parsed_at
if claim.batch is not None
else now
)
# ``paired_remit`` is the matched remittance we cleared in
# the unmatch; use its ``total_paid`` for the response shape
# so the UI sees what was paid before the unpair. May be
# ``None`` if the remittance was deleted since the match —
# default to 0.0 in that case.
received_total = (
float(paired_remit.total_paid or 0)
if paired_remit is not None
else 0.0
)
claim_dict = to_ui_claim_from_orm(
claim, batch_id=claim.batch_id, parsed_at=parsed_at,
received_total=received_total,
)
return {
"claim": claim_dict,
"deletedMatches": deleted_count,
}
# ------------------------------------------------------------------
+307
View File
@@ -0,0 +1,307 @@
"""Dashboard KPI aggregation — cross-table reads consumed by /api/dashboard/kpis.
``dashboard_kpis()`` returns a single dict that the React Dashboard page
consumes as the source of truth for the 4 KPI tiles + the recent-activity
panel. It reads from claims + remittances + batches in a single pass.
``_claim_state_str`` is a small mapping helper that translates ORM
status enum values to UI-friendly strings. It's private to this module.
"""
from datetime import datetime, timezone
from cyclone import db
from cyclone.db import Claim, Remittance
# ---------------------------------------------------------------------------
# SP27 Task 13: Dashboard aggregate KPIs.
#
# The Dashboard's "Billed / Received / Denial rate / Pending AR" tiles are
# computed from the *whole* claim population, not a sample. With 60k+ claims
# in production, fetching ``/api/claims?limit=100`` and reducing in JS would
# silently produce wrong numbers (denial rate sampled, billed summed from
# 100 rows). This module-level function does the aggregation server-side in
# a single session and returns a small structured payload the Dashboard
# can render directly.
#
# Performance: one ``SELECT * FROM claims`` (no pagination) + one
# ``SELECT id, total_paid FROM remittances WHERE id IN (...)`` for matched
# remits. SQLite returns 60k claim rows in ~30ms on the development
# machine; the Python reduce is microseconds. If the dataset grows past
# ~500k claims we'd want a SQL-side ``GROUP BY month`` instead — but at
# that volume the dashboard should probably be backed by a materialized
# view, not a live query.
# ---------------------------------------------------------------------------
def _claim_state_str(claim: Claim) -> str:
"""Stringify a Claim's ``state`` regardless of enum vs raw str storage."""
st = claim.state
return st.value if hasattr(st, "value") else str(st)
# Claim states counted toward the Dashboard's "pending" tile. SUBMITTED
# is the initial post-parse state; REJECTED is the 999 envelope-level
# rejection that means the payer never saw the claim. Both are
# "outstanding adjudication" from the operator's POV; ``denied`` is
# payer adjudication and lives in its own tile.
_DASHBOARD_PENDING_STATES: frozenset[str] = frozenset({"submitted", "rejected"})
def dashboard_kpis(
*,
months: int = 6,
top_n_providers: int = 4,
top_n_denials: int = 5,
) -> dict:
"""Compute Dashboard KPIs over the entire claim/remittance population.
Parameters
----------
months
Number of trailing calendar months to include in the ``monthly``
sparkline series (default 6 matches the existing frontend
``MONTHS_BACK`` constant).
top_n_providers
How many providers to include in the ``topProviders`` array
(default 4 matches the existing Dashboard layout).
top_n_denials
How many most-recent denied claims to include in the
``topDenials`` array (default 5).
Returns
-------
dict with keys:
- ``totals``: aggregate counts + dollar sums + rates for the whole DB.
- ``monthly``: list of ``{month, label, count, billed, received,
denied, denialRate, ar}`` dicts, oldest-first, length = ``months``.
``ar`` is the running outstanding accounts-receivable (billed -
received) carried forward across months, clamped at zero.
- ``topProviders``: list of ``{npi, label, claimCount, billed,
denied}`` dicts, sorted by claimCount desc.
- ``topDenials``: list of ``{id, patientName, billedAmount,
denialReason, submissionDate}`` dicts, sorted by submissionDate
desc, capped at ``top_n_denials``.
Notes
-----
- Empty DB returns zero-filled aggregates, an empty ``monthly`` array
(one entry per requested month), an empty ``topProviders`` array,
and an empty ``topDenials`` array.
- ``received`` for a claim is sourced from the matched Remittance's
``total_paid`` column. Claims without a matched remittance
contribute 0 to the received sum (and thus inflate the
outstanding-AR figure, which is the correct operator-visible
semantic "money we haven't been told was paid yet").
"""
# Pre-build the trailing-N-months skeleton so the response always has
# exactly ``months`` entries even when the DB is empty or sparse.
now = datetime.now(timezone.utc)
skeleton: list[dict] = []
for i in range(months - 1, -1, -1):
d = now.replace(day=1)
# Walk backwards N months without ``relativedelta``.
for _ in range(i):
prev_month = d.month - 1
if prev_month == 0:
d = d.replace(year=d.year - 1, month=12)
else:
d = d.replace(month=prev_month)
skeleton.append({
"month": f"{d.year:04d}-{d.month:02d}",
"label": d.strftime("%b"),
"count": 0,
"billed": 0.0,
"received": 0.0,
"denied": 0,
"ar": 0.0,
})
skeleton_index = {entry["month"]: entry for entry in skeleton}
with db.SessionLocal()() as s:
# ``Claim.batch`` is a lazy ``relationship`` (default
# ``lazy="select"``); without ``selectinload`` each access to
# ``r.batch`` in the reduce loop below issues a fresh
# ``SELECT ... FROM batches WHERE id=?``. ``selectinload``
# pulls every distinct batch in one round-trip instead of N+1
# — critical for the 60k-claim dataset.
from sqlalchemy.orm import selectinload
claims: list[Claim] = (
s.query(Claim)
.options(selectinload(Claim.batch))
.all()
)
# Bulk-load matched-remit total_paid so a 60k-claim DB doesn't
# produce a 60k-query N+1.
matched_ids = [
r.matched_remittance_id
for r in claims
if r.matched_remittance_id
]
received_by_remit: dict[str, float] = {}
if matched_ids:
for rid, total_paid in (
s.query(Remittance.id, Remittance.total_paid)
.filter(Remittance.id.in_(matched_ids))
.all()
):
received_by_remit[rid] = float(total_paid or 0)
# Per-provider accumulator for the topProviders leaderboard.
provider_counts: dict[str, int] = {}
provider_billed: dict[str, float] = {}
provider_denied: dict[str, int] = {}
# Per-month accumulator + totals in a single pass.
total_count = 0
total_billed = 0.0
total_received = 0.0
denied_count = 0
pending_count = 0
# Collect denied candidates as we walk so we don't issue a
# second pass for the topDenials array.
denied_candidates: list[dict] = []
for r in claims:
billed = float(r.charge_amount or 0)
received = received_by_remit.get(r.matched_remittance_id, 0.0)
state_str = _claim_state_str(r)
total_count += 1
total_billed += billed
total_received += received
if state_str == "denied":
denied_count += 1
# Drop denied claims with no ``submissionDate`` — they'd
# sort first under reverse-lex (empty string < ISO) and
# render as "Invalid Date" in the Dashboard. A denial
# without a batch is exceptional and operator-irrelevant
# for the "recent denials" widget.
if r.batch is None or r.batch.parsed_at is None:
pass
else:
raw = r.raw_json or {}
sub = raw.get("subscriber", {})
denied_candidates.append({
"id": r.id,
"patientName": (
f"{sub.get('first_name', '')} "
f"{sub.get('last_name', '')}".strip()
),
"billedAmount": billed,
"denialReason": r.rejection_reason,
"submissionDate": (
r.batch.parsed_at
.isoformat()
.replace("+00:00", "Z")
),
})
if state_str in _DASHBOARD_PENDING_STATES:
pending_count += 1
# Monthly bin. ``submissionDate`` lives on the parent Batch
# (all claims in a batch share a parsed_at). Use UTC year-month
# to match the skeleton.
if r.batch is not None and r.batch.parsed_at is not None:
pa = r.batch.parsed_at
key = f"{pa.year:04d}-{pa.month:02d}"
bucket = skeleton_index.get(key)
if bucket is not None:
bucket["count"] += 1
bucket["billed"] += billed
bucket["received"] += received
if state_str == "denied":
bucket["denied"] += 1
# Provider bin (keyed on NPI).
npi = r.provider_npi or ""
if npi:
provider_counts[npi] = provider_counts.get(npi, 0) + 1
provider_billed[npi] = provider_billed.get(npi, 0.0) + billed
if state_str == "denied":
provider_denied[npi] = provider_denied.get(npi, 0) + 1
# Compute denial rate per month + running AR.
running_ar = 0.0
for entry in skeleton:
if entry["count"] > 0:
entry["denialRate"] = (entry["denied"] / entry["count"]) * 100.0
else:
entry["denialRate"] = 0.0
running_ar = max(0.0, running_ar + entry["billed"] - entry["received"])
entry["ar"] = running_ar
# Resolve top provider labels from the Provider table in one
# round-trip. NPIs without a Provider row still appear in the
# leaderboard with an empty label so operators can see
# "unknown-NPI" claims are coming from somewhere.
# Local import keeps the module-level ``cyclone.providers.Provider``
# Pydantic DTO and the SQLAlchemy ``cyclone.db.Provider`` ORM
# separate (same pattern as ``list_providers`` / ``upsert_provider``).
provider_labels: dict[str, str] = {}
if provider_counts:
from cyclone.db import Provider as ProviderORM
for npi, label in (
s.query(ProviderORM.npi, ProviderORM.label).filter(
ProviderORM.npi.in_(provider_counts.keys())
).all()
):
provider_labels[npi] = label or ""
top_providers = sorted(
provider_counts.items(),
key=lambda kv: kv[1],
reverse=True,
)[: max(0, top_n_providers)]
top_providers_out = [
{
"npi": npi,
"label": provider_labels.get(npi, ""),
"claimCount": count,
"billed": round(provider_billed.get(npi, 0.0), 2),
"denied": provider_denied.get(npi, 0),
}
for npi, count in top_providers
]
# Top denials = most recently submitted denied claims, capped at
# ``top_n_denials``. submissionDate is ISO-8601 so lex sort ==
# chronological sort when timestamps share a tz.
denied_candidates.sort(key=lambda d: d["submissionDate"], reverse=True)
top_denials = denied_candidates[: max(0, top_n_denials)]
total_denial_rate = (
(denied_count / total_count) * 100.0 if total_count > 0 else 0.0
)
return {
"totals": {
"count": total_count,
"billed": round(total_billed, 2),
"received": round(total_received, 2),
"outstandingAr": round(max(0.0, total_billed - total_received), 2),
"denied": denied_count,
"denialRate": round(total_denial_rate, 4),
"pending": pending_count,
},
"monthly": [
{
"month": e["month"],
"label": e["label"],
"count": e["count"],
"billed": round(e["billed"], 2),
"received": round(e["received"], 2),
"denied": e["denied"],
"denialRate": round(e["denialRate"], 4),
"ar": round(e["ar"], 2),
}
for e in skeleton
],
"topProviders": top_providers_out,
"topDenials": top_denials,
}
+180
View File
@@ -0,0 +1,180 @@
"""ORM row builders — convert parser output into SQLAlchemy ORM rows.
Each function returns an unsaved ORM object (or inserts related rows
within an existing session). They never commit or close the session
the caller (``write.add_record``, ``acks.add_ack``, etc.) owns the
transaction boundary.
"""
from __future__ import annotations
import json
from decimal import Decimal
from cyclone import db
from cyclone.db import Claim, ClaimState, Remittance
from cyclone.parsers.models import ClaimOutput
from cyclone.parsers.models_835 import ClaimPayment
from . import utcnow
def _service_dates_from_claim(claim: ClaimOutput) -> tuple[date | None, date | None]:
"""Extract (service_date_from, service_date_to) from a ClaimOutput.
The 837P model has ``service_lines[*].service_date`` (one per SV1).
We use the earliest as ``from`` and the latest as ``to``; if there
are no service lines, both are ``None``.
"""
dates: list[date] = []
for sl in claim.service_lines:
if sl.service_date is not None:
dates.append(sl.service_date)
if not dates:
return None, None
return min(dates), max(dates)
def _claim_837_row(claim: ClaimOutput, batch_id: str) -> Claim:
"""Build a Claim ORM row from a ClaimOutput. NOT yet persisted."""
d_from, d_to = _service_dates_from_claim(claim)
return Claim(
id=claim.claim_id,
batch_id=batch_id,
# SP27 Task 17: Claim.patient_control_number must hold the CLM01
# claim_submittr's_identifier the 837 sent — that's the value the
# 835 echoes in CLP01, which the reconcile matcher joins on
# (reconcile.py:by_pcn), and what 999 / 277CA ACK lookups also
# use to cross-reference the original claim. Storing
# subscriber.member_id here (the 2010BA NM109) silently broke
# every auto-match in production.
patient_control_number=claim.claim_id or "",
service_date_from=d_from,
service_date_to=d_to,
charge_amount=Decimal(claim.claim.total_charge or 0),
provider_npi=claim.billing_provider.npi,
# SP32: wire NM1*82 (Loop 2420A) rendering provider NPI from
# ClaimOutput (T3 parser extraction) into the typed ORM column
# (T1 migration 0019). Falls back to None if absent.
rendering_provider_npi=claim.rendering_provider_npi,
payer_id=claim.payer.id,
state=ClaimState.SUBMITTED,
raw_json=json.loads(claim.model_dump_json()),
)
def _remittance_835_row(cp: ClaimPayment, batch_id: str) -> Remittance:
"""Build a Remittance ORM row from a ClaimPayment. NOT yet persisted."""
received_at = utcnow()
# Adjustment amount: sum the CAS rows for the first service line.
# NOTE: This is a best-effort placeholder used until the reconciliation
# pass (T10) overwrites it from the persisted CasAdjustment rows. The
# authoritative value comes from `reconcile.run()`, which sums
# ``CasAdjustment.amount`` per ``remittance_id`` and writes the result
# back to ``Remittance.adjustment_amount``. We keep this stub so the
# row has a sane value if reconciliation is disabled or fails.
adjustment = Decimal("0")
if cp.service_payments:
sp = cp.service_payments[0]
for adj in sp.adjustments:
adjustment += adj.amount
# Use the first service line's service_date as the remit service_date.
service_date: date | None = None
if cp.service_payments and cp.service_payments[0].service_date is not None:
service_date = cp.service_payments[0].service_date
return Remittance(
id=cp.payer_claim_control_number,
batch_id=batch_id,
payer_claim_control_number=cp.payer_claim_control_number,
claim_id=None,
status_code=cp.status_code,
status_label=cp.status_label,
total_charge=Decimal(cp.total_charge or 0),
total_paid=Decimal(cp.total_paid or 0),
patient_responsibility=cp.patient_responsibility,
adjustment_amount=adjustment,
# SP32: wire NM1*1P (Loop 2100) service provider NPI from
# ClaimPayment (T2 parser extraction) into the typed ORM column
# (T1 migration 0019). Falls back to None if absent.
rendering_provider_npi=cp.service_provider_npi,
received_at=received_at,
service_date=service_date,
is_reversal=cp.status_code in ("21", "22"),
raw_json=json.loads(cp.model_dump_json()),
)
def _persist_835_remit(session, cp: "ClaimPayment", remittance_id: str) -> None:
"""SP7: persist ServiceLinePayment + CAS rows for one CLP composite.
For each 835 SVC composite in ``cp.service_payments``:
- insert a ServiceLinePayment row (line_number, procedure, modifiers,
charge, payment, units, service_date).
- flush to populate slp.id.
- insert each per-SVC CAS adjustment with ``service_line_payment_id``
set to slp.id.
For CLP-level CAS adjustments (``cp.claim_adjustments``, a future
extension; not produced by today's 835 parser but allowed by the spec):
- insert CAS rows with ``service_line_payment_id IS NULL``.
The caller controls the transaction; this function does not commit.
The 835 ingest site calls this after ``_remittance_835_row`` is
flushed so the FK target is populated.
"""
import json as _json
from cyclone.db import ServiceLinePayment, CasAdjustment
for svc in cp.service_payments:
slp = ServiceLinePayment(
remittance_id=remittance_id,
line_number=svc.line_number,
procedure_qualifier=svc.procedure_qualifier,
procedure_code=svc.procedure_code,
modifiers_json=_json.dumps(svc.modifiers or []),
charge=Decimal(str(svc.charge)),
payment=Decimal(str(svc.payment)),
units=Decimal(str(svc.units)) if svc.units is not None else None,
unit_type=svc.unit_type,
service_date=svc.service_date,
ref_benefit_plan=svc.ref_benefit_plan,
)
session.add(slp)
session.flush() # populate slp.id for the FK below
for adj in svc.adjustments:
quantity = getattr(adj, "quantity", None)
session.add(CasAdjustment(
remittance_id=remittance_id,
group_code=adj.group_code,
reason_code=adj.reason_code,
amount=Decimal(str(adj.amount)),
quantity=Decimal(str(quantity)) if quantity is not None else None,
service_line_payment_id=slp.id,
))
# CLP-level CAS (no SVC composite to attach to). Today's parser does
# not produce these; the branch is forward-compatible.
for adj in getattr(cp, "claim_adjustments", []) or []:
quantity = getattr(adj, "quantity", None)
session.add(CasAdjustment(
remittance_id=remittance_id,
group_code=adj.group_code,
reason_code=adj.reason_code,
amount=Decimal(str(adj.amount)),
quantity=Decimal(str(quantity)) if quantity is not None else None,
service_line_payment_id=None,
))
def _claim_status_from_validation(claim: ClaimOutput) -> str:
"""Re-implement the in-memory status rules (sub-project 1 §6.2)."""
v = claim.validation
if not v.passed:
has_r050 = any(e.rule == "R050_diagnosis_present" for e in v.errors)
return "draft" if has_r050 else "denied"
if claim.claim.frequency_code == "1":
return "submitted"
if v.warnings:
return "pending"
return "draft"
+307
View File
@@ -0,0 +1,307 @@
"""Provider, payer, and clearhouse configuration reads and upserts.
The ``providers`` and ``payers`` modules in ``cyclone`` provide the
ORM-row DTOs; this module is the read/upsert surface over them.
``ensure_clearhouse_seeded`` is called at startup to insert the
default Clearhouse row if missing.
"""
import json
from datetime import datetime, timezone
from cyclone import db
from cyclone.providers import Clearhouse, Payer, Provider
from . import utcnow
# ------------------------------------------------------------------
# SP9: providers / payers / payer_configs / clearhouse
# ------------------------------------------------------------------
def list_providers(*, is_active: bool | None = True) -> list[Provider]:
"""List providers. ``is_active=None`` returns all."""
from cyclone.db import Provider as ProviderORM
from cyclone.providers import Provider
with db.SessionLocal()() as s:
q = s.query(ProviderORM)
if is_active is not None:
q = q.filter(ProviderORM.is_active == (1 if is_active else 0))
rows = q.order_by(ProviderORM.label).all()
return [Provider.model_validate(_provider_orm_to_dict(r)) for r in rows]
def get_provider(npi: str) -> Provider | None:
from cyclone.db import Provider as ProviderORM
from cyclone.providers import Provider
with db.SessionLocal()() as s:
row = s.get(ProviderORM, npi)
return Provider.model_validate(_provider_orm_to_dict(row)) if row else None
def upsert_provider(provider: Provider) -> Provider:
from cyclone.db import Provider as ProviderORM
with db.SessionLocal()() as s:
row = s.get(ProviderORM, provider.npi)
now = utcnow().isoformat()
if row is None:
row = ProviderORM(
npi=provider.npi, label=provider.label,
legal_name=provider.legal_name, tax_id=provider.tax_id,
taxonomy_code=provider.taxonomy_code,
address_line1=provider.address_line1,
address_line2=provider.address_line2,
city=provider.city, state=provider.state, zip=provider.zip,
is_active=1 if provider.is_active else 0,
created_at=provider.created_at.isoformat(),
updated_at=now,
)
s.add(row)
else:
row.label = provider.label
row.legal_name = provider.legal_name
row.tax_id = provider.tax_id
row.taxonomy_code = provider.taxonomy_code
row.address_line1 = provider.address_line1
row.address_line2 = provider.address_line2
row.city = provider.city
row.state = provider.state
row.zip = provider.zip
row.is_active = 1 if provider.is_active else 0
row.updated_at = now
s.commit()
return get_provider(provider.npi) # type: ignore[return-value]
def list_payers(*, is_active: bool | None = True) -> list[Payer]:
from cyclone.db import Payer as PayerORM
from cyclone.providers import Payer
with db.SessionLocal()() as s:
q = s.query(PayerORM)
if is_active is not None:
q = q.filter(PayerORM.is_active == (1 if is_active else 0))
rows = q.order_by(PayerORM.payer_id).all()
return [Payer.model_validate(_payer_orm_to_dict(r)) for r in rows]
def get_payer_config(payer_id: str, transaction_type: str) -> dict | None:
from cyclone.db import PayerConfigORM
with db.SessionLocal()() as s:
row = s.get(PayerConfigORM, (payer_id, transaction_type))
return dict(row.config_json) if row else None
def get_clearhouse() -> Clearhouse | None:
from cyclone.db import ClearhouseORM
from cyclone.providers import Clearhouse
with db.SessionLocal()() as s:
row = s.get(ClearhouseORM, 1)
if row is None:
return None
return Clearhouse.model_validate({
"id": 1,
"name": row.name,
"tpid": row.tpid,
"submitter_name": row.submitter_name,
"submitter_id_qual": row.submitter_id_qual,
"submitter_contact_name": row.submitter_contact_name,
"submitter_contact_email": row.submitter_contact_email,
"filename_block": dict(row.filename_block_json),
"sftp_block": dict(row.sftp_block_json),
"updated_at": row.updated_at,
})
def update_clearhouse(block: Clearhouse) -> Clearhouse:
"""Replace the singleton clearhouse row. SP25.
Used by ``PATCH /api/clearhouse`` to flip ``sftp_block.stub``
and adjust host/port/paths without touching SQLite directly.
Raises ``LookupError`` if the singleton row is missing the
caller is expected to run the lifespan seed first.
"""
from cyclone.db import ClearhouseORM
with db.SessionLocal()() as s:
row = s.get(ClearhouseORM, 1)
if row is None:
raise LookupError(
"clearhouse singleton row missing; run the lifespan "
"seed (ensure_clearhouse_seeded) before PATCH /api/clearhouse"
)
row.name = block.name
row.tpid = block.tpid
row.submitter_name = block.submitter_name
row.submitter_id_qual = block.submitter_id_qual
row.submitter_contact_name = block.submitter_contact_name
row.submitter_contact_email = block.submitter_contact_email
row.filename_block_json = json.loads(
json.dumps(block.filename_block.model_dump())
)
row.sftp_block_json = json.loads(
json.dumps(block.sftp_block.model_dump())
)
row.updated_at = datetime.now(timezone.utc).isoformat()
s.commit()
return Clearhouse.model_validate({
"id": 1,
"name": row.name,
"tpid": row.tpid,
"submitter_name": row.submitter_name,
"submitter_id_qual": row.submitter_id_qual,
"submitter_contact_name": row.submitter_contact_name,
"submitter_contact_email": row.submitter_contact_email,
"filename_block": dict(row.filename_block_json),
"sftp_block": dict(row.sftp_block_json),
"updated_at": row.updated_at,
})
def ensure_clearhouse_seeded() -> None:
"""Insert the default clearhouse singleton + 3 providers + CO_TXIX payer
if they don't exist. Idempotent. Called from the API lifespan."""
from cyclone.db import ClearhouseORM, Payer as PayerORM, PayerConfigORM, Provider as ProviderORM
from cyclone.providers import Clearhouse
with db.SessionLocal()() as s:
if s.get(ClearhouseORM, 1) is None:
ch = Clearhouse(
id=1,
name="dzinesco",
tpid="11525703",
submitter_name="Dzinesco",
submitter_id_qual="46",
submitter_contact_name="Tyler Martinez",
submitter_contact_email="tyler@dzinesco.com",
filename_block={
"tz": "America/Denver",
"outbound_template": "tp{tpid}-{tx}-{ts_mt}-1of1.{ext}",
"inbound_template": "TP{tpid}-{orig_tx}_M{tracking}-{ts}-1of1_{file_type}.x12",
},
sftp_block={
"host": "mft.gainwelltechnologies.com",
"port": 22,
"username": "colorado-fts\\coxix_prod_11525703",
"paths": {
"outbound": "/CO XIX/PROD/coxix_prod_11525703/ToHPE",
"inbound": "/CO XIX/PROD/coxix_prod_11525703/FromHPE",
},
"stub": True,
"staging_dir": "./var/sftp/staging",
"poll_seconds": 300,
"auth": {"method": "keychain", "secret_ref": "sftp.gainwell.password"},
},
updated_at=utcnow(),
)
s.add(ClearhouseORM(
id=1,
name=ch.name,
tpid=ch.tpid,
submitter_name=ch.submitter_name,
submitter_id_qual=ch.submitter_id_qual,
submitter_contact_name=ch.submitter_contact_name,
submitter_contact_email=ch.submitter_contact_email,
filename_block_json=ch.filename_block.model_dump(),
sftp_block_json=ch.sftp_block.model_dump(),
updated_at=ch.updated_at.isoformat(),
))
# Seed 3 providers (idempotent)
from cyclone.providers import Provider
now = utcnow().isoformat()
for npi, label in [
("1881068062", "Montrose"),
("1851446637", "Delta"),
("1467507269", "Salida"),
]:
if s.get(ProviderORM, npi) is None:
s.add(ProviderORM(
npi=npi,
label=label,
legal_name="TOC, Inc.",
tax_id="721587149",
taxonomy_code="251E00000X",
address_line1="1100 East Main St",
address_line2="Suite A",
city="Montrose",
state="CO",
zip="814014063",
is_active=1,
created_at=now,
updated_at=now,
))
# Seed CO_TXIX payer (idempotent)
if s.get(PayerORM, "CO_TXIX") is None:
s.add(PayerORM(
payer_id="CO_TXIX",
name="Colorado Medical Assistance Program",
receiver_name="COLORADO MEDICAL ASSISTANCE PROGRAM",
receiver_id="COMEDASSISTPROG",
is_active=1,
created_at=now,
updated_at=now,
))
# 837P config block
s.add(PayerConfigORM(
payer_id="CO_TXIX",
transaction_type="837P",
config_json={
"submitter_name": "Dzinesco",
"submitter_contact_name": "Tyler Martinez",
"submitter_contact_email": "tyler@dzinesco.com",
"receiver_name": "COLORADO MEDICAL ASSISTANCE PROGRAM",
"receiver_id_qualifier": "46",
"receiver_id": "COMEDASSISTPROG",
"bht06_allowed": ["CH", "RP"],
"bht06_default": "CH",
"sbr09_default": "MC",
"sbr09_allowed": ["MC", "16", "MA", "MB", "ZZ"],
"payer_id_qualifier": "PI",
"payer_id": "CO_TXIX",
"pwk_supported": False,
"cas_2320_group_allowed": False,
"claim_type_codes": {"11": "Office", "12": "Home", "99": "Other"},
},
updated_at=now,
))
# 835 config block
s.add(PayerConfigORM(
payer_id="CO_TXIX",
transaction_type="835",
config_json={
"expected_payer_tax_ids": [
"81-1725341", "811725341", "84-0644739",
"840644739", "1811725341",
],
"expected_payer_health_plan_id": "7912900843",
"payer_name_pattern": "^CO_(TXIX|BHA)$",
},
updated_at=now,
))
s.commit()
def _provider_orm_to_dict(row) -> dict:
return {
"npi": row.npi,
"label": row.label,
"legal_name": row.legal_name,
"tax_id": row.tax_id,
"taxonomy_code": row.taxonomy_code,
"address_line1": row.address_line1,
"address_line2": row.address_line2,
"city": row.city,
"state": row.state,
"zip": row.zip,
"is_active": bool(row.is_active),
"created_at": row.created_at,
"updated_at": row.updated_at,
}
def _payer_orm_to_dict(row) -> dict:
return {
"payer_id": row.payer_id,
"name": row.name,
"receiver_name": row.receiver_name,
"receiver_id": row.receiver_id,
"is_active": bool(row.is_active),
"created_at": row.created_at,
"updated_at": row.updated_at,
}
+84
View File
@@ -0,0 +1,84 @@
"""Pydantic models for parsed-batch records.
``BatchRecord`` is the union type; ``BatchRecord837`` and ``BatchRecord835``
narrow ``result`` to the parser-specific output types. Construction of
``BatchRecord(kind="837p", ...)`` dispatches to ``BatchRecord837`` via
``__new__`` so isinstance checks downstream narrow ``result`` correctly.
"""
from __future__ import annotations
from datetime import datetime
from typing import Any, Literal
from pydantic import BaseModel, ConfigDict, model_validator
from cyclone.parsers.models import ParseResult
from cyclone.parsers.models_835 import ParseResult835
BatchKind = Literal["837p", "835"]
# ---------------------------------------------------------------------------
# BatchRecord: value object preserved from sub-project 1.
# ---------------------------------------------------------------------------
class BatchRecord(BaseModel):
"""One parsed file, with a stable uuid4 id and the full ParseResult.
``result`` is a union: ``ParseResult`` for ``kind="837p"`` and
``ParseResult835`` for ``kind="835"``. The concrete subclasses
``BatchRecord837`` and ``BatchRecord835`` narrow those fields, so
callers that want type-checked access should use them and check
``isinstance`` rather than pattern-matching on ``kind``.
Constructing ``BatchRecord(kind="837p", ...)`` dispatches to
``BatchRecord837``; ``kind="835"`` dispatches to ``BatchRecord835``.
This lets the union-member narrowing work transparently.
"""
model_config = ConfigDict(extra="ignore")
id: str
kind: BatchKind
input_filename: str
parsed_at: datetime # tz-aware UTC
result: ParseResult | ParseResult835
def __new__(
cls, *args: Any, **kwargs: Any,
) -> BatchRecord837 | BatchRecord835 | BatchRecord:
# Dispatch base-class construction to the right concrete subclass
# so isinstance checks downstream narrow `result` correctly.
if cls is BatchRecord:
kind = kwargs.get("kind")
if kind is None and args and isinstance(args[0], dict):
kind = args[0].get("kind")
if kind == "837p":
return BatchRecord837(*args, **kwargs)
if kind == "835":
return BatchRecord835(*args, **kwargs)
return super().__new__(cls)
@model_validator(mode="after")
def _check_parsed_at_tz(self) -> BatchRecord:
if self.parsed_at.tzinfo is None:
raise ValueError(
"parsed_at must be tz-aware (use datetime.now(timezone.utc))"
)
return self
class BatchRecord837(BatchRecord):
"""A parsed 837P (professional claim) batch."""
kind: Literal["837p"] = "837p"
result: ParseResult
class BatchRecord835(BatchRecord):
"""A parsed 835 (remittance advice) batch."""
kind: Literal["835"] = "835"
result: ParseResult835
+683
View File
@@ -0,0 +1,683 @@
"""UI serialization helpers — convert ORM rows to API-shaped dicts.
These are the boundary between the persistence layer and the wire format
consumed by the React frontend. Keep them stable: any rename or
shape change ripples to every API consumer.
Also hosts ``utcnow()`` (the tz-aware UTC now) because it's a small
free function that several modules want and there's no better home.
SP25: ``to_ui_ack`` / ``to_ui_ta1_ack`` / ``to_ui_two77ca_ack`` were
moved here from ``api_routers/acks.py`` / ``api_routers/ta1_acks.py``
/ ``api.py`` so the live-tail event payload matches the list endpoint
shape byte-for-byte. The seam between persistence and streaming
depends on the two halves staying in sync.
SP28 adds ``to_ui_claim_ack`` for the same reason the
``claim_ack_written`` event payload must match the
``GET /api/acks/{kind}/{id}/claims`` list shape byte-for-byte.
moved here from ``api_routers/acks.py`` / ``api_routers/ta1_acks.py``
/ ``api.py`` so the live-tail event payload can match the list endpoint
shape byte-for-byte. The seam between persistence and streaming
depends on the two halves staying in sync.
"""
from __future__ import annotations
from datetime import datetime, timezone
from decimal import Decimal
from cyclone import db
from cyclone.db import Claim, ClaimAck, Remittance
from cyclone.parsers.models import ClaimOutput
from cyclone.parsers.models_835 import ClaimPayment
from cyclone.parsers.payer import PayerConfig835
from .orm_builders import _claim_status_from_validation
def to_ui_claim(
claim: ClaimOutput,
*,
batch_id: str,
parsed_at: datetime,
) -> dict:
"""Map a 837P ClaimOutput to the UI's `Claim` shape (preserved)."""
parsed_iso = parsed_at.isoformat().replace("+00:00", "Z")
return {
"id": claim.claim_id,
"patientName": f"{claim.subscriber.first_name} {claim.subscriber.last_name}".strip(),
"providerNpi": claim.billing_provider.npi,
"payerName": claim.payer.name,
"cptCode": (
claim.service_lines[0].procedure.code
if claim.service_lines
else ""
),
"billedAmount": float(claim.claim.total_charge or 0.0),
"receivedAmount": 0.0,
"status": _claim_status_from_validation(claim),
"denialReason": None,
"submissionDate": parsed_iso,
"batchId": batch_id,
"parsedAt": parsed_iso,
}
def to_ui_remittance(
cp: ClaimPayment,
*,
batch_id: str,
parsed_at: datetime,
payer_config: PayerConfig835 | None = None,
payer_name: str = "",
) -> dict:
"""Map an 835 ClaimPayment to the UI's `Remittance` shape (preserved)."""
code = cp.status_code
if code in {"21", "22"}:
status = "reconciled"
else:
status = "received"
denial_reason: str | None = None
if code == "4" and cp.service_payments:
sp = cp.service_payments[0]
if sp.adjustments:
adj = sp.adjustments[0]
denial_reason = (
f"{adj.group_code}-{adj.reason_code}: ${float(adj.amount):.2f}"
)
cfg = payer_config if payer_config is not None else PayerConfig835.generic_835()
validation_warnings: list[str] = []
if code not in cfg.allowed_status_codes:
validation_warnings.append(
f"CLP02 code {code} not in payer allowlist"
)
# Aggregate adjustmentAmount across ALL service-line CAS rows, not just
# the first line. Mirrors the SUM the reconcile aggregator (T10)
# computes against persisted CasAdjustment rows; this inline version
# is the write-path equivalent (used when streaming 835 NDJSON
# responses before persistence finishes).
adjustment_total = Decimal("0")
for sp in cp.service_payments:
for adj in sp.adjustments:
adjustment_total += adj.amount
parsed_iso = parsed_at.isoformat().replace("+00:00", "Z")
return {
"id": cp.payer_claim_control_number,
"claimId": cp.original_claim_id or "",
"payerName": payer_name,
"paidAmount": float(cp.total_paid or 0.0),
"adjustmentAmount": float(adjustment_total),
"status": status,
"denialReason": denial_reason,
"validationWarnings": validation_warnings,
"receivedDate": parsed_iso,
"batchId": batch_id,
"parsedAt": parsed_iso,
}
def to_ui_claim_from_orm(
row: Claim,
*,
batch_id: str,
parsed_at: datetime,
received_total: float = 0.0,
) -> dict:
"""Map an ORM ``Claim`` row to the UI's claim shape.
``to_ui_claim`` takes a Pydantic ``ClaimOutput`` (used on the write path
during 837 ingest). For read paths list_unmatched, manual_match return
values we already have the ORM row and the serialized fields it
carries in ``raw_json``. Reading from ``raw_json`` keeps the UI shape
in sync with the original 837 parse without re-deserializing to a
Pydantic model.
Adds two fields ``to_ui_claim`` doesn't emit: ``state`` (the
reconciliation state machine value) and ``matchedRemittanceId`` (the
FK to the paired remittance, or None). Both are required by the UI.
"""
raw = row.raw_json or {}
bp = raw.get("billing_provider", {})
payer_obj = raw.get("payer", {})
sub = raw.get("subscriber", {})
service_lines = raw.get("service_lines", [])
parsed_iso = parsed_at.isoformat().replace("+00:00", "Z")
cpt = (
service_lines[0].get("procedure", {}).get("code", "")
if service_lines
else ""
)
state_value = (
row.state.value if hasattr(row.state, "value") else str(row.state)
)
return {
"id": row.id,
"state": state_value,
"billedAmount": float(row.charge_amount or 0),
"patientName": (
f"{sub.get('first_name', '')} {sub.get('last_name', '')}".strip()
),
"providerNpi": bp.get("npi") or row.provider_npi or "",
"payerName": payer_obj.get("name") or "",
"cptCode": cpt,
"submissionDate": parsed_iso,
"parsedAt": parsed_iso,
"status": state_value,
"matchedRemittanceId": row.matched_remittance_id,
"batchId": batch_id,
# Parity with ``to_ui_claim``'s shape — the UI tolerates extra keys
# but expects these on freshly-loaded rows from /api/claims too.
# ``received_total`` comes from the matched Remittance row when one
# exists; callers that don't pre-compute it (write path, unmatched
# list) get the default of 0.0 — which matches the unmapped state.
"receivedAmount": float(received_total),
"denialReason": None,
}
# Max number of ActivityEvent rows surfaced in the detail drawer's
# state history. The spec caps it at 50; a higher claim volume (manual
# match/unmatch thrash) just shows the 50 most recent. Exposed as a
# module constant so the endpoint layer can pass it through as a
# default if it ever supports a `?limit=N` query param.
CLAIM_DETAIL_HISTORY_LIMIT = 50
def _iso_z(value: datetime | None) -> str:
"""Format a tz-aware-or-naive UTC datetime as ISO-8601 with trailing Z.
The DB columns are declared ``DateTime(timezone=True)`` and rows are
stored UTC at write time, but SQLite drops the tzinfo on read
(returning a naive ``datetime``). Re-attach UTC for naive values
so the spec contract holds: every ISO datetime field ends in Z.
"""
if value is None:
return ""
if value.tzinfo is None:
value = value.replace(tzinfo=timezone.utc)
return value.isoformat().replace("+00:00", "Z")
def to_ui_ack(row: db.Ack) -> dict:
"""Map an ``Ack`` ORM row to the UI shape used by ``/api/acks``.
Field names match the rest of the Cyclone API (snake_case). The
frontend ``useAcks`` hook re-shapes this to the camelCase ``Ack``
interface in ``src/types/index.ts``.
Adds ``patient_control_number`` pulled from ``raw_json`` so the
operator can correlate each 999 back to the original claim batch.
SP25: this was previously ``api_routers/acks._ack_to_ui``. Moved
here so the live-tail event payload (``ack_received``) matches the
list-endpoint shape the seam between persistence and streaming
depends on byte-for-byte equality.
"""
body = {
"id": row.id,
"source_batch_id": row.source_batch_id,
"accepted_count": row.accepted_count,
"rejected_count": row.rejected_count,
"received_count": row.received_count,
"ack_code": row.ack_code,
"parsed_at": (
row.parsed_at.isoformat().replace("+00:00", "Z")
if row.parsed_at is not None
else ""
),
"patient_control_number": None,
}
raw = row.raw_json or {}
try:
set_responses = raw.get("set_responses") or []
if set_responses:
body["patient_control_number"] = set_responses[0].get("set_control_number")
except (AttributeError, TypeError):
pass
return body
def to_ui_ta1_ack(row: db.Ta1Ack) -> dict:
"""Map a ``Ta1Ack`` ORM row to the UI shape used by ``/api/ta1-acks``.
SP25: this was previously ``api_routers/ta1_acks._ta1_to_ui``.
Moved here so the live-tail event payload (``ta1_ack_received``)
matches the list-endpoint shape byte-for-byte.
Note: ``parsed_at`` uses naive ``isoformat()`` (no ``Z`` suffix)
preserved from the original implementation. SQLite strips tzinfo
on read, so the field comes back as a naive datetime; we deliberately
keep the original (non-``Z``) rendering rather than re-attaching
UTC, because changing the wire shape now would drift both halves
of the live-tail contract.
"""
return {
"id": row.id,
"control_number": row.control_number,
"ack_code": row.ack_code,
"note_code": row.note_code,
"interchange_date": row.interchange_date.isoformat()
if row.interchange_date else None,
"interchange_time": row.interchange_time,
"sender_id": row.sender_id,
"receiver_id": row.receiver_id,
"source_batch_id": row.source_batch_id,
"parsed_at": row.parsed_at.isoformat() if row.parsed_at else None,
}
def to_ui_two77ca_ack(row: db.Two77caAck) -> dict:
"""Map a ``Two77caAck`` ORM row to the UI shape used by ``/api/277ca-acks``.
SP25: this was previously ``api._277ca_to_ui`` inlined alongside
the 277CA list endpoint. Moved here so the live-tail event
payload (``two77ca_ack_received``) matches the list-endpoint
shape byte-for-byte.
Note: ``parsed_at`` uses naive ``isoformat()`` (no ``Z`` suffix)
same SQLite-tzinfo caveat as :func:`to_ui_ta1_ack`.
"""
return {
"id": row.id,
"control_number": row.control_number,
"accepted_count": row.accepted_count,
"rejected_count": row.rejected_count,
"paid_count": row.paid_count,
"pended_count": row.pended_count,
"source_batch_id": row.source_batch_id,
"parsed_at": row.parsed_at.isoformat() if row.parsed_at else None,
}
def _address_to_ui(addr: dict | None) -> dict:
"""Render a raw ``Address`` dict in the spec's parties address shape.
Returns an empty dict when the source is missing so the UI can
branch on the field's presence rather than the value. The spec
shape is ``{line1, line2|null, city, state, zip}``.
"""
if not addr:
return {}
return {
"line1": addr.get("line1") or "",
"line2": addr.get("line2"),
"city": addr.get("city") or "",
"state": addr.get("state") or "",
"zip": addr.get("zip") or "",
}
def _validation_issues_to_ui(issues: list[dict] | None) -> list[dict]:
"""Project ValidationIssue dicts onto the spec's per-issue shape.
Source includes ``segment_index`` (a parser debug aid) which the
spec doesn't surface; we drop it. The endpoint contract is
``{rule, severity, message}`` per issue.
"""
if not issues:
return []
return [
{
"rule": issue.get("rule", ""),
"severity": issue.get("severity", "error"),
"message": issue.get("message", ""),
}
for issue in issues
]
def to_ui_claim_detail(
row: Claim,
*,
batch_id: str,
parsed_at: datetime,
) -> dict:
"""Map an ORM ``Claim`` row to the SP4 detail-drawer UI shape.
A superset of :func:`to_ui_claim_from_orm`: same top-level identity
fields, plus the full parties / validation / service-lines /
diagnoses / raw-segments / service-date / state-label payload that
the drawer needs. ``matchedRemittance`` and ``stateHistory`` are
*not* filled in here they require extra queries and are stitched
in by :meth:`CycloneStore.get_claim_detail`.
The mapper is deliberately a pure function (no DB I/O) so the
endpoint layer can call it from a worker thread or swap the
history/remittance sources for tests without re-implementing the
body.
"""
raw = row.raw_json or {}
bp = raw.get("billing_provider", {}) or {}
payer_obj = raw.get("payer", {}) or {}
sub = raw.get("subscriber", {}) or {}
service_lines = raw.get("service_lines", []) or []
diagnoses = raw.get("diagnoses", []) or []
validation = raw.get("validation", {}) or {}
raw_segments = raw.get("raw_segments", []) or []
parsed_iso = parsed_at.isoformat().replace("+00:00", "Z")
state_value = (
row.state.value if hasattr(row.state, "value") else str(row.state)
)
# Service dates come from the dedicated ORM columns (denormalized at
# ingest in _claim_837_row so the list views can sort/filter on
# them without a JSON parse). ``isoformat()`` on a ``date`` gives
# ``YYYY-MM-DD`` — the spec shape.
service_date_from_iso = (
row.service_date_from.isoformat() if row.service_date_from else None
)
service_date_to_iso = (
row.service_date_to.isoformat() if row.service_date_to else None
)
return {
# -- identity + state -----------------------------------------
"id": row.id,
"batchId": batch_id,
"state": state_value,
"stateLabel": state_value.capitalize(),
# -- money + dates --------------------------------------------
"billedAmount": float(row.charge_amount or 0),
"serviceDateFrom": service_date_from_iso,
"serviceDateTo": service_date_to_iso,
"submissionDate": parsed_iso,
"parsedAt": parsed_iso,
# -- patient / provider / payer -------------------------------
"patientName": (
f"{sub.get('first_name', '')} {sub.get('last_name', '')}".strip()
),
"providerNpi": bp.get("npi") or row.provider_npi or "",
"providerName": bp.get("name") or "",
"payerName": payer_obj.get("name") or "",
"payerId": payer_obj.get("id") or row.payer_id or "",
# -- diagnoses ------------------------------------------------
"diagnoses": [
{
"code": d.get("code", ""),
"qualifier": d.get("qualifier"),
}
for d in diagnoses
],
# -- service lines --------------------------------------------
# ``service_lines[i].procedure`` is a nested dict in the
# serialized raw_json; the spec flattens it into the line.
# ``charge`` and ``units`` are stored as Decimal via Pydantic
# and serialized to string — coerce defensively. ``modifiers``
# defaults to [] so the UI doesn't have to handle null.
"serviceLines": [
{
"lineNumber": sl.get("line_number"),
"procedureQualifier": (
sl.get("procedure", {}).get("qualifier", "") or ""
),
"procedureCode": (
sl.get("procedure", {}).get("code", "") or ""
),
"modifiers": list(
sl.get("procedure", {}).get("modifiers") or []
),
"charge": float(sl.get("charge") or 0),
"units": (
float(sl["units"])
if sl.get("units") is not None
else None
),
"unitType": sl.get("unit_type"),
"serviceDate": sl.get("service_date"),
}
for sl in service_lines
],
# -- parties --------------------------------------------------
"parties": {
"billingProvider": {
"name": bp.get("name") or "",
"npi": bp.get("npi") or "",
"taxId": bp.get("tax_id") or "",
"address": _address_to_ui(bp.get("address")),
},
"subscriber": {
"firstName": sub.get("first_name") or "",
"lastName": sub.get("last_name") or "",
"memberId": sub.get("member_id") or "",
"dob": sub.get("dob"),
"gender": sub.get("gender"),
},
"payer": {
"name": payer_obj.get("name") or "",
"id": payer_obj.get("id") or "",
},
},
# -- validation ----------------------------------------------
"validation": {
"passed": bool(validation.get("passed", True)),
"errors": _validation_issues_to_ui(validation.get("errors")),
"warnings": _validation_issues_to_ui(validation.get("warnings")),
},
# -- raw segments (debug aid) --------------------------------
"rawSegments": raw_segments,
# -- matched remittance (filled by get_claim_detail) ---------
"matchedRemittance": None,
# -- state history (filled by get_claim_detail) --------------
"stateHistory": [],
}
def to_ui_remittance_from_orm(
row: Remittance,
*,
batch_id: str,
parsed_at: datetime,
) -> dict:
"""Map an ORM ``Remittance`` row to the UI's remittance shape.
Same idea as ``to_ui_claim_from_orm``: read the PayerName from the
parent batch's ``raw_result_json`` (the ParseResult835 stashed at
insert time) since ``Remittance`` itself doesn't carry it.
"""
parsed_iso = parsed_at.isoformat().replace("+00:00", "Z")
payer_name = ""
if row.batch is not None and row.batch.raw_result_json:
payer_obj = row.batch.raw_result_json.get("payer", {}) or {}
payer_name = payer_obj.get("name") or ""
status = (
"reconciled" if row.status_code in ("21", "22") else "received"
)
return {
"id": row.id,
"payerClaimControlNumber": row.payer_claim_control_number,
"claimId": row.claim_id or "",
"payerName": payer_name,
"paidAmount": float(row.total_paid or 0),
"adjustmentAmount": float(row.adjustment_amount or 0),
"status": status,
"denialReason": None,
"validationWarnings": [],
"receivedDate": row.received_at.isoformat().replace("+00:00", "Z"),
"batchId": batch_id,
"parsedAt": parsed_iso,
}
def to_ui_remittance_with_adjustments(
row: Remittance,
*,
batch_id: str,
parsed_at: datetime,
cas_rows: list["db.CasAdjustment"] | None = None,
) -> dict:
"""Same shape as :func:`to_ui_remittance_from_orm` plus an ``adjustments`` array.
Each persisted ``CasAdjustment`` row is rendered as
``{"group", "reason", "label", "amount", "quantity"}``. The ``label``
is resolved through :func:`cyclone.parsers.cas_codes.reason_label`
so the UI does not have to ship its own CARC dictionary.
``cas_rows`` is optional so callers that don't have the rows handy
can still get the base dict; in that case ``adjustments`` is ``[]``.
Pass ``cas_rows`` to avoid an extra round-trip; the endpoint at
``GET /api/remittances/{id}`` passes them in to keep this mapper a
pure function.
"""
base = to_ui_remittance_from_orm(
row, batch_id=batch_id, parsed_at=parsed_at,
)
if not cas_rows:
base["adjustments"] = []
return base
# Lazy import to avoid the circular store ↔ parsers import that
# happens on cold start; mirrors the same pattern used elsewhere
# in this module.
from cyclone.parsers.cas_codes import reason_label
base["adjustments"] = [
{
"group": c.group_code,
"reason": c.reason_code,
"label": reason_label(c.group_code, c.reason_code),
"amount": float(c.amount or 0),
"quantity": (
float(c.quantity) if c.quantity is not None else None
),
}
for c in cas_rows
]
return base
def _svc_to_wire_dict(svc) -> dict:
"""Project an ORM ``ServiceLinePayment`` to the wire format used by
the remit drawer's ``serviceLinePayments`` array.
Mirrors the shape produced by the line-reconciliation endpoint so
the UI can render the same components from either source.
"""
import json as _json
return {
"id": svc.id,
"line_number": svc.line_number,
"procedure_qualifier": svc.procedure_qualifier,
"procedure_code": svc.procedure_code,
"modifiers": _json.loads(svc.modifiers_json or "[]"),
"charge": str(Decimal(str(svc.charge))),
"payment": str(Decimal(str(svc.payment))),
"units": str(Decimal(str(svc.units))) if svc.units is not None else None,
"unit_type": svc.unit_type,
"service_date": svc.service_date.isoformat() if svc.service_date else None,
}
def to_ui_provider(
*,
npi: str,
name: str,
tax_id: str | None = None,
address: str | None = None,
city: str | None = None,
state: str | None = None,
zip: str | None = None,
phone: str | None = None,
claim_count: int = 0,
outstanding_ar: float = 0.0,
) -> dict:
return {
"npi": npi,
"name": name,
"taxId": tax_id or "",
"address": address or "",
"city": city or "",
"state": state or "",
"zip": zip or "",
"phone": phone or "",
"claimCount": claim_count,
"outstandingAr": float(outstanding_ar),
}
def to_activity_event(
*,
id: str,
kind: str,
message: str,
timestamp: datetime,
npi: str | None = None,
amount: float | None = None,
) -> dict:
return {
"id": id,
"kind": kind,
"message": message,
"timestamp": timestamp.isoformat().replace("+00:00", "Z"),
"npi": npi,
"amount": amount,
}
def _date_in_bounds(
item: dict,
field: str,
date_from: str | None,
date_to: str | None,
) -> bool:
"""True if ``item[field]`` falls within ``[date_from, date_to]``."""
val = item.get(field)
if val is None:
return date_from is None and date_to is None
date_part = val[:10]
if date_from is not None and date_part < date_from:
return False
if date_to is not None and date_part > date_to:
return False
return True
def to_ui_claim_ack(row: ClaimAck) -> dict:
"""SP28: map a ClaimAck ORM row to the API shape.
Mirrors the rest of the to_ui_* serializers. The wire shape is
identical between the matching list endpoint (``GET /api/acks/{kind}/
{id}/claims``) and the ``claim_ack_written`` pubsub event so the
live-tail subscribers can rehydrate the snapshot from the bus
without diverging from the API response.
``claim_state`` is queried via a session round-trip
(``SELECT state FROM claims WHERE id = :claim_id``) so the drawer
panel can render the colored ClaimStateBadge inline. For TA1 rows
with ``claim_id IS NULL`` (batch-level envelope link),
``claim_state`` is ``"n/a"``.
"""
claim_state: str = "n/a"
if row.claim_id:
with db.SessionLocal()() as lookup_s:
crow = lookup_s.get(Claim, row.claim_id)
if crow is not None:
claim_state = (
crow.state.value
if hasattr(crow.state, "value")
else str(crow.state)
)
linked_iso = (
row.linked_at.isoformat().replace("+00:00", "Z")
if row.linked_at is not None
else ""
)
return {
"id": row.id,
"claim_id": row.claim_id,
"batch_id": row.batch_id,
"ack_id": row.ack_id,
"ack_kind": row.ack_kind,
"ak2_index": row.ak2_index,
"set_control_number": row.set_control_number,
"set_accept_reject_code": row.set_accept_reject_code,
"linked_at": linked_iso,
"linked_by": row.linked_by,
"claim_state": claim_state,
}
+252
View File
@@ -0,0 +1,252 @@
"""Write path for parsed batches — insert, reconcile, publish.
The single entry point is ``add_record(record, *, event_bus=None)``,
which replaces the body of the previous ``CycloneStore.add`` method.
It owns its own SQLAlchemy session, runs idempotency checks, persists
the batch + child rows, then (for 835 batches) triggers reconciliation
and (if ``event_bus`` is provided) publishes live-tail events.
Reconciliation runs OUTSIDE the persistence session, fail-soft: errors
are logged but do not roll back the persisted batch.
"""
from __future__ import annotations
import json
from cyclone import db
from cyclone.db import (
ActivityEvent,
Batch,
Claim,
Remittance,
)
from cyclone.parsers.models import ParseResult
from cyclone.parsers.models_835 import ParseResult835
from .orm_builders import _claim_837_row, _persist_835_remit, _remittance_835_row
from .records import BatchRecord, BatchRecord837, BatchRecord835
from .ui import to_ui_claim_from_orm, to_ui_remittance_from_orm
def add_record(record: BatchRecord, *, event_bus=None) -> None:
"""Persist a parsed batch (837P or 835) to the DB.
For 837P batches: inserts the Batch row, one Claim row per
claim, and a ``claim_submitted`` ActivityEvent per claim.
For 835 batches: inserts the Batch row, one Remittance row per
ClaimPayment, and a ``remit_received`` ActivityEvent per
ClaimPayment. Reconciliation (auto-match + per-pair CAS
aggregate) runs IN THE SAME SESSION before commit (SP27
Task 10) a single ``s.commit()`` covers both ingest and
reconciliation. If reconcile raises, the whole 835 ingest
rolls back; the batch never appears half-reconciled with
placeholder ``adjustment_amount`` values.
Idempotency: ``Claim.id`` and ``Remittance.id`` are PRIMARY KEYS,
so a re-ingest of the same fixture (e.g. ``/api/parse-837`` called
twice with the same file) would otherwise raise
``IntegrityError``. We do a per-row ``session.get(...)`` check
before each insert; if the row already exists, we log a warning
and skip. The batch row itself is still inserted (each parse
has a fresh ``uuid4`` id from the API). O(n) per row, but
acceptable for the small fixture sizes production load is
one batch at a time via the API, not bulk inserts.
When ``event_bus`` is provided, publishes one ``claim_written``
or ``remittance_written`` event per newly-inserted row plus an
``activity_recorded`` event per activity row, after commit. The
publish calls are best-effort failures are logged but do not
roll back the persisted batch.
"""
from cyclone.pubsub import EventBus
import logging
log = logging.getLogger(__name__)
# Track rows we actually inserted so we can publish events for them.
inserted_claim_ids: list[str] = []
inserted_remit_ids: list[str] = []
with db.SessionLocal()() as s:
batch_row = Batch(
id=record.id,
kind=record.kind,
input_filename=record.input_filename,
parsed_at=record.parsed_at,
totals_json=None,
validation_json=None,
raw_result_json=json.loads(record.result.model_dump_json()),
# SP37 Task 2: mirror the parsed 837's ST02 onto the batch
# row so 999 AK201 set_control_numbers can resolve back via
# Pass 1. The ``getattr`` chain handles the 835 path: the
# shared ``Envelope`` class is used by both 837P and 835
# parsers, but only ``parse_837`` populates this field — for
# 835 records it stays ``None`` and the column is NULL.
transaction_set_control_number=getattr(
getattr(record.result, "envelope", None),
"transaction_set_control_number",
None,
),
)
s.add(batch_row)
if isinstance(record, BatchRecord837):
result: ParseResult = record.result
for claim in result.claims:
if s.get(Claim, claim.claim_id) is not None:
log.warning(
"add: claim %s already exists; skipping (batch=%s)",
claim.claim_id, record.id,
)
continue
s.add(_claim_837_row(claim, record.id))
s.add(ActivityEvent(
ts=record.parsed_at,
kind="claim_submitted",
batch_id=record.id,
claim_id=claim.claim_id,
payload_json={
"message": (
f"Claim {claim.claim_id} submitted · "
f"{claim.payer.name}"
),
"npi": claim.billing_provider.npi,
"amount": float(claim.claim.total_charge or 0.0),
},
))
inserted_claim_ids.append(claim.claim_id)
elif isinstance(record, BatchRecord835):
result835: ParseResult835 = record.result
payer_name = result835.payer.name
for cp in result835.claims:
if s.get(Remittance, cp.payer_claim_control_number) is not None:
log.warning(
"add: remittance %s already exists; skipping (batch=%s)",
cp.payer_claim_control_number, record.id,
)
continue
remit_row = _remittance_835_row(cp, record.id)
s.add(remit_row)
# Flush so remit_row.id (FK target of cas_adjustments) is
# populated. SQLAlchemy assigns the PK on flush; without
# this the CasAdjustment inserts below would reference an
# unset id and violate the FK.
s.flush()
# SP7: persist per-line ServiceLinePayment + linked
# SVC-level CAS rows + claim-level CAS bucket. Replaces
# the previous per-SVC CAS insert loop so the
# service_line_payment_id FK is set correctly.
_persist_835_remit(s, cp, remit_row.id)
s.add(ActivityEvent(
ts=record.parsed_at,
kind="remit_received",
batch_id=record.id,
remittance_id=cp.payer_claim_control_number,
payload_json={
"message": (
f"Remit {cp.payer_claim_control_number} received"
),
"payerName": payer_name,
"amount": float(cp.total_paid or 0.0),
},
))
inserted_remit_ids.append(cp.payer_claim_control_number)
else:
raise TypeError(
f"Unsupported BatchRecord subclass: {type(record).__name__}"
)
# SP27 Task 10: run reconcile INSIDE the same session, before
# commit. The placeholder ``adjustment_amount`` set by
# ``_remittance_835_row`` is overwritten by reconcile's
# ``_reconcile_pair`` SUM-of-CAS-aggregate pass before the
# rows are ever visible. If reconcile raises, the whole
# 835 ingest rolls back and the batch never lands.
if record.kind == "835":
from cyclone import reconcile as _reconcile
_reconcile.run(s, record.id)
s.commit()
# Publish live-tail events synchronously. EventBus.publish is async
# but its body is purely synchronous ``put_nowait`` enqueues; we
# bypass the async wrapper and call the internal enqueue directly
# so callers (sync FastAPI endpoints, sync test harnesses) don't
# need to await.
if event_bus is not None and (inserted_claim_ids or inserted_remit_ids):
publish_events_sync(
event_bus, record, inserted_claim_ids, inserted_remit_ids,
)
def publish_events_sync(
event_bus,
record: BatchRecord,
claim_ids: list[str],
remit_ids: list[str],
) -> None:
"""Build UI-shaped payloads for newly-inserted rows and publish.
Runs after commit so subscribers can immediately re-fetch from
the API and see consistent data. Each ``claim_written`` /
``remittance_written`` payload is identical to what the matching
list endpoint would return for that row.
This is sync because EventBus's enqueue path is sync; we don't
need a coroutine for ``put_nowait``.
"""
from cyclone.pubsub import EventBus
import logging
log = logging.getLogger(__name__)
try:
with db.SessionLocal()() as s:
for cid in claim_ids:
row = s.get(Claim, cid)
if row is None:
continue
ui = to_ui_claim_from_orm(
row, batch_id=row.batch_id or record.id,
parsed_at=record.parsed_at,
# Fresh ingest — no remittance has been paired yet,
# so ``Received`` is necessarily 0.
received_total=0.0,
)
_sync_publish(event_bus, "claim_written", ui)
for rid in remit_ids:
row = s.get(Remittance, rid)
if row is None:
continue
ui = to_ui_remittance_from_orm(
row, batch_id=row.batch_id or record.id,
parsed_at=record.parsed_at,
)
_sync_publish(event_bus, "remittance_written", ui)
# Activity events for this batch.
from sqlalchemy import select
activity_rows = s.execute(
select(ActivityEvent).where(ActivityEvent.batch_id == record.id)
).scalars().all()
for arow in activity_rows:
ui = {
"kind": arow.kind,
"ts": arow.ts.isoformat().replace("+00:00", "Z"),
"batchId": arow.batch_id,
"claimId": arow.claim_id,
"remittanceId": arow.remittance_id,
"payload": arow.payload_json,
}
_sync_publish(event_bus, "activity_recorded", ui)
except Exception:
log.exception("add: event publish failed for batch %s", record.id)
def _sync_publish(event_bus, kind: str, payload: dict) -> None:
"""Synchronous fan-out helper. Mirrors ``EventBus.publish`` but
bypasses the async wrapper so callers don't need an event loop.
"""
event = {**payload, "_kind": kind}
for queue in list(event_bus._subscribers.get(kind, ())):
event_bus._enqueue_or_drop_oldest(queue, event)
@@ -0,0 +1,17 @@
"""SP37 Task 4: canonical 837P submission flow.
The single public helper is ``submit_file`` it owns parse DB
write SFTP upload per file. CLI (``cyclone submit-batch``) and HTTP
(``POST /api/submit-batch``) are thin wrappers; both call this helper
so the ordering, idempotency, and audit shape are identical.
DB-first, upload-second (per spec decision §2): if the DB write fails,
no SFTP call is made. If the SFTP call fails after a successful DB
write, the row exists but the file never landed re-running is safe
(idempotent DB write via add_record's s.get check; idempotent SFTP
upload via stat-then-put).
"""
from .core import submit_file
from .result import SubmitOutcome, SubmitResult
__all__ = ["submit_file", "SubmitOutcome", "SubmitResult"]
+225
View File
@@ -0,0 +1,225 @@
"""SP37 Task 4: parse → DB write → SFTP upload, per file.
Mirrors ``resubmit_rejected_claims`` but writes to the DB before
uploading. Same idempotency check (``stat().st_size == local_size``)
skips already-uploaded files without re-emitting audit events.
DB-first, upload-second (per spec decision §2): if the DB write fails,
no SFTP call is made. If the SFTP call fails after a successful DB
write, the row exists but the file never landed re-running is safe
(idempotent DB write via add_record's s.get check; idempotent SFTP
upload via stat-then-put).
The default SFTP factory uses paramiko directly (not the
``SftpClient`` wrapper) because the wrapper exposes ``write_file``,
``list_inbound``, and ``read_file`` but no ``stat()`` which makes
the SKIPPED outcome unreachable in production.
"""
from __future__ import annotations
import logging
import uuid
from datetime import datetime, timezone
from pathlib import Path
from typing import Any, Callable
from cyclone import db as db_mod
from cyclone.audit_log import AuditEvent, append_event
from cyclone.parsers.parse_837 import parse as parse_837_text
from cyclone.parsers.payer import PayerConfig
from cyclone.providers import SftpBlock
from cyclone.store import store as cycl_store
from cyclone.store.records import BatchRecord837
from .result import SubmitOutcome, SubmitResult
log = logging.getLogger(__name__)
# The companion-guide payer id we enforce for CO Medicaid submits.
# Same value the legacy ``resubmit-rejected-claims`` CLI gates on.
EXPECTED_PAYER_ID = "CO_TXIX"
def _default_sftp_factory(sftp_block: SftpBlock) -> Any:
"""Open a paramiko SFTP session to the real MFT. Mirrors
resubmit_rejected_claims._open_session (cli.py:620-640).
Caller is responsible for closing the session.
The returned ``sftp`` is a ``paramiko.SFTPClient`` it exposes
``stat(remote_path)`` (used for the idempotency check) and
``put(local_path, remote_path)`` (used for the upload). The
underlying SSH handle is stashed on ``sftp._cyclone_ssh`` so the
caller can close it cleanly via ``getattr(sftp, "_cyclone_ssh",
None).close()`` after the upload finishes.
Raises:
RuntimeError: if the SFTP block is in stub mode (the CLI/HTTP
layer should already guard against this, but we re-check
here because paramiko will try to connect even when
``stub=True``).
"""
if sftp_block.stub:
# Same posture as resubmit_rejected_claims in cli.py:592-595:
# the operator refuses to upload in stub mode, and the helper
# surfaces that as SFTP_FAILED with an explicit error.
raise RuntimeError("SFTP block is in stub mode")
from paramiko import AutoAddPolicy, SSHClient
from cyclone.secrets import get_secret
pw = get_secret(sftp_block.auth.get("password_keychain_account", ""))
ssh = SSHClient()
ssh.set_missing_host_key_policy(AutoAddPolicy())
ssh.connect(
sftp_block.host, port=sftp_block.port,
username=sftp_block.username, password=pw,
timeout=15, banner_timeout=15, auth_timeout=15,
)
sftp = ssh.open_sftp()
# Attach ssh to sftp so the caller can close it cleanly.
sftp._cyclone_ssh = ssh
return sftp
def submit_file(
path: Path,
*,
sftp_block: SftpBlock,
actor: str,
validate: bool = True,
sftp_client_factory: Callable[[SftpBlock], Any] | None = None,
) -> SubmitResult:
"""Submit one 837P file: parse → DB write → SFTP upload.
Args:
path: local 837P file.
sftp_block: the clearhouse SftpBlock (for paths + auth).
actor: audit-log actor tag (e.g. "api-submit-batch").
validate: parse the file before upload (default True). Catches
bad byte-fixes early.
sftp_client_factory: optional callable that returns an
SFTP-client-compatible object (must expose ``stat()`` and
``write_file()``). Defaults to :func:`_default_sftp_factory`
which opens a real paramiko session. Tests inject a fake.
Returns:
SubmitResult with file, outcome, batch_id (when written),
error (when failed).
"""
file_label = path.name
try:
content = path.read_bytes()
except OSError as exc:
return SubmitResult(file_label, SubmitOutcome.PARSE_FAILED, error=str(exc))
# 1. Validate via parse (optional but recommended).
parsed: Any = None
if validate:
try:
parsed = parse_837_text(content.decode(), PayerConfig.co_medicaid())
except Exception as exc: # noqa: BLE001
log.warning("submit_file %s: parse failed: %s", file_label, exc)
return SubmitResult(file_label, SubmitOutcome.PARSE_FAILED, error=str(exc))
mismatch = next(
(c for c in parsed.claims if c.payer.id != EXPECTED_PAYER_ID),
None,
)
if mismatch is not None:
return SubmitResult(
file_label,
SubmitOutcome.PAYER_MISMATCH,
error=f"payer.id={mismatch.payer.id!r} (expected {EXPECTED_PAYER_ID!r})",
)
# 2. DB write — DB-first, upload-second invariant.
# ``parsed`` is required to construct a BatchRecord837 (it embeds the
# full ParseResult). validate=False therefore isn't a real path —
# the CLI/HTTP always pass validate=True — so reject it loudly
# rather than silently building an empty row.
if parsed is None:
return SubmitResult(
file_label,
SubmitOutcome.PARSE_FAILED,
error="validate=False is not supported; must parse to construct a BatchRecord",
)
# Use the same uuid4().hex id the existing /api/parse-837 path uses.
batch_id = uuid.uuid4().hex
record = BatchRecord837(
id=batch_id,
input_filename=file_label,
parsed_at=datetime.now(timezone.utc),
result=parsed,
)
try:
# ``cycl_store.add`` is the public facade method (per the
# CycloneStore class in store/__init__.py:158). It delegates
# to ``store.write.add_record``, which is the underlying
# SQLAlchemy write path.
cycl_store.add(record)
except Exception as exc: # noqa: BLE001
log.warning("submit_file %s: DB write failed: %s", file_label, exc)
return SubmitResult(file_label, SubmitOutcome.DB_FAILED, error=str(exc))
# 3. SFTP upload. ``_default_sftp_factory`` opens a paramiko
# session directly because the SftpClient wrapper has no ``stat()``
# method — that omission made the SKIPPED outcome unreachable in
# production. Mirror cli.py:620-650.
remote_path = f"{sftp_block.paths['outbound']}/{file_label}"
factory = sftp_client_factory or _default_sftp_factory
sftp = factory(sftp_block)
local_size = len(content)
try:
try:
stat = sftp.stat(remote_path)
if stat.st_size == local_size:
# Already on remote at the right size — re-run is safe;
# do NOT emit a duplicate audit event.
return SubmitResult(file_label, SubmitOutcome.SKIPPED, batch_id=batch_id)
except (IOError, OSError):
pass # not on remote yet
sftp.write_file(remote_path, content)
except Exception as exc: # noqa: BLE001
log.warning("submit_file %s: SFTP failed: %s", file_label, exc)
return SubmitResult(
file_label, SubmitOutcome.SFTP_FAILED,
batch_id=batch_id, error=str(exc),
)
finally:
# Close the paramiko SSH handle the helper stashed on the
# SFTP client (paramiko does not auto-close on GC and a leak
# here costs a slot in MOVEit's per-IP session table).
ssh_handle = getattr(sftp, "_cyclone_ssh", None)
if ssh_handle is not None:
try:
ssh_handle.close()
except Exception: # noqa: BLE001
pass
# 4. Audit event — same shape the legacy resubmit_rejected_claims CLI
# uses (event_type="clearhouse.submitted", entity_type="claim_file",
# entity_id=filename, payload has remote_path + source + size).
# Best-effort: an audit failure must not roll back the upload.
try:
with db_mod.SessionLocal()() as session:
append_event(session, AuditEvent(
event_type="clearhouse.submitted",
entity_type="claim_file",
entity_id=file_label,
payload={
"remote_path": remote_path,
"source": "submit-batch",
"size": local_size,
"batch_id": batch_id,
},
actor=actor,
))
session.commit()
except Exception as exc: # noqa: BLE001
log.warning(
"submit_file %s: audit event failed: %s", file_label, exc,
)
return SubmitResult(file_label, SubmitOutcome.SUBMITTED, batch_id=batch_id)
+40
View File
@@ -0,0 +1,40 @@
"""SP37 Task 4: typed result for submit_file.
The CLI and HTTP layer both consume SubmitResult; keeping it in one
place means both surfaces agree on the response shape.
"""
from __future__ import annotations
from dataclasses import dataclass
from enum import Enum
class SubmitOutcome(str, Enum):
"""Outcome of a single submit_file invocation.
Audit-event invariant: ``SUBMITTED`` emits a ``clearhouse.submitted``
audit event; ``SKIPPED`` does NOT (re-running on an already-uploaded
file must not flood the audit log with duplicate events). All
failure outcomes (``PARSE_FAILED``, ``PAYER_MISMATCH``, ``DB_FAILED``,
``SFTP_FAILED``) emit no event either failures should be observable
via the helper's return value, not the audit log.
"""
SUBMITTED = "submitted"
SKIPPED = "skipped"
PARSE_FAILED = "parse_failed"
PAYER_MISMATCH = "payer_mismatch"
DB_FAILED = "db_failed"
SFTP_FAILED = "sftp_failed"
@dataclass(frozen=True)
class SubmitResult:
"""Return value of ``submit_file``: filename, outcome, and optional
``batch_id`` (populated when the DB write succeeded) and ``error``
(populated on any failure outcome; ``None`` on ``SUBMITTED`` /
``SKIPPED``).
"""
file: str
outcome: SubmitOutcome
batch_id: str | None = None
error: str | None = None

Some files were not shown because too many files have changed in this diff Show More