Commit Graph

94 Commits

Author SHA1 Message Date
Tyler 1942a22629 feat(sp20): NPI Luhn checksum + Tax ID (EIN) format validation
Adds pure local validators for the 10-digit NPI Luhn checksum (CMS-
published algorithm with the '80840' NPPES prefix) and 9-digit EIN
format (rejects reserved prefixes 00/07/80-89). No NPPES round-trip,
no IRS e-file lookup — catches the 99% typo case at parse time.

Surface:
- cyclone.npi.is_valid_npi / is_valid_tax_id / normalize_tax_id
- CLI: 'cyclone validate-npi <npi>' and 'cyclone validate-tax-id <ein>'
- API: GET /api/admin/validate-provider?npi=&tax_id=
- Parser validator: new R021_npi_checksum rule (warning, not error,
  to keep test fixtures with placeholder NPIs ingestible)
- minimal_837p.txt fixture NPI updated from '1234567890' to the
  Luhn-valid '1993999998' so strict-mode CLI parses still pass

Tests:
- test_npi.py — 27 cases (Luhn math, valid/invalid NPIs, EIN cases,
  normalize_tax_id edge cases)
- test_api_validate_provider.py — 4 cases (both valid, both invalid,
  omitted NPI, omitted tax_id)
- test_cli_validate.py — 8 cases (valid/invalid for both subcommands,
  exit codes, malformed inputs)
- test_validator.py — 4 new R021 cases (valid Luhn silent, bad Luhn
  warning, skipped when format bad, skipped when NPI missing)

Total: 923 tests pass.
2026-06-21 10:46:10 -06:00
Tyler c835996bd6 feat(sp19): security hardening + rich health probe
Three pure-ASGI middlewares close completeness-review gaps §3.1.4
(no body/rate limits) and §3.1.25 (no security headers):

- BodySizeLimitMiddleware — rejects oversized uploads (50 MB
  default, CYCLONE_MAX_BODY_BYTES override). 413 on over-cap
  Content-Length and on chunked reads that cross the cap.
- RateLimitMiddleware — sliding-window per-IP limiter (300/min
  default, CYCLONE_RATE_LIMIT_PER_MIN override). 429 over the
  window. /api/health is exempt.
- SecurityHeadersMiddleware — stamps X-Content-Type-Options,
  X-Frame-Options, Referrer-Policy, Permissions-Policy, and a
  strict Content-Security-Policy on every response.

Every 413/429 also writes a tamper-evident api.request_rejected
event into the SP11 audit chain so an operator can correlate
rejections with the SP18 JSON logs.

GET /api/health is rewritten to return a subsystem snapshot:
DB connectivity (SELECT 1), MFT scheduler state, backup scheduler
state, live pubsub subscriber counts, last batch id + timestamp.
Returns status='degraded' if any subsystem is unhappy; per-subsystem
errors surfaced in the respective dict.

Cyclone.pubsub.EventBus.stats() — new method for live subscriber
counts.

13 new tests (test_security.py) + 1 updated (test_api.py health
endpoint). All 883 backend tests pass.
2026-06-21 10:21:01 -06:00
Tyler 47e0f80786 feat(sp18): structured JSON logging + PII scrubber
Cyclone previously emitted stdlib default-formatted log lines that
operators couldn't parse with anything beyond grep. SP18 replaces
that with:

- JsonFormatter: newline-delimited JSON, ISO-8601 ms timestamps,
  structured "extra" dict, exception tracebacks serialized as a
  single string.
- CycloneDevFormatter: tabular format for tail -f in dev.
- PiiScrubber: logging.Filter that redacts NPIs, SSNs, DOBs,
  patient names — both inline in the message ("npi 1881068062")
  and via PHI-keyed extras ({"dob": "1980-04-12"}).
- setup_logging(): idempotent entry point used by the API lifespan
  and CLI main; respects CYCLONE_LOG_LEVEL/FILE/JSON/NO_PII_SCRUB.
- CLI --log-format=json|dev + --log-file=… flags.
- Migrate 9 highest-value log sites in scheduler/backup_scheduler/
  backup_service to use extra={...} (input_filename, claims, parser,
  backup_id, db_fingerprint, etc.).

34 new tests (test_logging_formatter + test_logging_scrubber +
test_logging_setup). All 867 backend tests pass.
2026-06-21 09:59:34 -06:00
Tyler f003c1f73a feat(sp17): encrypted DB backup automation
Adds automated encrypted backups of the live SQLite file. Closes the
'no backup automation' gap called out in the completeness review
(docs/reviews/2026-06-20-cyclone-completeness-review.md §3.1 #3) and
gives the SP16 MFT scheduler a recovery path when the MFT pipeline
loses days of inbound 999/277CA work in a single crash.

Architecture
------------
- AES-256-GCM with PBKDF2-HMAC-SHA256 (200,000 iters, 16-byte salt)
- Online backups via SQLite's .backup() API — no app downtime
- Salt + passphrase persisted to macOS Keychain (separate accounts
  backup.passphrase + backup.salt) so the key is reproducible across
  processes
- Two-step restore (initiate → confirm) with a one-shot 64-char hex
  token; the second call disposes + rebuilds the engine only if the
  token matches within a 5-minute TTL
- Tamper-evident audit chain (SP11) — db.backup_created,
  db.backup_failed, db.backup_pruned, db.backup_restored,
  db.backup_passphrase_set
- BackupService + BackupScheduler + module-level singletons
- 8 admin endpoints + 6 CLI subcommands
- Auto-start opt-in via CYCLONE_BACKUP_AUTOSTART=true; default
  interval 24h, default retention 30 days
- Fallback posture: if no separate passphrase is set and SQLCipher
  is enabled, the key is derived from the SQLCipher DB key with a
  fixed salt + WARNING log (degraded but never plaintext)

New modules
-----------
- cyclone.backup          — PBKDF2, AES-GCM, sidecar format
- cyclone.backup_service  — create_now / list / verify / restore / prune / status
- cyclone.backup_scheduler — async tick loop with audit hooks

New surface
-----------
- 8 admin endpoints under /api/admin/backup/*
- 6 CLI subcommands under cyclone backup (init-passphrase, create,
  list, verify, restore, prune, status)
- Migration 0012_backups.sql + DbBackup ORM
- store.add_backup_pending()

Tests
-----
- 14 unit tests in test_backup_crypto.py (key derivation, encrypt/
  decrypt round-trip, tampered ciphertext, wrong passphrase, sidecar
  round-trip, filename format)
- 19 tests in test_backup_service.py (create/list/verify/restore/
  prune/status, error handling, fallback key, module singleton)
- 14 API tests in test_api_backup.py (all 8 endpoints + scheduler
  endpoints, two-step restore, error responses)
- 10 tests in test_backup_scheduler.py (tick / start / stop /
  audit / coalescing / module singleton)
- 5 CLI tests in test_cli_backup.py (create / list / verify /
  restore confirm prompt / prune confirm prompt /
  init-passphrase minimum-length check)

Total new tests: 62. All pass. Full backend suite: 833 passed,
9 skipped (gitignored prodfiles), 1 warning.

Design doc: docs/superpowers/specs/2026-06-21-cyclone-encrypted-backup-design.md
README: new 'Encrypted Backups (SP17)' section, SP17 entry in
Roadmap, retention default documented in §Project layout.
2026-06-21 09:43:51 -06:00
Tyler 40f184c858 feat(sp16): live MFT polling scheduler
Adds an asyncio-based background scheduler that polls the Gainwell
MFT inbound path, downloads new files, and routes them through the
appropriate parser (999 / 835 / 277CA / TA1). Idempotent (re-ticks
and restarts skip already-processed files via the new
processed_inbound_files table). Crash-safe (per-file try/except so
one bad file doesn't stop the loop).

Lifespan auto-configures from the seeded dzinesco clearhouse's SFTP
block; auto-start is opt-in via CYCLONE_SCHEDULER_AUTOSTART.

Five admin endpoints added:
  GET  /api/admin/scheduler/status
  POST /api/admin/scheduler/start
  POST /api/admin/scheduler/stop
  POST /api/admin/scheduler/tick
  GET  /api/admin/scheduler/processed-files?status=&limit=

20 new tests (15 unit + 5 API).
2026-06-21 09:20:58 -06:00
sp15-bot 47902fd6b2 feat(sp15): SQLCipher key rotation via PRAGMA rekey
Adds in-place key rotation for the encrypted DB at rest (HIPAA
sec.164.308(a)(4) - periodic key rotation).

- db_crypto.rotate_db_key(): opens with old key, issues PRAGMA rekey,
  reopens with new key, verifies schema survived (table-count sanity).
- db_crypto.generate_db_key(): fresh 256-bit CSPRNG hex key.
- db_crypto.fingerprint(): SHA-256[:8] of a key, for the operator to
  compare across rotations.
- db.dispose_engine() + db.reinit_engine(): SP15 plumbing. The
  rotation endpoint disposes the pooled connections (SQLCipher
  refuses to rekey while another connection holds the file), runs
  the rekey, then rebuilds the engine with the new key from the
  Keychain.
- API: POST /api/admin/db/rotate-key with module-level threading.Lock
  to serialize rotations. 400 when encryption not enabled, 409 when
  a rotation is already in flight, 503 on rekey or Keychain failure
  with a reason that tells the operator what to do next.
- Engine uses NullPool when SQLCipher is enabled: the default
  QueuePool returns connections to a shared queue that any thread
  can pull from, which breaks SQLCipher's thread affinity. NullPool
  trades connection reuse for thread safety, the only correct
  behavior under FastAPI's per-request threadpool.
- Audit event db.key_rotated with old/new fingerprints and
  table_count, written after the engine is rebuilt so the new key
  proves it can take new writes.
- previous key is stashed to a second Keychain account so the
  operator can roll back if the new key turns out to be broken.

Tests:
- test_db_crypto.py: 12 new tests for generate/fingerprint/rekey
  mechanics (5 require SQLCipher at runtime; skipped otherwise).
- test_api_rotate_key.py: 6 new tests for endpoint wiring
  (encryption-required, Keychain update, audit event, rekey-failure
  rollback, Keychain-write-failure 503, concurrent-rotation 409).
2026-06-21 00:33:37 -06:00
Tyler 5c9365ec33 feat(sp14): 5-lane Inbox UI with Payer-Rejected acknowledge action
Closes the gap between the SP10 backend (5 lanes) and the SP6
frontend (4 lanes). The Payer-Rejected lane (277CA STC A4/A6/A7)
is now rendered alongside Rejected/Candidates/Unmatched/Done,
with an Acknowledge bulk action that drops claims from the
working surface without erasing the original 277CA rejection
event (audit log stays intact, SP11).

Backend:
* Migration 0010: add payer_rejected_acknowledged_at +
  payer_rejected_acknowledged_actor columns + partial index.
* db.py: surface the two new columns on the Claim model.
* inbox_lanes.py: filter acknowledged claims out of the
  payer_rejected lane; expose the new fields on the row payload
  for forward-compat (e.g. a future 'Recently acknowledged' view).
* api.py:
  - POST /api/inbox/payer-rejected/acknowledge
    Bulk-acknowledge. Idempotent. Returns transitioned /
    already_acked / not_found / not_rejected counts so the UI
    can show '3 of 5 were already acknowledged' on a noop bulk.
    Writes a 'claim.payer_rejected_acknowledged' event to the
    SP11 hash-chained audit log.
  - GET /api/inbox/export.csv: accept 'payer_rejected' lane.
* test_acks.py: bump user_version assertion to 10.
* test_lane_filter_acknowledged.py: 4 tests for the lane filter
  and forward-compat row payload.
* test_payer_rejected_acknowledge.py: 6 tests for the endpoint
  (happy path, idempotency, no-op on non-rejected, missing
  ids, 400 on empty, audit-log wiring + chain integrity).

Frontend:
* lib/inbox-api.ts: add payer_rejected to InboxLanes, add
  acknowledgePayerRejected(), update exportInboxCsvUrl union.
* hooks/useInboxLanes.ts: add payer_rejected to initial state.
* hooks/useInboxLanes.test.ts: add payer_rejected to mocks.
* components/inbox/BulkBar.tsx: add 'payer_rejected' lane with
  Acknowledge action (no Resubmit, no Dismiss — payer-rejected
  is not eligible for either).
* components/inbox/BulkBar.test.tsx: add payer_rejected test.
* pages/Inbox.tsx: render the 5th lane, hook up onAcknowledge,
  include payer_rejected in the needEyes count.
* pages/Inbox.test.tsx: 3 new tests (5-lane render, need-eyes
  count, acknowledge action hits the right endpoint).
* components/inbox/InboxHeader.tsx: doc comment now explains
  why payer_rejected rolls up into need-eyes.

Pre-existing typecheck warnings in BulkBar.test.tsx / InboxRow
.test.tsx / Lane.tsx / download.test.ts are unchanged from
main — not touched here.

Test counts: backend 724 -> 734 (+10). Frontend 350 -> 354 (+4).
2026-06-21 00:13:47 -06:00
Tyler a11e051f82 feat(sp13): paramiko-backed SftpClient wire-up
Replace SftpClient stub write_file/list_inbound/read_file
implementations with real paramiko SSHClient + SFTPClient
calls. The public API (SftpClient.write_file, list_inbound,
read_file, get_secret) is unchanged from SP9 — same signature,
same return types — so the API layer needs no changes.

Real-mode behavior:
* _connect() returns a context manager yielding (ssh, sftp);
  closes both on exit. Lazy-imports paramiko so the stub-only
  test path doesn't need the dependency.
* Auth resolves from SftpBlock.auth: password_keychain_account
  (MFT model) or key_file + optional key_passphrase_keychain_account.
  Missing Keychain entries fail loud (RuntimeError) rather than
  silently attempting empty-password auth.
* write_file: opens sftp.open(remote, 'wb') and writes bytes;
  mkdirs the parent dir (idempotent — MFT pre-creates FromHPE/ToHPE).
* list_inbound: listdir_attr + per-file download into local
  staging cache; skips directory entries (0o040000 mask).
* read_file: download via shutil.copyfileobj into BytesIO.

Stub mode is unchanged. AutoAddPolicy for first-time MFT host
fingerprint; operator should pin the key for production.

Adds tests/test_sftp_paramiko.py: 9 tests covering
* stub still works
* real-mode connect builds correct paramiko call from
  password_keychain_account, raises on missing Keychain,
  raises on missing auth config, raises on STUB_SECRET
* write_file opens 'wb' on the right path and writes bytes
* list_inbound translates attrs into InboundFile records and
  caches files locally; skips dirs

Removes 2 obsolete tests in test_sftp_stub.py that expected
SP13-mode to raise NotImplementedError.

pyproject.toml: new optional 'sftp' extra (paramiko>=3.4,<6).
2026-06-21 00:00:13 -06:00
Tyler d54c44f04a feat(sp12): SQLCipher encryption at rest (optional)
- New cyclone.db_crypto module:
  * is_sqlcipher_available() — capability check
  * is_encryption_enabled() — Keychain key + sqlcipher3 present
  * get_db_key() — reads 'cyclone.db.key' from Keychain
  * make_sqlcipher_connect_creator(url, key) — SQLAlchemy creator
- db._make_engine() now switches to SQLCipher when key is present
- pyproject.toml: optional 'sqlcipher' extra (sqlcipher3>=0.6,<1)
- Fallback: without Keychain key, DB stays plain SQLite (no surprise
  behavior for operators who haven't set up encryption yet)
- Verified: encrypted file is unreadable as plain SQLite, wrong key
  raises on first query, migrations + ORM work transparently
- HIPAA §164.312(a)(2)(iv) compliance note in docs

Tests: 705 -> 717 (12 new for SQLCipher). All 717 backend tests pass.
2026-06-20 23:52:41 -06:00
Tyler 62bb09f183 feat(sp11): tamper-evident hash-chained audit_log
- New audit_log table (migration 0009, user_version=9):
  * id, event_type, entity_type, entity_id, actor, payload_json,
    created_at, prev_hash, hash
  * Indexes on (entity_type, entity_id), event_type, created_at
- New cyclone.audit_log module:
  * append_event(session, AuditEvent) — appends one chained row
  * verify_chain(session) — walks the chain, returns first bad id
  * SHA-256 hash over canonical row form (unit-separator delimited)
  * Genesis prev_hash = 64 zeros (Bitcoin-style sentinel)
- New AuditLog ORM model
- New admin API endpoints:
  * GET /api/admin/audit-log (paginated, filterable)
  * GET /api/admin/audit-log/verify (returns ok/first_bad_id/reason)
- Hooked into existing endpoints to append events:
  * /api/parse-999 → 'claim.rejected' per matched claim
  * /api/parse-277ca → 'claim.payer_rejected' per matched claim
  * /api/clearhouse/submit → 'clearhouse.submitted' per claim
- HIPAA §164.316(b)(2) compliance note in docs

Tests: 688 -> 705 (9 audit + 8 audit-API). All 705 backend tests pass.
2026-06-20 23:45:43 -06:00
Tyler 2c0afbe9c5 feat(sp10): 277CA parser + Payer-Rejected Inbox lane
- Add cyclone.parsers.models_277ca + parse_277ca (X12 005010X214)
  - Per-Patient HL ClaimStatus with REF*1K (PCN), REF*EJ (tax ID),
    STC category code, amount, service date
  - STC classifier: A1-A3 accepted, A4/A6/A7 rejected, A8/A9 pended,
    P1-P5 paid, anything else unknown
  - Multiple STCs per Patient HL: last wins (canonical pattern for
    'first pended, then paid' acknowledgments)
  - Subscriber-level STCs surface in unscoped_statuses
- Add apply_277ca_rejections — stamps Claim.payer_rejected_* fields on
  matching rows (lookup by patient_control_number, mirrors 999 path)
- New /api/parse-277ca, /api/277ca-acks, /api/277ca-acks/{id} endpoints
- New two77ca_acks table + Two77caAck ORM model
- New Claim columns: payer_rejected_at, _reason, _status_code, _by_277ca_id
- New payer_rejected lane in /api/inbox/lanes (distinct from 999
  envelope rejected lane)
- New PayerConfig277CA block in config/payers.yaml + Pydantic model
- Migration 0008 bumps user_version to 8

Tests: 654 -> 688 (parser 22 + apply 6 + API 8 + config 6 + adjustments).
All 688 backend tests pass; 1 pre-existing skipped test class unaffected.
2026-06-20 23:41:01 -06:00
Tyler c62826daea feat(sp9): multi-payer, multi-NPI, SFTP stub for dzinesco/TOC
- providers table seeded with 3 NPIs (Montrose, Delta, Salida)
- payers + payer_configs (per-tx config) join table
- clearhouse singleton (dzinesco identity, SFTP block, filename block)
- config/payers.yaml loader with Pydantic schema validation
- cyclone/edi/filenames.py: HCPF X12 File Naming Standards helpers
- cyclone/clearhouse/sftp.py: stub that copies to ./var/sftp/staging/...
- cyclone/secrets.py: macOS Keychain wrapper via keyring
- 11 new validation rules R200-R210 (CO MAP + HCPF naming)
- 6 new API endpoints (/api/clearhouse/*, /api/config/*, /api/admin/reload-config)
- migration 0007
- 72 new tests (646 total passing, was 574)

See spec: docs/superpowers/specs/2026-06-20-cyclone-multi-payer-npi-sftp-design.md
2026-06-20 23:07:31 -06:00
Tyler ec9eae7a2c feat(sp8): resubmit endpoint supports ?download=zip for regenerated 837s
Add an optional ``?download=true`` query param to
``POST /api/inbox/rejected/resubmit`` that returns the same operation
result as a ZIP archive of regenerated 837P files (one
``claim-{id}.x12`` per successfully resubmitted claim) rather than the
JSON envelope.

Why: operators who mass-resubmit rejected claims want to hand the
files straight to their clearinghouse; round-tripping through copy-
paste is error-prone.

Implementation notes:
  * Uses ``serialize_837_for_resubmit`` so each X12 file in the bundle
    gets a unique interchange/group control number (back-to-back
    resubmits would otherwise collide on ISA13/GS06 = "000000001").
  * Conflicts and missing ids are deliberately excluded from the ZIP
    — the user already saw them in the JSON path on prior calls; the
    download is the "give me the files I asked for" payload.
  * Empty resubmit + download returns 200 with an empty ZIP so the UI
    can still hand the user a downloadable artifact.

Tests (test_inbox_endpoints.py): 2 new tests covering the success
shape (one .x12 per accepted claim) and the conflict-exclusion
contract.
2026-06-20 20:49:08 -06:00
Tyler 2893676c0b fix(835): correct SVC04/SVC05 mapping per X12 005010X221A1
X12 835 SVC segment:
  SVC01 = composite procedure
  SVC02 = charge
  SVC03 = payment
  SVC04 = Unit or Basis for Measurement Code (UN, MJ, DA, ...)
  SVC05 = Service Unit Count

The parser previously read SVC04 as the units count and SVC05 as the
unit type — backwards. On real 835s (and the canonical minimal
fixture), SVC04 carries the code 'UN' which fails Decimal parsing, so
the units always came out as None and the code string was assigned to
unit_type. SP7's line-level matcher couldn't compare units on the SVC
side against the claim side because of this.

- _consume_service_payment: SVC04 → unit_type, SVC05 → units count
- Default unit_type to 'UN' when only the count is present
- minimal_835.txt + unbalanced_835.txt: swap positions to match spec
- Add 2 regression tests (units-and-unit-type, default-unit-type-to-UN)
2026-06-20 20:44:28 -06:00
Tyler 3f4e6849c6 feat(sp8): GET /api/claims/{id}/serialize-837 endpoint
Returns the persisted ClaimOutput as a regenerated X12 837P file via
the new outbound serializer (Approach A — full rebuild from canonical
fields). 404 on missing claim, 422 if the stored raw_json cannot be
validated as a ClaimOutput. text/x12 content-type, attachment
disposition with the claim id as the filename.

3 tests:
- endpoint returns text/x12 attachment starting with ISA*
- 404 for missing claim id
- regenerated text round-trips back through parse()
2026-06-20 20:34:44 -06:00
Tyler 561018c690 feat(sp8): outbound 837P serializer — full rebuild + round-trip tests
- backend/src/cyclone/parsers/serialize_837.py — full-rebuild 837P serializer.
  Emits envelope (ISA/GS/ST/SE/GE/IEA + BHT) + submitter/receiver/billing
  provider/subscriber/payer hierarchy + editable segments (CLM/REF*G1/HI)
  + per-service-line LX/SV1/DTP*472/REF*6R — all from canonical
  ClaimOutput fields.

  Pivoted from spec §3.1 hybrid to full rebuild because ClaimOutput.raw_segments
  only captures post-CLM segments (CLM, REF*G1, HI, LX, SV1 pairs) — not the
  envelope or hierarchies. A pass-through approach cannot regenerate those
  without expanding raw_segments in parse_837.py (out of scope for this SP).

- backend/tests/test_serialize_837.py — 36 tests covering envelope shape,
  hierarchy segments, claim-level builders, service-line builders, edited-field
  propagation, round-trip, custom sender/receiver IDs, and resubmit helper.

- backend/tests/test_prodfiles_smoke.py::test_claims_prodfile_round_trip —
  every file in docs/prodfiles/claims/ (113 files) round-trips through
  serialize_837 → parse_837_text with deep-equal ClaimOutput (modulo
  validation, which is recomputed by the parser).

- docs/superpowers/plans/2026-06-20-cyclone-serialize-837.md — full plan
  with amendment note documenting the Approach A pivot.

Per session convention, plan note about unrelated modifications to
parse_835.py / fixtures stashed separately.
2026-06-20 20:27:48 -06:00
Tyler d033ce85db feat(sp7): wire _reconcile_pair into manual_match (T21)
manual_match previously only flipped the claim↔remit FK and the
claim state — it never ran line-level reconciliation, so manually-
paired claims surfaced empty line-reconciliation rows to the UI
and skipped CLP-level CAS aggregate recompute.

Refactor reconcile.run() to call a new per-pair helper
_reconcile_pair(session, claim, remittance) that:
  - clears any existing LineReconciliation rows for the claim
    (idempotent re-run; safe across manual_unmatch + manual_rematch
    cycles that may pair the claim with a different remittance),
  - reads 837 SV1 lines from Claim.raw_json and 835 SVC rows from
    ServiceLinePayment,
  - runs match_service_lines() and persists a LineReconciliation row
    per side,
  - recomputes Remittance.claim_level_adjustment_amount (CLP-level
    CAS) and Remittance.adjustment_amount (total CAS).

manual_match now calls this helper after the FK is set and before
commit, mirroring the auto-match path. Reversals are skipped (they
don't have SV1↔SVC line pairs; per §7.3).

Tests (test_store_reconcile.py):
  - test_manual_match_populates_line_reconciliation_rows: end-to-end
    check that a manual pair writes the expected matched +
    unmatched_837_only rows plus zero CAS aggregates.
  - test_manual_match_idempotent_line_reconciliation: after
    manual_unmatch + manual_rematch + inserting a CLP-level CAS row
    directly, the claim has exactly two fresh LineReconciliation rows
    (no duplicates) and the remittance aggregate reflects the new CAS.

Smoke tested end-to-end via TestClient: parse co_medicaid_837p.txt
+ co_medicaid_835.txt, auto-matcher skips (PCNs differ), manual
match via POST /api/reconciliation/match, then
GET /api/claims/{id}/line-reconciliation returns 2 rows and
GET /api/inbox/lanes shows matched_remittance.total_lines=2.

Note: matched_lines may be 0 on real 835 fixtures due to a
pre-existing parser bug in _consume_service_payment that swaps
SVC04 (units) and SVC05 (unit-of-measure) when reading units from
the segment. The SP7 strict-match criterion requires units parity,
so the misread produces None on the SVC side and strict-match
never succeeds. Surfaced as a follow-up; the SP7 work itself is
correct (line counts are computed and persisted).
2026-06-20 20:13:16 -06:00
Tyler b7bda0662f feat(sp7): inbox lanes matched_remittance includes matched/total line counts 2026-06-20 19:43:43 -06:00
Tyler 217f14a52a feat(sp7): claim detail includes lineReconciliation slim projection 2026-06-20 19:37:38 -06:00
Tyler 48d4739864 feat(sp7): GET /api/claims/{id}/line-reconciliation endpoint 2026-06-20 19:35:17 -06:00
Tyler c1323c44d0 feat(sp7): wire match_service_lines into reconcile.run 2026-06-20 19:33:02 -06:00
Tyler 9626eae5f1 feat(sp7): match_service_lines() pure function + tests 2026-06-20 19:29:21 -06:00
Tyler 8442bf91ce feat(sp7): persist ServiceLinePayment + link CAS rows on 835 ingest 2026-06-20 19:27:24 -06:00
Tyler 0dd6d39424 test(sp7): update migration version assertion to 6 (SP7 + TA1 WIP) 2026-06-20 19:25:26 -06:00
Tyler 76278ec9f6 feat(ta1): TA1 interchange ACK parser, persistence, and API
Adds full TA1 (Interchange Acknowledgment, X12 envelope-level) support
to mirror the existing 999 transaction-set ACK pipeline.

Parser & models
- parsers/models_ta1.py: Pydantic Ta1Ack + ParseResultTa1 models
  (AckCode = Literal['A','E','R'], date serializer mirroring 999)
- parsers/parse_ta1.py: TA1 segment parser
  - Tolerant YYMMDD (6-digit) + CCYYMMDD (8-digit) date handling
    for CO Medicaid interchange_date / ack_generated_date
  - 106-char ISA validation (15-char sender/receiver IDs)
  - source_batch_id = 'TA1-<ISA13>'

Persistence
- migrations/0005_create_ta1_acks.sql: ta1_acks table + indexes
  on source_batch_id and ack_code
- db.py: Ta1Ack ORM model (flat columns + raw_json, mirrors Ack)
- store.py: add_ta1_ack, list_ta1_acks (returns all rows),
  get_ta1_ack
- db version bumped 4 -> 5 (test_acks.py updated)

API (api.py)
- POST /api/parse-ta1: text/file ingest, persists ta1_ack,
  returns detail-ready payload
- GET /api/ta1-acks: list with limit + total (mirrors 999 pattern)
- GET /api/ta1-acks/{ack_id}: detail
- _ta1_to_ui / _serialize_ta1 / _serialize_ta1_from_row helpers

Tests (17 new, 508 total passing)
- tests/test_parse_ta1.py (8): CCYYMMDD + YYMMDD acceptance,
  E/R codes, source_batch_id, missing ISA/TA1, short defaults
- tests/test_api_ta1.py (9): happy path, rejected persists,
  empty/malformed 400, missing file 422, detail regenerates
  segment, 404, empty list, newest-first
- tests/test_prodfiles_smoke.py: extended to smoke-test 352
  production TA1 files (A=0, R=352, E=0)
- tests/test_api_parse_persists.py: cross-pipeline reconciliation
  test asserting invariants across 837P + 835 prod files

Real-data finding: all 352 production TA1s are R (rejected);
operator follow-up warranted.
2026-06-20 18:58:56 -06:00
Tyler f4baceb574 feat(sp6): inbox endpoints — lanes, match, dismiss, resubmit, csv export
T7-T10 combined (single edit: all endpoints share a section).

- GET /api/inbox/lanes — four lanes in one call
- POST /api/inbox/candidates/{remit_id}/match — 409 on conflict
- POST /api/inbox/candidates/dismiss — session-scoped dismissed set
- POST /api/inbox/rejected/resubmit — 200 with conflicts list
- GET /api/inbox/export.csv?lane=<lane> — streams CSV

Also adds module-level imports (db, Claim, ClaimState, Remittance,
csv, io, datetime) that the new endpoints need; cleans up the
duplicated local imports in the parse-999 SP6 T4 block.
2026-06-20 18:34:09 -06:00
Tyler cdffb3faf3 feat(sp6): compute_lanes — 4 lanes on read with scoring 2026-06-20 18:31:01 -06:00
Tyler 3432c76561 feat(sp6): 4-field weighted scoring module 2026-06-20 18:30:05 -06:00
Tyler f4910fd94d feat(sp6): wire 999 ingest → claim rejected state
T4 — calls apply_999_rejections inside /api/parse-999, then
publishes a claim.rejected event for each transitioned claim so
the Inbox live-tail refetches.
2026-06-20 18:29:28 -06:00
Tyler 78ed75a31d feat(sp6): apply_999_rejections state helper
T3 — moves claims to ClaimState.REJECTED on 999 AK5 R/E/X.

- New module: src/cyclone/inbox_state.py
- New tests: tests/test_inbox_state.py (4 cases)
- Adds ORM mappings for rejection_reason / rejected_at / resubmit_count
  / state_changed_at on Claim (db.py) — these were added to the
  schema by migration 0004 but not yet exposed to the ORM.
2026-06-20 18:25:25 -06:00
Tyler 1b1534d8d2 feat(sp6): migration 0004 — rejection columns + state-history index
Adds:
- claims.rejection_reason, claims.rejected_at, claims.resubmit_count
- claims.state_changed_at (was missing, needed for Done-today lane)
- ix_claims_state_changed_at composite index

Also fixes the ClaimState count assertion in test_db_models.py
(7 → 8) to match the REJECTED enum value added in the previous
commit, and bumps the user_version expectation in test_acks.py
(3 → 4) for the same reason.
2026-06-20 18:21:29 -06:00
Tyler de09959377 test(835): update prodfile test for PCN dedup behavior
The store dedupes remittances by PCN. Refine the assertions:
- total_clps counts the raw CLP segments across all files (3374)
- unique_pcns counts distinct PCNs (persisted row count)
- Read claimId via the API response shape, not the SQLAlchemy attribute
2026-06-20 18:12:59 -06:00
Tyler b5e27927e0 fix(835): drop over-constraining UNIQUE constraints, add multi-BPR warning
Three related changes for real CO Medicaid data:

1. Drop UNIQUE(batch_id, patient_control_number) on claims and
   UNIQUE(batch_id, payer_claim_control_number) on remittances. The X12
   spec allows multiple CLM segments per 2000B subscriber loop and 835
   ERAs can repeat a payer_claim_control_number for reversals. Claim/
   remittance identity is provided by the primary key (claims.id = CLM01,
   remittances.id = CLP01).

2. Add validator rule R835_MULTI_BPR warning for files with multiple BPR
   segments (CO Medicaid split-payment pattern). The parser already sums
   BPR02 paid_amounts; this surfaces the non-standard data to operators.

3. Skip R835_BAL_BPR_vs_CLP04 when BPR01='I' (Information Only 835).
   In that mode BPR02 is informational and the per-claim CLP04 totals
   are authoritative — a diff is expected, not an error.

Migration 0003 handles the drop with IF EXISTS so fresh DBs skip cleanly.
Updates affected tests to reflect new schema (no UNIQUE constraint on
batch_id + patient_control_number / payer_claim_control_number).

Fixes test_api_835::test_prodfile_round_trip_persists_separately which
was failing on real production data.
2026-06-20 18:10:07 -06:00
Tyler f4a7cd99b3 merge: ui/batch-diff 2026-06-20 17:30:30 -06:00
Tyler 7b394fff1a feat(batch-diff): side-by-side claim diff between two batches 2026-06-20 17:17:09 -06:00
Tyler da29188dd0 feat(api): GET /api/{claims,remittances,activity}/stream + unsubscribe
Adds three live-tail streaming endpoints that emit an NDJSON snapshot
then forward new event-bus events as they arrive, with a 15s idle
heartbeat (overridable via CYCLONE_TAIL_HEARTBEAT_S for tests).

Each endpoint:
  1. yields a snapshot of existing rows as {"type":"item","data":<row>}
  2. terminates the snapshot with {"type":"snapshot_end","data":{"count":N}}
  3. subscribes to its event kind and forwards each new event as an
     {"type":"item","data":<event>} line
  4. emits a {"type":"heartbeat","data":{"ts":<iso>}} line every
     CYCLONE_TAIL_HEARTBEAT_S seconds when idle
  5. checks request.is_disconnected() before each yield and unsubscribes
     from the bus on cleanup so a closed stream releases its queue

The shared tail loop lives in api._tail_events, which polls
bus.subscribe_raw()'s queue directly instead of using the bus's
async-iterator wrapper — wait_for on an async generator cancels the
inner future on timeout, which poisons subsequent __anext__ calls with
StopAsyncIteration. Queue.get() is idempotent under cancellation, so
heartbeats don't break the subscription.

EventBus gains an unsubscribe(queue, kinds) method (idempotent) so
the tail loop can release its queue in a try/finally. The disconnect
test asserts the subscriber list is empty after the body iterator is
closed, validating no queue leak per open stream.

Tests in test_api_stream_live.py: 8 tests covering snapshot shape,
post-snapshot publish, heartbeat timing, multi-item snapshots, and
client disconnect cleanup. Plus 2 tests in test_pubsub.py for the
new unsubscribe method.
2026-06-20 16:57:30 -06:00
Tyler 8bcb23c78d feat(api): parse endpoints pass EventBus into store writes
- api.parse_837 / parse_835: pass request.app.state.event_bus into store.add()
- conftest: autouse fixture wires a fresh EventBus onto app.state for every
  test, since TestClient does not invoke the FastAPI lifespan handler
  unless used as a context manager
- test_pubsub: split get_event_bus coverage into a raises-when-missing test
  and a returns-attached-bus test, both save/restore app.state.event_bus
  around the assertion so the autouse fixture's bus is preserved

Phase 2 complete: db init moved to lifespan, EventBus is the process-wide
publish point, and the two ingest endpoints publish claim_written /
remittance_written / activity_recorded events on every store.add().
2026-06-20 15:51:18 -06:00
Tyler 8d02ed3204 test(store): 835 batch publishes remittance_written + activity_recorded 2026-06-20 15:47:35 -06:00
Tyler ac7f3283d6 feat(store): add publishes claim_written + activity_recorded events 2026-06-20 15:42:56 -06:00
Tyler 25a76a515d feat(pubsub): get_event_bus() late-import accessor 2026-06-20 15:35:02 -06:00
Tyler b2f5a16541 feat(pubsub): EventBus with drop-oldest overflow, per-kind fan-out 2026-06-20 15:28:13 -06:00
Tyler 14d61f82b8 test(api)+docs: assert matchedRemittance=null; fail-loud on fixture drift; tighten docstring 2026-06-20 09:56:24 -06:00
Tyler 129599f0f4 feat(api): GET /api/claims/{claim_id} returns full claim context 2026-06-20 09:43:08 -06:00
Tyler 423415ab64 feat(store): get_claim_detail returns full claim context with history + matched remit 2026-06-20 09:21:07 -06:00
Tyler a665881d24 feat(parsers+api): 270/271 eligibility request + response (SP3 P4)
Adds API-only 270/271 eligibility flow per spec section 3.4 (no UI, no
DB persistence). Operator pastes a 270 query, gets back raw 270 text,
submits manually, pastes the 271 back.

New models (270 + 271):
  - InformationSource, InformationReceiver, Subscriber, Patient
  - EligibilityBenefitInquiry (EQ) and CoverageBenefit (EB)
  - Inline SERVICE_TYPE_CODES dict (566 codes, snapshot 2026-06-20)
  - service_type_description() lookup helper

Parsers (single segment walker, defensive, mirrors 835 pattern):
  - parse_270: HL*20/21/22/23 loops + EQ, NM1, DMG, TRN
  - parse_271: HL*20/21/22/23 loops + EB, DTP, MSG, AAA

Serializer:
  - serialize_270: ISA/GS/ST/BHT/HL*20/21/22/patient/EQ/SE/GE/IEA
  - round-trips: serialize -> parse -> serialize == identical

API:
  - POST /api/eligibility/request (JSON body -> {raw_270_text, parsed})
  - POST /api/eligibility/parse-271 (file upload -> structured summary)
  - 400 on missing fields / garbage; never 500.

Tests: 365 -> 408 backend (+43 new), 10 frontend unchanged.
TypeScript clean.
2026-06-20 08:22:31 -06:00
Tyler f3002e4cbe fix(api): regenerate raw_999_text in /api/acks/{id} detail response (SP3 P3 follow-up)
The P3 subagent shipped a Download button on /acks that fell back to
'no raw text' because the detail endpoint returned raw_json (the parsed
model dump) but never the regenerated X12.

Fix: detail endpoint now re-serializes raw_json via serialize_999 and
returns it as raw_999_text, so the operator can actually download the
999 file. List endpoint unchanged (keeps payload small).
2026-06-20 08:06:21 -06:00
Tyler fb2a98fc7a feat(parsers+api+ui+db): 999 ACK transaction set end-to-end (SP3 P3) 2026-06-20 08:03:37 -06:00
Tyler 7a20f732f2 fix(api): include adjustments array in /api/remittances list response (SP3 P2 follow-up)
The P2 subagent shipped a UI chevron+expansion in Remittances.tsx that
read adjustments from the list payload, but iter_remittances() didn't
include the CasAdjustment rows. Result: the expansion was always empty
when a remittance was opened from the list (the detail endpoint worked).

Fix: bulk-fetch all CasAdjustment rows for the remittance_ids in one
query, then attach them as the 'adjustments' array. N+1-free.
2026-06-20 07:49:20 -06:00
Tyler 2a853857b1 feat(parsers+api+ui): expose CARC-labeled CAS adjustments (SP3 P2) 2026-06-20 07:44:41 -06:00
Tyler 716bfb0a26 feat(parsers): R034 REF*G1 enforcement + R035 BHT06 transaction type code (SP3 P1)
- R034 _r034_ref_g1_required: when payer opts in (require_ref_g1_for_adjustments),
  emit error if frequency_code is 7/8 and no REF*G1 segment appears in raw_segments.
  CO Medicaid stays lenient in v1 (gate is False by default).
- R035 _r035_bht06_allowed: BHT06 transaction type code must be in cfg.allowed_bht06.
  Skipped silently when transaction_type_code is None.
- Add transaction_type_code (str|None) to Envelope and ClaimOutput.
- parse_837: read BHT06 (seg[6]) in _build_envelope; mirror onto each ClaimOutput
  via the existing model_copy(update={...}) call site.
- 9 new tests in test_validator.py: 5 R034 (incl. lenient no-op) + 4 R035.
  213 passed, 1 skipped (prodfile corpus present-conditional).
2026-06-20 07:25:29 -06:00