From f21201a22e523cd6c0ff970df9c300a2d83e9155 Mon Sep 17 00:00:00 2001 From: Nora Date: Tue, 7 Jul 2026 18:02:58 -0600 Subject: [PATCH] docs(spec): SP40 Edifabric validation gate + 837P byte-defect sweep Covers three things: 1. _build_per / _build_sbr call-site fixes so the serializer always emits PER-02/03/04 + SBR-09 (Edifabric rejects the current shape). 2. New cyclone.edifabric HTTP client wrapping the two-step /x12/read -> /x12/validate flow against api.edination.com. 3. Pre-upload gate inside resubmit-rejected-claims so an Edifabric-invalid file can't reach Gainwell. Public eval API key 3ecf6b1c5cf34bd797a5f4c57951a1cf for dev. Production is operator-supplied via cyclone secrets set edifabric.api_key . --- ...yclone-edifabric-validation-gate-design.md | 124 ++++++++++++++++++ 1 file changed, 124 insertions(+) create mode 100644 docs/superpowers/specs/2026-07-07-cyclone-edifabric-validation-gate-design.md diff --git a/docs/superpowers/specs/2026-07-07-cyclone-edifabric-validation-gate-design.md b/docs/superpowers/specs/2026-07-07-cyclone-edifabric-validation-gate-design.md new file mode 100644 index 0000000..63439cc --- /dev/null +++ b/docs/superpowers/specs/2026-07-07-cyclone-edifabric-validation-gate-design.md @@ -0,0 +1,124 @@ +# SP40 — Edifabric validation gate + 837P byte-defect sweep + +**Date:** 2026-07-07 +**Status:** Draft, awaiting user sign-off +**Branch:** `sp40-edifabric-validation-gate` +**Aesthetic direction:** No new UI; one new CLI command + one HTTP endpoint + serializer hardening + a defensive HTTP client. Three new tests, two modified files, two new docs. + +--- + +## 1. Scope + +Edifabric's `POST /v2/x12/validate` (the operator's reference parser for CO Medicaid compliance) rejected the SP39-regenerated 837P files with three byte-level errors: + +``` +Line 1: Element PER-03 is required +Line 1: Element PER-04 is required +Line 1: Element SBR-09 is required +``` + +Root cause: `serialize_837._build_per` emits `PER*IC` (only PER01), and `_build_sbr` emits `SBR*P*18*******` (no SBR09) when the caller passes no contact / SBR09 kwargs. The clearhouse config has `submitter_contact_name="Tyler Martinez"` and `submitter_contact_email="tyler@dzinesco.com"` seeded, and `PayerConfig.co_medicaid().sbr09_claim_filing = "MC"` — both are available to the serializer today but not wired into the default emit. SP33-era callers and the SP39 regen script both bypass the kwargs, so every file they emit is invalid. + +SP40 lands (a) the serializer fixes that close the gap, (b) the Edifabric integration that catches the same class of defect for any future payer / schema drift, (c) a pre-upload gate on `resubmit-rejected-claims` so the operator can't push an Edifabric-invalid file to Gainwell, and (d) the API key wiring through the existing `secrets.get_secret()` plumbing. + +**In scope:** + +- `_build_per` always emits PER02 (Name) + at least one PER03/PER04 pair (email or phone), sourced from `claim.billing_provider.contact` or the clearhouse config when the caller passes nothing. +- `_build_sbr` always emits SBR09 from `payer_cfg.sbr09_claim_filing` (already `"MC"` for CO Medicaid). +- A new `cyclone.edifabric` module — thin `httpx` client wrapping the two-step Edifabric flow (`/x12/read` → `/x12/validate`). +- A new CLI `cyclone validate-837 ` — reads the file, hits Edifabric, prints `OperationResult`; exits 0 on success / warning-only, 2 on any error. +- A new HTTP `POST /api/admin/validate-837` (admin-gated) — multipart upload of the .x12 file; same response shape. +- A pre-upload gate inside `resubmit-rejected-claims` — every file is validated before the SFTP `put`; `Status == "error"` aborts the file but continues the batch (other files still push). Bypass flag `--skip-edifabric-validate` for dev. +- API key wiring: `secrets.get_secret('edifabric.api_key')` with a new `_ENV_NAME_FOR` entry mapping it to `CYCLONE_EDIFABRIC_API_KEY`. +- A dev-fixture API key in `tests/fixtures/edifabric_api_key.txt` so CI doesn't need a real subscription (the public eval key `3ecf6b1c5cf34bd797a5f4c57951a1cf`). +- A `PERMISSIONS` matrix entry for `("POST", "/api/admin/validate-837"): ADMIN_ONLY`. +- RUNBOOK updates for the env var, the new CLI/HTTP, and the bypass flag. + +**Out of scope:** + +- 999 / 277CA / TA1 / 835 / 270 / 271 Edifabric validation (only 837P this SP). +- Building a custom EDI → X12Interchange JSON converter (we use Edifabric's `/x12/read` to do the conversion — see D1). +- Auto-retry on transient Edifabric 5xx — the gate surfaces the error; the operator decides whether to retry. +- Refreshing the 363 regen'd `ingest/corrected-v2/` files — the regen script already runs cleanly through the fixed serializer once Task 1 lands. + +## 2. Decisions + +- **D1.** Two-step Edifabric flow: `/x12/read` (raw EDI → `X12Interchange` JSON, `Content-Type: application/octet-stream`) then `/x12/validate` (JSON → `OperationResult`, `Content-Type: application/json`). Building our own EDI → JSON converter would be ~150 fields; `/x12/read` is the pragmatic path and the same library parses files we generate. +- **D2.** API key via `secrets.get_secret('edifabric.api_key')` → `CYCLONE_EDIFABRIC_API_KEY` env var or Keychain (`cyclone` service, `edifabric.api_key` account). The eval key in `tests/fixtures/edifabric_api_key.txt` is dev-only; production is operator-supplied via `cyclone secrets set edifabric.api_key `. +- **D3.** Pre-upload gate is **fail-closed**: any `Status == "error"` item in `OperationResult.Details` aborts the file (file is recorded as `failed`, batch continues). Warnings are logged at INFO level but don't block. This matches the operator's stated goal ("claims arent passing verification"). +- **D4.** Serializer hardening is minimal + backward-compatible: + - `_build_per(contact_name, contact_phone, contact_email)` already handles all three; the fix is at the call site (where `submitter_kwargs` is built) — fall back to `clearhouse.submitter_contact_*` when the caller passes no kwargs. When none of those are set either, emit `PER*IC*CUSTOMER SERVICE*TE*8005550100` (a safe placeholder; documented in the function docstring as a temporary fallback so future per-billing-office contact info lands at the call site instead). + - `_build_sbr(individual_relationship_code, claim_filing_indicator_code)` already takes the SBR09 value; the fix is at the call site — pass `payer_cfg.sbr09_claim_filing` (default `"MC"` for CO Medicaid). When `payer_cfg` is unavailable, fall back to `"MC"` (matches the SP9-era seeding default). +- **D5.** HTTP client uses `httpx` (already in `pyproject.toml:35` `httpx>=0.27,<1`). One new module `cyclone.edifabric` with three functions: + - `read_interchange(edi_bytes, *, api_key) -> X12Interchange` + - `validate_interchange(x12_json, *, api_key) -> OperationResult` + - `validate_edi(edi_bytes, *, api_key) -> OperationResult` — composes the two. + Custom 4xx/5xx handling raises `EdifabricError(status, body)` so the caller can surface the Edifabric error verbatim. +- **D6.** HTTP endpoint is `POST /api/admin/validate-837` (matches existing `/api/admin/*` admin-only pattern from `cyclone/api_routers/admin.py:38-39`). Body is `multipart/form-data` with a single `file` part (FastAPI's `UploadFile`); response is the raw `OperationResult` JSON. The CLI uses the same module functions directly so the wire shape stays consistent. +- **D7.** Test mock strategy: `httpx.MockTransport` registered against the `httpx.Client` instance inside the `cyclone.edifabric` module (factory pattern lets tests inject the transport). No live HTTP in CI; the eval API key in `tests/fixtures/` is referenced but never sent — the mock intercepts first. +- **D8.** Pre-upload gate bypass: `--skip-edifabric-validate` flag on the CLI; no env var. Bypassing is an explicit operator action, not a global setting, so it's hard to forget about. + +## 3. Files expected to change + +| Path | Type | Purpose | +|---|---|---| +| `backend/src/cyclone/parsers/serialize_837.py` | modify | `_build_per` call site: fall back to clearhouse contact info. `_build_sbr` call site: always pass `sbr09_claim_filing`. | +| `backend/src/cyclone/edifabric.py` | create | `read_interchange`, `validate_interchange`, `validate_edi`, `EdifabricError`. | +| `backend/src/cyclone/cli.py` | modify | New `validate_837_cmd` (read file → call `validate_edi` → print `OperationResult`). Pre-upload gate call inside `resubmit_rejected_claims`. New `--skip-edifabric-validate` flag. | +| `backend/src/cyclone/api_routers/admin.py` | modify | New `POST /api/admin/validate-837` (multipart upload, returns `OperationResult`). | +| `backend/src/cyclone/secrets.py` | modify | Add `"edifabric.api_key": "CYCLONE_EDIFABRIC_API_KEY"` to `_ENV_NAME_FOR`. | +| `backend/src/cyclone/auth/permissions.py` | modify | Add `("POST", "/api/admin/validate-837"): ADMIN_ONLY`. | +| `backend/tests/test_serialize_837.py` | extend | 3 new regression tests: PER-02/03/04 always present, SBR-09 always present, no-kwargs fallback still emits valid PER + SBR. | +| `backend/tests/test_edifabric.py` | create | 4 tests with `httpx.MockTransport`: read succeeds, validate succeeds, validate returns errors, 5xx surfaces EdifabricError. | +| `backend/tests/test_validate_837_cli.py` | create | 4 CliRunner tests: clean file → exit 0 + status printed, error → exit 2, missing API key → exit 1, file not found → exit 2. | +| `backend/tests/test_api_validate_837.py` | create | 3 TestClient tests: admin auth gate (401 / 403), valid upload returns OperationResult, missing file part returns 422. | +| `backend/tests/test_resubmissions_cli.py` | extend | 1 new test: validation fail mid-batch aborts that file but continues the others; `--skip-edifabric-validate` flag bypasses the gate. | +| `backend/tests/fixtures/edifabric_api_key.txt` | create | The public eval key `3ecf6b1c5cf34bd797a5f4c57951a1cf` (file-based secret, read via `_FILE` env var in tests). | +| `docs/RUNBOOK.md` | modify | New "After SP40 lands" section: env var, secret setup, CLI/HTTP usage, bypass flag, expected OperationResult shape. | +| `docs/superpowers/specs/2026-07-07-cyclone-edifabric-validation-gate-design.md` | create | This spec, committed on the branch. | + +## 4. Auth boundary + +The HTTP endpoint inherits the existing admin router's `Depends(matrix_gate)` gate (`cyclone/api_routers/admin.py:39`); `PERMISSIONS` entry is `ADMIN_ONLY`. The API key never leaves the backend — the CLI and the HTTP handler both call `cyclone.edifabric.validate_edi(...)` server-side. The CLI is operator-only (no auth boundary in the CLI itself, per house convention). File-system threats unchanged: SQLCipher at rest, macOS Keychain for the secret. No new ports. + +## 5. Open questions + +- Should the `validate-837` HTTP endpoint accept a JSON body `{content: "ISA*..."}` as an alternative to multipart upload? Pro: easier UI integration. Con: more attack surface (request body in DB-sized chunks). Defer to a future SP if needed. +- Should the gate emit a per-claim `Resubmission` row update when validation fails? The current `Resubmission` row is only written on successful SFTP push, so a fail-closed gate would skip the row entirely — clean. If the operator wants an audit trail of *attempted* pushes, that's a follow-up column on `Resubmission`. Defer. + +## 6. Test impact + +- `test_serialize_837.py`: 3 new regression tests. Total file: 53 + 3 = 56 tests. +- `test_edifabric.py`: 4 new tests with mocked transport. +- `test_validate_837_cli.py`: 4 new CliRunner tests. +- `test_api_validate_837.py`: 3 new TestClient tests (auth gate, valid upload, missing file). +- `test_resubmissions_cli.py`: 1 new gate-fail test + 1 bypass-flag test. +- Total new tests: 13. Full suite target: 1458 + 13 = 1471 passing. + +## 7. Acceptance criteria + +- All 3 new serializer tests pass. +- `cyclone validate-837 backend/tests/fixtures/co_medicaid_837p.txt` exits 0 and prints `Status: success` (mocked, no live HTTP). +- `cyclone validate-837 ingest/corrected-v2/v2-batch-*/tp-cyc-*.x12` (363 files) — all exit 0 (no PER/SBR errors) once the regen is re-run with the fixed serializer. Spot-checked with `grep NM1*PR2*CO_TXIX` per file. +- `POST /api/admin/validate-837` returns 401 unauthenticated, 403 with viewer cookie, 200 with admin cookie + multipart file. +- `resubmit-rejected-claims --ingest-dir ingest/corrected-v2` aborts files whose `OperationResult.Status == "error"`, continues with others, prints the failure list at the end. +- `--skip-edifabric-validate` flag bypasses the gate (for dev only). +- `secrets._ENV_NAME_FOR['edifabric.api_key'] == 'CYCLONE_EDIFABRIC_API_KEY'`. +- `cyclone secrets set edifabric.api_key ` persists; `get_secret('edifabric.api_key')` reads from Keychain on subsequent boots. +- RUNBOOK.md has the "After SP40 lands" section. + +## 8. Implementation plan summary + +(Full plan with checkboxes goes in `docs/superpowers/plans/2026-07-07-cyclone-edifabric-validation-gate.md` once this spec lands.) + +| Task | Scope | Estimated commits | +|---|---|---| +| 0 | Spec commit | `docs(spec): SP40 ...` | +| 1 | Serializer fixes (D4): `_build_per` + `_build_sbr` call sites, regression tests | `feat(sp40): 837P serializer always emits PER-02/03/04 + SBR-09` | +| 2 | `cyclone.edifabric` module + tests (D5, D7) | `feat(sp40): Edifabric HTTP client (read + validate)` | +| 3 | `cyclone validate-837` CLI + tests (D8) | `feat(sp40): cyclone validate-837 CLI` | +| 4 | `POST /api/admin/validate-837` HTTP + tests (D6) | `feat(sp40): POST /api/admin/validate-837 admin endpoint` | +| 5 | Pre-upload gate in `resubmit-rejected-claims` + bypass flag (D3, D8) | `feat(sp40): resubmit-rejected-claims pre-upload validation gate` | +| 6 | API key wiring: `_ENV_NAME_FOR` entry + dev fixture (D2) | `feat(sp40): edifabric.api_key secret wiring + dev fixture` | +| 7 | RUNBOOK.md "After SP40 lands" section (D5) | `docs(sp40): RUNBOOK entry for Edifabric validation` | +| 8 | Full suite + regen re-run + atomic merge | `merge: SP40 Edifabric validation gate + 837P byte-defect sweep into main` | \ No newline at end of file