tyler/cyclone
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(spec): SP40 Edifabric validation gate + 837P byte-defect sweep

Browse Source 
Covers three things:
1. _build_per / _build_sbr call-site fixes so the serializer always
   emits PER-02/03/04 + SBR-09 (Edifabric rejects the current shape).
2. New cyclone.edifabric HTTP client wrapping the two-step
   /x12/read -> /x12/validate flow against api.edination.com.
3. Pre-upload gate inside resubmit-rejected-claims so an
   Edifabric-invalid file can't reach Gainwell.

Public eval API key 3ecf6b1c5cf34bd797a5f4c57951a1cf for dev.
Production is operator-supplied via cyclone secrets set
edifabric.api_key <paid-key>.
This commit is contained in:
Nora
2026-07-07 18:02:58 -06:00
parent 0436820023
commit f21201a22e
1 changed files with 124 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/superpowers/specs/2026-07-07-cyclone-edifabric-validation-gate-design.md
+124
View File
@@ -0,0 +1,124 @@
# SP40 — Edifabric validation gate + 837P byte-defect sweep
**Date:** 2026-07-07
**Status:** Draft, awaiting user sign-off
**Branch:** `sp40-edifabric-validation-gate`
**Aesthetic direction:** No new UI; one new CLI command + one HTTP endpoint + serializer hardening + a defensive HTTP client. Three new tests, two modified files, two new docs.
---
## 1. Scope
Edifabric's `POST /v2/x12/validate` (the operator's reference parser for CO Medicaid compliance) rejected the SP39-regenerated 837P files with three byte-level errors:
```
Line 1: Element PER-03 is required
Line 1: Element PER-04 is required
Line 1: Element SBR-09 is required
```
Root cause: `serialize_837._build_per` emits `PER*IC` (only PER01), and `_build_sbr` emits `SBR*P*18*******` (no SBR09) when the caller passes no contact / SBR09 kwargs. The clearhouse config has `submitter_contact_name="Tyler Martinez"` and `submitter_contact_email="tyler@dzinesco.com"` seeded, and `PayerConfig.co_medicaid().sbr09_claim_filing = "MC"` — both are available to the serializer today but not wired into the default emit. SP33-era callers and the SP39 regen script both bypass the kwargs, so every file they emit is invalid.
SP40 lands (a) the serializer fixes that close the gap, (b) the Edifabric integration that catches the same class of defect for any future payer / schema drift, (c) a pre-upload gate on `resubmit-rejected-claims` so the operator can't push an Edifabric-invalid file to Gainwell, and (d) the API key wiring through the existing `secrets.get_secret()` plumbing.
**In scope:**
- `_build_per` always emits PER02 (Name) + at least one PER03/PER04 pair (email or phone), sourced from `claim.billing_provider.contact` or the clearhouse config when the caller passes nothing.
- `_build_sbr` always emits SBR09 from `payer_cfg.sbr09_claim_filing` (already `"MC"` for CO Medicaid).
- A new `cyclone.edifabric` module — thin `httpx` client wrapping the two-step Edifabric flow (`/x12/read` → `/x12/validate`).
- A new CLI `cyclone validate-837 <file>` — reads the file, hits Edifabric, prints `OperationResult`; exits 0 on success / warning-only, 2 on any error.
- A new HTTP `POST /api/admin/validate-837` (admin-gated) — multipart upload of the .x12 file; same response shape.
- A pre-upload gate inside `resubmit-rejected-claims` — every file is validated before the SFTP `put`; `Status == "error"` aborts the file but continues the batch (other files still push). Bypass flag `--skip-edifabric-validate` for dev.
- API key wiring: `secrets.get_secret('edifabric.api_key')` with a new `_ENV_NAME_FOR` entry mapping it to `CYCLONE_EDIFABRIC_API_KEY`.
- A dev-fixture API key in `tests/fixtures/edifabric_api_key.txt` so CI doesn't need a real subscription (the public eval key `3ecf6b1c5cf34bd797a5f4c57951a1cf`).
- A `PERMISSIONS` matrix entry for `("POST", "/api/admin/validate-837"): ADMIN_ONLY`.
- RUNBOOK updates for the env var, the new CLI/HTTP, and the bypass flag.
**Out of scope:**
- 999 / 277CA / TA1 / 835 / 270 / 271 Edifabric validation (only 837P this SP).
- Building a custom EDI → X12Interchange JSON converter (we use Edifabric's `/x12/read` to do the conversion — see D1).
- Auto-retry on transient Edifabric 5xx — the gate surfaces the error; the operator decides whether to retry.
- Refreshing the 363 regen'd `ingest/corrected-v2/` files — the regen script already runs cleanly through the fixed serializer once Task 1 lands.
## 2. Decisions
- **D1.** Two-step Edifabric flow: `/x12/read` (raw EDI → `X12Interchange` JSON, `Content-Type: application/octet-stream`) then `/x12/validate` (JSON → `OperationResult`, `Content-Type: application/json`). Building our own EDI → JSON converter would be ~150 fields; `/x12/read` is the pragmatic path and the same library parses files we generate.
- **D2.** API key via `secrets.get_secret('edifabric.api_key')` → `CYCLONE_EDIFABRIC_API_KEY` env var or Keychain (`cyclone` service, `edifabric.api_key` account). The eval key in `tests/fixtures/edifabric_api_key.txt` is dev-only; production is operator-supplied via `cyclone secrets set edifabric.api_key <paid-key>`.
- **D3.** Pre-upload gate is **fail-closed**: any `Status == "error"` item in `OperationResult.Details` aborts the file (file is recorded as `failed`, batch continues). Warnings are logged at INFO level but don't block. This matches the operator's stated goal ("claims arent passing verification").
- **D4.** Serializer hardening is minimal + backward-compatible:
- `_build_per(contact_name, contact_phone, contact_email)` already handles all three; the fix is at the call site (where `submitter_kwargs` is built) — fall back to `clearhouse.submitter_contact_*` when the caller passes no kwargs. When none of those are set either, emit `PER*IC*CUSTOMER SERVICE*TE*8005550100` (a safe placeholder; documented in the function docstring as a temporary fallback so future per-billing-office contact info lands at the call site instead).
- `_build_sbr(individual_relationship_code, claim_filing_indicator_code)` already takes the SBR09 value; the fix is at the call site — pass `payer_cfg.sbr09_claim_filing` (default `"MC"` for CO Medicaid). When `payer_cfg` is unavailable, fall back to `"MC"` (matches the SP9-era seeding default).
- **D5.** HTTP client uses `httpx` (already in `pyproject.toml:35` `httpx>=0.27,<1`). One new module `cyclone.edifabric` with three functions:
- `read_interchange(edi_bytes, *, api_key) -> X12Interchange`
- `validate_interchange(x12_json, *, api_key) -> OperationResult`
- `validate_edi(edi_bytes, *, api_key) -> OperationResult` — composes the two.
Custom 4xx/5xx handling raises `EdifabricError(status, body)` so the caller can surface the Edifabric error verbatim.
- **D6.** HTTP endpoint is `POST /api/admin/validate-837` (matches existing `/api/admin/*` admin-only pattern from `cyclone/api_routers/admin.py:38-39`). Body is `multipart/form-data` with a single `file` part (FastAPI's `UploadFile`); response is the raw `OperationResult` JSON. The CLI uses the same module functions directly so the wire shape stays consistent.
- **D7.** Test mock strategy: `httpx.MockTransport` registered against the `httpx.Client` instance inside the `cyclone.edifabric` module (factory pattern lets tests inject the transport). No live HTTP in CI; the eval API key in `tests/fixtures/` is referenced but never sent — the mock intercepts first.
- **D8.** Pre-upload gate bypass: `--skip-edifabric-validate` flag on the CLI; no env var. Bypassing is an explicit operator action, not a global setting, so it's hard to forget about.
## 3. Files expected to change
| Path | Type | Purpose |
|---|---|---|
| `backend/src/cyclone/parsers/serialize_837.py` | modify | `_build_per` call site: fall back to clearhouse contact info. `_build_sbr` call site: always pass `sbr09_claim_filing`. |
| `backend/src/cyclone/edifabric.py` | create | `read_interchange`, `validate_interchange`, `validate_edi`, `EdifabricError`. |
| `backend/src/cyclone/cli.py` | modify | New `validate_837_cmd` (read file → call `validate_edi` → print `OperationResult`). Pre-upload gate call inside `resubmit_rejected_claims`. New `--skip-edifabric-validate` flag. |
| `backend/src/cyclone/api_routers/admin.py` | modify | New `POST /api/admin/validate-837` (multipart upload, returns `OperationResult`). |
| `backend/src/cyclone/secrets.py` | modify | Add `"edifabric.api_key": "CYCLONE_EDIFABRIC_API_KEY"` to `_ENV_NAME_FOR`. |
| `backend/src/cyclone/auth/permissions.py` | modify | Add `("POST", "/api/admin/validate-837"): ADMIN_ONLY`. |
| `backend/tests/test_serialize_837.py` | extend | 3 new regression tests: PER-02/03/04 always present, SBR-09 always present, no-kwargs fallback still emits valid PER + SBR. |
| `backend/tests/test_edifabric.py` | create | 4 tests with `httpx.MockTransport`: read succeeds, validate succeeds, validate returns errors, 5xx surfaces EdifabricError. |
| `backend/tests/test_validate_837_cli.py` | create | 4 CliRunner tests: clean file → exit 0 + status printed, error → exit 2, missing API key → exit 1, file not found → exit 2. |
| `backend/tests/test_api_validate_837.py` | create | 3 TestClient tests: admin auth gate (401 / 403), valid upload returns OperationResult, missing file part returns 422. |
| `backend/tests/test_resubmissions_cli.py` | extend | 1 new test: validation fail mid-batch aborts that file but continues the others; `--skip-edifabric-validate` flag bypasses the gate. |
| `backend/tests/fixtures/edifabric_api_key.txt` | create | The public eval key `3ecf6b1c5cf34bd797a5f4c57951a1cf` (file-based secret, read via `_FILE` env var in tests). |
| `docs/RUNBOOK.md` | modify | New "After SP40 lands" section: env var, secret setup, CLI/HTTP usage, bypass flag, expected OperationResult shape. |
| `docs/superpowers/specs/2026-07-07-cyclone-edifabric-validation-gate-design.md` | create | This spec, committed on the branch. |
## 4. Auth boundary
The HTTP endpoint inherits the existing admin router's `Depends(matrix_gate)` gate (`cyclone/api_routers/admin.py:39`); `PERMISSIONS` entry is `ADMIN_ONLY`. The API key never leaves the backend — the CLI and the HTTP handler both call `cyclone.edifabric.validate_edi(...)` server-side. The CLI is operator-only (no auth boundary in the CLI itself, per house convention). File-system threats unchanged: SQLCipher at rest, macOS Keychain for the secret. No new ports.
## 5. Open questions
- Should the `validate-837` HTTP endpoint accept a JSON body `{content: "ISA*..."}` as an alternative to multipart upload? Pro: easier UI integration. Con: more attack surface (request body in DB-sized chunks). Defer to a future SP if needed.
- Should the gate emit a per-claim `Resubmission` row update when validation fails? The current `Resubmission` row is only written on successful SFTP push, so a fail-closed gate would skip the row entirely — clean. If the operator wants an audit trail of *attempted* pushes, that's a follow-up column on `Resubmission`. Defer.
## 6. Test impact
- `test_serialize_837.py`: 3 new regression tests. Total file: 53 + 3 = 56 tests.
- `test_edifabric.py`: 4 new tests with mocked transport.
- `test_validate_837_cli.py`: 4 new CliRunner tests.
- `test_api_validate_837.py`: 3 new TestClient tests (auth gate, valid upload, missing file).
- `test_resubmissions_cli.py`: 1 new gate-fail test + 1 bypass-flag test.
- Total new tests: 13. Full suite target: 1458 + 13 = 1471 passing.
## 7. Acceptance criteria
- All 3 new serializer tests pass.
- `cyclone validate-837 backend/tests/fixtures/co_medicaid_837p.txt` exits 0 and prints `Status: success` (mocked, no live HTTP).
- `cyclone validate-837 ingest/corrected-v2/v2-batch-*/tp-cyc-*.x12` (363 files) — all exit 0 (no PER/SBR errors) once the regen is re-run with the fixed serializer. Spot-checked with `grep NM1*PR2*CO_TXIX` per file.
- `POST /api/admin/validate-837` returns 401 unauthenticated, 403 with viewer cookie, 200 with admin cookie + multipart file.
- `resubmit-rejected-claims --ingest-dir ingest/corrected-v2` aborts files whose `OperationResult.Status == "error"`, continues with others, prints the failure list at the end.
- `--skip-edifabric-validate` flag bypasses the gate (for dev only).
- `secrets._ENV_NAME_FOR['edifabric.api_key'] == 'CYCLONE_EDIFABRIC_API_KEY'`.
- `cyclone secrets set edifabric.api_key <key>` persists; `get_secret('edifabric.api_key')` reads from Keychain on subsequent boots.
- RUNBOOK.md has the "After SP40 lands" section.
## 8. Implementation plan summary
(Full plan with checkboxes goes in `docs/superpowers/plans/2026-07-07-cyclone-edifabric-validation-gate.md` once this spec lands.)
| Task | Scope | Estimated commits |
|---|---|---|
| 0 | Spec commit | `docs(spec): SP40 ...` |
| 1 | Serializer fixes (D4): `_build_per` + `_build_sbr` call sites, regression tests | `feat(sp40): 837P serializer always emits PER-02/03/04 + SBR-09` |
| 2 | `cyclone.edifabric` module + tests (D5, D7) | `feat(sp40): Edifabric HTTP client (read + validate)` |
| 3 | `cyclone validate-837` CLI + tests (D8) | `feat(sp40): cyclone validate-837 CLI` |
| 4 | `POST /api/admin/validate-837` HTTP + tests (D6) | `feat(sp40): POST /api/admin/validate-837 admin endpoint` |
| 5 | Pre-upload gate in `resubmit-rejected-claims` + bypass flag (D3, D8) | `feat(sp40): resubmit-rejected-claims pre-upload validation gate` |
| 6 | API key wiring: `_ENV_NAME_FOR` entry + dev fixture (D2) | `feat(sp40): edifabric.api_key secret wiring + dev fixture` |
| 7 | RUNBOOK.md "After SP40 lands" section (D5) | `docs(sp40): RUNBOOK entry for Edifabric validation` |
| 8 | Full suite + regen re-run + atomic merge | `merge: SP40 Edifabric validation gate + 837P byte-defect sweep into main` |