cyclone/docs/superpowers/specs/2026-07-07-cyclone-edifabric-validation-gate-design.md

SP40 — Edifabric validation gate + 837P byte-defect sweep

Date: 2026-07-07 Status: Draft, awaiting user sign-off Branch: sp40-edifabric-validation-gate Aesthetic direction: No new UI; one new CLI command + one HTTP endpoint + serializer hardening + a defensive HTTP client. Three new tests, two modified files, two new docs.

---

1. Scope

Edifabric's POST /v2/x12/validate (the operator's reference parser for CO Medicaid compliance) rejected the SP39-regenerated 837P files with three byte-level errors:

Line 1: Element PER-03 is required
Line 1: Element PER-04 is required
Line 1: Element SBR-09 is required

Root cause: serialize_837._build_per emits PER*IC (only PER01), and _build_sbr emits SBR*P*18******* (no SBR09) when the caller passes no contact / SBR09 kwargs. The clearhouse config has submitter_contact_name="Tyler Martinez" and submitter_contact_email="tyler@dzinesco.com" seeded, and PayerConfig.co_medicaid().sbr09_claim_filing = "MC" — both are available to the serializer today but not wired into the default emit. SP33-era callers and the SP39 regen script both bypass the kwargs, so every file they emit is invalid.

SP40 lands (a) the serializer fixes that close the gap, (b) the Edifabric integration that catches the same class of defect for any future payer / schema drift, (c) a pre-upload gate on resubmit-rejected-claims so the operator can't push an Edifabric-invalid file to Gainwell, and (d) the API key wiring through the existing secrets.get_secret() plumbing.

In scope:

• _build_per always emits PER02 (Name) + at least one PER03/PER04 pair (email or phone), sourced from claim.billing_provider.contact or the clearhouse config when the caller passes nothing.
• _build_sbr always emits SBR09 from payer_cfg.sbr09_claim_filing (already "MC" for CO Medicaid).
• A new cyclone.edifabric module — thin httpx client wrapping the two-step Edifabric flow (/x12/read → /x12/validate).
• A new CLI cyclone validate-837 <file> — reads the file, hits Edifabric, prints OperationResult; exits 0 on success / warning-only, 2 on any error.
• A new HTTP POST /api/admin/validate-837 (admin-gated) — multipart upload of the .x12 file; same response shape.
• A pre-upload gate inside resubmit-rejected-claims — every file is validated before the SFTP put; Status == "error" aborts the file but continues the batch (other files still push). Bypass flag --skip-edifabric-validate for dev.
• API key wiring: secrets.get_secret('edifabric.api_key') with a new _ENV_NAME_FOR entry mapping it to CYCLONE_EDIFABRIC_API_KEY.
• A dev-fixture API key in tests/fixtures/edifabric_api_key.txt so CI doesn't need a real subscription (the public eval key 3ecf6b1c5cf34bd797a5f4c57951a1cf).
• A PERMISSIONS matrix entry for ("POST", "/api/admin/validate-837"): ADMIN_ONLY.
• RUNBOOK updates for the env var, the new CLI/HTTP, and the bypass flag.

Out of scope:

• 999 / 277CA / TA1 / 835 / 270 / 271 Edifabric validation (only 837P this SP).
• Building a custom EDI → X12Interchange JSON converter (we use Edifabric's /x12/read to do the conversion — see D1).
• Auto-retry on transient Edifabric 5xx — the gate surfaces the error; the operator decides whether to retry.
• Refreshing the 363 regen'd ingest/corrected-v2/ files — the regen script already runs cleanly through the fixed serializer once Task 1 lands.

2. Decisions

• D1. Two-step Edifabric flow: /x12/read (raw EDI → X12Interchange JSON, Content-Type: application/octet-stream) then /x12/validate (JSON → OperationResult, Content-Type: application/json). Building our own EDI → JSON converter would be ~150 fields; /x12/read is the pragmatic path and the same library parses files we generate.
• D2. API key via secrets.get_secret('edifabric.api_key') → CYCLONE_EDIFABRIC_API_KEY env var or Keychain (cyclone service, edifabric.api_key account). The eval key in tests/fixtures/edifabric_api_key.txt is dev-only; production is operator-supplied via cyclone secrets set edifabric.api_key <paid-key>.
• D3. Pre-upload gate is fail-closed: any Status == "error" item in OperationResult.Details aborts the file (file is recorded as failed, batch continues). Warnings are logged at INFO level but don't block. This matches the operator's stated goal ("claims arent passing verification").
• D4. Serializer hardening is minimal + backward-compatible:
  • _build_per(contact_name, contact_phone, contact_email) already handles all three; the fix is at the call site (where submitter_kwargs is built) — fall back to clearhouse.submitter_contact_* when the caller passes no kwargs. When none of those are set either, emit PER*IC*CUSTOMER SERVICE*TE*8005550100 (a safe placeholder; documented in the function docstring as a temporary fallback so future per-billing-office contact info lands at the call site instead).
  • _build_sbr(individual_relationship_code, claim_filing_indicator_code) already takes the SBR09 value; the fix is at the call site — pass payer_cfg.sbr09_claim_filing (default "MC" for CO Medicaid). When payer_cfg is unavailable, fall back to "MC" (matches the SP9-era seeding default).
• D5. HTTP client uses httpx (already in pyproject.toml:35 httpx>=0.27,<1). One new module cyclone.edifabric with three functions:
  • read_interchange(edi_bytes, *, api_key) -> X12Interchange
  • validate_interchange(x12_json, *, api_key) -> OperationResult
  • validate_edi(edi_bytes, *, api_key) -> OperationResult — composes the two. Custom 4xx/5xx handling raises EdifabricError(status, body) so the caller can surface the Edifabric error verbatim.
• D6. HTTP endpoint is POST /api/admin/validate-837 (matches existing /api/admin/* admin-only pattern from cyclone/api_routers/admin.py:38-39). Body is multipart/form-data with a single file part (FastAPI's UploadFile); response is the raw OperationResult JSON. The CLI uses the same module functions directly so the wire shape stays consistent.
• D7. Test mock strategy: httpx.MockTransport registered against the httpx.Client instance inside the cyclone.edifabric module (factory pattern lets tests inject the transport). No live HTTP in CI; the eval API key in tests/fixtures/ is referenced but never sent — the mock intercepts first.
• D8. Pre-upload gate bypass: --skip-edifabric-validate flag on the CLI; no env var. Bypassing is an explicit operator action, not a global setting, so it's hard to forget about.

3. Files expected to change

Path	Type	Purpose
backend/src/cyclone/parsers/serialize_837.py	modify	_build_per call site: fall back to clearhouse contact info. _build_sbr call site: always pass sbr09_claim_filing.
backend/src/cyclone/edifabric.py	create	read_interchange, validate_interchange, validate_edi, EdifabricError.
backend/src/cyclone/cli.py	modify	New validate_837_cmd (read file → call validate_edi → print OperationResult). Pre-upload gate call inside resubmit_rejected_claims. New --skip-edifabric-validate flag.
backend/src/cyclone/api_routers/admin.py	modify	New POST /api/admin/validate-837 (multipart upload, returns OperationResult).
backend/src/cyclone/secrets.py	modify	Add "edifabric.api_key": "CYCLONE_EDIFABRIC_API_KEY" to _ENV_NAME_FOR.
backend/src/cyclone/auth/permissions.py	modify	Add ("POST", "/api/admin/validate-837"): ADMIN_ONLY.
backend/tests/test_serialize_837.py	extend	3 new regression tests: PER-02/03/04 always present, SBR-09 always present, no-kwargs fallback still emits valid PER + SBR.
backend/tests/test_edifabric.py	create	4 tests with httpx.MockTransport: read succeeds, validate succeeds, validate returns errors, 5xx surfaces EdifabricError.
backend/tests/test_validate_837_cli.py	create	4 CliRunner tests: clean file → exit 0 + status printed, error → exit 2, missing API key → exit 1, file not found → exit 2.
backend/tests/test_api_validate_837.py	create	3 TestClient tests: admin auth gate (401 / 403), valid upload returns OperationResult, missing file part returns 422.
backend/tests/test_resubmissions_cli.py	extend	1 new test: validation fail mid-batch aborts that file but continues the others; --skip-edifabric-validate flag bypasses the gate.
backend/tests/fixtures/edifabric_api_key.txt	create	The public eval key 3ecf6b1c5cf34bd797a5f4c57951a1cf (file-based secret, read via _FILE env var in tests).
docs/RUNBOOK.md	modify	New "After SP40 lands" section: env var, secret setup, CLI/HTTP usage, bypass flag, expected OperationResult shape.
docs/superpowers/specs/2026-07-07-cyclone-edifabric-validation-gate-design.md	create	This spec, committed on the branch.

4. Auth boundary

The HTTP endpoint inherits the existing admin router's Depends(matrix_gate) gate (cyclone/api_routers/admin.py:39); PERMISSIONS entry is ADMIN_ONLY. The API key never leaves the backend — the CLI and the HTTP handler both call cyclone.edifabric.validate_edi(...) server-side. The CLI is operator-only (no auth boundary in the CLI itself, per house convention). File-system threats unchanged: SQLCipher at rest, macOS Keychain for the secret. No new ports.

5. Open questions

• Should the validate-837 HTTP endpoint accept a JSON body {content: "ISA*..."} as an alternative to multipart upload? Pro: easier UI integration. Con: more attack surface (request body in DB-sized chunks). Defer to a future SP if needed.
• Should the gate emit a per-claim Resubmission row update when validation fails? The current Resubmission row is only written on successful SFTP push, so a fail-closed gate would skip the row entirely — clean. If the operator wants an audit trail of attempted pushes, that's a follow-up column on Resubmission. Defer.

6. Test impact

• test_serialize_837.py: 3 new regression tests. Total file: 53 + 3 = 56 tests.
• test_edifabric.py: 4 new tests with mocked transport.
• test_validate_837_cli.py: 4 new CliRunner tests.
• test_api_validate_837.py: 3 new TestClient tests (auth gate, valid upload, missing file).
• test_resubmissions_cli.py: 1 new gate-fail test + 1 bypass-flag test.
• Total new tests: 13. Full suite target: 1458 + 13 = 1471 passing.

7. Acceptance criteria

• All 3 new serializer tests pass.
• cyclone validate-837 backend/tests/fixtures/co_medicaid_837p.txt exits 0 and prints Status: success (mocked, no live HTTP).
• cyclone validate-837 ingest/corrected-v2/v2-batch-*/tp-cyc-*.x12 (363 files) — all exit 0 (no PER/SBR errors) once the regen is re-run with the fixed serializer. Spot-checked with grep NM1*PR2*CO_TXIX per file.
• POST /api/admin/validate-837 returns 401 unauthenticated, 403 with viewer cookie, 200 with admin cookie + multipart file.
• resubmit-rejected-claims --ingest-dir ingest/corrected-v2 aborts files whose OperationResult.Status == "error", continues with others, prints the failure list at the end.
• --skip-edifabric-validate flag bypasses the gate (for dev only).
• secrets._ENV_NAME_FOR['edifabric.api_key'] == 'CYCLONE_EDIFABRIC_API_KEY'.
• cyclone secrets set edifabric.api_key <key> persists; get_secret('edifabric.api_key') reads from Keychain on subsequent boots.
• RUNBOOK.md has the "After SP40 lands" section.

8. Implementation plan summary

(Full plan with checkboxes goes in docs/superpowers/plans/2026-07-07-cyclone-edifabric-validation-gate.md once this spec lands.)

Task	Scope	Estimated commits
0	Spec commit	docs(spec): SP40 ...
1	Serializer fixes (D4): _build_per + _build_sbr call sites, regression tests	feat(sp40): 837P serializer always emits PER-02/03/04 + SBR-09
2	cyclone.edifabric module + tests (D5, D7)	feat(sp40): Edifabric HTTP client (read + validate)
3	cyclone validate-837 CLI + tests (D8)	feat(sp40): cyclone validate-837 CLI
4	POST /api/admin/validate-837 HTTP + tests (D6)	feat(sp40): POST /api/admin/validate-837 admin endpoint
5	Pre-upload gate in resubmit-rejected-claims + bypass flag (D3, D8)	feat(sp40): resubmit-rejected-claims pre-upload validation gate
6	API key wiring: _ENV_NAME_FOR entry + dev fixture (D2)	feat(sp40): edifabric.api_key secret wiring + dev fixture
7	RUNBOOK.md "After SP40 lands" section (D5)	docs(sp40): RUNBOOK entry for Edifabric validation
8	Full suite + regen re-run + atomic merge	merge: SP40 Edifabric validation gate + 837P byte-defect sweep into main