From d9dc8a86bd34ace476c7b788fd255f5644ec04f3 Mon Sep 17 00:00:00 2001 From: Cyclone Date: Fri, 19 Jun 2026 15:07:52 -0600 Subject: [PATCH] Add 837P parser design spec - Approved design for the Python 837P parser module under backend/ - Pydantic v2, click CLI, structural + CO Medicaid validation - One JSON file per claim, summary.json, continue-on-failure - Includes .gitignore for the Python + Node stack --- .gitignore | 33 ++ .../2026-06-19-cyclone-837p-parser-design.md | 281 ++++++++++++++++++ 2 files changed, 314 insertions(+) create mode 100644 .gitignore create mode 100644 docs/superpowers/specs/2026-06-19-cyclone-837p-parser-design.md diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..3b20bf8 --- /dev/null +++ b/.gitignore @@ -0,0 +1,33 @@ +# Dependencies +node_modules/ +.venv/ +venv/ +__pycache__/ +*.py[cod] +*.egg-info/ +.pytest_cache/ +.ruff_cache/ + +# Build output +dist/ +build/ +*.tsbuildinfo + +# Editor / OS +.DS_Store +.idea/ +.vscode/ +*.swp +*.swo + +# Local config +.env +.env.local +.env.*.local + +# Production data (handled by ops, not committed) +docs/prodfiles/*/ +*.production.txt + +# Local parser output +claims_output/ diff --git a/docs/superpowers/specs/2026-06-19-cyclone-837p-parser-design.md b/docs/superpowers/specs/2026-06-19-cyclone-837p-parser-design.md new file mode 100644 index 0000000..3f1bfc9 --- /dev/null +++ b/docs/superpowers/specs/2026-06-19-cyclone-837p-parser-design.md @@ -0,0 +1,281 @@ +# Cyclone 837P Parser — Design + +**Date:** 2026-06-19 +**Status:** Approved (pending user review of this doc) +**Scope:** v1 — Python module + CLI for parsing X12 837P professional claim files into per-claim JSON, with structural + Colorado-Medicaid-specific validation. + +--- + +## 1. Overview + +The Cyclone EDI suite (CuNtx) needs a Python module that ingests X12 837P files exported by AxisCare and produces one validated JSON file per claim. The output drives the existing Vite/React frontend (Cyclone) through a future FastAPI service that does not exist yet. + +This design covers the parser, the data model, validation, and the CLI. It does not cover the HTTP service, database, or the 835 ERA parser — those are separate, future work items that this module is structured to enable. + +## 2. Goals + +1. Parse a real X12 837P file (e.g. `docs/prodfiles/837p-from-axiscare/tp11525703-837P-20260618153358831-1of1.txt`) into one structured JSON file per `CLM` segment. +2. Validate each claim against structural rules and Colorado Medical Assistance Program companion-guide rules, collect errors, and continue processing the rest of the batch. +3. Preserve the original raw segments inside each JSON for debugging and round-tripping. +4. Provide a CLI: `python -m cyclone.cli parse-837 --output-dir `. +5. Be configurable so a future 835 parser can reuse `segments`, `validator`, and `models` with a different `PayerConfig`. + +## 3. Non-goals (v1) + +- FastAPI server / HTTP endpoints. The existing `src/lib/api.ts` is a stub; it stays a stub. +- Database persistence. +- Automatic claim repair (e.g. synthesizing a missing NPI). A `--max-retries` flag exists for v2 hook integration but no fixers ship in v1. +- 835 ERA parsing (separate spec). +- Code-set validation (CPT/HCPCS/ICD-10 vocab). Only structural shape checks. +- UI work. The frontend reads JSON files or — later — API responses; it does not change in this spec. + +## 4. Stack + +- **Python:** 3.11+ (uses `match`, `tomllib`, `Self` from typing). +- **Pydantic:** v2 (`BaseModel`, `field_validator`, `model_config = ConfigDict(frozen=True, extra="forbid")` for immutable input models). +- **CLI:** `click` 8.x. +- **Tests:** `pytest` 8.x. +- **Linting (optional, not blocking v1):** `ruff`. + +No third-party EDI libraries. The X12 format is regular enough to tokenize in ~50 lines. + +## 5. Directory layout + +``` +cyclone/backend/ +├── pyproject.toml # PEP 621 metadata, src layout, click + pydantic + pytest deps +├── README.md # install + usage +├── requirements.txt # pinned for non-pipenv installs +├── .gitignore # __pycache__, .pytest_cache, .venv, *.egg-info +├── src/cyclone/ +│ ├── __init__.py +│ ├── cli.py # click entrypoint: `python -m cyclone.cli parse-837 ...` +│ └── parsers/ +│ ├── __init__.py # public re-exports: parse_837, Claim, PayerConfig, ... +│ ├── exceptions.py # CycloneParseError, CycloneValidationError +│ ├── segments.py # tokenize, parse_isa_delimiters, EDIStream iterator +│ ├── models.py # all Pydantic models +│ ├── payer.py # PayerConfig + co_medicaid() factory +│ ├── validator.py # structural + CO rules, returns ValidationReport +│ └── parse_837.py # orchestrator: text -> list[Claim] +└── tests/ + ├── __init__.py + ├── conftest.py # shared fixtures: minimal 837P, CO Medicaid 837P + ├── fixtures/ + │ ├── minimal_837p.txt + │ └── co_medicaid_837p.txt + ├── test_segments.py + ├── test_parse_837.py + ├── test_validator.py + └── test_cli.py # CLI smoke test; skips prodfile integration if absent +``` + +## 6. Data flow + +``` +input file (text) + │ + ▼ +segments.tokenize(text) + │ detects ISA delimiters, returns list[list[str]] + │ raises CycloneParseError on malformed ISA + ▼ +parse_837.parse(stream, payer_config) + │ walks HL hierarchy, groups 2400 service lines under 2300 claims, + │ emits one Claim per CLM segment + │ collect errors; do not raise on per-claim failure + ▼ +validator.validate(claim, payer_config) → ValidationReport + │ append-only; never raises + ▼ +list[ClaimOutput] (Pydantic) + │ + ▼ +cli writes: + ./claims/claim-{ISA13}-{BHT04}.json (per claim, even if failed) + ./claims/summary.json (batch summary) +``` + +## 7. Data model + +All models live in `parsers/models.py`. Models are Pydantic v2 `BaseModel`s with `model_config = ConfigDict(extra="ignore", str_strip_whitespace=True)`. + +### Top-level + +| Model | Notes | +|---|---| +| `ParseResult` | The orchestrator's return type: `claims: list[ClaimOutput]`, `summary: BatchSummary`, `envelope: Envelope`. | +| `Envelope` | ISA / GS / ST / BHT, used to write filenames. Fields: `sender_id`, `receiver_id`, `control_number` (ISA13), `transaction_date` (BHT04), `transaction_time` (BHT05), `implementation_guide` (ST03). | +| `BatchSummary` | `total_claims: int`, `passed: int`, `failed: int`, `failed_claim_ids: list[str]`, `output_dir: str`. | + +### `ClaimOutput` + +| Field | Type | Source | +|---|---|---| +| `claim_id` | `str` | CLM01 | +| `control_number` | `str` | ISA13 | +| `transaction_date` | `str` (ISO date) | BHT04 | +| `billing_provider` | `BillingProvider` | Loop 2010AA (NM1*85, N3, N4, REF*EI) | +| `subscriber` | `Subscriber` | Loop 2010BA (NM1*IL, N3, N4, DMG) | +| `payer` | `Payer` | Loop 2010BB (NM1*PR) | +| `claim` | `ClaimHeader` | CLM, REF*G1 | +| `diagnoses` | `list[Diagnosis]` | HI segment | +| `service_lines` | `list[ServiceLine]` | Loop 2400 (LX → SV1 → DTP*472 → REF*6R) | +| `validation` | `ValidationReport` | `passed: bool`, `errors: list[ValidationIssue]`, `warnings: list[ValidationIssue]` | +| `raw_segments` | `list[list[str]]` | Original tokenized segments; **always populated** so JSON files are debuggable | + +### Sub-models + +- `BillingProvider`: `npi`, `name`, `tax_id`, `address: Address` +- `Subscriber`: `first_name`, `last_name` (split from NM103), `member_id`, `dob` (ISO), `gender` ("M"|"F"|"U"), `address: Address` +- `Address`: `line1`, `line2?`, `city`, `state`, `zip` +- `Payer`: `name`, `id` +- `ClaimHeader`: `claim_id` (CLM01), `total_charge` (CLM02, `Decimal`), `place_of_service` (CLM05-1), `frequency_code` (CLM05-3), `provider_signature` (CLM06), `assignment` (CLM07), `release_of_info` (CLM09), `prior_auth?` (REF*G1, optional — see assumption #5 in §14) +- `Diagnosis`: `code`, `qualifier` +- `ServiceLine`: `line_number` (LX01), `procedure: Procedure`, `charge` (SV102, `Decimal`), `unit_type?` (SV103), `units?` (SV104, `Decimal`), `place_of_service?` (SV105), `service_date?` (ISO, from DTP*472), `provider_reference?` (REF*6R) +- `Procedure`: `qualifier` (SV101-1), `code` (SV101-2), `modifiers: list[str]` (SV101-3..6) +- `ValidationIssue`: `rule: str`, `severity: Literal["error","warning"]`, `message: str`, `segment_index?: int` +- `ValidationReport`: `passed: bool`, `errors: list[ValidationIssue]`, `warnings: list[ValidationIssue]` + +### `PayerConfig` (`payer.py`) + +```python +class PayerConfig(BaseModel): + name: str + sbr09_claim_filing: str # "MC" for CO Medicaid + allowed_claim_frequencies: set[int] # {1, 7, 8} for CO + require_ref_g1_for_adjustments: bool + allowed_bht06: set[str] # {"CH"} for FFS CO; {"RP"} for encounter-only + payer_id: str # "SKCO0" + payer_name: str # "COHCPF" + no_patient_loop: bool # True for CO + encounter_claim_in_same_batch: bool # False for CO (CH and RP must be separate) +``` + +Built-ins: +- `PayerConfig.co_medicaid()` — defaults for Colorado Medical Assistance Program, derived from `docs/companionguides/837p.md`. +- `PayerConfig.generic_837p()` — relaxed defaults for unknown payers; only structural rules apply. + +## 8. Parser internals (`parse_837.py`) + +A single `parse(text: str, payer_config: PayerConfig) -> ParseResult` function: + +1. Tokenize via `segments.tokenize`. +2. Validate the envelope: ISA / IEA, GS / GE, ST / SE. Mismatched control numbers are reported as a single fatal error on the `ParseResult` (the function never raises for envelope problems; the CLI decides what to do with `summary.total_claims == 0`). +3. Find the billing-provider HL (`HL*1**20*1`). Read the provider from 2010AA. +4. For each subsequent HL (`HL*n*1*22*0`): + - Read subscriber from 2010BA. + - Read payer from 2010BB. + - Scan forward until the next HL or `SE`. For each `CLM`: + - Start a new `ClaimOutput`. + - Capture `REF*G1` (prior auth / Payer Claim Control Number). + - Capture `HI` (diagnoses). + - For each subsequent `LX`/`SV1` block, append a `ServiceLine`. + - Hand the populated `ClaimOutput` to `validator.validate`; store the report back on the model. +5. Build `BatchSummary`. +6. Return `ParseResult`. + +Per-claim failures (e.g. malformed `CLM` line) are caught with a `try/except` and recorded as a single error inside that claim's `validation.errors`. The parser never aborts mid-batch. + +## 9. Validation rules (`validator.py`) + +A list of pure functions `Rule = Callable[[ClaimOutput, PayerConfig], Iterable[ValidationIssue]]`. The validator runs all rules and concatenates results. + +| Rule ID | Severity | Description | +|---|---|---| +| `R001_envelope_balanced` | error | ISA/IEA, GS/GE, ST/SE control numbers match | +| `R010_clm01_present` | error | `CLM01` non-empty | +| `R011_total_charge_positive` | error | `CLM02 > 0` | +| `R020_npi_format` | error | All `XX`-qualified NPIs match `^\d{10}$` | +| `R030_frequency_allowed` | error | `CLM05-3 ∈ payer_config.allowed_claim_frequencies` | +| `R031_ref_g1_optional` | warning | REF\*G1 is captured but **not enforced** in v1. The companion guide describes G1 as the encounter TCN, but the production data shows G1 used on every fee-for-service claim with values that look like prior-auth numbers. The rule exists as a placeholder for the payer to tighten the policy in a follow-up; for now it surfaces as informational. | +| `R040_sbr09_matches_payer` | error | `SBR09 == payer_config.sbr09_claim_filing` | +| `R050_diagnosis_present` | error | At least one `HI` diagnosis present | +| `R060_service_dates_present` | error | Every `ServiceLine` has a `service_date` | +| `R070_charges_sum` | warning | `abs(sum(SV1.charge) - CLM02) < 0.01` | +| `R080_no_patient_loop` | error | When `payer_config.no_patient_loop`, fail if a Loop 2000C (patient) HL appears | +| `R090_bht06_allowed` | error | `BHT06 ∈ payer_config.allowed_bht06` | +| `R100_payer_id_matches` | warning | `Payer.id == payer_config.payer_id` (mismatch is non-fatal; informational) | + +`--strict` upgrades warnings to errors at the CLI level (post-validation, not inside the rules). + +## 10. Output files + +The CLI writes to `--output-dir`: + +``` +/ +├── claim-991102977-20260611.json # one per CLM, named - +├── claim-991102977-20260611.json +├── ... +└── summary.json +``` + +`summary.json` shape: +```json +{ + "input_file": "/abs/path/to/input.txt", + "control_number": "991102977", + "transaction_date": "2026-06-11", + "total_claims": 12, + "passed": 11, + "failed": 1, + "failed_claim_ids": ["t991102977o1c3d"], + "issues_by_rule": {"R070_charges_sum": 1} +} +``` + +If two claims in the same file share `ISA13` and `BHT04` (unusual but legal), the second file gets a numeric suffix: `claim-991102977-20260611-2.json`. Filenames are sanitized; control characters are stripped. + +## 11. CLI + +`cli.py` exposes a single command group: + +```bash +python -m cyclone.cli parse-837 \ + --output-dir # required + [--payer co_medicaid] # default: co_medicaid + [--strict] # warnings -> errors + [--max-retries N] # default 0; re-runs validation, does not patch + [--include-raw-segments / --no-raw-segments] # default: include + [--log-level INFO] +``` + +Exit codes: +- `0` — every claim parsed (failures are written to JSON; this is by design) +- `2` — envelope-level failure (e.g. ISA not found, no `CLM` segments in the file) +- `1` — unexpected exception (bug); message printed to stderr + +The CLI prints a one-line summary table to stdout: +``` +parsed=12 passed=11 failed=1 output=./claims +``` + +## 12. Testing + +| Test file | Coverage | +|---|---| +| `test_segments.py` | ISA delimiter detection, edge cases (long ISA IDs, missing terminator), `EDIStream` iteration, raises on malformed input | +| `test_parse_837.py` | Parses `fixtures/minimal_837p.txt` and `fixtures/co_medicaid_837p.txt`; asserts claim count, model field values, raw segments preserved, errors collected (not raised) for known-bad input | +| `test_validator.py` | Each `R0xx` rule, both pass and fail cases; `PayerConfig.co_medicaid()` vs `generic_837p()` | +| `test_cli.py` | `CliRunner` invokes `parse-837` on the synthetic fixture and asserts files written. An additional test reads one file from `docs/prodfiles/837p-from-axiscare/` if any are present and is `pytest.skip`-ed otherwise. | + +Fixtures: +- `minimal_837p.txt` — hand-written, smallest valid 837P (one subscriber, one claim, one service line). ≤ 30 segments. +- `co_medicaid_837p.txt` — generated from a sanitized subset of a production file. Names are randomized, member IDs are stubbed, addresses use the same ZIP code with a generic street. + +## 13. Out of scope / future work + +- **FastAPI server** — separate spec. The `ClaimOutput` schema is the contract; the API will serialize the same models. +- **835 ERA parser** — separate spec. Will share `segments.py`, `validator.py` (with a different rule list), and parts of `models.py`. +- **Database persistence** — separate spec. +- **Auto-fix / claim repair** — a `--max-retries` flag exists in v1 to test the wiring, but no fixers ship. +- **Code-set lookups** — CPT/HCPCS/ICD-10 vocab validation deferred; out of scope for v1. + +## 14. Open assumptions (will be verified during implementation) + +1. Each input file contains exactly one ISA / GS / ST envelope. **Likely true** for AxisCare exports but should be asserted. +2. Subscriber = patient (no 2000C loop) is enforced for CO Medicaid via the companion guide and the production file; this is captured by `R080_no_patient_loop`. +3. The production file uses the standard `*` element and `~` segment delimiters. **Confirmed** in the first 200 lines of `tp11525703-837P-20260618153358831-1of1.txt`. +4. `REF*6R` and `REF*G1` are the only REF qualifiers we care about in v1. Other qualifiers (D9 for ICN, etc.) are captured as opaque strings in `raw_segments` for now. +5. **REF*G1 semantic ambiguity** — the CO companion guide describes G1 as the encounter TCN, but the production data shows G1 used on every FFS claim with small integer values that look like prior-auth numbers. v1 captures the value into `claim.prior_auth` and surfaces its presence as a warning; the rule is intentionally lenient and documented as a follow-up.