cyclone/docs/superpowers/specs/2026-06-19-cyclone-837p-parser-design.md

Cyclone 837P Parser — Design

Date: 2026-06-19 Status: Approved (pending user review of this doc) Scope: v1 — Python module + CLI for parsing X12 837P professional claim files into per-claim JSON, with structural + Colorado-Medicaid-specific validation.

---

1. Overview

The Cyclone EDI suite (CuNtx) needs a Python module that ingests X12 837P files exported by AxisCare and produces one validated JSON file per claim. The output drives the existing Vite/React frontend (Cyclone) through a future FastAPI service that does not exist yet.

This design covers the parser, the data model, validation, and the CLI. It does not cover the HTTP service, database, or the 835 ERA parser — those are separate, future work items that this module is structured to enable.

2. Goals

1. Parse a real X12 837P file (e.g. docs/prodfiles/837p-from-axiscare/tp11525703-837P-20260618153358831-1of1.txt) into one structured JSON file per CLM segment.
2. Validate each claim against structural rules and Colorado Medical Assistance Program companion-guide rules, collect errors, and continue processing the rest of the batch.
3. Preserve the original raw segments inside each JSON for debugging and round-tripping.
4. Provide a CLI: python -m cyclone.cli parse-837 <input> --output-dir <dir>.
5. Be configurable so a future 835 parser can reuse segments, validator, and models with a different PayerConfig.

3. Non-goals (v1)

• FastAPI server / HTTP endpoints. The existing src/lib/api.ts is a stub; it stays a stub.
• Database persistence.
• Automatic claim repair (e.g. synthesizing a missing NPI). A --max-retries flag exists for v2 hook integration but no fixers ship in v1.
• 835 ERA parsing (separate spec).
• Code-set validation (CPT/HCPCS/ICD-10 vocab). Only structural shape checks.
• UI work. The frontend reads JSON files or — later — API responses; it does not change in this spec.

4. Stack

• Python: 3.11+ (uses match, tomllib, Self from typing).
• Pydantic: v2 (BaseModel, field_validator, model_config = ConfigDict(frozen=True, extra="forbid") for immutable input models).
• CLI: click 8.x.
• Tests: pytest 8.x.
• Linting (optional, not blocking v1): ruff.

No third-party EDI libraries. The X12 format is regular enough to tokenize in ~50 lines.

5. Directory layout

cyclone/backend/
├── pyproject.toml            # PEP 621 metadata, src layout, click + pydantic + pytest deps
├── README.md                 # install + usage
├── requirements.txt          # pinned for non-pipenv installs
├── .gitignore                # __pycache__, .pytest_cache, .venv, *.egg-info
├── src/cyclone/
│   ├── __init__.py
│   ├── cli.py                # click entrypoint: `python -m cyclone.cli parse-837 ...`
│   └── parsers/
│       ├── __init__.py       # public re-exports: parse_837, Claim, PayerConfig, ...
│       ├── exceptions.py     # CycloneParseError, CycloneValidationError
│       ├── segments.py       # tokenize, parse_isa_delimiters, EDIStream iterator
│       ├── models.py         # all Pydantic models
│       ├── payer.py          # PayerConfig + co_medicaid() factory
│       ├── validator.py      # structural + CO rules, returns ValidationReport
│       └── parse_837.py      # orchestrator: text -> list[Claim]
└── tests/
    ├── __init__.py
    ├── conftest.py           # shared fixtures: minimal 837P, CO Medicaid 837P
    ├── fixtures/
    │   ├── minimal_837p.txt
    │   └── co_medicaid_837p.txt
    ├── test_segments.py
    ├── test_parse_837.py
    ├── test_validator.py
    └── test_cli.py           # CLI smoke test; skips prodfile integration if absent

6. Data flow

input file (text)
   │
   ▼
segments.tokenize(text)
   │  detects ISA delimiters, returns list[list[str]]
   │  raises CycloneParseError on malformed ISA
   ▼
parse_837.parse(stream, payer_config)
   │  walks HL hierarchy, groups 2400 service lines under 2300 claims,
   │  emits one Claim per CLM segment
   │  collect errors; do not raise on per-claim failure
   ▼
validator.validate(claim, payer_config) → ValidationReport
   │  append-only; never raises
   ▼
list[ClaimOutput] (Pydantic)
   │
   ▼
cli writes:
   ./claims/claim-{ISA13}-{BHT04}.json   (per claim, even if failed)
   ./claims/summary.json                 (batch summary)

7. Data model

All models live in parsers/models.py. Models are Pydantic v2 BaseModels with model_config = ConfigDict(extra="ignore", str_strip_whitespace=True).

Top-level

Model	Notes
ParseResult	The orchestrator's return type: claims: list[ClaimOutput], summary: BatchSummary, envelope: Envelope.
Envelope	ISA / GS / ST / BHT, used to write filenames. Fields: sender_id, receiver_id, control_number (ISA13), transaction_date (BHT04), transaction_time (BHT05), implementation_guide (ST03).
BatchSummary	total_claims: int, passed: int, failed: int, failed_claim_ids: list[str], output_dir: str.

ClaimOutput

Field	Type	Source
claim_id	str	CLM01
control_number	str	ISA13
transaction_date	str (ISO date)	BHT04
billing_provider	BillingProvider	Loop 2010AA (NM185, N3, N4, REFEI)
subscriber	Subscriber	Loop 2010BA (NM1*IL, N3, N4, DMG)
payer	Payer	Loop 2010BB (NM1*PR)
claim	ClaimHeader	CLM, REF*G1
diagnoses	list[Diagnosis]	HI segment
service_lines	list[ServiceLine]	Loop 2400 (LX → SV1 → DTP472 → REF6R)
validation	ValidationReport	passed: bool, errors: list[ValidationIssue], warnings: list[ValidationIssue]
raw_segments	list[list[str]]	Original tokenized segments; always populated so JSON files are debuggable

Sub-models

• BillingProvider: npi, name, tax_id, address: Address
• Subscriber: first_name, last_name (split from NM103), member_id, dob (ISO), gender ("M"|"F"|"U"), address: Address
• Address: line1, line2?, city, state, zip
• Payer: name, id
• ClaimHeader: claim_id (CLM01), total_charge (CLM02, Decimal), place_of_service (CLM05-1), frequency_code (CLM05-3), provider_signature (CLM06), assignment (CLM07), release_of_info (CLM09), prior_auth? (REF*G1, optional — see assumption #5 in §14)
• Diagnosis: code, qualifier
• ServiceLine: line_number (LX01), procedure: Procedure, charge (SV102, Decimal), unit_type? (SV103), units? (SV104, Decimal), place_of_service? (SV105), service_date? (ISO, from DTP472), provider_reference? (REF6R)
• Procedure: qualifier (SV101-1), code (SV101-2), modifiers: list[str] (SV101-3..6)
• ValidationIssue: rule: str, severity: Literal["error","warning"], message: str, segment_index?: int
• ValidationReport: passed: bool, errors: list[ValidationIssue], warnings: list[ValidationIssue]

PayerConfig (payer.py)

class PayerConfig(BaseModel):
    name: str
    sbr09_claim_filing: str            # "MC" for CO Medicaid
    allowed_claim_frequencies: set[int]  # {1, 7, 8} for CO
    require_ref_g1_for_adjustments: bool
    allowed_bht06: set[str]              # {"CH"} for FFS CO; {"RP"} for encounter-only
    payer_id: str                        # "SKCO0"
    payer_name: str                      # "COHCPF"
    no_patient_loop: bool                # True for CO
    encounter_claim_in_same_batch: bool  # False for CO (CH and RP must be separate)

Built-ins:

• PayerConfig.co_medicaid() — defaults for Colorado Medical Assistance Program, derived from docs/companionguides/837p.md.
• PayerConfig.generic_837p() — relaxed defaults for unknown payers; only structural rules apply.

8. Parser internals (parse_837.py)

A single parse(text: str, payer_config: PayerConfig) -> ParseResult function:

1. Tokenize via segments.tokenize.
2. Validate the envelope: ISA / IEA, GS / GE, ST / SE. Mismatched control numbers are reported as a single fatal error on the ParseResult (the function never raises for envelope problems; the CLI decides what to do with summary.total_claims == 0).
3. Find the billing-provider HL (HL*1**20*1). Read the provider from 2010AA.
4. For each subsequent HL (HL*n*1*22*0):
   • Read subscriber from 2010BA.
   • Read payer from 2010BB.
   • Scan forward until the next HL or SE. For each CLM:
     • Start a new ClaimOutput.
     • Capture REF*G1 (prior auth / Payer Claim Control Number).
     • Capture HI (diagnoses).
     • For each subsequent LX/SV1 block, append a ServiceLine.
     • Hand the populated ClaimOutput to validator.validate; store the report back on the model.
5. Build BatchSummary.
6. Return ParseResult.

Per-claim failures (e.g. malformed CLM line) are caught with a try/except and recorded as a single error inside that claim's validation.errors. The parser never aborts mid-batch.

9. Validation rules (validator.py)

A list of pure functions Rule = Callable[[ClaimOutput, PayerConfig], Iterable[ValidationIssue]]. The validator runs all rules and concatenates results.

Rule ID	Severity	Description
R001_envelope_balanced	error	ISA/IEA, GS/GE, ST/SE control numbers match
R010_clm01_present	error	CLM01 non-empty
R011_total_charge_positive	error	CLM02 > 0
R020_npi_format	error	All XX-qualified NPIs match ^\d{10}$
R030_frequency_allowed	error	CLM05-3 ∈ payer_config.allowed_claim_frequencies
R031_ref_g1_optional	warning	REF*G1 is captured but not enforced in v1. The companion guide describes G1 as the encounter TCN, but the production data shows G1 used on every fee-for-service claim with values that look like prior-auth numbers. The rule exists as a placeholder for the payer to tighten the policy in a follow-up; for now it surfaces as informational.
R040_sbr09_matches_payer	error	SBR09 == payer_config.sbr09_claim_filing
R050_diagnosis_present	error	At least one HI diagnosis present
R060_service_dates_present	error	Every ServiceLine has a service_date
R070_charges_sum	warning	abs(sum(SV1.charge) - CLM02) < 0.01
R080_no_patient_loop	error	When payer_config.no_patient_loop, fail if a Loop 2000C (patient) HL appears
R090_bht06_allowed	error	BHT06 ∈ payer_config.allowed_bht06
R100_payer_id_matches	warning	Payer.id == payer_config.payer_id (mismatch is non-fatal; informational)

--strict upgrades warnings to errors at the CLI level (post-validation, not inside the rules).

10. Output files

The CLI writes to --output-dir:

<output-dir>/
├── claim-991102977-20260611.json     # one per CLM, named <ISA13>-<BHT04 YYYYMMDD>
├── claim-991102977-20260611.json
├── ...
└── summary.json

summary.json shape:

{
  "input_file": "/abs/path/to/input.txt",
  "control_number": "991102977",
  "transaction_date": "2026-06-11",
  "total_claims": 12,
  "passed": 11,
  "failed": 1,
  "failed_claim_ids": ["t991102977o1c3d"],
  "issues_by_rule": {"R070_charges_sum": 1}
}

If two claims in the same file share ISA13 and BHT04 (unusual but legal), the second file gets a numeric suffix: claim-991102977-20260611-2.json. Filenames are sanitized; control characters are stripped.

11. CLI

cli.py exposes a single command group:

python -m cyclone.cli parse-837 <input> \
    --output-dir <dir>             # required
    [--payer co_medicaid]          # default: co_medicaid
    [--strict]                     # warnings -> errors
    [--max-retries N]              # default 0; re-runs validation, does not patch
    [--include-raw-segments / --no-raw-segments]  # default: include
    [--log-level INFO]

Exit codes:

• 0 — every claim parsed (failures are written to JSON; this is by design)
• 2 — envelope-level failure (e.g. ISA not found, no CLM segments in the file)
• 1 — unexpected exception (bug); message printed to stderr

The CLI prints a one-line summary table to stdout:

parsed=12  passed=11  failed=1  output=./claims

12. Testing

Test file	Coverage
test_segments.py	ISA delimiter detection, edge cases (long ISA IDs, missing terminator), EDIStream iteration, raises on malformed input
test_parse_837.py	Parses fixtures/minimal_837p.txt and fixtures/co_medicaid_837p.txt; asserts claim count, model field values, raw segments preserved, errors collected (not raised) for known-bad input
test_validator.py	Each R0xx rule, both pass and fail cases; PayerConfig.co_medicaid() vs generic_837p()
test_cli.py	CliRunner invokes parse-837 on the synthetic fixture and asserts files written. An additional test reads one file from docs/prodfiles/837p-from-axiscare/ if any are present and is pytest.skip-ed otherwise.

Fixtures:

• minimal_837p.txt — hand-written, smallest valid 837P (one subscriber, one claim, one service line). ≤ 30 segments.
• co_medicaid_837p.txt — generated from a sanitized subset of a production file. Names are randomized, member IDs are stubbed, addresses use the same ZIP code with a generic street.

13. Out of scope / future work

• FastAPI server — separate spec. The ClaimOutput schema is the contract; the API will serialize the same models.
• 835 ERA parser — separate spec. Will share segments.py, validator.py (with a different rule list), and parts of models.py.
• Database persistence — separate spec.
• Auto-fix / claim repair — a --max-retries flag exists in v1 to test the wiring, but no fixers ship.
• Code-set lookups — CPT/HCPCS/ICD-10 vocab validation deferred; out of scope for v1.

14. Open assumptions (will be verified during implementation)

1. Each input file contains exactly one ISA / GS / ST envelope. Likely true for AxisCare exports but should be asserted.
2. Subscriber = patient (no 2000C loop) is enforced for CO Medicaid via the companion guide and the production file; this is captured by R080_no_patient_loop.
3. The production file uses the standard * element and ~ segment delimiters. Confirmed in the first 200 lines of tp11525703-837P-20260618153358831-1of1.txt.
4. REF*6R and REF*G1 are the only REF qualifiers we care about in v1. Other qualifiers (D9 for ICN, etc.) are captured as opaque strings in raw_segments for now.
5. REF*G1 semantic ambiguity — the CO companion guide describes G1 as the encounter TCN, but the production data shows G1 used on every FFS claim with small integer values that look like prior-auth numbers. v1 captures the value into claim.prior_auth and surfaces its presence as a warning; the rule is intentionally lenient and documented as a follow-up.