Add 837P parser design spec
- Approved design for the Python 837P parser module under backend/ - Pydantic v2, click CLI, structural + CO Medicaid validation - One JSON file per claim, summary.json, continue-on-failure - Includes .gitignore for the Python + Node stack
This commit is contained in:
Cyclone
+33
@@ -0,0 +1,33 @@
|
|||||||
|
# Dependencies
|
||||||
|
node_modules/
|
||||||
|
.venv/
|
||||||
|
venv/
|
||||||
|
__pycache__/
|
||||||
|
*.py[cod]
|
||||||
|
*.egg-info/
|
||||||
|
.pytest_cache/
|
||||||
|
.ruff_cache/
|
||||||
|
|
||||||
|
# Build output
|
||||||
|
dist/
|
||||||
|
build/
|
||||||
|
*.tsbuildinfo
|
||||||
|
|
||||||
|
# Editor / OS
|
||||||
|
.DS_Store
|
||||||
|
.idea/
|
||||||
|
.vscode/
|
||||||
|
*.swp
|
||||||
|
*.swo
|
||||||
|
|
||||||
|
# Local config
|
||||||
|
.env
|
||||||
|
.env.local
|
||||||
|
.env.*.local
|
||||||
|
|
||||||
|
# Production data (handled by ops, not committed)
|
||||||
|
docs/prodfiles/*/
|
||||||
|
*.production.txt
|
||||||
|
|
||||||
|
# Local parser output
|
||||||
|
claims_output/
|
||||||
@@ -0,0 +1,281 @@
|
|||||||
|
# Cyclone 837P Parser — Design
|
||||||
|
|
||||||
|
**Date:** 2026-06-19
|
||||||
|
**Status:** Approved (pending user review of this doc)
|
||||||
|
**Scope:** v1 — Python module + CLI for parsing X12 837P professional claim files into per-claim JSON, with structural + Colorado-Medicaid-specific validation.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. Overview
|
||||||
|
|
||||||
|
The Cyclone EDI suite (CuNtx) needs a Python module that ingests X12 837P files exported by AxisCare and produces one validated JSON file per claim. The output drives the existing Vite/React frontend (Cyclone) through a future FastAPI service that does not exist yet.
|
||||||
|
|
||||||
|
This design covers the parser, the data model, validation, and the CLI. It does not cover the HTTP service, database, or the 835 ERA parser — those are separate, future work items that this module is structured to enable.
|
||||||
|
|
||||||
|
## 2. Goals
|
||||||
|
|
||||||
|
1. Parse a real X12 837P file (e.g. `docs/prodfiles/837p-from-axiscare/tp11525703-837P-20260618153358831-1of1.txt`) into one structured JSON file per `CLM` segment.
|
||||||
|
2. Validate each claim against structural rules and Colorado Medical Assistance Program companion-guide rules, collect errors, and continue processing the rest of the batch.
|
||||||
|
3. Preserve the original raw segments inside each JSON for debugging and round-tripping.
|
||||||
|
4. Provide a CLI: `python -m cyclone.cli parse-837 <input> --output-dir <dir>`.
|
||||||
|
5. Be configurable so a future 835 parser can reuse `segments`, `validator`, and `models` with a different `PayerConfig`.
|
||||||
|
|
||||||
|
## 3. Non-goals (v1)
|
||||||
|
|
||||||
|
- FastAPI server / HTTP endpoints. The existing `src/lib/api.ts` is a stub; it stays a stub.
|
||||||
|
- Database persistence.
|
||||||
|
- Automatic claim repair (e.g. synthesizing a missing NPI). A `--max-retries` flag exists for v2 hook integration but no fixers ship in v1.
|
||||||
|
- 835 ERA parsing (separate spec).
|
||||||
|
- Code-set validation (CPT/HCPCS/ICD-10 vocab). Only structural shape checks.
|
||||||
|
- UI work. The frontend reads JSON files or — later — API responses; it does not change in this spec.
|
||||||
|
|
||||||
|
## 4. Stack
|
||||||
|
|
||||||
|
- **Python:** 3.11+ (uses `match`, `tomllib`, `Self` from typing).
|
||||||
|
- **Pydantic:** v2 (`BaseModel`, `field_validator`, `model_config = ConfigDict(frozen=True, extra="forbid")` for immutable input models).
|
||||||
|
- **CLI:** `click` 8.x.
|
||||||
|
- **Tests:** `pytest` 8.x.
|
||||||
|
- **Linting (optional, not blocking v1):** `ruff`.
|
||||||
|
|
||||||
|
No third-party EDI libraries. The X12 format is regular enough to tokenize in ~50 lines.
|
||||||
|
|
||||||
|
## 5. Directory layout
|
||||||
|
|
||||||
|
```
|
||||||
|
cyclone/backend/
|
||||||
|
├── pyproject.toml # PEP 621 metadata, src layout, click + pydantic + pytest deps
|
||||||
|
├── README.md # install + usage
|
||||||
|
├── requirements.txt # pinned for non-pipenv installs
|
||||||
|
├── .gitignore # __pycache__, .pytest_cache, .venv, *.egg-info
|
||||||
|
├── src/cyclone/
|
||||||
|
│ ├── __init__.py
|
||||||
|
│ ├── cli.py # click entrypoint: `python -m cyclone.cli parse-837 ...`
|
||||||
|
│ └── parsers/
|
||||||
|
│ ├── __init__.py # public re-exports: parse_837, Claim, PayerConfig, ...
|
||||||
|
│ ├── exceptions.py # CycloneParseError, CycloneValidationError
|
||||||
|
│ ├── segments.py # tokenize, parse_isa_delimiters, EDIStream iterator
|
||||||
|
│ ├── models.py # all Pydantic models
|
||||||
|
│ ├── payer.py # PayerConfig + co_medicaid() factory
|
||||||
|
│ ├── validator.py # structural + CO rules, returns ValidationReport
|
||||||
|
│ └── parse_837.py # orchestrator: text -> list[Claim]
|
||||||
|
└── tests/
|
||||||
|
├── __init__.py
|
||||||
|
├── conftest.py # shared fixtures: minimal 837P, CO Medicaid 837P
|
||||||
|
├── fixtures/
|
||||||
|
│ ├── minimal_837p.txt
|
||||||
|
│ └── co_medicaid_837p.txt
|
||||||
|
├── test_segments.py
|
||||||
|
├── test_parse_837.py
|
||||||
|
├── test_validator.py
|
||||||
|
└── test_cli.py # CLI smoke test; skips prodfile integration if absent
|
||||||
|
```
|
||||||
|
|
||||||
|
## 6. Data flow
|
||||||
|
|
||||||
|
```
|
||||||
|
input file (text)
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
segments.tokenize(text)
|
||||||
|
│ detects ISA delimiters, returns list[list[str]]
|
||||||
|
│ raises CycloneParseError on malformed ISA
|
||||||
|
▼
|
||||||
|
parse_837.parse(stream, payer_config)
|
||||||
|
│ walks HL hierarchy, groups 2400 service lines under 2300 claims,
|
||||||
|
│ emits one Claim per CLM segment
|
||||||
|
│ collect errors; do not raise on per-claim failure
|
||||||
|
▼
|
||||||
|
validator.validate(claim, payer_config) → ValidationReport
|
||||||
|
│ append-only; never raises
|
||||||
|
▼
|
||||||
|
list[ClaimOutput] (Pydantic)
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
cli writes:
|
||||||
|
./claims/claim-{ISA13}-{BHT04}.json (per claim, even if failed)
|
||||||
|
./claims/summary.json (batch summary)
|
||||||
|
```
|
||||||
|
|
||||||
|
## 7. Data model
|
||||||
|
|
||||||
|
All models live in `parsers/models.py`. Models are Pydantic v2 `BaseModel`s with `model_config = ConfigDict(extra="ignore", str_strip_whitespace=True)`.
|
||||||
|
|
||||||
|
### Top-level
|
||||||
|
|
||||||
|
| Model | Notes |
|
||||||
|
|---|---|
|
||||||
|
| `ParseResult` | The orchestrator's return type: `claims: list[ClaimOutput]`, `summary: BatchSummary`, `envelope: Envelope`. |
|
||||||
|
| `Envelope` | ISA / GS / ST / BHT, used to write filenames. Fields: `sender_id`, `receiver_id`, `control_number` (ISA13), `transaction_date` (BHT04), `transaction_time` (BHT05), `implementation_guide` (ST03). |
|
||||||
|
| `BatchSummary` | `total_claims: int`, `passed: int`, `failed: int`, `failed_claim_ids: list[str]`, `output_dir: str`. |
|
||||||
|
|
||||||
|
### `ClaimOutput`
|
||||||
|
|
||||||
|
| Field | Type | Source |
|
||||||
|
|---|---|---|
|
||||||
|
| `claim_id` | `str` | CLM01 |
|
||||||
|
| `control_number` | `str` | ISA13 |
|
||||||
|
| `transaction_date` | `str` (ISO date) | BHT04 |
|
||||||
|
| `billing_provider` | `BillingProvider` | Loop 2010AA (NM1*85, N3, N4, REF*EI) |
|
||||||
|
| `subscriber` | `Subscriber` | Loop 2010BA (NM1*IL, N3, N4, DMG) |
|
||||||
|
| `payer` | `Payer` | Loop 2010BB (NM1*PR) |
|
||||||
|
| `claim` | `ClaimHeader` | CLM, REF*G1 |
|
||||||
|
| `diagnoses` | `list[Diagnosis]` | HI segment |
|
||||||
|
| `service_lines` | `list[ServiceLine]` | Loop 2400 (LX → SV1 → DTP*472 → REF*6R) |
|
||||||
|
| `validation` | `ValidationReport` | `passed: bool`, `errors: list[ValidationIssue]`, `warnings: list[ValidationIssue]` |
|
||||||
|
| `raw_segments` | `list[list[str]]` | Original tokenized segments; **always populated** so JSON files are debuggable |
|
||||||
|
|
||||||
|
### Sub-models
|
||||||
|
|
||||||
|
- `BillingProvider`: `npi`, `name`, `tax_id`, `address: Address`
|
||||||
|
- `Subscriber`: `first_name`, `last_name` (split from NM103), `member_id`, `dob` (ISO), `gender` ("M"|"F"|"U"), `address: Address`
|
||||||
|
- `Address`: `line1`, `line2?`, `city`, `state`, `zip`
|
||||||
|
- `Payer`: `name`, `id`
|
||||||
|
- `ClaimHeader`: `claim_id` (CLM01), `total_charge` (CLM02, `Decimal`), `place_of_service` (CLM05-1), `frequency_code` (CLM05-3), `provider_signature` (CLM06), `assignment` (CLM07), `release_of_info` (CLM09), `prior_auth?` (REF*G1, optional — see assumption #5 in §14)
|
||||||
|
- `Diagnosis`: `code`, `qualifier`
|
||||||
|
- `ServiceLine`: `line_number` (LX01), `procedure: Procedure`, `charge` (SV102, `Decimal`), `unit_type?` (SV103), `units?` (SV104, `Decimal`), `place_of_service?` (SV105), `service_date?` (ISO, from DTP*472), `provider_reference?` (REF*6R)
|
||||||
|
- `Procedure`: `qualifier` (SV101-1), `code` (SV101-2), `modifiers: list[str]` (SV101-3..6)
|
||||||
|
- `ValidationIssue`: `rule: str`, `severity: Literal["error","warning"]`, `message: str`, `segment_index?: int`
|
||||||
|
- `ValidationReport`: `passed: bool`, `errors: list[ValidationIssue]`, `warnings: list[ValidationIssue]`
|
||||||
|
|
||||||
|
### `PayerConfig` (`payer.py`)
|
||||||
|
|
||||||
|
```python
|
||||||
|
class PayerConfig(BaseModel):
|
||||||
|
name: str
|
||||||
|
sbr09_claim_filing: str # "MC" for CO Medicaid
|
||||||
|
allowed_claim_frequencies: set[int] # {1, 7, 8} for CO
|
||||||
|
require_ref_g1_for_adjustments: bool
|
||||||
|
allowed_bht06: set[str] # {"CH"} for FFS CO; {"RP"} for encounter-only
|
||||||
|
payer_id: str # "SKCO0"
|
||||||
|
payer_name: str # "COHCPF"
|
||||||
|
no_patient_loop: bool # True for CO
|
||||||
|
encounter_claim_in_same_batch: bool # False for CO (CH and RP must be separate)
|
||||||
|
```
|
||||||
|
|
||||||
|
Built-ins:
|
||||||
|
- `PayerConfig.co_medicaid()` — defaults for Colorado Medical Assistance Program, derived from `docs/companionguides/837p.md`.
|
||||||
|
- `PayerConfig.generic_837p()` — relaxed defaults for unknown payers; only structural rules apply.
|
||||||
|
|
||||||
|
## 8. Parser internals (`parse_837.py`)
|
||||||
|
|
||||||
|
A single `parse(text: str, payer_config: PayerConfig) -> ParseResult` function:
|
||||||
|
|
||||||
|
1. Tokenize via `segments.tokenize`.
|
||||||
|
2. Validate the envelope: ISA / IEA, GS / GE, ST / SE. Mismatched control numbers are reported as a single fatal error on the `ParseResult` (the function never raises for envelope problems; the CLI decides what to do with `summary.total_claims == 0`).
|
||||||
|
3. Find the billing-provider HL (`HL*1**20*1`). Read the provider from 2010AA.
|
||||||
|
4. For each subsequent HL (`HL*n*1*22*0`):
|
||||||
|
- Read subscriber from 2010BA.
|
||||||
|
- Read payer from 2010BB.
|
||||||
|
- Scan forward until the next HL or `SE`. For each `CLM`:
|
||||||
|
- Start a new `ClaimOutput`.
|
||||||
|
- Capture `REF*G1` (prior auth / Payer Claim Control Number).
|
||||||
|
- Capture `HI` (diagnoses).
|
||||||
|
- For each subsequent `LX`/`SV1` block, append a `ServiceLine`.
|
||||||
|
- Hand the populated `ClaimOutput` to `validator.validate`; store the report back on the model.
|
||||||
|
5. Build `BatchSummary`.
|
||||||
|
6. Return `ParseResult`.
|
||||||
|
|
||||||
|
Per-claim failures (e.g. malformed `CLM` line) are caught with a `try/except` and recorded as a single error inside that claim's `validation.errors`. The parser never aborts mid-batch.
|
||||||
|
|
||||||
|
## 9. Validation rules (`validator.py`)
|
||||||
|
|
||||||
|
A list of pure functions `Rule = Callable[[ClaimOutput, PayerConfig], Iterable[ValidationIssue]]`. The validator runs all rules and concatenates results.
|
||||||
|
|
||||||
|
| Rule ID | Severity | Description |
|
||||||
|
|---|---|---|
|
||||||
|
| `R001_envelope_balanced` | error | ISA/IEA, GS/GE, ST/SE control numbers match |
|
||||||
|
| `R010_clm01_present` | error | `CLM01` non-empty |
|
||||||
|
| `R011_total_charge_positive` | error | `CLM02 > 0` |
|
||||||
|
| `R020_npi_format` | error | All `XX`-qualified NPIs match `^\d{10}$` |
|
||||||
|
| `R030_frequency_allowed` | error | `CLM05-3 ∈ payer_config.allowed_claim_frequencies` |
|
||||||
|
| `R031_ref_g1_optional` | warning | REF\*G1 is captured but **not enforced** in v1. The companion guide describes G1 as the encounter TCN, but the production data shows G1 used on every fee-for-service claim with values that look like prior-auth numbers. The rule exists as a placeholder for the payer to tighten the policy in a follow-up; for now it surfaces as informational. |
|
||||||
|
| `R040_sbr09_matches_payer` | error | `SBR09 == payer_config.sbr09_claim_filing` |
|
||||||
|
| `R050_diagnosis_present` | error | At least one `HI` diagnosis present |
|
||||||
|
| `R060_service_dates_present` | error | Every `ServiceLine` has a `service_date` |
|
||||||
|
| `R070_charges_sum` | warning | `abs(sum(SV1.charge) - CLM02) < 0.01` |
|
||||||
|
| `R080_no_patient_loop` | error | When `payer_config.no_patient_loop`, fail if a Loop 2000C (patient) HL appears |
|
||||||
|
| `R090_bht06_allowed` | error | `BHT06 ∈ payer_config.allowed_bht06` |
|
||||||
|
| `R100_payer_id_matches` | warning | `Payer.id == payer_config.payer_id` (mismatch is non-fatal; informational) |
|
||||||
|
|
||||||
|
`--strict` upgrades warnings to errors at the CLI level (post-validation, not inside the rules).
|
||||||
|
|
||||||
|
## 10. Output files
|
||||||
|
|
||||||
|
The CLI writes to `--output-dir`:
|
||||||
|
|
||||||
|
```
|
||||||
|
<output-dir>/
|
||||||
|
├── claim-991102977-20260611.json # one per CLM, named <ISA13>-<BHT04 YYYYMMDD>
|
||||||
|
├── claim-991102977-20260611.json
|
||||||
|
├── ...
|
||||||
|
└── summary.json
|
||||||
|
```
|
||||||
|
|
||||||
|
`summary.json` shape:
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"input_file": "/abs/path/to/input.txt",
|
||||||
|
"control_number": "991102977",
|
||||||
|
"transaction_date": "2026-06-11",
|
||||||
|
"total_claims": 12,
|
||||||
|
"passed": 11,
|
||||||
|
"failed": 1,
|
||||||
|
"failed_claim_ids": ["t991102977o1c3d"],
|
||||||
|
"issues_by_rule": {"R070_charges_sum": 1}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
If two claims in the same file share `ISA13` and `BHT04` (unusual but legal), the second file gets a numeric suffix: `claim-991102977-20260611-2.json`. Filenames are sanitized; control characters are stripped.
|
||||||
|
|
||||||
|
## 11. CLI
|
||||||
|
|
||||||
|
`cli.py` exposes a single command group:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python -m cyclone.cli parse-837 <input> \
|
||||||
|
--output-dir <dir> # required
|
||||||
|
[--payer co_medicaid] # default: co_medicaid
|
||||||
|
[--strict] # warnings -> errors
|
||||||
|
[--max-retries N] # default 0; re-runs validation, does not patch
|
||||||
|
[--include-raw-segments / --no-raw-segments] # default: include
|
||||||
|
[--log-level INFO]
|
||||||
|
```
|
||||||
|
|
||||||
|
Exit codes:
|
||||||
|
- `0` — every claim parsed (failures are written to JSON; this is by design)
|
||||||
|
- `2` — envelope-level failure (e.g. ISA not found, no `CLM` segments in the file)
|
||||||
|
- `1` — unexpected exception (bug); message printed to stderr
|
||||||
|
|
||||||
|
The CLI prints a one-line summary table to stdout:
|
||||||
|
```
|
||||||
|
parsed=12 passed=11 failed=1 output=./claims
|
||||||
|
```
|
||||||
|
|
||||||
|
## 12. Testing
|
||||||
|
|
||||||
|
| Test file | Coverage |
|
||||||
|
|---|---|
|
||||||
|
| `test_segments.py` | ISA delimiter detection, edge cases (long ISA IDs, missing terminator), `EDIStream` iteration, raises on malformed input |
|
||||||
|
| `test_parse_837.py` | Parses `fixtures/minimal_837p.txt` and `fixtures/co_medicaid_837p.txt`; asserts claim count, model field values, raw segments preserved, errors collected (not raised) for known-bad input |
|
||||||
|
| `test_validator.py` | Each `R0xx` rule, both pass and fail cases; `PayerConfig.co_medicaid()` vs `generic_837p()` |
|
||||||
|
| `test_cli.py` | `CliRunner` invokes `parse-837` on the synthetic fixture and asserts files written. An additional test reads one file from `docs/prodfiles/837p-from-axiscare/` if any are present and is `pytest.skip`-ed otherwise. |
|
||||||
|
|
||||||
|
Fixtures:
|
||||||
|
- `minimal_837p.txt` — hand-written, smallest valid 837P (one subscriber, one claim, one service line). ≤ 30 segments.
|
||||||
|
- `co_medicaid_837p.txt` — generated from a sanitized subset of a production file. Names are randomized, member IDs are stubbed, addresses use the same ZIP code with a generic street.
|
||||||
|
|
||||||
|
## 13. Out of scope / future work
|
||||||
|
|
||||||
|
- **FastAPI server** — separate spec. The `ClaimOutput` schema is the contract; the API will serialize the same models.
|
||||||
|
- **835 ERA parser** — separate spec. Will share `segments.py`, `validator.py` (with a different rule list), and parts of `models.py`.
|
||||||
|
- **Database persistence** — separate spec.
|
||||||
|
- **Auto-fix / claim repair** — a `--max-retries` flag exists in v1 to test the wiring, but no fixers ship.
|
||||||
|
- **Code-set lookups** — CPT/HCPCS/ICD-10 vocab validation deferred; out of scope for v1.
|
||||||
|
|
||||||
|
## 14. Open assumptions (will be verified during implementation)
|
||||||
|
|
||||||
|
1. Each input file contains exactly one ISA / GS / ST envelope. **Likely true** for AxisCare exports but should be asserted.
|
||||||
|
2. Subscriber = patient (no 2000C loop) is enforced for CO Medicaid via the companion guide and the production file; this is captured by `R080_no_patient_loop`.
|
||||||
|
3. The production file uses the standard `*` element and `~` segment delimiters. **Confirmed** in the first 200 lines of `tp11525703-837P-20260618153358831-1of1.txt`.
|
||||||
|
4. `REF*6R` and `REF*G1` are the only REF qualifiers we care about in v1. Other qualifiers (D9 for ICN, etc.) are captured as opaque strings in `raw_segments` for now.
|
||||||
|
5. **REF*G1 semantic ambiguity** — the CO companion guide describes G1 as the encounter TCN, but the production data shows G1 used on every FFS claim with small integer values that look like prior-auth numbers. v1 captures the value into `claim.prior_auth` and surfaces its presence as a warning; the rule is intentionally lenient and documented as a follow-up.
|
||||||
Reference in New Issue
Block a user