diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 0000000..25cb22f --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,56 @@ +# Changelog + +## 1.1.0 — 2026-10-06 + +### Fixes + +- Restore the published tool-reference path in the Docker image. The directory + alias `/opt/healthverse-task-assets` points to `/opt/chi-bench-task-assets`, so + existing task instructions see the selected per-task tool reference. +- Allow a completed payer intake with failed member eligibility to end in the + existing `denied` state with reason `ELIG_TERMED`. The agent generates and + delivers a denial letter. Approval and clinical routing are blocked after a + failed eligibility check. +- Add coverage status and known coverage dates to the existing request-form + template. The data repair restores pa_t035's termination date from the original + synthesis facts and replaces its unreachable `requirement_checked` expectation + with denial and a denial letter. No separate evidence document is added. +- Provide scripts to repair downloaded pa_t035 task copies and regenerate the + 25 saved payer request forms and their marathon copies before rebuilding. + Remove rendering timestamps from regenerated PDFs so repeat runs preserve + their hashes when content is unchanged. +- Align the local competition Rules and Overview references with the published + Kaggle pages. + +### Upgrade + +After obtaining handbook access and downloading the task data: + +```bash +uv sync --extra dev +uv run playwright install chromium +uv run python scripts/repair_pa_t035.py --data-dir data +uv run python scripts/regenerate_payer_request_forms.py --data-dir data +uv run cb data verify +uv run cb docker build +``` + +On Linux, use `uv run playwright install --with-deps chromium` if browser system +libraries are missing. + +This is runtime version **1.1.0**. The downloaded dataset remains pinned to +**chi-bench-v1.0.0**; the scripts apply local corrections. This change does not +publish a new Hugging Face dataset revision. Preserve earlier trial results and +record the runtime commit and image digest for new runs. + +### P2P clarification + +The P2P workflow is unchanged. The original `MdCaseDecision` records the referral +to P2P; the completed P2P request supplies the outcome, and +`determination.finalize` records the final determination. No second physician +submission is required. + +Among the six public payer tasks requiring P2P, t026 and t032 expect the referral +to have `signed_off=false`. Tasks t016, t019, t031, and t036 exclude MD review +from their scored stages. A direct-decision task may require signed physician +review even if an agent chooses P2P instead. diff --git a/CLAUDE.md b/CLAUDE.md index dbf06a1..afa6138 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -151,6 +151,20 @@ Configs: ## Things to remember +- **Write Kaggle FAQs as direct participant answers in ASD-STE100 style, using the known + CHI-Bench dates. Omit organizer-reply citations, review notes, repeated requirements, and + verbatim restatements of user directions. Keep contact links in FAQs and only necessary + source links in announcements. + Assess existing rules against the intended workflow before proposing them.** +- **Use the CHI-Bench Cup sequence: competition deadline, private evaluation, result release, + then the IEEE-format camera-ready report. Do not add a paper-acceptance stage.** **Why:** + CHI-Bench evaluates competition systems; the report follows competition results. Use its + own schedule and participation policy; do not import another Cup's instructions. Use + organizer-specified submission routes; do not infer a requirement from a portal listing. +- **Preserve published task paths with image compatibility aliases when fixing runtime renames.** + **Why:** Rewriting downloaded `instruction.md` files changes task checksums. For tool references, + alias `/opt/healthverse-task-assets` to `/opt/chi-bench-task-assets` as a directory so per-task + selection and tool-name rewriting remain visible through the published path. - **For simple, explicitly scoped maintenance PRs, work directly on a `codex/` branch in the current clean checkout unless the user asks for a worktree or design spec.** **Why:** The user prefers direct execution over extra process for low-risk changes such as a model-default update. diff --git a/README.md b/README.md index d966964..e43c368 100644 --- a/README.md +++ b/README.md @@ -89,10 +89,23 @@ uv run huggingface-cli download actava/managed-care-operations-handbook \ **5. Build the Docker image** (~5 min, one-time). +Apply the `pa_t035` data correction before building. It restores the coverage +termination date in the existing request form and requires a payer denial letter. +The script changes only local data; it does not publish a dataset revision. + ```bash +uv run playwright install chromium +uv run python scripts/repair_pa_t035.py +uv run python scripts/regenerate_payer_request_forms.py uv run cb docker build ``` +The repair scripts render PDFs on the host. On Linux, use +`uv run playwright install --with-deps chromium` if browser system libraries are missing. +Runtime version `1.1.0` and dataset revision `chi-bench-v1.0.0` are separate; +keep the dataset pin at the revision you downloaded and record the runtime commit +and image digest with your results. See [the v1.1.0 changes](CHANGELOG.md). + > [!NOTE] > `cb` is the short alias for `chi-bench`; both commands resolve to the same CLI. Pick whichever you prefer (the rest of this README uses `cb`). If your shell already aliases `cb` to something else (e.g. a clipboard tool), use `chi-bench`. For the full command surface and flag reference, read [`docs/cli.md`](docs/cli.md). diff --git a/configs/submission_example.yaml b/configs/submission_example.yaml index 7ad40a4..5fc6de9 100644 --- a/configs/submission_example.yaml +++ b/configs/submission_example.yaml @@ -5,9 +5,11 @@ # cb submission validate -f .yaml # cb submission run -f .yaml # cb submission status -f .yaml -# cb submission package -f .yaml +# cb submission prepare -f .yaml # -# The packet (/.zip) is what you upload to the leaderboard. +# The prepared packet is under /packet/YYYY-MM-DD-/. +# Follow the Kaggle organizer instructions for private-final package delivery; +# the separate evidence leaderboard accepts the same packet through a GitHub PR. schema: chi-bench/submission/v1 diff --git a/docker/Dockerfile b/docker/Dockerfile index a1d88f0..282386b 100644 --- a/docker/Dockerfile +++ b/docker/Dockerfile @@ -99,6 +99,9 @@ COPY data/marathon /opt/chi-bench/tasks/marathon COPY data/skills/managed-care-operations-handbook /workspace/skills/managed-care-operations-handbook RUN mkdir -p /opt/chi-bench-task-assets COPY data/care_management/shared/tool_reference.md /opt/chi-bench-task-assets/tool_reference.md +# Published task instructions still use the healthverse asset path. Alias the +# directory so per-task selection and tool-name rewriting remain visible there. +RUN ln -s /opt/chi-bench-task-assets /opt/healthverse-task-assets COPY docker/entrypoint.sh /usr/local/bin/entrypoint.sh RUN chmod +x /usr/local/bin/entrypoint.sh diff --git a/docs/architecture.md b/docs/architecture.md index 309e94d..b9060f6 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -48,6 +48,11 @@ 5. After the agent stops (success / timeout / abstain), Harbor invokes the verifier in the same container. 6. Verifier writes `verifier/scorecard.json` and `verifier/verdicts.json`; Harbor produces `result.json`. +When the payer confirms failed member eligibility, processing stops at intake. +`determination.finalize` can use the completed intake record to set the existing +case status `denied`, with reason `ELIG_TERMED`. The agent then generates and +delivers the denial letter. Clinical review is not required for this result. + ## Why the LLM judge needs Anthropic credits The verifier always uses `claude-opus-4-7` (configurable but paper-faithful default). diff --git a/docs/cli.md b/docs/cli.md index 246f41c..1f8b8a6 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -250,6 +250,29 @@ echo "$REV" > data/.chi-bench-version `cb data download` exists so the pin file is written in one step; the README documents the raw HF CLI invocation as the canonical path. +For the `pa_t035` correction, run the following after downloading data and before +building the Docker image: + +```bash +uv run playwright install chromium +uv run python scripts/repair_pa_t035.py --data-dir data +uv run python scripts/regenerate_payer_request_forms.py --data-dir data +``` + +These scripts render PDFs on the host, so Playwright needs its Chromium browser +there. On Linux, use `uv run playwright install --with-deps chromium` when system +libraries are missing. + +This restores the coverage termination date in the existing request form and +changes the expected payer result to `denied`, with a denial letter. It updates +the task's expectation copies and its marathon copy. It does not publish data +or change the dataset version pin. + +The second script updates all saved payer request forms with the same template, +including their marathon copies. Coverage status appears on each form. Coverage +dates appear only when the shared world provides them. Case facts and scoring +expectations are unchanged by this regeneration. + --- ## `cb submission` diff --git a/docs/competition-rules-draft.md b/docs/competition-rules-draft.md new file mode 100644 index 0000000..c454b3f --- /dev/null +++ b/docs/competition-rules-draft.md @@ -0,0 +1,305 @@ +# χ-Bench Healthcare AI Agents Challenge + +## IEEE Big Data Cup 2026 — Competition Rules + +This local reference was aligned with the [published Kaggle Rules](https://www.kaggle.com/competitions/chi-bench/rules) and [Overview](https://www.kaggle.com/competitions/chi-bench/overview) on October 6, 2026. Kaggle hosts public submissions; organizer-run private evaluation determines final standings and prizes. The live Rules tab, including Kaggle's Foundational Competition Rules, remains authoritative. + +### 1. Competition + +The χ-Bench Healthcare AI Agents Challenge is part of the IEEE Big Data Cup 2026 and is sponsored and organized by actAVA.ai. + +The Competition evaluates autonomous AI agents on long-horizon U.S. healthcare-administration workflows. It is hosted through [Kaggle](https://www.kaggle.com/competitions/chi-bench). + +Participation is also subject to the [Kaggle Terms of Service](https://www.kaggle.com/terms) and the Foundational Competition Rules on the [Rules tab](https://www.kaggle.com/competitions/chi-bench/rules). Kaggle states that its foundational rules take precedence over conflicting competition-specific provisions. + +### 2. Challenge scope + +The Competition covers three χ-Bench tracks: + +1. Provider Prior Authorization: verify coverage, assemble clinical evidence, submit an authorization request, and handle requests for information, peer-to-peer review, and appeals. +2. Payer Utilization Management: intake requests, apply plan medical policy, conduct nurse and physician review, and issue coverage determinations. +3. Care Management: review patient charts, conduct outreach and assessments, and create NANDA-I/NOC/NIC-aligned care plans. + +The public development benchmark contains 75 tasks: 25 tasks in each track. The private final benchmark is expected to contain approximately 75 additional clinician-validated tasks that are not released during the Competition. + +The combined three-track result on the private set determines the overall winners. Per-track results will also be reported. + +### 3. Important dates + +All deadlines are at 11:59 PM UTC unless otherwise announced. + +| Event | Date | +|---|---| +| Cup data-release program began | June 1, 2026 | +| Kaggle competition opens | August 15, 2026 | +| Entry deadline | November 7, 2026 | +| Team-merger deadline | November 7, 2026 | +| Final Kaggle submission deadline | November 15, 2026 | +| Final results announced | November 23, 2026 | +| Finalist materials due | December 7, 2026 | +| IEEE Big Data 2026 | December 14–17, 2026 | +| Conference location | Phoenix, Arizona, USA | + +The organizers may adjust dates when necessary. Material changes will be announced on Kaggle. + +### 4. Eligibility and teams + +The Competition is open to individuals and teams worldwide, subject to applicable laws, sanctions, export controls, employer policies, and Kaggle eligibility requirements. + +- Each participant may use only one Kaggle account. +- A participant may belong to only one team. +- The maximum team size is five people. +- Team mergers must be completed before the team-merger deadline. +- Each team member must accept these Rules. +- Organizers, competition administrators, and anyone with advance access to private tasks, expectations, verifier assets, or solutions may participate for demonstration purposes but are not eligible for rankings or prizes. +- Participants under the age of majority in their jurisdiction must obtain any required parental or guardian consent. + +Private sharing of competition code or private competition data between separate teams is prohibited. Public sharing available equally to all participants is permitted, subject to the handbook restrictions in Section 5. + +### 5. Data and Managed-Care Operations Handbook + +The χ-Bench code and public task fixtures are available through the official repository and dataset release under their stated licenses. + +The Managed-Care Operations Handbook is a separate, gated artifact. Access is manually reviewed and governed by the actAVA clinical-collaborator license. It is restricted to non-commercial research use and may not be redistributed. The complete access conditions are published on the [official handbook page](https://huggingface.co/datasets/actava/managed-care-operations-handbook). + +The following requirements are mandatory: + +- Participants must request and receive handbook access before running χ-Bench. +- Every team member who accesses the handbook must be separately authorized. +- Teams may not share the handbook with other people or teams. +- The handbook, substantial excerpts, or derived confidential content must not be uploaded to Kaggle, GitHub, public model repositories, public notebooks, or other public services without prior written permission from actAVA. +- Hugging Face tokens, model API keys, and other credentials must never be included in submissions, notebooks, logs, or source repositories. +- Public reports should refer to handbook section identifiers or summarize methods without reproducing protected material. +- Private test tasks and their associated data must remain confidential. + +χ-Bench uses simulated patients and contains no real patient data or protected health information. Participants must not introduce real patient information into their systems or submissions. + +### 6. Permitted systems + +Participants may use any lawful model, agent harness, orchestration framework, prompting method, hardware, cloud provider, local environment, or external API. + +There is no required execution environment and no restriction to the agent harnesses shipped with χ-Bench. Proprietary models and custom agents are permitted. + +External models, tools, and datasets must: + +- be lawfully accessible to the team; +- comply with their applicable licenses and terms; +- be disclosed in the qualifying report; and +- not contain leaked private χ-Bench tasks, expectations, or solutions. + +Participants are responsible for ensuring that any third-party API or service they use is compatible with the handbook’s access and confidentiality terms. + +### 7. Automation and human involvement + +Human work is permitted when developing, configuring, testing, and debugging an agent. + +Once an evaluated trial begins: + +- The task must be completed by the submitted automated system. +- Humans may not select actions, supply task-specific answers, repair artifacts, or intervene based on intermediate task state. +- Teams may not manually change verifier outputs, rewards, scorecards, trajectories, or result files. +- The same frozen system configuration must be used for all private-final tasks. + +Systems may evolve during the public development period. The version selected for final judging must be frozen at the final submission deadline. + +### 8. Prohibited conduct + +The following may result in submission invalidation or team disqualification: + +- Accessing private tasks, hidden expectations, verifier solutions, or organizer-only materials. +- Making hidden verifier files available to the agent during execution. +- Hard-coding private task answers or task-specific final outputs. +- Submitting a reward that was not produced by the official verifier. +- Submitting an all-passing result file when the corresponding trials did not pass. +- Fabricating, deleting, or materially altering execution evidence. +- Using multiple Kaggle accounts or proxy accounts. +- Sharing private competition code or private data between teams. +- Attempting to interfere with Kaggle, χ-Bench infrastructure, other participants, or the evaluation process. +- Introducing real patient data or other unlawfully obtained information. +- Failing to disclose material external models, tools, or data. + +General workflow logic, reusable healthcare knowledge, and improvements derived from public development tasks are permitted. The prohibition concerns answer leakage, falsification, and task-specific circumvention. + +### 9. Public Kaggle evaluation + +The public competition uses the 75 core tasks from the announced χ-Bench dataset version, initially `chi-bench-v1.0.0`. + +Each task must be run once for the submitted result. Its value is taken directly from `verifier/reward.json`: + +- `1` means the trial passed every required non-N/A check. +- `0` means one or more required checks failed. + +Fractional diagnostic scores are not accepted as competition rewards. + +The Kaggle submission file must contain exactly one row for every required task: + +```csv +task_id,reward +cm_afib_moderate_anxious_001,1 +pa_t008_t008_o002_p01_mdreview_payer,0 +``` + +Requirements: + +- `task_id` must match the official task identifier. +- `reward` must be either `0` or `1`. +- Every required task must appear exactly once. +- No additional rows or duplicate task identifiers are permitted. + +Kaggle Accuracy will be configured against a target value of `1` for every row. With binary rewards, this equals: + +\[ +\mathrm{Public\ Pass@1} +=\frac{\text{number of tasks with reward 1}}{75}. +\] + +Because participants run the benchmark themselves, the Kaggle score is a provisional, evidence-backed claim. It does not by itself establish prize eligibility. + +Teams may make up to five Kaggle submissions per day and must select one final submission before the deadline. + +### 10. Official verifier + +The official verifier combines: + +1. deterministic contract checks covering terminal state, routing, structured payloads, and required artifacts; and +2. rubric-based judging of policy grounding and clinical or operational reasoning. + +Unless announced otherwise before the final-system freeze: + +- the LLM judge will be pinned to `claude-opus-4-7`; +- the patient simulator will be pinned to `claude-sonnet-5`; +- each rubric will receive three independent judge votes; +- the majority result determines the rubric verdict; and +- a task receives reward `1` only when all required non-N/A checks pass. + +The primary score is strict binary pass@1. Fractional reward may be retained for diagnostics but will not affect ranking. + +An agent timeout, API failure, malformed output, or participant-infrastructure failure normally produces a failed trial. A trial may be rerun only when the organizers determine that an organizer-controlled infrastructure failure prevented valid execution. Reruns will not be granted because an agent made an incorrect decision. + +### 11. Finalist audit and private evaluation + +Every team that submits a complete runnable system, evidence packet, source code, qualifying report, and valid Kaggle entry by the deadline receives a private score. Organizer-run private results determine official standings. + +For eligibility, each team must privately provide: + +- the official χ-Bench audit packet produced through `cb submission prepare`; +- source code for the agent and orchestration logic; +- prompts, system instructions, tools, and configuration; +- dependency manifests and lock files; +- the exact model and API endpoint identifiers; +- a container image, image digest, or complete reproduction instructions; +- cost and token-usage records where available; +- a qualifying technical report using the IEEE conference template, no more than five pages excluding references; and +- reasonable assistance enabling organizer re-verification. + +“Source code” refers to the participant-created agent and orchestration components. It does not require disclosure of proprietary third-party model weights or source code that the participant does not own. + +The organizer's [submission clarification](https://www.kaggle.com/competitions/chi-bench/discussion/745236) +specifies delivery to `research@actava.ai`, with `[Kaggle]` in the email subject +and team information included. The complete package is due November 15; +December 7 is for prize winners' conference materials. Forum participation is +not required. Credentials must not be placed in Kaggle submissions or repositories. + +The organizer's [API-cost clarification](https://www.kaggle.com/competitions/chi-bench/discussion/741283) +states that participants fund development, with no sponsored token credits, and +that organizers pay all API costs during private evaluation. + +Failure to provide complete materials, or inability to execute the submitted system within the verification period, may result in removal from consideration. + +The organizers will evaluate all eligible systems on the private held-out task set. Private evaluation results supersede the provisional Kaggle public leaderboard and determine official standings and prizes. + +Submission to the separate [χ-Bench evidence leaderboard](https://github.com/actava-ai/leaderboard) is encouraged but is not a substitute for the required Kaggle entry. + +### 12. Final scoring and tie-breaking + +For each eligible team, the organizers will conduct three independent attempts per private task using the frozen submitted system. + +The primary score uses only the first attempt: + +\[ +\mathrm{Pass@1} +=\frac{1}{N}\sum_{i=1}^{N}r_{i,1}, +\] + +where \(r_{i,1}\in\{0,1\}\) and \(N\) is the number of private tasks. + +Ties in overall pass@1 will be resolved in this order: + +1. Higher pass³: + +\[ +\mathrm{Pass^3} +=\frac{1}{N}\sum_{i=1}^{N} +\mathbf{1}(r_{i,1}=r_{i,2}=r_{i,3}=1). +\] + +2. Lower mean agent inference cost, calculated from recorded usage and the organizer’s frozen price table. +3. Fewer mean agent tool calls per task. +4. Earlier valid final Kaggle submission. + +If inference costs are unavailable or not meaningfully comparable for either tied system, the cost tie-breaker will be skipped and tool-call count will be used next. + +Unrounded scores will be used for ranking. + +### 13. Prizes + +The stated prize pool is USD 3,000: + +- Gold — first overall: USD 1,500 +- Silver — second overall: USD 1,000 +- Bronze — third overall: USD 500 +- Reliability Award — highest private pass³ score + +The Reliability Award is a separate recognition and may be awarded to an overall medalist. No additional cash amount beyond the stated USD 3,000 pool is attached to it unless separately announced. + +Prize eligibility requires successful private verification, submission of source code and the qualifying report, and compliance with these Rules. + +Teams are responsible for taxes, reporting obligations, and any documentation required to receive a prize. Unless the team unanimously supplies different written instructions, a monetary team prize will be divided equally among eligible team members. + +### 14. Reports, presentation, and publication + +Finalists may be invited to: + +- present their systems at IEEE Big Data 2026 in Phoenix; +- contribute to a post-competition community results report; and +- extend their work into a conference or proceedings submission. + +At least one team representative should be available to present in person or through an organizer-approved remote format. Travel, accommodation, and conference registration are not guaranteed unless separately announced. + +Participants retain ownership of their original code and methods. By entering, participants grant the organizers permission to evaluate their submission and publish the team name, system description, scores, rankings, and qualifying report. Private source code will not be publicly released without the owner’s permission, except where a separately accepted award or publication agreement requires it. + +Authorship on any community report will follow normal scholarly contribution and approval requirements and is not automatically granted by placement alone. + +### 15. Organizer authority + +The organizers may: + +- inspect or re-judge submitted evidence; +- require clarification or additional reproduction materials; +- invalidate suspicious or unsupported submissions; +- correct scoring or infrastructure errors; +- disqualify participants who violate these Rules; +- withhold an award if no eligible submission satisfies the verification requirements; and +- amend these Rules when necessary for fairness, security, legal compliance, or platform requirements. + +Material changes will be announced publicly. Whenever practical, changes affecting evaluation will be made before the final-system freeze and applied consistently to all teams. + +Organizer decisions concerning eligibility, verification, scoring, and rule enforcement are final, subject to Kaggle requirements and applicable law. + +### 16. Medical disclaimer and contact + +χ-Bench is a research benchmark using simulated healthcare workflows. It is not a medical device, does not provide medical advice, and must not be used to make decisions about real patients. + +Questions should be posted in the Kaggle discussion forum or sent to [research@actava.ai](mailto:research@actava.ai). + +--- + +## Source notes + +The [Kaggle Rules](https://www.kaggle.com/competitions/chi-bench/rules) supply the competition policy; the [Overview](https://www.kaggle.com/competitions/chi-bench/overview) also specifies the simulator configuration. See the [local overview](kaggle-challenge-overview.md) for setup and submission commands. + +The Kaggle header says “IEEE Big Data Cup 2027,” while its rules, overview, and +dates describe 2026. The submission discussion clarifies the November 15 +final-package deadline and December 7 conference-material deadline. Its +follow-up about whether email delivery also requires a Kaggle CSV or Writeup +remains unanswered; Section 9 above retains the published CSV requirements. diff --git a/docs/judge.md b/docs/judge.md index a153500..82b60ea 100644 --- a/docs/judge.md +++ b/docs/judge.md @@ -14,6 +14,12 @@ non-N/A rubric verdicts in every domain. The scorecard also records `fractional_reward = passed_checks / total_checks` for diagnostics, including Care Management; fractional reward is not the published leaderboard axis. +A denial for failed member eligibility can use the completed intake record as +its source. The stage verifier checks that the record belongs to the same case, +that `decision_member_eligible` is false, and that the final decision is `denied`. +The task must also require the denial letter. A clinical review record is not +required for this intake result. + ## Why a single judge model? All paper numbers were collected with `claude-opus-4-7` as the judge. diff --git a/docs/kaggle-challenge-overview.md b/docs/kaggle-challenge-overview.md new file mode 100644 index 0000000..047d92d --- /dev/null +++ b/docs/kaggle-challenge-overview.md @@ -0,0 +1,304 @@ +# χ-Bench Healthcare AI Agents Challenge + +**Build an autonomous agent for prior authorization, utilization management, and care management in χ-World.** + +Aligned with the [published Kaggle Overview](https://www.kaggle.com/competitions/chi-bench/overview) and [Rules](https://www.kaggle.com/competitions/chi-bench/rules) on October 6, 2026. The live Rules tab, including Kaggle's Foundational Competition Rules, remains authoritative; the [local rules reference](competition-rules-draft.md) records the competition-specific requirements. + +## Description + +Can an AI agent complete a complex healthcare-administration workflow—not merely answer a question, but gather evidence, apply policy, use operational tools, communicate with simulated people, and drive a case to the correct outcome? + +χ-Bench evaluates long-horizon agents inside χ-World, a high-fidelity simulator of U.S. healthcare operations. Agents work across healthcare applications exposed through MCP and REST tools while grounding their decisions in the 1,279-document Managed-Care Operations Handbook. + +The benchmark contains only simulated patients and synthetic operational data. It contains no real patient data or protected health information and must not be used to make decisions about real patients. + +| | Competition at a glance | +|---|---| +| Public development set | 75 tasks, 25 per track | +| Private evaluation set | Approximately 75 unreleased clinician-validated tasks | +| Primary metric | Strict binary pass@1 | +| Reliability metric | pass³ across three independent attempts | +| Prize pool | USD 3,000 plus a Reliability Award | +| Conference | IEEE Big Data 2026, Phoenix, December 14–17, 2026 | + +### How the competition works + +1. Join the Kaggle competition and accept the Rules. +2. Request and receive access to the gated Managed-Care Operations Handbook **before running χ-Bench**. +3. Develop and evaluate your agent on the 75 public tasks. +4. Submit the public task results through Kaggle and prepare the required runnable final package. +5. The organizers run every complete, prize-eligible final entry on the unreleased private tasks. +6. Organizer-run private scores determine the final rankings and prizes. + +The Kaggle public leaderboard is a provisional development leaderboard because its task rewards are generated in participant-controlled environments. Every team that submits a complete runnable system, evidence packet, source code, and qualifying report by the deadline receives a private score. + +## Start here + +### Essential links + +- [Official χ-Bench repository](https://github.com/actava-ai/chi-bench) +- [Public χ-Bench task dataset](https://huggingface.co/datasets/actava/chi-bench) +- [Request access to the Managed-Care Operations Handbook](https://huggingface.co/datasets/actava/managed-care-operations-handbook) +- [Quickstart](https://www.actava.ai/benchmarks/docs/quickstart) +- [Running experiments and submissions](https://www.actava.ai/benchmarks/docs/run) +- [Browse all 75 public tasks](https://www.actava.ai/benchmarks/tasks) +- [Live χ-Bench leaderboard](https://www.actava.ai/benchmarks/leaderboards) +- [Evidence and submission-packet repository](https://github.com/actava-ai/leaderboard) +- [χ-Bench paper](https://arxiv.org/abs/2605.16679) +- [IEEE Big Data Cup 2026](https://bigdataieee.org/BigData2026/cup/) +- [Kaggle competition](https://www.kaggle.com/competitions/chi-bench) + +### Prerequisites + +- Python 3.12 or newer +- Docker +- [`uv`](https://docs.astral.sh/uv/) +- An approved Hugging Face token for the handbook +- `ANTHROPIC_API_KEY` for the official workspace judge +- The provider key required by your chosen agent model + +Any model, agent harness, prompting strategy, hardware, cloud provider, or local environment may be used, subject to the Rules and applicable licenses. You are not limited to the harnesses included in the repository. + +### Minimal local setup + +Clone and install χ-Bench: + +```bash +git clone https://github.com/actava-ai/chi-bench +cd chi-bench +uv sync --extra dev +``` + +Download the pinned public task release: + +```bash +uv run huggingface-cli login + +REV=chi-bench-v1.0.0 +uv run huggingface-cli download actava/chi-bench \ + --repo-type dataset \ + --revision "$REV" \ + --local-dir data/ +echo "$REV" > data/.chi-bench-version +``` + +Before continuing, request and obtain approval for the [Managed-Care Operations Handbook](https://huggingface.co/datasets/actava/managed-care-operations-handbook). Once approved, download it: + +```bash +uv run huggingface-cli download actava/managed-care-operations-handbook \ + --repo-type dataset \ + --local-dir data/skills/ +``` + +Configure credentials locally. Never upload `.env`, API keys, or Hugging Face tokens to Kaggle or a source repository: + +```bash +cp .env.example .env +``` + +Build and verify the local benchmark: + +```bash +uv run cb docker build +uv run cb data verify +``` + +Run a smoke-test task by following the [Quickstart](https://www.actava.ai/benchmarks/docs/quickstart). For Harbor and Modal alternatives, see the [run guide](https://www.actava.ai/benchmarks/docs/run). + +### Run a complete public submission + +Start from the [submission configuration template](https://github.com/actava-ai/chi-bench/blob/main/configs/submission_example.yaml): + +```bash +mkdir -p configs/submissions +cp configs/submission_example.yaml configs/submissions/my-team.yaml + +uv run cb submission validate -f configs/submissions/my-team.yaml +uv run cb submission run -f configs/submissions/my-team.yaml +uv run cb submission status -f configs/submissions/my-team.yaml +uv run cb submission prepare -f configs/submissions/my-team.yaml +``` + +Edit `my-team.yaml` before running it. Set your team, contact, agent, model, environment, and dataset version. The prepared evidence packet contains the manifest, per-domain results, verifier evidence, provenance, and compressed trajectories required for validation. + +The runtime v1.1.0 repair steps are in [the changelog](../CHANGELOG.md). They apply +local corrections to the downloaded `chi-bench-v1.0.0` dataset before rebuilding. + +### API costs and private package delivery + +The organizer's [API-cost clarification](https://www.kaggle.com/competitions/chi-bench/discussion/741283) +states that development has no sponsored credits and that organizers pay all API +costs during private evaluation. Local development uses the participant's keys +for the chosen agent, Anthropic judge, and simulators. + +The [submission clarification](https://www.kaggle.com/competitions/chi-bench/discussion/745236) +specifies email delivery to `research@actava.ai`, with `[Kaggle]` in the subject +and team information included. November 15 is the final-package deadline; +December 7 is for winners' conference materials. Discussion participation is +optional. The thread's follow-up about an additional Kaggle CSV or Writeup +requirement remains unanswered; the published Rules still specify a CSV. + +## Evaluation + +### Task reward + +Each task receives one strict binary reward: + +- `1` only when every required non-N/A verifier check passes. +- `0` when any required check fails. + +Fractional check completion is diagnostic only and does not affect competition ranking. + +The verifier combines deterministic workflow-contract checks with rubric-based review of policy and clinical reasoning. For official private evaluation, the judge is pinned to `claude-opus-4-7`, with three independent votes per rubric. + +The current simulator configuration is: + +| Component | Pinned model and configuration | +|---|---| +| Care Management patient simulator | `claude-sonnet-5`, adaptive thinking disabled, maximum 1,024 output tokens | +| Peer-to-peer physician counterpart | `claude-sonnet-5` | +| Workspace judge | `claude-opus-4-7`, three independent votes per rubric | + +If a provider introduces an immutable replacement snapshot identifier before launch, the organizers will announce and freeze that identifier before official evaluation. The frozen simulator and judge configuration will not change between teams. + +### Public score + +Run each of the 75 public tasks once. The public development score is: + +\[ +\mathrm{Public\ pass@1} +=\frac{\text{public tasks with reward 1}}{75}. +\] + +The Kaggle submission file contains exactly one binary reward for each required public task: + +```csv +task_id,reward +cm_afib_moderate_anxious_001,1 +pa_t008_t008_o002_p01_mdreview_payer,0 +``` + +The competition Data tab will provide `sample_submission.csv` containing the complete required task list. Do not submit a reward of `1` unless the corresponding official verifier output is `1`. + +### Private score and final ranking + +Every complete, runnable, prize-eligible final entry is evaluated for three independent attempts on the unreleased private set. + +Attempt 1 determines private pass@1: + +\[ +\mathrm{Pass@1} +=\frac{1}{N}\sum_{i=1}^{N}r_{i,1}. +\] + +All three attempts determine pass³: + +\[ +\mathrm{Pass^3} +=\frac{1}{N}\sum_{i=1}^{N} +\mathbf{1}(r_{i,1}=r_{i,2}=r_{i,3}=1). +\] + +Final entries are ranked by: + +1. Higher private pass@1. +2. Higher private pass³. +3. Lower mean agent inference cost. +4. Fewer mean agent tool calls. + +The Rules tab adds earlier valid final Kaggle submission as the remaining tie-breaker. When either tied system lacks comparable cost records, the rules skip cost and use tool-call count next. + +Organizer-run private results supersede the provisional Kaggle public score. The final private ranking will be announced on Kaggle and published through the official χ-Bench leaderboard. + +## Submission requirements + +### Public Kaggle submission + +Submit `submission.csv` through Kaggle. It must contain: + +- exactly one row for every required public task; +- an unchanged official `task_id` in every row; +- a `reward` value of either `0` or `1`; and +- no missing, additional, or duplicate task identifiers. + +### Prize-eligible final package + +To receive a private score and remain eligible for prizes, submit the following by the final deadline using the private delivery instructions in the Rules tab: + +- The final Kaggle `submission.csv`. +- The official audit packet produced by `cb submission prepare`. +- The agent and orchestration source code. +- Prompts, system instructions, tool definitions, and configuration. +- Dependency manifests and lock files. +- Exact model and API endpoint identifiers. +- A runnable container image, image digest, or complete reproduction instructions. +- Cost and token-usage records where available. +- A qualifying report using the IEEE conference template, no more than five pages excluding references. + +Source-code disclosure applies to participant-created agent and orchestration components. It does not require disclosure of proprietary third-party model weights or source code the team does not own. Do not include credentials in any submitted artifact. + +Teams may contain up to five people. Each person who accesses the handbook must have their own approved access. + +## Timeline + +All deadlines are 11:59 PM UTC unless otherwise announced. + +| Event | Date | +|---|---| +| Cup data-release program began | June 1, 2026 | +| Kaggle competition opens | August 15, 2026 | +| Entry deadline | November 7, 2026 | +| Team-merger deadline | November 7, 2026 | +| Final Kaggle submission deadline | November 15, 2026 | +| Final results announced | November 23, 2026 | +| Finalist materials due | December 7, 2026 | +| IEEE Big Data 2026 | December 14–17, 2026 | +| Conference location | Phoenix, Arizona, USA | + +These are the dates published on Kaggle. Its submission text also requires a complete package by the final deadline, although the timeline puts finalist materials after the results announcement. The header names 2027 while the rules and overview describe 2026. See the [rules source notes](competition-rules-draft.md#source-notes) and contact the organizers for clarification. + +## Rules, data use, and safety + +- Read and accept the Kaggle Rules before participating. +- Obtain handbook approval before running the benchmark. +- The [public χ-Bench dataset](https://huggingface.co/datasets/actava/chi-bench) is released under its stated Apache-2.0 license. +- The [Managed-Care Operations Handbook](https://huggingface.co/datasets/actava/managed-care-operations-handbook) is separately gated for non-commercial research and may not be redistributed. +- Do not upload the handbook, protected excerpts, credentials, or private evaluation materials to Kaggle or public repositories. +- Evaluated tasks must be completed autonomously. Human task-level intervention and manual modification of verifier results are prohibited. +- Agents may not access hidden expectations, private solutions, or verifier-only files. +- χ-Bench is a research benchmark, not a medical device or source of medical advice. + +The complete Competition Rules are available in the [Kaggle Rules tab](https://www.kaggle.com/competitions/chi-bench/rules). Questions should be posted in the Kaggle Discussion forum or sent to [research@actava.ai](mailto:research@actava.ai). + +## Resources + +- [χ-Bench documentation](https://www.actava.ai/benchmarks/docs) +- [Quickstart](https://www.actava.ai/benchmarks/docs/quickstart) +- [Run and submit](https://www.actava.ai/benchmarks/docs/run) +- [Task explorer](https://www.actava.ai/benchmarks/tasks) +- [Live leaderboard](https://www.actava.ai/benchmarks/leaderboards) +- [Official GitHub repository](https://github.com/actava-ai/chi-bench) +- [Public task dataset](https://huggingface.co/datasets/actava/chi-bench) +- [Managed-Care Operations Handbook access](https://huggingface.co/datasets/actava/managed-care-operations-handbook) +- [Evidence leaderboard repository](https://github.com/actava-ai/leaderboard) +- [Research paper](https://arxiv.org/abs/2605.16679) +- [IEEE Big Data Cup 2026](https://bigdataieee.org/BigData2026/cup/) + +## Citation + +If you use χ-Bench in a publication, cite: + +```bibtex +@misc{chen2026chibenchaiagentsautomate, + title = {CHI-Bench: Can AI Agents Automate End-to-End, + Long-Horizon, Policy-Rich Healthcare Workflows?}, + author = {Haolin Chen and Deon Metelski and Leon Qi and Tao Xia + and Joonyul Lee and Steve Brown and Kevin Riley and + Frank Wang and others}, + year = {2026}, + eprint = {2605.16679}, + archivePrefix = {arXiv}, + primaryClass = {cs.CL}, + url = {https://arxiv.org/abs/2605.16679} +} +``` diff --git a/pyproject.toml b/pyproject.toml index 6649cbe..41ccc95 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -1,6 +1,6 @@ [project] name = "chi-bench" -version = "0.1.0" +version = "1.1.0" description = "chi-Bench: long-horizon, policy-rich healthcare workflows for AI agents." readme = "README.md" requires-python = ">=3.12" diff --git a/scripts/regenerate_payer_request_forms.py b/scripts/regenerate_payer_request_forms.py new file mode 100644 index 0000000..ad746b9 --- /dev/null +++ b/scripts/regenerate_payer_request_forms.py @@ -0,0 +1,89 @@ +"""Regenerate saved payer request forms with the shared form template. + +Case facts come from the shared world. Only existing request PDFs and their +marathon copies are updated; world data and scoring expectations are unchanged. +""" + +import argparse +import json +from pathlib import Path +from tempfile import TemporaryDirectory + +from chi_bench.core.models import Coverage, Patient, PriorAuthCase, ServiceRequest +from chi_bench.core.pdf import generate_request_form_pdf +from chi_bench.core.pdf.maintenance import stable_pdf_bytes + + +def regenerate(data_dir: Path) -> list[Path]: + worlds = {} + changed = [] + with TemporaryDirectory() as temporary: + staged = [] + for manifest_path in sorted( + (data_dir / "prior_auth_um/tasks").glob("*/fixtures/manifest.json") + ): + fixtures = manifest_path.parent + form = fixtures / "request/prior-auth-request-form.pdf" + if not form.is_file(): + continue + manifest = json.loads(manifest_path.read_text()) + world_id = manifest["world_id"] + if world_id not in worlds: + world_path = data_dir / "prior_auth_um/shared/worlds" / f"{world_id}.json" + worlds[world_id] = json.loads(world_path.read_text()) + world = worlds[world_id] + case = PriorAuthCase.model_validate( + next(c for c in world["cases"] if c["id"] == manifest["target_case_id"]) + ) + if case.patient_id != manifest["target_patient_id"]: + raise ValueError(f"Case and patient do not match in {manifest_path}.") + patient = Patient.model_validate( + next(p for p in world["patients"] if p["id"] == case.patient_id) + ) + coverage = Coverage.model_validate( + next(c for c in world["coverages"] if c["patient_id"] == case.patient_id) + ) + service_request = ServiceRequest.model_validate( + next(s for s in world["service_requests"] if s["id"] == case.service_request_id) + ) + provider_name = next( + ( + p["name"] + for p in world["practitioners"] + if p["id"] == service_request.ordering_practitioner_id + ), + service_request.ordering_practitioner_id, + ) + output = Path(temporary) / f"{fixtures.parent.name}.pdf" + generate_request_form_pdf( + output, + case=case, + patient=patient, + coverage=coverage, + service_request=service_request, + provider_name=provider_name, + ) + staged.append((form, output)) + session_form = ( + data_dir + / "marathon/prior_auth_um/fixtures/tasks" + / fixtures.parent.name + / "request/prior-auth-request-form.pdf" + ) + if session_form.is_file(): + staged.append((session_form, output)) + + # Finish rendering every form before replacing any original file. + for destination, output in staged: + content = stable_pdf_bytes(output) + if destination.read_bytes() != content: + destination.write_bytes(content) + changed.append(destination) + return changed + + +if __name__ == "__main__": + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--data-dir", type=Path, default=Path("data")) + changed = regenerate(parser.parse_args().data_dir) + print(f"Updated {len(changed)} request PDFs.") diff --git a/scripts/repair_pa_t035.py b/scripts/repair_pa_t035.py new file mode 100644 index 0000000..072aaa2 --- /dev/null +++ b/scripts/repair_pa_t035.py @@ -0,0 +1,150 @@ +"""Repair pa_t035 coverage evidence and its payer denial expectations. + +Run after downloading data and before building the image. The termination +date comes from the task's original intake ground truth. No document is added. +""" + +import argparse +import json +import re +from pathlib import Path +from tempfile import TemporaryDirectory + +from pypdf import PdfReader +from pypdf.errors import PdfReadError + +from chi_bench.core.models import Coverage, Patient, PriorAuthCase, ServiceRequest +from chi_bench.core.pdf import generate_request_form_pdf +from chi_bench.core.pdf.maintenance import stable_pdf_bytes + +TASK_ID = "pa_t035_t035_o001_p01_intake_payer" +WORLD_NAME = "healthverse-pa-v2-payer.json" + + +def repair(data_dir: Path) -> list[Path]: + task = data_dir / "prior_auth_um/tasks" / TASK_ID + fixtures = task / "fixtures" + session = data_dir / "marathon/prior_auth_um/fixtures/tasks" / TASK_ID + expected = json.loads((fixtures / "expectations.json").read_text()) + intake_gt = next( + entry["expected_fields"] + for entry in expected["stage_ground_truth"] + if entry["model_name"] == "IntakeCase" + ) + if ( + expected["task_kind"] != "payer_not_covered" + or intake_gt["decision_member_eligible"] is not False + ): + raise ValueError("Expected a payer task with failed member eligibility.") + match = re.search(r"terminated effective (\d{4}-\d{2}-\d{2})", intake_gt["coverage_details"]) + if match is None: + raise ValueError("The original ground truth has no coverage termination date.") + termination_date = match.group(1) + updates: dict[Path, bytes] = {} + + def update_json(path, payload): + updates[path] = (json.dumps(payload, indent=2, ensure_ascii=False) + "\n").encode() + + for domain in ("prior_auth_provider", "prior_auth_um"): + path = data_dir / domain / "shared/worlds" / WORLD_NAME + world = json.loads(path.read_text()) + coverage = next( + c for c in world["coverages"] if c["patient_id"] == expected["target_patient_id"] + ) + if coverage["active"] or coverage.get("termination_date") not in (None, termination_date): + raise ValueError(f"Unexpected coverage state in {path}.") + coverage["termination_date"] = termination_date + update_json(path, world) + + # Use the existing models and form template for both copies of the request. + case = PriorAuthCase.model_validate( + next(c for c in world["cases"] if c["id"] == expected["target_case_id"]) + ) + patient = Patient.model_validate( + next(p for p in world["patients"] if p["id"] == case.patient_id) + ) + coverage = Coverage.model_validate(coverage) + service_request = ServiceRequest.model_validate( + next(s for s in world["service_requests"] if s["id"] == case.service_request_id) + ) + if termination_date >= service_request.requested_date.date().isoformat(): + raise ValueError("Coverage termination must precede the requested service date.") + provider_name = next( + ( + p["name"] + for p in world["practitioners"] + if p["id"] == service_request.ordering_practitioner_id + ), + service_request.ordering_practitioner_id, + ) + + for directory in (fixtures, task / "tests", task / "solution", session): + path = directory / "expectations.json" + if not path.exists(): + continue + payload = json.loads(path.read_text()) + if payload["target_case_id"] != case.id: + raise ValueError(f"Unexpected target case in {path}.") + payload["expected_target_status"] = "denied" + payload["expected_letter_types"] = ["denial"] + update_json(path, payload) + + for directory in (fixtures, session): + path = directory / "judge/canonical_case_record.json" + if not path.exists(): + continue + payload = json.loads(path.read_text()) + payload["canonical_determination"].update(decision="denied", terminal_case_status="denied") + for entry in payload.get("canonical_review_trajectory", []): + if entry.get("action") == "requirement_checked_member_ineligible": + entry["action"] = "deny_for_terminated_coverage" + update_json(path, payload) + + instruction = task / "instruction.md" + if instruction.exists(): + text = instruction.read_text().replace( + "You own this case end-to-end: walk the intake checklist, route it, then drive clinical review, MD decision, and any P2P through to final determination yourself.", + "Complete the intake checklist. If a failed check requires rejection, finalize the denial and issue the required letter. Otherwise, route the case and complete clinical review, the physician decision, and any required peer-to-peer review.", + ) + updates[instruction] = text.encode() + + form_paths = [ + directory / "request/prior-auth-request-form.pdf" + for directory in (fixtures, session) + if (directory / "request").is_dir() + ] + needs_pdf = False + for path in form_paths: + try: + text = " ".join(page.extract_text() or "" for page in PdfReader(path).pages) + needs_pdf |= not all( + fact in text for fact in ("Coverage Status", "Inactive", termination_date) + ) + except (FileNotFoundError, PdfReadError): + needs_pdf = True + if needs_pdf: + with TemporaryDirectory() as tmp: + output = Path(tmp) / "prior-auth-request-form.pdf" + generate_request_form_pdf( + output, + case=case, + patient=patient, + coverage=coverage, + service_request=service_request, + provider_name=provider_name, + ) + updates.update({path: stable_pdf_bytes(output) for path in form_paths}) + + changed = [] + for path, content in updates.items(): + if not path.exists() or path.read_bytes() != content: + path.write_bytes(content) + changed.append(path) + return changed + + +if __name__ == "__main__": + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--data-dir", type=Path, default=Path("data")) + for path in repair(parser.parse_args().data_dir): + print(path) diff --git a/src/chi_bench/__init__.py b/src/chi_bench/__init__.py index 00733a4..6f781d3 100644 --- a/src/chi_bench/__init__.py +++ b/src/chi_bench/__init__.py @@ -1,3 +1,3 @@ """ChiBench — healthcare simulation environment for AI agent evaluation.""" -__version__ = "0.1.0" +__version__ = "1.1.0" diff --git a/src/chi_bench/core/context.py b/src/chi_bench/core/context.py index 1bf9801..442605b 100644 --- a/src/chi_bench/core/context.py +++ b/src/chi_bench/core/context.py @@ -4030,6 +4030,11 @@ def route_case_internal( """ self.ensure_payer_mode_enabled() case = self.get_case(case_id) + intake = self.intake_case_for_case(case_id) + if intake is not None and intake.decision_member_eligible is False: + raise InvalidCaseActionError( + "Failed member eligibility must be denied at intake; do not route to review." + ) policy = self.get_policy(case_id) _linked_intake_case = ( None if case.payer_intake_case_id else self.intake_case_for_case(case_id) diff --git a/src/chi_bench/core/models.py b/src/chi_bench/core/models.py index 393b836..1e5f580 100644 --- a/src/chi_bench/core/models.py +++ b/src/chi_bench/core/models.py @@ -1031,7 +1031,9 @@ class DeterminationRecord(ChiBenchModel): id: str case_id: str intake_case_id: str - source: str # "nurse_recommendation" | "md_decision" + source: ( + str # "nurse_recommendation" | "md_decision" | "triage_auto_approve" | "intake_eligibility" + ) source_record_id: str original_recommendation: str final_decision: DeterminationDecision diff --git a/src/chi_bench/core/pdf/maintenance.py b/src/chi_bench/core/pdf/maintenance.py new file mode 100644 index 0000000..065f9c7 --- /dev/null +++ b/src/chi_bench/core/pdf/maintenance.py @@ -0,0 +1,21 @@ +"""Stable PDF bytes for maintenance of downloaded task artifacts.""" + +from io import BytesIO +from pathlib import Path + +from pypdf import PdfReader, PdfWriter + + +def stable_pdf_bytes(path: Path) -> bytes: + """Remove renderer timestamps so unchanged task PDFs keep the same hash.""" + reader = PdfReader(path) + writer = PdfWriter() + writer.clone_document_from_reader(reader) + writer.metadata = { + key: value + for key, value in (reader.metadata or {}).items() + if key not in {"/CreationDate", "/ModDate"} + } + output = BytesIO() + writer.write(output) + return output.getvalue() diff --git a/src/chi_bench/core/pdf/templates/pa_request_form.html.j2 b/src/chi_bench/core/pdf/templates/pa_request_form.html.j2 index 3d0edb0..68ec9ad 100644 --- a/src/chi_bench/core/pdf/templates/pa_request_form.html.j2 +++ b/src/chi_bench/core/pdf/templates/pa_request_form.html.j2 @@ -38,7 +38,9 @@ ("Payer", coverage.payer_name), ("Plan", coverage.plan_name), ("Network Status", coverage.network_status.replace("_", " ").title()), -]) }} + ("Coverage Status", "Active" if coverage.active else "Inactive"), +] + ([("Coverage Effective Date", coverage.effective_date)] if coverage.effective_date else []) + + ([("Coverage Termination Date", coverage.termination_date)] if coverage.termination_date else [])) }}
SECTION 3: REQUESTING PROVIDER
{{ metadata_table_1col([ diff --git a/src/chi_bench/server/routers/models.py b/src/chi_bench/server/routers/models.py index 1e3ded6..7e4e272 100644 --- a/src/chi_bench/server/routers/models.py +++ b/src/chi_bench/server/routers/models.py @@ -268,8 +268,8 @@ class IntakeDecisionsBody(BaseModel): case_id: str request_complete: bool member_eligible: bool - service_covered: bool - provider_network_status: ProviderNetworkStatus + service_covered: bool | None = None + provider_network_status: ProviderNetworkStatus | None = None class IntakeProcedureLookupBody(BaseModel): diff --git a/src/chi_bench/services/determination.py b/src/chi_bench/services/determination.py index aafe3c0..0903f3c 100644 --- a/src/chi_bench/services/determination.py +++ b/src/chi_bench/services/determination.py @@ -21,6 +21,7 @@ DeterminationRecord, DirectoryContact, MdCaseDecision, + PayerLetterRequest, PriorAuthCase, ReviewDecision, ) @@ -56,7 +57,7 @@ def __init__(self, ctx: ServiceContext): self.ctx = ctx def get_determination_summary(self, case_id: str) -> dict[str, Any]: - """Aggregate the upstream review state into a ready-to-finalize summary: source artifact (nurse recommendation vs MD decision), source recommendation, criteria tallies, documentation gaps, policy metadata, any existing determination, and the current workflow state. `source` is `nurse_recommendation`, `md_decision`, or null when no finalizable upstream exists. + """Aggregate the upstream review state into a ready-to-finalize summary: source artifact (nurse recommendation vs MD decision), source recommendation, criteria tallies, documentation gaps, policy metadata, any existing determination, and the current workflow state. `source` is `intake_eligibility` for a completed intake with failed member eligibility, `nurse_recommendation`, `md_decision`, or null when no finalizable upstream exists. Args: case_id: PriorAuthCase.id, such as 'CASE-A1B2C3D4'. @@ -172,6 +173,15 @@ def _preflight_finalization( if source_state["source"] is None: raise ValueError("No upstream recommendation or decision found for this case.") + # A failed eligibility check cannot authorize any part of the request. + intake_record = self.ctx.intake_case_for_case(case_id) + if ( + intake_record is not None + and intake_record.decision_member_eligible is False + and det_decision != DeterminationDecision.DENIED + ): + raise InvalidCaseActionError("Failed member eligibility only permits denial.") + # 5. Map + override validation recommended_decision = self._map_recommendation_to_final_decision( source_state["recommendation"] @@ -206,7 +216,6 @@ def _preflight_finalization( ) from exc # 7. Intake-complete preflight - intake_record = self.ctx.intake_case_for_case(case_id) if intake_record is None: raise InvalidCaseActionError( f"Cannot finalize determination for case {case_id!r}: no " @@ -236,6 +245,10 @@ def _preflight_finalization( policy_citations = [] supporting_basis = None + if source_state["source"] == "intake_eligibility": + policy_citations = ["Intake and Eligibility, section 5: ELIG_TERMED"] + supporting_basis = source_state["rationale"] + # 11. Build in-memory records record = DeterminationRecord( id=f"DET-{uuid4().hex[:8]}", @@ -307,9 +320,29 @@ def _persist_finalization(self, plan: _FinalizationPlan) -> dict[str, Any]: self.ctx.save_review_decision(plan.review_decision) # 3. Transition + save case (point of terminal status commit) + if plan.record.source == "intake_eligibility": + plan.case.outcome_reason = plan.review_decision.rationale case = self.ctx.transition(plan.case, plan.target_status, actor=plan.actor) self.ctx.save_case(case) + if plan.record.source == "intake_eligibility": + self.ctx.save_letter_request( + PayerLetterRequest( + id=f"LREQ-{uuid4().hex[:8]}", + case_id=case.id, + letter_type="denial", + rationale=plan.review_decision.rationale, + supporting_basis=plan.review_decision.supporting_basis, + policy_citations=plan.review_decision.policy_citations, + appeal_information=( + "You have the right to appeal this decision. Contact your health plan " + "for filing instructions and the applicable deadline." + ), + requested_by=plan.actor, + requested_at=self.ctx.now, + ) + ) + # 4. Generate authorization number (approved / partially_approved only) if plan.generate_auth: auth_result = self.ctx.review.generate_authorization_number(plan.case.id) @@ -478,7 +511,7 @@ def finalize_determination( """Finalize the case determination. Persists a PayerDetermination + ReviewDecision and transitions the case (`approved`, `denied`, or `partially_approved`); for approvals, an authorization number is generated. Does not compose or deliver provider correspondence. Args: - case_id: PriorAuthCase.id, such as 'CASE-A1B2C3D4'. Must have a finalizable upstream recommendation or decision; pending peer-to-peer / outstanding info / incomplete physician review block the call. + case_id: PriorAuthCase.id, such as 'CASE-A1B2C3D4'. A completed intake with member_eligible=False permits denial for ELIG_TERMED without clinical review. Otherwise a finalizable review recommendation or decision is required; pending peer-to-peer / outstanding info / incomplete physician review block the call. decision: final determination. `approved` = approve the request; `denied` = deny the request; `partially_approved` = approve some lines, deny or modify others. overridden: True when `decision` intentionally differs from the upstream-recommended decision. override_reason: free-text rationale; required when `overridden` is True. @@ -646,6 +679,21 @@ def _resolve_source_state(self, case_id: str) -> dict[str, Any]: else: state["blocking_reason"] = "awaiting_requested_information" + if nurse_rec is None: + intake = self.ctx.intake_case_for_case(case_id) + if ( + intake is not None + and intake.status == IntakeStatus.COMPLETE + and intake.decision_member_eligible is False + ): + state.update( + source="intake_eligibility", + source_record_id=intake.id, + recommendation="deny", + original_recommendation="deny", + rationale="ELIG_TERMED: Member coverage is inactive on the requested date of service.", + ) + return state def _map_recommendation_to_final_decision( diff --git a/src/chi_bench/services/intake.py b/src/chi_bench/services/intake.py index 4d94f0b..a401644 100644 --- a/src/chi_bench/services/intake.py +++ b/src/chi_bench/services/intake.py @@ -394,8 +394,8 @@ def confirm_intake_decisions( case_id: str, request_complete: bool, member_eligible: bool, - service_covered: bool, - provider_network_status: ProviderNetworkStatus, + service_covered: bool | None = None, + provider_network_status: ProviderNetworkStatus | None = None, ) -> dict[str, Any]: """Freeze the four intake-level decisions on an IntakeCase and mark intake `complete`. Validation of the decisions against intake-section content is caller-side — this call simply records what the caller supplies. @@ -403,16 +403,25 @@ def confirm_intake_decisions( case_id: PriorAuthCase.id, such as 'CASE-A1B2C3D4'. request_complete: True when all required intake fields are populated per the auth-type contract. member_eligible: True when the member's coverage is active for the requested service date. - service_covered: True when the requested procedure is within the member's benefit plan. - provider_network_status: network tier of the requesting provider. `in_network` = provider contracts with the payer; `out_of_network` = provider does not contract with the payer; `not_found` = provider identity could not be matched to the payer directory. + service_covered: True when the requested procedure is within the member's benefit plan. May be omitted when member_eligible is False, because processing stops at the failed eligibility check. + provider_network_status: network tier of the requesting provider. May be omitted when member_eligible is False. `in_network` = provider contracts with the payer; `out_of_network` = provider does not contract with the payer; `not_found` = provider identity could not be matched to the payer directory. """ self.ctx._require_loaded() intake = self._load_intake(case_id) + if member_eligible and service_covered is None: + raise ValueError("service_covered is required when member_eligible is True.") + if member_eligible and provider_network_status is None: + raise ValueError("provider_network_status is required when member_eligible is True.") + intake.decision_request_complete = request_complete intake.decision_member_eligible = member_eligible intake.decision_service_covered = service_covered - intake.decision_provider_network_status = ProviderNetworkStatus(provider_network_status) + intake.decision_provider_network_status = ( + ProviderNetworkStatus(provider_network_status) + if provider_network_status is not None + else None + ) intake.status = IntakeStatus.COMPLETE intake.completed_at = self.ctx.now diff --git a/src/chi_bench/services/letters.py b/src/chi_bench/services/letters.py index 64b3195..77d85ed 100644 --- a/src/chi_bench/services/letters.py +++ b/src/chi_bench/services/letters.py @@ -128,11 +128,19 @@ def _generate_letter( "letter": self.ctx.decorate_letter_payload(generated), "letter_request": request.model_dump(mode="json"), } + determination = self.ctx.latest_finalized_determination(case_id) letter = Letter( id=f"LET-{uuid4().hex[:8]}", case_id=case_id, letter_type=letter_type, subject=subject, + reason_code=( + "ELIG_TERMED" + if letter_type == "denial" + and determination is not None + and determination.source == "intake_eligibility" + else None + ), body=self._first_non_blank( body, request.rationale, diff --git a/src/chi_bench/services/triage.py b/src/chi_bench/services/triage.py index e7685ca..3a86b85 100644 --- a/src/chi_bench/services/triage.py +++ b/src/chi_bench/services/triage.py @@ -208,6 +208,10 @@ def set_disposition( disp = TriageDisposition(disposition) intake_data = self.ctx.intake.get_intake_case(case_id) + if intake_data.get("decision_member_eligible") is False: + raise InvalidCaseActionError( + "Failed member eligibility must be denied at intake; do not enter triage." + ) received_at = _parse_dt(intake_data["received_at"]) record = TriageRecord( diff --git a/src/chi_bench/verifier/stages/outcome.py b/src/chi_bench/verifier/stages/outcome.py index ec66768..5238450 100644 --- a/src/chi_bench/verifier/stages/outcome.py +++ b/src/chi_bench/verifier/stages/outcome.py @@ -69,6 +69,9 @@ def _lookup_source_record(exported: dict[str, Any], det: dict[str, Any]) -> dict elif source == "nurse_recommendation": table = exported.get("payer_nurse_case_recommendations") or [] decision_field = "recommendation" + elif source == "intake_eligibility": + table = exported.get("intake_cases") or [] + decision_field = None elif source == "triage_auto_approve": # Fast-track approval (gold_card_lane / notification_only_lane): # the routing record itself is the authorizing source. No @@ -79,7 +82,11 @@ def _lookup_source_record(exported: dict[str, Any], det: dict[str, Any]) -> dict else: return None for row in table: - if isinstance(row, dict) and normalized_str(row.get("id")) == source_record_id: + if ( + isinstance(row, dict) + and normalized_str(row.get("id")) == source_record_id + and normalized_str(row.get("case_id")) == normalized_str(det.get("case_id")) + ): return {"row": row, "decision_field": decision_field, "source": source} return None @@ -98,6 +105,16 @@ def _recommendation_maps_to_final(exported: dict[str, Any], det: dict[str, Any]) return False source = source_ref["source"] + if source == "intake_eligibility": + intake = source_ref["row"] + return ( + intake.get("status") == "complete" + and intake.get("decision_member_eligible") is False + and normalized_str(det.get("intake_case_id")) == normalized_str(intake.get("id")) + and det.get("original_recommendation") == "deny" + and final == "denied" + and not overridden + ) # Fast-track approval has no per-record decision field — the routing # record IS the authorization. `final_decision` must be 'approved', # `overridden` must be False (fast-track never overrides). No diff --git a/tests/unit/test_intake_denial.py b/tests/unit/test_intake_denial.py new file mode 100644 index 0000000..eb4d491 --- /dev/null +++ b/tests/unit/test_intake_denial.py @@ -0,0 +1,321 @@ +"""A failed member eligibility check can end a payer case before clinical review.""" + +from datetime import datetime + +import pytest + +from chi_bench.core.clock import WorldClock +from chi_bench.core.context import ServiceContext +from chi_bench.core.enums import DeterminationDecision, IntakeStatus, PayerMode, PayerRouteTarget +from chi_bench.core.errors import InvalidCaseActionError +from chi_bench.core.models import ( + Coverage, + HiddenCasePolicy, + IntakeCase, + MdCaseDecision, + NurseCaseRecommendation, + Patient, + PriorAuthCase, + ServiceRequest, +) +from chi_bench.core.pdf import render_request_form_html +from chi_bench.core.scenario import ScenarioDefinition +from chi_bench.verifier.stages.outcome import OutcomeVerifier + + +@pytest.fixture +def ctx(tmp_path): + context = ServiceContext.build(tmp_path) + context._scenario = ScenarioDefinition.model_construct(task_kind="payer_not_covered") + context._clock = WorldClock(datetime(2026, 3, 15, 9)) + context.payer_mode = PayerMode.AGENT + context.agent_owned_payer_workflow = True + case = PriorAuthCase( + id="CASE-TEST", + external_case_id="PA-TEST", + patient_id="PAT-TEST", + service_request_id="SR-TEST", + status="submitted", + urgency="standard", + requested_by="provider", + assigned_to="payer", + created_at=context.now, + updated_at=context.now, + decision_due_at=context.now, + ) + context.save_case(case) + context.save_hidden_policy( + HiddenCasePolicy(id="POLICY-TEST", case_id=case.id, route_target="standard_um") + ) + intake = IntakeCase(id="INTAKE-TEST", case_id=case.id, received_at=context.now) + context.store.save_resource("payer", "intake_cases", intake.id, intake.model_dump(mode="json")) + context.store.save_resource( + "provider", + "directory_contacts", + "PAYER-RN", + { + "id": "PAYER-RN", + "name": "Reviewer", + "role": "um_nurse", + "email": "", + "phone": "", + "organization_id": "ORG-PAYER", + "escalation_role": "um_nurse", + }, + ) + yield context + context.close() + + +def confirm(ctx, *, member_eligible=False, service_covered=None): + return ctx.intake.confirm_intake_decisions( + "CASE-TEST", + request_complete=True, + member_eligible=member_eligible, + service_covered=service_covered, + provider_network_status="in_network", + ) + + +def exported(ctx): + return { + "target_case": ctx.get_case("CASE-TEST").model_dump(mode="json"), + "intake_cases": ctx.store.list_resources("payer", "intake_cases"), + "payer_determinations": ctx.store.list_resources("payer", "payer_determinations"), + "payer_nurse_case_recommendations": ctx.store.list_resources( + "payer", "payer_nurse_case_recommendations" + ), + "payer_md_case_decisions": ctx.store.list_resources("payer", "payer_md_case_decisions"), + "review_decisions": ctx.store.list_resources("provider", "review_decisions"), + "state_transitions": [x.model_dump(mode="json") for x in ctx.store.list_transitions()], + "stage_traces": [x.model_dump(mode="json") for x in ctx.store.list_stage_traces()], + "audit_logs": [x.model_dump(mode="json") for x in ctx.store.list_audit_logs()], + } + + +def test_completed_ineligible_intake_can_be_denied(ctx): + intake = confirm(ctx) + result = ctx.determination.finalize_determination("CASE-TEST", DeterminationDecision.DENIED) + assert result["case"]["status"] == "denied" + assert "ELIG_TERMED" in result["case"]["outcome_reason"] + assert result["determination"]["source"] == "intake_eligibility" + assert result["determination"]["source_record_id"] == intake["id"] + assert result["case"]["authorization_number"] is None + assert intake["decision_service_covered"] is None + letter_request = ctx.pending_letter_request("CASE-TEST", "denial") + assert letter_request is not None + assert "ELIG_TERMED" in letter_request.rationale + assert "ELIG_TERMED" in letter_request.supporting_basis + assert letter_request.policy_citations + assert letter_request.appeal_information + checks = ( + OutcomeVerifier( + exported(ctx), + { + "target_case_id": "CASE-TEST", + "expected_target_status": "denied", + }, + [], + ) + .verify() + .checks + ) + assert checks["outcome.clean_determination"] is True, checks + + +@pytest.mark.parametrize("decision", ["approved", "partially_approved"]) +def test_ineligible_intake_cannot_be_approved_even_with_override(ctx, decision): + confirm(ctx) + with pytest.raises(InvalidCaseActionError, match="eligibility"): + ctx.determination.finalize_determination( + "CASE-TEST", + decision, + overridden=True, + override_reason="override", + ) + assert ctx.get_case("CASE-TEST").status.value == "submitted" + assert not ctx.store.list_resources("payer", "payer_determinations") + + +def test_eligible_intake_still_requires_service_coverage(ctx): + with pytest.raises(ValueError, match="service_covered"): + confirm(ctx, member_eligible=True) + assert ctx.intake_case_for_case("CASE-TEST").status == IntakeStatus.NEW + + +def test_intake_denial_does_not_generate_a_letter_automatically(ctx, monkeypatch): + # Keep real letter persistence; skip PDF rendering at the filesystem boundary. + monkeypatch.setattr(ctx, "_materialize_case_workspace_artifacts", lambda *args, **kwargs: None) + ctx.agent_owned_payer_workflow = False + confirm(ctx) + ctx.determination.finalize_determination("CASE-TEST", "denied") + assert not ctx.list_case_letters("CASE-TEST") + assert ctx.pending_letter_request("CASE-TEST", "denial") is not None + + +def test_denial_letter_uses_the_intake_reason_and_basis(ctx, monkeypatch): + monkeypatch.setattr(ctx, "_materialize_case_workspace_artifacts", lambda *args, **kwargs: None) + confirm(ctx) + ctx.determination.finalize_determination("CASE-TEST", "denied") + result = ctx.letters.generate_denial_letter("CASE-TEST") + assert result["letter"]["reason_code"] == "ELIG_TERMED" + assert "ELIG_TERMED" in result["letter"]["supporting_basis"] + assert ctx.letters.audit_letter_completeness(result["letter"]["id"])["complete"] is True + + +def test_eligible_intake_does_not_allow_denial_without_review(ctx): + confirm(ctx, member_eligible=True, service_covered=True) + with pytest.raises(ValueError, match="upstream"): + ctx.determination.finalize_determination("CASE-TEST", "denied") + + +@pytest.mark.parametrize( + "source, recommendation, decision", + [ + ("nurse", "approve", "approved"), + ("md", "approve", "approved"), + ("md", "deny", "denied"), + ], +) +def test_existing_clinical_determinations_still_pass(ctx, source, recommendation, decision): + confirm(ctx, member_eligible=True, service_covered=True) + if source == "nurse": + record = NurseCaseRecommendation( + id="REC-TEST", + case_id="CASE-TEST", + intake_case_id="INTAKE-TEST", + recommendation=recommendation, + created_at=ctx.now, + ) + table = "payer_nurse_case_recommendations" + else: + record = MdCaseDecision( + id="REC-TEST", + case_id="CASE-TEST", + intake_case_id="INTAKE-TEST", + decision=recommendation, + signed_off=True, + created_at=ctx.now, + ) + table = "payer_md_case_decisions" + ctx.store.save_resource("payer", table, record.id, record.model_dump(mode="json")) + result = ctx.determination.finalize_determination("CASE-TEST", decision) + assert result["case"]["status"] == decision + assert ( + OutcomeVerifier( + exported(ctx), + { + "target_case_id": "CASE-TEST", + "expected_target_status": decision, + }, + [], + ) + .verify() + .checks["outcome.clean_determination"] + is True + ) + + +def test_failed_eligibility_blocks_an_existing_approval_recommendation(ctx): + confirm(ctx) + record = NurseCaseRecommendation( + id="REC-TEST", + case_id="CASE-TEST", + intake_case_id="INTAKE-TEST", + recommendation="approve", + created_at=ctx.now, + ) + ctx.store.save_resource( + "payer", "payer_nurse_case_recommendations", record.id, record.model_dump(mode="json") + ) + with pytest.raises(InvalidCaseActionError, match="eligibility"): + ctx.determination.finalize_determination("CASE-TEST", "approved") + assert not ctx.store.list_resources("payer", "payer_determinations") + + +@pytest.mark.parametrize("lane", [None, "gold_card_lane"]) +def test_ineligible_member_cannot_be_routed_to_review_or_approval(ctx, lane): + confirm(ctx) + with pytest.raises(InvalidCaseActionError, match="eligibility"): + ctx.route_case_internal( + "CASE-TEST", override_target=PayerRouteTarget(lane) if lane else None + ) + assert ctx.get_case("CASE-TEST").status.value == "submitted" + assert not ctx.store.list_resources("payer", "payer_routing_records") + + +def test_ineligible_member_cannot_enter_triage(ctx): + confirm(ctx) + with pytest.raises(InvalidCaseActionError, match="eligibility"): + ctx.triage.set_disposition("CASE-TEST", "routine", "2026-03-22T09:00:00") + assert not ctx.store.list_resources("payer", "payer_triage_records") + + +def test_verifier_rejects_denial_with_incomplete_or_eligible_intake(): + from chi_bench.verifier.stages.outcome import _recommendation_maps_to_final + + determination = { + "source": "intake_eligibility", + "source_record_id": "INTAKE-TEST", + "case_id": "CASE-TEST", + "intake_case_id": "INTAKE-TEST", + "original_recommendation": "deny", + "final_decision": "denied", + } + for status, eligible in [("new", False), ("complete", True), ("complete", None)]: + assert not _recommendation_maps_to_final( + { + "intake_cases": [ + { + "id": "INTAKE-TEST", + "case_id": "CASE-TEST", + "status": status, + "decision_member_eligible": eligible, + } + ] + }, + determination, + ) + + +def test_unconfirmed_eligibility_does_not_allow_denial(ctx): + with pytest.raises(ValueError, match="upstream"): + ctx.determination.finalize_determination("CASE-TEST", "denied") + assert not ctx.store.list_resources("payer", "payer_determinations") + + +def test_request_form_displays_coverage_status_and_termination_date(ctx): + patient = Patient( + id="PAT-TEST", + member_id="MEMBER-TEST", + first_name="Test", + last_name="Patient", + dob="1971-08-14", + sex="female", + ) + coverage = Coverage( + id="COV-TEST", + patient_id=patient.id, + payer_name="The Insurance Company", + plan_name="HMO", + member_number=patient.member_id, + active=False, + termination_date="2025-11-30", + ) + request = ServiceRequest( + id="SR-TEST", + patient_id=patient.id, + ordering_practitioner_id="DR-TEST", + service_type="medical", + procedure_code="95816", + description="EEG", + priority="standard", + diagnosis_codes=["G40.009"], + site_of_service="inpatient_hospital", + requested_date=datetime(2026, 3, 18), + ) + html = render_request_form_html(ctx.get_case("CASE-TEST"), patient, coverage, request, "Doctor") + assert "Coverage Status" in html + assert "Inactive" in html + assert "Coverage Termination Date" in html + assert "2025-11-30" in html diff --git a/tests/unit/test_pdf_maintenance.py b/tests/unit/test_pdf_maintenance.py new file mode 100644 index 0000000..d824b16 --- /dev/null +++ b/tests/unit/test_pdf_maintenance.py @@ -0,0 +1,33 @@ +"""Task artifact hashes must not depend on PDF rendering timestamps.""" + +from io import BytesIO + +from pypdf import PdfReader, PdfWriter + +from chi_bench.core.pdf.maintenance import stable_pdf_bytes + + +def test_pdf_timestamps_do_not_change_artifact_bytes(tmp_path): + rendered = [] + for index, timestamp in enumerate(("D:20261006120000Z", "D:20261006120100Z")): + path = tmp_path / f"form-{index}.pdf" + writer = PdfWriter() + writer.add_blank_page(width=72, height=144) + writer.add_metadata( + { + "/Title": "Prior authorization request", + "/CreationDate": timestamp, + "/ModDate": timestamp, + } + ) + writer.write(path) + rendered.append(stable_pdf_bytes(path)) + + assert rendered[0] == rendered[1] + reader = PdfReader(BytesIO(rendered[0])) + assert reader.metadata.title == "Prior authorization request" + assert "/CreationDate" not in reader.metadata + assert "/ModDate" not in reader.metadata + assert len(reader.pages) == 1 + assert reader.pages[0].mediabox.width == 72 + assert reader.pages[0].mediabox.height == 144 diff --git a/tests/unit/test_repair_pa_t035.py b/tests/unit/test_repair_pa_t035.py new file mode 100644 index 0000000..09e4e84 --- /dev/null +++ b/tests/unit/test_repair_pa_t035.py @@ -0,0 +1,139 @@ +"""The data repair keeps the existing request form and restores its source facts.""" + +import json + +from pypdf import PdfWriter + +from scripts import repair_pa_t035 + + +def test_repair_updates_existing_form_and_all_expectations(tmp_path, monkeypatch): + task = tmp_path / "prior_auth_um/tasks" / repair_pa_t035.TASK_ID + world = { + "patients": [ + { + "id": "PAT-TEST", + "member_id": "MEMBER-TEST", + "first_name": "Test", + "last_name": "Patient", + "dob": "1971-08-14", + "sex": "female", + } + ], + "coverages": [ + { + "id": "COV-TEST", + "patient_id": "PAT-TEST", + "payer_name": "The Insurance Company", + "plan_name": "HMO", + "member_number": "MEMBER-TEST", + "active": False, + "termination_date": None, + } + ], + "cases": [ + { + "id": "CASE-TEST", + "external_case_id": "PA-TEST", + "patient_id": "PAT-TEST", + "service_request_id": "SR-TEST", + "status": "submitted", + "urgency": "standard", + "requested_by": "provider", + "assigned_to": "payer", + "created_at": "2026-03-15T09:00:00", + "updated_at": "2026-03-15T09:00:00", + "decision_due_at": "2026-03-29T09:00:00", + } + ], + "service_requests": [ + { + "id": "SR-TEST", + "patient_id": "PAT-TEST", + "ordering_practitioner_id": "DR-TEST", + "service_type": "medical", + "procedure_code": "95816", + "description": "EEG", + "priority": "standard", + "diagnosis_codes": ["G40.009"], + "site_of_service": "inpatient_hospital", + "requested_date": "2026-03-18T09:00:00", + } + ], + "practitioners": [], + "documents": [{"id": "DOC-TEST", "content": "Original clinical note"}], + } + for domain in ("prior_auth_provider", "prior_auth_um"): + path = tmp_path / domain / "shared/worlds" / repair_pa_t035.WORLD_NAME + path.parent.mkdir(parents=True) + path.write_text(json.dumps(world)) + expected = { + "target_case_id": "CASE-TEST", + "target_patient_id": "PAT-TEST", + "task_actor": "payer", + "task_kind": "payer_not_covered", + "expected_target_status": "requirement_checked", + "expected_letter_types": [], + "stage_ground_truth": [ + { + "model_name": "IntakeCase", + "expected_fields": { + "decision_member_eligible": False, + "coverage_details": "Member coverage was terminated effective 2025-11-30.", + }, + } + ], + } + session = tmp_path / "marathon/prior_auth_um/fixtures/tasks" / repair_pa_t035.TASK_ID + for directory in (task / "fixtures", task / "tests", task / "solution", session): + directory.mkdir(parents=True) + (directory / "expectations.json").write_text(json.dumps(expected)) + for directory in (task / "fixtures", session): + (directory / "request").mkdir() + (directory / "request/prior-auth-request-form.pdf").write_bytes(b"old form") + (directory / "judge").mkdir() + (directory / "judge/canonical_case_record.json").write_text( + json.dumps( + { + "canonical_determination": {"terminal_case_status": "requirement_checked"}, + "canonical_review_trajectory": [ + {"action": "requirement_checked_member_ineligible"} + ], + "documents_catalog": [], + } + ) + ) + calls = [] + + def render(path, **kwargs): + calls.append(kwargs) + writer = PdfWriter() + writer.add_blank_page(width=72, height=72) + writer.write(path) + + monkeypatch.setattr(repair_pa_t035, "generate_request_form_pdf", render) + + repair_pa_t035.repair(tmp_path) + + assert len(calls) == 1 + assert calls[0]["coverage"].termination_date == "2025-11-30" + for domain in ("prior_auth_provider", "prior_auth_um"): + actual = json.loads( + (tmp_path / domain / "shared/worlds" / repair_pa_t035.WORLD_NAME).read_text() + ) + assert actual["coverages"][0]["termination_date"] == "2025-11-30" + assert actual["documents"] == world["documents"] + assert actual["cases"][0]["status"] == "submitted" + for directory in (task / "fixtures", task / "tests", task / "solution", session): + actual = json.loads((directory / "expectations.json").read_text()) + assert actual["expected_target_status"] == "denied" + assert actual["expected_letter_types"] == ["denial"] + assert actual["stage_ground_truth"] == expected["stage_ground_truth"] + for directory in (task / "fixtures", session): + assert list((directory / "request").iterdir()) == [ + directory / "request/prior-auth-request-form.pdf" + ] + record = json.loads((directory / "judge/canonical_case_record.json").read_text()) + assert record["canonical_determination"]["decision"] == "denied" + assert record["canonical_determination"]["terminal_case_status"] == "denied" + assert record["documents_catalog"] == [] diff --git a/tests/unit/test_tool_reference_path.py b/tests/unit/test_tool_reference_path.py new file mode 100644 index 0000000..a0e8511 --- /dev/null +++ b/tests/unit/test_tool_reference_path.py @@ -0,0 +1,70 @@ +"""The path advertised by published tasks follows runtime reference selection.""" + +from __future__ import annotations + +import os +import subprocess +from pathlib import Path + +import pytest + +REPO_ROOT = Path(__file__).resolve().parents[2] + + +@pytest.mark.parametrize("per_task", [True, False], ids=["provider", "care-management"]) +@pytest.mark.parametrize("separator", [".", "__"], ids=["default-names", "rewritten-names"]) +def test_advertised_tool_reference_path(tmp_path: Path, per_task: bool, separator: str) -> None: + canonical = tmp_path / "opt/chi-bench-task-assets/tool_reference.md" + advertised = tmp_path / "opt/healthverse-task-assets/tool_reference.md" + canonical.parent.mkdir(parents=True) + canonical.write_text("Shared reference: cm_chart.get_member\nreport.v1.pdf\n") + + task_dir = tmp_path / "tasks/task" + task_dir.mkdir(parents=True) + if per_task: + (task_dir / "tool_reference.md").write_text( + "Task reference: chart.get_patient\nreport.v1.pdf\n" + ) + + # Execute the Dockerfile's asset setup, redirecting /opt into this test's + # filesystem. The shared file above stands in for its COPY instruction. + dockerfile = (REPO_ROOT / "docker/Dockerfile").read_text() + asset_setup = dockerfile.split("RUN mkdir -p /opt/chi-bench-task-assets\n", 1)[1] + asset_setup = asset_setup.split("COPY docker/entrypoint.sh", 1)[0] + for line in asset_setup.splitlines(): + if line.startswith("RUN "): + subprocess.run( + ["sh", "-ec", line[4:].replace("/opt/", f"{tmp_path}/opt/")], + check=True, + capture_output=True, + text=True, + timeout=10, + ) + + # Run the actual reference-selection and rewriting block without starting + # the simulator. No Docker daemon, downloaded dataset, or API key is needed. + entrypoint = (REPO_ROOT / "docker/entrypoint.sh").read_text() + setup = "TOOL_REF_SRC=" + entrypoint.split("TOOL_REF_SRC=", 1)[1] + setup = setup.split("# In docker-compose mode", 1)[0] + subprocess.run( + ["sh", "-ec", setup.replace("/opt/", f"{tmp_path}/opt/")], + env={ + **os.environ, + "TASKS_ROOT": str(task_dir.parent), + "TASK_ID": task_dir.name, + "CHI_BENCH_MCP_TOOL_SEP": separator, + }, + check=True, + capture_output=True, + text=True, + timeout=10, + ) + + assert advertised.is_file(), "The tool-reference path in task instructions is unreadable" + expected = ( + f"Task reference: chart{separator}get_patient\nreport.v1.pdf\n" + if per_task + else f"Shared reference: cm_chart{separator}get_member\nreport.v1.pdf\n" + ) + assert advertised.read_text() == expected + assert advertised.read_bytes() == canonical.read_bytes() diff --git a/uv.lock b/uv.lock index 9040050..13508a1 100644 --- a/uv.lock +++ b/uv.lock @@ -476,7 +476,7 @@ wheels = [ [[package]] name = "chi-bench" -version = "0.1.0" +version = "1.1.0" source = { editable = "." } dependencies = [ { name = "anthropic" },