Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
56 changes: 56 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
# Changelog

## 1.1.0 — 2026-10-06

### Fixes

- Restore the published tool-reference path in the Docker image. The directory
alias `/opt/healthverse-task-assets` points to `/opt/chi-bench-task-assets`, so
existing task instructions see the selected per-task tool reference.
- Allow a completed payer intake with failed member eligibility to end in the
existing `denied` state with reason `ELIG_TERMED`. The agent generates and
delivers a denial letter. Approval and clinical routing are blocked after a
failed eligibility check.
- Add coverage status and known coverage dates to the existing request-form
template. The data repair restores pa_t035's termination date from the original
synthesis facts and replaces its unreachable `requirement_checked` expectation
with denial and a denial letter. No separate evidence document is added.
- Provide scripts to repair downloaded pa_t035 task copies and regenerate the
25 saved payer request forms and their marathon copies before rebuilding.
Remove rendering timestamps from regenerated PDFs so repeat runs preserve
their hashes when content is unchanged.
- Align the local competition Rules and Overview references with the published
Kaggle pages.

### Upgrade

After obtaining handbook access and downloading the task data:

```bash
uv sync --extra dev
uv run playwright install chromium
uv run python scripts/repair_pa_t035.py --data-dir data
uv run python scripts/regenerate_payer_request_forms.py --data-dir data
uv run cb data verify
uv run cb docker build
```

On Linux, use `uv run playwright install --with-deps chromium` if browser system
libraries are missing.

This is runtime version **1.1.0**. The downloaded dataset remains pinned to
**chi-bench-v1.0.0**; the scripts apply local corrections. This change does not
publish a new Hugging Face dataset revision. Preserve earlier trial results and
record the runtime commit and image digest for new runs.

### P2P clarification

The P2P workflow is unchanged. The original `MdCaseDecision` records the referral
to P2P; the completed P2P request supplies the outcome, and
`determination.finalize` records the final determination. No second physician
submission is required.

Among the six public payer tasks requiring P2P, t026 and t032 expect the referral
to have `signed_off=false`. Tasks t016, t019, t031, and t036 exclude MD review
from their scored stages. A direct-decision task may require signed physician
review even if an agent chooses P2P instead.
14 changes: 14 additions & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -151,6 +151,20 @@ Configs:

## Things to remember

- **Write Kaggle FAQs as direct participant answers in ASD-STE100 style, using the known
CHI-Bench dates. Omit organizer-reply citations, review notes, repeated requirements, and
verbatim restatements of user directions. Keep contact links in FAQs and only necessary
source links in announcements.
Assess existing rules against the intended workflow before proposing them.**
- **Use the CHI-Bench Cup sequence: competition deadline, private evaluation, result release,
then the IEEE-format camera-ready report. Do not add a paper-acceptance stage.** **Why:**
CHI-Bench evaluates competition systems; the report follows competition results. Use its
own schedule and participation policy; do not import another Cup's instructions. Use
organizer-specified submission routes; do not infer a requirement from a portal listing.
- **Preserve published task paths with image compatibility aliases when fixing runtime renames.**
**Why:** Rewriting downloaded `instruction.md` files changes task checksums. For tool references,
alias `/opt/healthverse-task-assets` to `/opt/chi-bench-task-assets` as a directory so per-task
selection and tool-name rewriting remain visible through the published path.
- **For simple, explicitly scoped maintenance PRs, work directly on a `codex/` branch in the
current clean checkout unless the user asks for a worktree or design spec.** **Why:** The user
prefers direct execution over extra process for low-risk changes such as a model-default update.
Expand Down
13 changes: 13 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -89,10 +89,23 @@ uv run huggingface-cli download actava/managed-care-operations-handbook \

**5. Build the Docker image** (~5 min, one-time).

Apply the `pa_t035` data correction before building. It restores the coverage
termination date in the existing request form and requires a payer denial letter.
The script changes only local data; it does not publish a dataset revision.

```bash
uv run playwright install chromium
uv run python scripts/repair_pa_t035.py
uv run python scripts/regenerate_payer_request_forms.py
Comment on lines +98 to +99

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Repair pa_t035 in the Harbor build path

These commands repair only a user's local checkout. The published-task image built by docker/Dockerfile.harbor still downloads the unchanged chi-bench-v1.0.0 dataset and copies it directly without running either script, so a fresh Harbor export retains pa_t035's missing termination evidence and unreachable requirement_checked expectation. Invoke the repair during that build or publish and pin a repaired dataset revision before releasing this behavior.

Useful? React with 👍 / 👎.

uv run cb docker build
```

The repair scripts render PDFs on the host. On Linux, use
`uv run playwright install --with-deps chromium` if browser system libraries are missing.
Runtime version `1.1.0` and dataset revision `chi-bench-v1.0.0` are separate;
keep the dataset pin at the revision you downloaded and record the runtime commit
and image digest with your results. See [the v1.1.0 changes](CHANGELOG.md).

> [!NOTE]
> `cb` is the short alias for `chi-bench`; both commands resolve to the same CLI. Pick whichever you prefer (the rest of this README uses `cb`). If your shell already aliases `cb` to something else (e.g. a clipboard tool), use `chi-bench`. For the full command surface and flag reference, read [`docs/cli.md`](docs/cli.md).

Expand Down
6 changes: 4 additions & 2 deletions configs/submission_example.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,11 @@
# cb submission validate -f <your-sub>.yaml
# cb submission run -f <your-sub>.yaml
# cb submission status -f <your-sub>.yaml
# cb submission package -f <your-sub>.yaml
# cb submission prepare -f <your-sub>.yaml
#
# The packet (<output_root>/<id>.zip) is what you upload to the leaderboard.
# The prepared packet is under <output_root>/packet/YYYY-MM-DD-<id>/.
# Follow the Kaggle organizer instructions for private-final package delivery;
# the separate evidence leaderboard accepts the same packet through a GitHub PR.

schema: chi-bench/submission/v1

Expand Down
3 changes: 3 additions & 0 deletions docker/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -99,6 +99,9 @@ COPY data/marathon /opt/chi-bench/tasks/marathon
COPY data/skills/managed-care-operations-handbook /workspace/skills/managed-care-operations-handbook
RUN mkdir -p /opt/chi-bench-task-assets
COPY data/care_management/shared/tool_reference.md /opt/chi-bench-task-assets/tool_reference.md
# Published task instructions still use the healthverse asset path. Alias the
# directory so per-task selection and tool-name rewriting remain visible there.
RUN ln -s /opt/chi-bench-task-assets /opt/healthverse-task-assets

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Mirror the tool-reference alias in the Harbor image

This only fixes images built from docker/Dockerfile. I checked the published-task export path: scripts/export_harbor_tasks.py copies docker/Dockerfile.harbor, which creates /opt/chi-bench-task-assets but never creates /opt/healthverse-task-assets. Consequently, newly exported Harbor tasks still cannot read the path advertised in their instructions; add the same directory alias to that image definition.

Useful? React with 👍 / 👎.


COPY docker/entrypoint.sh /usr/local/bin/entrypoint.sh
RUN chmod +x /usr/local/bin/entrypoint.sh
Expand Down
5 changes: 5 additions & 0 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,11 @@
5. After the agent stops (success / timeout / abstain), Harbor invokes the verifier in the same container.
6. Verifier writes `verifier/scorecard.json` and `verifier/verdicts.json`; Harbor produces `result.json`.

When the payer confirms failed member eligibility, processing stops at intake.
`determination.finalize` can use the completed intake record to set the existing
case status `denied`, with reason `ELIG_TERMED`. The agent then generates and
delivers the denial letter. Clinical review is not required for this result.

## Why the LLM judge needs Anthropic credits

The verifier always uses `claude-opus-4-7` (configurable but paper-faithful default).
Expand Down
23 changes: 23 additions & 0 deletions docs/cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -250,6 +250,29 @@ echo "$REV" > data/.chi-bench-version

`cb data download` exists so the pin file is written in one step; the README documents the raw HF CLI invocation as the canonical path.

For the `pa_t035` correction, run the following after downloading data and before
building the Docker image:

```bash
uv run playwright install chromium
uv run python scripts/repair_pa_t035.py --data-dir data
uv run python scripts/regenerate_payer_request_forms.py --data-dir data
```

These scripts render PDFs on the host, so Playwright needs its Chromium browser
there. On Linux, use `uv run playwright install --with-deps chromium` when system
libraries are missing.

This restores the coverage termination date in the existing request form and
changes the expected payer result to `denied`, with a denial letter. It updates
the task's expectation copies and its marathon copy. It does not publish data
or change the dataset version pin.

The second script updates all saved payer request forms with the same template,
including their marathon copies. Coverage status appears on each form. Coverage
dates appear only when the shared world provides them. Case facts and scoring
expectations are unchanged by this regeneration.

---

## `cb submission`
Expand Down
Loading
Loading