Repository navigation
Release v1.1.0: fix payer eligibility and published tool paths - #53
Conversation
…ground truth" This reverts commit 7ab81de.
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 926e2fa498
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| COPY data/care_management/shared/tool_reference.md /opt/chi-bench-task-assets/tool_reference.md | ||
| # Published task instructions still use the healthverse asset path. Alias the | ||
| # directory so per-task selection and tool-name rewriting remain visible there. | ||
| RUN ln -s /opt/chi-bench-task-assets /opt/healthverse-task-assets |
There was a problem hiding this comment.
Mirror the tool-reference alias in the Harbor image
This only fixes images built from docker/Dockerfile. I checked the published-task export path: scripts/export_harbor_tasks.py copies docker/Dockerfile.harbor, which creates /opt/chi-bench-task-assets but never creates /opt/healthverse-task-assets. Consequently, newly exported Harbor tasks still cannot read the path advertised in their instructions; add the same directory alias to that image definition.
Useful? React with 👍 / 👎.
| uv run python scripts/repair_pa_t035.py | ||
| uv run python scripts/regenerate_payer_request_forms.py |
There was a problem hiding this comment.
Repair pa_t035 in the Harbor build path
These commands repair only a user's local checkout. The published-task image built by docker/Dockerfile.harbor still downloads the unchanged chi-bench-v1.0.0 dataset and copies it directly without running either script, so a fresh Harbor export retains pa_t035's missing termination evidence and unreachable requirement_checked expectation. Invoke the repair during that build or publish and pin a repaired dataset revision before releasing this behavior.
Useful? React with 👍 / 👎.
| source_record_id=intake.id, | ||
| recommendation="deny", | ||
| original_recommendation="deny", | ||
| rationale="ELIG_TERMED: Member coverage is inactive on the requested date of service.", |
There was a problem hiding this comment.
Derive the eligibility denial reason from coverage facts
Any completed intake with member_eligible=False is now labeled ELIG_TERMED, even when eligibility failed because no coverage was found or coverage starts after the requested date—states already represented by the coverage API/model. In those cases this rationale also propagates to the case and denial letter, producing a materially incorrect reason code and appeal record; restrict this source to terminated coverage or derive the reason from the actual coverage status and dates.
Useful? React with 👍 / 👎.
Published task instructions point to a tool-reference directory that the runtime image did not provide. The pa_t035 payer task also hides its coverage termination date and expects a final state that payer tools cannot produce. This PR restores the published path and makes the eligibility failure visible in the existing request form. The payer can end the case in the existing
deniedstate and deliver a denial letter.Changes
/opt/healthverse-task-assetsto/opt/chi-bench-task-assets. Per-task tool-reference selection and tool-name rewriting remain visible through the published path.ELIG_TERMED. Block approval and clinical routing after failed eligibility. Create a pending letter request; the agent must generate and deliver the letter.get_member_coverageoutside payer MCP.repair_pa_t035.pyto restore termination date 2025-11-30 from the original synthesis facts, correct the unreachablerequirement_checkedexpectation to denial with a denial letter, and repair task and marathon copies.regenerate_payer_request_forms.pyfor the 25 payer request forms and their marathon copies. Remove rendering timestamps so unchanged content retains the same PDF hash.packagecommand toprepare.Data upgrade
The downloaded fixtures are ignored and are not part of this PR. After downloading the public dataset and approved handbook, run:
Use
playwright install --with-deps chromiumon Linux if needed. Runtime 1.1.0 is separate from dataset revision chi-bench-v1.0.0. The scripts apply local corrections; this PR does not publish a new Hugging Face revision or update the Harbor hub listing. Preserve old trial evidence and record the runtime commit and image digest for new runs.Validation
git diff --check: passed.signed_offexpectation was found; P2P code is unchanged.Build limit: The normal Docker build stalled at Docker Hub frontend resolution. The syntax-free retry stalled at the
python:3.12-slimmetadata lookup. Both owned builds were stopped. The successful test image uses the already available local runtime base, with an offline package install. This does not verify a fresh base-image download or dependency installation. No agent-model trial or LLM judge call was made for these checks.Policy follow-up: Live inspection confirms CHI-Bench is a Community Hackathon with Writeup submissions, while Rules Section 9 describes CSV upload and Kaggle Accuracy scoring. Section 11 requires a Kaggle entry in addition to private materials. The email reply does not remove that entry requirement. The cost thread also says all 75 public results are not required for private evaluation. The communication drafts recommend correcting the Rules to require a Writeup plus the private email package; this is not adopted policy. This PR preserves the published CSV requirements in the local reference.
Announcements and FAQ drafts are outside the repository. No competition message has been posted.