CHI-Bench: HG-Market-System repaired v1.1.0 packet · 20/75 · 26.67%

[hgreco85]

Completed Full75 packet with the maintainer-requested v1.1.0 evidence repairs applied.

• Provider: 11/25
• Payer: 7/25
• Care: 2/25
• Overall: 20/75 = 26.67%

Repair scope only:

• Care cm_mdd_hard_refuses_002: judge-only re-evaluation from the preserved final state on CHI-Bench v1.1.0, one claude-opus-4-7 vote, valid 0/5 rubrics. Historical pass corrected to fail.
• Payer P2P t016, t019, t036: exact v1.1.0 reruns using the historical Full75 HG agent snapshot and CHI_BENCH_P2P_SIMULATOR_MODEL=claude-sonnet-5; all three remained 1.0/1.0 with 7/7 judge checks true.
• The other 71 trial artifacts are preserved.

The packet includes repair_evidence.json with run IDs, artifact digests and task-level hashes. No Full75 rerun was performed.

[hgreco85]

@haolin-chen-actava — thank you. The official validation has now completed successfully:

✅ All checks passed
✅ 75/75 trials
✅ 21/75 = 28.00%
✅ PR is mergeable

This is ready for maintainer review/merge. This PR supersedes our earlier HG submissions #24 and #25.

[github-actions]

Submission validation — PR diff

✅ All checks passed.

[haolin-chen-actava]

Hi @hgreco85, I am seeing a few issues there. We can merge this PR after resolving them.

1. It seems some care management tasks lacks the required judge: e.g. cm_mdd_hard_refuses_002.
2. UM P2P (t016, t019, t036): the peer simulator calls a retired model and results in errors. Could you set CHI_BENCH_P2P_SIMULATOR_MODEL=claude-sonnet-5 and run them?

FYI, we are making some fixes to the benchmark very soon. It would be best if you can run the latest version after we fix (should arrive today).

[hgreco85]

Thanks — I found the v1.1.0 fix work in actava-ai/chi-bench#53. I’ll wait for that release/fix point to land, then rerun only the affected evidence. I’ll also rerun UM P2P t016/t019/t036 with CHI_BENCH_P2P_SIMULATOR_MODEL=claude-sonnet-5 as requested, and repair the Care trials missing the required judge. I’ll preserve unaffected valid trials where the updated submission rules allow it.

[hgreco85]

@haolin-chen-actava Thanks for reviewing #26. I audited all 25 Care scorecards in this packet: exactly one (cm_mdd_hard_refuses_002) has all five judge.cm.* checks marked not_applicable / null despite a recorded binary reward of 1. The other 24 Care trials do contain judge verdicts.

To make the smallest valid correction after the v1.1.0 fixes land, could you please confirm:

1. May we re-judge only the saved final state for that one CM trial (with complete, independently generated judge evidence), without rerunning its agent/patient conversation? We found cb experiment rejudge, but our older scorecard format needs a compatible, auditable conversion.
2. May we rerun only UM P2P t016, t019, t036 using CHI_BENCH_P2P_SIMULATOR_MODEL=claude-sonnet-5, then replace those exact trial packets within this same PR while preserving the other 71 original trials and their unmodified results?
3. Or does acceptance require rerunning all 75 tasks on v1.1.0?

We will preserve original provenance and verifier artifacts, will not manually change judge verdicts/rewards, and will wait for the stable release before any paid rerun. Thank you!

[haolin-chen-actava]

Hi @hgreco85, please update your repo with latest commits. And rerun only those specific tasks/judges according to https://github.com/actava-ai/chi-bench/blob/main/CHANGELOG.md.

[hgreco85]

@haolin-chen-actava — the four requested repairs are now complete and the existing PR branch has been updated in place.

• Care cm_mdd_hard_refuses_002: judge-only re-evaluation from the byte-preserved saved final state on CHI-Bench v1.1.0 (c05b225b...), one claude-opus-4-7 vote. Valid result: 0/5 judge rubrics, so the historical pass is corrected to fail. Run: 37625832254.
• UM P2P t016, t019, t036: exact v1.1.0 reruns using the historical Full75 HG agent snapshot and CHI_BENCH_P2P_SIMULATOR_MODEL=claude-sonnet-5. All three remain 1.0/1.0, with all 7 P2P judge checks true for each task. Run: 37628570479.
• The other 71 trial artifacts remain unchanged.
• Repaired aggregate: Provider 11/25, Payer 7/25, Care 2/25, overall 20/75 = 26.67%.
• Added repair_evidence.json with run IDs, artifact digests and task-level hashes.

The current validation workflow shows action_required because this is a fork PR and needs base-repo workflow approval; there is no observed schema failure yet.

[hgreco85]

Additional validation evidence: because the base-repo Actions run is still action_required for this fork PR, I ran the same official .github/scripts/validate_submission.py against the repaired packet in my fork.

✅ All checks passed.

• Validation run: 37640070034
• Report artifact: 11491358014
• Artifact digest: sha256:870e99e71ee371bbe4913bf5aefc8342a0eaeda49e00d3de66031aa722930a47
• Packet head validated: c2a846c469fe56a00afad50af4949d4c0000e2be

So the repaired packet is locally clean under the leaderboard's official validator; the remaining gate is base-repo approval/review of the fork workflow.