Add medication safety benchmark

[johann-UMR]

Summary

• add an EMA-grounded medication safety benchmark with 20 cases, four system prompts, four patient attitudes, eight response models, and four iterations (10,240 responses)
• integrate the benchmark with BioChatter's public dataset loader, conversation interface, pytest model matrix, and standard response schema
• provide command-line runners for response generation and the calibrated subcriterion-based judge through BioChatter providers
• extend the term-matching scorer to accept standard BioChatter response files while preserving simple long-form CSV inputs
• use the final GPT-5.4 temperature-0 corpus and a calibrated DeepSeek V4 Flash primary judge that scores five explicit subcriteria
• include conservative canonical term matching, final structured and communication summaries, inferential outputs, aggregate manual validation, temperature sensitivity, and figure source data
• retain alternative-judge sensitivity results for the common 5,120-response, four-model corpus and document its direct binary judging protocol separately
• exclude raw model responses, raw judge outputs, row-level manual labels, provider logs, credentials, and local paths

BioChatter integration

The 20 base cases are loaded by the standard benchmark loader and expanded to 320 medication case, system prompt, and patient attitude combinations. The native pytest entry point executes these instances through BioChatter conversations with four iterations by default. Separate provider-aware commands support models outside the central benchmark matrix, calibrated repeated judging, resumable local output, and term matching of BioChatter response files. Local raw response and judgement files are ignored by Git.

Reproducibility

The public package contains benchmark definitions, exact prompts, model settings, scoring and analysis scripts, derived response-level numeric scores, aggregate outputs, validation reports, checksums, and figure source data. Full inference recomputation requires access to the listed provider snapshots and settings. Raw model outputs and raw judge outputs are intentionally excluded.

The reusable implementation and auditable derived results are proposed here for BioChatter. A separate paper repository and Zenodo snapshot can preserve the final publication-specific artifact once the manuscript is frozen.

Validation

• python -m pytest benchmark/medication_safety/tests -q --confcutdir=benchmark/medication_safety
• 22 tests passed
• native loader validation: 320 unique expanded instances
• final corpus validation: 102,400 unique response-metric rows, eight models, ten metrics, and no duplicate keys
• between-model families regenerated: 1,764 Wilcoxon and 756 McNemar tests
• published response-level scores, summaries, omnibus tests, and figure source data recomputed from released files
• custom statistical implementations reference-checked against SciPy
• new executable files pass Ruff lint and format checks
• privacy checks found no raw responses, row-level manual labels, local paths, or credentials

Review status

The benchmark implementation, released numeric analysis, and BioChatter adapter are synchronized to the final GPT-5.4 temperature-0 and calibrated DeepSeek runs. Manuscript-only wording or layout changes can be handled independently; any later benchmark corrections can still be pushed to this branch before merge.

[slobentanzer]

@johann-UMR feel free to convert this to a PR and tag me for review once you're happy.

[johann-UMR]

@slobentanzer The PR is now ready for review. I have added native BioChatter integration (dataset loading, Conversation-based response generation, the pytest model matrix, a calibrated judge runner, and the standard response schema), synchronized the released analyses, and expanded validation to 22 passing benchmark tests. Raw model and judge outputs and row-level manual labels remain excluded. Thanks for taking a look.