validation.md

Tool validation

Stability: Draft Last reviewed: 2026-10-27

What MedHarness is trusted to do, how much a wrong answer matters for each command, and the requirements and tests that back the commands rated high and medium. It is the start of iteration 4 of the charter.

ISO 13485 §4.1.6 asks a manufacturer to validate software used in its quality system in proportion to the risk, and IEC/TR 80002-2 describes how: state what the tool must do, rate the risk of it doing that wrongly, and keep evidence that it does it. This page does the first two for MedHarness; the repository's own DHF holds the requirements and the tests are the evidence.

Intended use

MedHarness checks the design-control evidence of a software change and produces the release's evidence from it. A verify command judges what is in the repository and on the machine and answers pass or fail; build release and build approvals write what a release is shown by; item records the DHF. It does not decide whether a requirement, a risk conclusion or a change is right, it is not an eQMS, and it does not replace the review of a pull request or the signatures of a release. The charter states the promises and the non-goals.

build plan and build code run an AI agent. They propose; the deterministic commands check what they propose, a person reviews it, and nothing relies on them being right. Their boundary is in ai-security.md.

Risk of each command

The risk is what a wrong pass would cause: a command that fails something sound costs time and is noticed at once; one that passes something unsound can reach a release unseen.

Risk	Commands	A wrong pass means
High	verify completion, verify tests, verify approval --release, build release, verify impact, verify changes, verify dhf	unverified, unapproved or untraced content is released, or an omission is hidden
Medium	verify soup, item transition, item create / update, verify approval without --release, build approvals, build soup, build validation	wrong data or a missed vulnerability, which a later gate checks again; for build validation, a user's validation evidence that overstates, while the release's own suite still verifies every gate
Low	item get / list, init	nothing evidence depends on
Out of scope	build plan, build code	AI proposals, not relied on

MedHarness's own DHF

MedHarness keeps its requirements the way it asks its users to: as items in a DHF, here DHF/ at the root of this repository, checked by MedHarness itself. A tool's requirements are software requirements, so the DHF holds one item type, the SRS, and nothing else: omit_doc_types in DHF/config/global.yaml leaves out the others, and with no other type there is no chain for coverage and no link to require (traceability_matrices: [], required_traceability: []; see configuration).

Its SRS type is the shipped one with a risk property in place of derives_from (DHF/config/doc_types/srs.yaml). risk is what a wrong pass of that behaviour would cause, rated high, medium or low by the table above. Each item states one behaviour of a gate, what it must pass and what it must stop, with verification_method: [Test], measurable verification_criteria and numbered test points T1, T2, … in testing, one for each distinct case.

The tests that verify an item carry its ID and the point they cover:

@pytest.mark.dhf_links("SRS-016")
@pytest.mark.dhf_testing("T2")
def test_a_signature_by_a_role_that_does_not_own_the_document_does_not_count(project): ...

The dhfkit pytest plugin writes both markers into the JUnit XML, and verify tests reads them. To check the whole thing locally:

pytest dhfkit/tests/ tests/ -q --junitxml=test-results/junit.xml
medharness --dhf DHF verify dhf --strict
medharness --dhf DHF verify tests --strict --junit test-results/junit.xml

CI runs exactly this in the job Check | Own DHF. The DHF is not part of the package: the wheel and the source distribution contain only medharness and dhfkit. To change a requirement, use medharness --dhf DHF item create|update, never the files; a change to behaviour changes its item and its test points in the same pull request.

What is covered

Command	Items	State
verify completion	SRS-002 to SRS-006	covered
verify impact	SRS-007 to SRS-015	covered
verify approval --release (and the approval log and approvers list it relies on)	SRS-016 to SRS-023	covered
verify tests	SRS-024 to SRS-029	covered
verify changes	SRS-030 to SRS-034	covered
verify dhf	SRS-035 to SRS-042	covered
build release	SRS-043 to SRS-050	covered
verify soup	SRS-051 to SRS-054	covered
item create / update	SRS-055 to SRS-059	covered
item transition	SRS-060 to SRS-062	covered
verify approval without --release	SRS-063 to SRS-065, with SRS-019, SRS-020 and SRS-022 for the log and the approvers list	covered
build approvals	SRS-066 and SRS-067	covered
build soup	SRS-068 to SRS-070	covered
build validation	SRS-071 to SRS-074	covered
Low risk commands	none	rated, not specified: nothing evidence depends on them

"Covered" means every test point of every item has at least one passing test, which verify tests --strict enforces. It does not mean the requirements are complete for the command: an omission a requirement does not name is not tested. All seven high-risk commands and all seven medium-risk ones are covered. The low-risk commands are rated and not specified.

The requirements were derived from what the commands do today (interface, the gates' docstrings and the tests that already existed), and the tests were annotated, not rewritten. Where a case was untested, a test was added.

Validation where it is used

The suite above is the vendor's verification: it runs in this repository, with pytest and a checkout. medharness build validation is what a user runs where MedHarness is installed (ADR-005, adopting). It runs the validation set in medharness/validation/, which ships in the wheel: each case runs the CLI against a scratch project in a temporary directory and checks test points of a high-risk requirement. A case checks with check(), never assert, which python -O would remove. The set covers the gates, not build validation itself, which cannot be validated by the set it runs; the suite verifies it (SRS-071 to SRS-074).

The set exercises every test point of the high-risk requirements, 171 points, each through the CLI:

Module	What it covers	Points
medharness/validation/seeded.py	The seeded-omission project: the honest change passes every gate, and each omission is stopped by the gate whose job it is	SRS-007 T4, SRS-008 T2, SRS-009 T2, SRS-010 T4, SRS-011 T3, SRS-012 T3, SRS-030 T4, SRS-031 T2, SRS-034 T3
medharness/validation/completion.py	verify completion, every point	SRS-002 to SRS-006 (22)
medharness/validation/impact.py	verify impact, every point the seeded project does not cover	SRS-007 to SRS-015 (22)
medharness/validation/approval.py	verify approval, every point, with throwaway SSH keys (medharness/validation/signing.py); needs ssh-keygen	SRS-016 to SRS-023 (31)
medharness/validation/verification.py	verify tests, every point	SRS-024 to SRS-029 (20)
medharness/validation/changes.py	verify changes, every point the seeded project does not cover; a store that keeps no item files is an adapter installed for the case through a real entry point	SRS-030 to SRS-034 (13)
medharness/validation/structure.py	verify dhf, every point; a check that could not run is shown by making the store fail after its first read	SRS-035 to SRS-042 (27)
medharness/validation/release.py	build release, every point	SRS-043 to SRS-050 (26)

Each case is checked against a broken gate as well as a sound one: run against a gate that always passes, every case that expects a refusal fails, and against one that always refuses, every case that expects a pass fails.

A run takes about two minutes one case after another, and about half a minute on four cores: --jobs N runs the cases N at a time (default: the number of CPUs, at most 8), each in a worker process that finds its case by name in the shipped set, and reports the same result for every point (SRS-074). Threads would not do: the cases patch process-wide state, and the in-process CLI runner swaps sys.stdout. tests/integration/test_the_validation_set_passes.py runs every case under pytest with its dhf_links and dhf_testing, so the release's JUnit contains each validation point. tests/integration/test_a_released_wheel_validates_itself.py builds a wheel as the release does and runs the command from it.

The release writes release.json into the wheel (scripts/write_release_record.py): its version and commit, the names of its evidence files, the dependency versions of medharness-<version>-tested-with.txt, and the text of each high-risk requirement. The command compares the installation against it without downloading anything.