Skip to content

test(rr_parity): retire three override keys that could never match, and assert it (#482) - #484

Merged
wshlavacek merged 1 commit into
mainfrom
parity/retire-stale-override-keys
Aug 25, 2026
Merged

test(rr_parity): retire three override keys that could never match, and assert it (#482)#484
wshlavacek merged 1 commit into
mainfrom
parity/retire-stale-override-keys

Conversation

@wshlavacek

Copy link
Copy Markdown
Collaborator

Follow-up to #482 / #483, independent of both — branched off main and mergeable
on its own.

The three keys

stale_keys reports three ODE override keys naming models the manifest does not
build. SSA is clean.

key dict corpus status
MODEL0913003363:ode TOL_OVERRIDES drop,no_model
MODEL1006230083:ode NO_ORACLE_ADJUDICATED drop,no_model
MODEL6963432821:ode NO_ORACLE_ADJUDICATED drop,no_model

All three are dropped by the stage-1 corpus filter
(benchmarks/suites/biomodels/manifest.csv) for want of a file: the recorded
sha256 is e3b0c442...b855, the empty string's, so the fetch produced zero
bytes. They are 3 of 305 corpus entries in that state. build_ode_jobs.py
therefore builds no job for them and the keys cannot match. No commit ever added
them to ode_jobs.json, so they have been stale since the initial public
release.

I checked that before removing anything, because the other thing a stale key
can be is a typo'd or renamed model_id — where deleting the entry would quietly
un-mask a real divergence rather than tidy a dead one. These are not that: the
ids appear in the corpus manifest, explicitly dropped.

Retired, not deleted

Each becomes a comment carrying its disposition forward — the tolerance rationale
for MODEL0913003363, the GH #117 scipy adjudications for the other two — so
none of it is re-derived if the corpus ever fetches the model successfully. Same
treatment MODEL2205030001:ode got when GH #119 retired it.

Why three accumulated unnoticed

Nothing fails on them. The only detection is a print("WARN: ...") at the end of
build_ode_jobs.py / build_ssa_jobs.py — advisory, and unreachable in a
checkout without $BIOMODELS_SEDML_DIR.

So the cleanup comes with a guard: test_committed_overrides_have_no_stale_keys
asserts the committed overrides against the committed manifests for both regimes,
needing neither the mirror nor the build. I verified it fails on an injected
stale key rather than passing vacuously.

Scope

Suite-side only; no bngsim behaviour changes, and no job's disposition moves —
none of these keys was reaching a job to begin with.

parity_checks/tests: 481 passed, 17 skipped. ruff clean.

…nd assert it (#482)

`stale_keys` reports three ODE override keys naming models the manifest does not
build: MODEL0913003363:ode (TOL_OVERRIDES), MODEL1006230083:ode and
MODEL6963432821:ode (both NO_ORACLE_ADJUDICATED). SSA is clean.

All three are `drop,no_model` in the stage-1 corpus filter
(benchmarks/suites/biomodels/manifest.csv), their fetched files zero bytes --
the recorded sha256 is e3b0c442...b855, the empty string's -- one of 305 corpus
entries in that state. So build_ode_jobs.py builds no job for them and the keys
cannot match. Checked before removing, because the other thing a stale key can
be is a typo'd or renamed model_id, where deleting the entry would quietly
un-mask a real divergence instead of tidying a dead one. These are not that: the
ids are in the corpus manifest, explicitly dropped for want of a file. They have
been stale since the initial public release (no commit ever added them to
ode_jobs.json).

Each is retired as a comment rather than deleted, carrying its disposition
forward -- the tolerance rationale for MODEL0913003363, the GH #117 scipy
adjudications for the other two -- so none of that is re-derived if the corpus
ever fetches the model successfully. Same treatment MODEL2205030001:ode got when
GH #119 retired it.

The reason three accumulated unnoticed is that nothing fails on them: the only
detection is a `print("WARN: ...")` at the end of build_ode_jobs.py /
build_ssa_jobs.py, which is advisory, and which cannot run at all in a checkout
without $BIOMODELS_SEDML_DIR. So the cleanup comes with a guard --
test_committed_overrides_have_no_stale_keys asserts the committed overrides
against the committed manifests for both regimes, needing neither the mirror nor
the build. Verified to fail on an injected stale key rather than passing
vacuously.

No job's disposition moves: none of these keys was reaching a job to begin with.

parity_checks/tests: 481 passed, 17 skipped.
@wshlavacek
wshlavacek force-pushed the parity/retire-stale-override-keys branch from 663bfd0 to 1625b89 Compare August 25, 2026 17:37
@wshlavacek
wshlavacek merged commit d4fcb4e into main Aug 25, 2026
4 checks passed
@wshlavacek
wshlavacek deleted the parity/retire-stale-override-keys branch August 25, 2026 18:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant