test(rr_parity): retire three override keys that could never match, and assert it (#482) - #484
Merged
Merged
Conversation
…nd assert it (#482) `stale_keys` reports three ODE override keys naming models the manifest does not build: MODEL0913003363:ode (TOL_OVERRIDES), MODEL1006230083:ode and MODEL6963432821:ode (both NO_ORACLE_ADJUDICATED). SSA is clean. All three are `drop,no_model` in the stage-1 corpus filter (benchmarks/suites/biomodels/manifest.csv), their fetched files zero bytes -- the recorded sha256 is e3b0c442...b855, the empty string's -- one of 305 corpus entries in that state. So build_ode_jobs.py builds no job for them and the keys cannot match. Checked before removing, because the other thing a stale key can be is a typo'd or renamed model_id, where deleting the entry would quietly un-mask a real divergence instead of tidying a dead one. These are not that: the ids are in the corpus manifest, explicitly dropped for want of a file. They have been stale since the initial public release (no commit ever added them to ode_jobs.json). Each is retired as a comment rather than deleted, carrying its disposition forward -- the tolerance rationale for MODEL0913003363, the GH #117 scipy adjudications for the other two -- so none of that is re-derived if the corpus ever fetches the model successfully. Same treatment MODEL2205030001:ode got when GH #119 retired it. The reason three accumulated unnoticed is that nothing fails on them: the only detection is a `print("WARN: ...")` at the end of build_ode_jobs.py / build_ssa_jobs.py, which is advisory, and which cannot run at all in a checkout without $BIOMODELS_SEDML_DIR. So the cleanup comes with a guard -- test_committed_overrides_have_no_stale_keys asserts the committed overrides against the committed manifests for both regimes, needing neither the mirror nor the build. Verified to fail on an injected stale key rather than passing vacuously. No job's disposition moves: none of these keys was reaching a job to begin with. parity_checks/tests: 481 passed, 17 skipped.
wshlavacek
force-pushed
the
parity/retire-stale-override-keys
branch
from
August 25, 2026 17:37
663bfd0 to
1625b89
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #482 / #483, independent of both — branched off
mainand mergeableon its own.
The three keys
stale_keysreports three ODE override keys naming models the manifest does notbuild. SSA is clean.
MODEL0913003363:odeTOL_OVERRIDESdrop,no_modelMODEL1006230083:odeNO_ORACLE_ADJUDICATEDdrop,no_modelMODEL6963432821:odeNO_ORACLE_ADJUDICATEDdrop,no_modelAll three are dropped by the stage-1 corpus filter
(
benchmarks/suites/biomodels/manifest.csv) for want of a file: the recordedsha256 is
e3b0c442...b855, the empty string's, so the fetch produced zerobytes. They are 3 of 305 corpus entries in that state.
build_ode_jobs.pytherefore builds no job for them and the keys cannot match. No commit ever added
them to
ode_jobs.json, so they have been stale since the initial publicrelease.
I checked that before removing anything, because the other thing a stale key
can be is a typo'd or renamed
model_id— where deleting the entry would quietlyun-mask a real divergence rather than tidy a dead one. These are not that: the
ids appear in the corpus manifest, explicitly dropped.
Retired, not deleted
Each becomes a comment carrying its disposition forward — the tolerance rationale
for
MODEL0913003363, the GH #117 scipy adjudications for the other two — sonone of it is re-derived if the corpus ever fetches the model successfully. Same
treatment
MODEL2205030001:odegot when GH #119 retired it.Why three accumulated unnoticed
Nothing fails on them. The only detection is a
print("WARN: ...")at the end ofbuild_ode_jobs.py/build_ssa_jobs.py— advisory, and unreachable in acheckout without
$BIOMODELS_SEDML_DIR.So the cleanup comes with a guard:
test_committed_overrides_have_no_stale_keysasserts the committed overrides against the committed manifests for both regimes,
needing neither the mirror nor the build. I verified it fails on an injected
stale key rather than passing vacuously.
Scope
Suite-side only; no bngsim behaviour changes, and no job's disposition moves —
none of these keys was reaching a job to begin with.
parity_checks/tests: 481 passed, 17 skipped. ruff clean.