test(rr_parity): MODEL2002070001's invalid-reference premise keys on the reference, not on bngsim (#485)

CHANGELOG.md

@@ -16,6 +16,66 @@ in `CMakeLists.txt`) is derived from it.
 
 ### Changed
 
+- **`MODEL2002070001`'s `INVALID_REFERENCE` premise is relaxed: a reference that
+  answers `NaN` is not a comparison basis, whatever bngsim did (#485).** The
+  0.15.0 sweep flagged the entry stale — `natural=DIFF, rr_finite=False` — and it
+  was half right. The entry claimed *both* engines were broken; bngsim's half of
+  that stopped being true in 0.14.0, when the #353 unit-volume substitution for a
+  non-finite compartment size let it integrate the model. RoadRunner's half is
+  unchanged. So the natural verdict resurfaced as a `DIFF` with `max_rel_err`
+  `inf`, 606/707 cells failing — RoadRunner's `NaN` scored against bngsim as a
+  divergence, the one thing it is not.
+
+  The model is well posed and the `NaN` is not in its dynamics. Both compartments
+  declare `size="NaN"`, but every species is `hasOnlySubstanceUnits="true"`, the
+  dynamics are 7 rate rules with no reaction, and no `<math>` block in the file
+  names `mucosa` or `lumen` — nothing reads a size, so the IVP is well posed in
+  amounts and bngsim's substitution warning describes a value nothing reads.
+  RoadRunner carries species as concentrations, so `x1..x6` — the six species in
+  those compartments — are `NaN` from `t = 0`, while `x7`, the one rate-rule
+  variable that is a *parameter* rather than a species, stays finite and agrees
+  with bngsim cell for cell. The `NaN` reaches RoadRunner's state through the
+  compartment, not through the dynamics.
+
+  At the sweep tolerance (rtol 1e-9 / atol 1e-12, `t` = 0..100, 101 points):
+  bngsim 707/707 cells finite; RoadRunner 2.9.2 runs without raising and returns
+  101/707; AMICI 1.0.1 confirms bngsim's trajectory independently, 0/707 cells
+  failing at `max_rel_err` 0.0. Two engines produce *a* trajectory here and the
+  third produces `NaN`, which is all the override needs to claim: RoadRunner's
+  output is not a usable comparison basis.
+
+  `_apply_invalid_reference` now keys the premise on the reference alone — RR ran
+  and emitted non-finite output — instead of also requiring that bngsim failed.
+  What that requirement was really doing was guarding against burying a bngsim
+  defect, and a model bngsim can now integrate needs a guard that still works:
+  `_reference_nonfinite_covers` re-runs the same verdict over just the columns
+  the reference kept finite, and the override holds on a `DIFF` only if that
+  passes. A real bngsim divergence beside the `NaN` leaves a failing cell in a
+  finite reference column, and the row stays a `DIFF`. Restricting the columns
+  can only tighten the re-run (`scale` is the peak over the retained block), so
+  the claim it licenses is never a loosening.
+
+  Two shapes the relaxation deliberately does **not** cover. `rr_finite is None`
+  — RoadRunner raised, or a segfaulted child left no per-engine status — stays
+  stale on anything but a natural `BAD_TEST`: "the reference raised" and "we
+  never learned" are indistinguishable there, and `REFERENCE_FAILED` is already
+  the right bucket for the first. A natural `PASS` stays stale too: if the
+  compared columns agreed, the reference was usable for the comparison whatever
+  it did elsewhere.
+
+  The row lands on `BAD_TEST`, not `PASS`. `PASS` is the #482/#483 treatment for
+  a divergence attributable to the reference engine, and it would score
+  RoadRunner as having *validated* a model it answered `NaN` on. `BAD_TEST` says
+  what happened: the model's own SBML put a `NaN` in the reference's state, and
+  no comparison exists. Both are non-scoring, so neither costs bngsim a model.
+
+  Suite-side only — no bngsim behavior changes. `overrides.py` rewrites the entry
+  (now citing #485, the AMICI confirmation, and the `x7` control),
+  `ode_jobs.json` re-bakes the one job record, and `test_outcome_classify.py`
+  pins the relaxed premise and the new guard. This was the last `DIFF` in the ODE
+  sweep: 1237 `PASS` / 27 `REFERENCE_FAILED` / 54 `BAD_TEST` / 5 `TIMEOUT` over
+  1323 jobs, no `DIFF` and no `EXCEPTION` row, and no override flagged stale.
+
 - **BIOMD0000000627's rr_parity divergence is allow-listed as a `KNOWN_ARTIFACT`:
   RoadRunner steps over a smooth pre-stimulus ramp, and bngsim is the correct
   engine (#482).** Reported as a 0.13.0 regression on the premise that the

parity_checks/amici_parity/README.md

@@ -140,9 +140,13 @@ Two dispositions, and the difference is which engine (if either) is at fault:
 agreed and they did not — this is the mirror of what #319/#323 did for bngsim's
 own declared refusals (`UNSUPPORTED` rather than a silent `EXCEPTION`): name the
 gap, do not claim the win. `REFERENCE_FAILED` and **not** `BAD_TEST`, because
-`BAD_TEST` means neither engine could run the model, and here bngsim did.
-(rr_parity's `invalid_reference` lands on `BAD_TEST` for exactly the reason it
-does not apply here: *there*, bngsim had also failed.) `COMPARISON_ARTIFACT` is
+`BAD_TEST` means the *model or test* is the problem, and here it is the reference
+engine: AMICI returns finite numbers that a defect of its own makes wrong.
+(rr_parity's `invalid_reference` lands on `BAD_TEST` because what it covers is
+the other case — a reference that answers `NaN` on a model whose own SBML is
+what put the `NaN` there. It no longer requires bngsim to have failed too, since
+GH #485; the guard against burying a bngsim defect is now that every surviving
+failing cell must sit in a column the reference left non-finite.) `COMPARISON_ARTIFACT` is
 correspondingly **narrower** than rr_parity's `KNOWN_ARTIFACT`, which also covers
 "the reference engine has the bug" — that case is `INVALID_REFERENCE` here,
 precisely so a reference defect is never dressed up as agreement.

parity_checks/rr_parity/ode_jobs.json

@@ -1359,7 +1359,7 @@
 {"model_id":"MODEL2001200002","input_format":"sbml","method":"ode","reference_engine":"roadrunner","model":"models/MODEL2001200002/MODEL2001200002.xml","oracle":{"metric":"max_rel_err","tol":0.0001},"params":{"sbml_origin":"biomodels_dir","sedml":null,"horizon_source":"invented","initial_time":0.0,"t_start":0.0,"t_end":100.0,"n_points":101,"rtol":1e-06,"atol":1e-09},"overrides":[{"field":"no_oracle_adjudicated","value":{"issue":"GH #117","verdict":"confirm"},"reason":"RR CVODE CV_CONV_FAILURE + COPASI failed. 7-species reaction net with pow(L/T,1.36) Hill terms; species underflow (1e-33..1e-41) makes a stiff solver's trial step overshoot negative -> pow(neg,1.36)=NaN. With species floored at 1e-100 in the RHS (60 orders below the smallest physical value), all three independent solvers (Radau, BDF, LSODA) agree with bngsim to max_rel=0 (peak-rel ~2e-7). bngsim confirmed."}],"notes":"invented horizon (placeholder)"},
 {"model_id":"MODEL2001310001","input_format":"sbml","method":"ode","reference_engine":"roadrunner","model":"models/MODEL2001310001/MODEL2001310001.xml","oracle":{"metric":"max_rel_err","tol":0.0001},"params":{"sbml_origin":"biomodels_dir","sedml":null,"horizon_source":"invented","initial_time":0.0,"t_start":0.0,"t_end":100.0,"n_points":101,"rtol":1e-06,"atol":1e-09},"overrides":[],"notes":"invented horizon (placeholder)"},
 {"model_id":"MODEL2002030001","input_format":"sbml","method":"ode","reference_engine":"roadrunner","model":"models/MODEL2002030001/MODEL2002030001.xml","oracle":{"metric":"max_rel_err","tol":0.0001},"params":{"sbml_origin":"biomodels_dir","sedml":null,"horizon_source":"invented","initial_time":0.0,"t_start":0.0,"t_end":100.0,"n_points":101,"rtol":1e-06,"atol":1e-09},"overrides":[],"notes":"invented horizon (placeholder)"},
-{"model_id":"MODEL2002070001","input_format":"sbml","method":"ode","reference_engine":"roadrunner","model":"models/MODEL2002070001/MODEL2002070001.xml","oracle":{"metric":"max_rel_err","tol":0.0001},"params":{"sbml_origin":"biomodels_dir","sedml":null,"horizon_source":"invented","initial_time":0.0,"t_start":0.0,"t_end":100.0,"n_points":101,"rtol":1e-06,"atol":1e-09},"overrides":[{"field":"invalid_reference","value":{"issue":null},"reason":"No valid reference: RoadRunner runs without raising but its trajectory is non-finite over the horizon (101/707 = 14.3% finite cells, verified 2026-06-03), so it is not a usable oracle. bngsim also fails to integrate this IVP \u2014 CVODE flag=-4 (error test failed repeatedly / |h|=hmin) at t~=0.0134. Neither engine produces a usable result, so this is a BAD_TEST (both broken), not the EXCEPTION the raised-vs-not taxonomy would record (which would mislabel a non-finite RR run as an actionable bngsim bug). If RR ever returns a finite trajectory here the override goes stale and the natural verdict \u2014 including any real bngsim bug \u2014 resurfaces."}],"notes":"invented horizon (placeholder)"},
+{"model_id":"MODEL2002070001","input_format":"sbml","method":"ode","reference_engine":"roadrunner","model":"models/MODEL2002070001/MODEL2002070001.xml","oracle":{"metric":"max_rel_err","tol":0.0001},"params":{"sbml_origin":"biomodels_dir","sedml":null,"horizon_source":"invented","initial_time":0.0,"t_start":0.0,"t_end":100.0,"n_points":101,"rtol":1e-06,"atol":1e-09},"overrides":[{"field":"invalid_reference","value":{"issue":485},"reason":"No valid reference: RoadRunner runs without raising but its trajectory is non-finite over the horizon (101/707 = 14.3% finite cells, re-verified 2026-08-25 against RR 2.9.2), so it is not a usable oracle. Both compartments declare size=\"NaN\"; every species is hasOnlySubstanceUnits=true and no <math> block in the file names either compartment, so the IVP is well posed in amounts and the size is inert \u2014 but RR carries species as concentrations, so x1..x6 (the species in those compartments) are NaN from t=0 while x7, the one rate-rule variable that is not a species, stays finite and agrees with bngsim cell for cell. The NaN reaches RR's state through the compartment, not through the dynamics. bngsim used to fail here too (CVODE flag=-4, error test failed repeatedly / |h|=hmin, at t~=0.0134); since the #353 unit-volume substitution for a non-finite size it integrates the model (707/707 finite) and AMICI 1.0.1 confirms that trajectory independently \u2014 0/707 cells failing, max_rel_err 0.0 at the sweep tolerance. So the half of this entry's original premise that read \"bngsim also failed\" is gone, and the half that matters is not: there is still no reference trajectory to compare against, which is BAD_TEST (no parity signal) rather than the DIFF the taxonomy would record \u2014 a DIFF would score RR's NaN as a bngsim divergence. NOT reclassified to PASS, which would credit RR with validating a model it answered NaN on. If RR ever returns a finite trajectory here, or any surviving failure lands in a column RR kept finite, the override goes stale and the natural verdict \u2014 including any real bngsim bug \u2014 resurfaces."}],"notes":"invented horizon (placeholder)"},
 {"model_id":"MODEL2003030002","input_format":"sbml","method":"ode","reference_engine":"roadrunner","model":"models/MODEL2003030002/MODEL2003030002.xml","oracle":{"metric":"max_rel_err","tol":0.0001},"params":{"sbml_origin":"biomodels_dir","sedml":null,"horizon_source":"invented","initial_time":0.0,"t_start":0.0,"t_end":100.0,"n_points":101,"rtol":1e-06,"atol":1e-09},"overrides":[],"notes":"invented horizon (placeholder)"},
 {"model_id":"MODEL2003030003","input_format":"sbml","method":"ode","reference_engine":"roadrunner","model":"models/MODEL2003030003/MODEL2003030003.xml","oracle":{"metric":"max_rel_err","tol":0.0001},"params":{"sbml_origin":"biomodels_dir","sedml":null,"horizon_source":"invented","initial_time":0.0,"t_start":0.0,"t_end":100.0,"n_points":101,"rtol":1e-06,"atol":1e-09},"overrides":[],"notes":"invented horizon (placeholder)"},
 {"model_id":"MODEL2003040001","input_format":"sbml","method":"ode","reference_engine":"roadrunner","model":"models/MODEL2003040001/MODEL2003040001.xml","oracle":{"metric":"max_rel_err","tol":0.0001},"params":{"sbml_origin":"biomodels_dir","sedml":null,"horizon_source":"invented","initial_time":0.0,"t_start":0.0,"t_end":100.0,"n_points":101,"rtol":1e-06,"atol":1e-09},"overrides":[],"notes":"invented horizon (placeholder)"},

parity_checks/rr_parity/overrides.py

@@ -20,16 +20,21 @@
                     non-finite output (NaN/Inf) over the horizon, so there is no
                     valid reference to compare against. The auto-derived taxonomy
                     keys only on raised-vs-not, so such a job lands in EXCEPTION
-                    (mislabeling it an actionable bngsim bug) when bngsim also
-                    failed; this narrow human disposition reclassifies it to
-                    ``Outcome.BAD_TEST`` (no parity signal). The runner applies it
-                    only while the premise holds — bngsim failed AND RR's output
-                    is non-finite — and otherwise flags the entry STALE (recovery
-                    stays visible; it can never silently mask a real bngsim bug).
-                    ODE-only in practice. Each entry MUST cite evidence (the RR
-                    finite fraction + the bngsim failure mode). The bar is the
-                    same as KNOWN_ARTIFACT: this is *not* a place to bury a real
-                    bngsim defect — a finite RR reference makes it stale at once.
+                    (mislabeling it an actionable bngsim bug) or, once bngsim can
+                    integrate the model, in DIFF (scoring the NaN against bngsim
+                    as a divergence); this narrow human disposition reclassifies
+                    it to ``Outcome.BAD_TEST`` (no parity signal). The runner
+                    applies it only while the premise holds — RR's output is
+                    non-finite, and on a DIFF that non-finite output accounts for
+                    the whole divergence (#485) — and otherwise flags the entry
+                    STALE (recovery stays visible; it can never silently mask a
+                    real bngsim bug). ODE-only in practice. Each entry MUST cite
+                    evidence (the RR finite fraction, and what settles bngsim's
+                    own trajectory: its failure mode, or an independent oracle).
+                    The bar is the same as KNOWN_ARTIFACT: this is *not* a place
+                    to bury a real bngsim defect — a finite RR reference makes it
+                    stale at once, and so does a failing cell in any column RR
+                    kept finite.
 
   NO_ORACLE_ADJUDICATED — a REFERENCE_FAILED row (bngsim ran, RoadRunner refused)
                     whose correctness blind spot was closed by an INDEPENDENT
@@ -283,23 +288,39 @@
 # --------------------------------------------------------------------------- #
 # Reference engine ran but produced no usable trajectory (non-finite output) ->
 # reclassified to BAD_TEST. Keyed "model_id:method". `issue` is optional.
-# Applied only while the premise holds (bngsim failed AND RR non-finite); a
-# finite RR reference or a now-running bngsim makes the entry STALE.
+# Applied only while the premise holds (RR non-finite, and on a DIFF only where
+# the non-finite columns account for every surviving failure); a finite RR
+# reference, or a failure in a column RR kept finite, makes the entry STALE.
 # --------------------------------------------------------------------------- #
 INVALID_REFERENCE: dict[str, dict] = {
     "MODEL2002070001:ode": {
-        "issue": None,
+        "issue": 485,
         "reason": (
             "No valid reference: RoadRunner runs without raising but its "
             "trajectory is non-finite over the horizon (101/707 = 14.3% finite "
-            "cells, verified 2026-06-03), so it is not a usable oracle. bngsim "
-            "also fails to integrate this IVP — CVODE flag=-4 (error test failed "
-            "repeatedly / |h|=hmin) at t~=0.0134. Neither engine produces a "
-            "usable result, so this is a BAD_TEST (both broken), not the "
-            "EXCEPTION the raised-vs-not taxonomy would record (which would "
-            "mislabel a non-finite RR run as an actionable bngsim bug). If RR "
-            "ever returns a finite trajectory here the override goes stale and "
-            "the natural verdict — including any real bngsim bug — resurfaces."
+            "cells, re-verified 2026-08-25 against RR 2.9.2), so it is not a "
+            'usable oracle. Both compartments declare size="NaN"; every '
+            "species is hasOnlySubstanceUnits=true and no <math> block in the "
+            "file names either compartment, so the IVP is well posed in amounts "
+            "and the size is inert — but RR carries species as concentrations, "
+            "so x1..x6 (the species in those compartments) are NaN from t=0 "
+            "while x7, the one rate-rule variable that is not a species, stays "
+            "finite and agrees with bngsim cell for cell. The NaN reaches RR's "
+            "state through the compartment, not through the dynamics. bngsim "
+            "used to fail here too (CVODE flag=-4, error test failed repeatedly "
+            "/ |h|=hmin, at t~=0.0134); since the #353 unit-volume substitution "
+            "for a non-finite size it integrates the model (707/707 finite) and "
+            "AMICI 1.0.1 confirms that trajectory independently — 0/707 cells "
+            "failing, max_rel_err 0.0 at the sweep tolerance. So the half of "
+            'this entry\'s original premise that read "bngsim also failed" is '
+            "gone, and the half that matters is not: there is still no reference "
+            "trajectory to compare against, which is BAD_TEST (no parity signal) "
+            "rather than the DIFF the taxonomy would record — a DIFF would score "
+            "RR's NaN as a bngsim divergence. NOT reclassified to PASS, which "
+            "would credit RR with validating a model it answered NaN on. If RR "
+            "ever returns a finite trajectory here, or any surviving failure "
+            "lands in a column RR kept finite, the override goes stale and the "
+            "natural verdict — including any real bngsim bug — resurfaces."
         ),
     },
 }