Skip to content

Commit 46de0fa

Browse files
committed
fix(test-harness): surface PR test failures whose main build failed
A PR run whose main counterpart failed to build was paired for the build comparison, but main's test count was never measured, so the PR's test failures were skipped without appearing anywhere in the summary. #715 hit this: main could not build swap-vm, the PR could, and the one failing swap-vm test showed up in neither the new, pre-existing nor unbaselined counts. Such runs now count as unbaselined when the PR's tests fail, so their failures reach the "No baseline" line and the table's failure cell. A repaired build whose tests all pass leaves nothing uncompared and stays out of it. The line now says "no `main` baseline", which holds for both an unpaired run and a paired one whose main side never ran its tests.
1 parent 0efe7c8 commit 46de0fa

6 files changed

Lines changed: 31 additions & 5 deletions

File tree

‎solx-benchmark-converter/src/output/summary/suite_stats.rs‎

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -52,7 +52,8 @@ pub struct SuiteStats {
5252
/// PR runs that found a `main` counterpart. The failure verdict only
5353
/// means something when at least one suite compared something.
5454
pub paired_runs: usize,
55-
/// PR runs with no main counterpart, and the failures recorded on them.
55+
/// PR runs with no main counterpart, or with failing tests whose main
56+
/// counterpart never reached its tests, and the failures recorded on them.
5657
/// They have nothing to compare against, so they are surfaced as
5758
/// unbaselined rather than counted as regressions against zero.
5859
pub unbaselined_runs: usize,
@@ -235,6 +236,12 @@ impl SuiteStats {
235236
continue;
236237
};
237238
stats.paired_runs += 1;
239+
if let (None, Some(pr_test_failures @ 1..)) =
240+
(main.test_failures_count(), pr.test_failures_count())
241+
{
242+
stats.unbaselined_runs += 1;
243+
stats.unbaselined_failures += pr_test_failures;
244+
}
238245
stats.baseline_build_failures += main.build_failures_count().unwrap_or_default();
239246
stats.baseline_test_failures += main.test_failures_count().unwrap_or_default();
240247
let mode = ToolchainMatrix::humanize_mode(key);

‎solx-benchmark-converter/src/output/summary/summary_template/health_issue.rs‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -27,7 +27,7 @@ pub enum HealthIssue {
2727
/// Recognized runs whose mode carries no recognized pipeline token, such
2828
/// as a new codegen letter the tables don't know yet.
2929
UnrecognizedPipelines { label: String, modes: Vec<String> },
30-
/// PR runs with no `main` counterpart; their failures are not compared.
30+
/// PR runs with no `main` baseline; their failures are not compared.
3131
Unbaselined {
3232
label: String,
3333
runs: usize,

‎solx-benchmark-converter/src/output/summary/summary_template/mod.rs‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -153,7 +153,7 @@ impl SummaryTemplate {
153153
let mut warn_lines = Vec::new();
154154
if !unbaselined.is_empty() {
155155
warn_lines.push(format!(
156-
"⚠️ **No baseline** — {} {} no `main` counterpart; {} {} not compared.",
156+
"⚠️ **No baseline** — {} {} no `main` baseline; {} {} not compared.",
157157
unbaselined.join("; "),
158158
crate::utils::agreeing(unbaselined_runs as u64, "has", "have"),
159159
crate::utils::agreeing(unbaselined_runs as u64, "its", "their"),

‎solx-benchmark-converter/src/tests/fixtures/degraded-harness.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@
55
❌ **Suite errored** — solx-tester produced no usable report.
66
❌ **Harness error** — Foundry: benchmark data matched no recognized toolchain naming.
77
❌ **Harness error** — Hardhat: runs matched no declared toolchain: `04.mason-legacy`.
8-
⚠️ **No baseline** — Hardhat: 1 run (5 failures) has no `main` counterpart; its failures are not compared.
8+
⚠️ **No baseline** — Hardhat: 1 run (5 failures) has no `main` baseline; its failures are not compared.
99

1010
| Suite | New failures | Size Δ | Gas Δ | Report |
1111
|---|---|---|---|---|

‎solx-benchmark-converter/src/tests/fixtures/new-failures.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,10 +2,11 @@
22

33
⚪ **No output data** — no size or gated-gas comparison had a `main` counterpart to compare against.
44
❌ **New failures** — Foundry: +2 build, +1 test.
5+
⚠️ **No baseline** — Foundry: 1 run (3 failures) has no `main` baseline; its failures are not compared.
56

67
| Suite | New failures | Size Δ | Gas Δ | Report |
78
|---|---|---|---|---|
8-
| Foundry · 5 proj | ❌ +2 build, +1 test (12 pre-existing) | ⚪ not collected | ⚪ not collected | — |
9+
| Foundry · 5 proj | ❌ +2 build, +1 test (12 pre-existing), ⚪ 3 unbaselined | ⚪ not collected | ⚪ not collected | — |
910

1011
**New failures (PR vs `main`):**
1112

‎solx-benchmark-converter/src/tests/summary.rs‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -317,6 +317,24 @@ fn tests_that_never_ran_are_not_a_measured_zero() {
317317
assert!(!out.contains("test failures 0 → 3"), "{out}");
318318
assert!(!out.contains("+3 test"), "{out}");
319319
assert!(out.contains("✅ **No new failures**"), "{out}");
320+
assert!(
321+
out.contains("⚠️ **No baseline** — Foundry: 1 run (3 failures) has no `main` baseline"),
322+
"{out}"
323+
);
324+
}
325+
326+
#[test]
327+
fn a_repaired_build_with_passing_tests_leaves_nothing_unbaselined() {
328+
let tests = vec![Test::failure(
329+
"p",
330+
&[
331+
("02.solx-main-legacy", RunFailures::Build(2)),
332+
("03.solx-legacy", RunFailures::Test(0)),
333+
],
334+
)];
335+
let out = Summary::new(vec![SummarySuite::merged(SuiteKind::Foundry, tests)]).render();
336+
assert!(!out.contains("No baseline"), "{out}");
337+
assert!(!out.contains("unbaselined"), "{out}");
320338
}
321339

322340
#[test]

0 commit comments

Comments
 (0)