Repository navigation
Commit 46de0fa
committed
fix(test-harness): surface PR test failures whose main build failed
A PR run whose main counterpart failed to build was paired for the build
comparison, but main's test count was never measured, so the PR's test
failures were skipped without appearing anywhere in the summary. #715 hit
this: main could not build swap-vm, the PR could, and the one failing
swap-vm test showed up in neither the new, pre-existing nor unbaselined
counts. Such runs now count as unbaselined when the PR's tests fail, so
their failures reach the "No baseline" line and the table's failure cell.
A repaired build whose tests all pass leaves nothing uncompared and stays
out of it. The line now says "no `main` baseline", which holds for both
an unpaired run and a paired one whose main side never ran its tests.1 parent 0efe7c8 commit 46de0fa
6 files changed
Lines changed: 31 additions & 5 deletions
File tree
- solx-benchmark-converter/src
- output/summary
- summary_template
- tests
- fixtures
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
52 | 52 | | |
53 | 53 | | |
54 | 54 | | |
55 | | - | |
| 55 | + | |
| 56 | + | |
56 | 57 | | |
57 | 58 | | |
58 | 59 | | |
| |||
235 | 236 | | |
236 | 237 | | |
237 | 238 | | |
| 239 | + | |
| 240 | + | |
| 241 | + | |
| 242 | + | |
| 243 | + | |
| 244 | + | |
238 | 245 | | |
239 | 246 | | |
240 | 247 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
27 | 27 | | |
28 | 28 | | |
29 | 29 | | |
30 | | - | |
| 30 | + | |
31 | 31 | | |
32 | 32 | | |
33 | 33 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
153 | 153 | | |
154 | 154 | | |
155 | 155 | | |
156 | | - | |
| 156 | + | |
157 | 157 | | |
158 | 158 | | |
159 | 159 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
5 | 5 | | |
6 | 6 | | |
7 | 7 | | |
8 | | - | |
| 8 | + | |
9 | 9 | | |
10 | 10 | | |
11 | 11 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
2 | 2 | | |
3 | 3 | | |
4 | 4 | | |
| 5 | + | |
5 | 6 | | |
6 | 7 | | |
7 | 8 | | |
8 | | - | |
| 9 | + | |
9 | 10 | | |
10 | 11 | | |
11 | 12 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
317 | 317 | | |
318 | 318 | | |
319 | 319 | | |
| 320 | + | |
| 321 | + | |
| 322 | + | |
| 323 | + | |
| 324 | + | |
| 325 | + | |
| 326 | + | |
| 327 | + | |
| 328 | + | |
| 329 | + | |
| 330 | + | |
| 331 | + | |
| 332 | + | |
| 333 | + | |
| 334 | + | |
| 335 | + | |
| 336 | + | |
| 337 | + | |
320 | 338 | | |
321 | 339 | | |
322 | 340 | | |
| |||
0 commit comments