ci(l1,l2): stop the merge queue satisfying the required integration checks with a skipped job - #7213
ci(l1,l2): stop the merge queue satisfying the required integration checks with a skipped job#7213ilitteri wants to merge 2 commits into
Conversation
…of skipping them.
`Integration Test` and `Integration Test L2` are required status checks on main,
but both gate jobs bailed out whenever a dependency was skipped:
if: ${{ ... && needs.run-hive.result != 'skipped' ... }}
Hive, assertoor and the L2 suites are all excluded from `merge_group` to keep the
queue cheap, so inside the queue that condition was always false and the gate job
was skipped. GitHub counts a skipped check run as satisfying a required status
check, so every merge group satisfied both requirements without running or even
consulting the suites they exist to enforce. A pull request queued while its hive
run was red, or while a re-run was still in flight, merged on that vacuous green.
Two of the three commits that landed on main on the day this was found had a red
`Hive - Devp2p tests` and a failed `Integration Test` on their own head, and main
went red on devp2p immediately afterwards.
The gate jobs now always run when the workflow is not skipped wholesale, and on
`merge_group` they read the queued pull request's own results for those suites
via `check-queued-pr-checks.sh`. That keeps the queue's cost profile unchanged
while making the requirement real, and because it runs at merge time it also
catches a suite that turned red or was re-triggered after the pull request was
added to the queue: a still-running suite blocks the group rather than passing.
The script resolves the queued pull requests from the merge group's commit
subjects, so a batched group is covered rather than only its last pull request,
and it keeps the most recently started check run per name so a superseded red run
cannot block a head that is now green. A skipped suite still counts as satisfied,
which is what an L2-only pull request looks like to the L1 workflow. Finding no
matching check run at all fails: a gate that cannot see what it is verifying must
not report success.
Not addressed here: `check-cargo-locks` is in the L1 gate's `needs` but its
result is still never inspected, so `Check Cargo.lock` remains unenforced by the
required check. That is a separate policy call.
…ude. Same bug class as the parent commit, one dependency further up. The gate keys off `needs.detect-changes.outputs.run_tests`, and a job that failed publishes no outputs, so `'' == 'true'` was false and the gate skipped — which GitHub counts as satisfying the required check. A broken change-detection step therefore turned both `Integration Test` and `Integration Test L2` green without anything having been evaluated. The gate now also runs when `detect-changes` did not succeed, and fails immediately in that case: with no `run_tests` answer there is no way to tell whether the suites were required, and an unanswerable gate must not report success.
|
🤖 Kimi Code ReviewThis PR correctly implements a fail-closed gate for GitHub merge queues to prevent merging PRs with failing/skipped expensive test suites (Hive, Assertoor, L2 integration tests). The approach of reading the original PR's check runs rather than relying on skipped jobs in the merge group is the correct solution to GitHub's behavior where skipped checks satisfy required status checks.
|
🤖 Codex Code ReviewFindings
This PR only touches CI/workflow code, so I did not find any Rust/EVM/consensus-path changes to review here. Automated review by OpenAI Codex · gpt-5.4 · custom prompt |
🤖 Claude Code ReviewReview: PR 7213 — merge queue enforcement of required integration checksThis is a CI/workflow change rather than core protocol code, so the review focuses on the correctness of the gating logic and the new shell script rather than EVM/consensus concerns. Findings1.
The result: in the merge queue, 2. In 3. Reliance on squash-commit-subject convention for identifying queued PRs (low severity, flagged for awareness)
Positive notes
Automated review by Claude (Anthropic) · sonnet · custom prompt |
MegaRedHand
left a comment
MegaRedHand
left a comment
There was a problem hiding this comment.
Reviewed the merge-queue gate change against live repo data. The problem it targets is real and I confirmed it end to end: PR #7209 merged on 2026-08-24 with Hive - Devp2p tests red, and its merge group reported Integration Test as skipped. Reading the queued pull request's own check runs is the right shape for a fix, and the bash is careful.
Two things stop it from actually closing the hole. Both are inline, both verified against real commits.
1. skipped is accepted, and that state is reachable with the suites genuinely not run. Head e104cdbc really exists: Build Docker = failure, Hive - ${{ matrix.name }} = skipped, Assertoor - ${{ matrix.name }} = skipped. Replaying this script's loop over that head prints All required suites passed.
The natural defense is that such a pull request cannot enter the queue because its own Integration Test is red. That does not hold: Integration Test is declared by three workflows (pr-main_l1.yaml, pr-main_l1_l2_dev.yaml, pr-main_levm.yaml), and on PR #7209's head the L1 one was failure while the L2-Dev one was success. It merged.
2. The gate still skips entirely in the queue for any non-Rust change. On merge_group both workflows set run_tests = code_changed, which matches only **/*.rs, **/*.toml, **/*.lock. PR #7193 (CI-only) merged with every job in its merge group skipped, Integration Test included, while its head carried seven real green Hive - * results the queue never read. This pull request is in that same class, so it would merge under the behavior it is fixing.
What it gets right. Dropping the needs.<job>.result != 'skipped' guards is a genuine fix on the pull_request side: run-hive and run-assertoor both need docker_build, so a broken image used to skip the gate into a green required check. I also checked and found no problems with injection (PR numbers are digits after grep -oE/tr, names are quoted, jq @tsv escapes), set -euo pipefail behavior (the empty-array guard catches the grep miss, [[ -z ... ]] && continue does not trip set -e, the here-string keeps matched/failed in scope), the --paginate --slurp shape, the (#N) title heuristic (the repo is squash-only with PR_TITLE), and the permissions: blocks, which are minimal and sufficient.
Remaining inline notes are smaller.
| if [[ "$status" != "completed" ]]; then | ||
| echo "PENDING ${name} (status=${status}) is still running, so it cannot be merged yet" | ||
| failed=1 | ||
| elif [[ "$conclusion" != "success" && "$conclusion" != "skipped" && "$conclusion" != "neutral" ]]; then |
There was a problem hiding this comment.
skipped is accepted unconditionally, and it is reachable with the suites genuinely not run.
Real head e104cdbccb2aad1c771ea4ab397fff8a10520f5b carries:
Build Docker failure
Hive - ${{ matrix.name }} skipped
Assertoor - ${{ matrix.name }} skipped
run-hive and run-assertoor both declare needs: [detect-changes, docker_build], so a broken image turns the whole suite into skips. Feeding that head's check runs through this loop prints All required suites passed on every queued pull request head.
Rejecting skipped outright is not the fix: an L1-only pull request legitimately shows all four L2 suites skipped (PR #7194 is exactly that), so it would become unmergeable. Telling the two apart needs a second signal, for example the head's own Integration Test / Integration Test L2 check run, or the pull_request run conclusion from /actions/runs?head_sha=....
| done | ||
| done <<<"$all_checks" | ||
|
|
||
| if [[ $matched -eq 0 ]]; then |
There was a problem hiding this comment.
matched is one counter across every prefix, so this net only fires when all prefixes come up empty. Rename or delete a single suite and it silently stops being enforced while the others keep the gate green.
Reproduced under bash with the L2 caller's four prefixes: rename state-diff-test's job, leave it red, and the script still prints All required suites passed and exits 0.
Worth noting that none of the six suite jobs carry the # "..." is a required check, don't change the name comment that all-tests has, and the prefix strings live in a third file with no link back to them. A per-prefix counter closes this.
| "repos/${GITHUB_REPOSITORY}/commits/${pr_head}/check-runs?per_page=100" | | ||
| jq -r '[.[].check_runs[]] | ||
| | group_by(.name) | ||
| | map(max_by(.started_at // "")) |
There was a problem hiding this comment.
This collapses same-named check runs from different workflows, keeping whichever started later.
daily_hive_report.yaml:24 declares name: Hive - ${{ matrix.test.name }} with a Rpc Compat tests matrix entry and continue-on-error: true, and it triggers on pull_request for .github/workflows/daily_hive_report.yaml and .github/scripts/publish_hive.sh. On PR #7170's head both workflows reported:
Hive - Rpc Compat tests success 2026-08-20T17:35:44Z (daily_hive_report)
Hive - Rpc Compat tests success 2026-08-20T17:53:25Z (pr-main_l1)
Which one survives is runner scheduling. Across ten recent daily runs its hive jobs concluded 104 success, 1 failure, 5 cancelled, so this can mask a red L1 hive, or fail the gate on a cancelled that blocks nothing.
Two smaller points on the same expression: the comment above attributes the dedupe to re-triggers, but commits/{sha}/check-runs already defaults to filter=latest, so that is not what it is doing. And started_at has second granularity, so ties resolve by API array order; .id descending would at least be deterministic. Matching on the workflow or .app.id alongside the name would be better than either.
|
|
||
| failed=0 | ||
| for pr in "${pr_numbers[@]}"; do | ||
| pr_head=$(gh api "repos/${GITHUB_REPOSITORY}/pulls/${pr}" --jq '.head.sha') |
There was a problem hiding this comment.
This resolves the pull request's head at gate time, not the SHA that was squashed into the merge group, and the merge_group payload carries no PR-head SHA to compare it against.
Every branch I could construct is fail-closed: a force-pushed head has no check runs, so matched == 0; a fresh push leaves them in_progress, so PENDING. What is lost is the property this file's header advertises, catching a result that turned red after queueing, during that window. Low priority, but a comment saying the live head is deliberate would help.
| # status check, so a gate that skips is a gate that always passes: inside the | ||
| # merge queue, where assertoor and hive do not run, that let a pull request | ||
| # whose suites were red merge on a vacuous green. | ||
| if: ${{ always() && (needs.detect-changes.result != 'success' || needs.detect-changes.outputs.run_tests == 'true') }} |
There was a problem hiding this comment.
This still skips the gate whenever run_tests is false, and on merge_group both workflows set run_tests = code_changed, which matches only **/*.rs, **/*.toml and **/*.lock. Any pull request touching only workflows, scripts, fixtures or configs keeps the old behavior.
PR #7193 (CI-only) is the worked example: merge group run 32502079792 skipped every job including Integration Test, and it merged. Its head was not untested; it carried seven green Hive - * runs, because the pull_request filter is !crates/l2/** and .github/** passes it. The queue just never read them.
This pull request changes only .github/** .yaml and .sh, so it is in that class itself.
Separately, always() also runs the job when the run is cancelled. The old expression short-circuited on outputs.run_tests == 'true' being empty, so a concurrency-cancelled run skipped the gate; now it runs and the step below exits 1, stamping a red Integration Test on superseded commits. !cancelled() && (...) avoids that.
| exit 1 | ||
|
|
||
| - name: Checkout sources | ||
| uses: actions/checkout@v6 |
There was a problem hiding this comment.
Only the merge_group step uses the checked-out tree; the other two steps are inline shell over the needs context. if: ${{ github.event_name == 'merge_group' }} here would skip a full clone on every pull_request and push run of both gates.
| if: ${{ github.event_name == 'merge_group' }} | ||
| env: | ||
| GH_TOKEN: ${{ github.token }} | ||
| run: ./.github/scripts/check-queued-pr-checks.sh "Hive - " "Assertoor - " |
There was a problem hiding this comment.
engine-ef-tests is in this job's needs, skips in the merge queue exactly like hive and assertoor, and is not a required context on its own, but it is missing from this prefix list. PR #7194's head carries a real Engine EF tests success run, so adding the prefix closes it in one line.
The Hive - prefix also over-matches: daily_hive_report.yaml emits check runs under that same prefix on any pull request touching its trigger paths. See the note on the jq dedupe in the script.
| GH_TOKEN: ${{ github.token }} | ||
| run: | | ||
| ./.github/scripts/check-queued-pr-checks.sh \ | ||
| "Integration Test - " \ |
There was a problem hiding this comment.
Integration Test - matches Integration Test - TDX (integration-test-tdx, line 509), but the Check if any job failed step below checks only integration-test, state-diff-test, uniswap-swap and integration-test-shared-bridge. TDX is in needs and has no continue-on-error.
So a red TDX leaves Integration Test L2 green on the pull request, the pull request is queueable, and the merge group then rejects it with nothing the author can turn green by re-running. Sampling twelve recent pr-main_l2 pull_request runs, TDX concluded 2 failure, 3 cancelled, 4 skipped, 1 success, so this is not a rare state.
Either add the tdx branch to the step below or narrow the prefix, but the two sides should agree on what is required.
2 participants
Motivation
A pull request whose
Hiverun was red, or whose re-run was still in flight, could mergethrough the merge queue without that result ever being consulted.
Integration TestandIntegration Test L2are required status checks onmain, and bothgate jobs bailed out whenever a dependency was skipped:
Hive, assertoor and the L2 suites are all excluded from
merge_groupto keep the queuecheap, so inside the queue that condition was always false and the gate job was skipped.
GitHub counts a skipped check run as satisfying a required status check, so every merge
group satisfied both requirements without running or even consulting the suites they exist to
enforce.
Merge when readycompounds it: GitHub decides queue eligibility when the pull request isenqueued and does not re-evaluate the head afterwards. A stale green
Integration Testisenough to get in, a later red result cannot evict it, and the merge group's own gate was
vacuous. So re-running a failed Hive job did not protect
main— the merge went ahead whilethe re-run was still going, and the re-run's failure landed after the merge.
Observed on three merges the same day:
Integration Teston its headfailureat 16:49:47failureat 17:03:50Both had a red
Hive - Devp2p tests, andmainwent red on devp2p immediately afterwardsuntil #7204 fixed the underlying discv5 bug.
Description
No suite is added to the merge queue. All fourteen
github.event_name != 'merge_group'exclusions are untouched, so hive, assertoor, reorg, engine-EF and the L2 suites still do not
run there, and the queue's cost profile is unchanged. What changes is what the gate does:
gate that always passes.
merge_groupthey run.github/scripts/check-queued-pr-checks.sh, which reads thequeued pull request's own check runs for those suites. A red suite fails the group; a suite
that is not yet completed also fails it, which is the case this PR is really about — a
Hive re-run in flight now blocks the merge group instead of being bypassed. Because this
runs at merge time rather than at enqueue time, it also catches a result that turned red
after the pull request was queued.
detect-changespublishesno outputs, so
'' == 'true'skipped the gate and turned both required checks green withnothing evaluated. The gate now runs and fails in that case.
Details in the script that are easy to get wrong, and why it is written this way:
compareAPI, not from the queue branch name, because a batched group's ref names only its last
pull request.
several check suites, and without this a superseded red run would keep blocking a head that
is now green.
skippedcounts as satisfied. That is what an L2-only pull request looks like to the L1workflow, and what a docs-only one looks like to both.
must not report success.
Verification
The gate's decision logic is not something CI can exercise, so it was checked directly:
ten cases through a stubbed API (all green, all skipped,
in_progress,queued,failure,cancelled,neutral, no match, and both stale-run orderings), then against live data —the #7204 merge group passes, a group containing #7201 blocks on its red
Hive - Devp2p tests, an L2-only pull request's skipped L1 suites pass, and a pull request with three Hivejobs genuinely in flight blocks.
Known limitation
If the merge group's gate has already reported green and a Hive job is re-run after that, the
merge can still complete. Closing that would need the suites to run in the queue, which is
the trade this PR deliberately does not make: it would add roughly twenty minutes per queued
pull request and let a hive flake evict pull requests from the queue.
Not addressed here
check-cargo-locksis in the L1 gate'sneedsbut its result is still never inspected, soCheck Cargo.lockremains unenforced by the required check. That is a separate policy call.Checklist
STORE_SCHEMA_VERSION(crates/storage/lib.rs) if the PR includes breaking changes to theStorerequiring a re-sync.