Run GPU tests only on explicit trigger

[JoshuaLampert]

This enables the possibility to start the AMD GPU tests only on an explicit trigger (commenting /amdgpu on a PR) rather than for each commit on a PR. This is helpful because there are not so many AMD runners available, which can lead to long waiting times in CI, see #3199 (comment). As far as I understand some settings in both GitHub and buildkite have to be adjusted before this should work. According to Claude these are the following (I cannot set those settings myself because I do not have the appropriate rights).

1. Merge the repo change first
   Safe ordering: with the new if: on main but the trigger not yet enabled, AMDGPU simply stops running except on main pushes and manual builds. Nothing breaks; CUDA is unaffected. Do the settings after.

2. GitHub: add the Issue comments event to the webhook
   github.com/trixi-framework/Trixi.jl → Settings → Webhooks → the Buildkite webhook → Let me select individual events → tick both Pull requests and Issue comments → Update webhook.
   Buildkite needs the pull_request event even though nothing is built from it directly — it's how Buildkite learns which branch and head commit a PR points at, so a later issue_comment can be resolved to a commit.
   If there is no per-repo webhook because the org uses the Buildkite GitHub App, the app already subscribes to these events and you can skip to step 3.

3. Buildkite: enable the comment trigger
   Pipeline → Settings → GitHub → Additional Webhooks → enable Issue comments:

• Command word: /amdgpu
• Match mode: exact — the comment must be exactly /amdgpu. Pick contains instead if you want /amdgpu please, this touches the ROCm kernels to work.

4. Leave the PR build settings alone
   In the main GitHub settings section, Build when pull request is opened or updated must stay enabled — that's what keeps CUDA running on every PR. [...]

5. Optional: stop duplicate CUDA builds
   Under Pull request webhook options, consider Skip when pull request has existing build for commit and branch. Trixi devs push topic branches to the main repo, so a push can produce both a branch build and a PR build — with CUDA unconditional, that's twice the GPU cost per push. Pre-existing behavior, but worth checking now that CUDA is the always-on suite.

6. Prime the open PRs
   GitHub does not resend pull_request events retroactively. Every PR that was already open when you changed the webhook needs one new push before /amdgpu works on it. New PRs are fine immediately.

7. Verify
   Comment /amdgpu on a test PR from an account GitHub reports as owner, member, or collaborator. Expect a new build to appear within a few seconds. If nothing happens, check in order: the webhook's Recent Deliveries tab in GitHub (did the issue_comment event fire and get a 200?), then the command word and match mode, then whether that PR was primed per step 6.

Instead of merging this PR first and then adjusting the settings I would change the settings first to see if it already works on this PR and if not we can still merge this PR and test it again after that on other PRs.

[github-actions]

Review checklist

This checklist is meant to assist creators of PRs (to let them know what reviewers will typically look for) and reviewers (to guide them in a structured review process). Items do not need to be checked explicitly for a PR to be eligible for merging.

Purpose and scope

• The PR has a single goal that is clear from the PR title and/or description.
• All code changes represent a single set of modifications that logically belong together.
• No more than 500 lines of code are changed or there is no obvious way to split the PR into multiple PRs.

Code quality

• The code can be understood easily.
• Newly introduced names for variables etc. are self-descriptive and consistent with existing naming conventions.
• There are no redundancies that can be removed by simple modularization/refactoring.
• There are no leftover debug statements or commented code sections.
• The code adheres to our conventions and style guide, and to the Julia guidelines.

Documentation

• New functions and types are documented with a docstring or top-level comment.
• Relevant publications are referenced in docstrings (see example for formatting).
• Inline comments are used to document longer or unusual code sections.
• Comments describe intent ("why?") and not just functionality ("what?").
• If the PR introduces a significant change or new feature, it is documented in NEWS.md with its PR number.

Testing

• The PR passes all tests.
• New or modified lines of code are covered by tests.
• New or modified tests run in less then 10 seconds.

Performance

• There are no type instabilities or memory allocations in performance-critical parts.
• If the PR intent is to improve performance, before/after time measurements are posted in the PR.

Verification

• The correctness of the code was verified using appropriate tests.
• If new equations/methods are added, a convergence test has been run and the results
  are posted in the PR.

Created with ❤️ by the Trixi.jl community.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 96.96%. Comparing base (70b1764) to head (7913d91).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #3200   +/-   ##
=======================================
  Coverage   96.96%   96.96%           
=======================================
  Files         673      673           
  Lines       52156    52156           
=======================================
  Hits        50569    50569           
  Misses       1587     1587           

Flag	Coverage Δ
unittests	96.96% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[JoshuaLampert]

As discussed yesterday, I switched the setup to the one from TrixiParticles.jl, i.e., with an explicit job, which catches the comment (now /run_gpu_tests for both CUDA and AMD tests). In contrast to TrixiParticles.jl I limited the right to execute the tests to users, who are members, collaborator, or owners to avoid anyone can run arbitrary code on the GPU runners from forked PRs.
In contrast to what I wrote above, the setup this needs is a bit different now. The steps I described are not necessary anymore. Instead the following two steps are necessary (from Claude):

1. Create a Buildkite API token with the write_builds scope, and add it as the TRIGGER_BK_BUILD_TOKEN secret on trixi-framework/Trixi.jl. [...]
2. In pipeline settings, disable PR and branch builds — with everything gated behind the trigger, webhook builds only create empty builds. Keep branch builds for main if you want post-merge coverage; the if: conditions allow it.

[JoshuaLampert]

That's interesting. Suddenly the broken tests due to #3136 pass again, which is nice (and why currently CI fails). I guess this is because of some recent work on the TestItems.jl framework. There has been a lot of work recently (see https://discourse.julialang.org/t/two-years-of-test-item-framework-updates/138792/17). I'll create a PR removing the @test_broken.
Edit: Should be fixed by #3203.

[JoshuaLampert]

On first glance, this error looks like it is introduced by trixi-framework/TrixiBase.jl#76. Can you maybe take a look, @efaulhaber?
edit: I guess, this error as well

[efaulhaber]

Yes, sorry! It appears that TrixiTest is doing something weird with Symbols that relied on the bug in TrixiBase. Working on a fix!

[JoshuaLampert]

Yes, sorry! It appears that TrixiTest is doing something weird with Symbols that relied on the bug in TrixiBase. Working on a fix!

Great, thanks! I am seeing similar failures also in another repo using TrixiBase.jl + TrixiTest.jl.

[ranocha]

Thanks, this looks reasonable to me. However, I cannot make the required changes on buildkite. @vchuravy: Could you please review this PR and make the required changes if everything looks good to you?