Remove Run 3 PU GPU workflows by makortel · Pull Request #51159 · cms-sw/cmssw

makortel · 2026-06-09T15:11:13Z

PR description:

These read non-RAW input files and are unlikely to be revived in 20_1_X, so they are removed.

I had missed these in #50954.

PR validation:

I checked runTheMatrix.py -w gpu -n without and with this PR. The 32 workflows failing in the 20_[01]_X GPU tests because of DictionaryNotFound are removed from the list of workflows. After limiting the PR to only the removal of Run 3 workflows, 24 workflows are removed.

If this PR is a backport please specify the original PR and why you need to backport that PR. If this PR will be backported please specify to which release cycle the backport is meant for:

To be backported to 20_0_X.

cmsbuild · 2026-06-09T15:11:40Z

cms-bot internal usage

cmsbuild · 2026-06-09T15:12:33Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-51159/49686

cmsbuild · 2026-06-09T15:12:57Z

A new Pull Request was created by @makortel for master.

It involves the following packages:

Configuration/PyReleaseValidation (pdmv)

@AdrianoDee, @DickyChant, @antoniovagnerini, @cmsbuild, @kfjack, @miquork, @sroychow can you please review it and eventually sign? Thanks.
@Martin-Grunewald, @fabiocos, @slomeo this is something you requested to watch as well.
@ftenchini, @mandrenguyen, @sextonkennedy you are the release manager for this.

cms-bot commands are listed here

makortel · 2026-06-09T15:16:12Z

enable gpu

makortel · 2026-06-09T15:16:17Z

@cmsbuild, please test

makortel · 2026-06-09T15:16:23Z

FYI @cms-sw/heterogeneous-l2

fwyzard · 2026-06-09T16:38:40Z

@@ -30,15 +27,6 @@
 #           Alpaka pixel-only quadruplets:                      ZMM: any backend, any backend vs cpu validation, profiling
 #           Alpaka pixel-only triplets:                         ZMM: any backend, any backend vs cpu validation, profiling
 #           Alpaka pixel-only quadruplets:                      Single Nu E10: any backend


shouldn't these be removed as well ?

I kept the Run 3 noPU workflows for now because they technically run and the exact route to dismantle Run 1-3 reconstruction (and simulation) capability in 20_1_X has not been agreed yet. I'm happy to remove them though if that is deemed appropriate.

@cms-sw/pdmv-l2 Any thoughts?

I had a similar doubt, not specifically related to these GPU wfs. In my personal opinion, we should start removing the Run1-3 wfs in general. Unless we want to keep Run1-3 capabilities still alive in 20_1_X for some reason.

At the time of the release planning XC a month ago (https://indico.cern.ch/event/1678193/) 4 use cases for running (parts of) Run 3 data/simulation in 20_1_X (and possibly beyond)

High-pileup /EGamma*/Run2025G-v1/RAW data for ECAL+EGamma phase 2 developments (Tests for processing /EGamma*/Run2025G-v1/RAW in CMSSW_20 #50773)

Workflow for (HLT, offline) algorithm benchmarking

Validation of big changes in MC truth infrastructure

PF reconstruction on Run 3 high PU data (Enabling PF reconstruction in Phase-2 CMSSW on Run-3 highPU RAW data #50885)

To my knowledge the fate of these use cases has not been decided. There was some discussion on the PF use case in https://indico.cern.ch/event/1681572/, but I'm not aware of further discussion on the others.

I suggest to move the general discussion on how long the Run 1-3 processing capability is needed to #51191 and the per use case issues linked therein.

In the meantime I think it would be nice to avoid reporting the failures of these workflows in the IBs.

cmsbuild · 2026-06-09T18:12:25Z

-1

Failed Tests: RelVals-AMD_W7900
Size: This PR adds an extra 20KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-e450bd/53779/summary.html
COMMIT: a63c69d
CMSSW: CMSSW_20_1_X_2026-06-09-1100/el9_amd64_gcc13
Additional Tests: GPU,AMD_MI300X,AMD_W7900,NVIDIA_H100,NVIDIA_L40S,NVIDIA_T4
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/51159/53779/install.sh to create a dev area with all the needed externals and cmssw changes.

Failed RelVals-AMD_W7900

34434.40434434.404_TTbar_14TeV+Run4D121_Patatrack_PixelOnlyAlpaka_Profiling/step2_TTbar_14TeV+Run4D121_Patatrack_PixelOnlyAlpaka_Profiling.log

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 44
DQMHistoTests: Total histograms compared: 3248390
DQMHistoTests: Total failures: 0
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3248372
DQMHistoTests: Total skipped: 18
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 43 files compared)
Checked 191 log files, 159 edm output root files, 44 DQM output files

AMD_MI300X Comparison Summary

There are some workflows for which there are errors in the baseline:
34434.403 step 2
The results for the comparisons for these workflows could be incomplete
This means most likely that the IB is having errors in the relvals.The error does NOT come from this pull request

Summary:

You potentially removed 72 lines from the logs
ROOTFileChecks: Some differences in event products or their sizes found
Reco comparison results: 106 differences found in the comparisons
DQMHistoTests: Total files compared: 6
DQMHistoTests: Total histograms compared: 147128
DQMHistoTests: Total failures: 15570
DQMHistoTests: Total nulls: 1
DQMHistoTests: Total successes: 131557
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 5 files compared)
Checked 23 log files, 18 edm output root files, 6 DQM output files

NVIDIA_H100 Comparison Summary

Summary:

You potentially removed 2 lines from the logs
Reco comparison results: 145 differences found in the comparisons
DQMHistoTests: Total files compared: 7
DQMHistoTests: Total histograms compared: 163207
DQMHistoTests: Total failures: 20613
DQMHistoTests: Total nulls: 1
DQMHistoTests: Total successes: 142593
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 6 files compared)
Checked 25 log files, 20 edm output root files, 7 DQM output files

NVIDIA_L40S Comparison Summary

Summary:

You potentially added 2 lines to the logs
Reco comparison results: 125 differences found in the comparisons
DQMHistoTests: Total files compared: 7
DQMHistoTests: Total histograms compared: 163207
DQMHistoTests: Total failures: 20491
DQMHistoTests: Total nulls: 1
DQMHistoTests: Total successes: 142715
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 6 files compared)
Checked 25 log files, 20 edm output root files, 7 DQM output files

NVIDIA_T4 Comparison Summary

Summary:

You potentially added 1 lines to the logs
Reco comparison results: 119 differences found in the comparisons
DQMHistoTests: Total files compared: 7
DQMHistoTests: Total histograms compared: 163207
DQMHistoTests: Total failures: 20541
DQMHistoTests: Total nulls: 1
DQMHistoTests: Total successes: 142665
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 6 files compared)
Checked 25 log files, 20 edm output root files, 7 DQM output files

Max Memory Comparisons exceeding threshold NVIDIA_L40S

@cms-sw/core-l2 , I found 1 workflow step(s) with memory usage exceeding the error threshold:

Expand to see workflows ...

Error: Workflow 34434.7503_TTbar_14TeV+Run4D121_HLTHeterogeneousValid step2 max memory diff -263.6 exceeds +/- 30.0 MiB

Max Memory Comparisons exceeding threshold NVIDIA_T4

@cms-sw/core-l2 , I found 1 workflow step(s) with memory usage exceeding the error threshold:

Expand to see workflows ...

Error: Workflow 34434.7503_TTbar_14TeV+Run4D121_HLTHeterogeneousValid step2 max memory diff 266.5 exceeds +/- 30.0 MiB

makortel · 2026-06-09T18:25:26Z

The failure of 34434.404 on AMD GPU seems to be related to #49570

makortel · 2026-06-10T18:47:55Z

ignore tests-rejected with ib-failure

makortel · 2026-06-11T21:25:15Z

How should this PR relate to #51199 ?

In principle after #51199 I guess the Run 4 PU (GPU) workflows could be kept. What about backports to 20_0_X?

AdrianoDee · 2026-06-12T08:51:18Z

In principle after #51199 I guess the Run 4 PU (GPU) workflows could be kept.

Indeed.

What about backports to 20_0_X?

I'll open a backport for #51199 as soon as the samples are done.

makortel · 2026-06-12T14:08:33Z

Just to clarify, is the preference to get #51199 merged first, and then rebase this PR on top of that leaving the Run 4 workflows untouched? (and likewise for backports to 20_0_X)

AdrianoDee · 2026-06-12T14:14:18Z

Just to clarify, is the preference to get #51199 merged first, and then rebase this PR on top of that leaving the Run 4 workflows untouched? (and likewise for backports to 20_0_X)

As you prefer. In principle, you could just remove the Run3 ones and leave the Run4 ones untouched here. Then whichever (among this and #51159) enters first, it shouldn't change much, right?

makortel · 2026-06-12T14:25:30Z

As you prefer. In principle, you could just remove the Run3 ones and leave the Run4 ones untouched here. Then whichever (among this and #51159) enters first, it shouldn't change much, right?

Good point. Ok, let's do that.

makortel · 2026-06-12T15:21:01Z

Now only Run 3 PU GPU workflows are removed.

cmsbuild · 2026-06-12T15:21:20Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-51159/49754

cmsbuild · 2026-06-12T15:21:23Z

Pull request #51159 was updated. @AdrianoDee, @DickyChant, @antoniovagnerini, @cmsbuild, @kfjack, @miquork, @sroychow can you please check and sign again.

makortel · 2026-06-12T15:21:27Z

@cmsbuild, please test

fwyzard · 2026-06-15T07:08:26Z

@smuzaffar looks like the workflow completed, but the bot didn't notice?

smuzaffar · 2026-06-15T07:17:56Z

bot is still waiting for https://cmssdt.cern.ch/jenkins/job/ib-run-pr-unittests/4000/ (unit tests for amd w7900) which failed due to rocmComputeCapabilities hanged for 3 hours

17:38:23 ++ rocmComputeCapabilities
20:38:16 Build timed out (after 180 minutes). Marking the build as aborted.

i am retrying that job now

cmsbuild · 2026-06-15T07:19:27Z

+1

Size: This PR adds an extra 24KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-e450bd/53910/summary.html
COMMIT: 3b08f73
CMSSW: CMSSW_20_1_X_2026-06-12-1100/el9_amd64_gcc13
Additional Tests: GPU,AMD_MI300X,AMD_W7900,NVIDIA_H100,NVIDIA_L40S,NVIDIA_T4
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/51159/53910/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 44
DQMHistoTests: Total histograms compared: 3267025
DQMHistoTests: Total failures: 16
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3266991
DQMHistoTests: Total skipped: 18
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 43 files compared)
Checked 191 log files, 159 edm output root files, 44 DQM output files

AMD_MI300X Comparison Summary

There are some workflows for which there are errors in the baseline:
34434.404 step 2
The results for the comparisons for these workflows could be incomplete
This means most likely that the IB is having errors in the relvals.The error does NOT come from this pull request

Summary:

You potentially removed 70 lines from the logs
ROOTFileChecks: Some differences in event products or their sizes found
Reco comparison results: 105 differences found in the comparisons
DQMHistoTests: Total files compared: 7
DQMHistoTests: Total histograms compared: 167007
DQMHistoTests: Total failures: 17373
DQMHistoTests: Total nulls: 2
DQMHistoTests: Total successes: 149632
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 6 files compared)
Checked 24 log files, 20 edm output root files, 7 DQM output files

AMD_W7900 Comparison Summary

Summary:

You potentially removed 2 lines from the logs
Reco comparison results: 132 differences found in the comparisons
DQMHistoTests: Total files compared: 7
DQMHistoTests: Total histograms compared: 167007
DQMHistoTests: Total failures: 17038
DQMHistoTests: Total nulls: 1
DQMHistoTests: Total successes: 149968
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 6 files compared)
Checked 25 log files, 20 edm output root files, 7 DQM output files

NVIDIA_H100 Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 121 differences found in the comparisons
DQMHistoTests: Total files compared: 7
DQMHistoTests: Total histograms compared: 167007
DQMHistoTests: Total failures: 19141
DQMHistoTests: Total nulls: 2
DQMHistoTests: Total successes: 147864
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 6 files compared)
Checked 25 log files, 20 edm output root files, 7 DQM output files

NVIDIA_L40S Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 145 differences found in the comparisons
DQMHistoTests: Total files compared: 7
DQMHistoTests: Total histograms compared: 167007
DQMHistoTests: Total failures: 22778
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 144229
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 6 files compared)
Checked 25 log files, 20 edm output root files, 7 DQM output files

NVIDIA_T4 Comparison Summary

Summary:

You potentially removed 1 lines from the logs
Reco comparison results: 177 differences found in the comparisons
DQMHistoTests: Total files compared: 7
DQMHistoTests: Total histograms compared: 167007
DQMHistoTests: Total failures: 20394
DQMHistoTests: Total nulls: 1
DQMHistoTests: Total successes: 146612
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 6 files compared)
Checked 25 log files, 20 edm output root files, 7 DQM output files

Max Memory Comparisons exceeding threshold NVIDIA_T4

@cms-sw/core-l2 , I found 1 workflow step(s) with memory usage exceeding the error threshold:

Expand to see workflows ...

Error: Workflow 34434.7503_TTbar_14TeV+Run4D121_HLTHeterogeneousValid step2 max memory diff 267.4 exceeds +/- 30.0 MiB

makortel · 2026-06-17T12:00:41Z

@cms-sw/pdmv-l2 Kind ping

AdrianoDee · 2026-06-17T12:03:04Z

+pdmv

cmsbuild · 2026-06-17T12:03:26Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @ftenchini, @mandrenguyen, @sextonkennedy (and backports should be raised in the release meeting by the corresponding L2)

makortel · 2026-06-17T12:13:11Z

Thanks. 20_0_X backport is in #51243

mandrenguyen · 2026-06-17T18:01:24Z

+1

Remove Run 3 PU GPU workflows

cmsbuild added this to the CMSSW_20_1_X milestone Jun 9, 2026

cmsbuild added pending-signatures tests-pending orp-pending pdmv-pending code-checks-pending labels Jun 9, 2026

cmsbuild added code-checks-approved and removed code-checks-pending labels Jun 9, 2026

cmsbuild added tests-started and removed tests-pending labels Jun 9, 2026

fwyzard reviewed Jun 9, 2026

View reviewed changes

cmsbuild added tests-rejected and removed tests-started labels Jun 9, 2026

cmsbuild added tests-approved tests-ib-failure and removed tests-rejected labels Jun 10, 2026

cmsbuild added the code-checks-pending label Jun 12, 2026

cmsbuild added code-checks-approved and removed code-checks-pending labels Jun 12, 2026

cmsbuild added tests-started and removed tests-pending labels Jun 12, 2026

cmsbuild added tests-approved and removed tests-started labels Jun 15, 2026

cmsbuild added fully-signed pdmv-approved and removed pending-signatures pdmv-pending labels Jun 17, 2026

makortel mentioned this pull request Jun 17, 2026

[20_0_X] Remove Run 3 PU GPU workflows #51243

Merged

cmsbuild added orp-approved and removed orp-pending labels Jun 17, 2026

cmsbuild merged commit 713ca89 into cms-sw:master Jun 17, 2026
46 of 47 checks passed

cmsbuild mentioned this pull request Jun 18, 2026

[Do not merge]Enable building with cuda for el9_amd64_gcc15 cms-sw/cms-bot#2785

Closed

makortel deleted the evoWorkflowsGPU branch June 18, 2026 08:54

NJManganelli pushed a commit to NJManganelli/cmssw that referenced this pull request Jul 25, 2026

Merge pull request cms-sw#51159 from makortel/evoWorkflowsGPU

6f85c08

Remove Run 3 PU GPU workflows

Uh oh!

Conversation

makortel commented Jun 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR description:

PR validation:

If this PR is a backport please specify the original PR and why you need to backport that PR. If this PR will be backported please specify to which release cycle the backport is meant for:

Uh oh!

cmsbuild commented Jun 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cmsbuild commented Jun 9, 2026

Uh oh!

cmsbuild commented Jun 9, 2026

Uh oh!

makortel commented Jun 9, 2026

Uh oh!

makortel commented Jun 9, 2026

Uh oh!

makortel commented Jun 9, 2026

Uh oh!

fwyzard Jun 9, 2026

Choose a reason for hiding this comment

Uh oh!

makortel Jun 9, 2026

Choose a reason for hiding this comment

Uh oh!

makortel Jun 10, 2026

Choose a reason for hiding this comment

Uh oh!

AdrianoDee Jun 10, 2026

Choose a reason for hiding this comment

Uh oh!

makortel Jun 10, 2026

Choose a reason for hiding this comment

Uh oh!

makortel Jun 10, 2026

Choose a reason for hiding this comment

Uh oh!

cmsbuild commented Jun 9, 2026

Failed RelVals-AMD_W7900

Comparison Summary

AMD_MI300X Comparison Summary

NVIDIA_H100 Comparison Summary

NVIDIA_L40S Comparison Summary

NVIDIA_T4 Comparison Summary

Max Memory Comparisons exceeding threshold NVIDIA_L40S

Max Memory Comparisons exceeding threshold NVIDIA_T4

Uh oh!

makortel commented Jun 9, 2026

Uh oh!

makortel commented Jun 10, 2026

Uh oh!

makortel commented Jun 11, 2026

Uh oh!

AdrianoDee commented Jun 12, 2026

Uh oh!

makortel commented Jun 12, 2026

Uh oh!

AdrianoDee commented Jun 12, 2026

Uh oh!

makortel commented Jun 12, 2026

Uh oh!

makortel commented Jun 12, 2026

Uh oh!

cmsbuild commented Jun 12, 2026

Uh oh!

cmsbuild commented Jun 12, 2026

Uh oh!

makortel commented Jun 12, 2026

Uh oh!

fwyzard commented Jun 15, 2026

Uh oh!

smuzaffar commented Jun 15, 2026

Uh oh!

cmsbuild commented Jun 15, 2026

Comparison Summary

AMD_MI300X Comparison Summary

AMD_W7900 Comparison Summary

NVIDIA_H100 Comparison Summary

makortel commented Jun 9, 2026 •

edited

Loading

cmsbuild commented Jun 9, 2026 •

edited

Loading