Drive build of rocm-system packages from a single build file.

[akritkbehera]

modular rocm-systems

[akritkbehera]

please test

[cmsbuild]

A new Pull Request was created by @akritkbehera for branch IB/CMSSW_20_1_X/rocm724.

@akritkbehera, @iarspider, @raoatifshad, @smuzaffar can you please review it and eventually sign? Thanks.
@ftenchini, @mandrenguyen, @sextonkennedy you are the release manager for this.
cms-bot commands are listed here

[cmsbuild]

cms-bot internal usage

[fwyzard]

Random comment, just thinking out loud.

Since most packages are built from the same sources, would it make sense to actually have a single spec file that builds multiple target %packages ?

[smuzaffar]

Random comment, just thinking out loud.

Since most packages are built from the same sources, would it make sense to actually have a single spec file that builds multiple target %packages ?

With current multi-spec files ( one per rocm core/library package) we can build multiple packages in parallel . With one spec we will build one package at a time and can takes 8-10 to build build that package

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-f67d9b/54637/summary.html
COMMIT: 1a541eb
CMSSW: CMSSW_20_1_ROCM724_X_2026-07-08-2300/el9_amd64_gcc13
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/10714/54637/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

• You potentially added 182 lines to the logs
• ROOTFileChecks: Some differences in event products or their sizes found
• Reco comparison results: 19 differences found in the comparisons
• DQMHistoTests: Total files compared: 45
• DQMHistoTests: Total histograms compared: 3548186
• DQMHistoTests: Total failures: 1578
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3546590
• DQMHistoTests: Total skipped: 18
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 44 files compared)
• Checked 195 log files, 163 edm output root files, 45 DQM output files
• TriggerResults: no differences found

[cmsbuild]

Pull request #10714 was updated.

[akritkbehera]

enable gpu

[akritkbehera]

please test

[akritkbehera]

@cmsbuild please abort

[akritkbehera]

please test

[cmsbuild]

Pull request #10714 was updated.

[cmsbuild]

Pull request #10714 was updated.

[akritkbehera]

please test

[cmsbuild]

-1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-f67d9b/54845/summary.html
COMMIT: 0e0b4ee
CMSSW: CMSSW_20_1_ROCM_X_2026-07-19-2300/el9_amd64_gcc13
Additional Tests: GPU,AMD_MI300X,AMD_W7900,NVIDIA_H100,NVIDIA_L40S,NVIDIA_T4
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/10714/54845/install.sh to create a dev area with all the needed externals and cmssw changes.

Failed External Build

I found compilation warning when building: See details on the summary page.

[akritkbehera]

please test

[cmsbuild]

-1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-f67d9b/54852/summary.html
COMMIT: 0e0b4ee
CMSSW: CMSSW_20_1_ROCM_X_2026-07-19-2300/el9_amd64_gcc13
Additional Tests: GPU,AMD_MI300X,AMD_W7900,NVIDIA_H100,NVIDIA_L40S,NVIDIA_T4
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/10714/54852/install.sh to create a dev area with all the needed externals and cmssw changes.

Failed External Build

I found compilation warning when building: See details on the summary page.

[akritkbehera]

please test

[akritkbehera]

How's it the URL is working when I changed the the base branch but it wasn't previously?

[akritkbehera]

so amd renamed or deleted the branch

[cmsbuild]

-1

Failed Tests: RelVals-AMD_W7900 amd_mi300xUnitTests
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-f67d9b/54860/summary.html
COMMIT: 0e0b4ee
CMSSW: CMSSW_20_1_X_2026-07-21-1100/el9_amd64_gcc13
Additional Tests: GPU,AMD_MI300X,AMD_W7900,NVIDIA_H100,NVIDIA_L40S,NVIDIA_T4
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/10714/54860/install.sh to create a dev area with all the needed externals and cmssw changes.

Failed RelVals-AMD_W7900

• 34434.40434434.404_TTbar_14TeV+Run4D121_Patatrack_PixelOnlyAlpaka_Profiling/step2_TTbar_14TeV+Run4D121_Patatrack_PixelOnlyAlpaka_Profiling.log

Comparison Summary

Summary:

• You potentially added 9 lines to the logs
• ROOTFileChecks: Some differences in event products or their sizes found
• Reco comparison results: 0 differences found in the comparisons
• DQMHistoTests: Total files compared: 45
• DQMHistoTests: Total histograms compared: 3548294
• DQMHistoTests: Total failures: 55
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3548221
• DQMHistoTests: Total skipped: 18
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 44 files compared)
• Checked 195 log files, 163 edm output root files, 45 DQM output files
• TriggerResults: found differences in 1 / 44 workflows

NVIDIA_H100 Comparison Summary

Summary:

• No significant changes to the logs found
• Reco comparison results: 47 differences found in the comparisons
• DQMHistoTests: Total files compared: 7
• DQMHistoTests: Total histograms compared: 167683
• DQMHistoTests: Total failures: 7810
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 159873
• DQMHistoTests: Total skipped: 0
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 6 files compared)
• Checked 25 log files, 20 edm output root files, 7 DQM output files
• TriggerResults: found differences in 1 / 6 workflows

NVIDIA_L40S Comparison Summary

Summary:

• No significant changes to the logs found
• Reco comparison results: 50 differences found in the comparisons
• DQMHistoTests: Total files compared: 7
• DQMHistoTests: Total histograms compared: 167683
• DQMHistoTests: Total failures: 9306
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 158377
• DQMHistoTests: Total skipped: 0
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 6 files compared)
• Checked 25 log files, 20 edm output root files, 7 DQM output files
• TriggerResults: found differences in 2 / 6 workflows

NVIDIA_T4 Comparison Summary

Summary:

• No significant changes to the logs found
• Reco comparison results: 41 differences found in the comparisons
• DQMHistoTests: Total files compared: 7
• DQMHistoTests: Total histograms compared: 167683
• DQMHistoTests: Total failures: 7124
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 160559
• DQMHistoTests: Total skipped: 0
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 6 files compared)
• Checked 25 log files, 20 edm output root files, 7 DQM output files
• TriggerResults: found differences in 1 / 6 workflows

Max Memory Comparisons exceeding threshold NVIDIA_T4

@cms-sw/core-l2 , I found 1 workflow step(s) with memory usage exceeding the error threshold:

Expand to see workflows ...

• Error: Workflow 34434.7503_TTbar_14TeV+Run4D121_HLTHeterogeneousValid step2 max memory diff 242.9 exceeds +/- 30.0 MiB

[akritkbehera]

@fwyzard I have integrated TheRockRocm 7.14 without much changes from our existing ROCM 7.2.4. As you mentioned since there are changes to ROCM in TheRock you would like to run some test. I have setup an separate TheRock_ROCM IBs. They should be available soon!

[fwyzard]

@fwyzard I have integrated TheRockRocm 7.14 without much changes from our existing ROCM 7.2.4. As you mentioned since there are changes to ROCM in TheRock you would like to run some test. I have setup an separate TheRock_ROCM IBs. They should be available soon!

Thanks @akritkbehera !
This week I am on vacation, next week I will try to check if there is any impact on performance.