Update CUDA to version 13.4.1 by fwyzard · Pull Request #10844 · cms-sw/cmsdist

fwyzard · 2026-09-10T14:42:47Z

Update CUDA to version 13.4.1 (general availability).

Major changes since CUDA 13.3:

add support for GCC 16 and clang 22 as host compilers;
the NVIDIA drivers are now packaged separately.

Patch the cuda-gdb shell script rather than adding a second wrapper.

The NVIDIA drivers are now packaged separately. Patch the cuda-gdb shell script rather than adding a second wrapper.

cmsbuild · 2026-09-10T14:43:16Z

A new Pull Request was created by @fwyzard for branch IB/CMSSW_20_1_X/master.

@akritkbehera, @cmsbuild, @iarspider, @raoatifshad, @smuzaffar can you please review it and eventually sign? Thanks.
@ftenchini, @mandrenguyen, @sextonkennedy you are the release manager for this.
cms-bot commands are listed here

cmsbuild · 2026-09-10T14:43:17Z

cms-bot internal usage

fwyzard · 2026-09-10T14:46:38Z

@smuzaffar @akritkbehera, if this works for GCC 14 in C++20 mode, how do I test it with GCC 15 and GCC 16 in C++23 mode ?

fwyzard · 2026-09-10T14:46:50Z

enable gpu

fwyzard · 2026-09-10T14:46:54Z

please test

fwyzard · 2026-09-10T14:56:35Z

please test for el9_amd64_gcc16 with cms-sw/cms-bot#2838

fwyzard · 2026-09-10T14:57:35Z

please test for el9_amd64_gcc15 with cms-sw/cms-bot#2838

cmsbuild · 2026-09-11T00:36:13Z

-1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-e693b7/55790/summary.html
COMMIT: 2077d12
CMSSW: CMSSW_20_1_X_2026-09-08-2300/el9_amd64_gcc16
Additional Tests: GPU,AMD_MI300X,AMD_W7900,NVIDIA_H100,NVIDIA_L40S,NVIDIA_T4
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/10844/55790/install.sh to create a dev area with all the needed externals and cmssw changes.

Failed External Build

I found compilation error when building:

cwd: /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el9_amd64_gcc16/external/py3-torch-cuda/2.13.0-c725d6eced9d91b59d33ddc3ba0b3142/external_py3-torch-cuda_2.13.0-c725d6eced9d91b59d33ddc3ba0b3142-1-build/cmsdist-pip-src/torch-2.13.0
Building wheel for torch (pyproject.toml): finished with status 'error'
ERROR: Failed building wheel for torch
Failed to build torch
ERROR: Failed to build one or more wheels
error: Bad exit status from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/tmp/rpm-tmp.xJbBlY (%build)

RPM build warnings:
Macro expanded in comment on line 674: %{pkginstroot}/bin/*

Macro expanded in comment on line 679: %{pkginstroot}/${PYTHON3_LIB_SITE_PACKAGES}

fwyzard · 2026-09-11T05:40:27Z

The failure with GCC 16 is a general CUDA issue, already reported here: https://forums.developer.nvidia.com/t/nvcc-13-4-rejects-conforming-libstdc-string-with-g-16/382852 .

cmsbuild · 2026-09-11T07:23:14Z

-1

Failed Tests: UnitTests nvidia_l40sUnitTests
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-e693b7/55789/summary.html
COMMIT: 2077d12
CMSSW: CMSSW_20_1_X_2026-09-10-1100/el9_amd64_gcc14
Additional Tests: GPU,AMD_MI300X,AMD_W7900,NVIDIA_H100,NVIDIA_L40S,NVIDIA_T4
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/10844/55789/install.sh to create a dev area with all the needed externals and cmssw changes.

Failed Unit Tests

I found 10 errors in the following unit tests:

---> test unitTestsGroup_1_u2 had ERRORS
---> test unitTestsGroup_4_u20 had ERRORS
---> test TestFWCoreIntegrationStandalone had ERRORS
and more ...

Comparison Summary

Summary:

You potentially added 8 lines to the logs
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 45
DQMHistoTests: Total histograms compared: 3592269
DQMHistoTests: Total failures: 105
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3592146
DQMHistoTests: Total skipped: 18
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 44 files compared)
Checked 195 log files, 163 edm output root files, 45 DQM output files
TriggerResults: no differences found

AMD_MI300X Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 62 differences found in the comparisons
DQMHistoTests: Total files compared: 7
DQMHistoTests: Total histograms compared: 173739
DQMHistoTests: Total failures: 8057
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 165682
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 6 files compared)
Checked 25 log files, 20 edm output root files, 7 DQM output files
TriggerResults: no differences found

AMD_W7900 Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 49 differences found in the comparisons
DQMHistoTests: Total files compared: 7
DQMHistoTests: Total histograms compared: 173739
DQMHistoTests: Total failures: 7175
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 166564
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 6 files compared)
Checked 25 log files, 20 edm output root files, 7 DQM output files
TriggerResults: no differences found

NVIDIA_H100 Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 66 differences found in the comparisons
DQMHistoTests: Total files compared: 7
DQMHistoTests: Total histograms compared: 173739
DQMHistoTests: Total failures: 8849
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 164890
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 6 files compared)
Checked 25 log files, 20 edm output root files, 7 DQM output files
TriggerResults: no differences found

NVIDIA_L40S Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 95 differences found in the comparisons
DQMHistoTests: Total files compared: 7
DQMHistoTests: Total histograms compared: 173739
DQMHistoTests: Total failures: 9167
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 164572
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 6 files compared)
Checked 25 log files, 20 edm output root files, 7 DQM output files
TriggerResults: no differences found

NVIDIA_T4 Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 85 differences found in the comparisons
DQMHistoTests: Total files compared: 7
DQMHistoTests: Total histograms compared: 173739
DQMHistoTests: Total failures: 9030
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 164709
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 6 files compared)
Checked 25 log files, 20 edm output root files, 7 DQM output files
TriggerResults: no differences found

cmsbuild · 2026-09-11T08:03:29Z

-1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-e693b7/55801/summary.html
COMMIT: 2077d12
CMSSW: CMSSW_20_1_X_2026-09-08-2300/el9_amd64_gcc15
Additional Tests: GPU,AMD_MI300X,AMD_W7900,NVIDIA_H100,NVIDIA_L40S,NVIDIA_T4
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/10844/55801/install.sh to create a dev area with all the needed externals and cmssw changes.

Failed External Build

I found compilation error when building:

cwd: /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el9_amd64_gcc15/external/py3-torch-cuda/2.13.0-69c963f7f69016829584583110c7f92b/external_py3-torch-cuda_2.13.0-69c963f7f69016829584583110c7f92b-1-build/cmsdist-pip-src/torch-2.13.0
Building wheel for torch (pyproject.toml): finished with status 'error'
ERROR: Failed building wheel for torch
Failed to build torch
ERROR: Failed to build one or more wheels
error: Bad exit status from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/tmp/rpm-tmp.JwEzSB (%build)

RPM build warnings:
Macro expanded in comment on line 674: %{pkginstroot}/bin/*

Macro expanded in comment on line 679: %{pkginstroot}/${PYTHON3_LIB_SITE_PACKAGES}

smuzaffar · 2026-09-11T08:36:51Z

torch still fails with Error: Internal Compiler Error (codegen): "unsupported float variant!" for gcc15 with c++23

fwyzard · 2026-09-11T08:45:28Z

torch still fails with Error: Internal Compiler Error (codegen): "unsupported float variant!" for gcc15 with c++23

Is that because of gcc15 or because of c++23 ?

smuzaffar · 2026-09-11T08:53:15Z

I think this is due to c++23, last time we tested cuda update with CPP23/GCC14 and we have seen the same ICE for that.

fwyzard · 2026-09-11T08:56:36Z

Would it be useful to have CUDA with gcc15 or gcc16 in c++20 mode ?

fwyzard · 2026-09-11T08:57:18Z

What I mean is that, given that CUDA still does not really support c++23, we could at least update the compiler to gcc 15 or gcc 16 and remain in c++20 mode 🤷🏻‍♂️

smuzaffar · 2026-09-11T09:34:39Z

I have no objections moving default gcc15/gcc16 to c++20. May be we can add extra CPP23 IBs for gcc15 and gcc16 just to make sure we build the sw stack with c++23 and newer compilers

fwyzard · 2026-09-11T09:38:46Z

I have no objections moving default gcc15/gcc16 to c++20.

OK, let's bring this up at the next Core Software meeting.

May be we can add extra CPP23 IBs for gcc15 and gcc16 just to make sure we build the sw stack with c++23 and newer compilers.

Yes, absolutely.

makortel · 2026-09-11T18:55:46Z

I have no objections moving default gcc15/gcc16 to c++20.

OK, let's bring this up at the next Core Software meeting.

FWIW I'm fine with the suggestion (in case you don't want to wait for the next core meeting that might take a while).

May be we can add extra CPP23 IBs for gcc15 and gcc16 just to make sure we build the sw stack with c++23 and newer compilers.

Yes, absolutely.

This sounds good to me as well.

smuzaffar · 2026-09-12T08:28:52Z

ok, I will change GCC15/16 OBs to use default c++20 and also open CPP23 IBs for GCC15/16

Update CUDA to version 13.4.1

2077d12

The NVIDIA drivers are now packaged separately. Patch the cuda-gdb shell script rather than adding a second wrapper.

cmsbuild added tests-pending externals-pending pending-signatures orp-pending labels Sep 10, 2026

cmsbuild added tests-started and removed tests-pending labels Sep 10, 2026

cmsbuild added the requires-external label Sep 10, 2026

cmsbuild added tests-rejected and removed tests-started labels Sep 11, 2026

Uh oh!

Conversation

fwyzard commented Sep 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cmsbuild commented Sep 10, 2026

Uh oh!

cmsbuild commented Sep 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fwyzard commented Sep 10, 2026

Uh oh!

fwyzard commented Sep 10, 2026

Uh oh!

fwyzard commented Sep 10, 2026

Uh oh!

fwyzard commented Sep 10, 2026

Uh oh!

fwyzard commented Sep 10, 2026

Uh oh!

cmsbuild commented Sep 11, 2026

Failed External Build

Uh oh!

fwyzard commented Sep 11, 2026

Uh oh!

cmsbuild commented Sep 11, 2026

Failed Unit Tests

Comparison Summary

AMD_MI300X Comparison Summary

AMD_W7900 Comparison Summary

NVIDIA_H100 Comparison Summary

NVIDIA_L40S Comparison Summary

NVIDIA_T4 Comparison Summary

Uh oh!

cmsbuild commented Sep 11, 2026

Failed External Build

Uh oh!

smuzaffar commented Sep 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fwyzard commented Sep 11, 2026

Uh oh!

smuzaffar commented Sep 11, 2026

Uh oh!

fwyzard commented Sep 11, 2026

Uh oh!

fwyzard commented Sep 11, 2026

Uh oh!

smuzaffar commented Sep 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fwyzard commented Sep 11, 2026

Uh oh!

makortel commented Sep 11, 2026

Uh oh!

smuzaffar commented Sep 12, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

fwyzard commented Sep 10, 2026 •

edited

Loading

cmsbuild commented Sep 10, 2026 •

edited

Loading

smuzaffar commented Sep 11, 2026 •

edited

Loading

smuzaffar commented Sep 11, 2026 •

edited

Loading