enable multi-vectorization for ucx version 1.20.1 by smuzaffar · Pull Request #10663 · cms-sw/cmsdist

smuzaffar · 2026-06-18T11:11:04Z

Follow up on #10658 , this PR fixes

multi-vectorization for UCX
Properly pass cuda gencode flags enable via out cuda-flags

cmsbuild · 2026-06-18T11:11:30Z

A new Pull Request was created by @smuzaffar for branch IB/CMSSW_20_1_X/master.

@akritkbehera, @cmsbuild, @iarspider, @raoatifshad, @smuzaffar can you please review it and eventually sign? Thanks.
@ftenchini, @mandrenguyen, @sextonkennedy you are the release manager for this.
cms-bot commands are listed here

cmsbuild · 2026-06-18T11:11:31Z

cms-bot internal usage

smuzaffar · 2026-06-18T11:13:02Z

please test

cmsbuild · 2026-06-18T11:13:11Z

Pull request #10663 was updated.

fwyzard · 2026-06-18T15:37:51Z

Thanks @smuzaffar .

I have the impression looking at the code that the ./configure for UCX is a bit weird, and even if one sets the x86-64-v2 arch it may try to check at runtime for AVX support - which should be be included not in x86-64-v2.

Anyway, let's look at the logs once the bot has completed the build.

cmsbuild · 2026-06-18T16:09:06Z

Pull request #10663 was updated.

smuzaffar · 2026-06-18T16:09:28Z

please test

build with make verbose output enabled

cmsbuild · 2026-06-19T03:54:15Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-20c18c/54082/summary.html
COMMIT: 7e23be7
CMSSW: CMSSW_20_1_X_2026-06-18-1100/el9_amd64_gcc13
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/10663/54082/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-20c18c/54082/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-20c18c/54082/git-merge-result

Comparison Summary

Summary:

You potentially added 9 lines to the logs
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 45
DQMHistoTests: Total histograms compared: 3414477
DQMHistoTests: Total failures: 20
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3414439
DQMHistoTests: Total skipped: 18
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 44 files compared)
Checked 195 log files, 163 edm output root files, 45 DQM output files

smuzaffar · 2026-06-19T06:41:42Z

@fwyzard , you are right. --with-march=x86-64-v3 via configure is not propogated to the compiler. I tested locally and the following is properly propogated to the compiler. I will update the ucx spec to use these flags.

+%ifarch x86_64
+  CFLAGS="%{selected_microarch}" \
+  CXXFLAGS="%{selected_microarch}" \
+%endif

about -g [a], it is hard coded in the build configuration of ucx. we can just remove it using sed before running confihgure

[a]

configure:             CPPFLAGS:   -DCPU_FLAGS="" -I${abs_top_srcdir}/src -I${abs_top_builddir} -I${abs_top_builddir}/src -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_amd64_gcc13/external/numactl/2.0.14-50cfd5594ba973c050bfc68aca5980c6/include
configure:               CFLAGS:   -O3 -g -Wall -Werror -funwind-tables -Wframe-larger-than=8192 -Wno-missing-field-initializers -Wno-unused-parameter -Wno-unused-label -Wno-long-long -Wno-endif-labels -Wno-sign-compare -Wno-multichar -Wno-deprecated-declarations -Winvalid-pch -Wno-pointer-sign -Werror-implicit-function-declaration -Wno-format-zero-length -Wnested-externs -Wshadow -Werror=declaration-after-statement 
configure:             CXXFLAGS:   -O3 -g -Wall -Werror -funwind-tables -Wframe-larger-than=8192 -Wno-missing-field-initializers -Wno-unused-parameter -Wno-unused-label -Wno-long-long -Wno-endif-labels -Wno-sign-compare -Wno-multichar -Wno-deprecated-declarations -Winvalid-pch 
configure:                 NVCC:   /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el9_amd64_gcc13/external/cuda/13.3.0-590e6ca5263b78d3dee75b67afe6a760/bin/nvcc
configure:            NVCCFLAGS:    -g  -I${abs_top_srcdir}/src -I${abs_top_builddir} -I${abs_top_builddir}/src -gencode=arch=compute_75,code=sm_75 -gencode=arch=compute_80,code=sm_80 -gencode=arch=compute_86,code=sm_86 -gencode=arch=compute_90,code=sm_90 -gencode=arch=compute_89,code=sm_89 -gencode=arch=compute_100,code=sm_100 -gencode=arch=compute_120,code=sm_120 -gencode=arch=compute_110,code=sm_110 -gencode=arch=compute_120,code=compute_120

smuzaffar · 2026-06-19T07:28:19Z

please test

cmsbuild · 2026-06-19T07:28:28Z

Pull request #10663 was updated.

ghyls · 2026-06-19T12:55:24Z

Hi @smuzaffar, thanks a lot. Thanks @fwyzard for the checks as well.

-g is indeed hard-coded in the UCX configuration. Let me run some tests on top of your changes to see if it removing it has any impact in performance.

cmsbuild · 2026-06-19T22:16:49Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-20c18c/54100/summary.html
COMMIT: 9b0bd3e
CMSSW: CMSSW_20_1_X_2026-06-18-2300/el9_amd64_gcc13
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/10663/54100/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

@cms-sw pyg-lib: Do not build for native arch #10664

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-20c18c/54100/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-20c18c/54100/git-merge-result

Comparison Summary

Summary:

You potentially removed 11 lines from the logs
ROOTFileChecks: Some differences in event products or their sizes found
Reco comparison results: 42774 differences found in the comparisons
DQMHistoTests: Total files compared: 45
DQMHistoTests: Total histograms compared: 3414477
DQMHistoTests: Total failures: 414640
DQMHistoTests: Total nulls: 301
DQMHistoTests: Total successes: 2999518
DQMHistoTests: Total skipped: 18
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 44 files compared)
Checked 195 log files, 163 edm output root files, 45 DQM output files

Max Memory Comparisons exceeding threshold

@cms-sw/core-l2 , I found 6 workflow step(s) with memory usage exceeding the error threshold:

Expand to see workflows ...

Error: Workflow 11634.0_TTbar_14TeV+2022 step5 max memory diff -359.8 exceeds +/- 30.0 MiB
Error: Workflow 12834.0_TTbar_14TeV+2024 step5 max memory diff -359.8 exceeds +/- 30.0 MiB
Error: Workflow 16834.0_TTbar_14TeV+2025 step5 max memory diff -313.6 exceeds +/- 30.0 MiB
Error: Workflow 18434.0_TTbar_14TeV+2026 step5 max memory diff -44.4 exceeds +/- 30.0 MiB
Error: Workflow 34634.0_TTbar_14TeV+Run4D121PU step2 max memory diff 163.9 exceeds +/- 30.0 MiB
Error: Workflow 34634.0_TTbar_14TeV+Run4D121PU step3 max memory diff -37.0 exceeds +/- 30.0 MiB

cmsbuild · 2026-06-21T17:11:38Z

Pull request #10663 was updated.

ghyls · 2026-06-21T19:00:36Z

Hi @smuzaffar and @fwyzard,

I saw no performance difference comparing with/without -g, and perhaps a tiny performance improvement when using -march=x86-64-v3 vs -march=x86-64-v2. See the various points significantly above the 1.00 line in the ratio plot below:

The size of the external in ./build/el9_amd64_gcc13/external/ucx/ is 5.5M without -g, and 32M with -g. Unless this is a problem (or there is another problem I am not aware of), I would suggest the leaving the -g flag enabled.

ghyls · 2026-06-21T19:08:56Z

Everything else, I think we can leave it as it is in this PR. Thanks @smuzaffar for the changes you made.

I checked that the basic UCX debug flags we typically use at runtime are still available with the changes this PR makes to the ./configure command. At the moment, I don't think we need a separate ucx build with the compile-time debug flags enabled.

smuzaffar · 2026-06-22T06:04:22Z

please test

cmsbuild · 2026-06-22T06:04:29Z

Pull request #10663 was updated.

cmsbuild · 2026-06-22T15:43:26Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-20c18c/54164/summary.html
COMMIT: 0cc09ea
CMSSW: CMSSW_20_1_X_2026-06-21-2300/el9_amd64_gcc13
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/10663/54164/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

@cms-sw added override_microarch_name for vectorized packages #10673

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-20c18c/54164/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-20c18c/54164/git-merge-result

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 45
DQMHistoTests: Total histograms compared: 3414477
DQMHistoTests: Total failures: 68
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3414391
DQMHistoTests: Total skipped: 18
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 44 files compared)
Checked 195 log files, 163 edm output root files, 45 DQM output files

smuzaffar · 2026-06-23T08:55:20Z

+externals

cmsbuild · 2026-06-23T08:55:45Z

This pull request is fully signed and it will be integrated in one of the next IB/CMSSW_20_1_X/master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @ftenchini, @sextonkennedy, @mandrenguyen (and backports should be raised in the release meeting by the corresponding L2)

ghyls · 2026-06-23T16:14:08Z

Thank you @smuzaffar

cmsbuild added tests-pending externals-pending pending-signatures orp-pending labels Jun 18, 2026

smuzaffar mentioned this pull request Jun 18, 2026

Add optimization flags to ucx.spec while keeping ucx portable #10658

Closed

cmsbuild added tests-started and removed tests-pending labels Jun 18, 2026

smuzaffar changed the title ~~enable multi-vectorization for ucx~~ enable multi-vectorization for ucx version 1.20.1 Jun 18, 2026

cmsbuild added tests-pending and removed tests-started labels Jun 18, 2026

cmsbuild added tests-started and removed tests-pending labels Jun 18, 2026

cmsbuild added tests-approved and removed tests-started labels Jun 19, 2026

smuzaffar force-pushed the ucx-multi-vec branch from 7e23be7 to 9b0bd3e Compare June 19, 2026 07:28

cmsbuild added tests-started and removed tests-approved labels Jun 19, 2026

cmsbuild added tests-approved tests-pending and removed tests-started tests-approved labels Jun 19, 2026

UCX version 1.20.1; enabled multi-vectorization

0cc09ea

smuzaffar force-pushed the ucx-multi-vec branch from 9b24186 to 0cc09ea Compare June 22, 2026 06:04

cmsbuild added tests-started and removed tests-pending labels Jun 22, 2026

cmsbuild added tests-approved and removed tests-started labels Jun 22, 2026

cmsbuild added externals-approved fully-signed and removed externals-pending pending-signatures labels Jun 23, 2026

smuzaffar merged commit 96b8688 into IB/CMSSW_20_1_X/master Jun 23, 2026
8 checks passed

cmsbuild mentioned this pull request Jun 23, 2026

Python packages update to get security fixes #10674

Merged

This was referenced Jun 23, 2026

PerfTools/Perfetto: in-process Perfetto tracing service cms-sw/cmssw#51271

Open

Update the NVIDIA compatibility and stub libraries for CUDA 13.3 #10678

Merged

fwyzard mentioned this pull request Jun 30, 2026

[17.0] Update UCX to version 1.21.0 #10689

Merged

smuzaffar deleted the ucx-multi-vec branch September 6, 2026 20:03

Uh oh!

Conversation

smuzaffar commented Jun 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cmsbuild commented Jun 18, 2026

Uh oh!

cmsbuild commented Jun 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

smuzaffar commented Jun 18, 2026

Uh oh!

cmsbuild commented Jun 18, 2026

Uh oh!

fwyzard commented Jun 18, 2026

Uh oh!

cmsbuild commented Jun 18, 2026

Uh oh!

smuzaffar commented Jun 18, 2026

Uh oh!

cmsbuild commented Jun 19, 2026

Comparison Summary

Uh oh!

smuzaffar commented Jun 19, 2026

Uh oh!

smuzaffar commented Jun 19, 2026

Uh oh!

cmsbuild commented Jun 19, 2026

Uh oh!

ghyls commented Jun 19, 2026

Uh oh!

cmsbuild commented Jun 19, 2026

Comparison Summary

Max Memory Comparisons exceeding threshold

Uh oh!

cmsbuild commented Jun 21, 2026

Uh oh!

ghyls commented Jun 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ghyls commented Jun 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

smuzaffar commented Jun 22, 2026

Uh oh!

cmsbuild commented Jun 22, 2026

Uh oh!

cmsbuild commented Jun 22, 2026

Comparison Summary

Uh oh!

smuzaffar commented Jun 23, 2026

Uh oh!

cmsbuild commented Jun 23, 2026

Uh oh!

Uh oh!

ghyls commented Jun 23, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

smuzaffar commented Jun 18, 2026 •

edited

Loading

cmsbuild commented Jun 18, 2026 •

edited

Loading

ghyls commented Jun 21, 2026 •

edited

Loading

ghyls commented Jun 21, 2026 •

edited

Loading