Skip to content

ci: Guard dynamic CUDART isolation #1360

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
bibrakc wants to merge 1 commit into
base: master
Choose a base branch
from
+180 −6
Open

ci: Guard dynamic CUDART isolation #1360

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 27 additions & 0 deletions .github/actions/build-plugin/action.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,9 @@ inputs:
build-type:
description: "release or debug"
required: true
cudart:
description: "static, dynamic, or none"
required: true
artifact-name:
description: "Name for the config.log artifact uploaded on failure"
required: true
Expand Down Expand Up @@ -61,6 +64,7 @@ runs:
--enable-werror=yes \
--enable-picky-compiler=yes \
--enable-platform-aws \
${{ inputs.cudart == 'dynamic' && '--enable-cudart-dynamic' || '' }} \
${{ inputs.sdk == 'neuron' && '--enable-neuron' || '--with-cuda=/usr/local/cuda' }} \
${{ inputs.build-type == 'debug' && '--enable-debug --enable-trace' || '' }} \
${{ contains(inputs.tracing, 'lttng') && '--with-lttng' || '' }} \
Expand All @@ -70,6 +74,29 @@ runs:
shell: bash
run: make V=1

- name: Verify dynamic CUDART isolation
if: inputs.cudart == 'dynamic'
shell: bash
run: |
set -euo pipefail

shopt -s nullglob
plugins=(src/.libs/libnccl*.so)
if [ ${#plugins[@]} -eq 0 ]; then
echo "No core plugin shared libraries were built" >&2
exit 1
fi

for plugin in "${plugins[@]}"; do
echo "Checking ${plugin}"

if readelf -d "${plugin}" | grep -Eq 'NEEDED.*\[libcudart\.so'; then
echo "${plugin} has a direct CUDART dependency" >&2
readelf -d "${plugin}" >&2
exit 1
fi
done

- name: Call `make check`
shell: bash
run: make check V=1 || (cat tests/unit/test-suite.log && exit 1)
Expand Down
159 changes: 153 additions & 6 deletions .github/workflows/distcheck.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,10 +30,32 @@ jobs:
build-type:
- release
- debug
cudart:
- static
- dynamic
- none
exclude:
# Neuron requires platform-aws, which requires newer EFA installer
- efainstaller: 1.25.0
sdk: neuron
# Neuron builds do not use CUDART
- sdk: neuron
cudart: static
- sdk: neuron
cudart: dynamic
# CUDA builds require a CUDART linkage mode
- sdk: cuda
cudart: none
# Keep one dynamic CUDA release build on the latest EFA installer
- sdk: cuda
cudart: dynamic
build-type: debug
- sdk: cuda
cudart: dynamic
efainstaller: 1.34.0
- sdk: cuda
cudart: dynamic
efainstaller: 1.25.0
Comment on lines +53 to +58

@hershys-aws hershys-aws Aug 20, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did you find out why we are running these versions specifically?

@bibrakc bibrakc Aug 20, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, not yet.

include:
- efainstaller: latest
platform-aws: enable
Expand All @@ -48,7 +70,7 @@ jobs:

runs-on: ubuntu-latest
container: ghcr.io/${{ github.repository }}/aws-ofi-nccl-al2023:${{ matrix.sdk }}-efa${{ matrix.efainstaller }}
name: al2023/${{ matrix.sdk }}/efa@${{ matrix.efainstaller }}/${{ matrix.build-type }}
name: al2023/${{ matrix.sdk }}/efa@${{ matrix.efainstaller }}/${{ matrix.build-type }}${{ matrix.sdk == 'cuda' && format('/cudart-{0}', matrix.cudart) || '' }}
steps:
- uses: actions/checkout@v6

Expand All @@ -67,12 +89,35 @@ jobs:
--enable-werror=yes \
--enable-picky-compiler=yes \
--${{ matrix.platform-aws }}-platform-aws \
${{ matrix.cudart == 'dynamic' && '--enable-cudart-dynamic' || '' }} \
${{ matrix.sdk == 'neuron' && '--enable-neuron' || '--with-cuda=/usr/local/cuda' }} \
${{ matrix.build-type == 'debug' && '--enable-debug --enable-trace' || '' }}

- name: Call `make`
run: make V=1

- name: Verify dynamic CUDART isolation
if: matrix.cudart == 'dynamic'
run: |
set -euo pipefail

shopt -s nullglob
plugins=(src/.libs/libnccl*.so)
if [ ${#plugins[@]} -eq 0 ]; then
echo "No core plugin shared libraries were built" >&2
exit 1
fi

for plugin in "${plugins[@]}"; do
echo "Checking ${plugin}"

if readelf -d "${plugin}" | grep -Eq 'NEEDED.*\[libcudart\.so'; then
echo "${plugin} has a direct CUDART dependency" >&2
readelf -d "${plugin}" >&2
exit 1
fi
done

- name: Call `make check`
run: make check V=1 || (cat tests/unit/test-suite.log && exit 1)

Expand Down Expand Up @@ -103,12 +148,43 @@ jobs:
build-type:
- release
- debug
cudart:
- static
- dynamic
- none
exclude:
# NVTX requires CUDA, so skip the nvtx variants on neuron.
- tracing: nvtx
sdk: neuron
- tracing: lttng-nvtx
sdk: neuron
# Neuron builds do not use CUDART
- sdk: neuron
cudart: static
- sdk: neuron
cudart: dynamic
# CUDA builds require a CUDART linkage mode
- sdk: cuda
cudart: none
# Keep one dynamic CUDA release/GCC/no-tracing build on this OS
- sdk: cuda
cudart: dynamic
build-type: debug
- sdk: cuda
cudart: dynamic
tracing: lttng
- sdk: cuda
cudart: dynamic
tracing: nvtx
- sdk: cuda
cudart: dynamic
tracing: lttng-nvtx
- sdk: cuda
cudart: dynamic
cc: clang
- sdk: cuda
cudart: dynamic
cc-variant: legacy
include:
- cc-variant: latest
cc: clang
Expand All @@ -117,7 +193,7 @@ jobs:
cc: gcc
cc-version: 13

name: u2204/${{ matrix.sdk }}/${{matrix.cc}}-${{matrix.cc-variant}}-${{matrix.tracing}}/${{ matrix.build-type }}
name: u2204/${{ matrix.sdk }}/${{matrix.cc}}-${{matrix.cc-variant}}-${{matrix.tracing}}/${{ matrix.build-type }}${{ matrix.sdk == 'cuda' && format('/cudart-{0}', matrix.cudart) || '' }}
# The nvtx and lttng-nvtx tracing variants reuse the lttng container
# image, which already has cuda-nvtx installed (added in PR #1224). Map
# any nvtx-containing variant to the lttng tag so the container reference
Expand All @@ -132,6 +208,7 @@ jobs:
sdk: ${{ matrix.sdk }}
tracing: ${{ matrix.tracing }}
build-type: ${{ matrix.build-type }}
cudart: ${{ matrix.cudart }}
artifact-name: ${{ matrix.cc }}-${{ matrix.cc-variant }}-${{ matrix.sdk }}-config.log

Ubuntu2404:
Expand All @@ -154,14 +231,48 @@ jobs:
build-type:
- release
- debug
cudart:
- static
- dynamic
- none
exclude:
# NVTX requires CUDA, so skip the nvtx variants on neuron.
- tracing: nvtx
sdk: neuron
- tracing: lttng-nvtx
sdk: neuron

name: U2404/${{ matrix.sdk }}/${{matrix.cc}}/${{matrix.tracing}}/${{ matrix.build-type }}
# Neuron builds do not use CUDART
- sdk: neuron
cudart: static
- sdk: neuron
cudart: dynamic
# CUDA builds require a CUDART linkage mode
- sdk: cuda
cudart: none
# Keep one dynamic CUDA release/GCC/no-tracing build on this OS
- sdk: cuda
cudart: dynamic
build-type: debug
- sdk: cuda
cudart: dynamic
tracing: lttng
- sdk: cuda
cudart: dynamic
tracing: nvtx
- sdk: cuda
cudart: dynamic
tracing: lttng-nvtx
- sdk: cuda
cudart: dynamic
cc: gcc-13
- sdk: cuda
cudart: dynamic
cc: clang-18
- sdk: cuda
cudart: dynamic
cc: clang-19

name: U2404/${{ matrix.sdk }}/${{matrix.cc}}/${{matrix.tracing}}/${{ matrix.build-type }}${{ matrix.sdk == 'cuda' && format('/cudart-{0}', matrix.cudart) || '' }}
# See comment on u2204 job: nvtx and lttng-nvtx reuse the lttng container.
container: ghcr.io/${{ github.repository }}/aws-ofi-nccl-ubuntu2404:${{ matrix.sdk }}-${{ matrix.cc }}-${{ (contains(matrix.tracing, 'nvtx') && 'lttng') || matrix.tracing }}-efalattest
steps:
Expand All @@ -173,6 +284,7 @@ jobs:
sdk: ${{ matrix.sdk }}
tracing: ${{ matrix.tracing }}
build-type: ${{ matrix.build-type }}
cudart: ${{ matrix.cudart }}
artifact-name: ${{ matrix.cc }}-${{ matrix.sdk }}-config.log

Ubuntu2604:
Expand All @@ -195,14 +307,48 @@ jobs:
build-type:
- release
- debug
cudart:
- static
- dynamic
- none
exclude:
# NVTX requires CUDA, so skip the nvtx variants on neuron.
- tracing: nvtx
sdk: neuron
- tracing: lttng-nvtx
sdk: neuron

name: U2604/${{ matrix.sdk }}/${{matrix.cc}}/${{matrix.tracing}}/${{ matrix.build-type }}
# Neuron builds do not use CUDART
- sdk: neuron
cudart: static
- sdk: neuron
cudart: dynamic
# CUDA builds require a CUDART linkage mode
- sdk: cuda
cudart: none
# Keep one dynamic CUDA release/GCC/no-tracing build on this OS
- sdk: cuda
cudart: dynamic
build-type: debug
- sdk: cuda
cudart: dynamic
tracing: lttng
- sdk: cuda
cudart: dynamic
tracing: nvtx
- sdk: cuda
cudart: dynamic
tracing: lttng-nvtx
- sdk: cuda
cudart: dynamic
cc: gcc-14
- sdk: cuda
cudart: dynamic
cc: clang-19
- sdk: cuda
cudart: dynamic
cc: clang-20

name: U2604/${{ matrix.sdk }}/${{matrix.cc}}/${{matrix.tracing}}/${{ matrix.build-type }}${{ matrix.sdk == 'cuda' && format('/cudart-{0}', matrix.cudart) || '' }}
# Host runner is ubuntu-24.04; the 26.04 environment comes from the container.
# See comment on u2204 job: nvtx and lttng-nvtx reuse the lttng container.
container: ghcr.io/${{ github.repository }}/aws-ofi-nccl-ubuntu2604:${{ matrix.sdk }}-${{ matrix.cc }}-${{ (contains(matrix.tracing, 'nvtx') && 'lttng') || matrix.tracing }}-efalattest
Expand All @@ -215,6 +361,7 @@ jobs:
sdk: ${{ matrix.sdk }}
tracing: ${{ matrix.tracing }}
build-type: ${{ matrix.build-type }}
cudart: ${{ matrix.cudart }}
artifact-name: ${{ matrix.cc }}-${{ matrix.sdk }}-u2604-config.log

thread-safety-check:
Expand Down
Loading