From 4af7323091eaec9e7b2c645bf510202dce44a96a Mon Sep 17 00:00:00 2001 From: jichuanh Date: Thu, 6 Aug 2026 10:58:36 -0700 Subject: [PATCH 01/11] Add multi-GPU training smoke tests with cross-socket xfail The multi-GPU workflow parametrizes each test over one device per shard, which cannot express a job that owns two GPUs at once. Nothing in CI therefore exercised multi-GPU rendering, and NVBUG#6565122 -- a heap corruption in Kit's Isaac Sim RTX renderer that only fires when the two GPUs span a cross-socket boundary -- shipped uncaught. Add smoke tests that launch real two-rank training runs and choose their GPU pair by interconnect class rather than taking whatever cuda:0,cuda:1 happens to be. On a two-socket 8-GPU host the default pick is a same-switch pair, so a fixed choice would never exercise the failing path. Each class is handled on its own terms: same-switch is strict and is the regression guard, cross-socket is expected-fail for the two Kit-renderer stacks and strict for the kitless newton_mjwarp,ovrtx stack, and the unmeasured PXB/PHB/NODE classes skip rather than assert either verdict. A physics-only case runs on any two visible devices so the step is never vacuous when every camera case skips. The expected-failure marker is applied only after the documented signature is confirmed, so an OOM or an argument error on a cross-socket pair is not recorded as the known crash. --- .github/actions/multi-gpu/multi_gpu_smoke.sh | 48 +++ .github/workflows/test-multi-gpu-pytest.yaml | 11 + .../jichuanh-mgpu-camera-smoke-xfail.rst | 9 + .../isaaclab/isaaclab/test/utils/__init__.py | 4 +- .../isaaclab/isaaclab/test/utils/devices.py | 109 +++++++ .../test_multi_gpu_training_smoke.py | 285 ++++++++++++++++++ .../isaaclab/test/utils/test_gpu_topology.py | 149 +++++++++ 7 files changed, 613 insertions(+), 2 deletions(-) create mode 100755 .github/actions/multi-gpu/multi_gpu_smoke.sh create mode 100644 source/isaaclab/changelog.d/jichuanh-mgpu-camera-smoke-xfail.rst create mode 100644 source/isaaclab/test/multi_gpu/test_multi_gpu_training_smoke.py create mode 100644 source/isaaclab/test/utils/test_gpu_topology.py diff --git a/.github/actions/multi-gpu/multi_gpu_smoke.sh b/.github/actions/multi-gpu/multi_gpu_smoke.sh new file mode 100755 index 000000000000..143032c730c4 --- /dev/null +++ b/.github/actions/multi-gpu/multi_gpu_smoke.sh @@ -0,0 +1,48 @@ +#!/usr/bin/env bash +# Copyright (c) 2022-2026, The Isaac Lab Project Developers (https://github.com/isaac-sim/IsaacLab/blob/main/CONTRIBUTORS.md). +# All rights reserved. +# +# SPDX-License-Identifier: BSD-3-Clause + +# Runs ON THE HOST runner. Launches ONE container that runs the multi-GPU +# training smoke tests, which spawn real two-rank torchrun jobs. +# +# Invoked by .github/workflows/test-multi-gpu-pytest.yaml's "Multi-GPU training +# smoke" step. Reads from the environment (set by that step): +# IMAGE_TAG — per-commit CI image to ``docker run`` +# +# Deliberately NOT sharded, unlike multi_gpu_host_launcher.sh. That launcher +# pins each shard to its own cuda:N via ISAACLAB_TEST_DEVICES, which models a +# test parametrized over a single device. A two-rank training run owns two GPUs +# at once, so a per-device shard cannot express it: the tests here pick their own +# GPU pair from ``nvidia-smi topo -m`` and pin it with CUDA_VISIBLE_DEVICES. +# +# Every GPU is exposed to the container for that reason -- the pair is selected +# inside, not by the runner. +set -euo pipefail + +: "${IMAGE_TAG:?IMAGE_TAG must be set}" + +CONTAINER="mgpu-smoke-${GITHUB_RUN_ID:-local}-${GITHUB_RUN_ATTEMPT:-1}" +TEST_PATH="source/isaaclab/test/multi_gpu/test_multi_gpu_training_smoke.py" + +cleanup() { docker rm -f "$CONTAINER" >/dev/null 2>&1 || true; } +trap cleanup EXIT + +echo "::group::GPU topology on this runner" +# Recorded because it decides which cases run: a host with no cross-socket pair +# skips the xfail case, and a host with no same-switch pair skips the strict +# camera guard. Without this the skips in the report have no visible cause. +nvidia-smi topo -m || echo "topology unavailable" +echo "::endgroup::" + +# --entrypoint bash is required: the image inherits /isaac-sim/runheadless.sh from +# the Isaac Sim base, which would swallow the command and launch Kit instead of +# running pytest. multi_gpu_host_launcher.sh overrides it for the same reason. +docker run --rm --name "$CONTAINER" \ + --entrypoint bash \ + --gpus all --network host --shm-size=16g \ + -e ACCEPT_EULA=Y -e PRIVACY_CONSENT=Y -e OMNI_KIT_ACCEPT_EULA=YES \ + -e NVIDIA_DRIVER_CAPABILITIES=all \ + "$IMAGE_TAG" \ + -lc "cd /workspace/isaaclab && ./isaaclab.sh -p -m pytest ${TEST_PATH} -v -rA --junitxml=/tmp/mgpu-smoke.xml" diff --git a/.github/workflows/test-multi-gpu-pytest.yaml b/.github/workflows/test-multi-gpu-pytest.yaml index 63c03a73779a..2ee812b291d0 100644 --- a/.github/workflows/test-multi-gpu-pytest.yaml +++ b/.github/workflows/test-multi-gpu-pytest.yaml @@ -179,6 +179,17 @@ jobs: PATHS: ${{ steps.discover.outputs.paths }} run: bash .github/actions/multi-gpu/multi_gpu_host_launcher.sh + - name: Multi-GPU training smoke + # Real two-rank training runs, outside the sharded device-parametrized + # path above: a multi-GPU job owns two GPUs at once, so a per-device + # shard cannot express it. Unsharded, all GPUs exposed; the tests pick + # their own GPU pair per interconnect class from ``nvidia-smi topo -m``. + # Runs on failure too, so a red shard does not hide smoke coverage. + if: always() + env: + IMAGE_TAG: ${{ env.CI_IMAGE_TAG }} + run: bash .github/actions/multi-gpu/multi_gpu_smoke.sh + - name: Aggregated test summary # Per-shard + per-file pass/total/walltime, plus a combined table, also # written to $GITHUB_STEP_SUMMARY. Runs on success or failure. diff --git a/source/isaaclab/changelog.d/jichuanh-mgpu-camera-smoke-xfail.rst b/source/isaaclab/changelog.d/jichuanh-mgpu-camera-smoke-xfail.rst new file mode 100644 index 000000000000..e678333879d6 --- /dev/null +++ b/source/isaaclab/changelog.d/jichuanh-mgpu-camera-smoke-xfail.rst @@ -0,0 +1,9 @@ +Added +^^^^^ + +* Added multi-GPU training smoke tests that launch real two-rank runs and select their GPU pair + by interconnect class, so cross-socket rendering is exercised instead of whichever pair happens + to be ``cuda:0,cuda:1``. The cross-socket camera case is marked ``xfail`` for NVBUG#6565122. + +* Added :func:`~isaaclab.test.utils.gpu_pairs_by_topology` to classify GPU pairs from + ``nvidia-smi topo -m`` as same-switch, cross-socket, or unknown. diff --git a/source/isaaclab/isaaclab/test/utils/__init__.py b/source/isaaclab/isaaclab/test/utils/__init__.py index c9631a080592..3fb555362a17 100644 --- a/source/isaaclab/isaaclab/test/utils/__init__.py +++ b/source/isaaclab/isaaclab/test/utils/__init__.py @@ -13,6 +13,6 @@ devices the run may use). """ -from .devices import DeviceScope, resolve_test_sim_device, test_devices +from .devices import DeviceScope, gpu_pairs_by_topology, resolve_test_sim_device, test_devices -__all__ = ["DeviceScope", "resolve_test_sim_device", "test_devices"] +__all__ = ["DeviceScope", "gpu_pairs_by_topology", "resolve_test_sim_device", "test_devices"] diff --git a/source/isaaclab/isaaclab/test/utils/devices.py b/source/isaaclab/isaaclab/test/utils/devices.py index 330964be6747..39fd66b95a22 100644 --- a/source/isaaclab/isaaclab/test/utils/devices.py +++ b/source/isaaclab/isaaclab/test/utils/devices.py @@ -81,6 +81,8 @@ def test_foo(device): ... from __future__ import annotations import os +import re +import subprocess from enum import Flag, auto _RUNTIME_DEVICES_ENV_VAR = "ISAACLAB_TEST_DEVICES" @@ -235,6 +237,113 @@ def _scope_mask(scope: str | DeviceScope) -> str: return scope if isinstance(scope, str) else scope.mask +#: How ``nvidia-smi topo -m`` link classes map to the interconnect distance that +#: multi-GPU rendering is sensitive to. ``UNKNOWN`` is not a gap in the parser -- +#: those classes are genuinely unmeasured for the defect this classification +#: exists to gate, so callers must skip rather than assume either verdict. +_TOPOLOGY_CLASS: dict[str, str] = { + "NV": "SAME_SWITCH", # NV1, NV2, ... bonded NVLinks + "PIX": "SAME_SWITCH", # at most a single PCIe bridge + "PXB": "UNKNOWN", # multiple PCIe bridges, not the host bridge + "PHB": "UNKNOWN", # traverses a PCIe host bridge + "NODE": "UNKNOWN", # between host bridges within one NUMA node + "SYS": "CROSS_SOCKET", # across the SMP interconnect between NUMA nodes +} + + +def gpu_pairs_by_topology() -> dict[str, tuple[int, int]]: + """Return one representative GPU pair per interconnect class on this host. + + Parses ``nvidia-smi topo -m`` and classifies every GPU pair by how far apart + the two devices sit. Only the first pair found per class is returned, which is + enough to parametrize a test that needs "a pair of this kind". + + Returns: + Mapping of class name (``SAME_SWITCH``, ``CROSS_SOCKET``, ``UNKNOWN``) to a + ``(index, index)`` pair. Classes with no such pair are absent. An empty + mapping means the topology could not be determined -- callers must treat + that as "skip", never as "no boundary present", so an unreadable topology + cannot silently turn a real failure into an expected one. + + Also empty on a MIG-enabled host: ``topo -m`` describes *physical* GPUs + while CUDA addresses MIG instances, so a physical index is not a device + the caller can select. + """ + if _mig_enabled(): + return {} + try: + out = subprocess.run(["nvidia-smi", "topo", "-m"], capture_output=True, text=True, timeout=30, check=False) + except (OSError, subprocess.SubprocessError): + return {} + if out.returncode != 0: + return {} + + lines = out.stdout.splitlines() + # Column count comes from the header. Rows also carry NIC and affinity columns, + # and NIC cells reuse the same tokens (NODE, SYS, PIX) -- reading past the GPU + # columns invents pairs against GPUs that do not exist. + # The header is the line listing the GPU columns: it carries several GPU + # labels and no link-class tokens (a data row carries exactly one label, + # followed by classes). + gpu_columns = 0 + for line in lines: + labels = re.findall(r"\bGPU\d+\b", line) + if len(labels) >= 2 and not re.search(r"\b(?:X|NV\d+|PIX|PXB|PHB|NODE|SYS)\b", line): + gpu_columns = len(labels) + break + if gpu_columns == 0: + return {} + + pairs: dict[str, tuple[int, int]] = {} + rows_seen: set[int] = set() + for line in lines: + # Data rows start with the GPU label; the legend and NIC rows do not. + match = re.match(r"^\s*GPU(\d+)\s+(.*)$", line) + if match is None: + continue + row = int(match.group(1)) + # A row naming a GPU the header does not list means the matrix is partial + # or inconsistent. Returning a pair from it would hand an index that is + # not a selectable device to CUDA_VISIBLE_DEVICES. + if row >= gpu_columns: + return {} + rows_seen.add(row) + # Cells run in GPU-index order; "X" marks self. + for col, cell in enumerate(match.group(2).split()[:gpu_columns]): + if col == row or cell == "X": + continue + # NVLink cells are NV1/NV2/...; every other class is a bare token. + key = "NV" if cell.startswith("NV") else cell + kind = _TOPOLOGY_CLASS.get(key) + if kind is None: + continue + pairs.setdefault(kind, (row, col) if row < col else (col, row)) + + # Require the full square: a truncated matrix can omit exactly the rows that + # carry the boundary, which would silently downgrade a cross-socket host to + # "same switch only" and turn the expected-failure case into a skip. + if len(rows_seen) != gpu_columns: + return {} + return pairs + + +def _mig_enabled() -> bool: + """Whether any GPU on this host is partitioned into MIG instances. + + Returns: + ``True`` when ``nvidia-smi -L`` lists a MIG device, and on any error -- + an unreadable device list is treated as MIG so callers skip rather than + select physical indices that may not be addressable. + """ + try: + out = subprocess.run(["nvidia-smi", "-L"], capture_output=True, text=True, timeout=30, check=False) + except (OSError, subprocess.SubprocessError): + return True + if out.returncode != 0: + return True + return "MIG " in out.stdout + + def _list_available_devices() -> list[str]: """Return the host's visible devices in mask order: ``cpu`` then ``cuda:0, cuda:1, ...``. diff --git a/source/isaaclab/test/multi_gpu/test_multi_gpu_training_smoke.py b/source/isaaclab/test/multi_gpu/test_multi_gpu_training_smoke.py new file mode 100644 index 000000000000..ed66b320e99b --- /dev/null +++ b/source/isaaclab/test/multi_gpu/test_multi_gpu_training_smoke.py @@ -0,0 +1,285 @@ +# Copyright (c) 2022-2026, The Isaac Lab Project Developers (https://github.com/isaac-sim/IsaacLab/blob/main/CONTRIBUTORS.md). +# All rights reserved. +# +# SPDX-License-Identifier: BSD-3-Clause + +"""Multi-GPU training smoke tests. + +Setup: + - none; each test launches a real two-rank training run as a subprocess +Tests: + - physics-only task on any 2 GPUs -> verify training completes + - each physics/renderer stack on a same-switch GPU pair -> verify training completes + - each physics/renderer stack on a cross-socket GPU pair -> Kit-renderer stacks + are expected to fail, NVBUG#6565122 + +Unlike the rest of the suite these are not parametrized over ``device``: a +multi-GPU run owns two devices at once, so the per-shard single-device +parametrization the multi-GPU workflow applies elsewhere does not model it. +They are driven by a dedicated workflow step instead. + +Which GPU pair a case uses is resolved from the host at runtime rather than +hardcoded. On an 8-GPU two-socket box the default ``cuda:0,cuda:1`` pick is a +*same-switch* pair and does not exercise the cross-socket path at all, so a +fixed pick would quietly stop testing the thing this file exists for. +""" + +from __future__ import annotations + +import contextlib +import os +import signal +import subprocess +import sys +import time +from pathlib import Path + +import pytest + +from isaaclab.test.utils import gpu_pairs_by_topology + +# Small on purpose: Kit boot dominates the runtime at this size, and the defect +# reproduces at 1024 envs exactly as it does at 2048. +_NUM_ENVS = "1024" +_MAX_ITERATIONS = "3" + +# A hung run goes silent with both GPUs at 0% utilisation, while a slow one keeps +# logging -- so silence, not elapsed time, is the signal. 90 s sits far above the +# observed inter-line gap for this task (~10 s for the first iteration, ~6 s +# after) and above any gap during Kit boot, which logs continuously. +_IDLE_TIMEOUT_S = 90 +# Backstop for a run that dribbles output forever. A passing run here is ~5 min +# (Kit boot ~4 min + ~30 s training), so this is ~2x headroom and caps the worst +# case a single parametrization can cost CI. +_HARD_TIMEOUT_S = 600 + +_PHYSICS_ONLY_TASK = "Isaac-Cartpole-Direct" +_CAMERA_TASK = "Isaac-Cartpole-Camera-Direct" + +# (id, presets) for each physics/renderer stack worth covering. +# +# ``isaacsim_physx,ovrtx`` is absent by design: IsaacLab rejects it, since ovrtx +# is a kitless renderer and cannot pair with Kit physics. +# +# TODO: add ``ovphysx,ovrtx`` once OvPhysX supports multi-GPU. It currently hangs +# at the first parameter sync on *any* GPU pair -- same-switch included -- so it +# is a separate defect from NVBUG#6565122 and would only cost CI a deliberate +# timeout while asserting something already known. See the process-global +# device-mode lock in ``isaaclab_ovphysx.physics.ovphysx_manager``. +_CAMERA_STACKS = [ + pytest.param("isaacsim_physx", id="isaacsim_physx-kit_rtx"), + pytest.param("newton_mjwarp,isaacsim_rtx", id="newton-kit_rtx"), + pytest.param("newton_mjwarp,ovrtx", id="newton-ovrtx"), +] + +# Stacks that drive Kit's Isaac Sim RTX renderer. These are the ones that fail on +# a cross-socket pair; the kitless ``ovrtx`` stack passes there. +_KIT_RENDERER_STACKS = frozenset({"isaacsim_physx", "newton_mjwarp,isaacsim_rtx"}) + +_XFAIL_REASON = ( + "Kit's Isaac Sim RTX renderer corrupts the host heap when a multi-GPU rendering job spans a" + " cross-socket (SYS) GPU pair, surfacing as SIGSEGV inside libcarb.cudainterop.plugin.so." + " Entered between Kit 110.0.0 and Kit 110.1.2. The same run passes on a same-switch pair, and" + " with presets=newton_mjwarp,ovrtx on the same pair. NVBUG#6565122" +) + + +def _repo_root() -> Path: + return Path(__file__).resolve().parents[4] + + +def _run_training(pair: tuple[int, int], task: str, presets: str) -> tuple[str, str]: + """Launch a two-rank training run pinned to ``pair`` and wait for it to settle. + + Streams the child's output so a stalled run is killed after + :data:`_IDLE_TIMEOUT_S` of silence rather than occupying a CI runner until the + hard timeout. + + Args: + pair: GPU indices to pin the two ranks to. + task: Gym task id to train. + presets: Value for the ``presets=`` selector (physics and/or renderer). + + Returns: + ``(outcome, output)`` where outcome is ``"passed"``, ``"failed"`` or + ``"hung"``, and output is the combined stdout/stderr captured so far. + """ + env = dict(os.environ) + # Docker's --gpus flag is not available from inside the test process, so the + # pair is selected with CUDA_VISIBLE_DEVICES. Verified to reproduce the + # canonical signature (exit 139 with cudainterop frames) rather than masking + # it behind a device-enumeration artifact. + env["CUDA_VISIBLE_DEVICES"] = f"{pair[0]},{pair[1]}" + env["PYTHONUNBUFFERED"] = "1" + + cmd = [ + sys.executable, + "scripts/reinforcement_learning/train_multigpu.py", + "--num_gpus", + "2", + # Without this only rank 0 is printed; when rank 1 is the one that dies + # the log carries no evidence of why. + "--log_all_ranks", + "--rl_library", + "rsl_rl", + "--task", + task, + f"presets={presets}", + "--num_envs", + _NUM_ENVS, + "--max_iterations", + _MAX_ITERATIONS, + ] + + started = time.monotonic() + last_output = started + lines: list[str] = [] + # Own process group: train_multigpu.py spawns torchrun, which spawns the rank + # workers. Killing only the wrapper skips its signal-forwarding handler and + # leaves Kit ranks alive holding GPU memory and the rendezvous port. + process = subprocess.Popen( + cmd, + cwd=_repo_root(), + env=env, + stdout=subprocess.PIPE, + stderr=subprocess.STDOUT, + text=True, + bufsize=1, + start_new_session=True, + ) + try: + os.set_blocking(process.stdout.fileno(), False) # type: ignore[union-attr] + while True: + line = process.stdout.readline() # type: ignore[union-attr] + now = time.monotonic() + # Checked on every iteration, not only when the pipe is quiet: a child + # that floods stdout would otherwise never reach the backstop and + # could grow ``lines`` without bound. + if now - started > _HARD_TIMEOUT_S or now - last_output > _IDLE_TIMEOUT_S: + _kill_process_group(process) + return "hung", "".join(lines) + if line: + lines.append(line) + last_output = now + elif process.poll() is not None: + break + else: + time.sleep(0.2) + lines.append(process.stdout.read() or "") # type: ignore[union-attr] + finally: + _kill_process_group(process) + + output = "".join(lines) + passed = ( + process.returncode == 0 + and "Traceback (most recent call last):" not in output + # The load-bearing check: a run that OOMs or exits early can still return 0. + and "Training time:" in output + ) + return ("passed" if passed else "failed"), output + + +def _kill_process_group(process: subprocess.Popen) -> None: + """Kill the child's whole process group and reap it; a no-op once it has exited.""" + if process.poll() is not None: + return + try: + os.killpg(os.getpgid(process.pid), signal.SIGKILL) + except (ProcessLookupError, PermissionError): + process.kill() + # Reaping is best-effort: the group is already SIGKILLed, and blocking the + # test run on an unreapable child would trade one hang for another. + with contextlib.suppress(subprocess.TimeoutExpired): + process.wait(timeout=30) + + +def _matches_known_crash(output: str) -> bool: + """Whether a failure carries the NVBUG#6565122 signature rather than some other fault. + + The crash is a SIGSEGV inside ``libcarb.cudainterop.plugin.so``. Without this + check an OOM, an argument error, or an unrelated hang on a cross-socket pair + would all be absorbed by the expected-failure marker. + """ + return "cudainterop" in output and ("exitcode : 139" in output or "Signal 11" in output) + + +def _assert_training_passed(outcome: str, output: str) -> None: + """Assert a training subprocess actually trained, not merely exited cleanly.""" + assert outcome == "passed", f"outcome={outcome}\n{output[-2000:]}" + + +def _visible_cuda_device_count() -> int: + """Return how many CUDA devices this process can address. + + Counts what CUDA exposes rather than what the host physically has, so a + ``CUDA_VISIBLE_DEVICES``-restricted runner or a MIG layout is reported as the + caller will actually see it. + """ + # Local import so collecting this module does not pull torch in before Kit. + import torch + + return torch.cuda.device_count() if torch.cuda.is_available() else 0 + + +def _pair_or_skip(kind: str) -> tuple[int, int]: + """Return a GPU pair of ``kind``, or skip with the reason it is unavailable.""" + pairs = gpu_pairs_by_topology() + if not pairs: + pytest.skip("GPU topology could not be determined from nvidia-smi topo -m") + if kind not in pairs: + pytest.skip(f"host has no {kind} GPU pair (available: {sorted(pairs)})") + return pairs[kind] + + +@pytest.mark.smoke +@pytest.mark.integration +class TestMultiGpuTrainingSmoke: + """Two-rank training smoke coverage across the stacks and interconnect classes the host offers.""" + + def test_physics_only_trains_on_any_pair(self) -> None: + """Physics-only multi-GPU training completes on the first two visible GPUs. + + This is the guard that always runs. It deliberately does NOT consult the + topology: the camera cases are gated on the host offering a pair of the + right interconnect class, so a host with an unreadable topology, a MIG + layout, or only unmeasured link classes skips all six of them. Were this + case gated too, the step would exit 0 having launched no training at all. + """ + count = _visible_cuda_device_count() + if count < 2: + pytest.skip(f"multi-GPU smoke needs 2 visible CUDA devices, host has {count}") + _assert_training_passed(*_run_training((0, 1), _PHYSICS_ONLY_TASK, "isaacsim_physx")) + + @pytest.mark.rendering + @pytest.mark.parametrize("presets", _CAMERA_STACKS) + def test_camera_trains_on_same_switch_pair(self, presets: str) -> None: + """Camera-based multi-GPU training completes when both GPUs share a PCIe switch. + + Strict for every stack. This is the regression guard: it is the + configuration NVBUG#6565122 does *not* affect, so a failure here is a new + defect rather than the known one. + """ + pair = _pair_or_skip("SAME_SWITCH") + _assert_training_passed(*_run_training(pair, _CAMERA_TASK, presets)) + + @pytest.mark.rendering + @pytest.mark.parametrize("presets", _CAMERA_STACKS) + def test_camera_trains_on_cross_socket_pair(self, presets: str, request) -> None: + """Camera-based multi-GPU training across a cross-socket GPU pair. + + Stacks driving Kit's RTX renderer are expected to fail while + NVBUG#6565122 is open; the kitless ``ovrtx`` stack must still pass, which + is what pins the defect to the renderer rather than to multi-GPU + rendering in general. The failing cases run rather than skip so a Kit fix + surfaces as XPASS instead of going unnoticed. + """ + pair = _pair_or_skip("CROSS_SOCKET") + outcome, output = _run_training(pair, _CAMERA_TASK, presets) + if presets in _KIT_RENDERER_STACKS and outcome == "failed" and _matches_known_crash(output): + # Marked only once the documented signature is confirmed, and applied + # after the run rather than as a decorator. A blanket marker would + # absorb an OOM, an argument error, or an unrelated hang on this pair + # as though it were NVBUG#6565122, and would also cover the ovrtx + # stack, which must stay strict. + pytest.xfail(_XFAIL_REASON) + _assert_training_passed(outcome, output) diff --git a/source/isaaclab/test/utils/test_gpu_topology.py b/source/isaaclab/test/utils/test_gpu_topology.py new file mode 100644 index 000000000000..dda19d904775 --- /dev/null +++ b/source/isaaclab/test/utils/test_gpu_topology.py @@ -0,0 +1,149 @@ +# Copyright (c) 2022-2026, The Isaac Lab Project Developers (https://github.com/isaac-sim/IsaacLab/blob/main/CONTRIBUTORS.md). +# All rights reserved. +# +# SPDX-License-Identifier: BSD-3-Clause + +"""Unit tests for the ``nvidia-smi topo -m`` parser behind :func:`gpu_pairs_by_topology`. + +Fixtures are verbatim output from three differently-shaped hosts, so the parser +is covered on any machine -- including single-GPU CI, where the multi-GPU smoke +tests that consume it cannot run. +""" + +from __future__ import annotations + +import subprocess +from unittest import mock + +import pytest + +from isaaclab.test.utils import gpu_pairs_by_topology + +# 4x L40, one GPU alone on socket 0. Carries a NIC column whose cells reuse the +# same link-class tokens -- reading past the GPU columns invents a pair against +# a GPU that does not exist. +_TOPO_1V3_WITH_NIC = """\t\tGPU0\tGPU1\tGPU2\tGPU3\tNIC0\tCPU Affinity\tNUMA Affinity +GPU0\t X \tSYS\tSYS\tSYS\tNODE\t0-31,64-95\t0 +GPU1\tSYS\t X \tPIX\tPIX\tSYS\t32-63,96-127\t1 +GPU2\tSYS\tPIX\t X \tPIX\tSYS\t32-63,96-127\t1 +GPU3\tSYS\tPIX\tPIX\t X \tSYS\t32-63,96-127\t1 +NIC0\tNODE\tSYS\tSYS\tSYS\t X +""" + +# 8x L40 split 4v4 across two sockets. +_TOPO_4V4 = """\t\tGPU0\tGPU1\tGPU2\tGPU3\tGPU4\tGPU5\tGPU6\tGPU7\tCPU Affinity +GPU0\t X \tPIX\tPIX\tPIX\tSYS\tSYS\tSYS\tSYS\t0-27 +GPU1\tPIX\t X \tPIX\tPIX\tSYS\tSYS\tSYS\tSYS\t0-27 +GPU2\tPIX\tPIX\t X \tPIX\tSYS\tSYS\tSYS\tSYS\t0-27 +GPU3\tPIX\tPIX\tPIX\t X \tSYS\tSYS\tSYS\tSYS\t0-27 +GPU4\tSYS\tSYS\tSYS\tSYS\t X \tPIX\tPIX\tPIX\t28-55 +GPU5\tSYS\tSYS\tSYS\tSYS\tPIX\t X \tPIX\tPIX\t28-55 +GPU6\tSYS\tSYS\tSYS\tSYS\tPIX\tPIX\t X \tPIX\t28-55 +GPU7\tSYS\tSYS\tSYS\tSYS\tPIX\tPIX\tPIX\t X \t28-55 +""" + +# 4x RTX 6000 Ada, single socket: every pair is PHB or NODE, neither of which is +# measured for the defect the classification gates. +_TOPO_SINGLE_SOCKET = """\t\tGPU0\tGPU1\tGPU2\tGPU3\tCPU Affinity\tNUMA Affinity +GPU0\t X \tPHB\tNODE\tNODE\t0-47\t0 +GPU1\tPHB\t X \tNODE\tNODE\t0-47\t0 +GPU2\tNODE\tNODE\t X \tNODE\t0-47\t0 +GPU3\tNODE\tNODE\tNODE\t X \t0-47\t0 +""" + +_TOPO_NVLINK = """\t\tGPU0\tGPU1\tCPU Affinity +GPU0\t X \tNV18\t0-95 +GPU1\tNV18\t X \t0-95 +""" + + +# A row naming a GPU the header never lists: the matrix is partial or inconsistent. +_TOPO_ROW_OUTSIDE_HEADER = """\t\tGPU0\tGPU1\tCPU Affinity +GPU0\t X \tSYS\t0-95 +GPU9\tSYS\t X \t0-95 +""" + +# Header advertises four GPUs but only two rows are present. +_TOPO_TRUNCATED = """\t\tGPU0\tGPU1\tGPU2\tGPU3\tCPU Affinity +GPU0\t X \tPIX\tPIX\tPIX\t0-27 +GPU1\tPIX\t X \tPIX\tPIX\t0-27 +""" + +_NVIDIA_SMI_L_DISCRETE = "GPU 0: NVIDIA L40 (UUID: GPU-aaa)\nGPU 1: NVIDIA L40 (UUID: GPU-bbb)\n" +_NVIDIA_SMI_L_MIG = ( + "GPU 0: NVIDIA A100-SXM4-40GB (UUID: GPU-aaa)\n" + " MIG 3g.20gb Device 0: (UUID: MIG-bbb)\n" + " MIG 3g.20gb Device 1: (UUID: MIG-ccc)\n" +) + + +def _with_topo(stdout: str, returncode: int = 0, devices: str = _NVIDIA_SMI_L_DISCRETE): + """Patch ``subprocess.run`` so the parser sees ``stdout`` as nvidia-smi output. + + ``devices`` stands in for ``nvidia-smi -L``, which the parser consults first to + rule out a MIG host. + """ + + def _run(cmd, *args, **kwargs): + out = devices if "-L" in cmd else stdout + code = 0 if "-L" in cmd else returncode + return subprocess.CompletedProcess(args=cmd, returncode=code, stdout=out, stderr="") + + return mock.patch("isaaclab.test.utils.devices.subprocess.run", side_effect=_run) + + +class TestGpuPairsByTopology: + """Classification of GPU pairs from nvidia-smi topology output.""" + + def test_two_socket_split_reports_both_classes(self) -> None: + with _with_topo(_TOPO_4V4): + assert gpu_pairs_by_topology() == {"SAME_SWITCH": (0, 1), "CROSS_SOCKET": (0, 4)} + + def test_nic_column_does_not_invent_a_gpu_pair(self) -> None: + """NIC cells carry the same tokens as GPU cells and must not be classified.""" + with _with_topo(_TOPO_1V3_WITH_NIC): + pairs = gpu_pairs_by_topology() + assert pairs == {"CROSS_SOCKET": (0, 1), "SAME_SWITCH": (1, 2)} + # The host has 4 GPUs; a pair naming index 4 would come from the NIC column. + assert all(max(pair) < 4 for pair in pairs.values()) + + def test_single_socket_reports_only_unknown(self) -> None: + """PHB/NODE are unmeasured for this defect, so neither camera case may run.""" + with _with_topo(_TOPO_SINGLE_SOCKET): + pairs = gpu_pairs_by_topology() + assert pairs == {"UNKNOWN": (0, 1)} + assert "CROSS_SOCKET" not in pairs + assert "SAME_SWITCH" not in pairs + + def test_nvlink_counts_as_same_switch(self) -> None: + with _with_topo(_TOPO_NVLINK): + assert gpu_pairs_by_topology() == {"SAME_SWITCH": (0, 1)} + + @pytest.mark.parametrize( + "stdout,returncode", + [("", 0), ("no topology here", 0), (_TOPO_4V4, 1)], + ids=["empty", "unparsable", "nvidia-smi failed"], + ) + def test_undeterminable_topology_returns_empty(self, stdout: str, returncode: int) -> None: + """Callers must skip rather than infer "no boundary present" from a parse failure.""" + with _with_topo(stdout, returncode): + assert gpu_pairs_by_topology() == {} + + def test_missing_nvidia_smi_returns_empty(self) -> None: + with mock.patch("isaaclab.test.utils.devices.subprocess.run", side_effect=FileNotFoundError): + assert gpu_pairs_by_topology() == {} + + def test_row_outside_header_returns_empty(self) -> None: + """A GPU index absent from the header must never reach CUDA_VISIBLE_DEVICES.""" + with _with_topo(_TOPO_ROW_OUTSIDE_HEADER): + assert gpu_pairs_by_topology() == {} + + def test_truncated_matrix_returns_empty(self) -> None: + """A partial matrix can omit exactly the rows carrying the boundary.""" + with _with_topo(_TOPO_TRUNCATED): + assert gpu_pairs_by_topology() == {} + + def test_mig_host_returns_empty(self) -> None: + """topo -m describes physical GPUs; CUDA addresses MIG instances.""" + with _with_topo(_TOPO_4V4, devices=_NVIDIA_SMI_L_MIG): + assert gpu_pairs_by_topology() == {} From 75c5721ac3a39c57e9e9fb02c57553a052fc6968 Mon Sep 17 00:00:00 2001 From: jichuanh Date: Fri, 7 Aug 2026 02:43:25 -0700 Subject: [PATCH 02/11] Mount the checkout into the multi-GPU smoke container The smoke step ran docker with no volume mount, so `cd /workspace/isaaclab` landed in the image's baked-in copy rather than the PR checkout. pytest collected 0 items and the step failed with 'file or directory not found'. Mount $PWD and run as the host user, matching what multi_gpu_host_launcher.sh already does for the sharded runs. Without --user the container writes root-owned files into the runner workspace. Also surface a ::warning:: when the runner's GPU topology cannot exercise a case, so a run that skips the NVBUG#6565122 coverage does not read as a normal green run. --- .github/actions/multi-gpu/multi_gpu_smoke.sh | 25 +++++++++++++++++++- 1 file changed, 24 insertions(+), 1 deletion(-) diff --git a/.github/actions/multi-gpu/multi_gpu_smoke.sh b/.github/actions/multi-gpu/multi_gpu_smoke.sh index 143032c730c4..283c0da36003 100755 --- a/.github/actions/multi-gpu/multi_gpu_smoke.sh +++ b/.github/actions/multi-gpu/multi_gpu_smoke.sh @@ -33,14 +33,37 @@ echo "::group::GPU topology on this runner" # Recorded because it decides which cases run: a host with no cross-socket pair # skips the xfail case, and a host with no same-switch pair skips the strict # camera guard. Without this the skips in the report have no visible cause. -nvidia-smi topo -m || echo "topology unavailable" +TOPO="$(nvidia-smi topo -m 2>/dev/null || true)" +echo "${TOPO:-topology unavailable}" echo "::endgroup::" +# A runner whose GPUs all share a switch cannot reproduce NVBUG#6565122 at all. +# That is a correct skip, but silently: three skips among many read as a normal +# green run, so the step would look like it covered the defect when it did not. +# Say so where the run summary shows it. +if [ -z "$TOPO" ]; then + echo "::warning::GPU topology unavailable -- multi-GPU smoke cases will skip; this run does not cover NVBUG#6565122" +elif ! grep -qw SYS <<<"$TOPO"; then + echo "::warning::No cross-socket (SYS) GPU pair on this runner -- the NVBUG#6565122 cases will skip; this run does not cover the cross-socket regression" +fi +if ! grep -qwE "PIX|NV[0-9]+" <<<"$TOPO"; then + echo "::warning::No same-switch (PIX/NVLink) GPU pair on this runner -- the strict camera regression guard will skip" +fi + # --entrypoint bash is required: the image inherits /isaac-sim/runheadless.sh from # the Isaac Sim base, which would swallow the command and launch Kit instead of # running pytest. multi_gpu_host_launcher.sh overrides it for the same reason. +# Mount the checkout and run as the host user, exactly as multi_gpu_host_launcher.sh +# does: the image's baked-in /workspace/isaaclab predates this commit, so without +# the mount pytest collects 0 items, and without --user the container writes +# root-owned files into the runner's workspace. +host_uid="$(id -u)" +host_gid="$(id -g)" docker run --rm --name "$CONTAINER" \ --entrypoint bash \ + --user "${host_uid}:${host_gid}" \ + -e USER="$(id -un)" \ + -v "$PWD:/workspace/isaaclab:rw" \ --gpus all --network host --shm-size=16g \ -e ACCEPT_EULA=Y -e PRIVACY_CONSENT=Y -e OMNI_KIT_ACCEPT_EULA=YES \ -e NVIDIA_DRIVER_CAPABILITIES=all \ From e9b2a72460271157d59e0177360063e86759b019 Mon Sep 17 00:00:00 2001 From: jichuanh Date: Fri, 7 Aug 2026 11:36:09 -0700 Subject: [PATCH 03/11] Run the multi-GPU smoke tests through the shared run-tests action The smoke step hand-rolled its own `docker run`, duplicating a fraction of the container contract that .github/actions/run-tests already provides -- and getting parts of it wrong. It had no workspace mount (so pytest collected 0 items against the image's baked-in copy), and once that was added, `--user` without a writable HOME failed on /root/.cache. Passing a file path rather than `tools` skips the conftest sharding orchestrator, so this is still a plain single pytest run on all GPUs, with no device restriction -- the tests continue to pick their own GPU pair per interconnect class. What it gains is the standard contract: writable HOME plus XDG_CACHE_HOME/XDG_DATA_HOME, the Isaac Sim cache mounts, warp cache, and junit results collected into reports/. Previously --junitxml wrote to /tmp inside a --rm container, so the smoke step contributed nothing to the aggregated test report. What remains smoke-specific is the topology report, split out into multi_gpu_topology_report.sh: the tests skip when the host offers no qualifying GPU pair, which is correct but silent, so the run summary needs to say the coverage was not exercised. Its guard now matches the matrix rows only -- `nvidia-smi topo -m` prints a legend naming every class, so grepping the whole output found SYS and PIX on a host that had neither and the warning never fired. Also reword the topology skip so "unreadable topology" is distinguishable from "no qualifying pair on this host"; the old message sent readers looking for a parsing bug that was not there. --- .github/actions/multi-gpu/multi_gpu_smoke.sh | 71 ------------------- .../multi-gpu/multi_gpu_topology_report.sh | 41 +++++++++++ .github/workflows/test-multi-gpu-pytest.yaml | 22 +++++- .../test_multi_gpu_training_smoke.py | 5 +- 4 files changed, 64 insertions(+), 75 deletions(-) delete mode 100755 .github/actions/multi-gpu/multi_gpu_smoke.sh create mode 100755 .github/actions/multi-gpu/multi_gpu_topology_report.sh diff --git a/.github/actions/multi-gpu/multi_gpu_smoke.sh b/.github/actions/multi-gpu/multi_gpu_smoke.sh deleted file mode 100755 index 283c0da36003..000000000000 --- a/.github/actions/multi-gpu/multi_gpu_smoke.sh +++ /dev/null @@ -1,71 +0,0 @@ -#!/usr/bin/env bash -# Copyright (c) 2022-2026, The Isaac Lab Project Developers (https://github.com/isaac-sim/IsaacLab/blob/main/CONTRIBUTORS.md). -# All rights reserved. -# -# SPDX-License-Identifier: BSD-3-Clause - -# Runs ON THE HOST runner. Launches ONE container that runs the multi-GPU -# training smoke tests, which spawn real two-rank torchrun jobs. -# -# Invoked by .github/workflows/test-multi-gpu-pytest.yaml's "Multi-GPU training -# smoke" step. Reads from the environment (set by that step): -# IMAGE_TAG — per-commit CI image to ``docker run`` -# -# Deliberately NOT sharded, unlike multi_gpu_host_launcher.sh. That launcher -# pins each shard to its own cuda:N via ISAACLAB_TEST_DEVICES, which models a -# test parametrized over a single device. A two-rank training run owns two GPUs -# at once, so a per-device shard cannot express it: the tests here pick their own -# GPU pair from ``nvidia-smi topo -m`` and pin it with CUDA_VISIBLE_DEVICES. -# -# Every GPU is exposed to the container for that reason -- the pair is selected -# inside, not by the runner. -set -euo pipefail - -: "${IMAGE_TAG:?IMAGE_TAG must be set}" - -CONTAINER="mgpu-smoke-${GITHUB_RUN_ID:-local}-${GITHUB_RUN_ATTEMPT:-1}" -TEST_PATH="source/isaaclab/test/multi_gpu/test_multi_gpu_training_smoke.py" - -cleanup() { docker rm -f "$CONTAINER" >/dev/null 2>&1 || true; } -trap cleanup EXIT - -echo "::group::GPU topology on this runner" -# Recorded because it decides which cases run: a host with no cross-socket pair -# skips the xfail case, and a host with no same-switch pair skips the strict -# camera guard. Without this the skips in the report have no visible cause. -TOPO="$(nvidia-smi topo -m 2>/dev/null || true)" -echo "${TOPO:-topology unavailable}" -echo "::endgroup::" - -# A runner whose GPUs all share a switch cannot reproduce NVBUG#6565122 at all. -# That is a correct skip, but silently: three skips among many read as a normal -# green run, so the step would look like it covered the defect when it did not. -# Say so where the run summary shows it. -if [ -z "$TOPO" ]; then - echo "::warning::GPU topology unavailable -- multi-GPU smoke cases will skip; this run does not cover NVBUG#6565122" -elif ! grep -qw SYS <<<"$TOPO"; then - echo "::warning::No cross-socket (SYS) GPU pair on this runner -- the NVBUG#6565122 cases will skip; this run does not cover the cross-socket regression" -fi -if ! grep -qwE "PIX|NV[0-9]+" <<<"$TOPO"; then - echo "::warning::No same-switch (PIX/NVLink) GPU pair on this runner -- the strict camera regression guard will skip" -fi - -# --entrypoint bash is required: the image inherits /isaac-sim/runheadless.sh from -# the Isaac Sim base, which would swallow the command and launch Kit instead of -# running pytest. multi_gpu_host_launcher.sh overrides it for the same reason. -# Mount the checkout and run as the host user, exactly as multi_gpu_host_launcher.sh -# does: the image's baked-in /workspace/isaaclab predates this commit, so without -# the mount pytest collects 0 items, and without --user the container writes -# root-owned files into the runner's workspace. -host_uid="$(id -u)" -host_gid="$(id -g)" -docker run --rm --name "$CONTAINER" \ - --entrypoint bash \ - --user "${host_uid}:${host_gid}" \ - -e USER="$(id -un)" \ - -v "$PWD:/workspace/isaaclab:rw" \ - --gpus all --network host --shm-size=16g \ - -e ACCEPT_EULA=Y -e PRIVACY_CONSENT=Y -e OMNI_KIT_ACCEPT_EULA=YES \ - -e NVIDIA_DRIVER_CAPABILITIES=all \ - "$IMAGE_TAG" \ - -lc "cd /workspace/isaaclab && ./isaaclab.sh -p -m pytest ${TEST_PATH} -v -rA --junitxml=/tmp/mgpu-smoke.xml" diff --git a/.github/actions/multi-gpu/multi_gpu_topology_report.sh b/.github/actions/multi-gpu/multi_gpu_topology_report.sh new file mode 100755 index 000000000000..a64f4cc2c9c0 --- /dev/null +++ b/.github/actions/multi-gpu/multi_gpu_topology_report.sh @@ -0,0 +1,41 @@ +#!/usr/bin/env bash +# Copyright (c) 2022-2026, The Isaac Lab Project Developers (https://github.com/isaac-sim/IsaacLab/blob/main/CONTRIBUTORS.md). +# All rights reserved. +# +# SPDX-License-Identifier: BSD-3-Clause + +# Record the runner's GPU interconnect topology and warn when it cannot exercise +# a multi-GPU smoke case. +# +# Invoked by .github/workflows/test-multi-gpu-pytest.yaml's "Multi-GPU topology +# report" step, immediately before the smoke tests run. +# +# Why this is a separate step: the smoke tests themselves pick their GPU pair per +# interconnect class, and skip when the host offers no qualifying pair. That is a +# correct skip, but a silent one -- skips among many read as a normal green run, +# so the step would look like it covered the defect when it did not. This makes +# the gap visible in the run summary. + +set -euo pipefail + +echo "::group::GPU topology on this runner" +TOPO="$(nvidia-smi topo -m 2>/dev/null || true)" +echo "${TOPO:-topology unavailable}" +echo "::endgroup::" + +# Match the MATRIX ROWS only (lines starting GPU). ``topo -m`` ends with a +# legend that spells out every class name, so grepping the whole output finds +# "SYS" and "PIX" on a host that has neither and the guard never fires. +TOPO_ROWS="$(grep -E '^GPU[0-9]+' <<<"$TOPO" || true)" + +if [ -z "$TOPO_ROWS" ]; then + echo "::warning::GPU topology unavailable -- multi-GPU smoke cases will skip; this run does not cover NVBUG#6565122" + exit 0 +fi + +if ! grep -qw SYS <<<"$TOPO_ROWS"; then + echo "::warning::No cross-socket (SYS) GPU pair on this runner -- the NVBUG#6565122 cases will skip; this run does not cover the cross-socket regression" +fi +if ! grep -qwE "PIX|NV[0-9]+" <<<"$TOPO_ROWS"; then + echo "::warning::No same-switch (PIX/NVLink) GPU pair on this runner -- the strict camera regression guard will skip" +fi diff --git a/.github/workflows/test-multi-gpu-pytest.yaml b/.github/workflows/test-multi-gpu-pytest.yaml index 2ee812b291d0..a0882b695a47 100644 --- a/.github/workflows/test-multi-gpu-pytest.yaml +++ b/.github/workflows/test-multi-gpu-pytest.yaml @@ -179,16 +179,32 @@ jobs: PATHS: ${{ steps.discover.outputs.paths }} run: bash .github/actions/multi-gpu/multi_gpu_host_launcher.sh + - name: Multi-GPU topology report + # Records the runner's interconnect classes and warns when it cannot + # exercise a case, so a run that legitimately skips the regression + # coverage does not read as a normal green run. + if: always() + run: bash .github/actions/multi-gpu/multi_gpu_topology_report.sh + - name: Multi-GPU training smoke # Real two-rank training runs, outside the sharded device-parametrized # path above: a multi-GPU job owns two GPUs at once, so a per-device # shard cannot express it. Unsharded, all GPUs exposed; the tests pick # their own GPU pair per interconnect class from ``nvidia-smi topo -m``. + # + # Uses the shared run-tests action rather than a bespoke ``docker run``: + # passing a file path (not ``tools``) skips the conftest sharding + # orchestrator, so this is a plain single pytest run, while still + # inheriting the standard container contract -- writable HOME, the Isaac + # Sim cache mounts, and junit results collected into ``reports/``. # Runs on failure too, so a red shard does not hide smoke coverage. if: always() - env: - IMAGE_TAG: ${{ env.CI_IMAGE_TAG }} - run: bash .github/actions/multi-gpu/multi_gpu_smoke.sh + uses: ./.github/actions/run-tests + with: + test-path: source/isaaclab/test/multi_gpu/test_multi_gpu_training_smoke.py + result-file: multi-gpu-smoke-report.xml + container-name: isaac-lab-mgpu-smoke-${{ github.run_id }}-${{ github.run_attempt }} + image-tag: ${{ env.CI_IMAGE_TAG }} - name: Aggregated test summary # Per-shard + per-file pass/total/walltime, plus a combined table, also diff --git a/source/isaaclab/test/multi_gpu/test_multi_gpu_training_smoke.py b/source/isaaclab/test/multi_gpu/test_multi_gpu_training_smoke.py index ed66b320e99b..79226b39ae28 100644 --- a/source/isaaclab/test/multi_gpu/test_multi_gpu_training_smoke.py +++ b/source/isaaclab/test/multi_gpu/test_multi_gpu_training_smoke.py @@ -225,7 +225,10 @@ def _pair_or_skip(kind: str) -> tuple[int, int]: """Return a GPU pair of ``kind``, or skip with the reason it is unavailable.""" pairs = gpu_pairs_by_topology() if not pairs: - pytest.skip("GPU topology could not be determined from nvidia-smi topo -m") + pytest.skip( + "GPU topology unreadable (nvidia-smi topo -m gave no usable matrix); " + "cannot tell whether this host has a qualifying pair" + ) if kind not in pairs: pytest.skip(f"host has no {kind} GPU pair (available: {sorted(pairs)})") return pairs[kind] From 4ba40a2fd3b3546521a81d09e6ffa86f29f86ac6 Mon Sep 17 00:00:00 2001 From: jichuanh Date: Wed, 12 Aug 2026 16:55:34 -0700 Subject: [PATCH 04/11] Fix renderer device index under non-identity CUDA_VISIBLE_DEVICES CUDA_VISIBLE_DEVICES renumbers devices for CUDA but not for the graphics stack. AppLauncher passed the same masked index to both /physics/cudaDevice and /renderer/activeGpu, so whenever the visible devices did not begin at zero the renderer requested a device outside the visible set, failed to create it, and the run aborted with CUDA error 700. Translate only the renderer index to a physical one. physics_gpu stays masked because /physics/cudaDevice is resolved by CUDA itself. Masks that already begin at zero are unaffected, and UUID or MIG masks fall back to the previous behavior since no physical index can be derived from them. --- ...fix-renderer-active-gpu-physical-index.rst | 9 ++++ source/isaaclab/isaaclab/app/app_launcher.py | 27 +++++++++- .../test/app/test_app_launcher_device.py | 54 +++++++++++++++++++ 3 files changed, 89 insertions(+), 1 deletion(-) create mode 100644 source/isaaclab/changelog.d/jichuanh-fix-renderer-active-gpu-physical-index.rst create mode 100644 source/isaaclab/test/app/test_app_launcher_device.py diff --git a/source/isaaclab/changelog.d/jichuanh-fix-renderer-active-gpu-physical-index.rst b/source/isaaclab/changelog.d/jichuanh-fix-renderer-active-gpu-physical-index.rst new file mode 100644 index 000000000000..d77b3c4a5ecf --- /dev/null +++ b/source/isaaclab/changelog.d/jichuanh-fix-renderer-active-gpu-physical-index.rst @@ -0,0 +1,9 @@ +Fixed +^^^^^ + +* Fixed rendering failing to start when ``CUDA_VISIBLE_DEVICES`` selects GPUs that do not begin at + zero, such as ``CUDA_VISIBLE_DEVICES=1,2``. Such runs aborted with ``CUDA error 700`` after + ``omni.gpu_foundation_factory`` reported "Failed to create any GPU devices". The renderer device + index is now translated to a physical index, since ``CUDA_VISIBLE_DEVICES`` renumbers devices for + CUDA but not for the graphics stack. Runs whose visible devices already begin at zero are + unaffected. diff --git a/source/isaaclab/isaaclab/app/app_launcher.py b/source/isaaclab/isaaclab/app/app_launcher.py index 83705d67b226..15bdad52b4aa 100644 --- a/source/isaaclab/isaaclab/app/app_launcher.py +++ b/source/isaaclab/isaaclab/app/app_launcher.py @@ -1065,6 +1065,29 @@ def _resolve_viewport_settings(self, launcher_args: dict): # avoid creating new stage at startup by default for performance reasons launcher_args["create_new_stage"] = False + @staticmethod + def _physical_device_id(logical_id: int) -> int: + """Map a CUDA device index to the physical index the renderer expects. + + ``CUDA_VISIBLE_DEVICES`` renumbers devices for CUDA but not for the graphics stack, so + ``cuda:1`` may be physical GPU 5. ``/renderer/activeGpu`` resolves against the unfiltered + enumeration and therefore needs the physical index. + + Args: + logical_id: CUDA device index, as reported after masking. + + Returns: + The physical device index. Falls back to ``logical_id`` when no mask is set, or when the + mask uses a non-numeric form such as ``GPU-`` or a MIG identifier. + """ + visible = [entry.strip() for entry in os.environ.get("CUDA_VISIBLE_DEVICES", "").split(",") if entry.strip()] + if not visible or logical_id >= len(visible): + return logical_id + try: + return int(visible[logical_id]) + except ValueError: + return logical_id + def _resolve_device_settings(self, launcher_args: dict): """Resolve simulation GPU device related settings.""" self.device_id = 0 @@ -1125,8 +1148,10 @@ def _resolve_device_settings(self, launcher_args: dict): # set rendering device. We do not need to set physics_gpu because it will automatically pick the same one # as the active_gpu device. Setting physics_gpu explicitly may result in a different device to be used. + # physics_gpu becomes /physics/cudaDevice, a CUDA setting, so it takes the masked index; active_gpu becomes + # /renderer/activeGpu, which the graphics stack resolves against the unmasked enumeration. launcher_args["physics_gpu"] = self.device_id - launcher_args["active_gpu"] = self.device_id + launcher_args["active_gpu"] = self._physical_device_id(self.device_id) # Defer importing torch until after SimulationApp starts. Importing # torch can import NumPy/OpenBLAS, whose at-fork handlers can crash diff --git a/source/isaaclab/test/app/test_app_launcher_device.py b/source/isaaclab/test/app/test_app_launcher_device.py new file mode 100644 index 000000000000..a3281c226f12 --- /dev/null +++ b/source/isaaclab/test/app/test_app_launcher_device.py @@ -0,0 +1,54 @@ +# Copyright (c) 2022-2026, The Isaac Lab Project Developers (https://github.com/isaac-sim/IsaacLab/blob/main/CONTRIBUTORS.md). +# All rights reserved. +# +# SPDX-License-Identifier: BSD-3-Clause + +"""Tests for mapping CUDA device indices to the physical indices the renderer expects.""" + +from isaaclab.app.app_launcher import AppLauncher + + +def test_physical_device_id_identity_mask_is_noop(monkeypatch): + """Return the index unchanged when the mask already starts at zero.""" + monkeypatch.setenv("CUDA_VISIBLE_DEVICES", "0,1,2,3") + + assert AppLauncher._physical_device_id(2) == 2 + + +def test_physical_device_id_maps_through_offset_mask(monkeypatch): + """Resolve against the mask so ``cuda:1`` becomes the second visible device.""" + monkeypatch.setenv("CUDA_VISIBLE_DEVICES", "4,5") + + assert AppLauncher._physical_device_id(1) == 5 + + +def test_physical_device_id_ignores_mask_order(monkeypatch): + """Follow the order given in the mask rather than sorting it.""" + monkeypatch.setenv("CUDA_VISIBLE_DEVICES", "7,6,5,4") + + assert AppLauncher._physical_device_id(0) == 7 + + +def test_physical_device_id_without_mask(monkeypatch): + """Return the index unchanged when no mask is set.""" + monkeypatch.delenv("CUDA_VISIBLE_DEVICES", raising=False) + + assert AppLauncher._physical_device_id(3) == 3 + + +def test_physical_device_id_falls_back_for_uuid_mask(monkeypatch): + """Fall back to the CUDA index when the mask names devices by UUID. + + ``CUDA_VISIBLE_DEVICES`` also accepts ``GPU-`` and MIG identifiers, for which no physical + index can be derived. Those runs keep the previous behavior instead of failing. + """ + monkeypatch.setenv("CUDA_VISIBLE_DEVICES", "GPU-0d1e2f3a,GPU-4b5c6d7e") + + assert AppLauncher._physical_device_id(1) == 1 + + +def test_physical_device_id_falls_back_when_index_exceeds_mask(monkeypatch): + """Fall back when the index is outside the mask rather than raising.""" + monkeypatch.setenv("CUDA_VISIBLE_DEVICES", "2,3") + + assert AppLauncher._physical_device_id(5) == 5 From 77ea288f4c10ac63cc120d365aa18a84ac21df85 Mon Sep 17 00:00:00 2001 From: jichuanh Date: Thu, 13 Aug 2026 14:34:47 -0700 Subject: [PATCH 05/11] Select the renderer device by CUDA index /renderer/activeGpu indexes the graphics device list, which CUDA_VISIBLE_DEVICES does not filter, so passing the CUDA device index there selected the wrong GPU whenever the visible devices did not start at zero. The renderer then failed to create a device and the run aborted with CUDA error 700 from a Warp launch. Select the device through /renderer/multiGpu/activeCudaGpus instead, which takes CUDA indices and is translated by the renderer itself, and stop setting activeGpu -- the translation is only applied when no explicit graphics index is given. physics_gpu is unchanged, since /physics/cudaDevice is resolved by CUDA. --- ...fix-renderer-active-gpu-physical-index.rst | 5 +- source/isaaclab/isaaclab/app/app_launcher.py | 41 ++++-------- .../test/app/test_app_launcher_device.py | 65 ++++++++++--------- 3 files changed, 51 insertions(+), 60 deletions(-) diff --git a/source/isaaclab/changelog.d/jichuanh-fix-renderer-active-gpu-physical-index.rst b/source/isaaclab/changelog.d/jichuanh-fix-renderer-active-gpu-physical-index.rst index d77b3c4a5ecf..bd7fd354a090 100644 --- a/source/isaaclab/changelog.d/jichuanh-fix-renderer-active-gpu-physical-index.rst +++ b/source/isaaclab/changelog.d/jichuanh-fix-renderer-active-gpu-physical-index.rst @@ -4,6 +4,7 @@ Fixed * Fixed rendering failing to start when ``CUDA_VISIBLE_DEVICES`` selects GPUs that do not begin at zero, such as ``CUDA_VISIBLE_DEVICES=1,2``. Such runs aborted with ``CUDA error 700`` after ``omni.gpu_foundation_factory`` reported "Failed to create any GPU devices". The renderer device - index is now translated to a physical index, since ``CUDA_VISIBLE_DEVICES`` renumbers devices for - CUDA but not for the graphics stack. Runs whose visible devices already begin at zero are + is now selected through ``/renderer/multiGpu/activeCudaGpus``, which takes a CUDA device index, + instead of ``/renderer/activeGpu``, which indexes the graphics device list that + ``CUDA_VISIBLE_DEVICES`` does not filter. Runs whose visible devices already begin at zero are unaffected. diff --git a/source/isaaclab/isaaclab/app/app_launcher.py b/source/isaaclab/isaaclab/app/app_launcher.py index 15bdad52b4aa..b4780ed9ccf9 100644 --- a/source/isaaclab/isaaclab/app/app_launcher.py +++ b/source/isaaclab/isaaclab/app/app_launcher.py @@ -721,6 +721,7 @@ def add_app_launcher_args(parser: argparse.ArgumentParser) -> None: "fast_shutdown": [bool], "limit_cpu_threads": [int], "experience": [str], + "extra_args": [list, type(None)], } """A dictionary containing the type of arguments passed to SimulationApp. @@ -1065,29 +1066,6 @@ def _resolve_viewport_settings(self, launcher_args: dict): # avoid creating new stage at startup by default for performance reasons launcher_args["create_new_stage"] = False - @staticmethod - def _physical_device_id(logical_id: int) -> int: - """Map a CUDA device index to the physical index the renderer expects. - - ``CUDA_VISIBLE_DEVICES`` renumbers devices for CUDA but not for the graphics stack, so - ``cuda:1`` may be physical GPU 5. ``/renderer/activeGpu`` resolves against the unfiltered - enumeration and therefore needs the physical index. - - Args: - logical_id: CUDA device index, as reported after masking. - - Returns: - The physical device index. Falls back to ``logical_id`` when no mask is set, or when the - mask uses a non-numeric form such as ``GPU-`` or a MIG identifier. - """ - visible = [entry.strip() for entry in os.environ.get("CUDA_VISIBLE_DEVICES", "").split(",") if entry.strip()] - if not visible or logical_id >= len(visible): - return logical_id - try: - return int(visible[logical_id]) - except ValueError: - return logical_id - def _resolve_device_settings(self, launcher_args: dict): """Resolve simulation GPU device related settings.""" self.device_id = 0 @@ -1146,12 +1124,19 @@ def _resolve_device_settings(self, launcher_args: dict): # pass command line variable to kit sys.argv.append(f"--/plugins/carb.tasking.plugin/threadCount={num_threads_per_process}") - # set rendering device. We do not need to set physics_gpu because it will automatically pick the same one - # as the active_gpu device. Setting physics_gpu explicitly may result in a different device to be used. - # physics_gpu becomes /physics/cudaDevice, a CUDA setting, so it takes the masked index; active_gpu becomes - # /renderer/activeGpu, which the graphics stack resolves against the unmasked enumeration. + # Set the rendering device. ``/physics/cudaDevice`` is resolved by CUDA, so the masked index is + # correct there. ``/renderer/activeGpu`` instead indexes the graphics device list, which + # ``CUDA_VISIBLE_DEVICES`` does not filter, so the same index selects the wrong GPU whenever the + # visible devices do not start at zero. ``/renderer/multiGpu/activeCudaGpus`` takes CUDA indices + # and the renderer translates them itself, so select the device through that instead and leave + # ``activeGpu`` unset -- the translation is only applied when no explicit graphics index is given. launcher_args["physics_gpu"] = self.device_id - launcher_args["active_gpu"] = self._physical_device_id(self.device_id) + launcher_args["active_gpu"] = None + extra_args = list(launcher_args.get("extra_args") or []) + # Trailing comma: the setting is parsed as a comma-separated string, and a bare integer is + # silently ignored. + extra_args.append(f"--/renderer/multiGpu/activeCudaGpus={self.device_id},") + launcher_args["extra_args"] = extra_args # Defer importing torch until after SimulationApp starts. Importing # torch can import NumPy/OpenBLAS, whose at-fork handlers can crash diff --git a/source/isaaclab/test/app/test_app_launcher_device.py b/source/isaaclab/test/app/test_app_launcher_device.py index a3281c226f12..eb394e598bfc 100644 --- a/source/isaaclab/test/app/test_app_launcher_device.py +++ b/source/isaaclab/test/app/test_app_launcher_device.py @@ -3,52 +3,57 @@ # # SPDX-License-Identifier: BSD-3-Clause -"""Tests for mapping CUDA device indices to the physical indices the renderer expects.""" - -from isaaclab.app.app_launcher import AppLauncher +"""Tests for selecting the renderer device by CUDA index.""" +import pytest -def test_physical_device_id_identity_mask_is_noop(monkeypatch): - """Return the index unchanged when the mask already starts at zero.""" - monkeypatch.setenv("CUDA_VISIBLE_DEVICES", "0,1,2,3") +from isaaclab.app.app_launcher import AppLauncher - assert AppLauncher._physical_device_id(2) == 2 +def _resolve(launcher_args: dict) -> dict: + """Run device resolution without constructing an ``AppLauncher``.""" + launcher = AppLauncher.__new__(AppLauncher) + launcher.device_id = 0 + launcher._deferred_cuda_device_id = None + launcher._xr = False + AppLauncher._resolve_device_settings(launcher, launcher_args) + return launcher_args -def test_physical_device_id_maps_through_offset_mask(monkeypatch): - """Resolve against the mask so ``cuda:1`` becomes the second visible device.""" - monkeypatch.setenv("CUDA_VISIBLE_DEVICES", "4,5") - assert AppLauncher._physical_device_id(1) == 5 +def test_active_gpu_is_not_set(): + """Leave ``activeGpu`` unset so the renderer applies the CUDA index translation.""" + args = _resolve({"device": "cuda:0"}) + assert args["active_gpu"] is None -def test_physical_device_id_ignores_mask_order(monkeypatch): - """Follow the order given in the mask rather than sorting it.""" - monkeypatch.setenv("CUDA_VISIBLE_DEVICES", "7,6,5,4") - assert AppLauncher._physical_device_id(0) == 7 +def test_renderer_selected_by_cuda_index(): + """Select the renderer device through the CUDA-indexed setting.""" + args = _resolve({"device": "cuda:1"}) + assert "--/renderer/multiGpu/activeCudaGpus=1," in args["extra_args"] -def test_physical_device_id_without_mask(monkeypatch): - """Return the index unchanged when no mask is set.""" - monkeypatch.delenv("CUDA_VISIBLE_DEVICES", raising=False) - assert AppLauncher._physical_device_id(3) == 3 +def test_physics_keeps_the_cuda_index(): + """Keep the masked index for physics, which CUDA resolves itself.""" + args = _resolve({"device": "cuda:1"}) + assert args["physics_gpu"] == 1 -def test_physical_device_id_falls_back_for_uuid_mask(monkeypatch): - """Fall back to the CUDA index when the mask names devices by UUID. - ``CUDA_VISIBLE_DEVICES`` also accepts ``GPU-`` and MIG identifiers, for which no physical - index can be derived. Those runs keep the previous behavior instead of failing. - """ - monkeypatch.setenv("CUDA_VISIBLE_DEVICES", "GPU-0d1e2f3a,GPU-4b5c6d7e") +@pytest.mark.parametrize("device", ["cuda:0", "cuda:3"]) +def test_cuda_index_setting_is_comma_terminated(device): + """Terminate the value with a comma: a bare integer is silently ignored by the renderer.""" + args = _resolve({"device": device}) - assert AppLauncher._physical_device_id(1) == 1 + cuda_gpu_args = [arg for arg in args["extra_args"] if "activeCudaGpus" in arg] + assert len(cuda_gpu_args) == 1 + assert cuda_gpu_args[0].endswith(",") -def test_physical_device_id_falls_back_when_index_exceeds_mask(monkeypatch): - """Fall back when the index is outside the mask rather than raising.""" - monkeypatch.setenv("CUDA_VISIBLE_DEVICES", "2,3") +def test_user_extra_args_are_preserved(): + """Append to caller-provided ``extra_args`` rather than replacing them.""" + args = _resolve({"device": "cuda:0", "extra_args": ["--/app/fastShutdown=False"]}) - assert AppLauncher._physical_device_id(5) == 5 + assert "--/app/fastShutdown=False" in args["extra_args"] + assert any("activeCudaGpus" in arg for arg in args["extra_args"]) From 33d65ab615d03f7ba07be8674c18f5d78d2d10b7 Mon Sep 17 00:00:00 2001 From: jichuanh Date: Thu, 13 Aug 2026 14:46:40 -0700 Subject: [PATCH 06/11] Leave activeGpu unset and fold device tests into the argv suite Setting active_gpu to None was redundant: SimulationApp already defaults it to None and only emits --/renderer/activeGpu when it is set, so not assigning it leaves the renderer at its own -1 default, which is what the CUDA index translation requires. It also stops overriding an active_gpu passed by the caller. Move the device selection tests into the existing app launcher argv suite, which already covers the arguments handed to Kit. --- source/isaaclab/isaaclab/app/app_launcher.py | 1 - .../test/app/test_app_launcher_argv.py | 55 ++++++++++++++++- .../test/app/test_app_launcher_device.py | 59 ------------------- 3 files changed, 53 insertions(+), 62 deletions(-) delete mode 100644 source/isaaclab/test/app/test_app_launcher_device.py diff --git a/source/isaaclab/isaaclab/app/app_launcher.py b/source/isaaclab/isaaclab/app/app_launcher.py index b4780ed9ccf9..0899c93639b8 100644 --- a/source/isaaclab/isaaclab/app/app_launcher.py +++ b/source/isaaclab/isaaclab/app/app_launcher.py @@ -1131,7 +1131,6 @@ def _resolve_device_settings(self, launcher_args: dict): # and the renderer translates them itself, so select the device through that instead and leave # ``activeGpu`` unset -- the translation is only applied when no explicit graphics index is given. launcher_args["physics_gpu"] = self.device_id - launcher_args["active_gpu"] = None extra_args = list(launcher_args.get("extra_args") or []) # Trailing comma: the setting is parsed as a comma-separated string, and a bare integer is # silently ignored. diff --git a/source/isaaclab/test/app/test_app_launcher_argv.py b/source/isaaclab/test/app/test_app_launcher_argv.py index a65068210761..e2a6e9631abb 100644 --- a/source/isaaclab/test/app/test_app_launcher_argv.py +++ b/source/isaaclab/test/app/test_app_launcher_argv.py @@ -3,11 +3,13 @@ # # SPDX-License-Identifier: BSD-3-Clause -"""Tests for filtering command-line arguments before Kit startup.""" +"""Tests for the command-line arguments passed to Kit at startup.""" import sys -from isaaclab.app.app_launcher import _sanitize_sys_argv_for_kit +import pytest + +from isaaclab.app.app_launcher import AppLauncher, _sanitize_sys_argv_for_kit def test_sanitize_sys_argv_removes_trailing_pytest_verbosity(monkeypatch): @@ -36,3 +38,52 @@ def test_sanitize_sys_argv_removes_pytest_marker_pair(monkeypatch): result = _sanitize_sys_argv_for_kit(["test_script.py", "-m", "not isaacsim_ci", "--keep"]) assert result == ["test_script.py", "--keep"] + + +def _resolve_device(launcher_args: dict) -> dict: + """Run device resolution without constructing an ``AppLauncher``.""" + launcher = AppLauncher.__new__(AppLauncher) + launcher.device_id = 0 + launcher._deferred_cuda_device_id = None + launcher._xr = False + AppLauncher._resolve_device_settings(launcher, launcher_args) + return launcher_args + + +def test_renderer_device_selected_by_cuda_index(): + """Select the renderer device through the CUDA-indexed setting.""" + args = _resolve_device({"device": "cuda:1"}) + + assert "--/renderer/multiGpu/activeCudaGpus=1," in args["extra_args"] + + +def test_active_gpu_is_left_unset(): + """Leave ``activeGpu`` unset: the renderer only applies the CUDA translation without it.""" + args = _resolve_device({"device": "cuda:1"}) + + assert args.get("active_gpu") is None + + +def test_physics_keeps_the_cuda_index(): + """Keep the CUDA index for physics, which CUDA resolves itself.""" + args = _resolve_device({"device": "cuda:1"}) + + assert args["physics_gpu"] == 1 + + +@pytest.mark.parametrize("device", ["cuda:0", "cuda:3"]) +def test_cuda_index_setting_is_comma_terminated(device): + """Terminate the value with a comma: a bare integer is silently ignored by the renderer.""" + args = _resolve_device({"device": device}) + + cuda_gpu_args = [arg for arg in args["extra_args"] if "activeCudaGpus" in arg] + assert len(cuda_gpu_args) == 1 + assert cuda_gpu_args[0].endswith(",") + + +def test_user_extra_args_are_preserved(): + """Append to caller-provided ``extra_args`` rather than replacing them.""" + args = _resolve_device({"device": "cuda:0", "extra_args": ["--/app/fastShutdown=False"]}) + + assert "--/app/fastShutdown=False" in args["extra_args"] + assert any("activeCudaGpus" in arg for arg in args["extra_args"]) diff --git a/source/isaaclab/test/app/test_app_launcher_device.py b/source/isaaclab/test/app/test_app_launcher_device.py deleted file mode 100644 index eb394e598bfc..000000000000 --- a/source/isaaclab/test/app/test_app_launcher_device.py +++ /dev/null @@ -1,59 +0,0 @@ -# Copyright (c) 2022-2026, The Isaac Lab Project Developers (https://github.com/isaac-sim/IsaacLab/blob/main/CONTRIBUTORS.md). -# All rights reserved. -# -# SPDX-License-Identifier: BSD-3-Clause - -"""Tests for selecting the renderer device by CUDA index.""" - -import pytest - -from isaaclab.app.app_launcher import AppLauncher - - -def _resolve(launcher_args: dict) -> dict: - """Run device resolution without constructing an ``AppLauncher``.""" - launcher = AppLauncher.__new__(AppLauncher) - launcher.device_id = 0 - launcher._deferred_cuda_device_id = None - launcher._xr = False - AppLauncher._resolve_device_settings(launcher, launcher_args) - return launcher_args - - -def test_active_gpu_is_not_set(): - """Leave ``activeGpu`` unset so the renderer applies the CUDA index translation.""" - args = _resolve({"device": "cuda:0"}) - - assert args["active_gpu"] is None - - -def test_renderer_selected_by_cuda_index(): - """Select the renderer device through the CUDA-indexed setting.""" - args = _resolve({"device": "cuda:1"}) - - assert "--/renderer/multiGpu/activeCudaGpus=1," in args["extra_args"] - - -def test_physics_keeps_the_cuda_index(): - """Keep the masked index for physics, which CUDA resolves itself.""" - args = _resolve({"device": "cuda:1"}) - - assert args["physics_gpu"] == 1 - - -@pytest.mark.parametrize("device", ["cuda:0", "cuda:3"]) -def test_cuda_index_setting_is_comma_terminated(device): - """Terminate the value with a comma: a bare integer is silently ignored by the renderer.""" - args = _resolve({"device": device}) - - cuda_gpu_args = [arg for arg in args["extra_args"] if "activeCudaGpus" in arg] - assert len(cuda_gpu_args) == 1 - assert cuda_gpu_args[0].endswith(",") - - -def test_user_extra_args_are_preserved(): - """Append to caller-provided ``extra_args`` rather than replacing them.""" - args = _resolve({"device": "cuda:0", "extra_args": ["--/app/fastShutdown=False"]}) - - assert "--/app/fastShutdown=False" in args["extra_args"] - assert any("activeCudaGpus" in arg for arg in args["extra_args"]) From 6940a1b1bac3d6927ea8727887cf4592ba3aa24e Mon Sep 17 00:00:00 2001 From: jichuanh Date: Thu, 13 Aug 2026 16:37:11 -0700 Subject: [PATCH 07/11] Select smoke GPUs by device order and split by renderer The camera cases picked their GPU pair by interconnect class, which never selected anything: the multi-GPU runner is four GPUs on one NUMA node with every pair PHB, so it has neither a same-switch nor a cross-socket pair and all six cases skipped. The class also does not decide the outcome -- a pair is exercised by whether CUDA_VISIBLE_DEVICES reorders it, since that renumbers devices for CUDA but not for the graphics stack. Pick the devices by order instead: one case with no mask, one exposing four GPUs as 3,1,2,0 so no rank's CUDA index matches its graphics index. Both run on any four-GPU host, including the runner. Drop gpu_pairs_by_topology() and its tests with the classification, and the workflow's topology report with them. Separately, the parser returned nothing on any real host: nvidia-smi underlines the header even into a pipe, so \bGPU0\b matched nothing, the header came up a column short and the last row tripped the partial-matrix guard. Split the stacks across two steps by what each needs installed, since ovrtx ships in neither image and would otherwise fail the whole step. --- .../multi-gpu/multi_gpu_topology_report.sh | 41 ---- .github/workflows/test-multi-gpu-pytest.yaml | 39 ++-- .../isaaclab/isaaclab/test/utils/__init__.py | 4 +- .../isaaclab/isaaclab/test/utils/devices.py | 105 --------- .../test_multi_gpu_training_smoke.py | 202 +++++++++--------- .../isaaclab/test/utils/test_gpu_topology.py | 149 ------------- 6 files changed, 134 insertions(+), 406 deletions(-) delete mode 100755 .github/actions/multi-gpu/multi_gpu_topology_report.sh delete mode 100644 source/isaaclab/test/utils/test_gpu_topology.py diff --git a/.github/actions/multi-gpu/multi_gpu_topology_report.sh b/.github/actions/multi-gpu/multi_gpu_topology_report.sh deleted file mode 100755 index a64f4cc2c9c0..000000000000 --- a/.github/actions/multi-gpu/multi_gpu_topology_report.sh +++ /dev/null @@ -1,41 +0,0 @@ -#!/usr/bin/env bash -# Copyright (c) 2022-2026, The Isaac Lab Project Developers (https://github.com/isaac-sim/IsaacLab/blob/main/CONTRIBUTORS.md). -# All rights reserved. -# -# SPDX-License-Identifier: BSD-3-Clause - -# Record the runner's GPU interconnect topology and warn when it cannot exercise -# a multi-GPU smoke case. -# -# Invoked by .github/workflows/test-multi-gpu-pytest.yaml's "Multi-GPU topology -# report" step, immediately before the smoke tests run. -# -# Why this is a separate step: the smoke tests themselves pick their GPU pair per -# interconnect class, and skip when the host offers no qualifying pair. That is a -# correct skip, but a silent one -- skips among many read as a normal green run, -# so the step would look like it covered the defect when it did not. This makes -# the gap visible in the run summary. - -set -euo pipefail - -echo "::group::GPU topology on this runner" -TOPO="$(nvidia-smi topo -m 2>/dev/null || true)" -echo "${TOPO:-topology unavailable}" -echo "::endgroup::" - -# Match the MATRIX ROWS only (lines starting GPU). ``topo -m`` ends with a -# legend that spells out every class name, so grepping the whole output finds -# "SYS" and "PIX" on a host that has neither and the guard never fires. -TOPO_ROWS="$(grep -E '^GPU[0-9]+' <<<"$TOPO" || true)" - -if [ -z "$TOPO_ROWS" ]; then - echo "::warning::GPU topology unavailable -- multi-GPU smoke cases will skip; this run does not cover NVBUG#6565122" - exit 0 -fi - -if ! grep -qw SYS <<<"$TOPO_ROWS"; then - echo "::warning::No cross-socket (SYS) GPU pair on this runner -- the NVBUG#6565122 cases will skip; this run does not cover the cross-socket regression" -fi -if ! grep -qwE "PIX|NV[0-9]+" <<<"$TOPO_ROWS"; then - echo "::warning::No same-switch (PIX/NVLink) GPU pair on this runner -- the strict camera regression guard will skip" -fi diff --git a/.github/workflows/test-multi-gpu-pytest.yaml b/.github/workflows/test-multi-gpu-pytest.yaml index a0882b695a47..458ca093c90a 100644 --- a/.github/workflows/test-multi-gpu-pytest.yaml +++ b/.github/workflows/test-multi-gpu-pytest.yaml @@ -179,18 +179,11 @@ jobs: PATHS: ${{ steps.discover.outputs.paths }} run: bash .github/actions/multi-gpu/multi_gpu_host_launcher.sh - - name: Multi-GPU topology report - # Records the runner's interconnect classes and warns when it cannot - # exercise a case, so a run that legitimately skips the regression - # coverage does not read as a normal green run. - if: always() - run: bash .github/actions/multi-gpu/multi_gpu_topology_report.sh - - - name: Multi-GPU training smoke - # Real two-rank training runs, outside the sharded device-parametrized - # path above: a multi-GPU job owns two GPUs at once, so a per-device - # shard cannot express it. Unsharded, all GPUs exposed; the tests pick - # their own GPU pair per interconnect class from ``nvidia-smi topo -m``. + - name: Multi-GPU training smoke (Kit renderer) + # Real multi-rank training runs, outside the sharded device-parametrized + # path above: a multi-GPU job owns several GPUs at once, so a per-device + # shard cannot express it. Unsharded, all GPUs exposed; the tests choose + # their own device order. # # Uses the shared run-tests action rather than a bespoke ``docker run``: # passing a file path (not ``tools``) skips the conftest sharding @@ -205,6 +198,28 @@ jobs: result-file: multi-gpu-smoke-report.xml container-name: isaac-lab-mgpu-smoke-${{ github.run_id }}-${{ github.run_attempt }} image-tag: ${{ env.CI_IMAGE_TAG }} + pytest-options: -m "not kitless" + + - name: Resolve OV pins + # The kitless renderer ships as an optional wheel that is in neither the + # Isaac Sim nor the Isaac Lab image, so its stack needs it installed first. + if: always() + id: ov_pins + uses: ./.github/actions/resolve-ov-pins + + - name: Multi-GPU training smoke (kitless renderer) + # Same image and same tests as the step above, with ovrtx installed and the + # marker inverted. Split into its own step so a missing optional wheel cannot + # take down the Kit-renderer coverage, which is what the fix under test guards. + if: always() + uses: ./.github/actions/run-tests + with: + test-path: source/isaaclab/test/multi_gpu/test_multi_gpu_training_smoke.py + result-file: multi-gpu-smoke-kitless-report.xml + container-name: isaac-lab-mgpu-smoke-kitless-${{ github.run_id }}-${{ github.run_attempt }} + image-tag: ${{ env.CI_IMAGE_TAG }} + extra-pip-packages: ${{ steps.ov_pins.outputs.ovrtx }} + pytest-options: -m kitless - name: Aggregated test summary # Per-shard + per-file pass/total/walltime, plus a combined table, also diff --git a/source/isaaclab/isaaclab/test/utils/__init__.py b/source/isaaclab/isaaclab/test/utils/__init__.py index 3fb555362a17..c9631a080592 100644 --- a/source/isaaclab/isaaclab/test/utils/__init__.py +++ b/source/isaaclab/isaaclab/test/utils/__init__.py @@ -13,6 +13,6 @@ devices the run may use). """ -from .devices import DeviceScope, gpu_pairs_by_topology, resolve_test_sim_device, test_devices +from .devices import DeviceScope, resolve_test_sim_device, test_devices -__all__ = ["DeviceScope", "gpu_pairs_by_topology", "resolve_test_sim_device", "test_devices"] +__all__ = ["DeviceScope", "resolve_test_sim_device", "test_devices"] diff --git a/source/isaaclab/isaaclab/test/utils/devices.py b/source/isaaclab/isaaclab/test/utils/devices.py index 39fd66b95a22..3bf6cb2cf690 100644 --- a/source/isaaclab/isaaclab/test/utils/devices.py +++ b/source/isaaclab/isaaclab/test/utils/devices.py @@ -81,8 +81,6 @@ def test_foo(device): ... from __future__ import annotations import os -import re -import subprocess from enum import Flag, auto _RUNTIME_DEVICES_ENV_VAR = "ISAACLAB_TEST_DEVICES" @@ -241,109 +239,6 @@ def _scope_mask(scope: str | DeviceScope) -> str: #: multi-GPU rendering is sensitive to. ``UNKNOWN`` is not a gap in the parser -- #: those classes are genuinely unmeasured for the defect this classification #: exists to gate, so callers must skip rather than assume either verdict. -_TOPOLOGY_CLASS: dict[str, str] = { - "NV": "SAME_SWITCH", # NV1, NV2, ... bonded NVLinks - "PIX": "SAME_SWITCH", # at most a single PCIe bridge - "PXB": "UNKNOWN", # multiple PCIe bridges, not the host bridge - "PHB": "UNKNOWN", # traverses a PCIe host bridge - "NODE": "UNKNOWN", # between host bridges within one NUMA node - "SYS": "CROSS_SOCKET", # across the SMP interconnect between NUMA nodes -} - - -def gpu_pairs_by_topology() -> dict[str, tuple[int, int]]: - """Return one representative GPU pair per interconnect class on this host. - - Parses ``nvidia-smi topo -m`` and classifies every GPU pair by how far apart - the two devices sit. Only the first pair found per class is returned, which is - enough to parametrize a test that needs "a pair of this kind". - - Returns: - Mapping of class name (``SAME_SWITCH``, ``CROSS_SOCKET``, ``UNKNOWN``) to a - ``(index, index)`` pair. Classes with no such pair are absent. An empty - mapping means the topology could not be determined -- callers must treat - that as "skip", never as "no boundary present", so an unreadable topology - cannot silently turn a real failure into an expected one. - - Also empty on a MIG-enabled host: ``topo -m`` describes *physical* GPUs - while CUDA addresses MIG instances, so a physical index is not a device - the caller can select. - """ - if _mig_enabled(): - return {} - try: - out = subprocess.run(["nvidia-smi", "topo", "-m"], capture_output=True, text=True, timeout=30, check=False) - except (OSError, subprocess.SubprocessError): - return {} - if out.returncode != 0: - return {} - - lines = out.stdout.splitlines() - # Column count comes from the header. Rows also carry NIC and affinity columns, - # and NIC cells reuse the same tokens (NODE, SYS, PIX) -- reading past the GPU - # columns invents pairs against GPUs that do not exist. - # The header is the line listing the GPU columns: it carries several GPU - # labels and no link-class tokens (a data row carries exactly one label, - # followed by classes). - gpu_columns = 0 - for line in lines: - labels = re.findall(r"\bGPU\d+\b", line) - if len(labels) >= 2 and not re.search(r"\b(?:X|NV\d+|PIX|PXB|PHB|NODE|SYS)\b", line): - gpu_columns = len(labels) - break - if gpu_columns == 0: - return {} - - pairs: dict[str, tuple[int, int]] = {} - rows_seen: set[int] = set() - for line in lines: - # Data rows start with the GPU label; the legend and NIC rows do not. - match = re.match(r"^\s*GPU(\d+)\s+(.*)$", line) - if match is None: - continue - row = int(match.group(1)) - # A row naming a GPU the header does not list means the matrix is partial - # or inconsistent. Returning a pair from it would hand an index that is - # not a selectable device to CUDA_VISIBLE_DEVICES. - if row >= gpu_columns: - return {} - rows_seen.add(row) - # Cells run in GPU-index order; "X" marks self. - for col, cell in enumerate(match.group(2).split()[:gpu_columns]): - if col == row or cell == "X": - continue - # NVLink cells are NV1/NV2/...; every other class is a bare token. - key = "NV" if cell.startswith("NV") else cell - kind = _TOPOLOGY_CLASS.get(key) - if kind is None: - continue - pairs.setdefault(kind, (row, col) if row < col else (col, row)) - - # Require the full square: a truncated matrix can omit exactly the rows that - # carry the boundary, which would silently downgrade a cross-socket host to - # "same switch only" and turn the expected-failure case into a skip. - if len(rows_seen) != gpu_columns: - return {} - return pairs - - -def _mig_enabled() -> bool: - """Whether any GPU on this host is partitioned into MIG instances. - - Returns: - ``True`` when ``nvidia-smi -L`` lists a MIG device, and on any error -- - an unreadable device list is treated as MIG so callers skip rather than - select physical indices that may not be addressable. - """ - try: - out = subprocess.run(["nvidia-smi", "-L"], capture_output=True, text=True, timeout=30, check=False) - except (OSError, subprocess.SubprocessError): - return True - if out.returncode != 0: - return True - return "MIG " in out.stdout - - def _list_available_devices() -> list[str]: """Return the host's visible devices in mask order: ``cpu`` then ``cuda:0, cuda:1, ...``. diff --git a/source/isaaclab/test/multi_gpu/test_multi_gpu_training_smoke.py b/source/isaaclab/test/multi_gpu/test_multi_gpu_training_smoke.py index 79226b39ae28..0203c039c0eb 100644 --- a/source/isaaclab/test/multi_gpu/test_multi_gpu_training_smoke.py +++ b/source/isaaclab/test/multi_gpu/test_multi_gpu_training_smoke.py @@ -6,22 +6,23 @@ """Multi-GPU training smoke tests. Setup: - - none; each test launches a real two-rank training run as a subprocess + - none; each test launches a real multi-rank training run as a subprocess Tests: - - physics-only task on any 2 GPUs -> verify training completes - - each physics/renderer stack on a same-switch GPU pair -> verify training completes - - each physics/renderer stack on a cross-socket GPU pair -> Kit-renderer stacks - are expected to fail, NVBUG#6565122 + - physics-only task on 2 GPUs -> verify training completes + - each stack on 4 GPUs with no device mask -> verify training completes + - each stack on 4 GPUs exposed as ``3,1,2,0`` -> verify training completes Unlike the rest of the suite these are not parametrized over ``device``: a -multi-GPU run owns two devices at once, so the per-shard single-device +multi-GPU run owns several devices at once, so the per-shard single-device parametrization the multi-GPU workflow applies elsewhere does not model it. -They are driven by a dedicated workflow step instead. - -Which GPU pair a case uses is resolved from the host at runtime rather than -hardcoded. On an 8-GPU two-socket box the default ``cuda:0,cuda:1`` pick is a -*same-switch* pair and does not exercise the cross-socket path at all, so a -fixed pick would quietly stop testing the thing this file exists for. +They are driven by dedicated workflow steps instead: the ``kitless`` marker +selects the stack whose renderer is an optional wheel the step installs first. + +Cases are split by whether the visible devices are exposed in their natural +order, because ``CUDA_VISIBLE_DEVICES`` renumbers devices for CUDA but not for +the graphics stack. Only a reordered mask makes the two indices disagree, which +is what exercises renderer device selection; the default ordering tests the case +where they coincide and cannot catch a selection defect at all. """ from __future__ import annotations @@ -36,8 +37,6 @@ import pytest -from isaaclab.test.utils import gpu_pairs_by_topology - # Small on purpose: Kit boot dominates the runtime at this size, and the defect # reproduces at 1024 envs exactly as it does at 2048. _NUM_ENVS = "1024" @@ -56,49 +55,54 @@ _PHYSICS_ONLY_TASK = "Isaac-Cartpole-Direct" _CAMERA_TASK = "Isaac-Cartpole-Camera-Direct" -# (id, presets) for each physics/renderer stack worth covering. +# Four ranks rather than two: with two, a single wrong device assignment can still land on a +# visible GPU by chance, and the rank-to-device mapping is too small to be wrong in an +# interesting way. +_CAMERA_RANKS = 4 + +# Deliberately not sorted and not contiguous-from-zero. Every rank's CUDA index differs from its +# graphics index, so no rank can be resolved by assuming the visible list is ordered. +_UNORDERED_DEVICES = (3, 1, 2, 0) + +# (id, presets) for each physics/renderer stack worth covering, split by what the run needs +# installed. Kit's RTX renderer ships in the image; ``ovrtx`` is an optional wheel that is in +# neither the Isaac Sim nor the Isaac Lab image, so its stack runs in a separate step that +# installs it first. The ``kitless`` marker is what selects between them. # # ``isaacsim_physx,ovrtx`` is absent by design: IsaacLab rejects it, since ovrtx # is a kitless renderer and cannot pair with Kit physics. # # TODO: add ``ovphysx,ovrtx`` once OvPhysX supports multi-GPU. It currently hangs -# at the first parameter sync on *any* GPU pair -- same-switch included -- so it -# is a separate defect from NVBUG#6565122 and would only cost CI a deliberate -# timeout while asserting something already known. See the process-global +# at the first parameter sync on *any* GPU pair, so it is a separate defect and would only cost +# CI a deliberate timeout while asserting something already known. See the process-global # device-mode lock in ``isaaclab_ovphysx.physics.ovphysx_manager``. -_CAMERA_STACKS = [ +_KIT_RENDERER_STACKS = [ pytest.param("isaacsim_physx", id="isaacsim_physx-kit_rtx"), pytest.param("newton_mjwarp,isaacsim_rtx", id="newton-kit_rtx"), - pytest.param("newton_mjwarp,ovrtx", id="newton-ovrtx"), ] -# Stacks that drive Kit's Isaac Sim RTX renderer. These are the ones that fail on -# a cross-socket pair; the kitless ``ovrtx`` stack passes there. -_KIT_RENDERER_STACKS = frozenset({"isaacsim_physx", "newton_mjwarp,isaacsim_rtx"}) - -_XFAIL_REASON = ( - "Kit's Isaac Sim RTX renderer corrupts the host heap when a multi-GPU rendering job spans a" - " cross-socket (SYS) GPU pair, surfacing as SIGSEGV inside libcarb.cudainterop.plugin.so." - " Entered between Kit 110.0.0 and Kit 110.1.2. The same run passes on a same-switch pair, and" - " with presets=newton_mjwarp,ovrtx on the same pair. NVBUG#6565122" -) +_KITLESS_STACKS = [ + pytest.param("newton_mjwarp,ovrtx", id="newton-ovrtx"), +] def _repo_root() -> Path: return Path(__file__).resolve().parents[4] -def _run_training(pair: tuple[int, int], task: str, presets: str) -> tuple[str, str]: - """Launch a two-rank training run pinned to ``pair`` and wait for it to settle. +def _run_training(devices: tuple[int, ...] | None, task: str, presets: str, num_gpus: int) -> tuple[str, str]: + """Launch a multi-rank training run and wait for it to settle. Streams the child's output so a stalled run is killed after :data:`_IDLE_TIMEOUT_S` of silence rather than occupying a CI runner until the hard timeout. Args: - pair: GPU indices to pin the two ranks to. + devices: GPU indices to expose, in the order given, or ``None`` to leave the inherited + visibility untouched. task: Gym task id to train. presets: Value for the ``presets=`` selector (physics and/or renderer). + num_gpus: Number of ranks to launch. Returns: ``(outcome, output)`` where outcome is ``"passed"``, ``"failed"`` or @@ -109,14 +113,18 @@ def _run_training(pair: tuple[int, int], task: str, presets: str) -> tuple[str, # pair is selected with CUDA_VISIBLE_DEVICES. Verified to reproduce the # canonical signature (exit 139 with cudainterop frames) rather than masking # it behind a device-enumeration artifact. - env["CUDA_VISIBLE_DEVICES"] = f"{pair[0]},{pair[1]}" + if devices is None: + # The naive case: whatever the host exposes, which is what a user gets by default. + env.pop("CUDA_VISIBLE_DEVICES", None) + else: + env["CUDA_VISIBLE_DEVICES"] = ",".join(str(index) for index in devices) env["PYTHONUNBUFFERED"] = "1" cmd = [ sys.executable, "scripts/reinforcement_learning/train_multigpu.py", "--num_gpus", - "2", + str(num_gpus), # Without this only rank 0 is printed; when rank 1 is the one that dies # the log carries no evidence of why. "--log_all_ranks", @@ -193,19 +201,10 @@ def _kill_process_group(process: subprocess.Popen) -> None: process.wait(timeout=30) -def _matches_known_crash(output: str) -> bool: - """Whether a failure carries the NVBUG#6565122 signature rather than some other fault. - - The crash is a SIGSEGV inside ``libcarb.cudainterop.plugin.so``. Without this - check an OOM, an argument error, or an unrelated hang on a cross-socket pair - would all be absorbed by the expected-failure marker. - """ - return "cudainterop" in output and ("exitcode : 139" in output or "Signal 11" in output) - - -def _assert_training_passed(outcome: str, output: str) -> None: +def _assert_training_passed(outcome: str, output: str, devices: tuple[int, ...] | None = None) -> None: """Assert a training subprocess actually trained, not merely exited cleanly.""" - assert outcome == "passed", f"outcome={outcome}\n{output[-2000:]}" + where = f" on CUDA_VISIBLE_DEVICES={devices}" if devices is not None else " with no device mask" + assert outcome == "passed", f"outcome={outcome}{where}\n{output[-2000:]}" def _visible_cuda_device_count() -> int: @@ -221,68 +220,77 @@ def _visible_cuda_device_count() -> int: return torch.cuda.device_count() if torch.cuda.is_available() else 0 -def _pair_or_skip(kind: str) -> tuple[int, int]: - """Return a GPU pair of ``kind``, or skip with the reason it is unavailable.""" - pairs = gpu_pairs_by_topology() - if not pairs: - pytest.skip( - "GPU topology unreadable (nvidia-smi topo -m gave no usable matrix); " - "cannot tell whether this host has a qualifying pair" - ) - if kind not in pairs: - pytest.skip(f"host has no {kind} GPU pair (available: {sorted(pairs)})") - return pairs[kind] +def _require_devices(count: int) -> None: + """Skip unless the host can address ``count`` CUDA devices.""" + available = _visible_cuda_device_count() + if available < count: + pytest.skip(f"needs {count} visible CUDA devices, host has {available}") @pytest.mark.smoke @pytest.mark.integration class TestMultiGpuTrainingSmoke: - """Two-rank training smoke coverage across the stacks and interconnect classes the host offers.""" + """Four-rank training smoke coverage across the physics and renderer stacks.""" - def test_physics_only_trains_on_any_pair(self) -> None: - """Physics-only multi-GPU training completes on the first two visible GPUs. + def test_physics_only_trains(self) -> None: + """Physics-only multi-GPU training completes with no device mask. - This is the guard that always runs. It deliberately does NOT consult the - topology: the camera cases are gated on the host offering a pair of the - right interconnect class, so a host with an unreadable topology, a MIG - layout, or only unmeasured link classes skips all six of them. Were this - case gated too, the step would exit 0 having launched no training at all. + The guard that always runs, and the cheapest signal that the launcher and NCCL are healthy + before any renderer is involved. Requires only two devices so it still runs on hosts too + small for the camera cases. """ - count = _visible_cuda_device_count() - if count < 2: - pytest.skip(f"multi-GPU smoke needs 2 visible CUDA devices, host has {count}") - _assert_training_passed(*_run_training((0, 1), _PHYSICS_ONLY_TASK, "isaacsim_physx")) + _require_devices(2) + _assert_training_passed(*_run_training(None, _PHYSICS_ONLY_TASK, "isaacsim_physx", num_gpus=2)) @pytest.mark.rendering - @pytest.mark.parametrize("presets", _CAMERA_STACKS) - def test_camera_trains_on_same_switch_pair(self, presets: str) -> None: - """Camera-based multi-GPU training completes when both GPUs share a PCIe switch. + @pytest.mark.parametrize("presets", _KIT_RENDERER_STACKS) + def test_kit_renderer_camera_trains_without_device_mask(self, presets: str) -> None: + """Kit-rendered training on four GPUs with no ``CUDA_VISIBLE_DEVICES`` set. + + What a user gets by default. CUDA and graphics device indices coincide here, so this passes + even when device selection is wrong -- it is the baseline the masked case is read against, + not a test of selection. + """ + _require_devices(_CAMERA_RANKS) + _assert_training_passed(*_run_training(None, _CAMERA_TASK, presets, num_gpus=_CAMERA_RANKS)) - Strict for every stack. This is the regression guard: it is the - configuration NVBUG#6565122 does *not* affect, so a failure here is a new - defect rather than the known one. + @pytest.mark.rendering + @pytest.mark.parametrize("presets", _KIT_RENDERER_STACKS) + def test_kit_renderer_camera_trains_on_unordered_devices(self, presets: str) -> None: + """Kit-rendered training on four GPUs exposed in a non-monotonic order. + + The regression guard for renderer device selection. ``CUDA_VISIBLE_DEVICES`` renumbers + devices for CUDA but not for the graphics stack, so with ``3,1,2,0`` every rank's CUDA index + differs from its graphics index and none of them can be recovered by assuming the list is + sorted or contiguous. Passing a CUDA index straight to ``/renderer/activeGpu`` selects a + device outside the visible set and the run dies with ``CUDA error 700``. """ - pair = _pair_or_skip("SAME_SWITCH") - _assert_training_passed(*_run_training(pair, _CAMERA_TASK, presets)) + _require_devices(_CAMERA_RANKS) + _assert_training_passed( + *_run_training(_UNORDERED_DEVICES, _CAMERA_TASK, presets, num_gpus=_CAMERA_RANKS), + devices=_UNORDERED_DEVICES, + ) @pytest.mark.rendering - @pytest.mark.parametrize("presets", _CAMERA_STACKS) - def test_camera_trains_on_cross_socket_pair(self, presets: str, request) -> None: - """Camera-based multi-GPU training across a cross-socket GPU pair. - - Stacks driving Kit's RTX renderer are expected to fail while - NVBUG#6565122 is open; the kitless ``ovrtx`` stack must still pass, which - is what pins the defect to the renderer rather than to multi-GPU - rendering in general. The failing cases run rather than skip so a Kit fix - surfaces as XPASS instead of going unnoticed. + @pytest.mark.kitless + @pytest.mark.parametrize("presets", _KITLESS_STACKS) + def test_kitless_camera_trains_without_device_mask(self, presets: str) -> None: + """Kitless-rendered training on four GPUs with no ``CUDA_VISIBLE_DEVICES`` set.""" + _require_devices(_CAMERA_RANKS) + _assert_training_passed(*_run_training(None, _CAMERA_TASK, presets, num_gpus=_CAMERA_RANKS)) + + @pytest.mark.rendering + @pytest.mark.kitless + @pytest.mark.parametrize("presets", _KITLESS_STACKS) + def test_kitless_camera_trains_on_unordered_devices(self, presets: str) -> None: + """Kitless-rendered training on four GPUs exposed in a non-monotonic order. + + ``ovrtx`` selects its device through CUDA, which ``CUDA_VISIBLE_DEVICES`` renumbers + consistently, so this stack was never affected by the graphics-index defect the Kit case + covers. It runs strict as the control: a failure here is a different defect. """ - pair = _pair_or_skip("CROSS_SOCKET") - outcome, output = _run_training(pair, _CAMERA_TASK, presets) - if presets in _KIT_RENDERER_STACKS and outcome == "failed" and _matches_known_crash(output): - # Marked only once the documented signature is confirmed, and applied - # after the run rather than as a decorator. A blanket marker would - # absorb an OOM, an argument error, or an unrelated hang on this pair - # as though it were NVBUG#6565122, and would also cover the ovrtx - # stack, which must stay strict. - pytest.xfail(_XFAIL_REASON) - _assert_training_passed(outcome, output) + _require_devices(_CAMERA_RANKS) + _assert_training_passed( + *_run_training(_UNORDERED_DEVICES, _CAMERA_TASK, presets, num_gpus=_CAMERA_RANKS), + devices=_UNORDERED_DEVICES, + ) diff --git a/source/isaaclab/test/utils/test_gpu_topology.py b/source/isaaclab/test/utils/test_gpu_topology.py deleted file mode 100644 index dda19d904775..000000000000 --- a/source/isaaclab/test/utils/test_gpu_topology.py +++ /dev/null @@ -1,149 +0,0 @@ -# Copyright (c) 2022-2026, The Isaac Lab Project Developers (https://github.com/isaac-sim/IsaacLab/blob/main/CONTRIBUTORS.md). -# All rights reserved. -# -# SPDX-License-Identifier: BSD-3-Clause - -"""Unit tests for the ``nvidia-smi topo -m`` parser behind :func:`gpu_pairs_by_topology`. - -Fixtures are verbatim output from three differently-shaped hosts, so the parser -is covered on any machine -- including single-GPU CI, where the multi-GPU smoke -tests that consume it cannot run. -""" - -from __future__ import annotations - -import subprocess -from unittest import mock - -import pytest - -from isaaclab.test.utils import gpu_pairs_by_topology - -# 4x L40, one GPU alone on socket 0. Carries a NIC column whose cells reuse the -# same link-class tokens -- reading past the GPU columns invents a pair against -# a GPU that does not exist. -_TOPO_1V3_WITH_NIC = """\t\tGPU0\tGPU1\tGPU2\tGPU3\tNIC0\tCPU Affinity\tNUMA Affinity -GPU0\t X \tSYS\tSYS\tSYS\tNODE\t0-31,64-95\t0 -GPU1\tSYS\t X \tPIX\tPIX\tSYS\t32-63,96-127\t1 -GPU2\tSYS\tPIX\t X \tPIX\tSYS\t32-63,96-127\t1 -GPU3\tSYS\tPIX\tPIX\t X \tSYS\t32-63,96-127\t1 -NIC0\tNODE\tSYS\tSYS\tSYS\t X -""" - -# 8x L40 split 4v4 across two sockets. -_TOPO_4V4 = """\t\tGPU0\tGPU1\tGPU2\tGPU3\tGPU4\tGPU5\tGPU6\tGPU7\tCPU Affinity -GPU0\t X \tPIX\tPIX\tPIX\tSYS\tSYS\tSYS\tSYS\t0-27 -GPU1\tPIX\t X \tPIX\tPIX\tSYS\tSYS\tSYS\tSYS\t0-27 -GPU2\tPIX\tPIX\t X \tPIX\tSYS\tSYS\tSYS\tSYS\t0-27 -GPU3\tPIX\tPIX\tPIX\t X \tSYS\tSYS\tSYS\tSYS\t0-27 -GPU4\tSYS\tSYS\tSYS\tSYS\t X \tPIX\tPIX\tPIX\t28-55 -GPU5\tSYS\tSYS\tSYS\tSYS\tPIX\t X \tPIX\tPIX\t28-55 -GPU6\tSYS\tSYS\tSYS\tSYS\tPIX\tPIX\t X \tPIX\t28-55 -GPU7\tSYS\tSYS\tSYS\tSYS\tPIX\tPIX\tPIX\t X \t28-55 -""" - -# 4x RTX 6000 Ada, single socket: every pair is PHB or NODE, neither of which is -# measured for the defect the classification gates. -_TOPO_SINGLE_SOCKET = """\t\tGPU0\tGPU1\tGPU2\tGPU3\tCPU Affinity\tNUMA Affinity -GPU0\t X \tPHB\tNODE\tNODE\t0-47\t0 -GPU1\tPHB\t X \tNODE\tNODE\t0-47\t0 -GPU2\tNODE\tNODE\t X \tNODE\t0-47\t0 -GPU3\tNODE\tNODE\tNODE\t X \t0-47\t0 -""" - -_TOPO_NVLINK = """\t\tGPU0\tGPU1\tCPU Affinity -GPU0\t X \tNV18\t0-95 -GPU1\tNV18\t X \t0-95 -""" - - -# A row naming a GPU the header never lists: the matrix is partial or inconsistent. -_TOPO_ROW_OUTSIDE_HEADER = """\t\tGPU0\tGPU1\tCPU Affinity -GPU0\t X \tSYS\t0-95 -GPU9\tSYS\t X \t0-95 -""" - -# Header advertises four GPUs but only two rows are present. -_TOPO_TRUNCATED = """\t\tGPU0\tGPU1\tGPU2\tGPU3\tCPU Affinity -GPU0\t X \tPIX\tPIX\tPIX\t0-27 -GPU1\tPIX\t X \tPIX\tPIX\t0-27 -""" - -_NVIDIA_SMI_L_DISCRETE = "GPU 0: NVIDIA L40 (UUID: GPU-aaa)\nGPU 1: NVIDIA L40 (UUID: GPU-bbb)\n" -_NVIDIA_SMI_L_MIG = ( - "GPU 0: NVIDIA A100-SXM4-40GB (UUID: GPU-aaa)\n" - " MIG 3g.20gb Device 0: (UUID: MIG-bbb)\n" - " MIG 3g.20gb Device 1: (UUID: MIG-ccc)\n" -) - - -def _with_topo(stdout: str, returncode: int = 0, devices: str = _NVIDIA_SMI_L_DISCRETE): - """Patch ``subprocess.run`` so the parser sees ``stdout`` as nvidia-smi output. - - ``devices`` stands in for ``nvidia-smi -L``, which the parser consults first to - rule out a MIG host. - """ - - def _run(cmd, *args, **kwargs): - out = devices if "-L" in cmd else stdout - code = 0 if "-L" in cmd else returncode - return subprocess.CompletedProcess(args=cmd, returncode=code, stdout=out, stderr="") - - return mock.patch("isaaclab.test.utils.devices.subprocess.run", side_effect=_run) - - -class TestGpuPairsByTopology: - """Classification of GPU pairs from nvidia-smi topology output.""" - - def test_two_socket_split_reports_both_classes(self) -> None: - with _with_topo(_TOPO_4V4): - assert gpu_pairs_by_topology() == {"SAME_SWITCH": (0, 1), "CROSS_SOCKET": (0, 4)} - - def test_nic_column_does_not_invent_a_gpu_pair(self) -> None: - """NIC cells carry the same tokens as GPU cells and must not be classified.""" - with _with_topo(_TOPO_1V3_WITH_NIC): - pairs = gpu_pairs_by_topology() - assert pairs == {"CROSS_SOCKET": (0, 1), "SAME_SWITCH": (1, 2)} - # The host has 4 GPUs; a pair naming index 4 would come from the NIC column. - assert all(max(pair) < 4 for pair in pairs.values()) - - def test_single_socket_reports_only_unknown(self) -> None: - """PHB/NODE are unmeasured for this defect, so neither camera case may run.""" - with _with_topo(_TOPO_SINGLE_SOCKET): - pairs = gpu_pairs_by_topology() - assert pairs == {"UNKNOWN": (0, 1)} - assert "CROSS_SOCKET" not in pairs - assert "SAME_SWITCH" not in pairs - - def test_nvlink_counts_as_same_switch(self) -> None: - with _with_topo(_TOPO_NVLINK): - assert gpu_pairs_by_topology() == {"SAME_SWITCH": (0, 1)} - - @pytest.mark.parametrize( - "stdout,returncode", - [("", 0), ("no topology here", 0), (_TOPO_4V4, 1)], - ids=["empty", "unparsable", "nvidia-smi failed"], - ) - def test_undeterminable_topology_returns_empty(self, stdout: str, returncode: int) -> None: - """Callers must skip rather than infer "no boundary present" from a parse failure.""" - with _with_topo(stdout, returncode): - assert gpu_pairs_by_topology() == {} - - def test_missing_nvidia_smi_returns_empty(self) -> None: - with mock.patch("isaaclab.test.utils.devices.subprocess.run", side_effect=FileNotFoundError): - assert gpu_pairs_by_topology() == {} - - def test_row_outside_header_returns_empty(self) -> None: - """A GPU index absent from the header must never reach CUDA_VISIBLE_DEVICES.""" - with _with_topo(_TOPO_ROW_OUTSIDE_HEADER): - assert gpu_pairs_by_topology() == {} - - def test_truncated_matrix_returns_empty(self) -> None: - """A partial matrix can omit exactly the rows carrying the boundary.""" - with _with_topo(_TOPO_TRUNCATED): - assert gpu_pairs_by_topology() == {} - - def test_mig_host_returns_empty(self) -> None: - """topo -m describes physical GPUs; CUDA addresses MIG instances.""" - with _with_topo(_TOPO_4V4, devices=_NVIDIA_SMI_L_MIG): - assert gpu_pairs_by_topology() == {} From 2931ce80a9448c26aada6cd02557d52dcbb0564e Mon Sep 17 00:00:00 2001 From: jichuanh Date: Thu, 13 Aug 2026 16:38:07 -0700 Subject: [PATCH 08/11] Update the smoke test changelog fragment for the new selection --- .../changelog.d/jichuanh-mgpu-camera-smoke-xfail.rst | 9 +++------ 1 file changed, 3 insertions(+), 6 deletions(-) diff --git a/source/isaaclab/changelog.d/jichuanh-mgpu-camera-smoke-xfail.rst b/source/isaaclab/changelog.d/jichuanh-mgpu-camera-smoke-xfail.rst index e678333879d6..5396ebbfa03f 100644 --- a/source/isaaclab/changelog.d/jichuanh-mgpu-camera-smoke-xfail.rst +++ b/source/isaaclab/changelog.d/jichuanh-mgpu-camera-smoke-xfail.rst @@ -1,9 +1,6 @@ Added ^^^^^ -* Added multi-GPU training smoke tests that launch real two-rank runs and select their GPU pair - by interconnect class, so cross-socket rendering is exercised instead of whichever pair happens - to be ``cuda:0,cuda:1``. The cross-socket camera case is marked ``xfail`` for NVBUG#6565122. - -* Added :func:`~isaaclab.test.utils.gpu_pairs_by_topology` to classify GPU pairs from - ``nvidia-smi topo -m`` as same-switch, cross-socket, or unknown. +* Added multi-GPU training smoke tests that launch real four-rank training runs, covering each + physics and renderer stack both with the host's default device order and with the devices + exposed out of order, which is the case that catches renderer device-selection defects. From 9c23e0afdb29e0d2711e5072f8992ee5fd77ea49 Mon Sep 17 00:00:00 2001 From: jichuanh Date: Thu, 13 Aug 2026 17:12:06 -0700 Subject: [PATCH 09/11] Provision ovrtx for the kitless smoke step from the pyproject pins Matches how the isaaclab_ov job provisions it: the ov extra is in MANUAL_EXTRA_FEATURES and the images do not carry it, so the pins are resolved and installed per run rather than baked into the image, which would bypass the wheelhouse path ovphysx uses. --- .github/workflows/test-multi-gpu-pytest.yaml | 13 +++++++------ .../test/multi_gpu/test_multi_gpu_training_smoke.py | 11 +++++------ 2 files changed, 12 insertions(+), 12 deletions(-) diff --git a/.github/workflows/test-multi-gpu-pytest.yaml b/.github/workflows/test-multi-gpu-pytest.yaml index 458ca093c90a..55be56bdcc17 100644 --- a/.github/workflows/test-multi-gpu-pytest.yaml +++ b/.github/workflows/test-multi-gpu-pytest.yaml @@ -200,17 +200,18 @@ jobs: image-tag: ${{ env.CI_IMAGE_TAG }} pytest-options: -m "not kitless" - - name: Resolve OV pins - # The kitless renderer ships as an optional wheel that is in neither the - # Isaac Sim nor the Isaac Lab image, so its stack needs it installed first. + - name: Resolve OV runtime pins from pyproject + # Same provisioning as the isaaclab_ov job: the ``ov`` extra is in + # MANUAL_EXTRA_FEATURES and the image does not carry it, so the pins are + # resolved and installed per run. if: always() id: ov_pins uses: ./.github/actions/resolve-ov-pins - name: Multi-GPU training smoke (kitless renderer) - # Same image and same tests as the step above, with ovrtx installed and the - # marker inverted. Split into its own step so a missing optional wheel cannot - # take down the Kit-renderer coverage, which is what the fix under test guards. + # Same image and tests as the step above with the marker inverted, plus ovrtx. + # Kept a separate step so a fault in the kitless renderer cannot take down the + # Kit-renderer coverage, which is what guards renderer device selection. if: always() uses: ./.github/actions/run-tests with: diff --git a/source/isaaclab/test/multi_gpu/test_multi_gpu_training_smoke.py b/source/isaaclab/test/multi_gpu/test_multi_gpu_training_smoke.py index 0203c039c0eb..04f5a2966bd6 100644 --- a/source/isaaclab/test/multi_gpu/test_multi_gpu_training_smoke.py +++ b/source/isaaclab/test/multi_gpu/test_multi_gpu_training_smoke.py @@ -15,8 +15,8 @@ Unlike the rest of the suite these are not parametrized over ``device``: a multi-GPU run owns several devices at once, so the per-shard single-device parametrization the multi-GPU workflow applies elsewhere does not model it. -They are driven by dedicated workflow steps instead: the ``kitless`` marker -selects the stack whose renderer is an optional wheel the step installs first. +They are driven by dedicated workflow steps instead, one per renderer, selected +by the ``kitless`` marker. Cases are split by whether the visible devices are exposed in their natural order, because ``CUDA_VISIBLE_DEVICES`` renumbers devices for CUDA but not for @@ -64,10 +64,9 @@ # graphics index, so no rank can be resolved by assuming the visible list is ordered. _UNORDERED_DEVICES = (3, 1, 2, 0) -# (id, presets) for each physics/renderer stack worth covering, split by what the run needs -# installed. Kit's RTX renderer ships in the image; ``ovrtx`` is an optional wheel that is in -# neither the Isaac Sim nor the Isaac Lab image, so its stack runs in a separate step that -# installs it first. The ``kitless`` marker is what selects between them. +# (id, presets) for each physics/renderer stack worth covering, split by renderer so a fault in +# one cannot mask the other: the Kit-renderer cases are what guard device selection. The +# ``kitless`` marker selects between them. # # ``isaacsim_physx,ovrtx`` is absent by design: IsaacLab rejects it, since ovrtx # is a kitless renderer and cannot pair with Kit physics. From 64f6a6560b140c0b7c68c48293b59231c2f01e50 Mon Sep 17 00:00:00 2001 From: jichuanh Date: Thu, 13 Aug 2026 17:42:03 -0700 Subject: [PATCH 10/11] Install the ov extra in the base image The ov extra is in MANUAL_EXTRA_FEATURES, so neither a bare --install nor -i all selects it and the image cannot import ovrtx. The ovrtx renderer is a supported backend, so install it as its own step, leaving the default selector untouched. This lets the kitless smoke step run on the same image as the rest of the job instead of resolving and installing the pins per run. --- .github/workflows/test-multi-gpu-pytest.yaml | 16 ++++------------ docker/Dockerfile.base | 7 +++++++ 2 files changed, 11 insertions(+), 12 deletions(-) diff --git a/.github/workflows/test-multi-gpu-pytest.yaml b/.github/workflows/test-multi-gpu-pytest.yaml index 55be56bdcc17..14ec629060d0 100644 --- a/.github/workflows/test-multi-gpu-pytest.yaml +++ b/.github/workflows/test-multi-gpu-pytest.yaml @@ -200,18 +200,11 @@ jobs: image-tag: ${{ env.CI_IMAGE_TAG }} pytest-options: -m "not kitless" - - name: Resolve OV runtime pins from pyproject - # Same provisioning as the isaaclab_ov job: the ``ov`` extra is in - # MANUAL_EXTRA_FEATURES and the image does not carry it, so the pins are - # resolved and installed per run. - if: always() - id: ov_pins - uses: ./.github/actions/resolve-ov-pins - - name: Multi-GPU training smoke (kitless renderer) - # Same image and tests as the step above with the marker inverted, plus ovrtx. - # Kept a separate step so a fault in the kitless renderer cannot take down the - # Kit-renderer coverage, which is what guards renderer device selection. + # Same image and tests as the step above with the marker inverted; the image + # carries ovrtx because Dockerfile.base installs the ``ov`` extra. Kept a separate + # step so a fault in the kitless renderer cannot take down the Kit-renderer + # coverage, which is what guards renderer device selection. if: always() uses: ./.github/actions/run-tests with: @@ -219,7 +212,6 @@ jobs: result-file: multi-gpu-smoke-kitless-report.xml container-name: isaac-lab-mgpu-smoke-kitless-${{ github.run_id }}-${{ github.run_attempt }} image-tag: ${{ env.CI_IMAGE_TAG }} - extra-pip-packages: ${{ steps.ov_pins.outputs.ovrtx }} pytest-options: -m kitless - name: Aggregated test summary diff --git a/docker/Dockerfile.base b/docker/Dockerfile.base index 77382f8c9e8b..4d8b4d9ac7d0 100644 --- a/docker/Dockerfile.base +++ b/docker/Dockerfile.base @@ -117,6 +117,13 @@ RUN --mount=type=cache,target=${DOCKER_USER_HOME}/.cache/pip \ ${ISAACLAB_PATH}/isaaclab.sh --install && \ ${ISAACLAB_PATH}/isaaclab.sh -p -m pip uninstall -y imageio-ffmpeg +# ``ov`` is in MANUAL_EXTRA_FEATURES, so neither a bare --install nor ``-i all`` selects it. +# Install it separately, leaving the default set above untouched: the ovrtx renderer is a +# supported backend, and an image that cannot import it cannot test it. Versions come from the +# ``ov`` extra, the same source of truth the per-run pin resolution reads. +RUN --mount=type=cache,target=${DOCKER_USER_HOME}/.cache/pip \ + ${ISAACLAB_PATH}/isaaclab.sh --install ov[all] + # HACK: Remove install of quadprog dependency RUN ${ISAACLAB_PATH}/isaaclab.sh -p -m pip uninstall -y quadprog From ef4151230cc7e3c5585f68f27092473ca08a37fd Mon Sep 17 00:00:00 2001 From: jichuanh Date: Wed, 19 Aug 2026 12:43:13 -0700 Subject: [PATCH 11/11] Drop the merged #7057 leftovers from the smoke-test branch #7057 landed on develop as e8dc0bf6da1, so the branch no longer needs its renderer changes: the orphaned extra_args entry (develop selects the device through _resolve_kit_args instead) and the changelog fragment, which the compile job already folded into the changelog. --- ...jichuanh-fix-renderer-active-gpu-physical-index.rst | 10 ---------- source/isaaclab/isaaclab/app/app_launcher.py | 1 - 2 files changed, 11 deletions(-) delete mode 100644 source/isaaclab/changelog.d/jichuanh-fix-renderer-active-gpu-physical-index.rst diff --git a/source/isaaclab/changelog.d/jichuanh-fix-renderer-active-gpu-physical-index.rst b/source/isaaclab/changelog.d/jichuanh-fix-renderer-active-gpu-physical-index.rst deleted file mode 100644 index bd7fd354a090..000000000000 --- a/source/isaaclab/changelog.d/jichuanh-fix-renderer-active-gpu-physical-index.rst +++ /dev/null @@ -1,10 +0,0 @@ -Fixed -^^^^^ - -* Fixed rendering failing to start when ``CUDA_VISIBLE_DEVICES`` selects GPUs that do not begin at - zero, such as ``CUDA_VISIBLE_DEVICES=1,2``. Such runs aborted with ``CUDA error 700`` after - ``omni.gpu_foundation_factory`` reported "Failed to create any GPU devices". The renderer device - is now selected through ``/renderer/multiGpu/activeCudaGpus``, which takes a CUDA device index, - instead of ``/renderer/activeGpu``, which indexes the graphics device list that - ``CUDA_VISIBLE_DEVICES`` does not filter. Runs whose visible devices already begin at zero are - unaffected. diff --git a/source/isaaclab/isaaclab/app/app_launcher.py b/source/isaaclab/isaaclab/app/app_launcher.py index efc4d3326cfd..7b212fbf18ab 100644 --- a/source/isaaclab/isaaclab/app/app_launcher.py +++ b/source/isaaclab/isaaclab/app/app_launcher.py @@ -721,7 +721,6 @@ def add_app_launcher_args(parser: argparse.ArgumentParser) -> None: "fast_shutdown": [bool], "limit_cpu_threads": [int], "experience": [str], - "extra_args": [list, type(None)], } """A dictionary containing the type of arguments passed to SimulationApp.