Add multi-GPU training smoke tests - #6946
Draft
hujc7 wants to merge 16 commits into
Draft
Conversation
The multi-GPU workflow parametrizes each test over one device per shard, which cannot express a job that owns two GPUs at once. Nothing in CI therefore exercised multi-GPU rendering, and NVBUG#6565122 -- a heap corruption in Kit's Isaac Sim RTX renderer that only fires when the two GPUs span a cross-socket boundary -- shipped uncaught. Add smoke tests that launch real two-rank training runs and choose their GPU pair by interconnect class rather than taking whatever cuda:0,cuda:1 happens to be. On a two-socket 8-GPU host the default pick is a same-switch pair, so a fixed choice would never exercise the failing path. Each class is handled on its own terms: same-switch is strict and is the regression guard, cross-socket is expected-fail for the two Kit-renderer stacks and strict for the kitless newton_mjwarp,ovrtx stack, and the unmeasured PXB/PHB/NODE classes skip rather than assert either verdict. A physics-only case runs on any two visible devices so the step is never vacuous when every camera case skips. The expected-failure marker is applied only after the documented signature is confirmed, so an OOM or an argument error on a cross-socket pair is not recorded as the known crash.
The smoke step ran docker with no volume mount, so `cd /workspace/isaaclab` landed in the image's baked-in copy rather than the PR checkout. pytest collected 0 items and the step failed with 'file or directory not found'. Mount $PWD and run as the host user, matching what multi_gpu_host_launcher.sh already does for the sharded runs. Without --user the container writes root-owned files into the runner workspace. Also surface a ::warning:: when the runner's GPU topology cannot exercise a case, so a run that skips the NVBUG#6565122 coverage does not read as a normal green run.
The smoke step hand-rolled its own `docker run`, duplicating a fraction of the container contract that .github/actions/run-tests already provides -- and getting parts of it wrong. It had no workspace mount (so pytest collected 0 items against the image's baked-in copy), and once that was added, `--user` without a writable HOME failed on /root/.cache. Passing a file path rather than `tools` skips the conftest sharding orchestrator, so this is still a plain single pytest run on all GPUs, with no device restriction -- the tests continue to pick their own GPU pair per interconnect class. What it gains is the standard contract: writable HOME plus XDG_CACHE_HOME/XDG_DATA_HOME, the Isaac Sim cache mounts, warp cache, and junit results collected into reports/. Previously --junitxml wrote to /tmp inside a --rm container, so the smoke step contributed nothing to the aggregated test report. What remains smoke-specific is the topology report, split out into multi_gpu_topology_report.sh: the tests skip when the host offers no qualifying GPU pair, which is correct but silent, so the run summary needs to say the coverage was not exercised. Its guard now matches the matrix rows only -- `nvidia-smi topo -m` prints a legend naming every class, so grepping the whole output found SYS and PIX on a host that had neither and the warning never fired. Also reword the topology skip so "unreadable topology" is distinguishable from "no qualifying pair on this host"; the old message sent readers looking for a parsing bug that was not there.
CUDA_VISIBLE_DEVICES renumbers devices for CUDA but not for the graphics stack. AppLauncher passed the same masked index to both /physics/cudaDevice and /renderer/activeGpu, so whenever the visible devices did not begin at zero the renderer requested a device outside the visible set, failed to create it, and the run aborted with CUDA error 700. Translate only the renderer index to a physical one. physics_gpu stays masked because /physics/cudaDevice is resolved by CUDA itself. Masks that already begin at zero are unaffected, and UUID or MIG masks fall back to the previous behavior since no physical index can be derived from them.
/renderer/activeGpu indexes the graphics device list, which CUDA_VISIBLE_DEVICES does not filter, so passing the CUDA device index there selected the wrong GPU whenever the visible devices did not start at zero. The renderer then failed to create a device and the run aborted with CUDA error 700 from a Warp launch. Select the device through /renderer/multiGpu/activeCudaGpus instead, which takes CUDA indices and is translated by the renderer itself, and stop setting activeGpu -- the translation is only applied when no explicit graphics index is given. physics_gpu is unchanged, since /physics/cudaDevice is resolved by CUDA.
…derer-active-gpu-physical-index
Setting active_gpu to None was redundant: SimulationApp already defaults it to None and only emits --/renderer/activeGpu when it is set, so not assigning it leaves the renderer at its own -1 default, which is what the CUDA index translation requires. It also stops overriding an active_gpu passed by the caller. Move the device selection tests into the existing app launcher argv suite, which already covers the arguments handed to Kit.
…ichuanh/mgpu-camera-smoke-xfail
The camera cases picked their GPU pair by interconnect class, which never selected anything: the multi-GPU runner is four GPUs on one NUMA node with every pair PHB, so it has neither a same-switch nor a cross-socket pair and all six cases skipped. The class also does not decide the outcome -- a pair is exercised by whether CUDA_VISIBLE_DEVICES reorders it, since that renumbers devices for CUDA but not for the graphics stack. Pick the devices by order instead: one case with no mask, one exposing four GPUs as 3,1,2,0 so no rank's CUDA index matches its graphics index. Both run on any four-GPU host, including the runner. Drop gpu_pairs_by_topology() and its tests with the classification, and the workflow's topology report with them. Separately, the parser returned nothing on any real host: nvidia-smi underlines the header even into a pipe, so \bGPU0\b matched nothing, the header came up a column short and the last row tripped the partial-matrix guard. Split the stacks across two steps by what each needs installed, since ovrtx ships in neither image and would otherwise fail the whole step.
hujc7
changed the title
6 tasks
Matches how the isaaclab_ov job provisions it: the ov extra is in MANUAL_EXTRA_FEATURES and the images do not carry it, so the pins are resolved and installed per run rather than baked into the image, which would bypass the wheelhouse path ovphysx uses.
The ov extra is in MANUAL_EXTRA_FEATURES, so neither a bare --install nor -i all selects it and the image cannot import ovrtx. The ovrtx renderer is a supported backend, so install it as its own step, leaving the default selector untouched. This lets the kitless smoke step run on the same image as the rest of the job instead of resolving and installing the pins per run.
ooctipus
added a commit
that referenced
this pull request
Aug 15, 2026
## Review Map - The "Exact changes" link pins a stacked PR's own commit range; its PR page shows the cumulative diff of the whole dependency chain. - SHA links can go stale after a branch update; this PR's copy is refreshed first. | PR | Status | Depends on | Exact changes | |---|---|---|---| | 📌 #7057 — select the renderer device by CUDA index (1/2) (this PR) |  | — | — | | #6946 — multi-GPU training smoke tests (2/2) (draft) |  | #7057 | [33d65ab..2931ce8](https://github.com/isaac-sim/IsaacLab/pull/6946/changes/33d65ab615d03f7ba07be8674c18f5d78d2d10b7..2931ce80a9448c26aada6cd02557d52dcbb0564e) | ## 1. Summary Multi-GPU rendering fails whenever `CUDA_VISIBLE_DEVICES` does not begin at zero (`1,2`, `4,5,6,7`, …), aborting with `CUDA error 700`. `AppLauncher` passed the masked CUDA index to `/renderer/activeGpu`, which indexes the graphics device list that the mask never filtered. Select the device by CUDA index instead — 4 lines, 17 configurations validated at 2-8 ranks. ## 2. Type of change - Bug fix (non-breaking change which fixes an issue) ## 3. Design notes `CUDA_VISIBLE_DEVICES` renumbers devices for CUDA but not for the graphics stack, so everything selecting *through* CUDA was always correct: torch, Warp, PhysX via `/physics/cudaDevice`, and the kitless `ovrtx` renderer. `/renderer/multiGpu/activeCudaGpus` takes CUDA indices and is translated by the renderer itself (Kit MR !42520); `activeGpu` is no longer set, since that translation only applies when no explicit graphics index is given. `physics_gpu` is unchanged. Dropping `activeGpu` entirely, or passing Kit's `-1` auto value, also stops the crash — but every rank then auto-selects the same device and the renderers stack: 380% memory spread at 8 ranks, 26.4 GB on GPU0 against 6.9 GB per peer. Auto-selection is per-process and cannot know the rank. ## 4. Validation 8x L40S, same command throughout, varying only the mask and rank count: ```bash CUDA_VISIBLE_DEVICES=<mask> ./isaaclab.sh -p \ scripts/reinforcement_learning/train_multigpu.py --num_gpus <ranks> \ --log_all_ranks --master_port <port> --rl_library rsl_rl \ --task Isaac-Cartpole-Camera-Direct presets=isaacsim_physx,isaacsim_rtx \ --num_envs 64 --max_iterations 5 ``` 17 configurations, all pass. Ranks 2-8; offset, interleaved (`0,4,1,5`) and reversed (`7,6,5,4,3,2,1,0`) masks, each of which fails before this change; identity masks at 2/3/4/8 ranks as regression checks. Per-GPU memory spread is 99-102% throughout (100% is even) and host RAM is flat at ~6.4 GB/rank, both matching the identity controls — the change costs nothing on either axis. ## 5. Screenshots None. ## 6. Checklist - [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with `./isaaclab.sh --format` - [ ] I have made corresponding changes to the documentation — no public API change - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] I have updated the changelog — `source/isaaclab/changelog.d/`; `config/extension.toml` is left to the version-bump job per `AGENTS.md` - [x] I have added my name to the `CONTRIBUTORS.md` or my name already exists there --------- Co-authored-by: Octi Zhang <zhengyuz@nvidia.com>
isaac-sim#7057 landed on develop as e8dc0bf, so the branch no longer needs its renderer changes: the orphaned extra_args entry (develop selects the device through _resolve_kit_args instead) and the changelog fragment, which the compile job already folded into the changelog.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Review Map
1. Summary
Nothing in CI exercised multi-GPU rendering. These are real multi-rank training runs that do, chosen to catch renderer device-selection defects: one case with no device mask, one exposing four GPUs as
3,1,2,0. Depends on #7057, without which the reordered case fails.2. Type of change
3. Design notes
CUDA_VISIBLE_DEVICESrenumbers devices for CUDA but not for the graphics stack, so a run is only exposed to a selection defect when the visible devices are reordered — with3,1,2,0no rank's CUDA index matches its graphics index, and no rank can be recovered by assuming the list is sorted or contiguous. The unmasked case is the baseline it is read against: it passes even when selection is wrong.Four ranks rather than two, because with two a wrong device can still land on a visible GPU by chance.
Selection is deliberately not by interconnect class. The multi-GPU runner is four GPUs on one NUMA node with every pair
PHB, so it has neither a same-switch nor a cross-socket pair — a topology-gated version of this file skipped all six camera cases there while reporting green. Interconnect also does not decide the outcome:PHBtrains fine, and the defect these guard is an index mismatch.The stacks are split across two steps by what each needs installed.
ovrtxships in neither the Isaac Sim nor the Isaac Lab image, so its step installs the wheel first; keeping it separate means a missing optional wheel cannot take down the Kit-renderer coverage.A pass requires
Training time:in the output, not merely exit 0 — a run that OOMs still exits 0 through the launcher. Silence is the hang signal: 90 s without output kills the run, with a 600 s backstop, and the child is killed by process group since killing only the wrapper leaves rank workers holding GPU memory.4. Validation
Run on 8x L40S in the shipped Isaac Lab container, four ranks per case:
3,1,2,0The
newton_mjwarp,ovrtxstack is unverified here: the wheel is absent from both images, which is what its separate step exists to handle.5. Screenshots
None.
6. Checklist
pre-commitchecks with./isaaclab.sh --formatCONTRIBUTORS.mdor my name already exists there