Select the renderer device by CUDA index

[hujc7]

Review Map

• The "Exact changes" link pins a stacked PR's own commit range; its PR page shows the cumulative diff of the whole dependency chain.
• SHA links can go stale after a branch update; this PR's copy is refreshed first.

PR	Status	Depends on	Exact changes
📌 #7057 — select the renderer device by CUDA index (1/2) (this PR)		—	—
#6946 — multi-GPU training smoke tests (2/2) (draft)		#7057	33d65ab..2931ce8

1. Summary

Multi-GPU rendering fails whenever CUDA_VISIBLE_DEVICES does not begin at zero (1,2, 4,5,6,7, …), aborting with CUDA error 700. AppLauncher passed the masked CUDA index to /renderer/activeGpu, which indexes the graphics device list that the mask never filtered. Select the device by CUDA index instead — 4 lines, 17 configurations validated at 2-8 ranks.

2. Type of change

• Bug fix (non-breaking change which fixes an issue)

3. Design notes

CUDA_VISIBLE_DEVICES renumbers devices for CUDA but not for the graphics stack, so everything selecting through CUDA was always correct: torch, Warp, PhysX via /physics/cudaDevice, and the kitless ovrtx renderer. /renderer/multiGpu/activeCudaGpus takes CUDA indices and is translated by the renderer itself (Kit MR !42520); activeGpu is no longer set, since that translation only applies when no explicit graphics index is given. physics_gpu is unchanged.

Dropping activeGpu entirely, or passing Kit's -1 auto value, also stops the crash — but every rank then auto-selects the same device and the renderers stack: 380% memory spread at 8 ranks, 26.4 GB on GPU0 against 6.9 GB per peer. Auto-selection is per-process and cannot know the rank.

4. Validation

8x L40S, same command throughout, varying only the mask and rank count:

CUDA_VISIBLE_DEVICES=<mask> ./isaaclab.sh -p \
  scripts/reinforcement_learning/train_multigpu.py --num_gpus <ranks> \
  --log_all_ranks --master_port <port> --rl_library rsl_rl \
  --task Isaac-Cartpole-Camera-Direct presets=isaacsim_physx,isaacsim_rtx \
  --num_envs 64 --max_iterations 5

17 configurations, all pass. Ranks 2-8; offset, interleaved (0,4,1,5) and reversed (7,6,5,4,3,2,1,0) masks, each of which fails before this change; identity masks at 2/3/4/8 ranks as regression checks. Per-GPU memory spread is 99-102% throughout (100% is even) and host RAM is flat at ~6.4 GB/rank, both matching the identity controls — the change costs nothing on either axis.

5. Screenshots

None.

6. Checklist

• I have run the pre-commit checks with ./isaaclab.sh --format
• I have made corresponding changes to the documentation — no public API change
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• I have updated the changelog — source/isaaclab/changelog.d/; config/extension.toml is left to the version-bump job per AGENTS.md
• I have added my name to the CONTRIBUTORS.md or my name already exists there

[greptile-apps]

Greptile Summary

The PR changes AppLauncher to select the renderer through its CUDA-indexed GPU setting while retaining the CUDA index for physics.

• Adds /renderer/multiGpu/activeCudaGpus=<device>, through SimulationApp.extra_args.
• Leaves /renderer/activeGpu unset so the renderer performs CUDA-to-graphics-device translation.
• Preserves existing caller-provided extra_args and adds focused argument-construction tests.
• Documents the fix for non-zero-based CUDA_VISIBLE_DEVICES masks in the changelog.

Confidence Score: 5/5

The PR appears safe to merge, with no concrete blocking or independently actionable non-blocking issue identified.

The launcher consistently forwards the resolved CUDA index to physics and the renderer’s CUDA-indexed setting, leaves the conflicting graphics-index setting unset, and preserves existing extra arguments with focused regression coverage.

Important Files Changed

Filename	Overview
source/isaaclab/isaaclab/app/app_launcher.py	Replaces direct graphics-index renderer selection with CUDA-index translation while preserving physics and caller extra arguments; no actionable defect was established.
source/isaaclab/test/app/test_app_launcher_argv.py	Adds focused tests for renderer argument formatting, activeGpu omission, physics indexing, and preservation of caller arguments.
source/isaaclab/changelog.d/jichuanh-fix-renderer-active-gpu-physical-index.rst	Accurately documents the masked-GPU startup failure and the new CUDA-indexed renderer selection.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart LR
    A["AppLauncher device (CUDA index)"] --> B["/physics/cudaDevice"]
    A --> C["/renderer/multiGpu/activeCudaGpus"]
    C --> D["Renderer CUDA-to-graphics index translation"]
    D --> E["Selected graphics GPU"]
    B --> F["Selected PhysX CUDA GPU"]

Loading

_{Reviews (1): Last reviewed commit: "Leave activeGpu unset and fold device te..." | Re-trigger Greptile}

[isaaclab-review-bot]

Isaac Lab Review Bot

The renderer is now selected through CUDA-indexed /renderer/multiGpu/activeCudaGpus, preserving masked-device behavior and caller-provided extra_args. However, the launcher does not remove an explicitly supplied active_gpu, allowing both renderer selectors to reach SimulationApp and defeating the intended CUDA-index translation.

• Design and architecture: Using the CUDA-indexed renderer setting aligns renderer placement with CUDA-visible indices used by physics, torch, and Warp. The selection path must also ensure the graphics-indexed active_gpu setting is absent, as the documented translation is only applied when no explicit graphics index is provided.
• API: No public CLI flag or default is renamed, and extra_args is added to the recognized SimulationApp configuration types. The existing accepted active_gpu input now has an ambiguous interaction with the new internally installed selector and should be cleared or explicitly rejected.
• Implementation: The comma-terminated renderer argument and non-destructive extra_args merge are covered by focused tests. A missing case is caller-supplied active_gpu: _resolve_device_settings leaves it in launcher_args, so activeGpu and activeCudaGpus can be forwarded together and the masked-device fix may not take effect.

Minor fixes needed. Posted 1 actionable finding inline.

Automated review; human maintainers own approval decisions.

[isaaclab-review-bot]

🟡 Warning · Api — Caller-supplied active_gpu no longer neutralized

active_gpu is still an accepted launcher setting (_SIM_APP_CFG_TYPES), and this path no longer overwrites it with device_id. A caller passing active_gpu now forwards an explicit /renderer/activeGpu alongside activeCudaGpus; per the invariant documented just above, the CUDA translation is then skipped, so the renderer can silently diverge from physics_gpu and the masked-device failure returns. Consider clearing or rejecting active_gpu when installing the CUDA-indexed selector.

[ooctipus]

can you reduce the comment verbosity?

[ooctipus]

can you combine these test all together? some of them seems can be pytest parameters

[ooctipus]

merge this as the rendering failure is exactly the motion vector due to acidently version bump which already fixed