Skip to content

perf: Order OVRTX render-var reads on the consuming stream, and block the host on Linux - #7074

Open
pv-nvidia wants to merge 1 commit into
isaac-sim:developfrom
pv-nvidia:pv/ovrtx-render-var-sync
Open

perf: Order OVRTX render-var reads on the consuming stream, and block the host on Linux #7074
pv-nvidia wants to merge 1 commit into
isaac-sim:developfrom
pv-nvidia:pv/ovrtx-render-var-sync

Conversation

@pv-nvidia

@pv-nvidia pv-nvidia commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

Description

Reading an OVRTX render var has to be ordered against render completion. On Linux, blocking the calling thread on the render-completion event measures faster end to end than a GPU-side wait, so that is now what Linux does.

Measured across three camera tasks at 256/1024/4096 envs, this is +14% to +73% throughput (mean +36%), and restores parity with ovrtx 0.3 on every case measured.

Other platforms order the read on the consuming Warp stream. ISAAC_LAB_OVRTX_DISABLE_LINUX_CUDA_CPU_SYNC=1 puts Linux on that ordering too — an escape hatch for when the trade-off changes; it is not currently faster.

All mapping sites now go through one helper, OVRTXRenderer._map_render_var_to_dlpack(). Camera outputs are unchanged — only the read ordering differs.

Reported in https://nvbugspro.nvidia.com/bug/6566453

Type of change

  • Bug fix (non-breaking change which fixes an issue)
  • Performance improvement

Screenshots

None — no visual change.

Checklist

  • I have run the pre-commit checks with ./isaaclab.sh --format
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • I have updated the changelog and the corresponding version in the extension's config/extension.toml file
  • I have added my name to the CONTRIBUTORS.md or my name already exists there

@github-actions github-actions Bot added the isaac-lab Related to Isaac Lab team label Aug 13, 2026
@pv-nvidia
pv-nvidia force-pushed the pv/ovrtx-render-var-sync branch from a249430 to 0c61aee Compare August 13, 2026 11:45
@pv-nvidia
pv-nvidia marked this pull request as ready for review August 13, 2026 12:16
@pv-nvidia
pv-nvidia requested a review from a team August 13, 2026 12:16
Comment thread source/isaaclab_ov/isaaclab_ov/renderers/ovrtx_renderer.py Outdated
@greptile-apps

greptile-apps Bot commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

The PR centralizes CUDA render-var mapping and changes Linux ordering to use a host-side render-completion wait, avoiding the reported GPU scheduling bottleneck while retaining an environment-controlled GPU-side path.

  • Routes CUDA camera render-var mappings through a shared context-managed helper.
  • Uses the consuming Warp stream for GPU-side synchronization and a no-sync mapping followed by a host wait on Linux.
  • Adds contract tests for platform and environment selection, synchronization ordering, DLPack conversion, and mapping release.
  • Documents the Linux throughput improvement and opt-out setting.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename Overview
source/isaaclab_ov/isaaclab_ov/renderers/ovrtx_renderer.py Centralizes render-var mapping and selects host-side synchronization on Linux while preserving consuming-stream synchronization elsewhere.
source/isaaclab_ov/test/test_ovrtx_renderer_contract.py Adds focused contract coverage for synchronization-mode selection, map arguments, wait ordering, DLPack conversion, and cleanup on failure.
source/isaaclab_ov/changelog.d/pv-ovrtx-render-var-sync.rst Documents the Linux camera-output throughput improvement and the environment-variable opt-out.

Sequence Diagram

sequenceDiagram
    participant Caller as Camera output processing
    participant Helper as _map_render_var_to_dlpack
    participant OVRTX as RenderVarOutput
    participant Warp as Consuming Warp stream
    alt Linux host synchronization
        Helper->>OVRTX: "map(CUDA, sync_stream=0)"
        Helper->>OVRTX: mapping.wait()
        OVRTX-->>Helper: render complete
    else GPU-side synchronization
        Helper->>Warp: get consuming CUDA stream
        Helper->>OVRTX: "map(CUDA, sync_stream=stream)"
        OVRTX-->>Warp: enqueue render-completion wait
    end
    Helper->>Warp: wp.from_dlpack(mapping)
    Helper-->>Caller: mapped Warp array
    Caller->>Warp: enqueue output extraction
    Helper->>OVRTX: release mapping
Loading

Reviews (2): Last reviewed commit: "Bot review feedback" | Re-trigger Greptile

Comment thread source/isaaclab_ov/isaaclab_ov/renderers/ovrtx_renderer.py
Comment thread source/isaaclab_ov/test/test_ovrtx_renderer_contract.py
@pv-nvidia
pv-nvidia force-pushed the pv/ovrtx-render-var-sync branch from 0c61aee to 4374b7c Compare August 13, 2026 15:02
@pv-nvidia pv-nvidia changed the title Improved OVRTX camera-output throughput on Linux perf: Order OVRTX render-var reads on the consuming stream, and block the host on Linux Aug 14, 2026
@pv-nvidia
pv-nvidia force-pushed the pv/ovrtx-render-var-sync branch from 4374b7c to b56335a Compare August 14, 2026 11:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

isaac-lab Related to Isaac Lab team

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants