feat(deviceio): identify the headset from its interaction profile - #926
feat(deviceio): identify the headset from its interaction profile#926jsepulveda-nvidia wants to merge 2 commits into
Conversation
A streaming runtime presents every headset through the same OpenXR surface,
so the obvious fields do not distinguish them -- CloudXR reports
NV_DEVICE_PROFILE=Quest3 with a PICO connected, and that string appears in
all 131 archived server logs regardless of hardware. Consumers that must know
the real device, such as a body-tracking skeleton correction, have had no way
to ask.
The interaction profile the runtime binds does track the hardware. Suggest
the PICO profile alongside the existing Oculus one and expose whichever the
runtime binds, via ControllerTracker.get_interaction_profile and a small
identify_headset mapping.
Measured over CloudXR, both hands, both sessions:
PICO 4 Ultra -> /interaction_profiles/bytedance/pico4_controller
Quest 3 -> /interaction_profiles/oculus/touch_controller
Three constraints shaped this:
The PICO profile is suggested in its own call and its failure is logged, not
thrown. xrSuggestInteractionProfileBindings rejects an entire profile if any
binding path is unsupported, and controller input has to keep working when a
profile is refused. It was accepted on both headsets, so this is belt and
braces rather than a live concern.
Nothing is bound until actions have synced, so the profile cannot be resolved
at construction; an earlier attempt to do so returned XR_NULL_PATH on both
headsets. It is read after sync and re-read only when the runtime reports
interactionProfileChanged, which keeps it off the per-frame IPC path and
handles a headset swap mid-session.
get_interaction_profile is virtual with a default rather than pure, so the
replay implementation and any future one inherit "unknown" unchanged.
identify_headset returns None for anything unrecognised, including the empty
string, rather than falling back to a default. Quest resolves through
oculus/touch_controller, which is the generic Meta-compatible fallback, so it
is an identification by elimination; enabling XR_META_touch_controller_plus
on the instance would make it a positive match too.
Signed-off-by: Juan Sepulveda <jsepulveda@nvidia.com>
|
Important Review skippedAuto incremental reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Enterprise Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
📝 WalkthroughWalkthroughThe change adds OpenXR interaction-profile retrieval to live controller tracking. The profile is cached after synchronization and exposed through the C++ tracker API and Python bindings. The deviceio package maps recognized PICO and Quest profiles to headset identifiers. Tests cover known profiles, unknown inputs, and mapping-table structure. Estimated code review effort: 3 (Moderate) | ~25 minutes Sequence Diagram(s)sequenceDiagram
participant LiveControllerTrackerImpl
participant ActionContextFunctions
participant OpenXRCoreFunctions
LiveControllerTrackerImpl->>ActionContextFunctions: synchronize actions and query current profile
ActionContextFunctions->>OpenXRCoreFunctions: convert profile path with xrPathToString
OpenXRCoreFunctions-->>LiveControllerTrackerImpl: return profile path string
LiveControllerTrackerImpl->>LiveControllerTrackerImpl: cache interaction profile
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 1📝 Generate docstrings 💡
🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/core/oxr_utils/cpp/inc/oxr_utils/oxr_funcs.hpp`:
- Around line 154-155: Update the initialization of
get_current_interaction_profile so xrGetCurrentInteractionProfile2NV is resolved
through a non-throwing optional lookup rather than loadExtensionFunction;
preserve nullptr when the runtime does not expose the symbol, allowing
query_interaction_profile() to handle its absence.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: a4515a35-ef41-4a6b-81fd-2e377003ab6c
📒 Files selected for processing (10)
src/core/deviceio_base/cpp/inc/deviceio_base/controller_tracker_base.hppsrc/core/deviceio_trackers/cpp/controller_tracker.cppsrc/core/deviceio_trackers/cpp/inc/deviceio_trackers/controller_tracker.hppsrc/core/deviceio_trackers/python/tracker_bindings.cppsrc/core/live_trackers/cpp/live_controller_tracker_impl.cppsrc/core/live_trackers/cpp/live_controller_tracker_impl.hppsrc/core/oxr_utils/cpp/inc/oxr_utils/oxr_funcs.hppsrc/core/retargeting_engine_tests/python/test_headset_identity.pysrc/python/isaacteleop/deviceio/__init__.pysrc/python/isaacteleop/deviceio/headset.py
| // Optional: absent on runtimes predating the query. Null-check before use. | ||
| PFN_xrGetCurrentInteractionProfile2NV get_current_interaction_profile; |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
rg -n -C 4 \
'xrGetCurrentInteractionProfile2NV|PFN_xrGetCurrentInteractionProfile2NV|XR_NVX1_ACTION_CONTEXT' \
srcRepository: NVIDIA/IsaacTeleop
Length of output: 3186
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '%s\n' '--- AGENTS files ---'
find .. -name AGENTS.md -print
printf '%s\n' '--- function loader and query call sites ---'
rg -n -C 6 \
'loadExtensionFunction|query_interaction_profile|get_current_interaction_profile|ActionContextFunctions::load|ActionContextFunctions' \
src/core/oxr_utils src/core/live_trackers
printf '%s\n' '--- relevant source slices ---'
for f in $(find src/core/oxr_utils src/core/live_trackers -type f \( -name '*.hpp' -o -name '*.cpp' -o -name '*.cc' \)); do
if rg -q 'loadExtensionFunction|query_interaction_profile|get_current_interaction_profile|ActionContextFunctions' "$f"; then
echo "### $f"
sed -n '1,240p' "$f"
fi
doneRepository: NVIDIA/IsaacTeleop
Length of output: 50375
🏁 Script executed:
#!/bin/bash
set -euo pipefail
for f in AGENTS.md src/core/AGENTS.md src/core/live_trackers/AGENTS.md; do
echo "### $f"
cat -n "$f"
done
echo '### action-context definitions and availability comments'
rg -n -C 8 \
'xrGetCurrentInteractionProfile2NV|XR_NVX1_action_context|INTERACTION_PROFILE_GET_INFO_2_NV|FUNCTION_UNSUPPORTED' \
. \
-g '*.h' -g '*.hpp' -g '*.cpp' -g '*.cc' -g '*.md' \
-g '!build/**' -g '!dist/**' -g '!node_modules/**'Repository: NVIDIA/IsaacTeleop
Length of output: 40679
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo '### extension header: declarations and function-pointer section'
sed -n '130,235p' deps/cloudxr/openxr_extensions/XR_NVX1_action_context.h
echo '### changes relative to main for the affected files'
git diff --stat main -- \
src/core/oxr_utils/cpp/inc/oxr_utils/oxr_funcs.hpp \
src/core/live_trackers/cpp/live_controller_tracker_impl.cpp
echo '### affected diff hunks'
git diff --unified=12 main -- \
src/core/oxr_utils/cpp/inc/oxr_utils/oxr_funcs.hpp \
src/core/live_trackers/cpp/live_controller_tracker_impl.cppRepository: NVIDIA/IsaacTeleop
Length of output: 19405
🌐 Web query:
OpenXR XR_NVX1_action_context xrGetCurrentInteractionProfile2NV optional function runtime predating query specification
💡 Result:
The function xrGetCurrentInteractionProfile2NV is a vendor-specific extension provided by NVIDIA as part of the XR_NVX1_action_context OpenXR extension [1]. It is not a standard part of the core OpenXR specification [2]. The core OpenXR API provides xrGetCurrentInteractionProfile [2], which retrieves the current interaction profile for a specified top-level user path [2]. Unlike the standard function, the extension function xrGetCurrentInteractionProfile2NV allows for more granular control by accepting an XrInteractionProfileGetInfo2NV structure [1]. This structure includes an XrSessionActionContextNV, enabling the retrieval of interaction profiles within a specific action context [1]. This is particularly useful for advanced use cases, such as isolating inputs from multiple sources or devices within a single session [3][4], which the standard core function does not natively support through a context parameter. Because XR_NVX1_action_context is a vendor extension, its use is optional and implementation-dependent [1]. Runtime support for this extension is typically limited to NVIDIA's specific hardware and software stacks, such as CloudXR, where it is used to manage device-specific input mapping and interaction states [1][5]. Applications requiring this functionality must check for the presence of the XR_NVX1_action_context extension during initialization [3][4]. If the extension is not supported by the runtime, applications should fall back to standard core OpenXR functions or adapt their interaction handling accordingly.
Citations:
- 1: https://github.com/NVIDIA/IsaacTeleop/blob/main/deps/cloudxr/openxr_extensions/XR_NVX1_action_context.h
- 2: https://registry.khronos.org/OpenXR/specs/1.1/man/html/xrGetCurrentInteractionProfile.html
- 3: https://github.com/NVIDIA/IsaacTeleop/blob/main/examples/oxr/python/test_controller_tracker.py
- 4: d128609
- 5: [Bug Report] Hand Tracking does not work with beta release isaac-sim/IsaacLab#6214
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo '### local extension history summary'
git log --oneline --all -- deps/cloudxr/openxr_extensions/XR_NVX1_action_context.h \
src/core/oxr_utils/cpp/inc/oxr_utils/oxr_funcs.hpp | head -40
echo '### all local references to the extension and query'
rg -n -C 3 \
'XR_NVX1_action_context_SPEC_VERSION|xrGetCurrentInteractionProfile2NV|XR_TYPE_INTERACTION_PROFILE_GET_INFO_2_NV' \
. \
-g '!build/**' -g '!dist/**' -g '!node_modules/**'
echo '### upstream header metadata and query references'
curl -L --fail --silent --show-error \
https://raw.githubusercontent.com/NVIDIA/IsaacTeleop/main/deps/cloudxr/openxr_extensions/XR_NVX1_action_context.h |
rg -n -C 8 \
'XR_NVX1_action_context_SPEC_VERSION|xrGetCurrentInteractionProfile2NV|INTERACTION_PROFILE_GET_INFO_2_NV'Repository: NVIDIA/IsaacTeleop
Length of output: 7649
🏁 Script executed:
#!/bin/bash
set -euo pipefail
python3 - <<'PY'
from pathlib import Path
funcs = Path("src/core/oxr_utils/cpp/inc/oxr_utils/oxr_funcs.hpp").read_text()
tracker = Path("src/core/live_trackers/cpp/live_controller_tracker_impl.cpp").read_text()
assert 'if (XR_FAILED(r))' in funcs
assert '"Failed to get " + name' in funcs
optional_load = (
'loadExtensionFunction(instance, getProcAddr, "xrGetCurrentInteractionProfile2NV",'
in funcs
)
assert optional_load
required_check_start = funcs.index("if (!f.create_instance_ctx")
required_check = funcs[required_check_start:funcs.index("return f;", required_check_start)]
assert "get_current_interaction_profile" not in required_check
guard = (
"if (action_ctx_funcs_.get_current_interaction_profile == nullptr)"
in tracker
)
assert guard
load_call = "ActionContextFunctions::load(handles.instance, handles.xrGetInstanceProcAddr)"
constructor_pos = tracker.index(load_call)
guard_pos = tracker.index(
"if (action_ctx_funcs_.get_current_interaction_profile == nullptr)"
)
assert constructor_pos < guard_pos
print("PASS: optional query lookup uses the throwing loader, is excluded from required validation, "
"and is null-checked only after construction.")
PYRepository: NVIDIA/IsaacTeleop
Length of output: 244
🏁 Script executed:
#!/bin/bash
set -euo pipefail
python3 - <<'PY'
from pathlib import Path
funcs = Path("src/core/oxr_utils/cpp/inc/oxr_utils/oxr_funcs.hpp").read_text()
tracker = Path("src/core/live_trackers/cpp/live_controller_tracker_impl.cpp").read_text()
assert "if (XR_FAILED(r))" in funcs
assert 'std::string("Failed to get ") + name' in funcs
assert (
'loadExtensionFunction(instance, getProcAddr, "xrGetCurrentInteractionProfile2NV",'
in funcs
)
required_check_start = funcs.index("if (!f.create_instance_ctx")
required_check_end = funcs.index("return f;", required_check_start)
required_check = funcs[required_check_start:required_check_end]
assert "get_current_interaction_profile" not in required_check
assert "if (action_ctx_funcs_.get_current_interaction_profile == nullptr)" in tracker
assert tracker.index(
"ActionContextFunctions::load(handles.instance, handles.xrGetInstanceProcAddr)"
) < tracker.index(
"if (action_ctx_funcs_.get_current_interaction_profile == nullptr)"
)
print("PASS")
PYRepository: NVIDIA/IsaacTeleop
Length of output: 161
Load xrGetCurrentInteractionProfile2NV as optional.
loadExtensionFunction throws when the symbol is unavailable, so controller tracker construction fails before query_interaction_profile() can handle nullptr. Use a non-throwing lookup and retain nullptr when the function is absent.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/core/oxr_utils/cpp/inc/oxr_utils/oxr_funcs.hpp` around lines 154 - 155,
Update the initialization of get_current_interaction_profile so
xrGetCurrentInteractionProfile2NV is resolved through a non-throwing optional
lookup rather than loadExtensionFunction; preserve nullptr when the runtime does
not expose the symbol, allowing query_interaction_profile() to handle its
absence.
|
Does that mean we can require |
| } | ||
|
|
||
|
|
||
| def identify_headset(interaction_profile: Optional[str]) -> Optional[str]: |
There was a problem hiding this comment.
I better place for this will be python/isaacteleop/cloudxr
context: the deviceio module is meant to be C++ only and the python bits are its bindings.
|
Hey @jiwenc-nv , So I finally finished all the real robot testing just now and I'll address all these concerns. First, in the case of NV_DEVICE, I found this to be highly unreliable and possibly either hard-coded or just not set. In all the sessions I ran the logs showed that NV_DEVICE_PROFILE=Quest3 appeared with a PICO connected, in something like 131 archived logs. It seems to be a configuration input rather than a detection output. After the partial success of this project, detection could plausibly set it — which could be for our users, but it's also a CloudXR change and I've been able to do all this work without touching CloudXR so far. |
…g to cloudxr Two review findings. loadExtensionFunction throws when a symbol is missing, so declaring xrGetCurrentInteractionProfile2NV optional and null-checking it at the call site was not enough: ActionContextFunctions::load would throw first and take controller tracker construction with it on any runtime lacking the function. Look the symbol up directly and leave the pointer null instead, which is what the null-check already expected. CloudXR provides the function, so this was latent rather than observed. identify_headset and its table move from isaacteleop.deviceio to isaacteleop.cloudxr: deviceio is a C++ module plus its bindings, and this is neither. The mapping is also specific to what a CloudXR runtime reports, so cloudxr is where a reader would look for it. Signed-off-by: Juan Sepulveda <jsepulveda@nvidia.com>
NVIDIA/IsaacTeleop#926 moved identify_headset out of isaacteleop.deviceio, on review feedback that deviceio is a C++ module plus its bindings. The cross-check sits inside a try/except ImportError, so the move would not have raised -- it would have silently stopped cross-checking, which is worse. Try the new location and fall back to the old one, so this works against isaacteleop builds from either side of that change.
2 participants
Description
Before conducting a code review, its best to learn how the big picture of headset detection and remapping works form this doc: https://docs.google.com/document/d/1dOX4aN6cnj9VV8iaMVsjhS7loJGrN_lSWixJVMpJelY/edit?usp=sharing
A streaming runtime presents every headset through the same OpenXR surface, so the obvious fields do not distinguish them. CloudXR reports
NV_DEVICE_PROFILE=Quest3with a PICO connected, and that same string appears in all 131 archived server logs regardless of hardware. Consumers that must know the real device — a body-tracking skeleton correction, say, where the wrong answer means permuted joint axes — have had no way to ask.The interaction profile the runtime binds for the controllers does track the hardware. This suggests the PICO profile alongside the Oculus one already suggested, and exposes whichever the runtime binds:
ControllerTracker.get_interaction_profile(session)→ the OpenXR path, or empty until actions have syncedisaacteleop.deviceio.identify_headset(profile)→"pico","quest", orNoneType of change
Additive. The existing Oculus suggestion, and every current code path, behave exactly as before.
Testing
Measured over CloudXR on real hardware, both hands, both sessions:
/interaction_profiles/bytedance/pico4_controller/interaction_profiles/oculus/touch_controller18 unit tests cover the mapping: the two measured profiles, that they do not collapse to one answer, that a
bytedancepath can never map toquest, and that unknown input returnsNone— including"quest3"(the misleading CloudXR device-profile string) and near-misses like a path missing its leading/interaction_profiles/.clang-format-14 --dry-run -Werror,pre-commit, and the mapping tests all pass.This was arrived at empirically. A throwaway probe first queried the profile at tracker construction and returned
XR_NULL_PATHon both headsets: nothing is bound until actions have synced. That is why the read happens after sync, and it is worth knowing before anyone tries to resolve this earlier.Design notes for reviewers
The PICO suggestion is deliberately tolerant.
xrSuggestInteractionProfileBindingsrejects an entire profile if any binding path is unsupported, and the binding list here is Oculus-shaped. A rejection must not take controller initialisation down with it, so it is a separate call whose failure is logged rather than thrown. It was accepted on both headsets, so this is belt-and-braces rather than a live concern.The profile is cached, not polled. It is read after a successful sync and re-read only when the runtime reports
interactionProfileChanged, which keeps it off the per-frame IPC path while still catching a headset swap mid-session.get_interaction_profileis virtual with a default, not pure, so the replay implementation and any future one inherit"unknown"without modification.identify_headsetnever guesses. Unrecognised input returnsNonerather than a default. Note that Quest resolves throughoculus/touch_controller, which is the generic Meta-compatible fallback — so PICO is a positive match while Quest is identification by elimination. EnablingXR_META_touch_controller_pluson the instance would make Quest positive too; it was rejected here because the extension is not enabled, and it is not needed for the two devices to be told apart.Checklist
SKIP=check-copyright-year pre-commit run --all-filesgit commit -s) per the DCONo
docs/page yet; the module and tests carry their own documentation. The C++ accessor itself has no unit test — it needs a live runtime and a headset — but the code path it uses was exercised on both devices via the probe described above.Summary by CodeRabbit