Detect the headset automatically and correct the skeleton to match - #2
Open
jsepulveda-nvidia wants to merge 24 commits into
Open
Detect the headset automatically and correct the skeleton to match#2jsepulveda-nvidia wants to merge 24 commits into
jsepulveda-nvidia wants to merge 24 commits into
Conversation
Quest 3 body tracking produces erratic, unsafe robot behaviour while the LOVR body_tracking sample renders a correct-looking skeleton from the same data. That asymmetry is explained by what each consumer reads: LOVR draws positions only (main.lua:271,283 -- quaternions are stored but never rendered), whereas compute_from_body_poses() derives the entire SMPL pose from the 24 quaternions and uses only positions[0] for root translation. Positions can therefore look correct while orientations are unusable. input_readers._body_data_to_24x7() discards per-joint is_valid -- invalid joints are skipped, leaving a zero position and a zero quaternion -- so no existing code path can distinguish a measured joint from an unmeasured one. This tool records the raw joints with validity preserved and reports, per joint, valid%, positional spread relative to the pelvis, orientation spread, and correlation with root motion. A joint reported 100% valid with near-zero orientation variance and root correlation ~1.0 is being synthesised rather than measured. --compare puts two recordings side by side. Relevant because Meta IOBT does not camera-track the legs; the BD 24-joint layout demands 8 lower-body joints (1,2,4,5,7,8,10,11) the Quest never measures, and the robot's failure mode is falling over. Upstream acknowledges reduced Quest lower-body accuracy in IsaacTeleop docs/source/device/body_tracking.rst:221-225. Analysis path validated against synthetic fixtures: correctly flags 8 frozen leg joints and no upper-body joints. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The --compare mode reports orientation *variance*, which is the wrong metric for a held pose. --rest-pose reports the per-joint mean-orientation offset of one capture relative to another, and tests whether a single rotation explains each anatomical group. Guards on staticness first: mean orientation is only the rest pose if the capture was actually held. During free motion the mean reflects whatever movement was performed, which invalidated an earlier round of this analysis. Also prints the caveat that one static pose cannot separate a constant rest-pose offset from a pose-dependent error -- that needs a second pose. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Quest body pose reaches this machine already converted to the ByteDance 24-joint layout by the closed CloudXR client SDK. Joint order and positions survive that conversion -- the LOVR body_tracking sample, which renders positions only, looks anatomically correct on both headsets -- but the joint orientations do not, and orientation is what drives the robot: compute_from_body_poses() builds the whole SMPL pose from the 24 quaternions and uses positions only for root translation. Adds a per-joint correction measured in parent-relative space from a held T-pose captured on both headsets, applied behind --skeleton-source quest. Default stays 'pico', so existing behaviour is untouched. The dominant defect fixed is LEFT_HIP arriving ~180 deg flipped (164.8 -> 10.2 deg) while RIGHT_HIP is nearly correct; that asymmetry inverts the left leg and is a plausible cause of the observed falling. Validation, held out against an independently captured A-pose: - local-frame joint error: 18/23 joints within 20 deg, vs 12/23 before - SMPL joint error vs Pico: 647 -> 305 mm mean (53% better) Still a partial fix. The ~300 mm residual is genuine pose error, not a facing artifact (yaw-aligning changes it by <5 mm), and is spread fairly evenly across legs, spine and arms rather than concentrated in the known-bad joints. RIGHT_SHOULDER, LEFT_ELBOW and both WRISTs remain pose-dependent and are flagged LOW_CONFIDENCE_JOINTS. Test in sim before hardware; the failure mode being corrected is the robot falling over. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Switching headsets is the expensive part of this calibration, so capture all poses in one session per device rather than one invocation per pose. --batch prompts for each pose, counts down, records, and reports staticness immediately so a bad take can be redone on the spot. Eight poses, chosen for REPRODUCIBILITY across sessions rather than coverage -- an earlier round was wasted because "arms down" is not a well-defined pose and the two captures did not match. Each has an unambiguous endpoint (straight, 90 degrees, seated). Between them every joint group lands in at least two clearly different configurations, which is what separates a constant rest-pose offset from a pose-dependent one: tpose baseline, matches the current correction table arms_forward shoulder flexion 90, elbows straight arms_forward_palms_up isolates wrist roll (only wrists change vs previous) goalpost elbow 90 with shoulder abducted arms_overhead far end of shoulder range bend_forward spine + hip flexion, never previously exercised lean_right lateral spine bend, breaks left/right symmetry sitting hip and knee flexion 90, the only leg-bending pose Prints the facing/position ground rules up front: a whole-body turn between captures previously corrupted an entire round of analysis. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The operator is wearing a headset and holding controllers while posing, so
pressing ENTER between poses is impossible. Two ways to advance now:
* Pull either controller trigger (or press ENTER -- whichever comes first).
Requires a release before a press, so a trigger still held from the
previous pose does not instantly advance the next one.
* --auto: fully hands-free, each pose gets --pose-delay seconds to assume.
Adds terminal-bell cues on the countdown and at the start of recording, since
the operator cannot read the terminal while posing.
_stdin_ready() requires a tty: a closed or redirected stdin selects as
readable immediately at EOF, which would silently auto-advance every pose.
Controller reads are wrapped so missing or unsupported controllers degrade to
the ENTER path rather than crashing mid-battery.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Two changes, the second of which was a real bug.
1. Refit across the full 8-pose battery instead of a single T-pose.
2. Fix the frame. compute_from_body_poses() post-multiplies every global
rotation by Ry(180 deg) before forming parent-relative rotations, so its
local frame is the raw skeleton's local frame conjugated by Ry180. The
previous table was fitted from raw captures and applied without that
conjugation. The stored values are now conjugated to match.
This was measurable, not theoretical: an 8-pose refit applied in the wrong
frame scored 643 mm versus 599 mm for no correction at all -- worse than
doing nothing -- while the same fit in the correct frame scores 216 mm.
Joint 0 is exempt: it is not parent-relative, so it takes the plain global
offset with no conjugation. Its correction also keeps yaw; the two headsets
reported consistently different pelvis facing (Pico ~6 deg, Quest ~88 deg,
each stable within ~10 deg across all 8 poses), so that is a device frame
convention rather than operator stance. Dropping yaw cost 722 mm vs 190 mm.
Leave-one-pose-out validation (fit on 7, evaluate on the held-out 8th),
mean SMPL joint error vs the Pico reference:
raw 599 mm
previous T-pose table 319 mm
this table 216 mm 64% better than raw, 32% better than v1
Wins on all 8 held-out poses individually.
LOW_CONFIDENCE_JOINTS is now LEFT_SHOULDER, RIGHT_SHOULDER and RIGHT_HAND --
offsets varying >30 deg across poses. Legs and spine, which govern balance, are
well corrected.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Fitting the main battery left LEFT_SHOULDER and RIGHT_SHOULDER pose-dependent
(leave-one-out residual 43 and 73 deg) while every other joint corrected well.
Two explanations were tested against the existing captures and both failed: the
collar+shoulder composite is no more stable than the shoulder alone (44/58 deg
vs 38/60), and the offset does not track arm elevation. So it is neither a
redistribution artefact nor a simple function of arm pose.
With one capture per pose we cannot distinguish a structured pose-dependent
offset (modellable with a richer fit) from an unrepeatable one (nothing will
fix it, and arm swing should instead be derived from positions, which agree
2-4x better than the shoulder quaternions).
Six poses:
tpose_r2, arms_forward_r2, goalpost_r2 exact repeats of main-battery poses,
captured in a fresh session -- comparing the same pose across sessions
measures repeatability directly.
elbow90_forward / _inward / _outward upper arm held in one place while
only shoulder axial roll varies. Roll is the component positions
constrain least and the likeliest place for the SDK conversion to fail.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The constant per-joint correction leaves both shoulders 50-69 deg out while
every other joint sits near 6 deg. Since the shoulder is upstream of the elbow,
that error rotates the whole forearm -- the reported symptom of a wrong left
elbow orientation with otherwise-good wrists.
Swing-from-position, the obvious fix, is provably a no-op and is NOT used.
Each device's shoulder quaternion already points exactly at its own elbow
(0.0 deg bone-axis spread over 8 poses), so positions and the quaternion carry
the same directional information. Measured end to end: 190 mm either way.
Also ruled out with the new captures: collar/shoulder redistribution (composite
no more stable than the shoulder alone), arm elevation (no relationship), and
shoulder axial roll (an elbow-at-side roll trio moved the offset only 15-22 deg,
within noise).
What does work: the offset is repeatable enough to interpolate. Repeating three
poses in a fresh session reproduced the shoulder offset to ~21 deg versus ~6 deg
for other joints, so the 50-69 deg pose-dependence is well above the noise
floor. Shoulder offsets are now stored as 14 (arm_direction, offset) samples and
blended with Gaussian weights on arm direction at runtime.
Leave-one-pose-out, mean SMPL joint error vs the Pico reference:
raw 599 mm
constant 190 mm
adaptive 163 mm 14% better than constant, held out
Elbows deliberately keep the constant offset -- blending did not help them
(37.4 -> 37.3 and 42.5 -> 35.3).
The blend is smooth by construction rather than nearest-neighbour: measured
max 2.67 deg change per 1 deg of arm motion, so it will not step the correction
and jerk the robot. Queries further than MAX_ANGLE_DEG from every sample fall
back to the constant offset rather than extrapolating.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Operator yaw was being turned into robot pitch, and operator pitch into robot roll: turning left made the robot lean forward, leaning forward made it drop a shoulder, and ~30 deg of body rotation was enough to topple it. Cause: the fitted root correction is a 117 deg rotation that cyclically permutes the axes. The root correction left-multiplies the body orientation, so it also remaps which robot axis an operator rotation drives -- corr.inv() sent world +Y to -X. Predicted axis mapping matches the reported symptoms exactly. Why the fit produced it: every capture was taken facing the same direction, as instructed, to avoid a different confound. The devices differ by both a world-frame rotation A and a body-frame rotation B, G_quest = A * G_pico * B. Only A should be corrected, since A is what maps rotation axes. From a single body orientation the fit returns A composed with a conjugated B -- still minimal static pose error, but the wrong axis mapping. Separating them needs captures at several body orientations (an AX = XB problem); that data does not exist yet. Root correction is now identity. This costs nothing for body pose -- 126 mm root-relative either way -- because it only ever affected absolute heading, which the teleop heading calibration (A+B+X+Y) establishes anyway. Also adds a startup guard: for each world axis a, corr.inv().apply(a) must return a. Anything else raises rather than starting, so a future refit cannot reintroduce this silently. Verified the guard fires on the old value. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The previous commit set the root correction to identity to stop it permuting
the rotation axes. That fixed the axis mapping but broke uprightness: the legs
collapsed immediately on connect.
Both failures come from trying to do two jobs with one rotation. The devices
differ by a world-frame rotation A and a body-frame rotation B:
G_quest = A * G_pico * B
A decides which robot axis an operator rotation drives. B decides whether the
body reads as upright. A single left-multiply can only be one of them:
* the original 117 deg fit kept the body upright but permuted the axes --
operator yaw became robot pitch, toppling the robot within ~30 deg of turn;
* identity restored the axis mapping but left B uncorrected, so SMPL was told
the body was tilted ~90 deg while the operator stood upright.
Measured pelvis frames confirm B is real and large: Pico's pelvis local +Y is
world-up (6 deg off vertical) while Quest's local +X is world-DOWN (177 deg).
Solved for A and B jointly across all 14 poses, constraining A to a rotation
about the vertical since both devices agree on gravity and their world frames
can therefore only differ in heading. A = -3.1 deg yaw, B = 122.7 deg, fit
residual median 9.6 deg over 14 poses. Applied as
corrected_root = ROOT_LEFT.inv() * local_rots[0] * ROOT_RIGHT
with ROOT_RIGHT folding in the Ry180 conjugation. Because A is near identity the
axis mapping is safe by construction; the startup guard now checks ROOT_LEFT and
still refuses to run on a permuting fit.
Mean SMPL joint error vs the Pico reference across the 8 main poses:
raw 599 mm
identity root (broken) 545 mm
two-sided root 145 mm
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Correcting a joint properly needs a two-sided fit, local_quest = A * local_pico * B, where A is the parent-frame rotation that decides which axis a motion emerges about and B is the child-frame relabelling. Separating them requires the joint to rotate about at least two independent axes. With one axis the fit still matches every static pose while reproducing motion about the wrong one -- which is how a correction can look good on every static metric and still destabilise the robot. Measured axis diversity from the first two batteries (0 = all motion about a single axis): SPINE1 0.08, SPINE2 0.15, SPINE3 0.13, PELVIS 0.61 over only 21 deg, hips/ankles/feet ~0. Arms scored 0.5-0.95, which is why they solve and the torso does not. That under-determination is the direct cause of two reported symptoms: operator waist flexion arriving as robot lateral bend (spine motion axis off by ~86 deg measured), and a persistent leg crouch. Twelve poses rotating the torso about all three axes (flexion, extension, lateral both ways, twist both ways), putting the legs through squat and stride, and standing at four body headings. The headings also condition the pelvis, which every earlier battery held fixed by instruction -- so this battery flips that rule and says so at startup. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…teral bend
Every joint now gets both a left and a right multiply:
local_quest = A * local_pico * B
corrected = LEFT.inv() * local * RIGHT (LEFT = A, RIGHT = B.inv())
A is the parent-frame rotation and decides which axis a motion comes out about.
B is the child-frame relabelling and decides rest orientation. One rotation
cannot do both, and getting it wrong is invisible to static metrics -- which is
how three separate versions of this bug survived every check I had:
* a fitted root kept the body upright but permuted the axes, so operator yaw
became robot pitch and the robot toppled within ~30 deg of turn;
* zeroing that root fixed the axes but dropped B, so SMPL was told the body
was tilted ~90 deg and the legs collapsed on connect;
* the spine stayed left-only and turned operator waist flexion into robot
lateral bend -- 87 deg between the operator's rotation axis and the robot's.
Separating A from B needs each joint to rotate about at least two independent
axes. The torso battery supplied that: SPINE1 axis diversity 0.08 -> 0.58, and
the pelvis went from one fixed heading to four. Fitted over 26 poses by
alternating SVD; median static residual 8 deg.
Motion-axis error over pose pairs with >30 deg of real motion:
SPINE1 87.0 -> 7.7 LEFT_SHOULDER 50.4 -> 11.4
LEFT_HIP 87.6 -> 8.5 RIGHT_SHOULDER 107.6 -> 12.8
RIGHT_HIP 85.9 -> 7.7 elbows 71/102 -> 30/29
LEFT_KNEE 103.9 -> 3.8 wrists 27/17 -> 14/14
RIGHT_KNEE 79.2 -> 2.4
Mean SMPL joint error vs the Pico reference across 20 poses: 629 mm raw,
168 mm with the previous left-only table, 127 mm now.
The pelvis A came out at 4.2 deg from an unconstrained fit, independently
confirming that both devices agree on gravity and their world frames differ only
in heading -- so the axis mapping is safe rather than merely constrained to be.
The startup guard still checks this and refuses to run on a permuting fit.
Removes quest_shoulder_adaptive.py. Its pose-dependent blending was compensating
for a left-only fit forced onto a two-sided problem; the two-sided fit beats it
outright (shoulder residual 11-13 deg vs 21-29) with no runtime interpolation.
RIGHT_HAND keeps a one-sided correction -- two-sided made its motion-axis error
worse (14 -> 64 deg) with no SMPL benefit.
Documents that Quest infers legs from vision: residual is 6.9 deg median but
12.4 deg on leg-bending poses, worst RIGHT_HIP at 27 deg in a deep squat.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
At the rest pose an operator expects to be neutral -- forearms forward, palms
facing each other, captured as 04_elbow90_forward -- the commanded G1 wrist
pitch is not neutral on either headset:
side Pico Quest delta
left -11.7 -14.3 -2.6
right -10.1 -16.4 -6.3
WRIST_PITCH_BIAS_RAD cancels only the delta, so the Quest lands where the Pico
already sits. Verified: residual after bias is 0.07 deg left, 0.02 deg right.
Pico is zero by intent, not oversight. Its teleop has been tuned by people over
a long time and its non-zero rest value may be correct for the robot's
mechanical neutral rather than an error; this data cannot distinguish those, so
changing it would risk invalidating that tuning on an assumption. The Pico path
is bit-identical to before.
This is a BIAS fix only and does not address the reported range asymmetry. Across
all 26 captured poses the commanded wrist pitch spans just 17-27 deg on an axis
with roughly 180 deg of travel: no pose in any battery flexes the wrist, and the
one wrist-specific pair (palms down / palms up) exercises roll rather than
flexion. Fitting a gain from data that never leaves the middle of the range
would be fitting noise.
Adds --battery wrist for that: neutral, flexion max, extension max, both
half-way points, and a deviation pose to separate the flexion axis from the
other two. The forearm is held fixed throughout so elbow motion does not leak
into the measurement, since the retargeting already folds elbow swing into the
wrist channels.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
… bias
The wrist entries in the correction table were fitted from poses that never
flexed the wrist, leaving them essentially unconstrained. The wrist battery
supplies the missing motion: L_WRIST axis diversity 0.46 -> 0.65 and range
42 -> 77 deg, R_WRIST 0.54 -> 0.73 and 39 -> 75 deg.
Refit over all 32 poses. On the wrist sweep specifically:
joint static residual motion-axis error
L_WRIST 30.6 -> 20.7 18.6 -> 12.7
R_WRIST 25.6 -> 15.2 23.0 -> 11.3
Also better across all 32 poses (L 23.3 -> 18.8, R 26.4 -> 16.1 axis error), so
this is not a trade against the rest of the body. Whole-body SMPL error over a
9-pose spread including the wrist sweep: 584 mm raw -> 124 mm corrected.
The pitch bias is re-anchored on 01_wrist_neutral, a genuine neutral wrist. The
previous anchor (04_elbow90_forward) was chosen before any battery flexed the
wrist, so it pinned the bias against a pose whose own wrist value was unverified.
Bias moves from +2.60/+6.30 to +11.37/+16.72 deg; residual at neutral is now
0.000 deg on both sides. Pico stays zero and its path is bit-identical.
Documents a separate limitation this table cannot fix. The battery showed the
commanded G1 wrist PITCH barely responds to flexion on BOTH headsets: a full
flex-to-extend sweep rotates the raw joint 132 deg (Pico) and 104 deg (Quest)
while commanded pitch moves ~25 deg, and is not even monotonic on the Pico. In
this arm configuration the flexion sits almost entirely in the Z euler component
of the SMPL wrist rotation -- Pico left wrist Z spans 131 deg while Y spans 19 --
and the retargeting takes Y as pitch while routing Z to wrist YAW. That is
shared, long-tuned code affecting the Pico identically, so it is left alone and
recorded instead.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The pitch-only bias was already exact at neutral (delta -0.0 and +0.1 deg) yet
the robot's hands still sat visibly extended there. Measuring all three channels
at 01_wrist_neutral showed why: the error was in YAW, ~29 deg per side.
channel Pico Quest bias applied
L roll -33.6 -24.5 -9.37
L pitch +7.2 +7.1 +11.44
L yaw +15.2 +44.1 -29.88
R roll +22.3 +19.7 +2.74
R pitch +9.4 +9.4 +16.87
R yaw -19.9 -48.2 +28.13
That is consistent with the routing already documented under "Wrist range":
operator wrist flexion largely lands in the yaw channel rather than pitch, so a
pitch-only bias could not reach the visible error. All six channels now match
the Pico reference at neutral to 0.00 deg.
Deliberate tradeoff, accepted after review. The wrist error is
configuration-dependent, so a constant bias tuned at one arm configuration is
wrong at others -- the same reason the shoulders needed pose-dependent handling.
Mean absolute wrist-channel error vs Pico:
pose group none pitch-only 3-channel
neutral 16.3 11.6 0.2
wrist battery (forearms forward) 14.8 15.1 10.0
other configs (tpose/goalpost/...) 12.3 15.4 21.9
all poses 13.5 15.6 15.9
Chosen because the operator judges correctness at neutral and spends most time
near it; the cost lands on arm configurations that are further from rest.
Pico stays zero on all three channels and its path is bit-identical.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The Pico commands -33.6 / +22.2 deg of wrist roll at a neutral wrist, which renders as visibly rolled palms, and it is asymmetric: the left sits 11.4 deg further round than the right. An operator reported exactly that -- both palms rolled up, the left more -- so the flaw is inherited from the reference, and matching the Pico faithfully reproduced it including the asymmetry. Roll therefore departs from the reference and is driven to zero: L +24.24, R -19.47 deg, giving -0.2 / +0.2 deg at neutral. Pitch and yaw still match the Pico, since those have been improving in trials. This costs nothing in responsiveness. A constant bias shifts the operating point without compressing travel, so roll span across the wrist sweep is unchanged at 91 / 86 deg -- unlike the yaw bias, which traded accuracy at other arm configurations. Zero commanded roll is a HYPOTHESIS about where the robot's neutral lies, not a measurement: there is no ground truth available for "thumbs facing each other" in joint terms. Recorded as such in the module, with a note that these two numbers can be nudged by whatever residual a trial shows, independently of the rest of the table. Pico stays zero on all three channels and its path is bit-identical. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Identifies the headset from the first frames of body tracking and applies the matching correction, so an operator who forgets the flag does not drive the robot with permuted joint axes. Detection uses the skeleton's own geometry. CloudXR normalises the Quest skeleton into the ByteDance layout, so joint order, positions, validity flags, sample rate and quantisation are all indistinguishable between the two headsets -- each was checked and none separates them. What does differ is the per-joint orientation convention, which is what the correction fixes. So the signature is each bone's direction (from positions, device-agnostic) expressed in its parent joint's frame (from the quaternions, device-specific). Over 68 recorded captures -- 34 paired poses on both headsets, 1020 random single frames -- classification is 100% correct with a worst margin of 50 deg against a 20 deg refusal threshold. Correcting a Quest capture makes it classify as PICO, which is what shows the signature tracks the orientation convention rather than something incidental. Deliberately does not gate startup. The earlier auto-detect waited for the headset to identify itself before proceeding, which made controllers look dead and the whole stack look broken. This reads data that is already flowing, so controllers, video and OpenXR come up exactly as with an explicit source. Five consecutive agreeing frames are required before committing, measured at exactly 5 frames for every capture -- about 80 ms at 60 Hz -- so a single degenerate frame cannot decide. A frame of identity quaternions does score as PICO rather than refusing, which the streak requirement is there to absorb. When the runtime can report its interaction profile, that is cross-checked against the geometry result: agreement is logged, disagreement warns and names both answers rather than silently picking one. The check needs an isaacteleop exposing get_interaction_profile (NVIDIA/IsaacTeleop#921's sibling branch); older builds skip it, so auto works today on 1.3.131 from geometry alone. 'pico' remains the default and is untouched: no correction, zero wrist bias. Also commits the quest-upstream A/B mode that was left uncommitted while the correction was being moved into isaacteleop. It shares the argparse choices line with this change, so the two could not be separated cleanly. It is scaffolding for comparing the in-repo and upstream corrections and should be removed once the in-repo copy is retired.
An operator who forgets the flag on a Quest currently drives the robot with permuted joint axes and nothing says so. Defaulting to auto removes that as a way to get it wrong. Safe in both directions. On a PICO, auto resolves to pico, which is no correction and no wrist bias -- identical to the old default. On a Quest it goes from silently wrong to corrected. If detection cannot decide, it leaves the skeleton uncorrected and logs, which is exactly the old default behaviour, so the failure mode is what shipped before rather than something new. Explicit values still pin the behaviour and are worth passing when the headset is known. run_pico_manager's signature default moves to auto with the CLI. PoseStreamer keeps "pico" because _pose_stream_common leaves it unset, so that default governs the non-manager path, where --skeleton-source is documented not to apply; noted at the definition so the mismatch reads as deliberate.
The correction now lives upstream (NVIDIA/IsaacTeleop#921) and is applied in the reader, so the in-repo copy is redundant. Removes SkeletonCorrection, its loader and startup axis guard, and the correction parameter on compute_from_body_poses; quest_skeleton_correction.py drops from 231 lines to 69. --skeleton-source loses 'quest-upstream'. It only ever existed to A/B the two implementations, which have been measured identical, so 'quest' now means the upstream correction and the distinction is gone from the interface. WRIST_BIAS_RAD stays. It is expressed in G1 wrist joint commands rather than skeleton space, so it is specific to this robot rather than to the headset's skeleton convention, and moving it upstream would mean re-deriving it as a skeleton-space rotation. Moving the correction into the reader creates an ordering constraint that did not exist before: auto-detection identifies the headset from the orientation convention the correction rewrites, so it has to see raw frames. Under auto the reader therefore starts with no correction and the manager installs one via the new set_skeleton_profile once detection resolves. An explicit --skeleton-source quest still corrects from the first frame. Verified against the recorded captures: the corrected path moves the pose by up to 1577 mm against uncorrected, so the correction is demonstrably still being applied; auto still resolves 68/68; pico is untouched, with no correction and zero wrist bias.
jsepulveda-nvidia
changed the base branch from
jsepulveda/quest_skeleton
to
main
August 8, 2026 02:14
…v_teleop --skeleton-source quest calls correct_body_orientations, which is added by NVIDIA/IsaacTeleop#921 and ships in isaacteleop 1.5+, while .venv_teleop pins 1.3.131. Until that PR merges and this repo moves to 1.5, the correction only reaches an operator who has a local IsaacTeleop checkout on the jsepulveda/quest_remapping branch plus three hand edits inside the venv. Those edits were undocumented and had to be reproduced from memory on each machine. Three steps: symlink full_body_transform.py out of the IsaacTeleop checkout, alias NUM_BODY_JOINTS (1.5 renamed it from NUM_BODY_JOINTS_PICO), and re-export the symbols from utilities/__init__. --check reports without touching anything and --revert undoes it. Reinstalling the venv against 1.5 would be the honest fix, but it drags in a C++ rebuild and a minor-version jump, which is a poor trade while the stack is being tested. The failure this is really guarding against is silent: the symlink target only exists on one branch, so checking that repo out elsewhere leaves it dangling and teleop fails with an unrelated-looking ImportError. The script names that cause directly, and always finishes by importing the function and asserting that pico is identity and quest is not -- a green checklist alone would not catch a stale symlink or a version skew. Verified by reverting the hand-made bridge and rebuilding it from scratch: the corrected pose still differs from uncorrected by 1577 mm across the recorded captures and auto-detection still resolves 68/68, matching the numbers before the rebuild.
…w data On a Quest with --skeleton-source auto, the robot turned sharply on the first POSE -> PLANNER toggle, and never again in that session. Explicit --skeleton-source quest was unaffected, and so was the PICO. Detection resolved inside PoseStreamer.run_once, which only runs in POSE mode. A+B+X+Y goes OFF -> PLANNER directly and calls ThreePointPose.calibrate_now() straight away, so on that path the correction had not been installed yet and calibration captured its neck reference from uncorrected Quest orientations. That reference is captured once and deliberately never recaptured. Entering POSE then resolved detection and installed the correction, and the next PLANNER entry met corrected data against a stale calibration. Measured on the recorded captures, installing the correction rotates global_orient_quat by ~121 deg with a pose-dependent axis, so the turn an operator sees depends on how they were standing when they calibrated. Move detection into the reader, which sees raw frames in every stream mode: AutoSkeletonSource now sits beside classify() in skeleton_source_detect.py, the reader resolves and installs the correction itself and publishes resolved_skeleton_source, and the manager only follows that for the wrist bias and the cross-check. The correction is settled within about five frames of body data arriving, long before an operator can press anything, so no consumer can latch a reference from uncorrected data. Detection accuracy is unchanged at 68/68 over the recorded captures, and explicit --skeleton-source values still bypass detection entirely.
The correction ships in isaacteleop 1.5+ while this repo pins 1.3.131, so any machine that has not run install_isaacteleop_skeleton_correction.sh cannot correct a Quest skeleton. Previously that surfaced as a bare ImportError raised inside the reader thread, which says nothing about the cause or the fix. Raise SkeletonCorrectionUnavailable instead, naming the script to run and the branch the correction lives on. Explicit --skeleton-source quest now refuses at startup rather than beginning a session it cannot correct: driving a robot from an uncorrected Quest skeleton gives it permuted joint axes, which is worse than not starting. Under auto the session is already running by the time the headset is known, so it stays unresolved and logs the same message, throttled. Detection resolves within about a second of body data, well before an operator can start the policy, so the message is seen before the robot can move. PICO is untouched and silent. Detection resolves to "pico", which needs no correction, so the import is never attempted and an operator who knows nothing about this work sees nothing new. Verified by removing the bridge and running both paths: PICO resolves normally with no error, Quest reports and refuses.
NVIDIA/IsaacTeleop#926 moved identify_headset out of isaacteleop.deviceio, on review feedback that deviceio is a C++ module plus its bindings. The cross-check sits inside a try/except ImportError, so the move would not have raised -- it would have silently stopped cross-checking, which is worse. Try the new location and fall back to the old one, so this works against isaacteleop builds from either side of that change.
With --skeleton-source auto on a machine without the bridge, a Quest was detected, the correction could not be loaded, and the previous handling logged the problem and kept going. Continuing means streaming an uncorrected Quest skeleton, which drives the robot with permuted joint axes -- observed on hardware, the robot fell over. Logging is not enough when the next step is the robot moving. The reader now records the failure, stops, and re-raises from get_latest() so it surfaces on the calling thread rather than dying in a background one. That matches what an explicit --skeleton-source quest already did at startup: refuse rather than begin a session it cannot correct. PICO is unchanged and silent: detection resolves to "pico", which needs no correction, so the import is never attempted.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Before conducting a code review, its best to learn how the big picture of headset detection and remapping works form this doc: https://docs.google.com/document/d/1dOX4aN6cnj9VV8iaMVsjhS7loJGrN_lSWixJVMpJelY/edit?usp=sharing
Supersedes and replaces PR #1, which should be closed. This targets
mainand carries the whole of the Quest work: capture tooling, the wrist bias, automatic headset detection, and taking the skeleton correction from isaacteleop.The per-joint skeleton correction itself is not here — it moved upstream to NVIDIA/IsaacTeleop#921, which this depends on. Until that merges and
.venv_teleopis upgraded to an isaacteleop containing it,--skeleton-source questand thequestbranch ofautorequire that build.Why
An operator who forgets
--skeleton-source queston a Quest drives the robot with permuted joint axes, and nothing says so. The failure looks like bad tracking rather than a missing flag. This makesautothe default so that mistake is no longer possible.Safe in both directions:
pico— no correction, zero wrist bias, identical to the old default.Explicit values still pin the behaviour.
How the headset is identified
From the body data itself. CloudXR normalises the Quest skeleton into the ByteDance layout, so I checked the obvious fields first and none separate the two headsets: joint order, positions, validity flags, sample rate, quantisation, bone-length variance. What does differ is the per-joint orientation convention — precisely what the correction fixes.
So the signature is each bone's direction (from positions, device-agnostic) expressed in its parent joint's frame (from the quaternions, device-specific).
Over 68 recorded captures — 34 paired poses on both headsets, 1020 random single frames — classification is 100% correct, worst margin 50° against a 20° refusal threshold. Correcting a Quest capture makes it classify as PICO, which is what shows the signature tracks the orientation convention rather than something incidental.
It does not gate startup
This matters more than the accuracy. An earlier auto-detect attempt waited for the headset to identify itself before letting the session proceed; controllers looked dead and the whole stack looked broken, which operators cannot distinguish from a real failure. That version was removed.
This one reads data that is already flowing, so controllers, video and OpenXR come up exactly as with an explicit source. Five consecutive agreeing frames are required before committing — measured at exactly 5 for every one of the 68 captures, about 80 ms at 60 Hz.
The streak requirement exists for a specific weakness: a frame of identity quaternions scores as PICO rather than refusing. Verified that four such frames commit nothing and real frames afterwards still resolve correctly.
Cross-check
When the runtime can report its interaction profile (NVIDIA/IsaacTeleop#926), it is compared against the geometry result: agreement is logged, disagreement warns and names both answers rather than silently picking one. Two independent signals with different failure modes.
That check is dormant today —
.venv_teleopruns isaacteleop 1.3.131, which has noget_interaction_profile— and activates automatically once that venv moves to a build containing #926. Auto works today from geometry alone.Testing
Validated against the robot simulator on both headsets. Also replayed frame by frame over all 68 captures: 68/68 resolve to the correct headset.
picoverified unchanged: no correction, zero wrist bias.Reviewer notes
Detection must run on raw data. It identifies the headset from the orientation convention — the very thing the correction changes. That is what makes it accurate, and it is also why a future refactor that applied a correction before detection would make the detector confidently report the wrong headset.
One caveat on generality. The reference signatures come from one operator on two headsets. The 20° refusal threshold against a 50° worst observed margin is the safety band; an unusual body should produce a refusal, not a wrong answer, but that has not been tested with a second operator.
The correction comes from isaacteleop, not from here. The in-repo copy is removed:
SkeletonCorrection, its loader and startup axis guard are gone, andquest_skeleton_correction.pydrops from 231 lines to 69. What remains there is the wrist bias, expressed in G1 wrist joint commands rather than skeleton space — robot-specific rather than headset-specific — so moving it upstream would mean re-deriving it as a skeleton-space rotation.An ordering constraint comes with that move. The correction is applied in the reader, but detection identifies the headset from the orientation convention the correction rewrites. So under
autothe reader starts uncorrected and the manager installs the correction viaset_skeleton_profileonce detection resolves;--skeleton-source questcorrects from the first frame. A refactor that corrected before detecting would make the detector confidently report the wrong headset.