AI v2 A3: retarget gesture end-effector IK (wrist targeting) — the missing half of Retarget v2

[fernandotonon]

Part of #818 (Track A3). Status: PARTIAL — foot half shipped, gesture half missing.

Done (#856)

src/FootContact.{h,cpp} ships contact detection (detectContacts), analytic two-bone IK (solveKnee) and edge blending (blendWeight), wired through AnimationMerger::pinFeet with GUI/CLI/MCP surfaces.

Remaining — the gesture end-effector pass

AnimationMerger.cpp:5654 still carries the limitation this issue exists to remove:

"Good-but-imperfect: locomotion looks right; arm-precise gestures place the arm approximately. A true IK/look-at retarget ... is the follow-up for exact gesture placement."

This is the observed "wave goes to the back / arm stops mid-throw / wave hits the face" class of bug on retargeted clips.

Design: for upper_body actions, solve two-bone arm IK so the retargeted WRIST matches the canonical wrist position (scaled by limb-length ratio), instead of only chaining local deltas down the arm.

The machinery already exists and should be reused rather than rewritten:

• FootContact::solveKnee is generic two-bone IK — shoulder→elbow→wrist is geometrically identical to hip→knee→foot.
• TargetBindFrame::roleBoneIdx already resolves arm roles (7/8/9 right, 11/12/13 left) exactly as it resolves the leg roles pinFeet uses.
• pinFeet already does the manual FK over tracks, the canonical-frame mapping (tb.Ct), and the "fold a world-space correction into parent-relative keyframes" rewrite. The arm pass is the same shape with a different target rule (match a per-frame canonical position rather than pin to a span anchor).

Suggested surface: a sibling post-process (AnimationMerger::pinGestures or a shared src/RetargetIK.{h,cpp}), default ON behind the epic's QTMESH_T2M_IK=0 escape hatch, CLI --no-ik, mirroring pinFeet's GUI/CLI/MCP wiring.

Acceptance

• Wrist target error ≤ 5% of arm length on a wave clip retargeted to a rig with different arm proportions (fails today).
• Existing foot-pin behaviour and non-gesture clips unchanged (locomotion must not regress).
• Headless unit tests on a synthetic rig, in the FootContact_test.cpp style.

[fernandotonon]

Attempted and reverted — with the measurements, so the next attempt doesn't repeat it

I implemented AnimationMerger::pinGestures following pinFeet's exact shape (manual FK over the tracks, canonical-frame targets, FootContact::solveKnee two-bone IK, world-premultipliers folded into parent-relative keyframes, _keyFrameDataChanged()), wired behind QTMESH_T2M_GESTURE_IK=1, and reverted it. It made a retargeted wave visibly worse: the arm crossed in front of the face and the torso distorted.

Recording what was measured, because the mechanical half works and only the TARGET derivation is wrong.

What works

• Role resolution is correct: roleBoneIdx[7/8/9] → RightArm/RightForeArm/RightHand, [11/12/13] → LeftArm/LeftForeArm/LeftHand, 73 keyframes each on a generated wave.
• FootContact::solveKnee is genuinely reusable — shoulder→elbow→wrist is geometrically identical to hip→knee→foot.
• The FK + keyframe-rewrite path converges: mean wrist error to the supplied target went 0.2369 → 0.0000 over 146 keyframes across both arms.

Why it was wrong

Error-against-my-own-target is circular: it proves the solver converges, not that the target is anatomically right. Rendering showed it wasn't.

The target needs the canonical arm direction at frame f. I measured every candidate against the direction the WORKING retarget actually produces (dot product, canonical frame, generated wave on Goblin_rigged.glb, left arm = the one that moves):

frame	actual upper-arm dir	q*restDir	q*+Y	q*+X	q*+Z
0 (rest)	(0.05, −1.00, 0.04)	−0.990	0.986	−0.090	0.140
12	(0.22, 0.16, −0.96)	0.164	−0.081	0.982	−0.170
24	(0.14, 0.15, −0.98)	0.171	−0.087	0.992	−0.091
48	(0.21, 0.11, −0.97)	0.118	−0.035	0.987	−0.154
72 (rest)	(0.05, −1.00, 0.04)	−0.990	0.986	−0.090	0.140

The winning axis changes mid-clip — +Y at the rest frames, +X while the arm is raised. So the canonical quats are not absolute world orientations of a fixed-axis bone, and no constant axis (nor q * restDir) reconstructs the wrist position. That matches how the retarget itself works: tgtBindDir.getRotationTo(Ct⁻¹ · srcDir) maps a DIRECTION to a DIRECTION and never assumes a bone axis.

Two traps for the next attempt

1. Probe the arm that actually moves. On a wave, role 7 (right) is completely static — its shoulder keyframe is bit-identical across all 73 frames. I initially read that frozen arm as an FK bug in my own code; it wasn't. Role 11 carries the motion.
2. Do not validate with wrist-error-to-target. It is circular and reported a perfect 0.0000 for a visibly broken result. Render it.

Suggested direction

Derive the wrist target the way the retarget derives directions, rather than reconstructing positions from quats:

• feed pinGestures the per-frame canonical DIRECTIONS the retarget already computes internally (Ct⁻¹ · srcDir per role), instead of re-deriving them from clipQuats + an assumed axis; or
• express the target as the canonical shoulder→wrist OFFSET measured from the source clip's own FK (which requires canonical bone lengths the library does not currently ship — possibly an A2/AI v2 A2: motion library loop metadata (loopable, loop_start/end, in_place, tags) #1009 addition), scaled by the target's arm length.

The first is strictly cheaper and reuses a quantity that is already correct by construction.

#1010 (MotionComposer) shipped and is unaffected — see #1022.

[fernandotonon]

Follow-up: the "reuse the retarget's own directions" lead does NOT hold

My earlier comment suggested feeding pinGestures the per-frame canonical directions "the retarget already computes internally". I checked, and that quantity does not exist on the template path.

• liveCanon / collectCanonicalLiveDirections (AnimationMerger.cpp:2860) live in BodyRetargeter::evaluateFrame — the mocap path, ENABLE_MOCAP-guarded. Its directions come from MediaPipe landmark segments, which a library clip does not have.

• applyMotionClip's template path is the roll-corrected pure-local retarget (AnimationMerger.cpp:5648):

  local(f) = bind · (M_c⁻¹ · cmuLocalDelta[c][f] · M_c)
  

  It composes parent-relative rotation DELTAS conjugated by a constant per-joint frame map. It never derives a world position, and never assumes a bone axis — which is exactly why no fixed axis reproduced the wrist position in the measurements above, and why q * restDir failed too.

So option 1 from my earlier comment is off the table. What remains:

Option 2 (unchanged): give the library canonical bone LENGTHS so the source clip's own FK can produce a true canonical shoulder→wrist offset, which is then scaled by the target's arm length. That is a schema addition (an A2/#1009-style library change) plus a builder change, not a retarget-only fix.

Option 3 (new, cheapest): skip the position target entirely. Measure the wrist error the current retarget produces on a rig with known-different arm proportions, and if the dominant error is a single accumulated rotation offset rather than a reach error, correct THAT instead — no IK, no canonical positions. Worth measuring before building anything.

Either way this is design work, not another implementation attempt. Leaving the issue open and unassigned rather than half-building it again; the measurements in the previous comment are the starting point.