Video-to-Kinematics for robotics: capture human demonstrations with RGB-D cameras, retarget motions to robots, and generate LeRobot datasets.
ViKi turns multi-view RGB-D video of a human doing a manipulation task into a robot-ready demonstration dataset — no teleoperation rig required.
🚧 Work in progress. The capture → skeleton → retarget pipeline runs end to end, but tracking accuracy, multi-camera sync, and the hand-fit model are under active tuning. Interfaces and defaults are still moving.
Multi-view RGB-D capture ← RealSense D435i + Azure Kinect DK (HW-synced)
│
▼
3D hand skeleton ← MediaPipe + multi-view triangulation + depth
│
▼
Smoothing & fusion ← cross-camera fuse, Savitzky-Golay, articulated hand model
│
▼
Robot retargeting ← object-relative IK via PINK / Pinocchio
│
▼
LeRobot dataset ← for ACT / Diffusion Policy training [export: stub]
Human video is cheap and abundant, but naive retargeting from human to robot kinematics produces noisy, jerky trajectories. ViKi's optimisation stage exists to close that gap.
See SETUP_GUIDE.md for USB configuration, Docker setup, and multi-Kinect sync wiring.
sudo ./scripts/host_setup.sh # once per host
docker compose up --build
# open http://localhost:8000The web UI (viki/server/static/) is plain HTML/CSS/JS served directly by the FastAPI server.
docker compose run --rm test # full test suite
docker compose run --rm test tests/unit_tests/gripper # a subset (args append to pytest)
docker compose run --rm cli <verb> ... # viki record/extract/prepare/retarget/replay/label/export/run
docker compose run --rm terminal # bash shell in the container| stage | package | status |
|---|---|---|
| record | viki/cameras |
done |
| calibrate | viki/calibration |
done |
| extract (skeleton) | viki/perception |
in progress — accuracy tuning |
| prepare (fuse + smooth) | viki/prepare |
in progress |
| retarget (IK) | viki/retarget |
done |
| replay (hardware validation) | viki/replay |
stub |
| export (LeRobot dataset) | viki/export |
stub |
See viki/README.md for the full package map and cross-cutting rules.
ViKi's source code is licensed under Apache-2.0. Third-party dependencies, hardware SDKs, model weights, and robot assets retain their own terms. See the open license audit and remediation checklist before distributing the application or its Docker image.