This document describes how to train and evaluate StarVLA models on the Calvin benchmark, including dataset preparation, training configuration, and evaluation procedures.
Note: Calvin benchmark experiments were conducted by the UNT team. For inquiries, please contact Zhijie Song (1600013008@pku.edu.cn) or Feng Yan (bphengyan@163.com).
This directory hosts two independent training pipelines that share only the eval scripts in eval_files/:
| Pipeline | Format | Training entry | Training config |
|---|---|---|---|
| QwenPI (existing) | LeRobot v3.0 | run_calvin_train.sh |
starvla_train_calvin.yaml |
| UamVLA (this section) | UAM unified format (data.jsonl + statistics.yaml) |
run_uamvla_calvin_train.sh |
starVLA/config/training/uamvla_calvin.yaml |
The two pipelines do not share preprocessing — pick one and stay in lane.
-
Preprocess a CALVIN split into UAM format (requires the
calvin_envconda environment):Single-scene splits (
task_D_D,calvin_debug_dataset):python runners/preprocess_calvin.py \ --input_dir /path/to/calvin/task_D_D/training \ --output_dir datasets/uamvla_calvin/task_D_D \ --dataset_source task_D_DMulti-scene splits (
task_ABC_D,task_ABCD_D): the single-shot preprocessor would render every frame against ONE scene's URDF, placing scene-A frames' blocks inside scene-D fixtures (visible as bodies clipping through the switch / drawer inrgb_static). Use the multi-scene driver instead — it splits by scene, runs the preprocessor once per scene with its own URDF, then merges:python runners/preprocess_calvin_multiscene.py \ --input_dir /path/to/calvin/task_ABCD_D/training \ --work_dir /tmp/abcd_split_work \ --output_dir datasets/uamvla_calvin/task_ABCD_D/training \ --dataset_source task_ABCD_D \ --num_workers 8 \ --on_missing_target skipIf
calvin_env's scene configs aren't auto-discoverable from the installed package, pass them explicitly:--scene_config_dir /path/to/calvin_env/conf/scene.The preprocessor regenerates
statistics.yamlfrom the mergeddata.jsonlon every run, so re-running refreshes stats ifCalvinAdapterever changes. -
Train:
bash examples/calvin/train_files/run_uamvla_calvin_train.sh
Verify the paths inside the script first (
base_vlm,calvin_data_root,run_root_dir). -
Evaluate — uses the same eval scripts as QwenPI (
run_policy_server.sh+eval_calvin.sh); see §3 below.
Known limitation (CALVIN eval state distribution gap). UamVLA training feeds
canonical_stateto the model every step, but the current CALVIN eval client (eval_calvin.py) sends onlyimage+langto the policy server. A trained UamVLA-CALVIN checkpoint loads cleanly into the eval pipeline (after theunnorm_keyrename ineval_calvin.sh) but inference uses the state-less branch — a real distribution mismatch from training. Expect avoidable success-rate loss until a CALVIN parallel ofdocs/superpowers/specs/2026-04-30-uamvla-eval-state-passthrough-design.mdlands. Tracked as a follow-up spec.
| Model | Avg. Length | Task 1 | Task 2 | Task 3 | Task 4 | Task 5 |
|---|---|---|---|---|---|---|
| PI0* | 2.954 | 84.8% | 70.4% | 55.9% | 46.6% | 37.7% |
| PI0.5* | 3.885 | 92.5% | 84.0% | 76.6% | 71.0% | 64.4% |
| qwenpi (qwen2.5-vl-3B-instruct-action) | 3.576 | 90.9% | 79.5% | 69.6% | 62.2% | 55.4% |
| qwenpi (qwen3-vl-4B-instruct) | 3.472 | 87.7% | 75.2% | 67.4% | 61.8% | 55.1% |
| qwengr00t (qwen2.5-vl-3B-instruct) | 3.697 | 91.7% | 81.9% | 72.7% | 65.3% | 58.1% |
| qwengr00t (qwen2.5-vl-3B-instruct-action) | 3.786 | 92.5% | 83.9% | 74.4% | 67.9% | 59.9% |
| qwengr00t (qwen3-vl-4B-instruct) | 3.645 | 89.8% | 79.9% | 71.8% | 64.9% | 58.1% |
| qwengr00t (qwen3-vl-4B-instruct-action) | 3.757 | 91.1% | 81.8% | 74.1% | 67.6% | 61.1% |
- Models marked with
*are trained by us. Other experimental results will be released soon.
For training, the Calvin dataset must be converted into LeRobot format.
-
Convert the original Calvin dataset to LeRobot format. Please refer to RoboTron-Mani (ICCV 2025) for detailed instructions: 👉 https://github.com/EmbodiedAI-RoboTron/RoboTron-Mani/tree/lerobot/examples/calvin
-
Copy the modality definition file:
cp examples/calvin/train_files/modality.json <lerobot_dataset_path>/meta/modality.json
For evaluation, use the original Calvin dataset format.
- The dataset path should directly contain the
validation/directory.
Training and evaluation use different dataset formats:
- Training: LeRobot format (converted from original Calvin)
- Evaluation: Original Calvin format (with
validation/folder)
Configure the Calvin data mix in:
starVLA/dataloader/gr00t_lerobot/mixtures.py
Example configuration:
"calvin_task_D_D": [
("task_D_D", 1.0, "libero_franka"),
],Make sure the key name (e.g. calvin_task_D_D) matches the one used during training.
examples/calvin/train_files/run_calvin_train.sh:
calvin_data_root: Path to the LeRobot-format Calvin datasetdata_mix: Must match the key defined inmixtures.py
Start training with:
bash examples/calvin/train_files/run_calvin_train.shEvaluation requires two terminals, both launched from the repository root, but using different conda environments.
In the first terminal, activate the starVLA conda environment and start the inference server:
bash examples/calvin/eval_files/run_policy_server.shrun_policy_server.sh is correct.
In the second terminal, activate the Calvin conda environment and run:
bash examples/calvin/eval_files/eval_calvin.sheval_calvin.sh:
dataset_path: Path to the original Calvin dataset (containingvalidation/)your_ckpt: Checkpoint path (must match the one used by the policy server)
eval_calvin.py:
dataset_path: Path to Calvin dataset (default:"/path/to/calvin/task_D_D")calvin_config_path: Path to Calvin models configuration directory (default:"/path/to/calvin/calvin_models/conf")eval_sequences_path: Path to evaluation sequences JSON file (default:"/path/to/calvin/eval_sequences.json")
For convenience, we provide a reference evaluation sequence file at examples/calvin/eval_files/eval_sequences.json, which can be used directly.