WM4A repurposes pretrained video-generation world models as visual encoders for robot action prediction. Instead of using a Vision-Language Model (VLM) backbone, WM4A feeds images through a Diffusion Transformer (DiT) — the same architecture used for video prediction — and attaches a lightweight action head on top of its intermediate representations.
Video prediction models learn rich spatiotemporal representations of physics, object permanence, and dynamics. WM4A exploits these representations directly:
VLM approach: Image → VLM encoder → language-aligned features → Action Head
WM4A approach: Image → Video DiT encoder → physics-aligned features → Action Head
| Backbone | Params | Layers | Hidden Dim | Source |
|---|---|---|---|---|
| Cosmos-Predict2-2B | 2B | 28 | 2048 | NVIDIA Cosmos |
| Wan2.2-T2V | 5B | 30 | 3072 | Alibaba Wan |
Input: {images: [PIL], instruction: str, actions: [T, action_dim]}
│
├─ Text Encoder (T5 / UMT5)
│ instruction → [B, L_text, text_hidden]
│
├─ VAE Encoder
│ images → latents [B, 16, T_latent, H/8, W/8]
│
└─ DiT Transformer (frozen or fine-tuned)
latents + text_embeds → hidden_states [B, N_tokens, hidden_dim]
│
└─ Action Head (one of three variants below)
→ predicted actions [B, chunk_len, action_dim]
Each backbone can be paired with three action head types:
| Variant | Action Head | What it uses | Speed | Frameworks |
|---|---|---|---|---|
| OFT | MLP regression | Last layer only | Fastest | CosmoPredict2OFT, WanOFT |
| GR00T | Flow-matching diffusion (single-layer) | Last layer only | Medium | CosmoPredict2GR00T, WanGR00T |
| PI | Layer-wise cross-attention DiT | All transformer layers | Slowest | CosmoPredict2PI, WanPI |
This gives 7 framework combinations (including a generic WM4A_OFT).
Run the built-in demo to see the full forward pass (training + inference):
python starVLA/model/framework/WM4A/CosmoPredict2GR00T.pyThis script:
- Loads a config and instantiates the
CosmoPredict2_GR00Tmodel - Creates a synthetic batch with multi-view images and random actions
- Runs the training forward and prints the action loss
- Runs the inference forward and prints the predicted actions
Note: You need the Cosmos-Predict2-2B weights downloaded locally. Set the path in the script or download via:
huggingface-cli download nvidia/Cosmos-Predict2-2B-Video2World \ --local-dir ./playground/Pretrained_models/nvidia/Cosmos-Predict2-2B-Video2World
The simplest way to train a WM4A model is with the OFT (MLP) action head on the LIBERO benchmark:
# From the starVLA project root
bash examples/LIBERO/train_files/run_libero_train.shBefore running, edit the script to set your local paths:
Framework_name=CosmoPredict2OFT # or CosmoPredict2GR00T, WanOFT, etc.
base_wm=nvidia/Cosmos-Predict2-2B-Video2World> # still needed for tokenizer/processor
config_yaml=./examples/LIBERO/train_files/starvla_cotrain_libero.yaml
libero_data_root=<path_to_LIBERO_dataset>
data_mix=libero_all # or libero_goalKey training arguments:
| Argument | Description | Recommended |
|---|---|---|
--framework.name |
Framework class name | CosmoPredict2OFT |
--framework.action_model.future_action_window_size |
Action prediction horizon | 7 |
--datasets.vla_data.per_device_batch_size |
Per-GPU batch size | 16 (OFT) / 8 (GR00T) |
--trainer.max_train_steps |
Total training steps | 80000 |
--trainer.save_interval |
Checkpoint save frequency | 10000 |
To use Wan instead of Cosmos, simply change the framework name:
Framework_name=WanOFT # or WanGR00T, WanPIThe config YAML remains the same — the framework class handles backbone loading automatically based on its registered name.
starVLA/model/
├── framework/
│ └── WM4A/
│ ├── CosmoPredict2GR00T.py # Cosmos + flow-matching head
│ ├── CosmoPredict2OFT.py # Cosmos + MLP head
│ ├── CosmoPredict2PI.py # Cosmos + layer-wise cross-DiT head
│ ├── WanGR00T.py # Wan + flow-matching head
│ ├── WanOFT.py # Wan + MLP head
│ ├── WanPI.py # Wan + layer-wise cross-DiT head
│ └── WM4A_OFT.py # Generic WM backend + MLP head
└── modules/
├── world_model/
│ └── CosmoPredict2.py # Cosmos backbone wrapper (VAE + T5 + DiT)
└── action_model/
├── MLP_ActionHeader.py # OFT action head
├── flow_matching_head/ # GR00T action head (single-layer FM)
└── LayerwiseFM_ActionHeader.py # PI action head (layer-wise FM)
- Precision: DiT forward runs in
bfloat16for speed; action head runs infloat32for numerical stability. - Feature extraction: Uses PyTorch hooks on selected DiT blocks to capture intermediate hidden states without modifying the backbone code.
- Timestep trick: Sets
timestep=0during feature extraction — this extracts representations at the near-clean noise level rather than denoising. - Shape reshaping: DiT outputs
[B, C, T, H, W]video tensors, which are reshaped to[B, T*H*W, C]token sequences for the action head. - Framework registry: All WM4A classes register themselves via
@FRAMEWORK_REGISTRY.register("FrameworkName")and are auto-discovered at import time — no manual imports needed.