Skip to content

Lyra-2

Directory actions

More options

Directory actions

More options

Latest commit

 

History

History

Lyra-2

README.md

Lyra 2.0: Explorable Generative 3D Worlds

Lyra-2.0.mp4

Lyra 2.0: Explorable Generative 3D Worlds
Tianchang Shen*, Sherwin Bahmani, Kai He, Sangeetha Grama Srinivasan, Tianshi Cao, Jiawei Ren, Ruilong Li, Zian Wang, Nicholas Sharp, Zan Gojcic, Sanja Fidler, Jiahui Huang, Huan Ling, Jun Gao, Xuanchi Ren*
* indicates equal contribution
NVIDIA
Paper, Project Page, HuggingFace

TL;DR: Lyra 2.0 turns an image into a 3D world you can walk through, look back, and drop a robot into for real-time rendering, simulation, and immersive applications.

Abstract: Recent advances in video generation enable a new paradigm for 3D scene creation: generating camera-controlled videos that simulate scene walkthroughs, then lifting them to 3D via feed-forward reconstruction techniques. This generative reconstruction approach combines the visual fidelity and creative capacity of video models with 3D outputs ready for real-time rendering and simulation. Scaling to large, complex environments requires 3D-consistent video generation over long camera trajectories with large viewpoint changes and location revisits, a setting where current video models degrade quickly. Existing methods for long-horizon generation are fundamentally limited by two forms of degradation: spatial forgetting and temporal drifting. As exploration proceeds, previously observed regions fall outside the model's temporal context, forcing the model to hallucinate structures when revisited. Meanwhile, autoregressive generation accumulates small synthesis errors over time, gradually distorting scene appearance and geometry. We present Lyra 2.0, a framework for generating persistent, explorable 3D worlds at scale. To address spatial forgetting, we maintain per-frame 3D geometry and use it solely for information routing—retrieving relevant past frames and establishing dense correspondences with the target viewpoints—while relying on the generative prior for appearance synthesis. To address temporal drifting, we train with self-augmented histories that expose the model to its own degraded outputs, teaching it to correct drift rather than propagate it. Together, these enable substantially longer and 3D-consistent video trajectories, which we leverage to fine-tune feed-forward reconstruction models that reliably recover high-quality 3D scenes.

News

Installation

See INSTALL.md for full step-by-step instructions.

Inference

Download Checkpoints

Download the pretrained checkpoints from HuggingFace:

# Install huggingface_hub if needed
pip install huggingface_hub

# Download all checkpoints into the checkpoints/ directory
huggingface-cli download nvidia/Lyra-2.0 --include "checkpoints/*" --local-dir .

Step 1 — Generate Video

Lyra 2.0 takes a starting image, a user-defined camera trajectory, and per-chunk text captions as input to generate an exploration video. For ease of testing, we provide several options for video generation.

Option 1: Preset Trajectory (Zoom-in / Zoom-out)

Generate a zoom-in then zoom-out exploration video from a single input image. Because outpainting is gradual in this mode, a single text prompt describing the starting image is sufficient—each image needs a paired .txt file with its caption in the same directory (see assets/samples/ for examples). Change --sample_id (0–14) to run different examples.

export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True

PYTHONPATH=. python -m lyra_2._src.inference.lyra2_zoomgs_inference \
  --input_image_path assets/samples \
  --sample_id 4 \
  --experiment lyra2 \
  --checkpoint_dir checkpoints/model \
  --prompt_dir assets/samples \
  --output_path outputs/zoomgs \
  --num_frames_zoom_in 81 \
  --num_frames_zoom_out 241 \
  --zoom_in_strength 0.5 \
  --zoom_out_strength 1.5

Outputs written to outputs/zoomgs/<sample_id>/:

  • zoom_in.mp4 — zoom-in clip
  • zoom_out.mp4 — zoom-out clip
  • videos/<sample_id>.mp4 — combined video (zoom-in → zoom-out)

Tips:

  • The scale of the zoom-in and zoom-out trajectories can be adjusted via --zoom_in_strength and --zoom_out_strength. Note that a built-in collision detection mechanism prevents the camera from moving into objects, so increasing --zoom_in_strength may not always change the trajectory.
  • If the camera movement appears too fast, increase --num_frames_zoom_in or --num_frames_zoom_out to spread the motion over more frames. Frame counts must be of the form 1 + 80k (i.e., 81, 161, 241, …) to align with autoregressive chunk boundaries.

Runtime on 1× H100 80GB: ~9 min per 80 frames (~35s with --use_dmd).

Example outputs (Step 1 video generation | Step 2 GS reconstruction):

video_8x.mp4
gs_render_8x.mp4

Option 2: Custom Trajectory

For evaluating Lyra 2.0 on a custom, predefined camera trajectory, prepare your inputs following the examples under assets/custom_trajectory_examples/. Each example folder contains:

  • first_frame.png — the starting image
  • trajectory.npz — camera poses (w2c: N×4×4 world-to-camera matrices, intrinsics: N×3×3, image_height, image_width)
  • captions.json — per-chunk text captions keyed by frame index (e.g. {"0": "...", "81": "...", ...})

The script loads the first --num_frames poses from the trajectory file and runs autoregressive generation. When --captions_path is provided with a multi-entry JSON, each chunk uses the caption whose frame index is closest to (but not exceeding) the current chunk start.

export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True

PYTHONPATH=. python -m lyra_2._src.inference.lyra2_custom_traj_inference \
  --input_image_path assets/custom_trajectory_examples/example_0/first_frame.png \
  --trajectory_path assets/custom_trajectory_examples/example_0/trajectory.npz \
  --experiment lyra2 \
  --checkpoint_dir checkpoints/model \
  --captions_path assets/custom_trajectory_examples/example_0/captions.json \
  --num_frames 481 \
  --output_path outputs/custom_traj

Output: outputs/custom_traj/<image_name>.mp4

Note: The model may hallucinate objects along the predefined trajectory, which can lead to degenerated results. Additionally, the input trajectory is always relative to the monocular depth estimated from the first frame, so the actual camera path may differ from what you expect. You can adjust --pose_scale to scale your trajectory.

Example outputs (Step 1 video generation | Step 2 GS reconstruction):

first_frame.mp4
gs_trajectory.mp4

Option 3: Interactive GUI

Our interactive GUI supports camera-path authoring in a 3D viewer, autoregressive video extension, and text prompts for specifying the content to generate, as demonstrated in the demos on our website. See gui/README.md for instructions.

gui.mp4

Fast Inference with DMD Distillation

Pass --use_dmd to either video generation command (lyra2_zoomgs_inference or lyra2_custom_traj_inference) to enable the DMD distillation LoRA. This reduces sampling to 4 steps and distills away CFG, giving roughly a ~15× speedup.

Example outputs generated with --use_dmd on custom_trajectory_examples/example_1 (Step 1 video generation | Step 2 GS reconstruction):

first_frame_dmd.mp4
gs_trajectory_dmd.mp4

Note: DMD may weaken prompt following and sometimes produces scenes with repetitive patterns. For best quality and scale, we recommend running without --use_dmd.

Step 2 — 3D Gaussian Splatting Reconstruction

Lift the generated video to a 3D Gaussian Splatting scene using VIPE pose estimation and DA3 depth.

PYTHONPATH=. python -m lyra_2._src.inference.vipe_da3_gs_recon \
  --input_video_path outputs/zoomgs/videos/<sample_id>.mp4

For long videos (for example, more than 500 frames), reconstructing all frames in a single DA3 pass may produce blurry results because the DA3 Gaussian head was not fine-tuned at such sequence lengths. For best quality, reconstruct the video as multiple overlapping chunks. Set --num_chunks to 2 or greater:

PYTHONPATH=. python -m lyra_2._src.inference.vipe_da3_chunked_gs_recon \
  --input_video_path <video_path> \
  --num_chunks 2 \
  --chunk_size 128 \
  --chunk_overlap 10

The single-pass command writes <sample_id>_gs_ours/, while the chunked command writes <sample_id>_gs_chunked/. Both output directories contain:

  • reconstructed_scene.ply — Gaussian point cloud
  • gs_trajectory.mp4 — rendered flythrough video

Runtime on 1× H100 80GB: ~1 min (VIPE + DA3 depth + GS render).

Training

See TRAINING.md for pretrained weights, dataset format, and a runnable example using the included sample data.

Acknowledgement

We thank the following open-source projects that this work builds upon:

Some example input images included in assets/ are sourced from the Tanks and Temples benchmark and Marble by World Labs. We thank the respective authors for making these resources available.

Citation

@article{shen2026lyra2,
    title={Lyra 2.0: Explorable Generative 3D Worlds},
    author={Shen, Tianchang and Bahmani, Sherwin and He, Kai and Srinivasan, Sangeetha Grama and Cao, Tianshi and Ren, Jiawei and Li, Ruilong and Wang, Zian and Sharp, Nicholas and Gojcic, Zan and Fidler, Sanja and Huang, Jiahui and Ling, Huan and Gao, Jun and Ren, Xuanchi},
    journal={arXiv preprint arXiv:2604.13036},
    year={2026}
}

License and Contact

This project will download and install additional third-party open source software projects. Review the license terms of these open source projects before use.

Lyra 2.0 source code is released under the Apache 2 License.

Lyra 2.0 models are released under the NVIDIA Internal Scientific Research and Development Model License. For a custom license, please visit our website and submit the form: NVIDIA Research Licensing.

Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.