Add VLM support to the dynamic-batching inference server - #6260

Open

RPrenger wants to merge 36 commits into

NVIDIA:mainfrom

RPrenger:vlm-inference

Commits on Aug 15, 2026

Add VLM support to the dynamic-batching inference server
RPrenger
committed
a392b42
NeMo-RL working branch.

cspades
authored and
RPrenger
committed
2402b1d
Match vLLM multimodal API.

cspades
authored and
RPrenger
committed
23fad30
Contain SUBMIT_REQUEST errors so one bad payload doesn't kill the engine
RPrenger
committed
5080610
Clean up VLM request data if _add_request rejects the request
RPrenger
committed
e43f374
Reject incomplete static-tiling image payloads at the wire boundary
RPrenger
committed
c373111
Preserve VLM subtype and image data across suspend/resume
RPrenger
committed
c7833b4
Fix placeholder token bug with RL + Gym.

cspades
authored and
RPrenger
committed
0299f03
Address PR comments for cspades' changes.

cspades
authored and
RPrenger
committed
5ee057b
Apply black/isort autoformatter to the current changeset
RPrenger
committed
25dccea
Guard the truncation branch for final_position_ids being None
RPrenger
committed
bdf13de
Stop threading a non-image-aware arange into the language model as position_ids
RPrenger
committed
592fd9c
Restore h/w kwargs on pixel_shuffle to keep existing callers working
RPrenger
committed
d513661
Fail loudly for audio / video kwargs on LLaVAModel.forward instead of silently ignoring
RPrenger
committed
3a39431
Raise instead of assert on image-embedding count mismatch
RPrenger
committed
4418ab6
Fix RADIO class_token_len override chain and self._class_token_len
RPrenger
committed
bd99491
Fold two D2H syncs in packed_seq_params rebuild into one
RPrenger
committed
591f560
Default inference_wrapper_cls to GPTInferenceWrapper on MegatronLLM / MegatronAsyncLLM
RPrenger
committed
151c8ce
Make _generate_impl.multi_modal_data_list optional (backward-compat)
RPrenger
committed
76a89ef
Match the decode-branch rank in current_image_token_mask to the prefill branch
RPrenger
committed
2aba70e
Short-circuit current_image_embeddings on decode steps
RPrenger
committed
fdff099
Skip VLM context helpers entirely on text-only workloads
RPrenger
committed
3ed1164
Collapse per-image .item() syncs to a single tolist() in expand_image_tokens
RPrenger
committed
78adb52
Skip num_tiles.sum() D2H sync on dynamic-resolution admissions
RPrenger
committed
7909b4f
Reject HTTP redirects in the image_url fetch path
RPrenger
committed
643a615
Note the n>1 image re-preprocessing cost as a known limitation
RPrenger
committed
084c7be
Drop the wire-side static-tiling helper to remove megatron/core → examples/ import
RPrenger
committed
67c7239
Stop mutating sys.path inside megatron/core
RPrenger
committed
5e61e13
Assert MoE hybrid config has num_moe_experts set in get_hybrid_layer_spec_te
RPrenger
committed
d96819c
Give the CLS token a position embedding in the learned_absolute ViT path
RPrenger
committed
ebc35b7
Move KimiLearned2DPosEmbed weight to device on both forward paths
RPrenger
committed
a972aa2
Pin apply_chat_template return_dict=False to keep the tensor/list return shape
RPrenger
committed
b20d25e
Declare Pillow as a megatron-core dev dep for VLM image preprocessing
RPrenger
committed
4eb95a4
Guard the torchvision import in image_preprocessing with an install-me error
RPrenger
committed
4915df7
Re-run isort/black on files touched in this round of review fixes
RPrenger
committed
1af7efc
Refresh uv.lock so it matches the Pillow entry added to pyproject.toml
RPrenger
committed
a9464c7