Skip to content

fix(model): use dynamic temporal video token counts - #5590

Draft
cuichenx wants to merge 1 commit into
NVIDIA-NeMo:mainfrom
cuichenx:chcui/sage/fix-omni-dynamic-video-counts
Draft

fix(model): use dynamic temporal video token counts#5590
cuichenx wants to merge 1 commit into
NVIDIA-NeMo:mainfrom
cuichenx:chcui/sage/fix-omni-dynamic-video-counts

Conversation

@cuichenx

Copy link
Copy Markdown
Contributor

What does this PR do ?

Adds an explicit, processor-driven temporal-video resize mode for Nemotron Omni Energon training so each tubelet receives the number of image placeholders its actual post-resize vision grid produces.

Changelog

  • Preserve fixed_512 as the compatibility default and enable processor mode only in the shipped Nemotron Omni VALOR Energon recipe.
  • Reuse the public Nemotron Omni processor's aspect-preserving video pixels, imgs_sizes, and reported token counts instead of reimplementing its resize policy.
  • Derive prompt-expansion counts independently for every tubelet while keeping logical num_image_tiles metadata separate.
  • Validate frame/grid metadata and reject processor-driven sizing with the deprecated LLaVA collapse/expand contract.
  • Cover 252- and 242-token rectangular grids, ragged packing, incomplete tubelets, text-only temporal batches, config propagation, and a real rectangular RADIO forward.
  • Document the new mode and its compatibility boundary. Direct-HF behavior remains unchanged.

GitHub Actions CI

Focused validation on one H100:

  • uv run --no-sync pre-commit run --all-files
  • A real VALOR MP4 through the locked public processor: ten 384x672 frames, five 252-token tubelets, and 1,260 placeholders matching 1,260 expected vision rows.
  • Focused Nemotron Omni builder, collator, Energon, recipe, utility, and rectangular RADIO GPU tests: 159 passed.
  • git diff --check

Before your PR is "Ready for review"

Pre checks:

  • Make sure you read and followed Contributor guidelines
  • Did you write any new necessary tests?
  • Did you add or update any necessary documentation?
  • Does the PR affect components that are optional to install? No.

Additional Information

This PR intentionally scopes dynamic temporal-video sizing to the canonical Energon path. The fixed mode remains available for compatibility, and the deprecated LLaVA contract remains fixed-resolution.

Signed-off-by: Chen Cui <chcui@nvidia.com>
@copy-pr-bot

copy-pr-bot Bot commented Aug 14, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant