Skip to content

fix(model): normalize Qwen3-Omni integer masks for TE - #5544

Open
hbhflw2000 wants to merge 3 commits into
NVIDIA-NeMo:mainfrom
hbhflw2000:pr/qwen3-omni-mask-mapping-followup
Open

fix(model): normalize Qwen3-Omni integer masks for TE#5544
hbhflw2000 wants to merge 3 commits into
NVIDIA-NeMo:mainfrom
hbhflw2000:pr/qwen3-omni-mask-mapping-followup

Conversation

@hbhflw2000

Copy link
Copy Markdown
Contributor

Summary

  • Normalize 2D integral Qwen3-Omni valid-token masks to the boolean
    masked-token format [B, 1, 1, S] expected by Transformer Engine.
  • Preserve the existing behavior for boolean, packed-sequence, and
    unsupported mask formats.
  • Add regression coverage for asymmetric QKV and gate/up tensor mappings,
    including expert-parallel export.

Problem

The processor and collator may emit 2D integer valid-token masks, while
the existing conversion only handled boolean masks. These integer masks
therefore bypassed normalization even though Transformer Engine expects
boolean masks with masked-token semantics.

The existing conversion tests also relied mainly on registry checks and
symmetric round trips, which could miss incorrect tensor ordering.

Scope

This follow-up is limited to:

  • Qwen3-Omni attention-mask normalization.
  • Regression tests for QKV, gate/up, and EP=2 expert mappings.

It does not change the production tensor-mapping implementation,
packed-sequence handling, conversion APIs, dependencies, or Megatron Core.

Validation

  • 15 focused unit tests passed.
  • Local 4-GPU pruned-model SFT completed one finite training step with
    zero skipped or NaN iterations.
  • A 4-node / 32-GPU full-model SFT run with TP2, PP2, and EP8 completed
    one finite training step:
    • LM loss: 4.563817
    • Load-balancing loss: 2.480726
    • Skipped iterations: 0
    • NaN iterations: 0
    • All distributed ranks exited cleanly.

Related

Follow-up to #4988, addressing the post-merge review comments on integral
attention masks and semantic tensor-mapping coverage.

Signed-off-by: hbhflw2000 <417911774@qq.com>
@copy-pr-bot

copy-pr-bot Bot commented Aug 13, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@yaoyu-33 yaoyu-33 added area:model Model implementations and HF bridge logic bug Something isn't working needs-more-tests Requires additional L0 and L1 test coverage before merge needs-review PR is ready for code review and waiting on a reviewer labels Aug 13, 2026
@yaoyu-33 yaoyu-33 added ready-to-merge PR is approved, current, and only waiting for CI to pass before merge and removed needs-review PR is ready for code review and waiting on a reviewer labels Aug 13, 2026
@yaoyu-33

Copy link
Copy Markdown
Contributor

/ok to test 9801179

@kamran-nvidia

Copy link
Copy Markdown
Contributor

/ok to test 0690c89

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:model Model implementations and HF bridge logic bug Something isn't working community-request needs-more-tests Requires additional L0 and L1 test coverage before merge ready-to-merge PR is approved, current, and only waiting for CI to pass before merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants