feat(qwen): verify Qwen 3.8 27B and tune GB200 pretraining

[cuichenx]

What does this PR do?

Adds the GB200 model verification card for Qwen/Qwen3.8-27B at immutable Hugging Face revision 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0, fixes multimodal HF/Megatron model comparison for processors that emit mm_token_type_ids, and adds a tuned canonical 16-GB200 Qwen 3.5/3.8 27B projector-pretraining recipe.

The comparison helper previously retained only the legacy token_type_ids field. Qwen 3.8 supplies mm_token_type_ids, which Transformers requires alongside image_grid_thw to compute multimodal RoPE positions, so the HF forward failed before logits were produced. The fix preserves and forwards both fields under their exact processor keys, pads each consistently with TP-padded input IDs, and carries mm_token_type_ids through the Megatron single-batch path. It does not change token selection, the required next-token match, the cosine-similarity threshold, or absolute-difference reporting.

The original pretrain sanity run used the H100 canonical topology in deterministic mode. For this 48-Gated-DeltaNet-layer model, deterministic mode selects the torch-native GDN reference path instead of the fused implementation. TP4/PP4 also consumed the entire 16-GPU world as pipeline parallelism, while full activation recompute repeated expensive GDN forward work.

The canonical qwen35_vl_27b_pretrain_16gpu_gb200_bf16_mock_config now keeps the same projector-only objective, BF16 precision, sequence length, and global batch while using TP2/PP1/CP1, DP8, MBS4, fused GDN execution, and no activation recompute. This executes one microbatch per DP rank without pipeline bubbles. TP1/PP1 with MBS2 was rejected because cold GDN autotuning exhausted the approximately 185-GiB device memory; TP2 completed with about 11 GiB of allocated-memory headroom. Evaluation and checkpoint output remain disabled for the bounded performance run. No file under src/megatron/bridge/perf_recipes/ is added or modified.

Changelog

• Forward current mm_token_type_ids and legacy token_type_ids through the comparison helper without reinterpreting either field.
• Add focused preprocessing, HF-forward, and Megatron-forward regression coverage.
• Add the Qwen 3.8 27B verification card with immutable model and Bridge revisions.
• Record conversion, strict HF reload, deterministic inference, training, resume, SFT/PEFT, post-SFT export, and 8K packed-context evidence collected on GB200 hardware.
• Verify manual forward parity on four GB200 GPUs: matching token 248045, cosine similarity 0.995276, maximum absolute logit difference 1.125000, and mean absolute difference 0.192757.
• Export and test the canonical 16-GB200 Qwen 3.5/3.8 27B projector-pretraining recipe.
• Record a separate pretrain_performance.GB200 result so the original deterministic functional pretrain/checkpoint-resume pair remains intact.

Performance result

On 16 GB200 GPUs in one NVL72 domain, the exact committed canonical recipe completed 20 steps with finite losses and gradients, zero skipped iterations, and zero NaN iterations. Steps 11–20 averaged 1,139.500 ms and 1,230.090 model TFLOP/s/GPU; peak allocated memory was 173.760 GiB. The first-iteration kernel-compilation outlier is excluded from the sustained average. This is 2.72x the prior 451.720 TFLOP/s/GPU canonical result. This mock projector-only result is throughput evidence, not full-model convergence evidence.

Validation

• uv run python -m pytest tests/unit_tests/test_compare_mask_handling.py -q — 20 passed on GB200.
• Final Qwen recipe suite — 124 passed on GB200 from the final branch head.
• Model-verification-card validator — passed on GB200 and locally.
• Canonical 20-step pretrain from clean recipe commit 7f9864c4712f401213ecd3c734a2bf627c306438 — passed on 16 GB200 GPUs; last-10 average 1,230.090 model TFLOP/s/GPU.
• Repository pre-commit hooks — passed.
• Public scripts/inference/infer.sh --task model-comparison workflow — passed from clean Bridge commit ab905ac72ccc9e45a081153655eda9ee469fdcff on four GB200 GPUs.

Before your PR is "Ready for review"

Pre checks:

• Make sure you read and followed Contributor guidelines.
• Did you write any new necessary tests?
• Did you add or update any necessary documentation?
• Does the PR affect components that are optional to install? No.
  • Reviewer: Does the PR have correct import guards for all optional libraries? Not applicable.

Additional Information

The comparison fix does not loosen the verification goal, and the optimized performance evidence is deliberately separate from deterministic convergence/resume evidence.

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.