feat(recipes): modernize DeepSeek V4 library configs

[cuichenx]

What does this PR do?

• adds real-training 64-GPU B200 Flash, 128-GPU GB200 Flash, and 256-GPU GB300 Pro library recipes; B200 uses portable all-to-all while GB200 and GB300 use hardware-local HybridEP topologies
• records the GB200 PP1 memory configuration: full-MoE selective recompute, attention activation offload, and MXFP8 parameter gather/buffer reuse
• adds a GB200-optimized offline-packed OpenMath SFT recipe with HybridEP, DSA fusion, uneven-dispatch padding, and export-compatible expert GEMMs
• moves supported precision-independent fusions, communication overlap, recompute, and activation-offload settings into the library recipe hierarchy
• uses native fused cross entropy after a matched TE/native screen and MCore's TE stability warning
• applies recipe environment variables before training imports and documents the required Megatron-LM development stack
• adds the DeepSeek-V4-Flash verification card from dsv4 flash verification card and export fixes #5360, updated for the latest library recipe and current evidence
• adds focused library recipe-contract, export, metadata, runner-bootstrap, SFT, and verification-card tests

The library variants retain natural routing, convergence batch sizes, validation and correctness checks, optimizer precision, and checkpoint behavior. They do not enable expert-rank capacity, paged stash, forced load balancing, full-iteration CUDA graphs, or reduced-precision gradient reduction. FP8 parameter gather/buffer reuse is enabled for the provisional B200 PP8 and GB200 PP1 MXFP8 recipes.

The performance recipes are unchanged and are used only as reference configurations. All training validation uses the library recipe through scripts/training/run_recipe.py.

Depends on NVIDIA/Megatron-LM#6509 for stateless grouped FP8 checkpoint saves.

Verification card

The card preserves the GPU conversion, packed SFT, SFT export/HF inference, and canonical performance evidence from #5360 at its item-pinned Bridge revision. It uses the current PP1/MXFP8 library recipe for the pretraining and resume commands, but keeps both items unverified because the exact PP1 candidate has only an eight-step screen. The imported 1K packed SFT run is not presented as long-context evidence, so sft_long_context.GB200 is also unverified.

No changes from #5360 were taken for verify_hf_inference.py, model conversion, DeepSeek V4 bridge implementation, or performance recipes.

Validation

• focused and adjacent recipe/runner tests: 256 passed, 1 deselected
• B200 and adjacent recipe-factory tests: 359 passed
• grouped checkpoint tests: 15 passed
• final DeepSeek recipe tests: 11 passed
• verification-card validator tests: 33 passed
• DeepSeek-V4-Flash card validator: passed
• uv run --no-project pre-commit run --all-files
• git diff --check
• matched native/TE cross-entropy screen: native averaged 139.700 versus 137.533 TFLOP/s/GPU over steps 2-4 and used the same 151.44 GiB peak allocated memory
• previous 128-GPU GB200 PP2 baseline: 100 finite real-data steps with zero skipped/NaN iterations; steps 81-100 averaged 135.38 TFLOP/s/GPU and 5538.41 ms
• model, distributed optimizer, and RNG checkpoint loaded at iteration 100 and resumed with finite losses through iteration 105

PP1 exploration

PP1 was evaluated separately to identify a lower-memory path. The combined configuration in this table is now recorded in the GB200 library recipe. The early-throughput column reports steps 2-4 in order.

Variant	Peak allocated	Saved vs. baseline	Early TFLOP/s/GPU	Outcome
Selective recompute baseline	176.95 GiB	-	144.2 / 158.3 / 110.2	OOM at step 5
Attention activation offload	160.30 GiB	16.65 GiB	145.5 / 171.6 / 153.0	Completed 8 steps
MXFP8 parameter gather/storage	168.11 GiB	8.84 GiB	143.5 / 160.6 / 140.2	Completed 8 steps
Combined	151.44 GiB	25.51 GiB	146.1 / 172.7 / 154.0	Completed 8 steps

The combined configuration removed the early PP1 OOM without an early throughput regression, but the unbounded-routing run slowed after step 4 as routing became imbalanced. The recipe remains a draft candidate and does not yet have a 100-step PP1 validation.

Capacity 1.5 was also evaluated as an explicit token-drop alternative:

Router bias update rate	Steps 2-8 TFLOP/s/GPU	Assignment drop	Routing-mass drop	Decision
0.001	206.56	66.16%	64.23%	Rejected
0.01	212.69	64.18%	62.85%	Rejected

The measured drop rates are too large for a convergence recipe. The library recipe therefore keeps unlimited capacity; PP1 requires routing stabilization rather than a small fixed capacity.

B200 NVL8 candidate

The 64-GPU B200 recipe records TP1/PP8/VPP2/EP8 with portable all-to-all dispatch, full-MoE selective recompute, attention activation offload, and MXFP8 parameter gather/buffer reuse. Expert capacity remains unlimited; paged stash and CUDA graphs remain disabled.

The conservative candidate completed eight finite real-data steps with zero skipped or NaN iterations. Steps 2-8 averaged 143.54 TFLOP/s/GPU (135.8, 148.1, 148.7, 149.0, 135.2, 144.5, 143.5) and reached 116.41 GiB peak allocated / 118.61 GiB peak reserved memory. A combined lighter-recompute/no-offload candidate OOMed before its first step, so the library recipe keeps the conservative recompute and activation-offload settings.

The screen used a 64-rank Blackwell topology with eight contiguous eight-rank communication domains. Each domain spanned two four-GPU hosts rather than one physical B200 NVL8 host, so this validates the all-to-all layout and memory candidate but not physical-B200 throughput or flex-dispatcher behavior. Cross-host CUDA IPC prevented meaningful DeepEP/HybridEP validation; neither backend is claimed by this recipe.

Remaining validation

• complete an exact 100-step PP1 natural-routing run with model/optimizer/RNG checkpoint save and direct resume

• complete physical 64-GPU B200 NVL8 100-step checkpoint/save/resume validation

• complete representative 100-step natural-routing validation for the 256-GPU GB300 Pro library recipe

• I, the PR author, have personally reviewed every line of this PR.

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.