feat(recipe): add Nemotron 3 Ultra verification recipes

[cuichenx]

What does this PR do?

Adds the initial Nemotron 3 Ultra 550B-A55B verification card and convergence-safe BF16 library pretraining recipes for 256 H100 and 256 GB200 GPUs.

The H100 recipe adopts the memory-feasible TP4/PP1/EP64 Megatron-FSDP execution policy. The GB200 recipe adopts the TP4/PP4/EP64 HybridEP and targeted activation-offload policy. Both preserve natural expert routing, numerical checks, and the library recipe optimizer/scheduler contract.

compare.py, the inference launchers, and src/megatron/bridge/perf_recipes/ are unchanged from main. The performance recipes are used only as execution-policy references; this PR places the correctness-oriented configurations in src/megatron/bridge/recipes/.

The verification card intentionally records partial progress. GPU checkpoint round-trip conversion and the exact 256-H100 BF16 real-data functional pretraining run are verified. The exact 256-GB200 real-data run remains in progress; CPU conversion, manual-forward, inference, SFT/export, PEFT, checkpoint resume, and separately scoped FSDP/performance leaves remain explicit rather than being overclaimed.

Changelog

• Add a 256-H100 BF16 Ultra library pretraining recipe with Megatron-FSDP, standard all-to-all dispatch, all-layer recompute/chunking, FP32 gradient communication, and numerical checks.
• Add a 256-GB200 BF16 Ultra library pretraining recipe with HybridEP, natural routing, targeted fused expert-MLP activation offload, and numerical checks.
• Export both configurations through the public recipe package and add focused construction/contract coverage.
• Add the Nemotron 3 Ultra verification card and record the completed H100 functional-pretraining evidence while leaving incomplete matrix entries unverified.

H100 pretraining verification

The public 256-H100 BF16 library recipe completed exactly 10 real RedPajama2 optimizer steps at TP4/PP1/CP1/EP64/ETP1, GBS/MBS 512/1, and sequence length 8192. The persisted post-setup configuration retained natural routing, standard all-to-all dispatch, Megatron-FSDP, all-108-layer recompute, FP32 gradient communication, numerical checks, and the library optimizer/constant-LR contract.

• Initial/final LM loss: 11.23305 / 9.966628
• Final-10 average: 388228.080 ms, 14.670 TFLOP/s/GPU
• Peak allocated/reserved memory: 41.934 / 45.188 GiB
• All 10 keyed rows had finite loss, throughput, and gradient norm with zero skipped and zero NaN iterations.

This is bounded functional support evidence, not a tuned-performance or full-convergence claim.

GitHub Actions CI

Local validation in nvcr.io/nvidia/nemo:26.06:

• 382 passed across final-head library-recipe factories, Ultra recipe tests, and model-card validator tests.
• After the H100 card promotion, 41 passed across focused card and Ultra recipe tests.
• Model-card validator passed.
• uv run pre-commit run --all-files passed in the container.
• git diff --check passed.

The required host uv run attempts stop before tests/hooks because nvidia-resiliency-ext==0.6.0 has no wheel compatible with the host older manylinux/glibc platform. The same commands pass in the card pinned public container.

Before your PR is "Ready for review"

Pre checks:

• Make sure you read and followed Contributor guidelines
• Did you write any new necessary tests?
• Did you add or update any necessary documentation?
• Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  • Reviewer: Does the PR have correct import guards for all optional libraries?

This remains a draft while the exact GB200 BF16 real-data pretraining run is monitored and its evidence is incorporated.

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.