feat(recipe): add Nemotron 3 Ultra verification recipes - #5568

Draft

cuichenx wants to merge 58 commits into

mainfrom

chcui/sol/nemotron-ultra-verification-card

Commits on Aug 2, 2026

feat(recipe): add Nemotron 3 Ultra FSDP performance recipes
cuichenx
committed
6488169
fix(recipe): disable unsupported Ultra virtual pipeline parallelism
cuichenx
committed
425586e

Commits on Aug 3, 2026

fix(recipe): use pipeline-free Ultra FSDP topology
cuichenx
committed
b722905
[recipe] fix: disable Ultra MoE recompute on H100
cuichenx
committed
d107212
[recipe] fix: recompute Ultra MoE on H100
cuichenx
committed
2a36913
[training] fix: log pretrain failures before cleanup
cuichenx
committed
04e7018
[recipe] fix: avoid replaying Ultra HybridEP
cuichenx
committed
ccfe801
[recipe] fix: use all-to-all for Ultra H100
cuichenx
committed
82df498
[recipe] fix: stabilize Ultra H100 FSDP gradients
cuichenx
committed
89840d7
[recipe] fix: fuse Ultra H100 FP32 gradients
cuichenx
committed
9950cd8
[recipe] fix: recompute Ultra H100 attention
cuichenx
committed
0d9bf22
[recipe] fix: bound Ultra H100 FSDP prefetch
cuichenx
committed
55d16fb
[recipe] fix: disable Ultra H100 FSDP prefetch
cuichenx
committed
310f5de
[recipe] fix: use async allocator for Ultra H100 FSDP
cuichenx
committed
0a5e358
[recipe] fix: disable Ultra H100 TE op fuser
cuichenx
committed
a7a32e9
[recipe] fix: use Ultra H100 activation recompute
cuichenx
committed
274ab5a
[recipe] fix: chunk Ultra H100 MLP training
cuichenx
committed
22f5fbd
[recipe] fix: increase Ultra H100 MLP chunks
cuichenx
committed
ffd97a6
[recipe] fix: use async allocator for chunked Ultra FSDP
cuichenx
committed
b867253
[recipe] fix: increase Ultra H100 MLP chunks to eight
cuichenx
committed
4eacbef
[recipe] fix: increase Ultra H100 MLP chunks to sixteen
cuichenx
committed
cab2a82
[recipe] fix: tune Ultra H100 Mamba chunk size
cuichenx
committed
8a4fe95
[recipe] fix: bound Ultra H100 CUDA connections
cuichenx
committed
c8a62f4
[recipe] fix: bound Ultra H100 NCCL buffers
cuichenx
committed
7332057
[recipe] fix: further chunk Ultra H100 MoE
cuichenx
committed
d598004
[recipe] fix: reduce Ultra H100 NCCL buffers
cuichenx
committed
b60b08a
[recipe] fix: further chunk Ultra H100 experts
cuichenx
committed
47de83e
[recipe] fix: checkpoint Ultra H100 hybrid layers
cuichenx
committed
bcb781d
[recipe] fix: checkpoint all Ultra H100 layers
cuichenx
committed
079276c
[recipe] fix: bound Ultra H100 recompute
cuichenx
committed
aaf057e
[recipe] fix: raise Ultra H100 recompute
cuichenx
committed
4f9aa9e
[recipe] fix: extend Ultra H100 recompute
cuichenx
committed
53509a2

Commits on Aug 4, 2026

Commits on Aug 5, 2026

[recipe] perf: add Ultra H100 TP4 CP2 FSDP candidate
cuichenx
committed
63aadec
[recipes] feat: add safe BF16 Ultra GB200 recipes
cuichenx
committed
b20e64b

Commits on Aug 6, 2026

Commits on Aug 7, 2026

[inference] feat: stage large model comparisons
cuichenx
committed
8f54c2d
[inference] feat: select staged comparison partition
cuichenx
committed
3d5ed8f
[inference] fix: reserve staged HF loading memory
cuichenx
committed
64b8a30

Commits on Aug 8, 2026

[performance] fix: align Ultra card with BF16 recipes
cuichenx
committed
98bce32

Commits on Aug 10, 2026

[perf recipes] fix: make Ultra GB200 BF16 runnable
cuichenx
committed
bb77c7e

Commits on Aug 13, 2026

[inference] feat: verify Ultra manual forward parity
cuichenx
committed
b0e5b0b
Merge origin/main into Nemotron Ultra verification work
cuichenx
committed
0677008
[inference] fix: align Ultra card with public launchers
cuichenx
committed
433fc57
fix(inference): preserve explicit HF device selection
cuichenx
committed
a572d99
docs(recipe): keep portable Ultra evidence unverified
cuichenx
committed
a8f2060

Commits on Aug 14, 2026

Merge remote-tracking branch 'origin/main' into chcui/sol/nemotron-ultra-verification-card
cuichenx
committed
be89f79
[recipes] fix: move Ultra verification configs to library recipes
cuichenx
committed
5e0587e
[recipes] fix: recompute all Ultra layers on H100
cuichenx
committed
69bd209
[recipes] docs: verify Ultra pretraining on H100
cuichenx
committed
f1f241b
[recipes] fix: reduce Ultra GB200 routing memory
cuichenx
committed
76ca2f2
[recipes] fix: add Ultra GB200 memory headroom
cuichenx
committed
89a0866
[recipes] fix: recompute Ultra GB200 expert activations
cuichenx
committed
e488132
[recipes] feat: add Ultra GB200 MXFP8 FSDP config
cuichenx
committed
3741693