Skip to content

feat(training): enable MFSDP V2 gradient clipping - #5540

Draft
wujingyue wants to merge 3 commits into
NVIDIA-NeMo:mainfrom
wujingyue:agent/mfsdp-v2-clip-grad
Draft

feat(training): enable MFSDP V2 gradient clipping#5540
wujingyue wants to merge 3 commits into
NVIDIA-NeMo:mainfrom
wujingyue:agent/mfsdp-v2-clip-grad

Conversation

@wujingyue

@wujingyue wujingyue commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

Summary

  • allow clip_grad > 0 for MFSDP V2
  • let the dense and EP=2 V2 smoke tests use the normal clipping default (1.0)

Dependencies

Validation

  • MCore two-rank dense and EP=2 clipping coverage in fix(optimizer): support MFSDP v2 gradient clipping NVIDIA/Megatron-LM#6489.
  • uv run pre-commit run --all-files
  • Bridge two-rank smokes were not runnable in this worktree: its committed MCore submodule is intentionally pinned before #6450, and the isolated local environment failed while building fast-hadamard-transform because it does not declare Torch as a build dependency.

Signed-off-by: Jingyue Wu <jingyuew@nvidia.com>
Signed-off-by: Jingyue Wu <jingyuew@nvidia.com>
Signed-off-by: Jingyue Wu <jingyuew@nvidia.com>
@copy-pr-bot

copy-pr-bot Bot commented Aug 12, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant