Skip to content

Validate MFSDP module phase transitions - #6359

Open
wujingyue wants to merge 1 commit into
NVIDIA:mainfrom
wujingyue:agent/clean-pr-6153
Open

Validate MFSDP module phase transitions#6359
wujingyue wants to merge 1 commit into
NVIDIA:mainfrom
wujingyue:agent/clean-pr-6153

Conversation

@wujingyue

@wujingyue wujingyue commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

What does this PR do?

Make MFSDP v2 module-phase handling explicit and validated after #6153:

  • Add a public FsdpModule.phase property that permits only RESTING → FORWARD, FORWARD → RESTING, RESTING → BACKWARD, and BACKWARD → RESTING.
  • Route forward and backward hooks through that property instead of writing _phase directly.
  • Finalize the MFSDP context in pre_forward(), the first MFSDP hook to run.
  • Separate activation-recomputation detection from the phase transitions. Reentrant recomputation follows RESTING → FORWARD → RESTING; non-reentrant recomputation occurs between pre_backward() and post_backward() and preserves BACKWARD through its nested forward hooks. In both cases, recomputation skips forward-order prefetch and parameter resharding.
  • Update the activation-recompute test to assert the public phase state after backward and after a subsequent forward/backward pass.

Validation

  • python -m torch.distributed.run --nproc-per-node 2 -m pytest -q tests/unit_tests/distributed/mfsdp_v2/test_fully_shard.py -k activation_recompute
  • ruff check megatron/core/distributed/fsdp/src/megatron_fsdp/experimental tests/unit_tests/distributed/mfsdp_v2/test_fully_shard.py

@copy-pr-bot

copy-pr-bot Bot commented Aug 7, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@wujingyue
wujingyue marked this pull request as ready for review August 7, 2026 22:24
@wujingyue
wujingyue requested review from a team as code owners August 7, 2026 22:24
@wujingyue

Copy link
Copy Markdown
Contributor Author

/ok to test 652fbe1

Signed-off-by: Jingyue Wu <jingyuew@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant