Skip to content

feat(dflash2): add MLA sliding-window training support - #181

Merged
yubofredwang merged 1 commit into
mainfrom
feat/dflash2-mla-swa
Aug 26, 2026
Merged

feat(dflash2): add MLA sliding-window training support#181
yubofredwang merged 1 commit into
mainfrom
feat/dflash2-mla-swa

Conversation

@yubofredwang

Copy link
Copy Markdown
Collaborator

Summary

  • add an MLA attention mode for DFlash2 training using the existing dual-source MLA attention implementation
  • support mixed per-layer sliding_attention and full_attention masks
  • reuse one block mask when all layers share the same attention policy
  • add regression coverage for MLA construction and mixed sliding/full mask policies

Testing

  • python -m pytest tests/test_dflash2.py -q (26 passed, 57 subtests passed)
  • ruff check on all changed Python files
  • python3 -m compileall -q torchspec tests

Signed-off-by: Yubo Wang <yubowang2019@gmail.com>
@yubofredwang
yubofredwang merged commit f306415 into main Aug 26, 2026
1 check passed
@yubofredwang
yubofredwang deleted the feat/dflash2-mla-swa branch August 26, 2026 19:38

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a3086fe961

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

"mscale": 1.0,
"mscale_all_dim": 0.0,
}.items():
rope_parameters.setdefault(key, default)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Replace null YaRN options with their defaults

When an MLA config explicitly contains null for a defaultable YaRN option such as beta_fast, setdefault preserves that None instead of applying the intended default. K3DSparkMLAAttention then forwards the value through build_rotary_embedding into yarn_find_correction_range, where arithmetic on None fails while constructing the model. The repository's existing RoPE normalizer and K3DSparkConfig both treat missing and null values equivalently, so this loop should likewise replace values whose .get(key) is None.

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant