feat(dflash2): add MLA sliding-window training support - #181
Conversation
Signed-off-by: Yubo Wang <yubowang2019@gmail.com>
077f935 to
a3086fe
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a3086fe961
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| "mscale": 1.0, | ||
| "mscale_all_dim": 0.0, | ||
| }.items(): | ||
| rope_parameters.setdefault(key, default) |
There was a problem hiding this comment.
Replace null YaRN options with their defaults
When an MLA config explicitly contains null for a defaultable YaRN option such as beta_fast, setdefault preserves that None instead of applying the intended default. K3DSparkMLAAttention then forwards the value through build_rotary_embedding into yarn_find_correction_range, where arithmetic on None fails while constructing the model. The repository's existing RoPE normalizer and K3DSparkConfig both treat missing and null values equivalently, so this loop should likewise replace values whose .get(key) is None.
Useful? React with 👍 / 👎.
Summary
sliding_attentionandfull_attentionmasksTesting
python -m pytest tests/test_dflash2.py -q(26 passed, 57 subtests passed)ruff checkon all changed Python filespython3 -m compileall -q torchspec tests