[Whisper] Decompose cross-attention SDPA for NPU too, matching CPU/GPU - #4306
Draft
parthmah14 wants to merge 1 commit into
Draft
[Whisper] Decompose cross-attention SDPA for NPU too, matching CPU/GPU#4306parthmah14 wants to merge 1 commit into
parthmah14 wants to merge 1 commit into
Conversation
DO NOT MERGE before the corresponding NPUW change that makes the Whisper transformation passes tolerant of already-decomposed cross-attention SDPA is merged - this makes GenAI decompose upfront for NPU, so until then the NPUW passes would crash (CVS-179287). Previously, cross-attention SDPA decomposition for word-level timestamps ran unconditionally before device branching, then NPU support was added by moving it into the non-NPU branch instead and signalling NPUW to decompose it internally via a "WHISPER_DECOMPOSE_SDPA" property (renamed to "NPUW_WHISPER_DECOMPOSE_SDPA" for the plugin). Since NPUW's Whisper passes now handle decomposed cross-attention SDPA directly, that split is no longer needed: - statefull_decoder.cpp: moved the two decomposition calls back out of the non-NPU branch, so they run once for every device. - pipeline.cpp: removed the NPU-only insertion of the "WHISPER_DECOMPOSE_SDPA" property. - utils.cpp: removed the rename of that property to "NPUW_WHISPER_DECOMPOSE_SDPA", which no longer exists as an NPUW option. CVS-184242
Contributor
There was a problem hiding this comment.
Pull request overview
This PR aligns Whisper stateful decoder behavior across devices by applying cross-attention SDPA decomposition for word-level timestamps on NPU as well (instead of skipping it and signaling NPUW via a dedicated property).
Changes:
- Removes the NPU-only
WHISPER_DECOMPOSE_SDPAproperty injection from the stateful Whisper pipeline. - Applies Whisper SDPA decomposition + QK-score outputs augmentation in the stateful decoder before device-specific compilation (so it also runs on NPU when
word_timestampsis enabled). - Removes NPU config key-renaming for the now-deleted decompose property.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| src/cpp/src/whisper/pipeline.cpp | Stops injecting WHISPER_DECOMPOSE_SDPA into NPU properties; decomposition is now controlled by the decoder flag only. |
| src/cpp/src/whisper/models/statefull_decoder.cpp | Moves SDPA decomposition logic to run before the device == "NPU" branch, enabling it for NPU too when requested. |
| src/cpp/src/utils.cpp | Drops renaming of WHISPER_DECOMPOSE_SDPA → NPUW_WHISPER_DECOMPOSE_SDPA, since the property is removed. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+37
to
+40
| if (m_decompose_cross_attention_spda_ops) { | ||
| ov::genai::decompose_scaled_dot_product_attention_for_whisper(model); | ||
| ov::genai::add_cross_attention_qk_scaled_scores_outputs_for_whisper(model); | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Reverts the NPU special-case added for CVS-179287: cross-attention SDPA decomposition for word-level timestamps now runs unconditionally (gated only by word_timestamps) instead of being skipped for NPU and signalled to NPUW via a WHISPER_DECOMPOSE_SDPA property. DO NOT MERGE before openvinotoolkit/openvino#37402 (makes NPUW's Whisper passes tolerant of decomposed SDPA) is merged - until then this would hit the CVS-179287 crash again.
Part 2 of 3 for CVS-184242
Tickets: