[NPUW] Remove NPUW-side Whisper cross-attention SDPA decomposition - #1
Draft
parthmah14 wants to merge 2 commits into
Draft
[NPUW] Remove NPUW-side Whisper cross-attention SDPA decomposition#1parthmah14 wants to merge 2 commits into
parthmah14 wants to merge 2 commits into
Conversation
DO NOT MERGE before the corresponding openvino.genai change that makes GenAI decompose cross-attention SDPA for NPU too (matching CPU/GPU) is merged - this removes NPUW's own decomposition, so until then Whisper word-level timestamps on NPU would break. With the previous commit making the Whisper transformation passes tolerant of already-decomposed cross-attention SDPA, NPUW no longer needs to decompose it internally after those passes have run: - Removed WhisperScaledDotProductAttentionDecomposition and its helper can_move_scale_after_matmul, decompose_scaled_dot_product_attention_ for_whisper(), and add_cross_attention_qk_scaled_scores_outputs_for_ whisper() from prepare_whisper_model.cpp. - Removed the NPUW_WHISPER_DECOMPOSE_SDPA option, PrepareWhisperPrefill Model's decompose_sdpa parameter, and the whisper_kvcache_size - 1 adjustment that only applied in that path. - Removed the fake dangling Parameter->Result pairs llm_compiled_model.cpp used to mock "cross_attention_qk_scaled_scores_N" outputs on the compiled model - once GenAI adds these as real outputs before NPUW compiles the model, no mocking is needed. - WhisperInferRequest now discovers those outputs by tensor name from m_prefill_out_ports instead of a precomputed decomposed-layer count, since NPUW itself no longer produces them. - Bumped NPUW_SERIALIZATION_VERSION (blob format changed: dropped m_decomposed_sdpa_size). CVS-184242
eshiryae
reviewed
Aug 13, 2026
| // For word-level timestamps: the prefill submodel exposes one | ||
| // "cross_attention_qk_scaled_scores_N" output per decoder layer (GenAI decomposes | ||
| // cross-attention SDPA and adds these outputs before NPUW ever sees the model), found | ||
| // by name rather than by a precomputed count since NPUW no longer decomposes anything |
| auto whisper_decompose_sdpa = m_cfg.get<::intel_npu::NPUW_WHISPER_DECOMPOSE_SDPA>(); | ||
| if (whisper_decompose_sdpa) { | ||
| m_kvcache_desc.max_prompt_size = whisper_kvcache_size - 1; | ||
| } |
There was a problem hiding this comment.
We still should use whisper_kvcache_size - 1 for reshape to static in case of decomposed SDPA. As when word_timestamps is enabled, the decoder may receive tokens decoded from the entire audio chunk.
Owner
Author
There was a problem hiding this comment.
Good catch - this is needed whenever word_timestamps is active, not just as an artifact of the removed option. Restored it, now detected structurally via a new has_decomposed_cross_attention_sdpa() helper (pushed in ea2553d).
…m comment Addresses eshiryae's review comments on the cross-attention SDPA decomposition cleanup: - Restored the whisper_kvcache_size - 1 static-prompt-length adjustment, which is needed whenever word-level timestamps are active (the decoder may receive tokens decoded from the entire audio chunk in one prefill call), not just as an artifact of NPUW's removed internal decompose option. Since that option is gone, detect the condition structurally via a new has_decomposed_cross_attention_sdpa() helper instead. - Trimmed an overly detailed comment in whisper_infer_request.cpp.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #1 (openvinotoolkit#37402). DO NOT MERGE before the GenAI PR that makes GenAI decompose cross-attention SDPA for NPU too is merged - this removes NPUW's own decomposition workaround, so word-level timestamps on NPU would break until then.
Removes WhisperScaledDotProductAttentionDecomposition, the NPUW_WHISPER_DECOMPOSE_SDPA option, and the fake-output hack that mocked the qk-scores outputs on the compiled model, now that the passes from #1 handle decomposed input directly.
Part 3 of 3 for CVS-184242
Tickets: