[NPUW] Enable vocabulary sharing for the asymmetric vocabulary case by AsyaPronina · Pull Request #37054 · openvinotoolkit/openvino

AsyaPronina · 2026-07-24T09:39:57Z

Details:

Vocab as input with just saving not all constants

Tickets:

EISW-227455

AI Assistance:

AI assistance used: no / yes
If yes, summarize how AI was used and what human validation was performed (build/tests/manual checks).

… WeightlessAttribute

Copilot

Pull request overview

Adds NPUW support for sharing asymmetric quantized vocabularies as inputs, including deferred u8-to-i8 shifting and LM-head rewrites.

Changes:

Adds i8 asymmetric unpacking and Sub128 lazy transformation.
Adds configurable vocabulary and MatMul-first graph rewrites.
Extends partitioning to preserve and process i8 vocabulary constants.

Key risks include an SIMD buffer overrun, ignored configuration, stale tests, and missing regression coverage.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
`src/plugins/intel_npu/src/plugin/npuw/util.cpp`	Routes i8 asymmetric unpacking.
`src/plugins/intel_npu/src/plugin/npuw/util_xarch.hpp`	Declares the i8 unpack kernel.
`src/plugins/intel_npu/src/plugin/npuw/util_xarch.cpp`	Implements AVX2 i8 dequantization.
`src/plugins/intel_npu/src/plugin/npuw/partitioning/patterns/opt.cpp`	Extends patterns to i8 weights.
`src/plugins/intel_npu/src/plugin/npuw/partitioning/partitioning.cpp`	Applies deferred shifts to closures.
`src/plugins/intel_npu/src/plugin/npuw/llm_compiled_model.cpp`	Adds vocabulary graph rewrites and configuration.
`src/plugins/intel_npu/src/plugin/npuw/lazy_tensor.hpp`	Defines the `Sub128` lazy operation.
`src/plugins/intel_npu/src/plugin/npuw/lazy_tensor.cpp`	Implements and serializes `Sub128`.
`src/plugins/intel_npu/src/plugin/CMakeLists.txt`	Registers the new cross-compiled kernel.
`src/plugins/intel_npu/src/al/include/intel_npu/config/npuw_option_defs.inc`	Defines new NPUW LLM options.

AlexanderKalistratov · 2026-08-10T20:51:12Z

+    NPUW_ASSERT(from->get_size() == to->get_size());
+    NPUW_ASSERT(from->get_size() % 8 == 0);


AlexanderKalistratov · 2026-08-10T20:50:29Z

+        auto vocab_as_input = get_option<bool>(other_props, std::string("NPUW_ASYM_VOCAB_AS_INPUT"));
+        if (m_cfg.get<::intel_npu::NPUW_LLM_ASYM_I8_VOCAB_AS_INPUT>() ||
+            m_cfg.get<::intel_npu::NPUW_LLM_ASYM_VOCAB_AS_INPUT>()) {


That's valid comment

AlexanderKalistratov · 2026-08-10T20:51:35Z

+        if ((ov::element::u8 == matched_qweight->get_element_type() ||
+             ov::element::i8 == matched_qweight->get_element_type()) &&
+            (standard_layout || pretransposed_layout)) {


please take a look

+    case TransformType::SUB128:
+        m_transform.emplace<op::Sub128>(ov::npuw::orc::load_versioned_payload<op::Sub128>(section));


AlexanderKalistratov · 2026-08-10T20:50:51Z

+class ConvertVocabAsymU8ToI8 : public ov::pass::MatcherPass {
+public:
+    OPENVINO_MATCHER_PASS_RTTI("ov::npuw::ConvertVocabU8ToI8");
+    explicit ConvertVocabAsymU8ToI8() {


Do we need tests here?

dmatveev

Do we need a new HOST_GATHER pattern to properly handle the 0th subgraph?

dmatveev · 2026-08-11T11:32:58Z

+INTEL_NPU_NPUW_SIMPLE_OPT(NPUW_LLM_ASYM_VOCAB_AS_INPUT, bool, false, ov::intel_npu::npuw::llm, asym_vocab_as_input, "NPUW_LLM_ASYM_VOCAB_AS_INPUT", LLM, EXPOSED, CACHED, ALL)
+INTEL_NPU_NPUW_SIMPLE_OPT(NPUW_LLM_ASYM_I8_VOCAB_AS_INPUT, bool, false, ov::intel_npu::npuw::llm, asym_i8_vocab_as_input, "NPUW_LLM_ASYM_I8_VOCAB_AS_INPUT", LLM, EXPOSED, CACHED, ALL)
+INTEL_NPU_NPUW_SIMPLE_OPT(NPUW_LLM_MATMUL_FIRST_VOCAB, bool, false, ov::intel_npu::npuw::llm, matmul_first_vocab, "NPUW_LLM_MATMUL_FIRST_VOCAB", LLM, EXPOSED, CACHED, ALL)


Do we need all these options? I don't think so.

If there was found a way to share asym (u4) vocabs, let just do it by default

dmatveev · 2026-08-11T11:34:04Z

+            } else if (ov::op::util::is_constant(input_node) &&
+                       input_node->get_rt_info().count(ov::npuw::weights::op::Sub128::rt_key) > 0) {
+                // A Sub128-marked Constant landed in consts_to_keep and stays inline
+                // in the function body - the shift would be silently skipped there,
+                // producing wrong numerics. Fail loudly instead.
+                OPENVINO_THROW("NPUW: Sub128-marked Constant ",
+                               input_node->get_friendly_name(),
+                               " is kept in the function body and won't be transformed");


It should be kept anyway, isn't it?

dmatveev · 2026-08-11T11:34:31Z

                                          << "] (via prototype " << proto_layer_name << ")");
-                    funcall._lazy_closure[param_idx - function._param_offset] =
-                        LazyTensor(std::static_pointer_cast<ov::op::v0::Constant>(input_node));  // (t)/1/c
+                    funcall._lazy_closure[param_idx - function._param_offset] = put_to_closure(input_node);  // (t)/1/c


Not sure why this special handling is required here.

dmatveev · 2026-08-11T11:34:51Z

-ov::Tensor Const::eval() const {
+ov::Tensor Const::eval_view() const {
    if (m_node) {
-        return ov::npuw::util::copy_tensor_from_const(m_node);
+        return ov::npuw::util::tensor_from_const(m_node);
    }


dmatveev · 2026-08-11T11:36:31Z

+ov::Tensor Sub128::eval() const {
+    const auto trs = tensor.get_transformations();
+
+    ov::Tensor src;
+    if (trs.size() == 1 && std::holds_alternative<op::Const>(trs.front())) {
+        // Fused path: read straight through a zero-copy view of the source,
+        // skipping the intermediate copy Const::eval() would make. The view is
+        // only ever READ here, so this is correct for all Const flavors,
+        // including the deserialized ones (read-only mmap / cached bin tensor)
+        src = std::get<op::Const>(trs.front()).eval_view();
+    } else {
+        src = tensor.eval();
+    }
+
+    const auto src_type = src.get_element_type();
+    NPUW_ASSERT(src_type == ov::element::u8 || src_type == ov::element::i8);
+
+    ov::Tensor dst(ov::element::i8, src.get_shape());
+    const auto* s = static_cast<const uint8_t*>(src.data());
+    auto* d = dst.data<int8_t>();
+    for (std::size_t i = 0, n = src.get_size(); i < n; ++i) {
+        d[i] = static_cast<int8_t>(static_cast<int8_t>(s[i]) - 128);
+    }
+    return dst;
+}


Honestly this could've been a generic subtract, I see no point in leaving it 128 only

dmatveev · 2026-08-11T11:39:10Z

+class ConvertVocabAsymU8ToI8 : public ov::pass::MatcherPass {
+public:


I believe this pass must be moved to the passes.

dmatveev · 2026-08-11T11:47:07Z

+        auto vocab_as_input = get_option<bool>(other_props, std::string("NPUW_ASYM_VOCAB_AS_INPUT"));
+        if (m_cfg.get<::intel_npu::NPUW_LLM_ASYM_I8_VOCAB_AS_INPUT>() ||
+            m_cfg.get<::intel_npu::NPUW_LLM_ASYM_VOCAB_AS_INPUT>()) {
+            lm_head_config["NPUW_HOST_GATHER"] = "NO";


Why? I believe HOST_GATHER was never a case for the lmhead submodel?

dmatveev · 2026-08-11T11:50:27Z

+ov::npuw::weights::LazyTensor put_to_closure(const std::shared_ptr<ov::Node>& input_node) {
+    auto const_node = std::static_pointer_cast<ov::op::v0::Constant>(input_node);
+    ov::npuw::weights::LazyTensor lt(const_node);
+    if (const_node->get_rt_info().count(ov::npuw::weights::op::Sub128::rt_key) > 0) {
+        LOG_DEBUG("Sub128 marker found on " << const_node->get_friendly_name() << " - applying to the LazyTensor");
+        lt = lt.sub128();
+    }
+    return lt;
+}


This sound very strange to me. Why we should check it here? Why cant it be part of the normal evaluation?

dmatveev · 2026-08-11T11:52:09Z

+        auto vocab_as_input = get_option<bool>(other_props, std::string("NPUW_ASYM_VOCAB_AS_INPUT"));
+        if (m_cfg.get<::intel_npu::NPUW_LLM_ASYM_I8_VOCAB_AS_INPUT>() ||
+            m_cfg.get<::intel_npu::NPUW_LLM_ASYM_VOCAB_AS_INPUT>()) {


We don't need these options I believe

AsyaPronina requested review from a team as code owners July 24, 2026 09:39

github-actions Bot added category: NPU OpenVINO NPU plugin category: NPUW NPUW plugin labels Jul 24, 2026

AsyaPronina marked this pull request as draft July 24, 2026 09:40

[NPUW] Share 1 vocab

4112c1e

AsyaPronina force-pushed the save_not_all_constants branch from baf31da to 4112c1e Compare August 3, 2026 18:29

U8 Vocab and ZP -> I8 Vocab and ZP

a5d3c8d

github-actions Bot added the category: build OpenVINO cmake script / infra label Aug 5, 2026

AsyaPronina added 7 commits August 6, 2026 00:25

Polishing

8d41dbb

Removed memory consumption by transient buffers and constants without…

3559c18

… WeightlessAttribute

Added I8 and U8 cases differently

df1db01

Fixed option issue

4e053db

Support of optimization. Temporarily default

3e9fd41

Working and performant version of sharing

e2a0a67

Fixed Clang-format

f6354b4

AsyaPronina marked this pull request as ready for review August 6, 2026 21:48

AsyaPronina requested review from AlexanderKalistratov and esmirno August 6, 2026 21:48

AlexanderKalistratov requested a balanced review from Copilot August 6, 2026 21:49

Copilot started reviewing on behalf of AlexanderKalistratov August 6, 2026 22:02 View session

Copilot AI reviewed Aug 6, 2026

View reviewed changes

dmatveev added this to the 2026.4 milestone Aug 11, 2026

dmatveev requested changes Aug 11, 2026

View reviewed changes

dmatveev changed the title ~~[NPUW] Share 1 vocab~~ [NPUW] Enable vocabulary sharing for the asymmetric vocabulary case Aug 11, 2026

		NPUW_ASSERT(from->get_size() == to->get_size());
		NPUW_ASSERT(from->get_size() % 8 == 0);

		case TransformType::SUB128:
		m_transform.emplace<op::Sub128>(ov::npuw::orc::load_versioned_payload<op::Sub128>(section));

		class ConvertVocabAsymU8ToI8 : public ov::pass::MatcherPass {
		public:

Uh oh!

Conversation

AsyaPronina commented Jul 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Details:

Tickets:

AI Assistance:

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dmatveev left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

AsyaPronina commented Jul 24, 2026 •

edited

Loading