Refactored Kleidiai executor and Support for INT4 groupwise asym Quantization

[hrushitfujitsu]

Details:

• Refactored kernel dispatch logic using a switch-case structure to improve readability, maintainability
• Added support for INT4 group-wise asymmetric quantization using Kleidiai kernels and test cases as well

AI Assistance:

• AI assistance used: no

This work was contributed by @hrushitfujitsu and @abhijain1204fujitsu

[maxnick]

@alvoron , could you please review?

[Copilot]

Pull request overview

This PR extends the Intel CPU ARM KleidiAI fully-connected/matmul path by refactoring kernel dispatch into a lookup-key + switch and adding INT4 group-wise asymmetric (u4 weights + u4 zero-points) support, along with new functional tests.

Changes:

• Added u4 compressed-weight support and validation for asymmetric group-wise quantization (zero-points + scale layout checks).
• Refactored KleidiAI executor kernel selection into a lookup-key driven switch and introduced asymmetric group-wise kernel tags.
• Added new ARM functional test instantiation for KleidiAI asymmetric group-wise decompression.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
src/plugins/intel_cpu/tests/functional/custom/subgraph_tests/src/arm/matmul_weights_decompression.cpp	Adds a new test instantiation covering u4 asymmetric group-wise decompression.
src/plugins/intel_cpu/src/nodes/fullyconnected.cpp	Expands supported compressed weight types to include u4; adds asymmetric zero-point validation; refactors attribute initialization.
src/plugins/intel_cpu/src/nodes/executors/kleidiai/kleidiai_mm.hpp	Introduces a kernel lookup-key enum and stores the lookup key in the executor.
src/plugins/intel_cpu/src/nodes/executors/kleidiai/kleidiai_mm.cpp	Refactors KleidiAI kernel dispatch into a switch on a computed key; adds asymmetric quantization detection.
src/plugins/intel_cpu/src/nodes/executors/kleidiai/kleidiai_common.hpp	Adds asymmetric group-wise INT4 kernel interfaces and packing/execution specializations.
src/plugins/intel_cpu/src/nodes/executors/fullyconnected_implementations.cpp	Adds a KleidiAI FC type mapping and expands KleidiAI supported weight precisions to include u4.

[alvoron]

Is it required to stop processing these cases to ComputeLibrary executor?

[hrushitfujitsu]

Restored the original cases

[alvoron]

This attribute is used by ACL executor. Is it required to stop initializing it?

[hrushitfujitsu]

Resolved, please check

[alvoron]

Please fix clang-format issues.
Copyright check failure could be ignored.
I also triggered Jenkins CI.

[Copilot]

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.

Suppressed comments (1)

src/plugins/intel_cpu/tests/functional/custom/subgraph_tests/src/arm/matmul_weights_decompression.cpp:99

• The second test case uses 32lu for the group size but the comment says group_size = 16, which makes the test intent unclear and can mislead future debugging.

    {{{}, {{1, 7, 256}}}, {256, 128}, 32lu},  // group_size = 32  
    {    {{-1, -1, -1}, {{10, 40, 64}, {11, 40, 64}}}, // data_shape
        {64, 128},  // weights_shape
        32lu},   // group_size = 16  
}; 

[Copilot]

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.

Suppressed comments (3)

src/plugins/intel_cpu/src/nodes/fullyconnected.cpp:213

• [BLOCKER] Adding u4 to the compressed-weight types also admits u4 graphs with no zero-point. This path does not reject them, so the executor classifies them as symmetric INT4 and routes them through the signed-i4 packers, which reinterpret values 8–15 as negative and silently changes inference results. Reject u4 unless a valid asymmetric zero-point is present, or add a separate unsigned symmetric kernel.

        if (hasWeightZeroPoints) {

src/plugins/intel_cpu/tests/functional/custom/subgraph_tests/src/arm/matmul_weights_decompression.cpp:107

• [HIGH] These tests do not exercise either new asymmetric KleidiAI kernel. ConvertMatMulToFC inserts a constant weights Transpose (convert_matmul_to_fc.cpp:240-242), then FuseFCAndTransposeOnWeights elides it and sets weightsNonTransposed=true (graph_optimizer.cpp:915-932); the new supports() check rejects every u4 configuration in that state. check_results() only checks compressed weight precision, so a fallback executor can make this suite pass. Add a test that builds the supported NxK FullyConnected layout without the elided Transpose and assert that fullyconnected_kleidiai was selected.

                                            ::testing::Values(true),  

src/plugins/intel_cpu/tests/functional/custom/subgraph_tests/src/arm/matmul_weights_decompression.cpp:101

• [HIGH] This suite name is not matched by the existing no-DOTPROD skip regex (skip_tests_config.cpp:689-691 only matches smoke_MatMulCompressedWeights_Kleidiai). On an AArch64 CPU without DOTPROD, compressed conversion is rejected while this test still expects u4 weights to remain fused, so it fails instead of being skipped. Rename the prefix to fall under the established skip pattern (or update that pattern).

INSTANTIATE_TEST_SUITE_P(smoke_MatMulCompressedWeightsGrp_Kleidiai_asym,  

[alvoron]

@hrushitfujitsu
you can ignore smoke_Conv1x1WeightCompressedToMatmul_TransposeInput failure in CI - we've seen this failed test in other PRs, so this failure is not caused by your changes. We'll fix it and ask you re rebase the PR later.

[v-Golubev]

@hrushitfujitsu fyi: the fix has been merged into master

[hrushitfujitsu]

Hi @alvoron, rebasing done

[Copilot]

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.

src/plugins/intel_cpu/src/nodes/fullyconnected.cpp:216

• [BLOCKER] u4 is accepted even when the zero-point input is empty. That configuration is classified as symmetric in kleidiai_mm.cpp and dispatched through the signed-I4 packers, which reinterpret values 8–15 as negative and can silently produce incorrect inference results. Require a non-empty zero point whenever the weights are u4 (or add a distinct unsigned-symmetric kernel) before accepting the compressed operation.

        if (hasWeightZeroPoints) {
            const auto weightsType = op->input(WEIGHTS).get_element_type();
            const auto zeroPointsType = op->input(WEIGHT_ZERO_POINTS).get_element_type();
            // KleidiAI supports asymmetric INT4 only for group-wise u4.
            if (weightsType != ov::element::u4 || zeroPointsType != ov::element::u4 || !isGroupWise) {

src/plugins/intel_cpu/src/nodes/executors/kleidiai/kleidiai_common.hpp:1235

• [HIGH] This constructor advertises f32 zero-points as supported, but packData() always obtains the buffer as uint8_t* and decodes packed nibbles. Passing the allowed f32 form therefore reads float object bytes as U4 values and produces incorrect results. Either restrict this assertion to u4 or add a precision-aware float path; the same issue must be corrected in the I8MM specialization at line 1443.

        const auto zpPrec = rhsZeroPointsMem->getDescPtr()->getPrecision();
        OPENVINO_ASSERT(zpPrec == ov::element::u4 || zpPrec == ov::element::f32,
                        "INT4 asymmetric group KleidiAI kernel expects u4 or f32 "
                        "zero-points. Got ",
                        zpPrec,

src/plugins/intel_cpu/tests/functional/custom/subgraph_tests/src/arm/matmul_weights_decompression.cpp:107

• [HIGH] This test sets transpose_weights=true, which makes FuseFCAndTransposeOnWeights set weightsNonTransposed=true (graph_optimizer.cpp:930). The updated supports() explicitly rejects every u4 configuration with that flag (kleidiai_mm.cpp:67), so these cases fall back and never exercise the new asymmetric KleidiAI kernels. Use the supported non-transposed test layout here; test the intentional fallback separately if needed.

                                            ::testing::Values(true),  

src/plugins/intel_cpu/src/nodes/executors/kleidiai/kleidiai_common.hpp:1404

• [MEDIUM] The I8MM specialization duplicates nearly the entire DOTPROD asymmetric implementation, including validation, weight/zero-point conversion, packing, allocation sizing, and execution loops. This already makes defects such as the unsupported f32 decoding easy to fix in only one path. Extract the shared asymmetric group implementation and parameterize only the kernel interface and tag.

template <>
class uKernel<KAIKernelTag::I4_NEON_IMM_GROUP_ASYM> : public uKernelBase {