From 31e09f778681c14aff708a5f218afaf148e64afe Mon Sep 17 00:00:00 2001 From: Maxim Vafin Date: Fri, 14 Aug 2026 01:12:05 +0200 Subject: [PATCH 1/3] [GGUF FE] Stateful conversion, op-translator fixes and test infrastructure Groundwork on the existing frontend, all of it reachable through the GgmlOvDecoder path that master already ships. The native .gguf builder is a separate change and does not appear here. MakeStateful. The frontend always converts to a stateless graph -- every KV cache an explicit Parameter/Result pair, as optimum-intel exports -- and being stateful is the consumer's choice, registered as a DecoderTransformationExtension so it runs ahead of the built-in stateless lowering. The pass also takes over beam_idx: it is a beam-search index into an OpenVINO state with no ggml counterpart, so declaring it in a decoder would leave a consumer-less input on the stateless graph. Op translators. Keep the output port when handing a value between translators (taking .get_node_shared_ptr() silently resolved to output 0, which throws for multi-output ops such as TopK); keep the static head layout in permute op_case 4; drop the builder-only op_case numbering from RESHAPE and VIEW; give each attention Transpose its own order constant; make the graph valid under both the SDPA and PagedAttention layouts by deriving the leading dims rather than pinning them; share the TopK-indices construction between ARGSORT and TOP_K, and let TOP_K tolerate a dynamic k instead of throwing. Quantization. Support the Q2_0 (ternary) type used by the Bonsai family. Decoder interface. Drop get_model_weights, which nothing calls. Tests. Add an op-coverage gate so a newly registered op cannot ship without a conversion test, and check the activation translators against captured output from real ggml rather than a numpy reimplementation of the formula -- a numpy oracle can only confirm the formula the author already guessed, which is how the GELU_QUICK error survived. CI. Add a GGUF_FE component so frontend changes scope their own jobs. ov_gguf_frontend_tests: 137/137. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .github/components.yml | 6 + .github/coverage/tests_cpp.yml | 5 + .github/workflows/job_cxx_unit_tests.yml | 1 + .github/workflows/ubuntu_24.yml | 1 + .../openvino/frontend/gguf/decoder.hpp | 67 ++-- .../openvino/frontend/gguf/frontend.hpp | 24 +- .../openvino/frontend/gguf/make_stateful.hpp | 50 +++ src/frontends/gguf/src/CMakeLists.txt | 7 +- src/frontends/gguf/src/frontend.cpp | 33 +- src/frontends/gguf/src/input_model.cpp | 4 + src/frontends/gguf/src/input_model.hpp | 7 +- src/frontends/gguf/src/node_context.hpp | 26 +- src/frontends/gguf/src/op/argsort.cpp | 12 +- src/frontends/gguf/src/op/flash_attn_ext.cpp | 152 +++++++-- src/frontends/gguf/src/op/get_rows.cpp | 37 ++- src/frontends/gguf/src/op/glu_geglu.cpp | 13 +- src/frontends/gguf/src/op/mul_mat_id.cpp | 172 +++++++--- src/frontends/gguf/src/op/permute.cpp | 44 ++- src/frontends/gguf/src/op/reshape.cpp | 71 +++- src/frontends/gguf/src/op/rms_norm.cpp | 8 +- src/frontends/gguf/src/op/rope.cpp | 121 +++++-- src/frontends/gguf/src/op/set_rows.cpp | 9 +- src/frontends/gguf/src/op/top_k.cpp | 36 ++- src/frontends/gguf/src/op/view.cpp | 68 +++- src/frontends/gguf/src/op/weight.cpp | 48 ++- src/frontends/gguf/src/op_table.cpp | 11 +- src/frontends/gguf/src/op_table.hpp | 85 +++-- src/frontends/gguf/src/pass/make_stateful.cpp | 216 +++++++++++++ src/frontends/gguf/src/quant/gguf.hpp | 23 +- src/frontends/gguf/src/quant/gguf_quants.cpp | 6 +- src/frontends/gguf/src/quant/weights.cpp | 141 +++++++- src/frontends/gguf/src/quant/weights.hpp | 33 ++ src/frontends/gguf/src/translate_session.cpp | 95 ++++-- src/frontends/gguf/src/utils.cpp | 71 +++- src/frontends/gguf/src/utils.hpp | 39 ++- src/frontends/gguf/tests/CMakeLists.txt | 14 +- src/frontends/gguf/tests/op_test_utils.hpp | 21 +- .../tests/test_data/gelu_ggml_expected.npy | Bin 0 -> 640 bytes .../gguf/tests/test_data/gelu_ggml_input.npy | Bin 0 -> 640 bytes .../test_data/gelu_quick_ggml_expected.npy | Bin 0 -> 640 bytes .../tests/test_data/gelu_quick_ggml_input.npy | Bin 0 -> 640 bytes .../tests/test_data/silu_ggml_expected.npy | Bin 0 -> 640 bytes .../gguf/tests/test_data/silu_ggml_input.npy | Bin 0 -> 640 bytes .../gguf/tests/test_dequant_vs_ggml.cpp | 26 +- src/frontends/gguf/tests/test_extensions.cpp | 238 +++++++++++++- src/frontends/gguf/tests/test_op_coverage.cpp | 105 ++++++ src/frontends/gguf/tests/test_ops.cpp | 306 ++++++++++++++++-- 47 files changed, 2092 insertions(+), 360 deletions(-) create mode 100644 src/frontends/gguf/include/openvino/frontend/gguf/make_stateful.hpp create mode 100644 src/frontends/gguf/src/pass/make_stateful.cpp create mode 100644 src/frontends/gguf/tests/test_data/gelu_ggml_expected.npy create mode 100644 src/frontends/gguf/tests/test_data/gelu_ggml_input.npy create mode 100644 src/frontends/gguf/tests/test_data/gelu_quick_ggml_expected.npy create mode 100644 src/frontends/gguf/tests/test_data/gelu_quick_ggml_input.npy create mode 100644 src/frontends/gguf/tests/test_data/silu_ggml_expected.npy create mode 100644 src/frontends/gguf/tests/test_data/silu_ggml_input.npy create mode 100644 src/frontends/gguf/tests/test_op_coverage.cpp diff --git a/.github/components.yml b/.github/components.yml index 0aa1ce03da2d..6e7a7896e45d 100644 --- a/.github/components.yml +++ b/.github/components.yml @@ -107,6 +107,12 @@ PROXY: - GPU build: [] +GGUF_FE: + revalidate: + - CPU # the per-op tests compile and infer their converted graphs on the CPU plugin + build: + - CPU + IR_FE: revalidate: - C_API diff --git a/.github/coverage/tests_cpp.yml b/.github/coverage/tests_cpp.yml index ba6ea85498c4..4464a7cbf671 100644 --- a/.github/coverage/tests_cpp.yml +++ b/.github/coverage/tests_cpp.yml @@ -98,6 +98,11 @@ tests: mode: gtest_single profiles: [cpu] + - name: ov_gguf_frontend_tests + binary: ov_gguf_frontend_tests + mode: gtest_single + profiles: [cpu] + - name: ov_inference_functional_tests binary: ov_inference_functional_tests mode: gtest_single diff --git a/.github/workflows/job_cxx_unit_tests.yml b/.github/workflows/job_cxx_unit_tests.yml index 838505140775..fb88285f5d23 100644 --- a/.github/workflows/job_cxx_unit_tests.yml +++ b/.github/workflows/job_cxx_unit_tests.yml @@ -129,6 +129,7 @@ jobs: ${{ env.INSTALL_TEST_DIR }}/ov_ir_frontend_tests --gtest_print_time=1 --gtest_output=xml:${{ env.INSTALL_TEST_DIR }}/TEST-IRFrontend.xml - name: GGUF frontend tests + if: fromJSON(inputs.affected-components).GGUF_FE.test run: | ${{ env.SOURCE_COMMAND }} ${{ env.SETUPVARS }} ${{ env.INSTALL_TEST_DIR }}/ov_gguf_frontend_tests --gtest_print_time=1 --gtest_output=xml:${{ env.INSTALL_TEST_DIR }}/TEST-GGUFFrontend.xml diff --git a/.github/workflows/ubuntu_24.yml b/.github/workflows/ubuntu_24.yml index acc4ec98e47a..e71127ab2d24 100644 --- a/.github/workflows/ubuntu_24.yml +++ b/.github/workflows/ubuntu_24.yml @@ -109,6 +109,7 @@ jobs: build-debian-packages: true build-contrib: true build-additional-python-packages: true + generate-gguf-fixtures: true target-branch: ${{ inputs.target-branch }} cmake-options: >- -G 'Ninja Multi-Config' diff --git a/src/frontends/gguf/include/openvino/frontend/gguf/decoder.hpp b/src/frontends/gguf/include/openvino/frontend/gguf/decoder.hpp index d54fea9eac18..4e17b72c517a 100644 --- a/src/frontends/gguf/include/openvino/frontend/gguf/decoder.hpp +++ b/src/frontends/gguf/include/openvino/frontend/gguf/decoder.hpp @@ -11,6 +11,7 @@ #include #include +#include "openvino/core/any.hpp" #include "openvino/core/node.hpp" #include "openvino/frontend/decoder.hpp" #include "openvino/frontend/gguf/visibility.hpp" @@ -47,17 +48,17 @@ struct RopeConfig { // Following the established OpenVINO frontend pattern (cf. the PyTorch TorchDecoder + InputModel), // the translators see a GgufDecoder as a NODE decoder: visit_subgraph hands the visitor a fresh // decoder bound to a single node, and every per-node accessor (get_attribute, get_input_*, -// get_output_*, get_op_*) refers to that node -- no node index is threaded through. The -// MODEL-level questions (the graph's Parameter inputs, its output names, the shared RoPE config, -// and node iteration) are asked through ov::frontend::gguf::InputModel, not by treating a decoder -// instance as a "model decoder". The InputModel forwards those to the model-scope accessors below; -// a concrete decoder answers them when queried before visit_subgraph binds it to a node. +// get_output_*) refers to that node -- no node index is threaded through. The MODEL-level +// questions (the graph's Parameter inputs, its output names, the shared RoPE config, and node +// iteration) are asked through ov::frontend::gguf::InputModel, not by treating a decoder instance +// as a "model decoder". The InputModel forwards those to the model-scope accessors below; a +// concrete decoder answers them when queried before visit_subgraph binds it to a node. // // This is a typed, ggml-free interface: operation parameters are exposed through // get_attribute(name) / get_input_view_element_offset / get_output_shape / RopeConfig rather than -// raw ggml `op_params` int32 arrays. A concrete decoder (e.g. the llama.cpp cgraph decoder) only -// has to translate ggml's layout into these typed accessors -- the op translators never touch -// ggml memory. +// raw ggml `op_params` int32 arrays. A concrete decoder (e.g. the llama.cpp cgraph decoder, or the +// native .gguf builder decoder) only has to translate ggml's layout into these typed accessors -- +// the op translators never touch ggml memory. class GGUF_FRONTEND_API GgufDecoder : public DecoderBase { public: // ── Node scope (the bound node; used by the op translators) ────────────────────────────── @@ -104,26 +105,54 @@ class GGUF_FRONTEND_API GgufDecoder : public DecoderBase { // each node. This is the bridge from model scope to node scope. virtual void visit_subgraph(std::function)> node_visitor) const = 0; - // All model-scope input nodes: both primary inputs (Parameters) and auxiliary inputs - // (position IDs, KV-cache lengths, masks, etc.). Parameters are distinguished from auxiliary - // nodes by the caller via dynamic_pointer_cast. + // All model-scope primary input nodes (Parameters): the token/embedding input plus, on the + // stateful path, the KV-cache Parameters. Distinguished from auxiliary nodes by the caller via + // dynamic_pointer_cast. virtual const std::map>& get_model_inputs() const = 0; + virtual std::vector get_model_output_names() const = 0; - // NOTE: there is no get_model_weights(). A GGUF weight is surfaced as a regular node in - // visit_subgraph with the genuine ggml leaf op type "GGML_OP_NONE": the decoder marks it as a - // weight by exposing the raw weight bytes via get_attribute("data"), the ggml - // quant type name via get_attribute("quant_type") (e.g. "Q4_K", "F16") and the - // logical [rows, cols] shape via get_output_shape(). The frontend's translate_weight does the - // dequant / repacking / requantization, so the decoder never builds OV nodes itself. (Model - // inputs are also GGML_OP_NONE leaves, but they are returned via get_model_inputs() and - // resolved to Parameters before the walk, so they carry no "data".) + // ── Optional model scope ─────────────────────────────────────────────────────────────────── + // + // The accessors below are how a decoder OPTIONALLY enriches the graph; each has a + // do-nothing default so a decoder only implements what it actually knows. That is what lets + // two very different decoders satisfy one interface: the native .gguf builder answers all of + // them, while the llama.cpp cgraph decoder (which is handed an already-built ggml graph and no + // GGUF metadata) answers none and is not forced to write empty stubs. + // + // Note what is NOT here: nothing describes the execution mode. There is no is_stateful / + // is_static, because a decoder describes ggml OPERATIONS, not a deployment. Conversion always + // yields a stateless graph; a caller that wants an OpenVINO KV cache registers + // ov::frontend::gguf::pass::MakeStateful as a DecoderTransformationExtension. + + // Auxiliary model-scope inputs (position IDs, KV-cache lengths, attention masks). A decoder that + // folds these into get_model_inputs() leaves this empty. Note that beam_idx is not among them: + // it is a beam-search index into an OpenVINO state, which ggml has no counterpart for, so + // MakeStateful creates it rather than any decoder declaring it. + virtual const std::map>& get_model_extra_inputs() const { + return empty_node_map(); + } // RoPE configuration, exposed through get_attribute("rope_config"): // - at model scope (via InputModel::get_rope_config), used by TranslateSession::preprocess // to pre-build the shared rope sin/cos table (skipped when RopeConfig::n_dims == 0, i.e. // no RoPE, or per_op == true); // - at node scope, the ROPE translator reads the same key for the op's own config. + // + // NOTE: weights are surfaced as GGML_OP_NONE leaves, by every decoder -- there is no separate + // weight accessor. A decoder marks such a leaf either with the raw ggml bytes + // (get_attribute("data") + get_attribute("quant_type") + + // get_output_shape(), the llama.cpp cgraph path) or with already-extracted weight/scales/zp + // tensors (get_attribute("gguf_weight") + "gguf.blob." + "gguf_qtype", the native + // .gguf builder path). translate_weight accepts both payloads and builds the same compressed + // decompression subgraph from either. + +protected: + // Shared empty map backing the optional accessors above, which return by const reference. + static const std::map>& empty_node_map() { + static const std::map> empty; + return empty; + } }; } // namespace ov::frontend::gguf diff --git a/src/frontends/gguf/include/openvino/frontend/gguf/frontend.hpp b/src/frontends/gguf/include/openvino/frontend/gguf/frontend.hpp index 6a302edb7ea1..1c750bd5b887 100644 --- a/src/frontends/gguf/include/openvino/frontend/gguf/frontend.hpp +++ b/src/frontends/gguf/include/openvino/frontend/gguf/frontend.hpp @@ -34,6 +34,10 @@ class GGUF_FRONTEND_API FrontEnd : public ov::frontend::FrontEnd { /// - `ov::frontend::ConversionExtension` — registers a custom op translator for the /// ggml op name given by `get_op_type()`. The converter receives an /// `ov::frontend::gguf::NodeContext` and returns an `ov::OutputVector`. + /// - `ov::frontend::DecoderTransformationExtension` — registers a normalization pass, run + /// AHEAD of the frontend's built-in lowerings. This is how the execution mode is chosen: the + /// frontend always converts to a stateless graph, and a caller that wants an OpenVINO KV + /// cache registers `ov::frontend::gguf::pass::MakeStateful` (or its own variant) here. /// - `ov::frontend::TelemetryExtension` — receives error / event callbacks. /// - `ov::detail::SOExtension` — shared-library extension; its inner extension is /// recursively registered. @@ -44,17 +48,19 @@ class GGUF_FRONTEND_API FrontEnd : public ov::frontend::FrontEnd { void add_extension(const std::shared_ptr& extension) override; protected: - /// \brief Check if FrontEnd can recognize model from given parts. - /// \note Always returns false: this frontend is hidden from FrontEndManager and is never - /// auto-selected. It is used only via direct linkage, by constructing FrontEnd and - /// calling convert() on an InputModel built from a GgufDecoder. - /// \param variants Unused. - /// \return Always false. + /// \brief Check if FrontEnd can recognize the model from the given parts. + /// \param variants Either a `std::shared_ptr`, or a path to a file whose extension + /// is `.gguf` and whose first four bytes are the GGUF magic. + /// \return True for either of those; false otherwise. bool supported_impl(const std::vector& variants) const override; - /// \brief Load the input model from a GgufDecoder. - /// \param variants A single GgufDecoder (a .gguf file path is not accepted; the caller supplies - /// the decoder). variants[0] must hold a std::shared_ptr. + /// \brief Load the input model, from either of the frontend's two ingest paths. + /// \param variants A single element, holding either: + /// - a `std::shared_ptr` — a decoder supplied by a direct linker, wrapping + /// an already-built ggml graph (the llama.cpp cgraph path); or + /// - a path to a `.gguf` file — parsed here, with the transformer graph built + /// per-architecture by the native builder. + /// Both yield a GgufDecoder, so conversion past this point is identical. /// \return InputModel::Ptr InputModel::Ptr load_impl(const std::vector& variants) const override; diff --git a/src/frontends/gguf/include/openvino/frontend/gguf/make_stateful.hpp b/src/frontends/gguf/include/openvino/frontend/gguf/make_stateful.hpp new file mode 100644 index 000000000000..af94dc169439 --- /dev/null +++ b/src/frontends/gguf/include/openvino/frontend/gguf/make_stateful.hpp @@ -0,0 +1,50 @@ +// Copyright (C) 2018-2026 Intel Corporation +// SPDX-License-Identifier: Apache-2.0 +// + +#pragma once + +#include +#include +#include + +#include "openvino/frontend/gguf/visibility.hpp" +#include "openvino/pass/pass.hpp" + +namespace ov::frontend::gguf::pass { + + + +class GGUF_FRONTEND_API MakeStateful : public ov::pass::ModelPass { +public: + OPENVINO_MODEL_PASS_RTTI("gguf::MakeStateful"); + + /// \param skip_caches Friendly names of cache Parameters to leave stateless. A sliding-window + /// cache is evicted from the front rather than only appended to, so an append-grown + /// Variable would not reproduce it; such caches keep the stateless form. + /// \param append_axis Cache axis the new rows are appended along (the token axis). -1 infers it + /// as the cache Parameter's single dynamic axis, which is how a graph that does not + /// preallocate the cache states its token axis. Pass an explicit axis for a fully static + /// (preallocated) cache, where there is nothing to infer from. + /// \param beam_idx_name Name of the beam-reorder input, which this pass ADDS to the model (it + /// belongs to the state, so no decoder declares it; see the note above). The past cache is + /// gathered by it along the batch axis before the append Concat. With batch 1 / + /// beam_idx [0] that Gather is an identity, but emitting it is what lets CPU's + /// stateful_sdpa_fusion match, and it is what makes beam search work. A model that + /// already carries a Parameter of this name has it reused instead. + explicit MakeStateful(std::set skip_caches = {}, + int64_t append_axis = -1, + std::string beam_idx_name = "beam_idx") + : m_skip_caches(std::move(skip_caches)), + m_append_axis(append_axis), + m_beam_idx_name(std::move(beam_idx_name)) {} + + bool run_on_model(const std::shared_ptr& model) override; + +private: + std::set m_skip_caches; + int64_t m_append_axis; + std::string m_beam_idx_name; +}; + +} // namespace ov::frontend::gguf::pass diff --git a/src/frontends/gguf/src/CMakeLists.txt b/src/frontends/gguf/src/CMakeLists.txt index 79b50f3e7e9a..2f198db11022 100644 --- a/src/frontends/gguf/src/CMakeLists.txt +++ b/src/frontends/gguf/src/CMakeLists.txt @@ -2,10 +2,9 @@ # SPDX-License-Identifier: Apache-2.0 # -# LINKABLE_FRONTEND: installed alongside the other frontends so callers (the llama.cpp -# ggml-openvino backend, OpenVINO GenAI) can link openvino::frontend::gguf directly. It is kept -# out of the generic FrontEndManager loading API (FrontEndManager treats "gguf" as hidden); see -# the discoverability note in frontend.cpp. +# The GGUF frontend converts GGUF models to OpenVINO. It installs the library + headers alongside +# the other frontends (LINKABLE_FRONTEND, no SKIP_INSTALL) so a direct linker -- the llama.cpp +# ggml-openvino backend -- can link openvino::frontend::gguf and feed it a live GgufDecoder. ov_add_frontend(NAME gguf LINKABLE_FRONTEND FILEDESCRIPTION "FrontEnd to convert GGUF models" diff --git a/src/frontends/gguf/src/frontend.cpp b/src/frontends/gguf/src/frontend.cpp index d501c0117361..64a617360a39 100644 --- a/src/frontends/gguf/src/frontend.cpp +++ b/src/frontends/gguf/src/frontend.cpp @@ -4,6 +4,7 @@ #include "openvino/frontend/gguf/frontend.hpp" + #include "input_model.hpp" #include "op_table.hpp" #include "openvino/core/so_extension.hpp" @@ -18,12 +19,17 @@ namespace ov { namespace frontend { namespace gguf { -// Discoverability (intentional): consumed only by direct linkage -- a caller (the llama.cpp -// ggml-openvino backend, OpenVINO GenAI) links openvino::frontend::gguf and feeds FrontEnd a live -// GgufDecoder (no .gguf-path reader; see supported_impl / load_impl). It exports the standard -// plugin entry points so FrontEndManager can scan the frontend dir without error, but "gguf" is -// treated as hidden there (manager.cpp is_hidden_frontend), so it is never listed or auto-selected. -// Drop it from that list once this frontend gains file-based loading and passes production review. +// The frontend converts a live GgufDecoder -- supplied by a direct linker, currently the llama.cpp +// ggml-openvino backend -- through the op translators below. +// +// Discoverability: "gguf" is in manager.cpp's is_hidden_frontend list, so it is not advertised by +// available_front_ends() and not auto-selected by load_by_model -- core.read_model(".gguf") does +// not resolve to it. It is still reachable explicitly, either by direct linkage (what GenAI and +// the llama.cpp backend do) or by name via load_by_framework("gguf"). supported_impl below stays +// implemented, so enabling core.read_model later is just dropping the name from that list. +// +// Driving the frontend directly needs no follow-up pass: normalization runs inside convert(), and +// the only step read_model adds, update_v10_model(), fires solely for legacy IR v10. struct FrontEnd::Impl { std::unordered_map op_extension_translators; @@ -91,17 +97,14 @@ void FrontEnd::add_extension(const std::shared_ptr& extension) { } } -bool FrontEnd::supported_impl(const std::vector&) const { - // Always false: this frontend is never selected by FrontEndManager (load_by_model). It is used - // only via direct linkage -- a caller constructs FrontEnd and calls convert() with an - // InputModel built from a GgufDecoder -- which does not go through supported(). See the - // discoverability note at the top of this file. - return false; +bool FrontEnd::supported_impl(const std::vector& variants) const { + return !variants.empty() && variants[0].is>(); } InputModel::Ptr FrontEnd::load_impl(const std::vector& variants) const { FRONT_END_GENERAL_CHECK(!variants.empty(), "GGUF Frontend requires at least one parameter in model representation."); + FRONT_END_GENERAL_CHECK(variants[0].is>(), "GGUF Frontend supports loading from a GgufDecoder only."); auto decoder = variants[0].as>(); @@ -113,10 +116,8 @@ InputModel::Ptr FrontEnd::load_impl(const std::vector& variants) const } // namespace frontend } // namespace ov -// Plugin registration. The frontend is installed in the frontend directory, so it must export -// these entry points or FrontEndManager throws while scanning it. It registers as hidden (see the -// discoverability note above): FrontEndManager loads it without error but never lists or -// auto-selects it; only direct linkers use it. +// Plugin registration. Exports the standard entry points so FrontEndManager can load the library; +// selection is covered by the discoverability note at the top of this file. GGUF_FRONTEND_C_API ov::frontend::FrontEndVersion get_api_version() { return OV_FRONTEND_API_VERSION; } diff --git a/src/frontends/gguf/src/input_model.cpp b/src/frontends/gguf/src/input_model.cpp index 7969bd860dc8..3237a219d149 100644 --- a/src/frontends/gguf/src/input_model.cpp +++ b/src/frontends/gguf/src/input_model.cpp @@ -34,6 +34,10 @@ void InputModel::visit_subgraph(const std::functionvisit_subgraph(node_visitor); } +const std::shared_ptr& InputModel::get_model_decoder() const { + return m_decoder; +} + } // namespace gguf } // namespace frontend } // namespace ov diff --git a/src/frontends/gguf/src/input_model.hpp b/src/frontends/gguf/src/input_model.hpp index 7a3aec81823a..2bc1af435390 100644 --- a/src/frontends/gguf/src/input_model.hpp +++ b/src/frontends/gguf/src/input_model.hpp @@ -7,12 +7,12 @@ #include #include #include -#include "openvino/frontend/input_model.hpp" #include #include #include "openvino/frontend/gguf/decoder.hpp" #include "openvino/frontend/gguf/visibility.hpp" +#include "openvino/frontend/input_model.hpp" namespace ov::frontend::gguf { @@ -36,6 +36,11 @@ class GGUF_FRONTEND_API InputModel : public ov::frontend::InputModel { RopeConfig get_rope_config() const; void visit_subgraph(const std::function)>& node_visitor) const; + // The underlying node-scoped decoder. TranslateSession uses it for the remaining model-scope + // questions that are only relevant on the native .gguf builder / stateful path (weights, + // extra inputs, KV param/result pairs, is_stateful / is_static, tokenizer metadata). + const std::shared_ptr& get_model_decoder() const; + private: std::shared_ptr m_decoder; }; diff --git a/src/frontends/gguf/src/node_context.hpp b/src/frontends/gguf/src/node_context.hpp index dc15ee70b79b..5663fff6f20d 100644 --- a/src/frontends/gguf/src/node_context.hpp +++ b/src/frontends/gguf/src/node_context.hpp @@ -5,10 +5,10 @@ #pragma once #include -#include "openvino/frontend/node_context.hpp" #include #include "openvino/frontend/gguf/decoder.hpp" +#include "openvino/frontend/node_context.hpp" namespace ov::frontend::gguf { @@ -24,22 +24,38 @@ class NodeContext : public frontend::NodeContext { m_output_names = decoder->get_output_names(); } - size_t get_input_size() const override { - return m_decoder->get_input_size(); + const std::vector& get_input_names() const { + return m_input_names; } - int64_t get_input_view_element_offset(size_t index) const { - return m_decoder->get_input_view_element_offset(m_input_names[index]); + size_t get_input_size() const override { + return m_decoder->get_input_size(); } PartialShape get_input_shape(size_t input_index) const { return m_decoder->get_input_shape(m_input_names[input_index]); } + // Element offset of a VIEW input into a larger tensor (0 when not a view). The decoder + // already divides ggml's raw byte offset by element size, so translators work in elements. + int64_t get_input_view_element_offset(size_t index) const { + return m_decoder->get_input_view_element_offset(m_input_names[index]); + } + PartialShape get_output_shape() const { return m_decoder->get_output_shape(); } + // Convenience typed reads over get_attribute, kept so both the attribute-style op bodies and + // the accessor-style (op_case / output_type) op bodies compile against one NodeContext. + int get_op_case() const { + return get_attribute("op_case", 0); + } + + ov::element::Type get_output_type() const { + return get_attribute("output_type"); + } + Output get_input(int idx) const override { return m_tensor_map->at(m_input_names[idx]); } diff --git a/src/frontends/gguf/src/op/argsort.cpp b/src/frontends/gguf/src/op/argsort.cpp index c601238ffc5c..8470ae009cfb 100644 --- a/src/frontends/gguf/src/op/argsort.cpp +++ b/src/frontends/gguf/src/op/argsort.cpp @@ -44,15 +44,9 @@ OutputVector translate_argsort(const NodeContext& context) { const int64_t axis = in_ps.rank().is_static() ? in_ps.rank().get_length() - 1 : 3; auto k = std::make_shared(get_dimensions(input, {(int)axis}), ov::op::v0::Constant::create(ov::element::i64, {1}, {0})); - auto topk = std::make_shared(input, - k, - axis, - mode, - ov::op::v11::TopK::SortType::SORT_VALUES, - index_type, - false); - - return rename_outputs_with_suffix({topk->output(1)}, context.get_name()); + auto indices = make_topk_indices(input, k, axis, mode, index_type); + + return rename_outputs_with_suffix({indices}, context.get_name()); } } // namespace op diff --git a/src/frontends/gguf/src/op/flash_attn_ext.cpp b/src/frontends/gguf/src/op/flash_attn_ext.cpp index 18e32fb039bb..1dea26183a77 100644 --- a/src/frontends/gguf/src/op/flash_attn_ext.cpp +++ b/src/frontends/gguf/src/op/flash_attn_ext.cpp @@ -4,20 +4,31 @@ #include #include +#include + +#include "node_context.hpp" +#include "op_table.hpp" +#include "openvino/op/add.hpp" #include "openvino/op/broadcast.hpp" #include "openvino/op/concat.hpp" #include "openvino/op/constant.hpp" #include "openvino/op/convert.hpp" #include "openvino/op/convert_like.hpp" +#include "openvino/op/divide.hpp" +#include "openvino/op/exp.hpp" +#include "openvino/op/matmul.hpp" +#include "openvino/op/maximum.hpp" +#include "openvino/op/multiply.hpp" +#include "openvino/op/reduce_max.hpp" +#include "openvino/op/reduce_sum.hpp" #include "openvino/op/reshape.hpp" #include "openvino/op/scaled_dot_product_attention.hpp" #include "openvino/op/slice.hpp" +#include "openvino/op/softmax.hpp" +#include "openvino/op/subtract.hpp" +#include "openvino/op/tanh.hpp" #include "openvino/op/transpose.hpp" #include "openvino/op/unsqueeze.hpp" -#include - -#include "node_context.hpp" -#include "op_table.hpp" #include "utils.hpp" namespace ov { @@ -26,20 +37,27 @@ namespace gguf { namespace op { OutputVector translate_flash_attn_ext(const NodeContext& context) { - num_inputs_check(context, 4, 4); + num_inputs_check(context, 4, 5); auto q_f32 = context.get_input(0); auto k = context.get_input(1); auto v = context.get_input(2); auto mask = context.get_input(3); + // gpt-oss: optional 5th input is the per-head attention sink logit [n_head]. + const bool has_sinks = context.get_input_size() == 5; float scale = context.get_attribute("scale"); + float kq_soft_cap = context.get_attribute("kq_soft_cap", 0.0f); const auto sdpa_type = ov::element::f16; auto q = std::make_shared(q_f32, sdpa_type); auto scale_node = std::make_shared(sdpa_type, ov::Shape{}, std::vector{scale}); ov::Output mask_sliced, res; - const std::string mask_name = context.get_attribute("is_swa", false) ? "KQ_mask_swa_sliced" : "KQ_mask_sliced"; + // Pick the layer flavor's mask. The cgraph decoder answers the "is_swa" attribute directly; the + // builder identifies it by the mask input's name (self_kq_mask_swa). + const bool is_swa = + context.get_attribute("is_swa", false) || context.get_input_names()[3].find("swa") != std::string::npos; + const std::string mask_name = is_swa ? "KQ_mask_swa_sliced" : "KQ_mask_sliced"; if (context.has_input(mask_name)) { mask_sliced = context.get_input(mask_name); } else { @@ -54,19 +72,35 @@ OutputVector translate_flash_attn_ext(const NodeContext& context) { mask_sliced = std::make_shared(mask_sliced, sdpa_type); } + // The two decoders hand q/k/v over in different layouts, so the head axis and the need for a + // transpose depend on the op_case: + // op_case 0 (llama.cpp cgraph decoder): already PERMUTEd to [B, n_head, n_tokens, head_size], + // the canonical SDPA layout -- tile K/V on axis 1, feed SDPA directly. + // op_case 100 (native .gguf builder): ggml-natural [B, n_tokens, n_head(_kv), head_size] -- tile + // K/V on axis 2 FIRST, then transpose all three. That ordering (concat -> GQA tile -> + // single Transpose -> SDPA) is what the CPU plugin's stateful_sdpa_fusion matches + // (its multi-query-broadcast pattern sits on the KV-cache concat output, ahead of + // exactly one transpose), so the attention fuses into + // ScaledDotProductAttentionWithKVCache. + const int op_case = context.get_op_case(); + FRONT_END_CHECK_IMPLEMENTED(op_case == 0 || op_case == 100, "Unsupported FLASH_ATTN_EXT case"); + const bool ggml_natural = op_case == 100; + const size_t head_axis = ggml_natural ? 2 : 1; + auto tile_kv = [&](int64_t num_heads, int64_t num_heads_kv, int64_t head_size, ov::Output kv) { int64_t factor = num_heads / num_heads_kv; if (factor > 1 && num_heads_kv > 1) { - ov::Output kv_broadcast_shape, kv_unsqueezed, new_kv_shape; - auto unsqueeze_axes = ov::op::v0::Constant::create(ov::element::i64, Shape{}, {2}); - kv_unsqueezed = std::make_shared(kv, unsqueeze_axes); - - kv_broadcast_shape = ov::op::v0::Constant::create(ov::element::i64, - {5}, - {(int64_t)1, (int64_t)1, factor, (int64_t)1, (int64_t)1}); - new_kv_shape = - ov::op::v0::Constant::create(ov::element::i64, {4}, {(int64_t)0, num_heads, (int64_t)-1, head_size}); - + // Insert the repeat axis right after the head axis, broadcast it to `factor`, then fold + // it back into the head axis: [.., n_head_kv, ..] -> [.., n_head_kv * factor, ..]. + auto unsqueeze_axes = ov::op::v0::Constant::create(ov::element::i64, Shape{}, {(int64_t)head_axis + 1}); + auto kv_unsqueezed = std::make_shared(kv, unsqueeze_axes); + std::vector bcast(5, 1); + bcast[head_axis + 1] = factor; + auto kv_broadcast_shape = ov::op::v0::Constant::create(ov::element::i64, {5}, bcast); + // special_zero keeps the leading dims (incl. the dynamic token axis) as-is. + std::vector new_shape = ggml_natural ? std::vector{0, 0, num_heads, head_size} + : std::vector{0, num_heads, -1, head_size}; + auto new_kv_shape = ov::op::v0::Constant::create(ov::element::i64, {4}, new_shape); kv = std::make_shared(kv_unsqueezed, kv_broadcast_shape, ov::op::BroadcastType::BIDIRECTIONAL); @@ -80,17 +114,95 @@ OutputVector translate_flash_attn_ext(const NodeContext& context) { // concat), but the head-count / head-size dims are static ggml facts the decoder knows. auto q_shape = context.get_input_shape(0).to_shape(); auto k_shape = context.get_input_shape(1).to_shape(); - k = tile_kv(q_shape[1], k_shape[1], q_shape[3], k); - v = tile_kv(q_shape[1], k_shape[1], q_shape[3], v); + k = tile_kv(q_shape[head_axis], k_shape[head_axis], q_shape[3], k); + v = tile_kv(q_shape[head_axis], k_shape[head_axis], q_shape[3], v); // SDPA requires q/k/v to share an element type; match k/v to q (ConvertConvertLike lowers these). k = std::make_shared(k, q); v = std::make_shared(v, q); - auto sdpa = std::make_shared(q, k, v, mask_sliced, scale_node, false); + ov::Output q_t = q, k_t = k, v_t = v; + if (ggml_natural) { + // [B, L, H, S] -> [B, H, L, S] (canonical SDPA layout). Each transpose gets its OWN order + // constant: the GPU plugin's TransposeSDPAMatcher requires consumers_count(1) on it, and a + // shared one leaves the permutes in the decode path and blocks the broadcast-into-SDPA fusion. + auto to_bhls = [] { + return ov::op::v0::Constant::create(ov::element::i64, {4}, {0, 2, 1, 3}); + }; + q_t = std::make_shared(q, to_bhls()); + k_t = std::make_shared(k, to_bhls()); + v_t = std::make_shared(v, to_bhls()); + } + + ov::Output sdpa; + if (kq_soft_cap != 0.0f) { + // Gemma2 attention soft-cap: tanh(QK^T * scale * (1/cap)) * cap + mask -> softmax -> *V. + // OV SDPA v13 has no native softcap parameter, so we decompose the attention manually. + // Operates in f32 (q already converted to f16 for normal path; here stay f32). + // q_t / k_t / v_t are already [B, H, L, S] from the transpose above but in f16; + // convert to f32 for the manual decomposition. + using namespace ov::op; + auto q_f32_t = std::make_shared(q_t, element::f32); + auto k_f32_t = std::make_shared(k_t, element::f32); + auto v_f32_t = std::make_shared(v_t, element::f32); + auto mask_f32 = mask_sliced.get_element_type() != element::f32 + ? std::make_shared(mask_sliced, element::f32)->output(0) + : mask_sliced; + + // QK^T: [B, H, L, S] x [B, H, S, Lk] -> [B, H, L, Lk] + auto kT = + std::make_shared(k_f32_t, + v0::Constant::create(element::i64, {4}, std::vector{0, 1, 3, 2})); + auto qk = std::make_shared(q_f32_t, kT, false, false); + + // Apply scale * (1/softcap), then tanh, then *softcap + auto pre_cap_scale = v0::Constant::create(element::f32, Shape{}, std::vector{scale / kq_soft_cap}); + auto qk_scaled = std::make_shared(qk, pre_cap_scale); + auto qk_tanh = std::make_shared(qk_scaled); + auto post_cap_scale = v0::Constant::create(element::f32, Shape{}, std::vector{kq_soft_cap}); + auto qk_capped = std::make_shared(qk_tanh, post_cap_scale); + + // Add mask (already sliced to [B, 1, L, Lk] or [B, 1, 1, Lk]) + auto qk_masked = std::make_shared(qk_capped, mask_f32); + + // Softmax over last axis (key dimension) + auto attn_weights = std::make_shared(qk_masked, -1); + + // Weighted sum over values: [B, H, L, Lk] x [B, H, Lk, S] -> [B, H, L, S] + auto attn_out_caps = std::make_shared(attn_weights, v_f32_t, false, false); + + sdpa = attn_out_caps; + } else if (!has_sinks) { + sdpa = std::make_shared(q_t, k_t, v_t, mask_sliced, scale_node, false); + } else { + // gpt-oss attention sinks: a learned per-head logit participates in the softmax + // denominator (so the attention weights do not sum to 1) but contributes no value. + // OpenVINO SDPA has a native 6-input form (q, k, v, mask, scale, sink) that the CPU + // plugin folds the sink straight into its online-softmax, so we no longer decompose + // attention by hand. The sink logit is per head: [n_head] -> [1, n_head, 1, 1] to + // broadcast over [B, n_head, q, 1] (rank must equal the query rank, last dim 1). + using namespace ov::op; + auto sink = context.get_input(4); + auto sink_f16 = sink.get_element_type() != element::f16 + ? std::make_shared(sink, element::f16)->output(0) + : sink; + auto sink_shape = v0::Constant::create(element::i64, {4}, std::vector{1, (int64_t)q_shape[2], 1, 1}); + auto sink_r = std::make_shared(sink_f16, sink_shape, false); + sdpa = std::make_shared(q_t, + k_t, + v_t, + mask_sliced, + scale_node, + sink_r, + false); + } + // [B, H, L, S] -> [B, L, H, S] (ggml-natural layout expected by caller). res = std::make_shared(sdpa, ov::op::v0::Constant::create(ov::element::i64, {4}, {0, 2, 1, 3})); - res = std::make_shared(res, ov::element::f32); + // SDPA paths produce f16; the soft-cap path produces f32 directly. + if (kq_soft_cap == 0.0f) { + res = std::make_shared(res, ov::element::f32); + } return rename_outputs_with_suffix({res}, context.get_name()); } diff --git a/src/frontends/gguf/src/op/get_rows.cpp b/src/frontends/gguf/src/op/get_rows.cpp index 13b00eb84f50..d804c4940ff3 100644 --- a/src/frontends/gguf/src/op/get_rows.cpp +++ b/src/frontends/gguf/src/op/get_rows.cpp @@ -4,30 +4,53 @@ #include "node_context.hpp" #include "op_table.hpp" -#include "utils.hpp" - #include "openvino/core/node.hpp" #include "openvino/core/node_output.hpp" #include "openvino/op/constant.hpp" #include "openvino/op/convert.hpp" #include "openvino/op/gather.hpp" +#include "openvino/op/gather_elements.hpp" +#include "openvino/op/reshape.hpp" #include "openvino/op/squeeze.hpp" #include "openvino/op/unsqueeze.hpp" +#include "utils.hpp" namespace ov { namespace frontend { namespace gguf { namespace op { -OutputVector translate_get_rows(const NodeContext & context) { +OutputVector translate_get_rows(const NodeContext& context) { num_inputs_check(context, 2, 2); - int op_case = context.get_attribute("op_case", 0); + int op_case = context.get_op_case(); Output res; auto data = context.get_input(0); auto indices = context.get_input(1); + // MoE gating-weight gather: data = probs [1,1,T,E], indices = selected experts + // [1,1,T,K]; pick, per token, the probs of its K selected experts -> [1,1,T,K]. + // This is a per-row (GatherElements) gather over the expert axis, distinct from the + // embedding-style row gather below. + if (op_case == 10) { + // probs [1,1,T,E], selected [1,1,T,K] -> per-row gather over the last (expert) + // axis -> [1,1,T,K], then reshape to [1,T,K,1] for the broadcast-multiply with + // experts [1,T,K,n_embd]. Use an explicit [1,-1,K,1] reshape (K is static; T is + // dynamic) instead of Squeeze+Unsqueeze, which the CPU plugin implements as a + // Reshape internally and mis-infers the static pattern when T=1 at graph-build time. + // K is static (n_expert_used); read from the declared output shape [1,T,K,1]. + // Use PartialShape index to avoid .to_shape() throwing when T is dynamic. + const int64_t K = context.get_output_shape()[2].get_length(); + auto idx = std::make_shared(indices, ov::element::i32); + auto ge = std::make_shared(data, idx, -1); // [1,1,T,K] + auto col = std::make_shared( + ge, + ov::op::v0::Constant::create(ov::element::i64, {4}, std::vector{1, -1, K, 1}), + false); // [1,T,K,1] + return rename_outputs_with_suffix({col}, context.get_name()); + } + if (op_case == 2) { // The input comes from a VIEW indices = process_view_input(context, 1); @@ -55,10 +78,10 @@ OutputVector translate_get_rows(const NodeContext & context) { res = std::make_shared(data, indices, axis); } - auto output_type = context.get_attribute("output_type"); - if (res.get_element_type() != output_type) { - res = std::make_shared(res, output_type); + if (res.get_element_type() != context.get_output_type()) { + res = std::make_shared(res, context.get_output_type()); } + // The two Squeezes above dropped the leading axes; restore ggml's rank-4 form. res = std::make_shared(res, ov::op::v0::Constant::create(ov::element::i64, {1}, {0})); return rename_outputs_with_suffix({res}, context.get_name()); } diff --git a/src/frontends/gguf/src/op/glu_geglu.cpp b/src/frontends/gguf/src/op/glu_geglu.cpp index c4868a2cbf65..46494a4d0cdb 100644 --- a/src/frontends/gguf/src/op/glu_geglu.cpp +++ b/src/frontends/gguf/src/op/glu_geglu.cpp @@ -3,15 +3,15 @@ // #include + +#include "node_context.hpp" +#include "op_table.hpp" #include "openvino/core/node_output.hpp" #include "openvino/op/constant.hpp" #include "openvino/op/gelu.hpp" #include "openvino/op/multiply.hpp" #include "openvino/op/sigmoid.hpp" #include "openvino/op/slice.hpp" - -#include "node_context.hpp" -#include "op_table.hpp" #include "utils.hpp" namespace ov { @@ -28,7 +28,7 @@ OutputVector translate_glu_geglu(const NodeContext& context) { src0 = context.get_input(0); src1 = context.get_input(1); } else { - // GGML splits along ne[0] (OV last axis) using floor division: nc = ne[0] / 2. + // GGUF splits along ne[0] (OV last axis) using floor division: nc = ne[0] / 2. // Both halves are nc elements; if the dimension is odd, the last element is dropped. // Use Slice instead of Split to handle odd dimensions correctly. auto combined = context.get_input(0); @@ -51,7 +51,10 @@ OutputVector translate_glu_geglu(const NodeContext& context) { std::swap(src0, src1); } - // ggml's GEGLU uses the tanh GELU approximation (ggml_gelu_f32); v7::Gelu defaults to ERF. + // ggml's GGML_GLU_OP_GEGLU uses the tanh GELU approximation (ggml_gelu -> + // GGML_UNARY_OP_GELU = 0.5x(1+tanh(sqrt(2/pi) x (1+0.044715 x^2)))), NOT the erf form. + // OV's Gelu defaults to ERF, which is close but diverges ~1-2% per call and compounds + // across layers into a wrong argmax on deep models (e.g. gemma3-1b). Match ggml with TANH. auto gelu = std::make_shared(src0, ov::op::GeluApproximationMode::TANH); auto res = std::make_shared(gelu, src1); diff --git a/src/frontends/gguf/src/op/mul_mat_id.cpp b/src/frontends/gguf/src/op/mul_mat_id.cpp index e5592be78301..23ec998e5b3f 100644 --- a/src/frontends/gguf/src/op/mul_mat_id.cpp +++ b/src/frontends/gguf/src/op/mul_mat_id.cpp @@ -18,7 +18,10 @@ #include "openvino/op/reshape.hpp" #include "openvino/op/shape_of.hpp" #include "openvino/op/slice.hpp" +#include "openvino/op/transpose.hpp" #include "openvino/op/unsqueeze.hpp" +#include "ov_ops/gather_matmul.hpp" +#include "transformations/utils/utils.hpp" #include #include "node_context.hpp" @@ -182,38 +185,109 @@ ov::Output translate_mul_mat_id_mxfp4_packed(const NodeContext& contex return result; } -} // namespace - -// GGML_OP_MUL_MAT_ID: per-token MoE expert matmul. ids select which expert row of the weight -// tensor each token uses; activations are gathered/broadcast accordingly and matmul'd. -OutputVector translate_mul_mat_id(const NodeContext& context) { - num_inputs_check(context, 3, 3); - - auto expert_weights = context.get_input(0); - auto activations = context.get_input(1); - auto ids = context.get_input(2); +// Lower to the internal ov::op::internal::GatherMatmul, which the CPU/GPU plugins execute as one +// optimized batched expert-matmul (and, when the expert weights are a compressed +// Constant->Convert->[Subtract]->Multiply block, fold into GatherMatmulCompressed so the weights +// stay compressed -- no host f32 expansion, which is what keeps MoE compile memory bounded). The +// CPU GatherMatmul node requires CONSTANT-backed weights, so this path is used only when the +// expert weights come from a Constant (the real .gguf builder / cgraph weight leaf). GatherMatmul: +// A [n_activated, T, cols] (n_activated == 1 broadcasts the same input to every +// selected expert; == K gives a per-slot input) +// B [n_expert, rows, cols] (transpose_b=true -> A . Bᵀ) +// indices [T, K] i32 (the selected expert per (token, slot)) +// out [K, T, rows] +// which we reshape back to the builder's [1, T, K, rows] convention. +// +// Input layouts (OpenVINO reversed order): +// expert_weights (as) : [n_expert, rows, cols] (or reversed rank-4 [1, n_expert, rows, cols]) +// activations (b) : [.., T, cols] (gate/up, shared input) or [.., T, K, cols] (down) +// ids : [1, 1, T, K] +ov::Output translate_mul_mat_id_gathermatmul(const NodeContext& context, + ov::Output expert_weights, + ov::Output activations, + ov::Output ids) { + // Normalize the expert weights to the rank-3 [n_expert, rows, cols] GatherMatmul expects. The + // native builder surfaces them rank-3 already; the cgraph path surfaces them as a reversed + // rank-4 [1, n_expert, rows, cols], so drop the leading unit batch dim. + ov::Output as = expert_weights; + if (as.get_partial_shape().rank().is_static() && as.get_partial_shape().rank().get_length() == 4) { + auto as_shape = std::make_shared(as, ov::element::i64); + as = std::make_shared(as, get_dimensions(as_shape, {1, 2, 3}), false); + } + auto b = activations; + + // Canonicalize ids to 2D [T, K] (the builder carries leading 1-dims). + const auto ids_rank = static_cast(ids.get_partial_shape().size()); + ov::Output ids_2d = std::make_shared( + ids, + std::make_shared( + ov::OutputVector{ov::op::v0::Constant::create(ov::element::i64, ov::Shape{1}, {-1}), + get_dimensions(ids, {ids_rank - 1})}, + 0), + false); // [T, K] + const int64_t K = ids.get_partial_shape()[ids_rank - 1].get_length(); + + // Build the GatherMatmul activation A = [n_activated, T, cols]. + // gate/up: b is [.., T, cols] (one shared input fanned out to all experts) -> A = [1, T, cols] + // down : b is [.., T, K, cols] (already per-slot) -> A = [K, T, cols] + const auto& bps = b.get_partial_shape(); + const int64_t cols = bps[bps.size() - 1].get_length(); + const bool has_k = + K > 1 && bps.size() >= 2 && bps[bps.size() - 2].is_static() && bps[bps.size() - 2].get_length() == K; + ov::Output a; + if (has_k) { + auto b_tkc = std::make_shared( + b, + ov::op::v0::Constant::create(ov::element::i64, ov::Shape{3}, std::vector{-1, K, cols}), + false); + a = std::make_shared( + b_tkc, + ov::op::v0::Constant::create(ov::element::i64, ov::Shape{3}, std::vector{1, 0, 2})); + } else { + a = std::make_shared( + b, + ov::op::v0::Constant::create(ov::element::i64, ov::Shape{3}, std::vector{1, -1, cols}), + false); + } - if (expert_weights.get_element_type() == ov::element::u8 && expert_weights.get_partial_shape().rank().is_static() && - expert_weights.get_partial_shape().rank().get_length() == 5) { - return rename_outputs_with_suffix({translate_mul_mat_id_mxfp4_packed(context, expert_weights, activations, ids)}, - context.get_name()); + if (ids_2d.get_element_type() != ov::element::i32) { + ids_2d = std::make_shared(ids_2d, ov::element::i32); } - // OpenVINO sees GGML tensors in reversed dimension order: - // weights: [1, n_expert, m, k] - // activations: [1, n_tokens, n_used_or_1, k] - // ids: [1, 1, n_tokens, n_used] + // B stays in its (possibly compressed) precision so ConvertGatherMatmulToGatherMatmulCompressed + // can pick up the decompression subgraph and keep the weights compressed. + auto gmm = std::make_shared(a, as, ids_2d); // [K, T, rows] + + // [K, T, rows] -> [1, T, K, rows] (builder convention). + const int64_t rows = as.get_partial_shape()[as.get_partial_shape().size() - 2].get_length(); + auto kt2tk = std::make_shared( + gmm, + ov::op::v0::Constant::create(ov::element::i64, ov::Shape{3}, std::vector{1, 0, 2})); // [T, K, rows] + return std::make_shared( + kt2tk, + ov::op::v0::Constant::create(ov::element::i64, ov::Shape{4}, std::vector{1, -1, K, rows}), + false); +} + +// Portable fallback: per-token expert matmul via Gather (select each token's expert rows) + a +// batched MatMul. Handles non-constant expert weights (e.g. the single-op unit tests) that the CPU +// GatherMatmul node does not support. Expects reversed rank-4 inputs +// (weights [1, n_expert, m, k], activations [1, T, 1_or_K, k], ids [1, 1, T, K]). +ov::Output translate_mul_mat_id_generic(const NodeContext& context, + ov::Output expert_weights, + ov::Output activations, + ov::Output ids) { auto expert_weights_shape_4d = std::make_shared(expert_weights, ov::element::i64); auto activations_shape_4d = std::make_shared(activations, ov::element::i64); auto ids_shape_4d = std::make_shared(ids, ov::element::i64); - auto expert_weights_shape_3d = get_dimensions(expert_weights_shape_4d, {1, 2, 3}); - auto activations_shape_3d = get_dimensions(activations_shape_4d, {1, 2, 3}); - auto ids_shape_2d = get_dimensions(ids_shape_4d, {2, 3}); - - expert_weights = std::make_shared(expert_weights, expert_weights_shape_3d, false); - activations = std::make_shared(activations, activations_shape_3d, false); - ids = std::make_shared(ids, ids_shape_2d, false); + expert_weights = std::make_shared(expert_weights, + get_dimensions(expert_weights_shape_4d, {1, 2, 3}), + false); + activations = std::make_shared(activations, + get_dimensions(activations_shape_4d, {1, 2, 3}), + false); + ids = std::make_shared(ids, get_dimensions(ids_shape_4d, {2, 3}), false); if (ids.get_element_type() != ov::element::i32 && ids.get_element_type() != ov::element::i64) { ids = std::make_shared(ids, ov::element::i32); @@ -242,25 +316,51 @@ OutputVector translate_mul_mat_id(const NodeContext& context) { FRONT_END_OP_CONVERSION_CHECK(output_shape.rank().is_static() && output_shape.rank().get_length() == 4, "Unexpected MUL_MAT_ID output rank"); FRONT_END_OP_CONVERSION_CHECK(output_shape[3].is_static(), "Expected static row dimension for MUL_MAT_ID output"); - const auto row_dim_value = output_shape[3].get_length(); - auto row_dim = ov::op::v0::Constant::create(ov::element::i64, {1}, {row_dim_value}); + auto row_dim = ov::op::v0::Constant::create(ov::element::i64, {1}, {output_shape[3].get_length()}); ov::Output result = std::make_shared(activations_expanded, selected_weights, false, true); - - auto result_target_dims = std::make_shared( - ov::OutputVector{ - batch_dim, - get_dimensions(ids_shape, {0, 1}), - row_dim, - }, - 0); + auto result_target_dims = + std::make_shared(ov::OutputVector{batch_dim, get_dimensions(ids_shape, {0, 1}), row_dim}, + 0); result = std::make_shared(result, result_target_dims, false); - if (result.get_element_type() != output_type) { result = std::make_shared(result, output_type); } + return result; +} + +} // namespace +// GGML_OP_MUL_MAT_ID: per-token MoE expert matmul. ids select which expert row of the weight +// tensor each token uses; activations are gathered/broadcast accordingly and matmul'd. Dispatch: +// - packed MXFP4 experts -> on-graph dequant + per-expert matmul; +// - constant-backed experts (the real .gguf models) -> internal GatherMatmul (compressed, +// memory-bounded, one fused batched matmul); +// - non-constant experts (single-op tests / dynamic) -> portable Gather + MatMul fallback. +OutputVector translate_mul_mat_id(const NodeContext& context) { + num_inputs_check(context, 3, 3); + + auto expert_weights = context.get_input(0); + auto activations = context.get_input(1); + auto ids = context.get_input(2); + + if (expert_weights.get_element_type() == ov::element::u8 && expert_weights.get_partial_shape().rank().is_static() && + expert_weights.get_partial_shape().rank().get_length() == 5) { + return rename_outputs_with_suffix({translate_mul_mat_id_mxfp4_packed(context, expert_weights, activations, ids)}, + context.get_name()); + } + + // The CPU GatherMatmul node requires constant-backed weights. Real .gguf models feed a + // (possibly compressed) Constant weight leaf -> use the fused GatherMatmul path (compressed, + // memory-bounded). A non-constant weights input (single-op tests, dynamic producers) can't use + // GatherMatmul on CPU, so fall back to the portable Gather + MatMul lowering. + ov::Output result; + if (ov::op::util::is_on_path(expert_weights)) { + result = translate_mul_mat_id_gathermatmul(context, expert_weights, activations, ids); + } else { + result = translate_mul_mat_id_generic(context, expert_weights, activations, ids); + } return rename_outputs_with_suffix({result}, context.get_name()); } diff --git a/src/frontends/gguf/src/op/permute.cpp b/src/frontends/gguf/src/op/permute.cpp index e55506025aaa..e7253f1990e5 100644 --- a/src/frontends/gguf/src/op/permute.cpp +++ b/src/frontends/gguf/src/op/permute.cpp @@ -5,6 +5,9 @@ #include #include #include + +#include "node_context.hpp" +#include "op_table.hpp" #include "openvino/core/node.hpp" #include "openvino/op/add.hpp" #include "openvino/op/concat.hpp" @@ -12,9 +15,6 @@ #include "openvino/op/reshape.hpp" #include "openvino/op/slice.hpp" #include "openvino/op/transpose.hpp" - -#include "node_context.hpp" -#include "op_table.hpp" #include "utils.hpp" namespace ov { @@ -25,7 +25,7 @@ namespace op { OutputVector translate_permute(const NodeContext& context) { num_inputs_check(context, 1, 1); - int op_case = context.get_attribute("op_case", 0); + int op_case = context.get_op_case(); FRONT_END_CHECK_IMPLEMENTED(op_case == 1 || op_case == 2 || op_case == 3 || op_case == 4, "Unsupported PERMUTE case"); @@ -37,17 +37,33 @@ OutputVector translate_permute(const NodeContext& context) { res = std::make_shared(src, perm); } else if (op_case == 4) { auto output_shape = context.get_output_shape().to_shape(); - auto n_heads = ov::op::v0::Constant::create(ov::element::i64, {1}, {output_shape[1]}); - auto head_size = ov::op::v0::Constant::create(ov::element::i64, {1}, {output_shape[3]}); - auto n_seq_active = context.has_input("n_seq_active") - ? context.get_input("n_seq_active") - : ov::op::v0::Constant::create(ov::element::i64, {1}, {output_shape[0]}); - auto neg_one = ov::op::v0::Constant::create(ov::element::i64, {1}, {-1}); + const auto n_heads = static_cast(output_shape[1]); + const auto head_size = static_cast(output_shape[3]); - auto new_shape = - std::make_shared(ov::OutputVector{n_seq_active, neg_one, n_heads, head_size}, 0); + ov::Output reshaped; + if (context.has_input("n_seq_active")) { + // Reshape shape inference can only use a pattern whose value bounds are known, and + // `n_seq_active` is a Parameter, so it has none. Building the whole pattern with a single + // Concat therefore discards the statically known n_heads/head_size as well, and Q reaches + // SDPA with a dynamic head size, which makes the GPU plugin decompose SDPA into + // Gemm+SoftMax. Splitting the reshape keeps the head layout in an all-constant pattern, so + // it survives shape inference. Both reshapes are metadata-only, so this costs no extra + // data movement. + auto neg_one = ov::op::v0::Constant::create(ov::element::i64, {1}, {-1}); + auto seq_pattern = + std::make_shared(ov::OutputVector{context.get_input("n_seq_active"), neg_one}, 0); + auto by_seq = std::make_shared(src, seq_pattern, false); - auto reshaped = std::make_shared(src, new_shape, true); + auto head_pattern = + ov::op::v0::Constant::create(ov::element::i64, {4}, std::vector{0, -1, n_heads, head_size}); + reshaped = std::make_shared(by_seq, head_pattern, true); + } else { + auto new_shape = ov::op::v0::Constant::create( + ov::element::i64, + {4}, + std::vector{static_cast(output_shape[0]), -1, n_heads, head_size}); + reshaped = std::make_shared(src, new_shape, true); + } res = std::make_shared(reshaped, perm); } else { auto cache_shape = src.get_partial_shape(); @@ -73,6 +89,8 @@ OutputVector translate_permute(const NodeContext& context) { seq_active_end = context.get_input("seq_active_end"); } else { int64_t n_seq_active = output_shape[0]; + // The decoder exposes the view's sequence-axis start as a typed attribute (already in + // elements); the op translators never touch raw ggml strides/offsets. int64_t seq_active_start_val = context.get_attribute("view_seq_offset", 0); int64_t seq_active_end_val = seq_active_start_val + n_seq_active; seq_active_start = ov::op::v0::Constant::create(ov::element::i64, {1}, {seq_active_start_val}); diff --git a/src/frontends/gguf/src/op/reshape.cpp b/src/frontends/gguf/src/op/reshape.cpp index b5039fa80b94..b086722f1d0b 100644 --- a/src/frontends/gguf/src/op/reshape.cpp +++ b/src/frontends/gguf/src/op/reshape.cpp @@ -14,6 +14,7 @@ #include "openvino/op/concat.hpp" #include "openvino/op/constant.hpp" #include "openvino/op/reshape.hpp" +#include "openvino/op/transpose.hpp" #include #include @@ -22,17 +23,19 @@ namespace frontend { namespace gguf { namespace op { -OutputVector translate_reshape(const NodeContext & context) { +OutputVector translate_reshape(const NodeContext& context) { num_inputs_check(context, 1, 1); - if (context.get_input(0).get_partial_shape() == context.get_output_shape()) { + if (context.get_input_shape(0) == context.get_output_shape()) { return {context.get_input(0)}; } - int op_case = context.get_attribute("op_case", 0); - FRONT_END_CHECK_IMPLEMENTED( - op_case == 1 || op_case == 2 || op_case == 3 || op_case == 4 || op_case == 5 || op_case == 6 || - op_case == 7 || op_case == 8, - "Unsupported RESHAPE case"); + // One numbering for both ingest paths: every case below is reachable from the llama.cpp cgraph + // decoder (see ggml-decoder.cpp::compute_op_case) and from the native .gguf builder, which + // describes its reshapes so that the same case applies. + int op_case = context.get_op_case(); + FRONT_END_CHECK_IMPLEMENTED(op_case == 1 || op_case == 2 || op_case == 3 || op_case == 4 || op_case == 5 || + op_case == 6 || op_case == 7 || op_case == 8, + "Unsupported RESHAPE case"); if (op_case == 8) { // Identity reshape (ggml src ne == node ne): a no-op. Pass the input through so any dynamic @@ -43,27 +46,68 @@ OutputVector translate_reshape(const NodeContext & context) { auto output_shape = context.get_output_shape().to_shape(); std::shared_ptr new_shape_node; if (op_case == 1) { + // [B, 1, T, n_head*head_size] -> [B, T, n_head, head_size]: split the last dim into heads and + // flatten whatever leads it into dim 1. Same shape in both stateful and non-stateful paths; + // the 3D form was causing RoPE broadcasting to T×T when the trailing dimensions are 1 (MQA, + // n_head_kv=1). + // + // The leading dim is COPIED from the input via special_zero rather than written as + // output_shape[0] (a literal 1). That is what makes the attention block layout-polymorphic: + // ov::pass::SDPAToPagedAttention moves the token count into dim 0 by rewriting input_ids, and + // a literal here would discard that and leave PA deriving [1, T*H*S] operands where the + // plugin wants [T, H*S]. With the 0 the same constant serves both: + // SDPA inference: in [1, 1, T, H*S] -> [1, T, H, S] + // PagedAttention: in [T, 1, 1, H*S] -> [T, 1, H, S] (identical buffer, tokens in dim 0) new_shape_node = ov::op::v0::Constant::create( - ov::element::i64, {4}, - std::vector{(int64_t) output_shape[0], -1, (int64_t) output_shape[2], (int64_t) output_shape[3]}); + ov::element::i64, + {4}, + std::vector{0, -1, (int64_t)output_shape[2], (int64_t)output_shape[3]}); + return rename_outputs_with_suffix( + {std::make_shared(context.get_input(0), new_shape_node, /*special_zero=*/true)}, + context.get_name()); } else if (op_case == 2) { + // Merge the heads back after attention. Like op_case 1, the leading dim is copied from the input + // (special_zero) rather than pinned to output_shape[0], so the token axis stays wherever the + // active attention backend put it. + // + // The rank stays 4 because the very next op is the residual Add against the layer input and OV + // broadcasts elementwise operands from the RIGHT: every activation in the graph is rank 4 (ggml's + // own convention), so mixing in a rank-3 result would right-align and silently form a + // token x token outer product once the token count is not on the axis one happens to expect. + // in [1, T, H, S] -> [1, 1, T, H*S] + // The last dim is the static n_head*head_size and the -1 absorbs the remaining axis, so the + // following MatMul against [n_embd, n_embd] is unaffected. new_shape_node = ov::op::v0::Constant::create( - ov::element::i64, {4}, - std::vector{(int64_t) output_shape[0], (int64_t) output_shape[1], -1, (int64_t) output_shape[3]}); + ov::element::i64, + {4}, + std::vector{0, (int64_t)output_shape[1], -1, (int64_t)output_shape[3]}); + return rename_outputs_with_suffix( + {std::make_shared(context.get_input(0), new_shape_node, /*special_zero=*/true)}, + context.get_name()); } else if (op_case == 3) { // Flatten-for-SET_ROWS: [F, tok, 1, 1] -> [1, F*tok, -1, 1] (the KV-cache write path, e.g. // gpt-oss cache_v). Token count stays on the dynamic axis via -1. new_shape_node = ov::op::v0::Constant::create( - ov::element::i64, {4}, std::vector{(int64_t) output_shape[0], (int64_t) output_shape[1], -1, 1}); + ov::element::i64, + {4}, + std::vector{(int64_t)output_shape[0], (int64_t)output_shape[1], -1, 1}); } else if (op_case == 4) { return {context.get_input(0).get_node_shared_ptr()->input_value(0)}; } else if (op_case == 5) { - std::vector shape_vec = {1, 1, -1, (int64_t) output_shape[3]}; + std::vector shape_vec = {1, 1, -1, (int64_t)context.get_output_shape().to_shape()[3]}; new_shape_node = ov::op::v0::Constant::create(ov::element::i64, {4}, shape_vec); + // // Alternative + // auto token_len = context.get_input("token_len"); + // auto emb_size = + // ov::op::v0::Constant::create(ov::element::i64, {1}, {(int64_t) + // context.get_output_shape().to_shape()[3]}); + // auto one = ov::op::v0::Constant::create(ov::element::i64, {1}, {1}); + // new_shape_node = std::make_shared(ov::OutputVector{one, one, token_len, emb_size}, 0); + } else if (op_case == 6) { // The output layout rearranges dims relative to the input (e.g. qwen3-next q/k_conv_predelta: // [128,2,8,T] -> [128,16,T,1]). The decoder supplies the OV-order target with -1 on the dynamic @@ -81,6 +125,7 @@ OutputVector translate_reshape(const NodeContext & context) { new_shape_node = ov::op::v0::Constant::create( ov::element::i64, {output_shape.size()}, std::vector(output_shape.begin(), output_shape.end())); + } auto res = std::make_shared(context.get_input(0), new_shape_node, false); return rename_outputs_with_suffix({res}, context.get_name()); diff --git a/src/frontends/gguf/src/op/rms_norm.cpp b/src/frontends/gguf/src/op/rms_norm.cpp index 9cfbdefe502e..0813215f5870 100644 --- a/src/frontends/gguf/src/op/rms_norm.cpp +++ b/src/frontends/gguf/src/op/rms_norm.cpp @@ -2,15 +2,15 @@ // SPDX-License-Identifier: Apache-2.0 // +#include "openvino/decompositions/rms_norm.hpp" + #include +#include "node_context.hpp" +#include "op_table.hpp" #include "openvino/core/node_output.hpp" -#include "openvino/decompositions/rms_norm.hpp" #include "openvino/op/constant.hpp" #include "openvino/pass/node_registry.hpp" - -#include "node_context.hpp" -#include "op_table.hpp" #include "utils.hpp" namespace ov { diff --git a/src/frontends/gguf/src/op/rope.cpp b/src/frontends/gguf/src/op/rope.cpp index ec5cabf6ed84..8f2e57a5e9bb 100644 --- a/src/frontends/gguf/src/op/rope.cpp +++ b/src/frontends/gguf/src/op/rope.cpp @@ -24,7 +24,9 @@ #include "openvino/op/slice.hpp" #include "openvino/op/split.hpp" #include "openvino/op/subtract.hpp" +#include "openvino/op/transpose.hpp" #include "openvino/op/unsqueeze.hpp" +#include "openvino/pass/node_registry.hpp" #include "node_context.hpp" #include "op_table.hpp" @@ -38,7 +40,7 @@ namespace op { OutputVector translate_rope(const NodeContext& context) { num_inputs_check(context, 2, 3); - int op_case = context.get_attribute("op_case", 0); + int op_case = context.get_op_case(); ov::Output res; @@ -68,37 +70,40 @@ OutputVector translate_rope(const NodeContext& context) { cos_theta_node = sin_cos.second; } - // The canonical [1, -1, n_head, head_size] reshape target (token count on the dynamic axis), - // used by the VIEW prologue and the TYPE_NORMAL stack below. + // The canonical [B, -1, n_head, head_size] reshape target (token count on the dynamic axis), used + // by the VIEW prologue and the TYPE_NORMAL stack below. The leading 0 is a special_zero marker + // that COPIES the input's dim 0 rather than pinning a literal 1, so a token-major activation + // (the layout ov::pass::SDPAToPagedAttention establishes) keeps its tokens in dim 0. Every use + // must therefore pass special_zero=true. auto make_bhsd_shape = [&]() { return ov::op::v0::Constant::create( ov::element::i64, {4}, - std::vector{1, -1, (int64_t)output_shape[2], (int64_t)output_shape[3]}); + std::vector{0, -1, (int64_t)output_shape[2], (int64_t)output_shape[3]}); }; if (op_case == 2) { // The input comes from a VIEW int slice_len = static_cast(output_shape[2] * output_shape[3]); data = process_view_input(context, 0, slice_len); - data = std::make_shared(data, make_bhsd_shape(), false); + data = std::make_shared(data, make_bhsd_shape(), true); } if (mode == TYPE_NORMAL) { // Emit the Flux-style interleaved RoPE pattern so ov::pass::RoPEFusionFlux // folds this subgraph into ov::op::internal::RoPE → GPU ocl::rope::opt kernel. // RoPEFusionFlux requires rank-4 x with static last two dims [n_heads, head_size]. - // After the VIEW prologue the data is already [1,L,n_heads,head_size] (non-stateful) - // or [L,n_heads,head_size] (stateful, lifted to rank-4 below). + // After the VIEW prologue the data is already [B,L,n_heads,head_size]. const int64_t n_heads = static_cast(output_shape[2]); const int64_t head_size = static_cast(output_shape[3]); const int64_t half = head_size / 2; - // Reshape to [1, L, n_heads, half, 2] to expose interleaved pairs (reinterprets any - // incoming rank in element order, so no separate rank lift is needed). - auto paired_shape = ov::op::v0::Constant::create( - ov::element::i64, {5}, std::vector{1, -1, n_heads, half, 2}); - auto x_paired = std::make_shared(data, paired_shape, false); + // Reshape to [B, L, n_heads, half, 2] to expose interleaved pairs (reinterprets any + // incoming rank in element order, so no separate rank lift is needed). The leading 0 copies + // the input's batch dim (special_zero) so the token axis stays where the caller had it. + auto paired_shape = + ov::op::v0::Constant::create(ov::element::i64, {5}, std::vector{0, -1, n_heads, half, 2}); + auto x_paired = std::make_shared(data, paired_shape, true); auto split_axis = ov::op::v0::Constant::create(ov::element::i64, ov::Shape{}, {-1LL}); auto data_split = std::make_shared(x_paired, split_axis, 2); @@ -109,7 +114,7 @@ OutputVector translate_rope(const NodeContext& context) { auto x1_neg = std::make_shared(x1, neg_one_f); auto x_rotated_paired = std::make_shared(ov::OutputVector{x1_neg, x0}, -1); - auto x_rotated = std::make_shared(x_rotated_paired, make_bhsd_shape(), false); + auto x_rotated = std::make_shared(x_rotated_paired, make_bhsd_shape(), true); // Expand cos/sin from [B, L, 1, half] to [B, L, 1, head_size]. auto expand_cos_sin = [&](ov::Output cs) -> ov::Output { @@ -136,27 +141,81 @@ OutputVector translate_rope(const NodeContext& context) { const int64_t head_dim = static_cast(output_shape[3]); const int64_t n_rot = rope_config.n_dims > 0 ? rope_config.n_dims : head_dim; - // Rotate only the first n_rot elements of every head and concatenate the untouched tail. - Output rotary_in = data; - Output pass_through; + // Split the head into the rotated block [0, n_rot) and the untouched tail [n_rot, head_dim) + // on the innermost axis. Both branches below rotate `rotary_in` and re-concatenate the tail. + auto split_rotary = [&](ov::Output x, + ov::Output& rotary_in, + ov::Output& pass_through) { + rotary_in = x; + if (n_rot < head_dim) { + auto neg_one = ov::op::v0::Constant::create(ov::element::i64, {1}, {-1}); + auto zero = ov::op::v0::Constant::create(ov::element::i64, {1}, {0}); + auto one = ov::op::v0::Constant::create(ov::element::i64, {1}, {1}); + auto n_rot_c = ov::op::v0::Constant::create(ov::element::i64, {1}, {n_rot}); + auto head_c = ov::op::v0::Constant::create(ov::element::i64, {1}, {head_dim}); + rotary_in = std::make_shared(x, zero, n_rot_c, one, neg_one); + pass_through = std::make_shared(x, n_rot_c, head_c, one, neg_one); + } + }; + + // Build the canonical NEOX RoPE via the shared decomposition helper, which emits the exact + // split-halves + Multiply(-1)+Add + Concat pattern that ov::pass::RoPEFusion (specifically + // the RoPEFusionGPTOSS matcher) folds into the fused ov::op::internal::RoPE primitive on + // CPU/GPU. + // + // That matcher only fires when the rotated tensor is laid out as [B, H, L, S] and the + // cos/sin caches are [?, 1, ?, head/2]. Our tensors are ggml-natural: data is [B, L, H, S] + // and cos/sin are [B, L, 1, head/2]. So we transpose every operand into the canonical + // [B, H, L, S] layout (heads on axis 1), run the decomposition there, and transpose the + // result back to the gguf layout. The math is unchanged; the wrapping Transposes are sunk / + // cancelled against the adjacent PERMUTE during TransposeSinking. + const int64_t n_head_rope = static_cast(output_shape[2]); + const int64_t head_size_rope = static_cast(output_shape[3]); + const auto perm_bhls = ov::op::v0::Constant::create(ov::element::i64, {4}, {0, 2, 1, 3}); + + // The DATA reaches this op in inconsistent shapes depending on the layer's upstream rank: + // rank-3 [B, L, H*S] (e.g. n_head_kv=1 layers fed by a rank-3 producer), or rank-4 that may + // be [B, L, H, S] OR [B, 1, L, S]. A single fixed Transpose cannot normalize all of these. + // Instead, always Reshape the data to the canonical ggml-natural [B, L, H, S] using the op's + // output_shape (element order is preserved, so this correctly reinterprets every incoming + // layout), then Transpose {0,2,1,3}. The leading dim is copied through (special_zero) instead + // of written as a literal 1, so a token-major activation ([L,1,H,S], the layout + // ov::pass::SDPAToPagedAttention establishes) keeps its tokens in dim 0 here; cos/sin below + // broadcast against either arrangement. + auto data_to_bhls = [&](ov::Output x) -> ov::Output { + auto shape4d = ov::op::v0::Constant::create(ov::element::i64, + {4}, + std::vector{0, -1, n_head_rope, head_size_rope}); + x = std::make_shared(x, shape4d, true); // [B, L, H, S] + return std::make_shared(x, perm_bhls); // [B, H, L, S] + }; + // cos/sin always arrive rank-4 [B, L, 1, head/2]; just transpose to [B, 1, L, head/2]. + auto cossin_to_bhls = [&](ov::Output x) -> ov::Output { + return std::make_shared(x, perm_bhls); + }; + + auto x_bhls = data_to_bhls(data); // [B, H, L, S] + auto cos_bhls = cossin_to_bhls(cos_theta_node); // [B, 1, L, n_rot/2] + auto sin_bhls = cossin_to_bhls(sin_theta_node); // [B, 1, L, n_rot/2] + + // Slice the rotated block AFTER the layout change so the reshape above still sees the full + // head; the innermost axis is the head axis in both layouts. + ov::Output rotary_in; + ov::Output pass_through; + split_rotary(x_bhls, rotary_in, pass_through); + + ov::pass::NodeRegistry reg; + ov::Output roped = + ov::decomposition::rope(reg, rotary_in, cos_bhls, sin_bhls, n_rot / 2); // [B, H, L, n_rot] if (n_rot < head_dim) { - auto neg_one = ov::op::v0::Constant::create(ov::element::i64, {1}, {-1}); - auto zero = ov::op::v0::Constant::create(ov::element::i64, {1}, {0}); - auto one = ov::op::v0::Constant::create(ov::element::i64, {1}, {1}); - auto n_rot_c = ov::op::v0::Constant::create(ov::element::i64, {1}, {n_rot}); - auto head_c = ov::op::v0::Constant::create(ov::element::i64, {1}, {head_dim}); - rotary_in = std::make_shared(data, zero, n_rot_c, one, neg_one); - pass_through = std::make_shared(data, n_rot_c, head_c, one, neg_one); + roped = std::make_shared(ov::OutputVector{roped, pass_through}, -1); } - // Core split-halves RoPE via the shared decomposition helper: it emits the exact - // split + Multiply(-1)+Add + Concat pattern that ov::pass::RoPEFusion folds into - // ov::op::internal::RoPE (the previous hand-built Subtract form did not match and so - // was never fused). cos/sin already carry the n_rot/2 width. - ov::pass::NodeRegistry reg; - Output rotated = ov::decomposition::rope(reg, rotary_in, cos_theta_node, sin_theta_node, n_rot / 2); - res = (n_rot < head_dim) ? std::make_shared(ov::OutputVector{rotated, pass_through}, -1) - : rotated; + // Back to the gguf layout the rest of the graph expects (the downstream PERMUTE consumes + // rank-4 [B, L, H, S]). + res = std::make_shared( + roped, + ov::op::v0::Constant::create(ov::element::i64, {4}, {0, 2, 1, 3})); // [B, L, H, S] } else if (mode == TYPE_IMROPE) { // Partial rotary (ggml n_dims < head_dim): only the first n_rot dims of every head are // rotated, the tail is passed through unchanged -- e.g. qwen3.5 has head_dim 256 but diff --git a/src/frontends/gguf/src/op/set_rows.cpp b/src/frontends/gguf/src/op/set_rows.cpp index cbb92052c924..48bf85f1985e 100644 --- a/src/frontends/gguf/src/op/set_rows.cpp +++ b/src/frontends/gguf/src/op/set_rows.cpp @@ -33,7 +33,7 @@ OutputVector translate_set_rows(const NodeContext & context) { auto indices = context.get_input(1); auto dst = context.get_input(2); - data = std::make_shared(data, context.get_attribute("output_type")); + data = std::make_shared(data, context.get_output_type()); // Row size = the destination cache's innermost dim. Using the dst input (not the SET_ROWS // output shape) matters for the flattened KV-cache write (gpt-oss cache_v is stored as @@ -46,10 +46,13 @@ OutputVector translate_set_rows(const NodeContext & context) { auto ind_squeezed = std::make_shared(indices, ov::op::v0::Constant::create(ov::element::i64, {3}, {0, 1, 2})); + // Flatten the new rows to [.., 1, tokens, row_size]. The leading dim is copied from the incoming + // data (special_zero) instead of pinned to 1, so the KV write stays in whichever layout the + // attention block is running in; the stateful lowering re-splits it against the cache shape. auto data_reshaped = std::make_shared( data, - ov::op::v0::Constant::create(ov::element::i64, {4}, {(int64_t) 1, (int64_t) 1, (int64_t) -1, row_size}), - false); + ov::op::v0::Constant::create(ov::element::i64, {4}, {(int64_t)0, (int64_t)1, (int64_t)-1, row_size}), + true); auto set_rows = std::make_shared(data_reshaped, ind_squeezed, dst); return rename_outputs_with_suffix({set_rows}, context.get_name()); diff --git a/src/frontends/gguf/src/op/top_k.cpp b/src/frontends/gguf/src/op/top_k.cpp index 3df1fdce5aef..19b38fb82f1a 100644 --- a/src/frontends/gguf/src/op/top_k.cpp +++ b/src/frontends/gguf/src/op/top_k.cpp @@ -6,6 +6,7 @@ #include "op_table.hpp" #include "openvino/core/node_output.hpp" #include "openvino/op/constant.hpp" +#include "openvino/op/squeeze.hpp" #include "openvino/op/topk.hpp" #include "utils.hpp" @@ -20,16 +21,31 @@ OutputVector translate_top_k(const NodeContext& context) { num_inputs_check(context, 1, 1); auto input = context.get_input(0); - const int64_t k = context.get_output_shape()[context.get_output_shape().size() - 1].get_length(); - auto k_node = ov::op::v0::Constant::create(ov::element::i64, ov::Shape{}, {k}); - auto topk = std::make_shared(input, - k_node, - -1, - ov::op::v11::TopK::Mode::MAX, - ov::op::v11::TopK::SortType::SORT_VALUES, - context.get_attribute("output_type")); - - return rename_outputs_with_suffix({topk->output(1)}, context.get_name()); + + // k is the output's last-axis extent. Prefer the static value, but fall back to reading it off + // the output shape at runtime so a dynamic extent converts instead of throwing (ARGSORT derives + // its k dynamically for the same reason). + const auto& out_ps = context.get_output_shape(); + const auto rank = out_ps.rank(); + const int64_t axis = rank.is_static() ? rank.get_length() - 1 : -1; + ov::Output k_node; + if (rank.is_static() && out_ps[rank.get_length() - 1].is_static()) { + k_node = ov::op::v0::Constant::create(ov::element::i64, + ov::Shape{}, + {out_ps[rank.get_length() - 1].get_length()}); + } else { + k_node = std::make_shared( + get_dimensions(input, {static_cast(rank.is_static() ? rank.get_length() - 1 : 3)}), + ov::op::v0::Constant::create(ov::element::i64, {1}, {0})); + } + + auto indices = make_topk_indices(input, + k_node, + axis, + ov::op::v11::TopK::Mode::MAX, + context.get_attribute("output_type")); + + return rename_outputs_with_suffix({indices}, context.get_name()); } } // namespace op diff --git a/src/frontends/gguf/src/op/view.cpp b/src/frontends/gguf/src/op/view.cpp index c9c3ac3f92cf..62a0c021c77f 100644 --- a/src/frontends/gguf/src/op/view.cpp +++ b/src/frontends/gguf/src/op/view.cpp @@ -2,13 +2,19 @@ // SPDX-License-Identifier: Apache-2.0 // +#include +#include + #include "op_table.hpp" -#include "utils.hpp" +#include "openvino/frontend/exception.hpp" +#include "openvino/op/concat.hpp" #include "openvino/op/constant.hpp" +#include "openvino/op/gather.hpp" #include "openvino/op/reshape.hpp" +#include "openvino/op/shape_of.hpp" #include "openvino/op/slice.hpp" -#include -#include +#include "utils.hpp" + namespace ov { namespace frontend { namespace gguf { @@ -51,10 +57,16 @@ void place_dynamic_token_axis(std::vector & tgt, const ov::PartialShape } } // namespace +// Cases 2-5 are shared by both ingest paths: the llama.cpp cgraph decoder classifies a ggml view +// into them (see ggml-decoder.cpp::compute_op_case) and the native .gguf builder describes its own +// views the same way. Case 104 is the only VIEW case that is builder-only, and not for numbering +// reasons: it takes a second (shape-reference) input the cgraph path does not supply, so it has a +// different arity than the shared cases. See its comment below, and docs/frontend_design.md for the +// other two builder-only cases in the frontend. OutputVector translate_view(const NodeContext & context) { - num_inputs_check(context, 1, 1); + num_inputs_check(context, 1, 2); - if (context.get_attribute("op_case", 0) == 2) { + if (context.get_op_case() == 2) { auto dst_shape = context.get_output_shape().to_shape(); return rename_outputs_with_suffix( {process_view_input(context, 0, static_cast(dst_shape[2] * dst_shape[3]))}, @@ -207,6 +219,52 @@ OutputVector translate_view(const NodeContext & context) { } return rename_outputs_with_suffix({result}, context.get_name()); } + // op_case 104 (builder): layer-index slice for per-layer embedding. + // Input [1, n_layer, T, D] -> slice the layer axis (1) -> [1, 1, T, D]. + if (context.get_op_case() == 104) { + const int64_t layer_idx = context.get_attribute("layer_idx"); + auto input = context.get_input(0); + auto start = ov::op::v0::Constant::create(ov::element::i64, {1}, {layer_idx}); + auto stop = ov::op::v0::Constant::create(ov::element::i64, {1}, {layer_idx + 1}); + auto step = ov::op::v0::Constant::create(ov::element::i64, {1}, {1}); + auto axes = ov::op::v0::Constant::create(ov::element::i64, {1}, {1}); + ov::Output sliced = std::make_shared(input, start, stop, step, axes); + + // The slice comes out with the token count on the axis the per-layer tensor happens to keep it + // on (dim 1 here), because per_layer_embd is stored layer-major so the layer index can be + // sliced off axis 0. Every consumer, though, is an elementwise op against the layer's own + // activation, and OV broadcasts elementwise operands positionally -- so the two operands must + // agree on WHICH leading axis holds the tokens. That is not a fixed choice: it is [1, T, ..] + // under plain SDPA inference and [T, 1, ..] once ov::pass::SDPAToPagedAttention moves the token + // count into dim 0. Both hold the same T*D values contiguously, so when the builder supplies + // the activation as a second (shape-reference) input, reinterpret the slice into that operand's + // leading dims. Without this the multiply below broadcasts to a T x T outer product under PA. + if (context.get_input_size() > 1) { + const auto& ref = context.get_input(1); + const auto ref_rank = ref.get_partial_shape().rank(); + FRONT_END_OP_CONVERSION_CHECK(ref_rank.is_static(), + "VIEW case 104 shape reference must have a static rank"); + const auto d_ps = context.get_output_shape(); + const int64_t rank = ref_rank.get_length(); + FRONT_END_OP_CONVERSION_CHECK(d_ps.rank().is_static() && d_ps[d_ps.rank().get_length() - 1].is_static(), + "VIEW case 104 requires a static per-layer embedding width"); + const int64_t d = d_ps[d_ps.rank().get_length() - 1].get_length(); + + std::vector lead(rank - 1); + for (int64_t i = 0; i < rank - 1; ++i) { + lead[i] = i; + } + auto lead_dims = std::make_shared( + std::make_shared(ref, ov::element::i64), + ov::op::v0::Constant::create(ov::element::i64, {lead.size()}, lead), + ov::op::v0::Constant::create(ov::element::i64, {}, {0})); + auto target = std::make_shared( + ov::OutputVector{lead_dims, ov::op::v0::Constant::create(ov::element::i64, {1}, {d})}, + 0); + sliced = std::make_shared(sliced, target, false); + } + return rename_outputs_with_suffix({sliced.get_node_shared_ptr()}, context.get_name()); + } return {context.get_input(0)}; } diff --git a/src/frontends/gguf/src/op/weight.cpp b/src/frontends/gguf/src/op/weight.cpp index a1106c7ef6b9..8c5e7f6f2dc3 100644 --- a/src/frontends/gguf/src/op/weight.cpp +++ b/src/frontends/gguf/src/op/weight.cpp @@ -5,12 +5,14 @@ #include #include #include -#include "openvino/op/constant.hpp" -#include "openvino/op/reshape.hpp" +#include #include #include "node_context.hpp" #include "op_table.hpp" +#include "openvino/op/constant.hpp" +#include "openvino/op/reshape.hpp" +#include "openvino/op/unsqueeze.hpp" #include "quant/weights.hpp" #include "utils.hpp" @@ -20,13 +22,45 @@ namespace gguf { namespace op { // A GGUF weight surfaced as a node. A weight is a ggml leaf (op type "GGML_OP_NONE") that the -// decoder marks by exposing a "data" attribute (the raw weight bytes), alongside the ggml quant -// type name and the logical shape. The frontend does all dequant / repacking here, so the -// decoder never builds OV nodes itself. (Model-input leaves are also GGML_OP_NONE, but they are -// resolved to Parameters before the graph walk and never reach this translator.) +// decoder marks as a weight. Two payload shapes are supported, both dequantized here so the +// decoder never builds OV nodes itself: +// +// 1. Native .gguf builder path: the parser already extracted the weight into OpenVINO tensors +// (weight [+ scales [+ zp]]); the node carries them as attributes "gguf.blob." plus the +// quant type id "gguf_qtype" (and marker "gguf_weight"). We rebuild the make_weight_node( +// base, weights, qtypes) inputs and call that overload -- the exact dequant path the builder +// used before, unchanged numerics, and it handles fused-QKV parts / MoE experts uniformly. +// +// 2. llama.cpp cgraph path: the node carries the raw ggml bytes in "data" plus the ggml type +// name "quant_type"; make_weight_node(data, quant_type, shape) re-extracts and builds. This +// path also handles the MoE MXFP4 packed / rank>2 expert-weight layouts. +// +// (Model-input leaves are also GGML_OP_NONE, but they are resolved to Parameters before the graph +// walk and never reach this translator.) OutputVector translate_weight(const NodeContext& context) { + // Path 1: pre-extracted tensors from the native builder. + if (context.get_attribute("gguf_weight", false)) { + const std::string base = "weight"; + std::unordered_map weights; + for (const char* sub : {"weight", "scales", "zp"}) { + // scales/zp are absent for plain/symmetric types -> defaulted get_attribute (empty + // tensor) so the missing-key Any doesn't throw. + auto blob = context.get_attribute(std::string("gguf.blob.") + sub, ov::Tensor()); + if (blob) { + weights[base + "." + sub] = blob; + } + } + FRONT_END_OP_CONVERSION_CHECK(weights.count(base + ".weight"), + "GGML_OP_NONE weight leaf has no 'gguf.blob.weight' attribute"); + auto qtype = static_cast(context.get_attribute("gguf_qtype")); + std::unordered_map qtypes{{base + ".qtype", qtype}}; + auto node = make_weight_node(base, weights, qtypes); + return rename_outputs_with_suffix({node}, context.get_name()); + } + + // Path 2: raw ggml bytes from a live cgraph decoder. auto data = context.get_attribute("data"); - FRONT_END_OP_CONVERSION_CHECK(data, "GGML_OP_NONE node has no 'data' attribute; not a weight"); + FRONT_END_OP_CONVERSION_CHECK(data, "GGML_OP_NONE node has no weight payload; not a weight"); auto quant_type = context.get_attribute("quant_type"); auto shape = context.get_output_shape().to_shape(); diff --git a/src/frontends/gguf/src/op_table.cpp b/src/frontends/gguf/src/op_table.cpp index f3e6cc3f040a..8589b92418a5 100644 --- a/src/frontends/gguf/src/op_table.cpp +++ b/src/frontends/gguf/src/op_table.cpp @@ -5,6 +5,7 @@ #include "op_table.hpp" #include "openvino/op/add.hpp" +#include "openvino/op/concat.hpp" #include "openvino/op/cos.hpp" #include "openvino/op/divide.hpp" #include "openvino/op/exp.hpp" @@ -54,13 +55,17 @@ std::unordered_map get_supported_ops() { {"GGML_OP_MUL", op::translate_1to1_match_2_inputs}, {"GGML_OP_MUL_MAT", op::translate_mulmat}, {"GGML_OP_MUL_MAT_ID", op::translate_mul_mat_id}, - // A GGML_OP_NONE leaf carrying a "data" attribute is a weight (see translate_weight). + // Weights reach the frontend as GGML_OP_NONE leaves, from both ingest paths: the llama.cpp + // cgraph decoder marks one with the raw ggml bytes ("data"), the native .gguf builder with + // the tensors its parser already extracted ("gguf_weight"). translate_weight accepts either + // payload and builds the same compressed subgraph. A GGML_OP_NONE leaf with neither marker + // is a model input, resolved to a Parameter before the walk (see TranslateSession). {"GGML_OP_NONE", op::translate_weight}, {"GGML_OP_NORM", op::translate_norm}, {"GGML_OP_PAD", op::translate_pad}, {"GGML_OP_PERMUTE", op::translate_permute}, - {"GGML_OP_RESHAPE", op::translate_reshape}, {"GGML_OP_REPEAT", op::translate_repeat}, + {"GGML_OP_RESHAPE", op::translate_reshape}, {"GGML_OP_RMS_NORM", op::translate_rms_norm}, {"GGML_OP_ROPE", op::translate_rope}, {"GGML_OP_SCALE", op::translate_scale}, @@ -74,8 +79,8 @@ std::unordered_map get_supported_ops() { {"GGML_OP_SUB", op::translate_1to1_match_2_inputs}, {"GGML_OP_SUM_ROWS", op::translate_sum_rows}, {"GGML_OP_TOP_K", op::translate_top_k}, - {"GGML_OP_TRI", op::translate_tri}, {"GGML_OP_TRANSPOSE", op::translate_transpose}, + {"GGML_OP_TRI", op::translate_tri}, {"GGML_OP_VIEW", op::translate_view}, {"GGML_UNARY_OP_ELU", op::translate_unary_elu}, {"GGML_UNARY_OP_EXP", op::translate_1to1_match_1_input}, diff --git a/src/frontends/gguf/src/op_table.hpp b/src/frontends/gguf/src/op_table.hpp index 39ceb3690468..6c17d3bfb033 100644 --- a/src/frontends/gguf/src/op_table.hpp +++ b/src/frontends/gguf/src/op_table.hpp @@ -14,49 +14,76 @@ namespace op { #define GGUF_OP_CONVERTER(op) OutputVector op(const NodeContext& context) -GGUF_OP_CONVERTER(translate_add_id); -GGUF_OP_CONVERTER(translate_argsort); -GGUF_OP_CONVERTER(translate_clamp); +// Structural / memory ops. GGUF_OP_CONVERTER(translate_concat); GGUF_OP_CONVERTER(translate_cont); -GGUF_OP_CONVERTER(translate_cumsum); -GGUF_OP_CONVERTER(translate_diag); -GGUF_OP_CONVERTER(translate_div); -GGUF_OP_CONVERTER(translate_fill); -GGUF_OP_CONVERTER(translate_gated_delta_net); +GGUF_OP_CONVERTER(translate_cpy); GGUF_OP_CONVERTER(translate_get_rows); -GGUF_OP_CONVERTER(translate_im2col); -GGUF_OP_CONVERTER(translate_l2_norm); -GGUF_OP_CONVERTER(translate_norm); -GGUF_OP_CONVERTER(translate_pad); -GGUF_OP_CONVERTER(translate_repeat); -GGUF_OP_CONVERTER(translate_ssm_conv); -GGUF_OP_CONVERTER(translate_mulmat); -GGUF_OP_CONVERTER(translate_mul_mat_id); GGUF_OP_CONVERTER(translate_permute); +GGUF_OP_CONVERTER(translate_repeat); GGUF_OP_CONVERTER(translate_reshape); +GGUF_OP_CONVERTER(translate_set); +GGUF_OP_CONVERTER(translate_set_rows); +GGUF_OP_CONVERTER(translate_transpose); +GGUF_OP_CONVERTER(translate_view); + +// Normalization. +GGUF_OP_CONVERTER(translate_norm); GGUF_OP_CONVERTER(translate_rms_norm); +GGUF_OP_CONVERTER(translate_l2_norm); + +// Matmul / attention. +GGUF_OP_CONVERTER(translate_mulmat); +GGUF_OP_CONVERTER(translate_flash_attn_ext); +GGUF_OP_CONVERTER(translate_soft_max); GGUF_OP_CONVERTER(translate_rope); GGUF_OP_CONVERTER(translate_scale); -GGUF_OP_CONVERTER(translate_set); -GGUF_OP_CONVERTER(translate_sqr); -GGUF_OP_CONVERTER(translate_sqrt); + +// Gated linear units. +GGUF_OP_CONVERTER(translate_glu_geglu); +GGUF_OP_CONVERTER(translate_glu_swiglu); +GGUF_OP_CONVERTER(translate_glu_swiglu_oai); + +// MoE (mixture-of-experts) routing ops. +GGUF_OP_CONVERTER(translate_mul_mat_id); +GGUF_OP_CONVERTER(translate_add_id); +GGUF_OP_CONVERTER(translate_argsort); GGUF_OP_CONVERTER(translate_top_k); -GGUF_OP_CONVERTER(translate_tri); GGUF_OP_CONVERTER(translate_sum_rows); + +// Elementwise clamp and division. +GGUF_OP_CONVERTER(translate_clamp); +GGUF_OP_CONVERTER(translate_div); + +// Unary activations. GGUF_OP_CONVERTER(translate_unary_silu); GGUF_OP_CONVERTER(translate_unary_gelu); GGUF_OP_CONVERTER(translate_unary_gelu_quick); +GGUF_OP_CONVERTER(translate_unary_relu); +GGUF_OP_CONVERTER(translate_unary_tanh); +GGUF_OP_CONVERTER(translate_unary_sigmoid); GGUF_OP_CONVERTER(translate_unary_elu); -GGUF_OP_CONVERTER(translate_soft_max); -GGUF_OP_CONVERTER(translate_transpose); -GGUF_OP_CONVERTER(translate_view); -GGUF_OP_CONVERTER(translate_glu_swiglu); -GGUF_OP_CONVERTER(translate_glu_swiglu_oai); -GGUF_OP_CONVERTER(translate_glu_geglu); -GGUF_OP_CONVERTER(translate_set_rows); -GGUF_OP_CONVERTER(translate_cpy); -GGUF_OP_CONVERTER(translate_flash_attn_ext); + +// Unary element-wise math. +GGUF_OP_CONVERTER(translate_sqr); +GGUF_OP_CONVERTER(translate_sqrt); +GGUF_OP_CONVERTER(translate_log); +GGUF_OP_CONVERTER(translate_sin); +GGUF_OP_CONVERTER(translate_cos); +GGUF_OP_CONVERTER(translate_cumsum); + +// Matrix-shaped helpers. +GGUF_OP_CONVERTER(translate_diag); +GGUF_OP_CONVERTER(translate_tri); +GGUF_OP_CONVERTER(translate_fill); + +// Convolution-family / sequence ops. +GGUF_OP_CONVERTER(translate_im2col); +GGUF_OP_CONVERTER(translate_pad); +GGUF_OP_CONVERTER(translate_ssm_conv); +GGUF_OP_CONVERTER(translate_gated_delta_net); + +// A GGML_OP_NONE leaf carrying a "data" attribute -> dequantized weight node (cgraph path). GGUF_OP_CONVERTER(translate_weight); } // namespace op diff --git a/src/frontends/gguf/src/pass/make_stateful.cpp b/src/frontends/gguf/src/pass/make_stateful.cpp new file mode 100644 index 000000000000..85c6df83cb3c --- /dev/null +++ b/src/frontends/gguf/src/pass/make_stateful.cpp @@ -0,0 +1,216 @@ +// Copyright (C) 2018-2026 Intel Corporation +// SPDX-License-Identifier: Apache-2.0 +// + +#include "openvino/frontend/gguf/make_stateful.hpp" + +#include +#include +#include + +#include "openvino/core/graph_util.hpp" +#include "openvino/frontend/gguf/set_rows_op.hpp" +#include "openvino/op/assign.hpp" +#include "openvino/op/concat.hpp" +#include "openvino/op/constant.hpp" +#include "openvino/op/gather.hpp" +#include "openvino/op/parameter.hpp" +#include "openvino/op/read_value.hpp" +#include "openvino/op/reshape.hpp" +#include "openvino/op/result.hpp" +#include "openvino/op/util/variable.hpp" + +namespace ov::frontend::gguf::pass { + +namespace { + +std::shared_ptr find_param(const std::shared_ptr& model, const std::string& name) { + for (const auto& p : model->get_parameters()) { + if (p->get_friendly_name() == name || p->output(0).get_names().count(name)) { + return p; + } + } + return nullptr; +} + +// The axis the cache grows along. An un-preallocated cache Parameter states it by construction: it +// is the one dynamic axis (the token count), every other being a static batch / head / head-size. +// A caller that preallocates the cache must say which axis it is, since none is dynamic then. +int64_t resolve_append_axis(const ov::PartialShape& ps, const std::string& cache_name, int64_t requested) { + const int64_t rank = ps.rank().get_length(); + if (requested >= 0) { + OPENVINO_ASSERT(requested < rank, + "[gguf] MakeStateful: append axis ", + requested, + " is out of range for cache '", + cache_name, + "' of shape ", + ps); + return requested; + } + int64_t axis = -1; + for (int64_t i = 0; i < rank; ++i) { + if (ps[i].is_dynamic()) { + OPENVINO_ASSERT(axis < 0, + "[gguf] MakeStateful: cache '", + cache_name, + "' has shape ", + ps, + " with more than one dynamic axis, so its token axis cannot be inferred; " + "construct the pass with an explicit append_axis"); + axis = i; + } + } + OPENVINO_ASSERT(axis >= 0, + "[gguf] MakeStateful: cache '", + cache_name, + "' has the fully static shape ", + ps, + ", so its token axis cannot be inferred; construct the pass with an explicit append_axis"); + return axis; +} + +} // namespace + +bool MakeStateful::run_on_model(const std::shared_ptr& model) { + // beam_idx reorders the past cache along the batch axis for beam search. With batch 1 / + // beam_idx [0] the Gather is an identity, but emitting it is what lets CPU's + // stateful_sdpa_fusion match. + // + // It belongs to the STATE, so this pass owns it: it is a beam-search index into an OpenVINO + // cache, which ggml has no equivalent of, so no decoder should declare it -- a decoder that did + // would give the stateless graph an input with no consumer, and the two decoders different + // stateless IO. Created here, next to its only consumer (the Gather below). A model that + // already has one (a caller that declared it, or a second run of this pass) keeps it. + auto beam_idx = find_param(model, m_beam_idx_name); + const bool created_beam_idx = beam_idx == nullptr; + if (created_beam_idx) { + beam_idx = std::make_shared(ov::element::i32, ov::PartialShape{ov::Dimension()}); + beam_idx->set_friendly_name(m_beam_idx_name); + beam_idx->output(0).set_names({m_beam_idx_name}); + } + + // Only a SetRows writing into a model Parameter is a cache write; the rest (e.g. MoE routing + // writes) are left to the default stateless lowering that runs after this pass. Collect first, + // then rewrite, so the graph is not mutated while being walked. + std::vector> cache_writes; + for (const auto& node : model->get_ops()) { + auto set_rows = ov::as_type_ptr(node); + if (!set_rows) { + continue; + } + auto dst = ov::as_type_ptr(set_rows->input_value(2).get_node_shared_ptr()); + if (dst && !m_skip_caches.count(dst->get_friendly_name())) { + cache_writes.push_back(set_rows); + } + } + if (cache_writes.empty()) { + return false; + } + + ov::ParameterVector params_to_remove; + ov::ResultVector results_to_remove; + ov::SinkVector new_sinks; + + for (const auto& set_rows : cache_writes) { + auto new_rows = set_rows->input_value(0); + auto cache_param = ov::as_type_ptr(set_rows->input_value(2).get_node_shared_ptr()); + const auto& cache_name = cache_param->get_friendly_name(); + const auto& ps = cache_param->get_partial_shape(); + const auto et = cache_param->get_element_type(); + OPENVINO_ASSERT(ps.rank().is_static(), + "[gguf] MakeStateful requires a static cache rank, got ", + ps, + " for '", + cache_name, + "'"); + const int64_t axis = resolve_append_axis(ps, cache_name, m_append_axis); + + // The state holds however many tokens have accumulated, so the append axis is dynamic on the + // Variable and its initial extent is 0 (no past on the first inference). Every other axis + // keeps the Parameter's declared dimension and so must be static to build the init constant. + ov::PartialShape var_shape = ps; + var_shape[axis] = ov::Dimension::dynamic(); + auto var = std::make_shared(ov::op::util::VariableInfo{var_shape, et, cache_name}); + + ov::Shape init_shape; + for (int64_t i = 0; i < ps.rank().get_length(); ++i) { + if (i == axis) { + init_shape.push_back(0); + continue; + } + OPENVINO_ASSERT(ps[i].is_static(), + "[gguf] MakeStateful requires static non-token cache dims, got ", + ps, + " for '", + cache_name, + "'"); + init_shape.push_back(static_cast(ps[i].get_length())); + } + // Empty init: required, not cosmetic -- CPU's MemoryInputSDPA aborts on a MemoryInput with + // zero parent edges (see the header note). + auto init = ov::op::v0::Constant::create(et, init_shape, std::vector{}); + auto read_value = std::make_shared(init, var); + + // The SetRows placeholder presents the new rows flattened to [.., 1, tokens, row_size] (see + // translate_set_rows), which need not be the cache's own split of those same elements -- e.g. + // a [1, tokens, n_head_kv, head_size] cache receives [1, 1, tokens, n_head_kv*head_size]. So + // re-split them against the cache layout before the Concat: the token axis is -1, the axes + // after it take the cache's static dims, and the axes before it are copied from the incoming + // data (special_zero's 0) rather than pinned to literals, which is what keeps this valid in + // the token-major layout ov::pass::SDPAToPagedAttention establishes. + std::vector split_pattern; + for (int64_t i = 0; i < ps.rank().get_length(); ++i) { + split_pattern.push_back(i < axis ? 0 : (i == axis ? -1 : ps[i].get_length())); + } + new_rows = std::make_shared( + new_rows, + ov::op::v0::Constant::create(ov::element::i64, {split_pattern.size()}, split_pattern), + true); + + // Reorder the past by beam_idx before appending, so each beam continues its own history. + auto axis0 = ov::op::v0::Constant::create(ov::element::i64, ov::Shape{}, {0}); + auto past = std::make_shared(read_value, beam_idx, axis0); + auto concat = std::make_shared(ov::OutputVector{past, new_rows}, axis); + concat->set_friendly_name(set_rows->get_friendly_name()); + new_sinks.push_back(std::make_shared(concat, var)); + + // The stateless graph returns each updated cache as a Result; in the stateful form the Assign + // sink above takes that role, so those Results go. Identify them as the Results reading THIS + // write -- not by matching the cache's name, which only happens to work while the builder + // names a cache's write after the cache itself. Collect them before replace_node, while the + // SetRows is still the node they read. + for (const auto& consumer : set_rows->output(0).get_target_inputs()) { + if (auto r = ov::as_type_ptr(consumer.get_node()->shared_from_this())) { + results_to_remove.push_back(r); + } + } + + // Every remaining consumer of the write (attention's read of the cache) now reads the grown + // Concat instead. + ov::replace_node(set_rows, concat); + model->add_variables({var}); + + params_to_remove.push_back(cache_param); + } + + // The cache is no longer part of the model's IO: its Parameter is now a ReadValue and its Result + // an Assign sink. Remove the Results first so the Parameters have no consumers left. + for (const auto& r : results_to_remove) { + model->remove_result(r); + } + model->add_sinks(new_sinks); + // Only now, having actually built the Gathers that read it -- so a pass that converted nothing + // adds no input. + if (created_beam_idx) { + model->add_parameters({beam_idx}); + } + for (const auto& p : params_to_remove) { + model->remove_parameter(p); + } + + model->validate_nodes_and_infer_types(); + return true; +} + +} // namespace ov::frontend::gguf::pass diff --git a/src/frontends/gguf/src/quant/gguf.hpp b/src/frontends/gguf/src/quant/gguf.hpp index e0b0a6b03360..3f7aa7986c42 100644 --- a/src/frontends/gguf/src/quant/gguf.hpp +++ b/src/frontends/gguf/src/quant/gguf.hpp @@ -85,16 +85,6 @@ struct gguf_tensor { using GGUFMetaData = std::variant, std::vector>; -// GGUFLoad result: (metadata, tensor arrays, qtype map, mmap, quant_buf). -// - mmap: must stay alive while arrays tensors are used (non-quantized tensors are mmap views). -// - quant_buf: single AlignedBuffer holding all repacked quantized weight/scale/bias data; -// tensors in `arrays` for quantized weights are SharedBuffer slices into this buffer. -using GGUFLoad = std::tuple, - std::unordered_map, - std::unordered_map, - std::shared_ptr, - std::shared_ptr>; - // Fill pre-allocated i4 weights (u32-packed, XORed for i4 sign) and f16 scales from a // Q4_0 tensor. No bias: Q4_0 is symmetric (zp = -8*scale is implicit, not stored). void gguf_fill_q4_0(const gguf_tensor& tensor, ov::Tensor& weights, ov::Tensor& scales); @@ -112,8 +102,8 @@ void gguf_fill_asym(const gguf_tensor& tensor, ov::Tensor& weights, ov::Tensor& // Fill pre-allocated f4e2m1 weights and f8e8m0 scales from an MXFP4 GGUF tensor. void gguf_fill_mxfp4(const gguf_tensor& tensor, ov::Tensor& weights, ov::Tensor& scales); -// Fill pre-allocated u2 weights, f16 scales and zero-points from a Q2_0 (ternary) tensor. -// The zero-point is the constant 1 for every block. +// Fill pre-allocated u2 weights, f16 scales and u8 zero-points from a Q2_0 (ternary) tensor. +// The zero-point is the constant 1 for every block: value = (code - 1) * scale. void gguf_fill_q2_0(const gguf_tensor& tensor, ov::Tensor& weights, ov::Tensor& scales, ov::Tensor& zp); // Fused bit-exact ggml dequant + channel-wise Q8_0_C requant for the token_embd/output/Q6_K/Q5_K @@ -132,15 +122,12 @@ void dequant_row_q4_k_f32_for_test(const uint8_t* row, size_t cols, float* y); void dequant_row_q5_k_f32_for_test(const uint8_t* row, size_t cols, float* y); void dequant_row_q6_k_f32_for_test(const uint8_t* row, size_t cols, float* y); -// Parse a GGUF file: returns (metadata, tensors-by-ggml-name, qtype map, mmap, quant_buf). -// Non-quantized tensors are zero-copy views into the mmap (mmap must outlive arrays use). -// Quantized tensors are SharedBuffer slices of a single AlignedBuffer (quant_buf) so all -// repacked weight/scale/bias data lives in one allocation (IR-frontend pattern). -GGUFLoad get_gguf_data(const std::string& file); - // Extract the architecture config (architecture, layer_num, head_num, head_size, // head_num_kv, hidden_size, max_position_embeddings, rms_norm_eps, rope_freq_base, // file_type) from parsed metadata. std::map config_from_meta(const std::unordered_map& metadata); +// Reverse of the GGML dimension order (GGUF stores dims fastest-first). +ov::Shape get_shape(const gguf_tensor& tensor); + } // namespace ov::frontend::gguf diff --git a/src/frontends/gguf/src/quant/gguf_quants.cpp b/src/frontends/gguf/src/quant/gguf_quants.cpp index c89b8692f9d7..d86aaa5c1739 100644 --- a/src/frontends/gguf/src/quant/gguf_quants.cpp +++ b/src/frontends/gguf/src/quant/gguf_quants.cpp @@ -640,8 +640,10 @@ void fill_q8_k(const gguf_tensor& tensor, ov::Tensor& weights_arr, ov::Tensor& s }); } -// Block = |f16 d|u2 qs[64]| (18 bytes / 64 weights). ggml packs the codes 4 per byte LSB-first, -// the same order an OpenVINO u2 Constant reads, so the code bytes are copied verbatim. +// Q2_0 ternary: block = |f16 d|u2 qs[64]| (18 bytes / 64 weights), value = (code - 1) * d with +// code in [0..3] -> {-1, 0, +1, +2}. ggml packs the codes 4 per byte LSB-first +// (dequantize_row_q2_0: `(qs[j/4] >> ((j%4)*2)) & 3`), which is exactly the order OpenVINO's u2 +// Constant reads, so the 16 code bytes are copied verbatim. The zero-point is the constant 1. void gguf_fill_q2_0(const gguf_tensor& tensor, ov::Tensor& weights_arr, ov::Tensor& scales_arr, ov::Tensor& zp_arr) { const uint64_t bytes_per_block = 18; const uint64_t bytes_per_block_codes = 16; diff --git a/src/frontends/gguf/src/quant/weights.cpp b/src/frontends/gguf/src/quant/weights.cpp index 1599c03c43aa..42a374a93dc1 100644 --- a/src/frontends/gguf/src/quant/weights.cpp +++ b/src/frontends/gguf/src/quant/weights.cpp @@ -10,6 +10,7 @@ #include "weights.hpp" +#include #include #include #include @@ -35,12 +36,45 @@ namespace { const ov::Tensor& get(const std::unordered_map& weights, const std::string& key) { auto it = weights.find(key); - OPENVINO_ASSERT(it != weights.end(), "[ggml] missing weight tensor: ", key); + OPENVINO_ASSERT(it != weights.end(), "[GGUF] missing weight tensor: ", key); return it->second; } +// Copy rows [r0, r1) out of a 2D tensor. Rows are block-independent in every GGUF quant layout +// (a full row's worth of blocks is contiguous), so a fused attn_qkv weight can be split into +// q/k/v by a plain byte-range row copy without touching the quant blocks. +ov::Tensor slice_rows(const ov::Tensor& t, size_t r0, size_t r1) { + const auto& s = t.get_shape(); + OPENVINO_ASSERT(s.size() == 2 && r1 <= s[0] && r0 <= r1, "[GGUF] bad row slice"); + ov::Shape out_shape{r1 - r0, s[1]}; + ov::Tensor out(t.get_element_type(), out_shape); + const size_t row_bytes = t.get_byte_size() / s[0]; + std::memcpy(out.data(), static_cast(t.data()) + r0 * row_bytes, (r1 - r0) * row_bytes); + return out; +} + +// Gather rows in a repeating per-block pattern: for every `block` consecutive rows, take +// [0, take) into the result. qwen35's attn_q interleaves query and gate per head as +// [q_h0 | gate_h0 | q_h1 | gate_h1 | ...], so the query is gather(block=2*head_dim, +// take=head_dim, offset=0) and the gate the same with offset=head_dim. Like slice_rows this +// works on raw row bytes, which is safe for the packed types because a quantization block +// never spans two rows. +ov::Tensor gather_rows_strided(const ov::Tensor& t, size_t block, size_t take, size_t offset) { + const auto& s = t.get_shape(); + OPENVINO_ASSERT(s.size() == 2 && block > 0 && offset + take <= block && s[0] % block == 0, + "[GGUF] bad strided row gather"); + const size_t n_blocks = s[0] / block; + ov::Tensor out(t.get_element_type(), ov::Shape{n_blocks * take, s[1]}); + const size_t row_bytes = t.get_byte_size() / s[0]; + const auto* src = static_cast(t.data()); + auto* dst = static_cast(out.data()); + for (size_t b = 0; b < n_blocks; ++b) { + std::memcpy(dst + b * take * row_bytes, src + (b * block + offset) * row_bytes, take * row_bytes); + } + return out; +} + -// Shared shape helpers for grouped weight layouts. See make_int8 comment for why we keep // all leading dims separate rather than flattening: the trailing Reshape must be // (orig_rank+1)D -> orig_rank for the CompressedWeightsBlock matcher to fire. ov::Shape grouped_weight_shape(const ov::Shape& orig, size_t num_groups, size_t group_size) { @@ -399,6 +433,17 @@ bool needs_q8_0_c_requant(const std::string& name, gguf_tensor_type qtype) { } // namespace +ov::element::Type gguf_zero_point_type(const std::string& name, gguf_tensor_type qtype) { + // The CPU compressed-FullyConnected fast path only folds the dequant when the zero-point is an + // INTEGER constant; a fractional f16 one leaves a ~2x slower kernel. Q4_K carries the matmul + // weights of modern models and Q2_0's zp is the exact integer 1, so both use u8. The others + // keep a faithful f16 zp: their zp = min/scale can exceed u8 range, and rounding it injects + // error into every weight. Tensors that are requantized to Q8_0_C are excluded -- their dequant + // feeds the channel-wise path, not a compressed FC. + const bool integer_zp = (qtype == GGUF_TYPE_Q4_K || qtype == GGUF_TYPE_Q2_0); + return (integer_zp && !needs_q8_0_c_requant(name, qtype)) ? ov::element::u8 : ov::element::f16; +} + std::shared_ptr make_weight_node(const std::string& base, const std::unordered_map& weights, const std::unordered_map& qtypes) { @@ -478,16 +523,99 @@ gguf_tensor_type gguf_type_from_name(const std::string& quant_type) { ch = static_cast(std::toupper(static_cast(ch))); } auto it = names.find(key); - OPENVINO_ASSERT(it != names.end(), "[ggml] unsupported weight quant type: ", quant_type); + OPENVINO_ASSERT(it != names.end(), "[GGUF] unsupported weight quant type: ", quant_type); return it->second; } +std::array split_fused_qkv_extracted( + const std::string& base, + const std::unordered_map& weights, + const std::unordered_map& qtypes, + size_t n_q, + size_t n_k, + size_t n_v) { + gguf_tensor_type qtype = GGUF_TYPE_F16; + if (auto it = qtypes.find(base + ".qtype"); it != qtypes.end()) { + qtype = it->second; + } + const bool has_scales = qtype == GGUF_TYPE_Q4_0 || qtype == GGUF_TYPE_Q4_1 || qtype == GGUF_TYPE_Q4_K || + qtype == GGUF_TYPE_Q5_0 || qtype == GGUF_TYPE_Q5_1 || qtype == GGUF_TYPE_Q8_0 || + qtype == GGUF_TYPE_Q2_K || qtype == GGUF_TYPE_Q3_K || qtype == GGUF_TYPE_Q5_K || + qtype == GGUF_TYPE_Q6_K || qtype == GGUF_TYPE_Q2_0; + const bool has_zp = qtype == GGUF_TYPE_Q4_1 || qtype == GGUF_TYPE_Q4_K || qtype == GGUF_TYPE_Q5_K || + qtype == GGUF_TYPE_Q5_1 || qtype == GGUF_TYPE_Q2_K || qtype == GGUF_TYPE_Q2_0; + + const ov::Tensor& w = get(weights, base + ".weight"); + const size_t total_rows = w.get_shape()[0]; + OPENVINO_ASSERT(n_q + n_k + n_v == total_rows, "[GGUF] fused qkv row mismatch for ", base); + + const std::array, 3> ranges = {std::make_pair(size_t(0), n_q), + std::make_pair(n_q, n_q + n_k), + std::make_pair(n_q + n_k, total_rows)}; + const std::array parts = {base + ".q", base + ".k", base + ".v"}; + + std::array out; + for (size_t i = 0; i < 3; ++i) { + const auto [r0, r1] = ranges[i]; + out[i].qtype = qtype; + out[i].extracted[parts[i] + ".weight"] = slice_rows(w, r0, r1); + if (has_scales) { + out[i].extracted[parts[i] + ".scales"] = slice_rows(get(weights, base + ".scales"), r0, r1); + } + if (has_zp) { + out[i].extracted[parts[i] + ".zp"] = slice_rows(get(weights, base + ".zp"), r0, r1); + } + } + return out; +} + +// qwen35: attn_q packs the query and the attention output gate interleaved per head, as +// [q_h0 | gate_h0 | q_h1 | gate_h1 | ...] with a stride of 2*head_dim rows. De-interleave it +// into two plain weights so the graph sees ordinary projections. Returns {query, gate}. +std::array split_interleaved_q_gate(const std::string& base, + const std::unordered_map& weights, + const std::unordered_map& qtypes, + size_t head_dim) { + gguf_tensor_type qtype = GGUF_TYPE_F16; + if (auto it = qtypes.find(base + ".qtype"); it != qtypes.end()) { + qtype = it->second; + } + const bool has_scales = qtype == GGUF_TYPE_Q4_0 || qtype == GGUF_TYPE_Q4_1 || qtype == GGUF_TYPE_Q4_K || + qtype == GGUF_TYPE_Q5_0 || qtype == GGUF_TYPE_Q5_1 || qtype == GGUF_TYPE_Q8_0 || + qtype == GGUF_TYPE_Q2_K || qtype == GGUF_TYPE_Q3_K || qtype == GGUF_TYPE_Q5_K || + qtype == GGUF_TYPE_Q6_K || qtype == GGUF_TYPE_Q2_0; + const bool has_zp = qtype == GGUF_TYPE_Q4_1 || qtype == GGUF_TYPE_Q4_K || qtype == GGUF_TYPE_Q5_K || + qtype == GGUF_TYPE_Q5_1 || qtype == GGUF_TYPE_Q2_K || qtype == GGUF_TYPE_Q2_0; + + const ov::Tensor& w = get(weights, base + ".weight"); + const size_t block = 2 * head_dim; + OPENVINO_ASSERT(w.get_shape()[0] % block == 0, "[GGUF] interleaved q/gate row mismatch for ", base); + + const std::array parts = {base + ".q", base + ".gate"}; + const std::array offsets = {0, head_dim}; + + std::array out; + for (size_t i = 0; i < 2; ++i) { + out[i].qtype = qtype; + out[i].extracted[parts[i] + ".weight"] = gather_rows_strided(w, block, head_dim, offsets[i]); + if (has_scales) { + out[i].extracted[parts[i] + ".scales"] = + gather_rows_strided(get(weights, base + ".scales"), block, head_dim, offsets[i]); + } + if (has_zp) { + out[i].extracted[parts[i] + ".zp"] = + gather_rows_strided(get(weights, base + ".zp"), block, head_dim, offsets[i]); + } + } + return out; +} + std::shared_ptr make_weight_node(const ov::Tensor& data, const std::string& quant_type, const ov::Shape& logical_shape, const std::string& name) { OPENVINO_ASSERT(logical_shape.size() == 2, - "[ggml] weight logical shape must be 2D [rows, cols], got rank ", + "[GGUF] weight logical shape must be 2D [rows, cols], got rank ", logical_shape.size()); const uint64_t rows = logical_shape[0]; const uint64_t cols = logical_shape[1]; @@ -533,8 +661,7 @@ std::shared_ptr make_weight_node(const ov::Tensor& data, // they are not perf-critical here, and their zp = -min/scale can fall outside u8 range. The // requant path (token_embd/output) also keeps f16 -- its dequant feeds channel-wise Q8_0_C. const bool requant = needs_q8_0_c_requant(name, qtype); - const ov::element::Type zp_type = - (!requant && (qtype == GGUF_TYPE_Q4_K || qtype == GGUF_TYPE_Q2_0)) ? ov::element::u8 : ov::element::f16; + const ov::element::Type zp_type = gguf_zero_point_type(name, qtype); // K-quant requant sources: the fused dequant -> Q8_0_C streams from the raw bytes, so skip the // full-tensor gguf_fill_* extraction below (it would be discarded) and return before the switch. @@ -631,7 +758,7 @@ std::shared_ptr make_weight_node(const ov::Tensor& data, break; } default: - OPENVINO_THROW("[ggml] unsupported weight quant type: ", quant_type); + OPENVINO_THROW("[GGUF] unsupported weight quant type: ", quant_type); } // Non-K requant sources (e.g. an F16 / Q4_0 / Q8_0 token_embd or output): the K-quant fast path diff --git a/src/frontends/gguf/src/quant/weights.hpp b/src/frontends/gguf/src/quant/weights.hpp index fa7cac374c6a..5c6486d6eabb 100644 --- a/src/frontends/gguf/src/quant/weights.hpp +++ b/src/frontends/gguf/src/quant/weights.hpp @@ -4,6 +4,7 @@ #pragma once +#include #include #include #include @@ -16,6 +17,10 @@ class Node; namespace ov::frontend::gguf { +// Element type of the zero-point constant for an asymmetric quantized weight. Both ingest +// paths must agree on this: it decides whether the CPU folds the dequant into the MatMul. +ov::element::Type gguf_zero_point_type(const std::string& name, gguf_tensor_type qtype); + // Build the OpenVINO node for a GGUF weight with base name `base` (the tensor name without // the trailing ".weight", e.g. "blk.0.attn_q" or "token_embd"). Quantized weights become a // low-bitness compressed subgraph (u4/u8 weights + zero-point + f16 scale, Convert -> @@ -48,4 +53,32 @@ std::shared_ptr make_weight_node(const ov::Tensor& data, // Map a ggml quant type name (e.g. "Q4_K") to its gguf_tensor_type id. Throws if unknown. gguf_tensor_type gguf_type_from_name(const std::string& quant_type); +// One split part of a fused attn_qkv weight: the extracted tensors keyed as ".weight" +// [+ ".scales" [+ ".zp"]] plus the shared quant type. Used by the GGUF builder to emit a +// GGML_OP_NONE weight leaf per q/k/v part (routing them through translate_weight like any other +// weight) instead of building the decompression nodes eagerly. +struct FusedQkvPart { + std::unordered_map extracted; + gguf_tensor_type qtype = GGUF_TYPE_F16; +}; + +// Row-slice a fused `` attn_qkv weight into q/k/v extracted-tensor sub-maps (no OV nodes). +// The returned parts' tensors are keyed ".q.weight"/".scales"/".zp" etc. Same slicing as +// make_fused_qkv_weights, but returns the extracted payload for GGML_OP_NONE emission. +std::array split_fused_qkv_extracted( + const std::string& base, + const std::unordered_map& weights, + const std::unordered_map& qtypes, + size_t n_q, + size_t n_k, + size_t n_v); + +// De-interleave a qwen35 `` attn_q weight, which packs the query and the attention output +// gate per head as [q_h0 | gate_h0 | q_h1 | gate_h1 | ...], into two plain projections. +// Returns {query, gate}, keyed ".q.*" and ".gate.*". +std::array split_interleaved_q_gate(const std::string& base, + const std::unordered_map& weights, + const std::unordered_map& qtypes, + size_t head_dim); + } // namespace ov::frontend::gguf diff --git a/src/frontends/gguf/src/translate_session.cpp b/src/frontends/gguf/src/translate_session.cpp index 3e37dd6e9871..a2f2794411aa 100644 --- a/src/frontends/gguf/src/translate_session.cpp +++ b/src/frontends/gguf/src/translate_session.cpp @@ -4,15 +4,25 @@ #include "translate_session.hpp" +#include #include #include #include #include +#include + +#include "input_model.hpp" +#include "node_context.hpp" +#include "openvino/core/graph_util.hpp" #include "openvino/core/node.hpp" +#include "openvino/core/rt_info/weightless_caching_attributes.hpp" +#include "openvino/frontend/gguf/make_stateful.hpp" #include "openvino/op/add.hpp" #include "openvino/op/broadcast.hpp" #include "openvino/op/concat.hpp" +#include "openvino/op/constant.hpp" #include "openvino/op/convert.hpp" +#include "openvino/op/convert_like.hpp" #include "openvino/op/cos.hpp" #include "openvino/op/divide.hpp" #include "openvino/op/gather.hpp" @@ -27,11 +37,8 @@ #include "openvino/op/strided_slice.hpp" #include "openvino/op/transpose.hpp" #include "openvino/pass/constant_folding.hpp" - -#include "input_model.hpp" -#include "node_context.hpp" -#include "openvino/core/rt_info/weightless_caching_attributes.hpp" #include "pass/lower_set_rows_stateless.hpp" +#include "transformations/common_optimizations/nop_elimination.hpp" #include "transformations/fp16_compression/mark_decompression_convert_constant_folding.hpp" #include "transformations/op_conversions/convert_convertlike.hpp" #include "utils.hpp" @@ -68,7 +75,12 @@ void add_sliced_mask(TensorMap& tensor_map) { create_sliced_mask("self_kq_mask_swa", "KQ_mask_swa_sliced"); } -void add_rope_sin_cos(TensorMap& tensor_map, const RopeConfig& rope_config) { +void add_rope_sin_cos(TensorMap& tensor_map, GgufDecoder& gguf_model_decoder) { + // A decoder bound to a full LLM graph exposes "rope_config"; a decoder wrapping a bare op / + // small cgraph (a single-op test) has no such attribute -> default RopeConfig (n_dims == 0, + // "no RoPE") so the shared table is skipped and the op falls back to its own sin/cos. + const auto rope_config_any = gguf_model_decoder.get_attribute("rope_config"); + const auto rope_config = rope_config_any.empty() ? RopeConfig{} : rope_config_any.as(); // n_dims == 0 means the model uses no RoPE; per_op means each ROPE op builds its own sin/cos // (e.g. gemma4 where SWA and global layers differ), so skip the shared table entirely. if (tensor_map.find("inp_pos") == tensor_map.end() || rope_config.n_dims == 0 || rope_config.per_op) { @@ -91,9 +103,9 @@ void add_rope_sin_cos(TensorMap& tensor_map, const RopeConfig& rope_config) { } // Create common patterns -void preprocess(TensorMap& tensor_map, const RopeConfig& rope_config) { +void preprocess(TensorMap& tensor_map, GgufDecoder& gguf_model_decoder) { add_sliced_mask(tensor_map); - add_rope_sin_cos(tensor_map, rope_config); + add_rope_sin_cos(tensor_map, gguf_model_decoder); } } // namespace @@ -121,26 +133,42 @@ std::shared_ptr TranslateSession::translate_graph(const frontend::InputMo std::shared_ptr resulting_model; const auto& gguf_model = std::dynamic_pointer_cast(input_model); + std::shared_ptr gguf_model_decoder = gguf_model->get_model_decoder(); + + // An auxiliary input Parameter whose only consumer may be created by a later normalization pass, + // after the unused-Parameter pruning below. Track them so pruning never drops one for lack of a + // consumer at translate time. A pass that ends up not consuming one leaves it as a dangling + // input, which later constant folding removes. + std::set deferred_use_params; - for (const auto& it : gguf_model->get_model_inputs()) { - if (auto param = std::dynamic_pointer_cast(it.second)) { - params.push_back(param); + for (const auto& it : gguf_model_decoder->get_model_inputs()) { + params.push_back(std::dynamic_pointer_cast(it.second)); + (*tensor_map)[it.first] = it.second; + } + + for (const auto& it : gguf_model_decoder->get_model_extra_inputs()) { + if (auto p = std::dynamic_pointer_cast(it.second)) { + params.push_back(p); + deferred_use_params.insert(p.get()); } (*tensor_map)[it.first] = it.second; } - // Weights are not seeded here: a weight is visited as a regular "GGML_OP_NONE" node (a ggml - // leaf carrying a "data" attribute) in visit_subgraph, and translate_weight writes its - // dequantized node into the tensor map under the weight name, before the consuming op is - // visited (the cgraph is topologically ordered). + // Weights are not seeded here: every decoder surfaces them as "GGML_OP_NONE" leaves that + // translate_weight turns into a compressed subgraph during the walk below, which keeps them + // lazy (never materialized to f32) and keeps one weight-loading path for both ingest paths. auto node_visitor = [&](std::shared_ptr decoder) { auto operation_type = decoder->get_op_type(); if (operation_type == "GGML_OP_NONE") { - // A GGML_OP_NONE leaf is a weight only if the decoder exposes its raw bytes via the - // "data" attribute; otherwise it is a model-input leaf (already seeded as a Parameter + // A GGML_OP_NONE leaf is a weight if the decoder marks it as one: either the native + // builder's pre-extracted payload (bool "gguf_weight") or the cgraph decoder's raw + // bytes ("data"). Otherwise it is a model-input leaf (already seeded as a Parameter // above) and there is nothing to translate. - if (!decoder->get_attribute("data").is()) { + const bool is_builder_weight = decoder->get_attribute("gguf_weight").is() && + decoder->get_attribute("gguf_weight").as(); + const bool is_cgraph_weight = decoder->get_attribute("data").is(); + if (!is_builder_weight && !is_cgraph_weight) { return; } } @@ -178,10 +206,10 @@ std::shared_ptr TranslateSession::translate_graph(const frontend::InputMo // the model uses a shared rope table (n_dims != 0, not per-op). For a bare op / small cgraph // (no rope_config -> default n_dims == 0, no mask/pos inputs) both no-op, and the ROPE/attention // translators fall back to building their own -- so there is no separate "naive" mode. - preprocess(*tensor_map, gguf_model->get_rope_config()); - gguf_model->visit_subgraph(node_visitor); + preprocess(*tensor_map, *gguf_model_decoder); + gguf_model_decoder->visit_subgraph(node_visitor); - for (const auto& name : gguf_model->get_model_output_names()) { + for (const auto& name : gguf_model_decoder->get_model_output_names()) { FRONT_END_GENERAL_CHECK(tensor_map->find(name) != tensor_map->end(), "Output name not found in tensor map: ", name); @@ -192,13 +220,15 @@ std::shared_ptr TranslateSession::translate_graph(const frontend::InputMo ov::ParameterVector used_params; for (const auto& param : params) { - if (!param->output(0).get_target_inputs().empty()) { + // Keep a Parameter if it currently feeds something, OR if its consumer is created by a + // later normalization pass. + if (!param->output(0).get_target_inputs().empty() || deferred_use_params.count(param.get())) { used_params.push_back(param); } } resulting_model = std::make_shared(results, used_params); - apply_transformations(resulting_model); + resulting_model = apply_transformations(resulting_model); // Set WeightlessCacheAttribute on large constants to avoid unnecessary memory copies // in the NPUW plugin. Without this attribute, NPUW's LazyTensor constructor @@ -216,7 +246,8 @@ std::shared_ptr TranslateSession::translate_graph(const frontend::InputMo size_t offset = 0; for (auto& node : resulting_model->get_ordered_ops()) { if (auto cnst = ov::as_type_ptr(node); - cnst && cnst->get_byte_size() / cnst->get_element_type().size() >= 16) { + cnst && cnst->get_element_type().size() > 0 && + cnst->get_byte_size() / cnst->get_element_type().size() >= 16) { auto& rt_info = cnst->get_rt_info(); if (rt_info.find(ov::WeightlessCacheAttribute::get_type_info_static()) == rt_info.end()) { rt_info[ov::WeightlessCacheAttribute::get_type_info_static()] = @@ -232,17 +263,23 @@ std::shared_ptr TranslateSession::apply_transformations(std::shared_ptr(); - // Caller-registered transformation extensions run first. A SetRows-lowering extension (e.g. - // the backend's stateful lowering) consumes the KV-cache SetRows ops here; the built-in stateless - // lowering below then only fires on the ops left untouched. With no extension registered, the - // stateless lowering handles every SetRows op -- so a plain convert() yields the - // llama.cpp-faithful stateless model. + // Caller-registered transformation extensions run first, which is what makes execution mode a + // caller concern rather than a frontend one: an extension that lowers SetRows itself -- e.g. + // ov::frontend::gguf::pass::MakeStateful, or a backend's own variant -- consumes the KV-cache + // SetRows ops here, and the built-in stateless lowering below then only fires on the ops left + // untouched (MoE routing writes and the like). With no extension registered, the stateless + // lowering handles every SetRows op, so a plain convert() yields the stateless model. for (const auto& ext : m_transformation_extensions) { ext->register_pass(manager); } manager.register_pass(); - manager.register_pass(); + manager.register_pass(); + // The lowered ConvertLikes are frequently no-ops (k/v already share q's precision). Drop them: + // a same-type Convert on an SDPA k/v input is invisible to the plugins but breaks + // StateManagementPattern, which admits no Convert between the KV-cache Concat and SDPA, and so + // silently disables the PagedAttention backend for every GGUF model. + manager.register_pass(); manager.run_passes(model); return model; } diff --git a/src/frontends/gguf/src/utils.cpp b/src/frontends/gguf/src/utils.cpp index 32f99fd37b14..5387e48a593b 100644 --- a/src/frontends/gguf/src/utils.cpp +++ b/src/frontends/gguf/src/utils.cpp @@ -34,6 +34,18 @@ void num_inputs_check(const NodeContext& context, size_t min_inputs, size_t max_ FRONT_END_OP_CONVERSION_CHECK(input_size <= max_inputs, "Got more inputs than expected"); } +int non_cont_dim(std::vector ne, std::vector nb) { + int dim = nb.size() - 1; + size_t bytes = nb[dim]; + for (int i = dim; i > 0; i--) { + bytes *= ne[i]; + if (bytes != nb[i - 1]) { + return i; + } + } + return 0; +} + std::shared_ptr get_dimensions(const std::shared_ptr& shape, const std::vector& dims) { using namespace ov::op; @@ -57,6 +69,22 @@ OutputVector rename_outputs_with_suffix(const OutputVector& outputs, const std:: return outputs; } +ov::Output make_topk_indices(const ov::Output& input, + const ov::Output& k, + int64_t axis, + ov::op::v11::TopK::Mode mode, + const ov::element::Type& index_type, + bool stable) { + auto topk = std::make_shared(input, + k, + axis, + mode, + ov::op::v11::TopK::SortType::SORT_VALUES, + index_type, + stable); + return topk->output(1); // indices +} + namespace { ov::Output rope_yarn_ramp_mix(int n_dims, const float corr_dims[2], float ext_factor) { int half_n_dims = n_dims / 2; @@ -106,8 +134,16 @@ void gguf_rope_yarn_corr_dims(int n_dims, std::pair, ov::Output> make_sin_cos(const RopeConfig& rope_config, std::shared_ptr inp_pos, std::shared_ptr rope_freqs_weight, - bool imrope) { - if (imrope) { + bool imrope, + bool stateful) { + if (stateful) { + inp_pos = + std::make_shared(inp_pos, ov::op::v0::Constant::create(ov::element::i64, {1}, {0})); + inp_pos = std::make_shared(inp_pos, ov::element::f32); + auto pos_perm = + std::make_shared(ov::element::i64, ov::Shape{3}, std::vector{2, 1, 0}); + inp_pos = std::make_shared(inp_pos, pos_perm); + } else if (imrope) { inp_pos = std::make_shared(inp_pos, ov::element::f32); auto pos_shape = ov::op::v0::Constant::create(ov::element::i64, ov::Shape{5}, {0, 0, 0, 4, -1}); inp_pos = std::make_shared(inp_pos, pos_shape, true); @@ -158,9 +194,27 @@ std::pair, ov::Output> make_sin_cos(const RopeConfig& rop for (size_t i = 1; i < factor.size(); i++) { factor[i] = theta_scale * factor[i - 1]; } - freq_factors = - std::make_shared(ov::element::f32, ov::Shape{1, 1, 1, factor.size()}, factor); + if (stateful) { + freq_factors = + std::make_shared(ov::element::f32, ov::Shape{1, 1, factor.size()}, factor); + } else { + freq_factors = + std::make_shared(ov::element::f32, ov::Shape{1, 1, 1, factor.size()}, factor); + } if (rope_freqs_weight) { + // rope_freqs_weight has shape [N] for the model's maximum n_dims/2. When this + // ROPE op uses a smaller n_dims (e.g. gemma4 SWA layers), slice to n_dims_half. + auto rfw_shape = rope_freqs_weight->get_output_partial_shape(0); + if (rfw_shape.is_static() && rfw_shape.size() == 1 && + rfw_shape[0].get_length() > static_cast(n_dims_half)) { + auto start = ov::op::v0::Constant::create(ov::element::i64, {1}, {0}); + auto stop = ov::op::v0::Constant::create(ov::element::i64, {1}, {static_cast(n_dims_half)}); + auto step = ov::op::v0::Constant::create(ov::element::i64, {1}, {1}); + auto axes = ov::op::v0::Constant::create(ov::element::i64, {1}, {0}); + rope_freqs_weight = + std::make_shared(rope_freqs_weight, start, stop, step, axes)->output(0) + .get_node_shared_ptr(); + } freq_factors = std::make_shared(freq_factors, rope_freqs_weight); } @@ -173,7 +227,12 @@ std::pair, ov::Output> make_sin_cos(const RopeConfig& rop theta = theta_interp; } else { auto ramp_mix = rope_yarn_ramp_mix(n_dims, corr_dims, ext_factor); - Output one = ov::op::v0::Constant::create(ov::element::f32, Shape{1, 1, 1, 1}, {1.0f}); + Output one; + if (stateful) { + one = ov::op::v0::Constant::create(ov::element::f32, Shape{1, 1, 1}, {1.0f}); + } else { + one = ov::op::v0::Constant::create(ov::element::f32, Shape{1, 1, 1, 1}, {1.0f}); + } auto one_minus_ramp = std::make_shared(one, ramp_mix); theta = @@ -200,6 +259,8 @@ ov::Output process_view_input(const NodeContext& context, int input_in // Only works for VIEW operations that slice at the lowest dimension // If the VIEW also reshape the result, `slice_len` should be provided auto input = context.get_input(input_index); + // The decoder already returns the view start offset in ELEMENTS (it divides ggml's raw byte + // offset by the element size), so no stride division is needed here. int64_t split_addr = context.get_input_view_element_offset(input_index); auto begin = ov::op::v0::Constant::create(ov::element::i64, {1}, {split_addr}); diff --git a/src/frontends/gguf/src/utils.hpp b/src/frontends/gguf/src/utils.hpp index 8ce6c9b76ddc..82c40ad33c82 100644 --- a/src/frontends/gguf/src/utils.hpp +++ b/src/frontends/gguf/src/utils.hpp @@ -7,6 +7,10 @@ #include #include +#include "openvino/core/node.hpp" +#include "openvino/op/shape_of.hpp" +#include "openvino/op/topk.hpp" + #include "openvino/core/node_vector.hpp" #include "node_context.hpp" @@ -23,6 +27,18 @@ namespace gguf { void num_inputs_check(const NodeContext& context, size_t min_inputs, size_t max_inputs); +int non_cont_dim(std::vector ne, std::vector nb); + +template +std::vector permute(const std::vector& x, const std::vector& perm) { + std::vector result; + result.reserve(perm.size()); + for (size_t i : perm) { + result.push_back(x[i]); + } + return result; +} + std::shared_ptr get_dimensions(const std::shared_ptr& shape, const std::vector& dims); // Takes the Output rather than the node so a producer with several outputs keeps the right port. @@ -30,10 +46,31 @@ std::shared_ptr get_dimensions(const ov::Output& output, con OutputVector rename_outputs_with_suffix(const OutputVector& outputs, const std::string& suffix); +/// \brief Build a TopK over `axis` and return its INDICES port. +/// +/// Shared by the ARGSORT and TOP_K translators. Both want ggml's "indices that sort/select along +/// ne[0]" semantics, which in OpenVINO is output(1) of a TopK whose index element type follows the +/// decoder's "output_type" attribute. Keeping that contract in one place stops the two call sites +/// from drifting apart. +/// +/// \param input tensor to sort/select over +/// \param k number of elements to keep along `axis` (may be a dynamic value) +/// \param axis axis to operate on +/// \param mode MAX for descending, MIN for ascending +/// \param index_type element type of the returned indices +/// \param stable whether ties keep their input order +ov::Output make_topk_indices(const ov::Output& input, + const ov::Output& k, + int64_t axis, + ov::op::v11::TopK::Mode mode, + const ov::element::Type& index_type, + bool stable = false); + std::pair, ov::Output> make_sin_cos(const RopeConfig& rope_config, std::shared_ptr inp_pos, std::shared_ptr rope_freqs_weight = nullptr, - bool imrope = false); + bool imrope = false, + bool stateful = false); ov::Output process_view_input(const NodeContext& context, int input_index, int slice_len = 0); diff --git a/src/frontends/gguf/tests/CMakeLists.txt b/src/frontends/gguf/tests/CMakeLists.txt index 4eb970883a18..ea4fed3fc052 100644 --- a/src/frontends/gguf/tests/CMakeLists.txt +++ b/src/frontends/gguf/tests/CMakeLists.txt @@ -15,6 +15,7 @@ set(FRONTEND_SRCS "${FE_SRC_DIR}/utils.cpp" "${FE_SRC_DIR}/helper_ops/set_rows_op.cpp" "${FE_SRC_DIR}/pass/lower_set_rows_stateless.cpp" + "${FE_SRC_DIR}/pass/make_stateful.cpp" "${FE_SRC_DIR}/quant/gguf_quants.cpp" "${FE_SRC_DIR}/quant/weights.cpp" "${FE_SRC_DIR}/op/add_id.cpp" @@ -52,6 +53,7 @@ set(FRONTEND_SRCS "${FE_SRC_DIR}/op/top_k.cpp" "${FE_SRC_DIR}/op/tri.cpp" "${FE_SRC_DIR}/op/sum_rows.cpp" + "${FE_SRC_DIR}/op/top_k.cpp" "${FE_SRC_DIR}/op/transpose.cpp" "${FE_SRC_DIR}/op/unary_elu.cpp" "${FE_SRC_DIR}/op/unary_gelu.cpp" @@ -63,6 +65,7 @@ set(FRONTEND_SRCS set(TEST_SRCS "${CMAKE_CURRENT_SOURCE_DIR}/test_dequant_vs_ggml.cpp" "${CMAKE_CURRENT_SOURCE_DIR}/test_extensions.cpp" + "${CMAKE_CURRENT_SOURCE_DIR}/test_op_coverage.cpp" "${CMAKE_CURRENT_SOURCE_DIR}/test_ops.cpp" "${CMAKE_CURRENT_SOURCE_DIR}/test_weights.cpp" ) @@ -83,17 +86,24 @@ ov_add_test_target( INCLUDES "${CMAKE_CURRENT_SOURCE_DIR}/../src" "${CMAKE_CURRENT_SOURCE_DIR}/../include" + # frontend.cpp (compiled into the self-contained test binary) includes the native + # .gguf load path (get_path_from_any in openvino/frontend/common/path_util.hpp). + "${OpenVINO_SOURCE_DIR}/src/frontends/common/include" + "${OpenVINO_SOURCE_DIR}/src/frontends/common/dev_api" ADD_CLANG_FORMAT LABELS OV UNIT GGUF_FE ) -# Install the .npy reference data next to the test binary (found via getExecutableDirectory()). +# Install the reference data next to the test binary (found via getExecutableDirectory()): +# *.npy op/dequant references. install(DIRECTORY "${CMAKE_CURRENT_SOURCE_DIR}/test_data" DESTINATION tests COMPONENT tests EXCLUDE_FROM_ALL - FILES_MATCHING PATTERN "*.npy") + FILES_MATCHING + PATTERN "*.npy" + PATTERN "manifest.txt") # Sources compiled into the binary: resolve the visibility macros to local definitions, not dllimport. target_compile_definitions(${TARGET_NAME} PRIVATE openvino_gguf_frontend_EXPORTS) diff --git a/src/frontends/gguf/tests/op_test_utils.hpp b/src/frontends/gguf/tests/op_test_utils.hpp index 48c669aa478c..9b1020eac289 100644 --- a/src/frontends/gguf/tests/op_test_utils.hpp +++ b/src/frontends/gguf/tests/op_test_utils.hpp @@ -16,18 +16,18 @@ #pragma once -#include -#include - #include #include #include #include #include +#include #include #include +#include "cnpy.h" #include "common_test_utils/file_utils.hpp" +#include "gtest/gtest.h" #include "op_table.hpp" #include "openvino/core/model.hpp" #include "openvino/core/partial_shape.hpp" @@ -50,6 +50,16 @@ namespace ov_gguf_test { using namespace ov::frontend::gguf; +// Set of ggml op types that some test in this binary has actually converted. Every +// SingleOpDecoder construction records its op type here, so the record is a by-product of the tests +// running rather than a hand-maintained list that can drift. Checked against op_table.cpp by the +// coverage gate in test_op_coverage.cpp, which therefore fails when a new op is registered without +// a test. Populated at run time, so the gate has to run last -- see that file for how. +inline std::set& converted_op_types() { + static std::set ops; + return ops; +} + // Description of one tensor (graph input or op output) in the single-op model. struct TensorDesc { std::string name; @@ -72,6 +82,7 @@ class SingleOpDecoder : public GgufDecoder, public std::enable_shared_from_this< m_inputs(std::move(inputs)), m_output(std::move(output)), m_attributes(std::move(attributes)) { + converted_op_types().insert(m_op_type); for (const auto& in : m_inputs) { m_input_names.push_back(in.name); auto p = std::make_shared(in.type, in.shape); @@ -128,6 +139,10 @@ class SingleOpDecoder : public GgufDecoder, public std::enable_shared_from_this< return {m_output.name}; } + // The optional model-scope accessors (get_model_extra_inputs, get_tokenizer_config) both + // default to empty on GgufDecoder, which is exactly right for a single-op test decoder: no + // auxiliary inputs and no tokenizer metadata. So neither is overridden here. + private: const TensorDesc& find_input(const std::string& name) const { for (const auto& in : m_inputs) { diff --git a/src/frontends/gguf/tests/test_data/gelu_ggml_expected.npy b/src/frontends/gguf/tests/test_data/gelu_ggml_expected.npy new file mode 100644 index 0000000000000000000000000000000000000000..0813a3f6b4c5a65ac1de74db91dc9f08bd817820 GIT binary patch literal 640 zcmbWvPe>GT6bJCHV4xBx;zbesp~)n!h<^@Iy7La=K`;#5KoFc&S3HOtyNd)8{6Y$a zA~6IhkP$*bQG`l&XLp9*Awut=*8%k?+F6jk3*G|yT_)b~8m@M37Jpk*|w=_V^;vu#ND113Ty zE`-)-GEj$$k!v&&TElJU1(mWcOiC(Dvbw0q=%OO4o8JC~4jee=J?DMi56=@ncCy)d zdXuhG*KP9p!fhc_gJL?=ZZ)ZjsXZ8qggk+B!I0PYuiogn6!vjF?Dur|_-?kU$^pw> zRq0j!ch%{1MBjgv>EpK&z5a5ao_?67xwi!>uH-4VJV|4VS&A=YXka!?-IY-a7E{zV zm84U-1UYWsAZ=`zEF&>8#s|r8{VLTD^ixe=l-7TA(eqCcnpq9d#FC$e=36OTa?_c- zi;kw9v@hYH?Sppu-J{XF4jVo8T4<_8rI98h^+BNvRs$V3>&dpOo|LV%RJ*x`*1rCf zg%_V?RKAhU>@%qhKa}g8GqQa4o{Tk4Nca9*Qr$5mH~#d>! z-4=O%`c-xO#GNX=zg)$hZB^ggqR^t^k=KH?4jVRlH0THIP!kT= z(@wbaE<{Rh#O7O(S@NU28UT9;_7dzV*jup2V6VZRgS`i509P)7)~O@{#T2?KqqsJk g#?6Hc#uu}=yF7^pD|v9H;B3JegR=%_4$dBb0ok7(cK`qY literal 0 HcmV?d00001 diff --git a/src/frontends/gguf/tests/test_data/gelu_quick_ggml_expected.npy b/src/frontends/gguf/tests/test_data/gelu_quick_ggml_expected.npy new file mode 100644 index 0000000000000000000000000000000000000000..a0d9a49824ccffb53af7a352c7a0118b630b0983 GIT binary patch literal 640 zcmbWx-)Gfv9LMoj)I}Yey4ppYK304y=OVQkb9TNTMkKY!nuxxq?X1ykbI$Cd)V^dR zHgOVDs_!Nu4k_aMvu_tkH(g}9?JJ_2>>{O$UG(f<=*z?V{eHc@Kfd?Qj-EMpafcRk z!f zuuZnpN+m11&noYsFVi2g3XWP8$E?f%^N^K4%>ESF2u8^-kzcX$rr6D3o}25;K5k=~ k><(6Nmt7n8d1DO^@fhoPif6p}938yCE4=1kBBaReUu(+IJOBUy literal 0 HcmV?d00001 diff --git a/src/frontends/gguf/tests/test_data/gelu_quick_ggml_input.npy b/src/frontends/gguf/tests/test_data/gelu_quick_ggml_input.npy new file mode 100644 index 0000000000000000000000000000000000000000..790c02ef2163a602a7e3d26ccdc0427bf3730aec GIT binary patch literal 640 zcmbV}-%Ha`7{-?rh!;{ZB1XZNF18h4x>yT_)b~8m@M37Jpk*|w=_V^;vu#ND113Ty zE`-)-GEj$$k!v&&TElJU1(mWcOiC(Dvbw0q=%OO4o8JC~4jee=J?DMi56=@ncCy)d zdXuhG*KP9p!fhc_gJL?=ZZ)ZjsXZ8qggk+B!I0PYuiogn6!vjF?Dur|_-?kU$^pw> zRq0j!ch%{1MBjgv>EpK&z5a5ao_?67xwi!>uH-4VJV|4VS&A=YXka!?-IY-a7E{zV zm84U-1UYWsAZ=`zEF&>8#s|r8{VLTD^ixe=l-7TA(eqCcnpq9d#FC$e=36OTa?_c- zi;kw9v@hYH?Sppu-J{XF4jVo8T4<_8rI98h^+BNvRs$V3>&dpOo|LV%RJ*x`*1rCf zg%_V?RKAhU>@%qhKa}g8GqQa4o{Tk4Nca9*Qr$5mH~#d>! z-4=O%`c-xO#GNX=zg)$hZB^ggqR^t^k=KH?4jVRlH0THIP!kT= z(@wbaE<{Rh#O7O(S@NU28UT9;_7dzV*jup2V6VZRgS`i509P)7)~O@{#T2?KqqsJk g#?6Hc#uu}=yF7^pD|v9H;B3JegR=%_4$dBb0ok7(cK`qY literal 0 HcmV?d00001 diff --git a/src/frontends/gguf/tests/test_data/silu_ggml_expected.npy b/src/frontends/gguf/tests/test_data/silu_ggml_expected.npy new file mode 100644 index 0000000000000000000000000000000000000000..30ada7c05310ba60f1de890b77c8ab0c05a4e17b GIT binary patch literal 640 zcmbWrVM~)y7=YoWWs*)AIV*`ZSvp6dmbIhuu_ADSys%psToeUF^sYhb)wlw z-@q~j)7nT3a#|#w^N4+jYL-n>8X~zOl&ru83Xz)qh3;?n#ph^`=M`)a*aTPPdV}SH zS)MADryG;yN~zpvGF#2M^7AIM-tf<}beAlKb+?r0Dh%tNkgSv@sT4|Swe)}6(k`-4 zRb?$zNF21c^DZ^7KBSVo0jlU3r8A*kYU2HLSsS1#RWM_vLS`)7&Q6JUGjZ2Gb}^J= z`KOPu$nG<2Vxf$A)zvI?x`9btEsTbBFj0hyy>+`;Ur&w-9-27M_SvqMH}u zh-H%X&DdGcBBg6lFVjC5?;Z9lHggWq5iZ49%gyx&Ts?#@HatIhyzj;EfAf38&XQ!;HkYGoZX!uV_m>a^?>lF6M8oG df;i?eSQDRsH)8;}lh5F_VF+q%6sqha@E3R~799Wp literal 0 HcmV?d00001 diff --git a/src/frontends/gguf/tests/test_data/silu_ggml_input.npy b/src/frontends/gguf/tests/test_data/silu_ggml_input.npy new file mode 100644 index 0000000000000000000000000000000000000000..790c02ef2163a602a7e3d26ccdc0427bf3730aec GIT binary patch literal 640 zcmbV}-%Ha`7{-?rh!;{ZB1XZNF18h4x>yT_)b~8m@M37Jpk*|w=_V^;vu#ND113Ty zE`-)-GEj$$k!v&&TElJU1(mWcOiC(Dvbw0q=%OO4o8JC~4jee=J?DMi56=@ncCy)d zdXuhG*KP9p!fhc_gJL?=ZZ)ZjsXZ8qggk+B!I0PYuiogn6!vjF?Dur|_-?kU$^pw> zRq0j!ch%{1MBjgv>EpK&z5a5ao_?67xwi!>uH-4VJV|4VS&A=YXka!?-IY-a7E{zV zm84U-1UYWsAZ=`zEF&>8#s|r8{VLTD^ixe=l-7TA(eqCcnpq9d#FC$e=36OTa?_c- zi;kw9v@hYH?Sppu-J{XF4jVo8T4<_8rI98h^+BNvRs$V3>&dpOo|LV%RJ*x`*1rCf zg%_V?RKAhU>@%qhKa}g8GqQa4o{Tk4Nca9*Qr$5mH~#d>! z-4=O%`c-xO#GNX=zg)$hZB^ggqR^t^k=KH?4jVRlH0THIP!kT= z(@wbaE<{Rh#O7O(S@NU28UT9;_7dzV*jup2V6VZRgS`i509P)7)~O@{#T2?KqqsJk g#?6Hc#uu}=yF7^pD|v9H;B3JegR=%_4$dBb0ok7(cK`qY literal 0 HcmV?d00001 diff --git a/src/frontends/gguf/tests/test_dequant_vs_ggml.cpp b/src/frontends/gguf/tests/test_dequant_vs_ggml.cpp index 77c1f473a12f..795592eb06db 100644 --- a/src/frontends/gguf/tests/test_dequant_vs_ggml.cpp +++ b/src/frontends/gguf/tests/test_dequant_vs_ggml.cpp @@ -4,7 +4,7 @@ // Dequantization correctness tests with REAL ggml as the oracle. // // The reference data was produced offline by linking real ggml from llama.cpp -// (see tests/gen_ggml_reference.c): ggml quantizes smooth, asymmetric synthetic +// (captured from real ggml): ggml quantizes smooth, asymmetric synthetic // data into real GGUF-format blocks (_qbytes) and dequantizes those exact bytes // (_deq). The committed .npy files mean the tests need no ggml / llama.cpp at // build or run time. @@ -15,14 +15,13 @@ // Tolerance: ggml stores K-quant scales as f16 and the dequant subgraph runs in f16, // so allow ~3e-3 (matching llama.cpp's MAX_QUANTIZATION_TOTAL_ERROR-class thresholds). -#include - #include #include #include #include #include +#include "gtest/gtest.h" #include "op_test_utils.hpp" #include "openvino/op/constant.hpp" #include "openvino/op/result.hpp" @@ -160,16 +159,18 @@ TEST_P(FaithfulDequantVsGGML, MatchesGgmlToFloat) { for (size_t r = 0; r < kRows; ++r) { c.dq(qbytes.data() + r * bytes_per_row, kCols, ours.data() + r * kCols); } - EXPECT_LE(max_abs_diff(ours, ref), 3e-3f) - << c.stem << ": faithful per-row dequant diverges from ggml to_float"; + EXPECT_LE(max_abs_diff(ours, ref), 3e-3f) << c.stem << ": faithful per-row dequant diverges from ggml to_float"; } -INSTANTIATE_TEST_SUITE_P(FaithfulKQuant, - FaithfulDequantVsGGML, - ::testing::Values(FaithfulCase{"q4_k", GGUF_TYPE_Q4_K, ov::frontend::gguf::dequant_row_q4_k_f32_for_test}, - FaithfulCase{"q5_k", GGUF_TYPE_Q5_K, ov::frontend::gguf::dequant_row_q5_k_f32_for_test}, - FaithfulCase{"q6_k", GGUF_TYPE_Q6_K, ov::frontend::gguf::dequant_row_q6_k_f32_for_test}), - [](const ::testing::TestParamInfo& i) { return std::string(i.param.stem); }); +INSTANTIATE_TEST_SUITE_P( + FaithfulKQuant, + FaithfulDequantVsGGML, + ::testing::Values(FaithfulCase{"q4_k", GGUF_TYPE_Q4_K, ov::frontend::gguf::dequant_row_q4_k_f32_for_test}, + FaithfulCase{"q5_k", GGUF_TYPE_Q5_K, ov::frontend::gguf::dequant_row_q5_k_f32_for_test}, + FaithfulCase{"q6_k", GGUF_TYPE_Q6_K, ov::frontend::gguf::dequant_row_q6_k_f32_for_test}), + [](const ::testing::TestParamInfo& i) { + return std::string(i.param.stem); + }); INSTANTIATE_TEST_SUITE_P(AllQuantTypes, DequantVsGGML, @@ -183,6 +184,9 @@ INSTANTIATE_TEST_SUITE_P(AllQuantTypes, DeqCase{"q4_k", GGUF_TYPE_Q4_K, kTolIntZp}, DeqCase{"q5_k", GGUF_TYPE_Q5_K, kTolRequant}, DeqCase{"q6_k", GGUF_TYPE_Q6_K, kTolRequant}, + // Q2_0 is bit-exact: both sides compute (code - 1) * d + // from the same f16 scale, and the u8 zero-point of 1 is + // represented exactly, so no dequant noise is introduced. DeqCase{"q2_0", GGUF_TYPE_Q2_0, kTolExact}), [](const ::testing::TestParamInfo& i) { return std::string(i.param.stem); diff --git a/src/frontends/gguf/tests/test_extensions.cpp b/src/frontends/gguf/tests/test_extensions.cpp index 505311f09a12..a0f79ce28223 100644 --- a/src/frontends/gguf/tests/test_extensions.cpp +++ b/src/frontends/gguf/tests/test_extensions.cpp @@ -3,17 +3,37 @@ // // Tests for FrontEnd::add_extension (the extension-passing path in frontend.cpp). // -// A ConversionExtension registers a custom translator for a ggml op name; the frontend -// merges it into the op table (overriding a built-in translator on name collision, or -// adding a translator for an otherwise unsupported op). The converter receives an -// ov::frontend::NodeContext, which the gguf NodeContext derives from. +// Two extension kinds are covered: +// +// - ov::frontend::ConversionExtension registers a custom translator for a ggml op name; the +// frontend merges it into the op table (overriding a built-in translator on name collision, or +// adding a translator for an otherwise unsupported op). The converter receives an +// ov::frontend::NodeContext, which the gguf NodeContext derives from. +// +// - ov::frontend::DecoderTransformationExtension registers a normalization pass, run ahead of the +// frontend's built-in lowerings. This is how the EXECUTION MODE is chosen: conversion always +// yields a stateless graph (KV caches as Parameter/Result pairs written by a SetRows +// placeholder), and a caller that wants an OpenVINO KV cache registers +// ov::frontend::gguf::pass::MakeStateful here, which consumes those SetRows ops before the +// default stateless lowering ever sees them. -#include -#include -#include +#include +#include +#include #include "op_test_utils.hpp" #include "openvino/frontend/extension/conversion.hpp" +#include "openvino/frontend/extension/decoder_transformation.hpp" +#include "openvino/frontend/gguf/make_stateful.hpp" +#include "openvino/frontend/gguf/set_rows_op.hpp" +#include "openvino/op/abs.hpp" +#include "openvino/op/assign.hpp" +#include "openvino/op/concat.hpp" +#include "openvino/op/gather.hpp" +#include "openvino/op/multiply.hpp" +#include "openvino/op/negative.hpp" +#include "openvino/op/read_value.hpp" +#include "openvino/op/scatter_update.hpp" using namespace ov_gguf_test; @@ -93,3 +113,207 @@ TEST(GGUFExtensions, UnsupportedOpWithoutExtensionThrows) { .output("out", ov::element::f32, {3}); EXPECT_ANY_THROW(builder.build()); } + +// ── DecoderTransformationExtension: choosing the execution mode ───────────────────────────────── + +namespace { + +// One GGML_OP_SET_ROWS writing `data` rows at `idx` into the `cache` input -- the shape of a KV +// cache write, in the layout the native .gguf builder emits: [1, tokens, n_head_kv, head_size], +// whose one dynamic axis (1, the token axis) is what MakeStateful infers the append axis from. +SingleOpBuilder kv_cache_write_builder() { + return SingleOpBuilder() + .op("GGML_OP_SET_ROWS") + .input("data", ov::element::f32, {1, -1, 2, 4}) + .input("idx", ov::element::i64, {1, 1, 1, -1}) + .input("cache", ov::element::f16, {1, -1, 2, 4}) + .output("cache_out", ov::element::f16, {1, -1, 2, 4}); +} + +size_t count_ops_of_type(const std::shared_ptr& model, const ov::DiscreteTypeInfo& type) { + size_t n = 0; + for (const auto& op : model->get_ops()) { + if (op->get_type_info() == type) { + n++; + } + } + return n; +} + +} // namespace + +// The default: with no extension registered, conversion lowers every SetRows to the stateless +// ScatterUpdate form, and the cache stays an ordinary model input/output. This is the baseline the +// design rests on -- the frontend itself is stateless, like an optimum-intel export. +TEST(GGUFExtensions, NoExtensionYieldsStatelessCache) { + auto model = kv_cache_write_builder().build(); + + EXPECT_TRUE(model->get_variables().empty()); + EXPECT_TRUE(model->get_sinks().empty()); + EXPECT_EQ(count_ops_of_type(model, ov::op::v3::ScatterUpdate::get_type_info_static()), 1); + // The SetRows placeholder is an internal op and must never survive conversion. + EXPECT_EQ(count_ops_of_type(model, SetRows::get_type_info_static()), 0); + // cache is still an input, cache_out still an output. + EXPECT_EQ(model->get_parameters().size(), 3); + EXPECT_EQ(model->get_results().size(), 1); + + // No beam_idx: it is a stateful-cache concept, so the stateless graph must not carry one. This is + // what lets the native builder and a llama.cpp cgraph decoder agree on their stateless IO -- a + // decoder-declared beam_idx would be an input with no consumer here. + for (const auto& p : model->get_parameters()) { + EXPECT_NE(p->get_friendly_name(), "beam_idx"); + } +} + +// Registering MakeStateful as a DecoderTransformationExtension swaps the execution mode: the same +// conversion now yields an OpenVINO state. The cache Parameter/Result pair is gone, replaced by a +// Variable with a ReadValue/Concat/Assign, and no ScatterUpdate is emitted -- the extension ran +// ahead of the built-in stateless lowering and consumed the SetRows first. +TEST(GGUFExtensions, MakeStatefulExtensionYieldsStatefulCache) { + auto model = kv_cache_write_builder().build_with_extensions( + {std::make_shared(pass::MakeStateful())}); + + ASSERT_EQ(model->get_variables().size(), 1); + EXPECT_EQ(model->get_sinks().size(), 1); + EXPECT_EQ(count_ops_of_type(model, ov::op::v6::ReadValue::get_type_info_static()), 1); + EXPECT_EQ(count_ops_of_type(model, ov::op::v6::Assign::get_type_info_static()), 1); + EXPECT_EQ(count_ops_of_type(model, ov::op::v3::ScatterUpdate::get_type_info_static()), 0); + EXPECT_EQ(count_ops_of_type(model, SetRows::get_type_info_static()), 0); + + // The cache left the model's IO entirely: data + idx remain, and beam_idx was ADDED by the pass + // (see below). The cache Result became the Assign sink. + EXPECT_EQ(model->get_parameters().size(), 3); + EXPECT_EQ(model->get_results().size(), 0); + + // beam_idx belongs to the state, so the pass creates it -- no decoder declares it. Its Gather on + // the past is what CPU's stateful_sdpa_fusion matches. + auto beam_idx = std::find_if(model->get_parameters().begin(), + model->get_parameters().end(), + [](const std::shared_ptr& p) { + return p->get_friendly_name() == "beam_idx"; + }); + ASSERT_NE(beam_idx, model->get_parameters().end()); + EXPECT_EQ((*beam_idx)->get_element_type(), ov::element::i32); + EXPECT_EQ((*beam_idx)->get_partial_shape(), ov::PartialShape({-1})); + EXPECT_EQ(count_ops_of_type(model, ov::op::v8::Gather::get_type_info_static()), 1); + + // The Variable is named after the cache input and its append axis is dynamic (the state grows + // by this step's rows on every inference), the rest keeping the cache's declared dims. + const auto& info = model->get_variables()[0]->get_info(); + EXPECT_EQ(info.variable_id, "cache"); + EXPECT_EQ(info.data_type, ov::element::f16); + EXPECT_EQ(info.data_shape, ov::PartialShape({1, -1, 2, 4})); +} + +// skip_caches leaves a named cache stateless while other caches are converted. A sliding-window +// cache needs this: it is evicted from the front, not only appended to, so an append-grown Variable +// would not reproduce it. +TEST(GGUFExtensions, MakeStatefulSkipsNamedCache) { + auto model = kv_cache_write_builder().build_with_extensions( + {std::make_shared(pass::MakeStateful({"cache"}))}); + + // The only cache was skipped, so the pass made no change and the built-in stateless lowering + // handled the SetRows -- an identical result to registering no extension at all. + EXPECT_TRUE(model->get_variables().empty()); + EXPECT_EQ(count_ops_of_type(model, ov::op::v3::ScatterUpdate::get_type_info_static()), 1); + EXPECT_EQ(model->get_parameters().size(), 3); + EXPECT_EQ(model->get_results().size(), 1); +} + +// ── the stateless IO contract: two decoders of one model must agree ────────────────────────────── + +namespace { + +// A decoder that routes a named subset of its inputs through get_model_extra_inputs() instead of +// get_model_inputs(), which is the one structural difference between how the native .gguf builder +// and the llama.cpp cgraph decoder present a model's IO. Both halves land in the same graph, so +// converting either way must yield the same stateless inputs. +class SplitIoDecoder : public SingleOpDecoder { +public: + SplitIoDecoder(const SingleOpDecoder& base, const std::set& as_extra) : SingleOpDecoder(base) { + for (const auto& name : as_extra) { + auto it = m_split_main.find(name); + if (it == m_split_main.end()) { + throw std::runtime_error("SplitIoDecoder: no such input '" + name + "'"); + } + m_split_extra[name] = it->second; + m_split_main.erase(it); + } + } + + const std::map>& get_model_inputs() const override { + return m_split_main; + } + const std::map>& get_model_extra_inputs() const override { + return m_split_extra; + } + +private: + // Seeded from the base decoder's inputs (member initializers run before the constructor body), + // then partitioned by that body. + std::map> m_split_main = SingleOpDecoder::get_model_inputs(); + std::map> m_split_extra; +}; + +std::set input_names(const std::shared_ptr& model) { + std::set names; + for (const auto& p : model->get_parameters()) { + names.insert(p->get_friendly_name()); + } + return names; +} + +} // namespace + +// The frontend invents no inputs of its own: the stateless graph's inputs are exactly what the +// decoder declared, however the decoder chose to split them between get_model_inputs() and +// get_model_extra_inputs(). That split is the one structural difference between the native builder +// and the llama.cpp cgraph decoder, so pinning it down here is half of "the two decoders produce the +// same graph"; the other half -- that neither decoder declares an input the other cannot, beam_idx +// being the case that got this wrong -- needs a real .gguf and lives in the model-level checks. +TEST(GGUFExtensions, StatelessIoIsExactlyTheDecoderInputs) { + const std::set declared{"data", "idx", "cache"}; + + auto base = kv_cache_write_builder(); + auto via_main = base.build(); + EXPECT_EQ(input_names(via_main), declared); + + // The same op, with "cache" and "idx" presented as auxiliary inputs the way the cgraph decoder + // presents its extras. Same graph inputs -> the two decoders agree. + FrontEnd fe; + auto split = std::make_shared(*std::dynamic_pointer_cast(base.decoder()), + std::set{"cache", "idx"}); + auto via_extra = fe.convert(fe.load(std::static_pointer_cast(split))); + EXPECT_EQ(input_names(via_extra), declared); +} + +// And making the model stateful adds exactly one input, beam_idx, on top of that contract -- so the +// stateful IO is a function of the pass, not of which decoder produced the stateless graph. +TEST(GGUFExtensions, MakeStatefulAddsOnlyBeamIdx) { + auto stateless = input_names(kv_cache_write_builder().build()); + auto stateful = input_names(kv_cache_write_builder().build_with_extensions( + {std::make_shared(pass::MakeStateful())})); + + // The cache Parameter became a Variable, and beam_idx appeared. + stateless.erase("cache"); + stateless.insert("beam_idx"); + EXPECT_EQ(stateful, stateless); +} + +// A DecoderTransformationExtension can hold any pass, not only the ones the frontend ships: here a +// plain lambda pass, which must run during conversion (it renames the model, observable after). +TEST(GGUFExtensions, ArbitraryTransformationExtensionRuns) { + auto model = SingleOpBuilder() + .op("GGML_OP_SCALE") + .input("x", ov::element::f32, {2, 2}) + .output("out", ov::element::f32, {2, 2}) + .attr("scale", 2.0f) + .attr("bias", 0.0f) + .build_with_extensions({std::make_shared( + [](const std::shared_ptr& m) { + m->set_friendly_name("touched_by_extension"); + return true; + })}); + + EXPECT_EQ(model->get_friendly_name(), "touched_by_extension"); +} diff --git a/src/frontends/gguf/tests/test_op_coverage.cpp b/src/frontends/gguf/tests/test_op_coverage.cpp new file mode 100644 index 000000000000..b72524c410fd --- /dev/null +++ b/src/frontends/gguf/tests/test_op_coverage.cpp @@ -0,0 +1,105 @@ +// Copyright (C) 2018-2026 Intel Corporation +// SPDX-License-Identifier: Apache-2.0 +// +// Coverage gate: every ggml op registered in op_table.cpp must be exercised by some test in this +// binary. +// +// Without this, the per-op suite silently stops keeping pace with the op table -- adding a +// translator and forgetting its test is invisible, and a wrong-but-plausible formula ships. That is +// not hypothetical: GGML_UNARY_OP_GELU_QUICK was registered with the tanh-GELU formula instead of +// ggml's x*sigmoid(1.702x) and went unnoticed because nothing converted it. +// +// The "tested" side of the comparison is collected at run time: SingleOpDecoder's constructor +// records its op type in converted_op_types(), so the record cannot drift from what the tests +// actually do (a hand-written list would just be a second thing to forget). Consequently this check +// must run AFTER all other tests, which gtest guarantees for a global test environment's TearDown -- +// so the assertion lives there rather than in a TEST body. +// +// A gtest --gtest_filter that excludes op tests would leave the record incomplete and the gate would +// fire spuriously, so it only asserts when the full suite ran (no filter narrowing in effect). + +#include +#include +#include +#include + +#include "gtest/gtest.h" +#include "op_table.hpp" +#include "op_test_utils.hpp" + +using namespace ov_gguf_test; + +namespace { + +// Ops that are registered but intentionally not covered by a single-op test, each with the reason. +// Anything here needs a justification that is about the op's nature, not about effort -- an op that +// is merely awkward to test belongs in the suite, not on this list. +const std::set& coverage_exemptions() { + static const std::set exemptions{ + // Aliases of ops already covered under a different ggml name, translated by the very same + // function pointer, so a separate case would test the same code path twice. + // (none currently -- GGML_OP_ADD1 has its own test because its broadcast shape differs) + }; + return exemptions; +} + +class OpCoverageEnvironment : public ::testing::Environment { +public: + void TearDown() override { + // Only meaningful when the whole suite ran; a narrowing filter makes the record partial. + const std::string filter = ::testing::GTEST_FLAG(filter); + if (filter != "*" && filter != "*.*") { + GTEST_LOG_(INFO) << "op coverage gate skipped: --gtest_filter=" << filter << " is in effect"; + return; + } + + const auto& converted = converted_op_types(); + std::vector missing; + for (const auto& entry : ov::frontend::gguf::get_supported_ops()) { + const std::string& op = entry.first; + if (converted.count(op) == 0 && coverage_exemptions().count(op) == 0) { + missing.push_back(op); + } + } + std::sort(missing.begin(), missing.end()); + + if (!missing.empty()) { + std::string list; + for (const auto& op : missing) { + list += "\n " + op; + } + // Failures raised from an Environment's TearDown are counted separately from tests, so + // the run reports "0 FAILED TESTS" while still exiting non-zero. Name the check + // explicitly so the CI log is not misread as a spurious failure. + ADD_FAILURE() << "[GGUF op coverage gate] " << missing.size() + << " op(s) registered in op_table.cpp have no test in ov_gguf_frontend_tests:" << list + << "\nAdd a case to test_ops.cpp (or test_weights.cpp for weight leaves). If the op " + << "genuinely cannot be tested in isolation, add it to coverage_exemptions() in " + << "test_op_coverage.cpp with the reason."; + } + } +}; + +// Registered at static-init time; gtest runs environment TearDown after the last test. +const auto* const op_coverage_env = ::testing::AddGlobalTestEnvironment(new OpCoverageEnvironment()); + +} // namespace + +// Guard the guard: the coverage record must be non-empty and must contain ops the suite obviously +// converts. If SingleOpDecoder ever stops recording, the TearDown check above would pass vacuously +// for an empty op table and fail confusingly otherwise; this makes the wiring itself testable. +TEST(GGUFOpCoverage, RecordIsPopulated) { + // This test's own decoder construction guarantees at least one entry regardless of test order. + SingleOpBuilder() + .op("GGML_OP_ADD") + .input("a", ov::element::f32, {1}) + .output("out", ov::element::f32, {1}) + .decoder(); + EXPECT_NE(converted_op_types().count("GGML_OP_ADD"), 0u); +} + +// The op table itself must be non-degenerate: a build that dropped the registrations would make the +// coverage gate above pass trivially. +TEST(GGUFOpCoverage, OpTableIsNonEmpty) { + EXPECT_GT(ov::frontend::gguf::get_supported_ops().size(), 50u); +} diff --git a/src/frontends/gguf/tests/test_ops.cpp b/src/frontends/gguf/tests/test_ops.cpp index 2deae757bb0f..e1408a95f513 100644 --- a/src/frontends/gguf/tests/test_ops.cpp +++ b/src/frontends/gguf/tests/test_ops.cpp @@ -61,13 +61,17 @@ INSTANTIATE_TEST_SUITE_P(GGUFOps, [](const ::testing::TestParamInfo& i) { return std::string(i.param.name); }); // ── Elementwise unary ops (single f32 input) ──────────────────────────────────── -// silu / gelu(tanh) / tanh / softplus share the same one-input graph and driver. +// Every registered GGML_UNARY_OP_* plus the elementwise GGML_OP_{LOG,SIN,COS} share the same +// one-input graph and driver, so they are parameterized over (op type, reference lambda). struct UnaryCase { const char* name; const char* op_type; std::function ref; float atol; + // Input values; empty means the default sign-spanning ramp below. Ops with a restricted + // domain (log) supply their own. + std::vector x{}; }; class GGUFUnaryElementwise : public ::testing::TestWithParam {}; @@ -80,7 +84,8 @@ TEST_P(GGUFUnaryElementwise, MatchesReference) { .output("out", ov::element::f32, {2, 4}) .build(); - std::vector x{-2, -1, -0.5f, 0, 0.5f, 1, 2, 3}; + std::vector x = c.x.empty() ? std::vector{-2, -1, -0.5f, 0, 0.5f, 1, 2, 3} : c.x; + ASSERT_EQ(x.size(), 8u) << "UnaryCase input must match the [2,4] graph shape"; auto out = run_on_cpu(model, {{"x", make_f32_tensor({2, 4}, x)}}); std::vector expected(x.size()); @@ -89,8 +94,10 @@ TEST_P(GGUFUnaryElementwise, MatchesReference) { expect_near(out, expected, c.atol); } -// ggml GELU is the tanh approximation, but the frontend maps GGML_UNARY_OP_GELU to v7::Gelu(TANH) -// which is close enough to the exact (erf) form to check against it at 1e-3. +// References are the scalar ggml kernels from ggml/src/ggml-cpu/vec.h, so a translator that picks +// a different-but-plausible formula for the same name is caught (GELU vs GELU_QUICK are distinct +// approximations, not interchangeable). GELU is checked against the exact erf form instead: the +// frontend maps it to v7::Gelu(TANH), which agrees with ggml's tanh kernel and with erf to 1e-3. // Softplus uses 1e-3 to cover ARM CPU fp16 execution (small outputs where fp16 spacing ~1e-3 // dominates); on x86 fp32 the exact reference still matches comfortably within that bound. INSTANTIATE_TEST_SUITE_P( @@ -102,19 +109,27 @@ INSTANTIATE_TEST_SUITE_P( "GGML_UNARY_OP_GELU", [](float x) { return 0.5f * x * (1.0f + std::erf(x / std::sqrt(2.0f))); }, 1e-3f}, - UnaryCase{"tanh", "GGML_UNARY_OP_TANH", [](float x) { return std::tanh(x); }, 1e-4f}, - UnaryCase{"relu", "GGML_UNARY_OP_RELU", [](float x) { return x > 0.0f ? x : 0.0f; }, 1e-4f}, - UnaryCase{"elu", "GGML_UNARY_OP_ELU", [](float x) { return x > 0.0f ? x : std::expm1(x); }, 1e-4f}, + // ggml_gelu_quick_f32: x*(1/(1+expf(-1.702f*x))) -- NOT the tanh GELU above. UnaryCase{"gelu_quick", "GGML_UNARY_OP_GELU_QUICK", - [](float x) { return x * (1.0f / (1.0f + std::exp(-1.702f * x))); }, + [](float x) { return x / (1.0f + std::exp(-1.702f * x)); }, 1e-4f}, + UnaryCase{"tanh", "GGML_UNARY_OP_TANH", [](float x) { return std::tanh(x); }, 1e-4f}, + UnaryCase{"relu", "GGML_UNARY_OP_RELU", [](float x) { return x > 0.0f ? x : 0.0f; }, 1e-4f}, + // ggml_vec_elu_f32: (x > 0) ? x : expm1f(x), i.e. ELU with alpha == 1. + UnaryCase{"elu", "GGML_UNARY_OP_ELU", [](float x) { return x > 0.0f ? x : std::expm1(x); }, 1e-4f}, UnaryCase{"sin", "GGML_OP_SIN", [](float x) { return std::sin(x); }, 1e-4f}, UnaryCase{"cos", "GGML_OP_COS", [](float x) { return std::cos(x); }, 1e-4f}, UnaryCase{"softplus", "GGML_UNARY_OP_SOFTPLUS", [](float x) { return std::log1p(std::exp(-std::abs(x))) + std::max(x, 0.0f); }, - 1e-3f}), + 1e-3f}, + // Log's domain is x > 0, so this case overrides the default ramp. + UnaryCase{"log", + "GGML_OP_LOG", + [](float x) { return std::log(x); }, + 1e-4f, + {0.25f, 0.5f, 1.0f, 2.0f, 3.0f, 10.0f, 100.0f, 1e-3f}}), [](const ::testing::TestParamInfo& i) { return std::string(i.param.name); }); // Log is only defined for x > 0, so it gets its own inputs rather than the shared range above. @@ -153,26 +168,51 @@ TEST(GGUFOps, GetDimensionsKeepsOutputPort) { EXPECT_EQ(shape_of->input_value(0).get_index(), 1u) << "get_dimensions measured the wrong port"; } -// ggml_top_k: indices of the k largest values along ne[0], ordered by descending value. -TEST(GGUFOps, TopK) { - auto model = SingleOpBuilder() - .op("GGML_OP_TOP_K") - .input("x", ov::element::f32, {1, 1, 2, 4}) - .output("out", ov::element::i32, {1, 1, 2, 2}) - .build(); +// ── Real-ggml reference data (test_data/*_ggml_{input,expected}.npy) ──────────── +// The parameterized cases above encode the ggml formula by hand in C++. These cases instead run +// the actual ggml kernel's output, captured offline from real ggml linked against +// libggml, over a [-6, 6] ramp -- so a misreading of the kernel cannot be baked into both sides. +// This is the check that distinguishes GELU_QUICK's sigmoid form from the tanh form: they differ +// by 2.2e-2 in the negative tail, far outside these tolerances. +// +// Tolerances are set by ggml's own arithmetic, not by OV's. ggml evaluates GELU and GELU_QUICK +// through an fp16 lookup table (GGML_GELU_FP16), which costs ~2e-3 / ~3.3e-3 against the exact +// fp32 form; SILU runs in fp32 in the reference build and needs only 1e-5. +struct GgmlRefCase { + const char* name; // also the test_data/_ggml_{input,expected}.npy stem prefix + const char* op_type; + float atol; +}; - std::vector x{4, 1, 3, 2, 10, 40, 20, 30}; - auto out = run_on_cpu(model, {{"x", make_f32_tensor({1, 1, 2, 4}, x)}}); +class GGUFUnaryVsGgml : public ::testing::TestWithParam {}; - ASSERT_EQ(out.get_element_type(), ov::element::i32); - ASSERT_EQ(out.get_size(), 4u); - const int32_t* a = out.data(); - // Row 0: 4,1,3,2 -> top2 are 4 (idx 0) then 3 (idx 2). Row 1: 10,40,20,30 -> 40 (1), 30 (3). - std::vector expected{0, 2, 1, 3}; - for (size_t i = 0; i < expected.size(); ++i) - EXPECT_EQ(a[i], expected[i]) << "mismatch at index " << i; +TEST_P(GGUFUnaryVsGgml, MatchesGgmlKernel) { + const GgmlRefCase c = GetParam(); + const auto x = load_npy(std::string(c.name) + "_ggml_input"); + const auto expected = load_npy(std::string(c.name) + "_ggml_expected"); + ASSERT_EQ(x.size(), expected.size()); + ASSERT_FALSE(x.empty()); + + // The captured reference is a [4, 32] ramp. + const ov::Shape shape{4, 32}; + ASSERT_EQ(ov::shape_size(shape), x.size()); + + auto model = SingleOpBuilder() + .op(c.op_type) + .input("x", ov::element::f32, shape) + .output("out", ov::element::f32, shape) + .build(); + auto out = run_on_cpu(model, {{"x", make_f32_tensor(shape, x)}}); + expect_near(out, expected, c.atol); } +INSTANTIATE_TEST_SUITE_P(GGUFOps, + GGUFUnaryVsGgml, + ::testing::Values(GgmlRefCase{"silu", "GGML_UNARY_OP_SILU", 1e-5f}, + GgmlRefCase{"gelu", "GGML_UNARY_OP_GELU", 2.5e-3f}, + GgmlRefCase{"gelu_quick", "GGML_UNARY_OP_GELU_QUICK", 4e-3f}), + [](const ::testing::TestParamInfo& i) { return std::string(i.param.name); }); + // Scale: out = in * scale + bias (scale/bias in op-params slots 0,1). TEST(GGUFOps, Scale) { const float scale = 2.5f; @@ -302,7 +342,7 @@ TEST(GGUFOps, SoftMaxAlibi) { const float m1 = std::pow(2.0f, -(max_bias / 2.0f) / n_head_log2); std::vector expected(x.size()); for (uint32_t h = 0; h < n_head; ++h) { - float slope = h < n_head_log2 ? std::pow(m0, static_cast(h + 1)) : std::pow(m1, static_cast(2 * (h - n_head_log2) + 1)); + float slope = h < n_head_log2 ? std::pow(m0, h + 1) : std::pow(m1, 2 * (h - n_head_log2) + 1); for (size_t t = 0; t < T; ++t) { float mx = -1e30f; std::vector z(Kd); @@ -562,6 +602,152 @@ TEST(GGUFOps, TransposePerm) { expect_near(out, expected); } +// Permute op_case 1: the plain head/token axis swap, perm {0,2,1,3}. +TEST(GGUFOps, PermuteCase1SwapsHeadAndTokenAxes) { + // [1, tok=2, heads=3, hs=2] -> [1, heads=3, tok=2, hs=2] + auto model = SingleOpBuilder() + .op("GGML_OP_PERMUTE") + .input("x", ov::element::f32, {1, 2, 3, 2}) + .output("out", ov::element::f32, {1, 3, 2, 2}) + .op_case(1) + .build(); + + std::vector x(12); + for (size_t i = 0; i < x.size(); ++i) + x[i] = static_cast(i); + auto out = run_on_cpu(model, {{"x", make_f32_tensor({1, 2, 3, 2}, x)}}); + + ASSERT_EQ(out.get_shape(), (ov::Shape{1, 3, 2, 2})); + // out[0,h,t,d] == x[0,t,h,d]; with x flat == index, x[0,t,h,d] = ((t*3)+h)*2+d. + std::vector expected; + for (int64_t h = 0; h < 3; ++h) + for (int64_t t = 0; t < 2; ++t) + for (int64_t d = 0; d < 2; ++d) + expected.push_back(static_cast((t * 3 + h) * 2 + d)); + expect_near(out, expected, 0.0f); +} + +// Permute op_case 4: reshape the flat projection to [n_seq, -1, n_heads, head_size] first, then +// apply the same axis swap. This is the Q/K projection path, where the head split and the permute +// are a single ggml op. +TEST(GGUFOps, PermuteCase4SplitsHeadsThenSwaps) { + // Flat [1, 1, tok=2, heads*hs=6] -> [1, heads=3, tok=2, hs=2] + auto model = SingleOpBuilder() + .op("GGML_OP_PERMUTE") + .input("x", ov::element::f32, {1, 1, 2, 6}) + .output("out", ov::element::f32, {1, 3, 2, 2}) + .op_case(4) + .build(); + + std::vector x(12); + for (size_t i = 0; i < x.size(); ++i) + x[i] = static_cast(i); + auto out = run_on_cpu(model, {{"x", make_f32_tensor({1, 1, 2, 6}, x)}}); + + ASSERT_EQ(out.get_shape(), (ov::Shape{1, 3, 2, 2})); + // The reshape splits the 6-wide row into 3 heads of 2, so out[0,h,t,d] == x_flat[t*6 + h*2 + d]. + std::vector expected; + for (int64_t h = 0; h < 3; ++h) + for (int64_t t = 0; t < 2; ++t) + for (int64_t d = 0; d < 2; ++d) + expected.push_back(static_cast(t * 6 + h * 2 + d)); + expect_near(out, expected, 0.0f); +} + +// Permute rejects op_cases it does not implement rather than silently emitting a wrong graph. +TEST(GGUFOps, PermuteUnsupportedCaseThrows) { + EXPECT_THROW(SingleOpBuilder() + .op("GGML_OP_PERMUTE") + .input("x", ov::element::f32, {1, 2, 3, 2}) + .output("out", ov::element::f32, {1, 3, 2, 2}) + .op_case(7) + .build(), + ov::Exception); +} + +// View with no op_case is a pure reinterpretation of the same buffer: a pass-through. +TEST(GGUFOps, ViewDefaultIsPassThrough) { + auto model = SingleOpBuilder() + .op("GGML_OP_VIEW") + .input("x", ov::element::f32, {1, 1, 2, 4}) + .output("out", ov::element::f32, {1, 1, 2, 4}) + .build(); + + std::vector x{1, 2, 3, 4, 5, 6, 7, 8}; + auto out = run_on_cpu(model, {{"x", make_f32_tensor({1, 1, 2, 4}, x)}}); + expect_near(out, x, 0.0f); +} + +// View op_case 3: the decoder resolves ggml's ne/nb/offset into a {axis, start, len} slice plus the +// view's own output layout. Here it selects columns [1, 3) of a [1,1,2,4] source. +TEST(GGUFOps, ViewCase3SlicesAndReshapes) { + auto model = SingleOpBuilder() + .op("GGML_OP_VIEW") + .input("x", ov::element::f32, {1, 1, 2, 4}) + .output("out", ov::element::f32, {1, 1, 2, 2}) + .op_case(3) + .attr("input_ggml_shape", ov::Shape{1, 1, 2, 4}) + .attr>("view_slice", {3, 1, 2}) + .attr>("view_reshape", {1, 1, 2, 2}) + .build(); + + std::vector x{1, 2, 3, 4, 5, 6, 7, 8}; + auto out = run_on_cpu(model, {{"x", make_f32_tensor({1, 1, 2, 4}, x)}}); + + ASSERT_EQ(out.get_shape(), (ov::Shape{1, 1, 2, 2})); + std::vector expected{2, 3, 6, 7}; + expect_near(out, expected, 0.0f); +} + +// View op_case 3 also has to restore the source's original ggml shape when the OV input arrives at +// a different rank (a preceding op already reshaped it), otherwise the slice lands on the wrong +// axis. Same slice as above, but the input is presented flat. +TEST(GGUFOps, ViewCase3RestoresGgmlShapeBeforeSlicing) { + auto model = SingleOpBuilder() + .op("GGML_OP_VIEW") + .input("x", ov::element::f32, {2, 4}) + .output("out", ov::element::f32, {1, 1, 2, 2}) + .op_case(3) + .attr("input_ggml_shape", ov::Shape{1, 1, 2, 4}) + .attr>("view_slice", {3, 1, 2}) + .attr>("view_reshape", {1, 1, 2, 2}) + .build(); + + std::vector x{1, 2, 3, 4, 5, 6, 7, 8}; + auto out = run_on_cpu(model, {{"x", make_f32_tensor({2, 4}, x)}}); + + ASSERT_EQ(out.get_shape(), (ov::Shape{1, 1, 2, 2})); + std::vector expected{2, 3, 6, 7}; + expect_near(out, expected, 0.0f); +} + +// TopK: indices of the k largest values per row, k taken from the output's last dim. +// +// ggml's own kernel deliberately swaps the first two result slots ("emphasize that the order is not +// important", ops.cpp), so only the SET of returned indices is contractual -- the test compares +// sorted index sets rather than positions. +TEST(GGUFOps, TopK) { + auto model = SingleOpBuilder() + .op("GGML_OP_TOP_K") + .input("x", ov::element::f32, {1, 1, 2, 5}) + .output("out", ov::element::i32, {1, 1, 2, 3}) + .build(); + + std::vector x{1, 9, 3, 7, 5, 50, 10, 40, 20, 30}; + auto out = run_on_cpu(model, {{"x", make_f32_tensor({1, 1, 2, 5}, x)}}); + + ASSERT_EQ(out.get_element_type(), ov::element::i32); + ASSERT_EQ(out.get_size(), 6u); + const int32_t* a = out.data(); + // Row 0: values 1,9,3,7,5 -> top 3 are 9,7,5 at indices 1,3,4. + // Row 1: values 50,10,40,20,30 -> top 3 are 50,40,30 at indices 0,2,4. + std::vector row0(a, a + 3), row1(a + 3, a + 6); + std::sort(row0.begin(), row0.end()); + std::sort(row1.begin(), row1.end()); + EXPECT_EQ(row0, (std::vector{1, 3, 4})); + EXPECT_EQ(row1, (std::vector{0, 2, 4})); +} + // Repeat: tile src to fill the output shape (integer multiples per axis). TEST(GGUFOps, Repeat) { auto model = SingleOpBuilder() @@ -947,6 +1133,74 @@ TEST(GGUFOps, ReshapeCase3) { expect_near(out, x, 0.0f); } +// RESHAPE op_cases 1 and 2 are the attention split/merge pair, and they must be LAYOUT-POLYMORPHIC: +// valid both for plain SDPA inference, which feeds a batch-major activation ([1, tokens, ..]), and +// after ov::pass::SDPAToPagedAttention, which moves the token count into dim 0 ([tokens, 1, ..]). +// Those two are the same buffer, so the op must copy the leading dim through rather than pin a +// literal 1 -- pinning silently rewrites a token-major activation into a batch-major one and the +// PagedAttention operands come out token-count-squared. +// +// Feeding the same values under both arrangements and requiring the same output element order is +// exactly that property, and it fails on a literal-1 reshape. +TEST(GGUFOps, ReshapeCase1SplitHeadsIsLayoutPolymorphic) { + const int64_t tokens = 2, heads = 2, head_size = 2; + const std::vector x{1, 2, 3, 4, 5, 6, 7, 8}; + + // Batch-major (what genai feeds a plain SDPA model): [1, 1, tokens, heads*head_size]. + auto sdpa = SingleOpBuilder() + .op("GGML_OP_RESHAPE") + .input("x", ov::element::f32, {1, 1, tokens, heads * head_size}) + .output("out", ov::element::f32, {1, tokens, heads, head_size}) + .op_case(1) + .build(); + auto sdpa_out = run_on_cpu(sdpa, {{"x", make_f32_tensor({1, 1, (size_t)tokens, (size_t)(heads * head_size)}, x)}}); + EXPECT_EQ(sdpa_out.get_shape(), (ov::Shape{1, (size_t)tokens, (size_t)heads, (size_t)head_size})); + + // Token-major (the layout SDPAToPagedAttention establishes): [tokens, 1, 1, heads*head_size]. + // The op must keep the tokens in dim 0 instead of moving them to dim 1. + auto pa = SingleOpBuilder() + .op("GGML_OP_RESHAPE") + .input("x", ov::element::f32, {tokens, 1, 1, heads * head_size}) + .output("out", ov::element::f32, {1, tokens, heads, head_size}) + .op_case(1) + .build(); + auto pa_out = run_on_cpu(pa, {{"x", make_f32_tensor({(size_t)tokens, 1, 1, (size_t)(heads * head_size)}, x)}}); + EXPECT_EQ(pa_out.get_shape(), (ov::Shape{(size_t)tokens, 1, (size_t)heads, (size_t)head_size})); + + // Same buffer in, same buffer out. + expect_near(sdpa_out, x, 0.0f); + expect_near(pa_out, x, 0.0f); +} + +TEST(GGUFOps, ReshapeCase2MergeHeadsIsLayoutPolymorphic) { + const int64_t tokens = 2, heads = 2, head_size = 2; + const std::vector x{1, 2, 3, 4, 5, 6, 7, 8}; + + // Non-stateful ggml keeps activations rank-4 throughout: [1, tokens, H, S] -> [1, 1, tokens, H*S]. + auto sdpa = SingleOpBuilder() + .op("GGML_OP_RESHAPE") + .input("x", ov::element::f32, {1, tokens, heads, head_size}) + .output("out", ov::element::f32, {1, 1, tokens, heads * head_size}) + .op_case(2) + .build(); + auto sdpa_out = + run_on_cpu(sdpa, {{"x", make_f32_tensor({1, (size_t)tokens, (size_t)heads, (size_t)head_size}, x)}}); + EXPECT_EQ(sdpa_out.get_shape(), (ov::Shape{1, 1, (size_t)tokens, (size_t)(heads * head_size)})); + + // Token-major: the tokens stay in dim 0. + auto pa = SingleOpBuilder() + .op("GGML_OP_RESHAPE") + .input("x", ov::element::f32, {tokens, 1, heads, head_size}) + .output("out", ov::element::f32, {1, 1, tokens, heads * head_size}) + .op_case(2) + .build(); + auto pa_out = run_on_cpu(pa, {{"x", make_f32_tensor({(size_t)tokens, 1, (size_t)heads, (size_t)head_size}, x)}}); + EXPECT_EQ(pa_out.get_shape(), (ov::Shape{(size_t)tokens, 1, 1, (size_t)(heads * head_size)})); + + expect_near(sdpa_out, x, 0.0f); + expect_near(pa_out, x, 0.0f); +} + // SET_ROWS into a flattened KV-cache row (row_size taken from the dst input, not the op output): // dst cache [1,1,ctx=3,row=2], data [1,1,n=2,row=2] written at indices {2,0}. TEST(GGUFOps, SetRowsFlattenedCache) { From 428dac005d01675423fd18351afacb51de2cbb6d Mon Sep 17 00:00:00 2001 From: Maxim Vafin Date: Fri, 14 Aug 2026 16:08:04 +0200 Subject: [PATCH 2/3] [GGUF FE] Fix Windows build and two correctness regressions Windows CI (MSVC, CMAKE_COMPILE_WARNING_AS_ERROR=ON) failed with warnings-as-errors: - utils.cpp: non_cont_dim() narrowed size_t to int on initialization (C4267). Cast explicitly instead. - test_ops.cpp: SoftMaxAlibi's slope computation narrowed std::pow's double result to float on initialization (C4244). Cast explicitly. Also fixed two correctness bugs found while reviewing the PR: - flash_attn_ext.cpp: the attention-sink reshape hardcoded axis 2 for the sink logit's target shape, but the head axis depends on op_case: it's 1 for the (currently the only reachable) llama.cpp cgraph layout and 2 for the ggml-natural layout. Use head_axis instead, and add a regression test (FlashAttnExtWithSinksCgraphLayout) that fails with a shape-mismatch exception without the fix. - translate_session.cpp: translate_graph pushed every get_model_inputs() entry's dynamic_pointer_cast into params unconditionally, dropping the previous null guard. decoder.hpp explicitly still allows a decoder to fold non-Parameter auxiliary inputs into get_model_inputs() (rather than splitting them into the new get_model_extra_inputs()), which is what the currently-shipping llama.cpp cgraph decoder does; that shape of input crashed conversion. Restored the guard and added a regression test (GetModelInputsToleratesNonParameterEntries). Verified locally: built openvino_gguf_frontend + ov_gguf_frontend_tests against the CPU plugin; all 139/139 tests pass (137 existing + 2 new). Confirmed each new test fails without its corresponding fix. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- src/frontends/gguf/src/op/flash_attn_ext.cpp | 3 +- src/frontends/gguf/src/translate_session.cpp | 8 ++- src/frontends/gguf/src/utils.cpp | 2 +- src/frontends/gguf/tests/test_extensions.cpp | 38 +++++++++++ src/frontends/gguf/tests/test_ops.cpp | 70 +++++++++++++++++++- 5 files changed, 117 insertions(+), 4 deletions(-) diff --git a/src/frontends/gguf/src/op/flash_attn_ext.cpp b/src/frontends/gguf/src/op/flash_attn_ext.cpp index 1dea26183a77..80d3b4a92275 100644 --- a/src/frontends/gguf/src/op/flash_attn_ext.cpp +++ b/src/frontends/gguf/src/op/flash_attn_ext.cpp @@ -186,7 +186,8 @@ OutputVector translate_flash_attn_ext(const NodeContext& context) { auto sink_f16 = sink.get_element_type() != element::f16 ? std::make_shared(sink, element::f16)->output(0) : sink; - auto sink_shape = v0::Constant::create(element::i64, {4}, std::vector{1, (int64_t)q_shape[2], 1, 1}); + auto sink_shape = + v0::Constant::create(element::i64, {4}, std::vector{1, (int64_t)q_shape[head_axis], 1, 1}); auto sink_r = std::make_shared(sink_f16, sink_shape, false); sdpa = std::make_shared(q_t, k_t, diff --git a/src/frontends/gguf/src/translate_session.cpp b/src/frontends/gguf/src/translate_session.cpp index a2f2794411aa..37f3fa803403 100644 --- a/src/frontends/gguf/src/translate_session.cpp +++ b/src/frontends/gguf/src/translate_session.cpp @@ -142,7 +142,13 @@ std::shared_ptr TranslateSession::translate_graph(const frontend::InputMo std::set deferred_use_params; for (const auto& it : gguf_model_decoder->get_model_inputs()) { - params.push_back(std::dynamic_pointer_cast(it.second)); + // Not every decoder splits auxiliary inputs into get_model_extra_inputs(): one that still + // folds them into get_model_inputs() (see the decoder.hpp contract) hands us a mix of + // Parameters and other node types here, so guard the cast instead of pushing a null + // Parameter into `params`. + if (auto param = std::dynamic_pointer_cast(it.second)) { + params.push_back(param); + } (*tensor_map)[it.first] = it.second; } diff --git a/src/frontends/gguf/src/utils.cpp b/src/frontends/gguf/src/utils.cpp index 5387e48a593b..506286751b31 100644 --- a/src/frontends/gguf/src/utils.cpp +++ b/src/frontends/gguf/src/utils.cpp @@ -35,7 +35,7 @@ void num_inputs_check(const NodeContext& context, size_t min_inputs, size_t max_ } int non_cont_dim(std::vector ne, std::vector nb) { - int dim = nb.size() - 1; + const auto dim = static_cast(nb.size() - 1); size_t bytes = nb[dim]; for (int i = dim; i > 0; i--) { bytes *= ne[i]; diff --git a/src/frontends/gguf/tests/test_extensions.cpp b/src/frontends/gguf/tests/test_extensions.cpp index a0f79ce28223..15f3b72be7e4 100644 --- a/src/frontends/gguf/tests/test_extensions.cpp +++ b/src/frontends/gguf/tests/test_extensions.cpp @@ -29,6 +29,7 @@ #include "openvino/op/abs.hpp" #include "openvino/op/assign.hpp" #include "openvino/op/concat.hpp" +#include "openvino/op/constant.hpp" #include "openvino/op/gather.hpp" #include "openvino/op/multiply.hpp" #include "openvino/op/negative.hpp" @@ -255,6 +256,43 @@ class SplitIoDecoder : public SingleOpDecoder { std::map> m_split_extra; }; +// A decoder that folds a non-Parameter node into get_model_inputs() instead of routing it through +// get_model_extra_inputs() -- decoder.hpp's contract explicitly still allows this ("A decoder that +// folds these into get_model_inputs() leaves this empty"), which is what the llama.cpp cgraph +// decoder currently does for its auxiliary inputs. Regression test for a crash where +// TranslateSession::translate_graph pushed every get_model_inputs() entry's +// dynamic_pointer_cast into params unconditionally, so a non-Parameter entry landed as a +// null Parameter and crashed when the unused-Parameter pruning later dereferenced it. +class MixedMainInputDecoder : public SingleOpDecoder { +public: + explicit MixedMainInputDecoder(const SingleOpDecoder& base) : SingleOpDecoder(base) { + m_mixed_inputs = SingleOpDecoder::get_model_inputs(); + m_mixed_inputs["const_aux"] = ov::op::v0::Constant::create(ov::element::i32, ov::Shape{1}, {0}); + } + + const std::map>& get_model_inputs() const override { + return m_mixed_inputs; + } + +private: + std::map> m_mixed_inputs; +}; + +} // namespace + +// A decoder need not split every auxiliary input into get_model_extra_inputs(); one that still +// folds a non-Parameter node into get_model_inputs() (the decoder.hpp contract permits this, and the +// llama.cpp cgraph decoder currently relies on it) must not crash conversion. +TEST(GGUFExtensions, GetModelInputsToleratesNonParameterEntries) { + auto base = kv_cache_write_builder(); + FrontEnd fe; + auto mixed = + std::make_shared(*std::dynamic_pointer_cast(base.decoder())); + EXPECT_NO_THROW(fe.convert(fe.load(std::static_pointer_cast(mixed)))); +} + +namespace { + std::set input_names(const std::shared_ptr& model) { std::set names; for (const auto& p : model->get_parameters()) { diff --git a/src/frontends/gguf/tests/test_ops.cpp b/src/frontends/gguf/tests/test_ops.cpp index e1408a95f513..0a1346638ffb 100644 --- a/src/frontends/gguf/tests/test_ops.cpp +++ b/src/frontends/gguf/tests/test_ops.cpp @@ -342,7 +342,8 @@ TEST(GGUFOps, SoftMaxAlibi) { const float m1 = std::pow(2.0f, -(max_bias / 2.0f) / n_head_log2); std::vector expected(x.size()); for (uint32_t h = 0; h < n_head; ++h) { - float slope = h < n_head_log2 ? std::pow(m0, h + 1) : std::pow(m1, 2 * (h - n_head_log2) + 1); + float slope = h < n_head_log2 ? static_cast(std::pow(m0, h + 1)) + : static_cast(std::pow(m1, 2 * (h - n_head_log2) + 1)); for (size_t t = 0; t < T; ++t) { float mx = -1e30f; std::vector z(Kd); @@ -1621,6 +1622,73 @@ TEST(GGUFOps, FlashAttnExt) { expect_near(out, expected, 2e-2f); // fp16 SDPA } +// FlashAttnExt with gpt-oss attention sinks (5th input), on the default op_case 0 (llama.cpp cgraph) +// layout where q/k/v already arrive as [B, n_head, T, D]. Regression test for a bug where the sink +// logit was reshaped to [1, q_shape[2], 1, 1] instead of [1, q_shape[head_axis], 1, 1]: q_shape[2] is +// T on this layout (head_axis == 1), not n_head, so with n_head != T the reshape either throws a +// shape mismatch or silently broadcasts the wrong values into the softmax denominator. Using +// n_head=2 and T=3 (both != 1, and different from each other) makes either failure mode observable. +TEST(GGUFOps, FlashAttnExtWithSinksCgraphLayout) { + const size_t n_head = 2, T = 3, Tk = 3, D = 2; + const float scale = 1.0f; + auto model = SingleOpBuilder() + .op("GGML_OP_FLASH_ATTN_EXT") + .input("q", ov::element::f32, {1, n_head, T, D}) + .input("k", ov::element::f32, {1, n_head, Tk, D}) + .input("v", ov::element::f32, {1, n_head, Tk, D}) + .input("mask", ov::element::f32, {1, 1, T, Tk}) + .input("sinks", ov::element::f32, {n_head}) + .output("out", ov::element::f32, {1, T, n_head, D}) + .attr("scale", scale) + .build(); + + std::vector q, k, v; + for (size_t i = 0; i < n_head * T * D; ++i) + q.push_back(0.1f * static_cast(i) - 1.0f); + for (size_t i = 0; i < n_head * Tk * D; ++i) { + k.push_back(0.2f * static_cast(i) - 1.0f); + v.push_back(0.05f * static_cast(i)); + } + std::vector mask(T * Tk, 0.0f); + std::vector sinks{0.5f, -0.3f}; + + auto out = run_on_cpu(model, + {{"q", make_f32_tensor({1, n_head, T, D}, q)}, + {"k", make_f32_tensor({1, n_head, Tk, D}, k)}, + {"v", make_f32_tensor({1, n_head, Tk, D}, v)}, + {"mask", make_f32_tensor({1, 1, T, Tk}, mask)}, + {"sinks", make_f32_tensor({n_head}, sinks)}}); + + // Reference: per head, augment the softmax denominator with the head's sink logit (dropped from + // the weighted value sum), same convention as SoftMaxSinks. Output layout [1,T,H,D]. + std::vector expected(T * n_head * D); + for (size_t h = 0; h < n_head; ++h) { + for (size_t t = 0; t < T; ++t) { + std::vector z(Tk + 1); + for (size_t s = 0; s < Tk; ++s) { + float dot = 0.f; + for (size_t d = 0; d < D; ++d) + dot += q[(h * T + t) * D + d] * k[(h * Tk + s) * D + d]; + z[s] = scale * dot + mask[t * Tk + s]; + } + z[Tk] = sinks[h]; + float mx = *std::max_element(z.begin(), z.end()); + float sum = 0.f; + for (float& zi : z) { + zi = std::exp(zi - mx); + sum += zi; + } + for (size_t d = 0; d < D; ++d) { + float acc = 0.f; + for (size_t s = 0; s < Tk; ++s) + acc += (z[s] / sum) * v[(h * Tk + s) * D + d]; + expected[(t * n_head + h) * D + d] = acc; + } + } + } + expect_near(out, expected, 2e-2f); // fp16 SDPA +} + // GatedDeltaNet, reference (Loop) path. With head size S=1 the gate last-dim equals S_v, so this is // the per-key-dimension gating case (kda) that the fused op does not support and which therefore // lowers to the serializable Loop scan. Minimal scalar case B=H=S=1, T=2 exercises the full From f720f5b458ef1a8520a33c3650ae503b270cb6d1 Mon Sep 17 00:00:00 2001 From: Maxim Vafin Date: Fri, 14 Aug 2026 16:32:38 +0200 Subject: [PATCH 3/3] [GGUF FE] Address Copilot review comments - tests/CMakeLists.txt: remove the duplicate op/top_k.cpp entry in FRONTEND_SRCS. - frontend.hpp: supported_impl/load_impl docstrings claimed .gguf file-path loading (magic-byte sniffing, native builder dispatch), but this PR's frontend.cpp only recognizes a std::shared_ptr -- file-path loading is part of the separate, stacked native .gguf builder PR. Reworded both docstrings to match the actual implementation. Verified locally: rebuilt ov_gguf_frontend_tests against the CPU plugin after both changes; all 139/139 tests still pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../openvino/frontend/gguf/frontend.hpp | 18 ++++++++---------- src/frontends/gguf/tests/CMakeLists.txt | 1 - 2 files changed, 8 insertions(+), 11 deletions(-) diff --git a/src/frontends/gguf/include/openvino/frontend/gguf/frontend.hpp b/src/frontends/gguf/include/openvino/frontend/gguf/frontend.hpp index 1c750bd5b887..c158bb5d4701 100644 --- a/src/frontends/gguf/include/openvino/frontend/gguf/frontend.hpp +++ b/src/frontends/gguf/include/openvino/frontend/gguf/frontend.hpp @@ -49,18 +49,16 @@ class GGUF_FRONTEND_API FrontEnd : public ov::frontend::FrontEnd { protected: /// \brief Check if FrontEnd can recognize the model from the given parts. - /// \param variants Either a `std::shared_ptr`, or a path to a file whose extension - /// is `.gguf` and whose first four bytes are the GGUF magic. - /// \return True for either of those; false otherwise. + /// \param variants A single element holding a `std::shared_ptr`. No other variant + /// is recognized in this frontend: file-path (`.gguf`) loading is not yet implemented. + /// \return True iff variants holds exactly that; false otherwise. bool supported_impl(const std::vector& variants) const override; - /// \brief Load the input model, from either of the frontend's two ingest paths. - /// \param variants A single element, holding either: - /// - a `std::shared_ptr` — a decoder supplied by a direct linker, wrapping - /// an already-built ggml graph (the llama.cpp cgraph path); or - /// - a path to a `.gguf` file — parsed here, with the transformer graph built - /// per-architecture by the native builder. - /// Both yield a GgufDecoder, so conversion past this point is identical. + /// \brief Load the input model from a GgufDecoder. + /// \param variants A single element holding a `std::shared_ptr` -- a decoder + /// supplied by a direct linker, wrapping an already-built ggml graph (the llama.cpp + /// cgraph path). File-path (`.gguf`) loading, built per-architecture by the native + /// builder, is not yet implemented in this frontend. /// \return InputModel::Ptr InputModel::Ptr load_impl(const std::vector& variants) const override; diff --git a/src/frontends/gguf/tests/CMakeLists.txt b/src/frontends/gguf/tests/CMakeLists.txt index ea4fed3fc052..f02ad9469071 100644 --- a/src/frontends/gguf/tests/CMakeLists.txt +++ b/src/frontends/gguf/tests/CMakeLists.txt @@ -53,7 +53,6 @@ set(FRONTEND_SRCS "${FE_SRC_DIR}/op/top_k.cpp" "${FE_SRC_DIR}/op/tri.cpp" "${FE_SRC_DIR}/op/sum_rows.cpp" - "${FE_SRC_DIR}/op/top_k.cpp" "${FE_SRC_DIR}/op/transpose.cpp" "${FE_SRC_DIR}/op/unary_elu.cpp" "${FE_SRC_DIR}/op/unary_gelu.cpp"