diff --git a/.claude/skills/ov-gguf-add-architecture/SKILL.md b/.claude/skills/ov-gguf-add-architecture/SKILL.md new file mode 100644 index 00000000000000..8e4bfb20099084 --- /dev/null +++ b/.claude/skills/ov-gguf-add-architecture/SKILL.md @@ -0,0 +1,15 @@ +--- +name: ov-gguf-add-architecture +description: > + Enable a new model architecture/family in the OpenVINO GGUF frontend's native .gguf builder, + or check whether a GGUF model is supported. Use when the user asks to enable, support or bring + up a GGUF/llama.cpp model (llama, qwen, phi, gemma, MoE and similar decoder-only families), + when a .gguf file is rejected as an unsupported architecture, or when working on + arch_registry / DecoderBuilder in src/frontends/gguf/src/builder/. Do NOT use for adding + a single ggml op translator or for debugging wrong output from an already-supported model. +--- + +1. Read [src/frontends/gguf/docs/adding_an_architecture.md](../../../src/frontends/gguf/docs/adding_an_architecture.md) — most architectures in the transformer family need **no code**, only a `general.architecture` string added to `arch_registry.cpp` plus the correct RoPE type. It also lists exactly which structural features are auto-detected from the tensor table and metadata, and how to verify a new arch. +2. Check [src/frontends/gguf/docs/supported_models.md](../../../src/frontends/gguf/docs/supported_models.md) first to see whether the architecture is already accepted and how support was verified. +3. Only if the family is structurally novel, consult the "10% case" section and [frontend_design.md](../../../src/frontends/gguf/docs/frontend_design.md) before adding builder code. +4. Verify as that document prescribes, including the graph-fingerprint check, and re-run the other supported architectures after any change to shared builder logic. diff --git a/.claude/skills/ov-gguf-debug-accuracy/SKILL.md b/.claude/skills/ov-gguf-debug-accuracy/SKILL.md new file mode 100644 index 00000000000000..503b9815e0accb --- /dev/null +++ b/.claude/skills/ov-gguf-debug-accuracy/SKILL.md @@ -0,0 +1,15 @@ +--- +name: ov-gguf-debug-accuracy +description: > + Debug wrong or degraded output from a GGUF model running through the OpenVINO GGUF frontend or + the ggml-openvino backend. Use when a GGUF model converts but produces garbage, repeated or + drifting tokens, a cosine-similarity cliff between layers, output that diverges from llama.cpp + CPU, or a broadcast/shape crash that appears only at decode or only for one architecture. Do + NOT use for ops that fail to convert at all ("Translation for operation type ... is not + implemented"), or for build and CMake failures. +--- + +1. Read [src/frontends/gguf/docs/debugging_accuracy.md](../../../src/frontends/gguf/docs/debugging_accuracy.md) — the coarse-to-fine bisection strategy, the ggml-CPU oracle technique, the catalogue of bug archetypes with their generalizable lessons, the debug env vars, and the checklist. +2. Apply its one governing rule before anything else: every accuracy claim is a comparison against the **real llama.cpp CPU implementation**, never a hand-derived reference. If you are about to write out an op's math to form an expectation, generate it from ggml instead. +3. Work the bisection steps in order and do not open a debugger until they have cornered the bug. Check the bug archetypes first when a cosine cliff or dynamic-shape crash points near one. +4. After any fix to a shared path — especially a VIEW or `op_case` predicate — re-run the other supported architectures and confirm their classification and output are unchanged. diff --git a/.claude/skills/ov-gguf-enable-op/SKILL.md b/.claude/skills/ov-gguf-enable-op/SKILL.md new file mode 100644 index 00000000000000..7bbe765ef04cde --- /dev/null +++ b/.claude/skills/ov-gguf-enable-op/SKILL.md @@ -0,0 +1,15 @@ +--- +name: ov-gguf-enable-op +description: > + Enable a ggml operation in the OpenVINO GGUF frontend by adding or fixing an op translator. + Use when conversion fails with "Translation for operation type GGML_OP_* is not implemented", + when the user asks to add/implement/fix a GGUF or ggml op translator or an op_case, or when + working on src/frontends/gguf/src/op/. Do NOT use for enabling a new model architecture + (that is usually just a name in arch_registry.cpp), for wrong numerical output from an op that + already converts, or for GGUF quantization format work. +--- + +1. Read [src/frontends/gguf/docs/how_to_add_op.md](../../../src/frontends/gguf/docs/how_to_add_op.md) — the five-file checklist (including the test CMake source list, which is not globbed), the `NodeContext` API, the mandatory op-coverage gate, where reference values must come from, and the build/test commands (`-DENABLE_OV_GGUF_FRONTEND=ON`). +2. Confirm a translator is really what is missing: a new architecture usually needs only a name in `arch_registry.cpp`, and a structurally different use of an existing op is an `op_case`. Both are covered in that document. +3. Find the closest existing op with `grep -n "GGML_" src/frontends/gguf/src/op_table.cpp` and the closest test with `grep -n "^TEST(" src/frontends/gguf/tests/test_ops.cpp`, then read only those ranges — do not read `test_ops.cpp` in full. +4. Implement, then build and run `ov_gguf_frontend_tests` — filtered while iterating, unfiltered before finishing so the coverage gate runs. diff --git a/.github/components.yml b/.github/components.yml index 0aa1ce03da2d9c..6e7a7896e45d39 100644 --- a/.github/components.yml +++ b/.github/components.yml @@ -107,6 +107,12 @@ PROXY: - GPU build: [] +GGUF_FE: + revalidate: + - CPU # the per-op tests compile and infer their converted graphs on the CPU plugin + build: + - CPU + IR_FE: revalidate: - C_API diff --git a/.github/coverage/tests_cpp.yml b/.github/coverage/tests_cpp.yml index ba6ea85498c4c6..4464a7cbf671c6 100644 --- a/.github/coverage/tests_cpp.yml +++ b/.github/coverage/tests_cpp.yml @@ -98,6 +98,11 @@ tests: mode: gtest_single profiles: [cpu] + - name: ov_gguf_frontend_tests + binary: ov_gguf_frontend_tests + mode: gtest_single + profiles: [cpu] + - name: ov_inference_functional_tests binary: ov_inference_functional_tests mode: gtest_single diff --git a/.github/labeler.yml b/.github/labeler.yml index 00a3ca40bc24c9..53172b5b27891d 100644 --- a/.github/labeler.yml +++ b/.github/labeler.yml @@ -71,6 +71,11 @@ - 'src/frontends/common/include/openvino/frontend/extension.hpp' - 'src/frontends/common/include/openvino/frontend/extension/**/*' +'category: GGUF FE': +- 'src/frontends/gguf/**/*' +- 'tests/model_hub_tests/gguf/**/*' +- 'tests/requirements_gguf' + 'category: GPU': - 'src/plugins/intel_gpu/**/*' - 'thirdparty/ocl/**/*' @@ -154,7 +159,8 @@ - 'tests/requirements_tensorflow' - any: ['tests/model_hub_tests/**', '!tests/model_hub_tests/pytorch/**/*', - '!tests/model_hub_tests/jax/**/*'] + '!tests/model_hub_tests/jax/**/*', + '!tests/model_hub_tests/gguf/**/*'] 'category: TFL FE': - 'src/frontends/tensorflow_lite/**/*' @@ -170,7 +176,8 @@ - 'tests/requirements_pytorch' - any: ['tests/model_hub_tests/**', '!tests/model_hub_tests/tensorflow/**/*', - '!tests/model_hub_tests/jax/**/*'] + '!tests/model_hub_tests/jax/**/*', + '!tests/model_hub_tests/gguf/**/*'] 'category: JAX FE': - 'src/frontends/jax/**/*' @@ -178,7 +185,8 @@ - 'tests/layer_tests/jax_tests/**/*' - any: ['tests/model_hub_tests/**', '!tests/model_hub_tests/tensorflow/**/*', - '!tests/model_hub_tests/pytorch/**/*'] + '!tests/model_hub_tests/pytorch/**/*', + '!tests/model_hub_tests/gguf/**/*'] 'category: tools': - any: ['tools/**', diff --git a/.github/workflows/job_build_linux.yml b/.github/workflows/job_build_linux.yml index adbb18dec0d5fa..bf3b43ebb63df9 100644 --- a/.github/workflows/job_build_linux.yml +++ b/.github/workflows/job_build_linux.yml @@ -85,6 +85,11 @@ on: type: boolean required: false default: true + generate-gguf-fixtures: + description: 'Whether to generate the GGUF per-architecture test fixtures from llama.cpp into the tests artifact. Off by default: only useful where this build feeds job_cxx_unit_tests.yml' + type: boolean + required: false + default: false permissions: read-all @@ -249,6 +254,25 @@ jobs: cmake --install ${BUILD_DIR} --config ${{ env.CMAKE_BUILD_TYPE }} --prefix ${INSTALL_TEST_DIR} --component tests cmake --install ${BUILD_DIR} --config ${{ env.CMAKE_BUILD_TYPE }} --prefix ${DEVELOPER_PACKAGE_DIR} --component developer_package + # The GGUF per-architecture fixtures are GGUF headers emitted by llama.cpp's test-llama-archs. + # They are generated here rather than committed, and land in the tests artifact so the test job + # (which has neither a source checkout nor cmake) can consume them like any other test data. + # Where they are absent the arch test suite skips itself, so this is test data, not a build + # dependency -- which is what keeps llama.cpp out of OpenVINO's dependency graph. + # + # llama.cpp is pinned inside gen_arch_fixtures.py and only moves when the fixtures are + # deliberately refreshed: the generator writes whatever KVs llama.cpp currently defines, so + # following upstream would change every fixture's bytes -- and break the pinned graph + # fingerprints -- on an unrelated llama.cpp commit. + - name: Generate GGUF arch fixtures + if: ${{ inputs.generate-gguf-fixtures && fromJSON(inputs.affected-components).GGUF_FE.test }} + run: | + python3 -m pip install --no-cache-dir gguf + python3 ${OPENVINO_REPO}/src/frontends/gguf/tests/gen_arch_fixtures.py \ + --fetch \ + --out-dir ${INSTALL_TEST_DIR}/tests/test_data/arch_fixtures \ + -j $(nproc) + - name: Install Python wheels for the main Python if: ${{ ! inputs.build-additional-python-packages }} run: cmake --install ${BUILD_DIR} --config ${{ env.CMAKE_BUILD_TYPE }} --prefix ${INSTALL_WHEELS_DIR} --component python_wheels diff --git a/.github/workflows/job_cxx_unit_tests.yml b/.github/workflows/job_cxx_unit_tests.yml index 8385051407758f..fb88285f5d2383 100644 --- a/.github/workflows/job_cxx_unit_tests.yml +++ b/.github/workflows/job_cxx_unit_tests.yml @@ -129,6 +129,7 @@ jobs: ${{ env.INSTALL_TEST_DIR }}/ov_ir_frontend_tests --gtest_print_time=1 --gtest_output=xml:${{ env.INSTALL_TEST_DIR }}/TEST-IRFrontend.xml - name: GGUF frontend tests + if: fromJSON(inputs.affected-components).GGUF_FE.test run: | ${{ env.SOURCE_COMMAND }} ${{ env.SETUPVARS }} ${{ env.INSTALL_TEST_DIR }}/ov_gguf_frontend_tests --gtest_print_time=1 --gtest_output=xml:${{ env.INSTALL_TEST_DIR }}/TEST-GGUFFrontend.xml diff --git a/.github/workflows/job_gguf_models_tests.yml b/.github/workflows/job_gguf_models_tests.yml new file mode 100644 index 00000000000000..954be7740416fa --- /dev/null +++ b/.github/workflows/job_gguf_models_tests.yml @@ -0,0 +1,116 @@ +name: GGUF Models tests + +on: + workflow_call: + inputs: + runner: + description: 'Machine on which the tests would run' + type: string + required: true + image: + description: 'Docker image to use for the job' + type: string + required: false + default: null + model_scope: + description: 'Scope of models for testing.' + type: string + required: true + +permissions: read-all + +env: + HF_HUB_CACHE_LIN: /mount/caches/huggingface + HF_HUB_CACHE_WIN: "C:\\mount\\caches\\huggingface" + HF_TOKEN_PATH_LIN: /secrets/huggingface/token-secondary + HF_TOKEN_PATH_WIN: "C:\\mount\\secrets\\huggingface\\token-secondary" + +jobs: + GGUF_Models_Tests: + name: GGUF Models tests + # The nightly scope downloads several 7B-30B checkpoints, so it needs far more time + # than the precommit scope (18 models of <=~4B). + timeout-minutes: ${{ inputs.model_scope == 'precommit' && 60 || 240 }} + runs-on: ${{ inputs.runner }} + container: + image: ${{ inputs.image }} + volumes: + - /mount:/mount + - /home/runner/secrets/:/secrets:ro + - ${{ github.workspace }}:${{ github.workspace }} # Needed as ${{ github.workspace }} is not working correctly when using Docker + defaults: + run: + shell: bash + env: + DEBIAN_FRONTEND: noninteractive # to prevent apt-get from waiting user input + INSTALL_DIR: ${{ github.workspace }}/install + INSTALL_TEST_DIR: ${{ github.workspace }}/install/tests + MODEL_HUB_TESTS_INSTALL_DIR: ${{ github.workspace }}/install/tests/model_hub_tests + USE_SYSTEM_CACHE: False # Using remote HuggingFace cache + HF_HUB_VERBOSITY: debug + steps: + # checkout action cleans up the workspace and have to be the first step + - name: Fetch custom actions + uses: ababushk/checkout@dd591a6a2ac25618db4eda86e7e0d938f88cf01b # cherry_pick_retries + timeout-minutes: 15 + with: + sparse-checkout: | + .github/actions + sparse-checkout-cone-mode: false + submodules: 'false' + + - name: Download OpenVINO artifacts (wheels) + uses: akashchi/download-artifact@d59a9c15fec3fdb7c9adf09464124d00f9c11415 # main + with: + name: openvino_wheels + path: ${{ env.INSTALL_DIR }} + + - name: Download OpenVINO artifacts (tests) + uses: akashchi/download-artifact@d59a9c15fec3fdb7c9adf09464124d00f9c11415 # main + with: + name: openvino_tests + path: ${{ env.INSTALL_DIR }} + + - name: Setup Variables + run: | + echo "HF_HUB_CACHE=${{ runner.os == 'Linux' && env.HF_HUB_CACHE_LIN || env.HF_HUB_CACHE_WIN }}" >> "$GITHUB_ENV" + echo "HUGGINGFACE_HUB_CACHE=${{ runner.os == 'Linux' && env.HF_HUB_CACHE_LIN || env.HF_HUB_CACHE_WIN }}" >> "$GITHUB_ENV" + echo "HF_TOKEN_PATH=${{ runner.os == 'Linux' && env.HF_TOKEN_PATH_LIN || env.HF_TOKEN_PATH_WIN }}" >> "$GITHUB_ENV" + + - name: Extract OpenVINO packages and tests + run: pigz -dc openvino_tests.tar.gz | tar -xf - -v + working-directory: ${{ env.INSTALL_DIR }} + + - name: Setup Python 3.11 + uses: ./.github/actions/setup_python + with: + version: '3.11' + set-pip-install-path: 'false' + self-hosted-runner: ${{ contains(inputs.runner, 'aks') }} + use-pip-proxy: ${{ contains(inputs.runner, 'aks') }} + + - name: Install OpenVINO Python wheels + uses: ./.github/actions/install_ov_wheels + with: + wheels-dir-path: ${{ env.INSTALL_DIR }} + wheels-to-install: 'openvino' + + - name: Install GGUF Models tests requirements + run: python3 -m pip install -r ${INSTALL_TEST_DIR}/requirements_gguf + + - name: GGUF Models Tests from Hugging Face + run: | + export PYTHONPATH=${MODEL_HUB_TESTS_INSTALL_DIR}:$PYTHONPATH + python3 -m pytest ${MODEL_HUB_TESTS_INSTALL_DIR}/gguf/ -m ${TYPE} --html=${INSTALL_TEST_DIR}/TEST-gguf_model_"${TYPE}"_tests.html --self-contained-html -v + env: + TYPE: ${{ inputs.model_scope == 'precommit' && 'precommit' || 'nightly' }} + TEST_DEVICE: CPU + + - name: Upload Test Results + uses: ababushk/upload-artifact@ebc7d74ace101c08868aed05dba2aaf274b9a2c7 # main + if: ${{ !cancelled() }} + with: + name: test-results-gguf-models-${{ inputs.model_scope }} + path: | + ${{ env.INSTALL_TEST_DIR }}/TEST*.html + if-no-files-found: 'error' diff --git a/.github/workflows/linux_sanitizers.yml b/.github/workflows/linux_sanitizers.yml index 258618af73f15c..fcc1aa853a4cdc 100644 --- a/.github/workflows/linux_sanitizers.yml +++ b/.github/workflows/linux_sanitizers.yml @@ -220,6 +220,17 @@ jobs: cmake -DCMAKE_INSTALL_PREFIX=${INSTALL_DIR} -P ${BUILD_DIR}/cmake_install.cmake --config ${{ env.CMAKE_BUILD_TYPE }} cmake -DCMAKE_INSTALL_PREFIX=${INSTALL_TEST_DIR} -DCOMPONENT=tests -P ${BUILD_DIR}/cmake_install.cmake --config ${{ env.CMAKE_BUILD_TYPE }} + # Same generated test data as the regular Linux build (see job_build_linux.yml); running the 101 + # per-architecture conversions under the sanitizers is the highest-value place for them. The + # generator's own llama.cpp build is not instrumented, which is fine -- only its output is used. + - name: Generate GGUF arch fixtures + run: | + python3 -m pip install --no-cache-dir gguf + python3 ${OPENVINO_REPO}/src/frontends/gguf/tests/gen_arch_fixtures.py \ + --fetch \ + --out-dir ${INSTALL_TEST_DIR}/tests/test_data/arch_fixtures \ + -j $(nproc) + - name: Remove unused files to free space run: rm -rf ${BUILD_DIR}/* @@ -350,6 +361,13 @@ jobs: ${INSTALL_TEST_DIR}/ov_conditional_compilation_tests --gtest_print_time=1 \ --gtest_output=xml:${INSTALL_TEST_DIR}/TEST-ConditionalCompilation.xml + - name: GGUF frontend tests + if: ${{ !cancelled() }} + run: | + source ${INSTALL_DIR}/setupvars.sh + ${INSTALL_TEST_DIR}/ov_gguf_frontend_tests --gtest_print_time=1 \ + --gtest_output=xml:${INSTALL_TEST_DIR}/TEST-GGUFFrontend.xml + - name: IR frontend tests if: ${{ !cancelled() }} run: | diff --git a/.github/workflows/ubuntu_22.yml b/.github/workflows/ubuntu_22.yml index fe67b90f408417..08cf18e1e7583e 100644 --- a/.github/workflows/ubuntu_22.yml +++ b/.github/workflows/ubuntu_22.yml @@ -131,6 +131,7 @@ jobs: build-debian-packages: true build-rpm-packages: false build-additional-python-packages: true + generate-gguf-fixtures: true target-branch: ${{ inputs.target-branch }} cmake-options: >- -G 'Ninja Multi-Config' @@ -493,6 +494,26 @@ jobs: model_scope: 'precommit' image: ${{ fromJSON(needs.docker.outputs.images).ov_test.ubuntu_22_04_x64 }} + GGUF_Models_Tests_Precommit: + name: GGUF Models tests + if: ${{ github.event_name != 'schedule' && fromJSON(needs.smart_ci.outputs.affected_components).GGUF_FE.test }} + needs: [ Docker, Build, Smart_CI ] + uses: ./.github/workflows/job_gguf_models_tests.yml + with: + runner: 'aks-linux-8-cores-32gb' + model_scope: 'precommit' + image: ${{ fromJSON(needs.docker.outputs.images).ov_test.ubuntu_22_04_x64 }} + + GGUF_Models_Tests_Nightly: + name: GGUF Models Nightly tests + if: ${{ github.event_name == 'schedule' }} + needs: [ Docker, Build, Smart_CI ] + uses: ./.github/workflows/job_gguf_models_tests.yml + with: + runner: 'aks-linux-16-cores-64gb' + model_scope: 'nightly' + image: ${{ fromJSON(needs.docker.outputs.images).ov_test.ubuntu_22_04_x64 }} + NVIDIA_Plugin: name: NVIDIA plugin needs: [ Docker, Build, Smart_CI ] @@ -660,7 +681,8 @@ jobs: Overall_Status: name: ci/gha_overall_status needs: [Smart_CI, Build, Debian_Packages, Samples, Conformance, CXX_Unit_Tests, Python_Unit_Tests, TensorFlow_Layer_Tests, Pytorch_Layer_Tests, - CPU_Functional_Tests, TensorFlow_Models_Tests_Precommit, PyTorch_Models_Tests, JAX_Models_Tests_Precommit, NVIDIA_Plugin, Openvino_tokenizers, iGPU, + CPU_Functional_Tests, TensorFlow_Models_Tests_Precommit, PyTorch_Models_Tests, JAX_Models_Tests_Precommit, GGUF_Models_Tests_Precommit, + NVIDIA_Plugin, Openvino_tokenizers, iGPU, Keras3_OpenVINO_Backend, iGPU_RoPE_Tests] if: ${{ always() }} runs-on: ubuntu-latest diff --git a/.github/workflows/ubuntu_24.yml b/.github/workflows/ubuntu_24.yml index acc4ec98e47afc..e71127ab2d2422 100644 --- a/.github/workflows/ubuntu_24.yml +++ b/.github/workflows/ubuntu_24.yml @@ -109,6 +109,7 @@ jobs: build-debian-packages: true build-contrib: true build-additional-python-packages: true + generate-gguf-fixtures: true target-branch: ${{ inputs.target-branch }} cmake-options: >- -G 'Ninja Multi-Config' diff --git a/src/frontends/common/src/manager.cpp b/src/frontends/common/src/manager.cpp index a97616829fbcbf..9ea2c93d3f7de3 100644 --- a/src/frontends/common/src/manager.cpp +++ b/src/frontends/common/src/manager.cpp @@ -19,9 +19,10 @@ using namespace ov; using namespace ov::frontend; namespace { -// Frontends for direct linkage only: loadable, but never listed by available_front_ends() nor -// auto-selected by load_by_model / load_by_framework. A manager-side list (not a plugin-info flag) -// keeps the public FrontEndPluginInfo struct / ABI unchanged. +// Hidden frontends are absent from available_front_ends() and skipped by load_by_model, so they +// are never picked up implicitly (notably by core.read_model), but stay loadable on explicit +// request via load_by_framework(). Kept manager-side to leave FrontEndPluginInfo's ABI unchanged. +// "gguf" is listed because reading GGUF through core.read_model is not supported yet. bool is_hidden_frontend(const std::string& name) { static const std::set hidden_frontends = {"gguf"}; return hidden_frontends.count(name) != 0; @@ -71,10 +72,7 @@ class FrontEndManager::Impl { // Load plugins until we found the right one for (auto& plugin : m_plugins) { OPENVINO_ASSERT(plugin.load(), "Cannot load frontend ", plugin.get_name_from_file()); - // Hidden frontends are not selectable by name through the generic API. - if (is_hidden_frontend(plugin.get_creator().m_name)) { - continue; - } + // Not filtered by is_hidden_frontend: asking for a frontend by name is explicit. if (plugin.get_creator().m_name == framework) { return make_frontend(plugin); } diff --git a/src/frontends/gguf/.gitignore b/src/frontends/gguf/.gitignore new file mode 100644 index 00000000000000..f671f608a0a3d5 --- /dev/null +++ b/src/frontends/gguf/.gitignore @@ -0,0 +1,3 @@ +# The root .gitignore's "[Bb]uild*/" rule (meant for CMake build output dirs) also matches +# the src/builder/ source directory by name. Re-include it; this is tracked source code. +!src/builder diff --git a/src/frontends/gguf/docs/adding_an_architecture.md b/src/frontends/gguf/docs/adding_an_architecture.md new file mode 100644 index 00000000000000..5da6fccd3999c3 --- /dev/null +++ b/src/frontends/gguf/docs/adding_an_architecture.md @@ -0,0 +1,153 @@ +# Adding a new architecture to the GGUF frontend + +The native `.gguf` path builds an OpenVINO graph from a GGUF file with **no llama.cpp +dependency**. It emits nodes in the GGML op vocabulary (`GGML_OP_MUL_MAT`, `GGML_OP_ROPE`, +`GGML_OP_FLASH_ATTN_EXT`, ...) that reproduce llama.cpp's cgraph topology, so the same op +translators (`src/op/*.cpp`) run for both the native path and the llama.cpp cgraph path. + +## How the builder is laid out + +`src/builder/` is layered; each layer knows strictly less than the one above it: + +| File | Responsibility | Knows about | +|---|---|---| +| [`graph_emitter.hpp`](../src/builder/graph_emitter.hpp) | `add_op` / `add_input` / `add_weight` + shape & type bookkeeping | nothing about transformers | +| [`blocks/`](../src/builder/blocks) | reusable graph fragments: `common` (norm/scale/bias), `ffn` (dense/GeGLU/MoE), `attention`, `gated_delta_net`, `qkv_repack` | a decoder layer | +| [`decoder_config.hpp`](../src/builder/decoder_config.hpp) | all per-architecture detection + per-layer accessors | one model's hyperparameters | +| [`arch/decoder_builder.cpp`](../src/builder/arch/decoder_builder.cpp) | the order a decoder is assembled in | the whole decoder family | +| [`arch_registry.cpp`](../src/builder/arch_registry.cpp) | which architectures are accepted, and their RoPE mode | names only | +| [`model_kind.hpp`](../src/builder/model_kind.hpp) | which model *family* a file holds | raw metadata | +| [`gguf_builder.cpp`](../src/builder/gguf_builder.cpp) | parse → detect family → dispatch to a `ModelBuilder` | the entry point | + +A single generic `DecoderBuilder` covers the whole "llama family" of decoder-only transformers. +This is deliberately **not** llama.cpp's one-file-per-architecture layout: llama.cpp needs that +because every architecture enumerates its tensors by hand, whereas this builder derives them from +the tensor table, so a same-family architecture costs zero lines of code. + +## The 90% case: add a name + +Most new architectures in the transformer family need **no code** — the builder auto-detects +their structure from the GGUF tensor table and metadata. To enable one, add its +`general.architecture` string to `verified_archs()` (or `experimental_archs()`) in +[`arch_registry.cpp`](../src/builder/arch_registry.cpp): + +```cpp +const std::set& experimental_archs() { + static const std::set archs = { + "llama-embed", "exaone4", ..., + "your-arch", // <-- add here + }; + return archs; +} +``` + +Then check whether RoPE is NEOX (rotate-halves) or NORMAL (rotate consecutive pairs) for the +arch and, if NEOX, add it to `arch_uses_neox_rope()` in the same file (mirror +`llama_model_rope_type` in llama.cpp). That is the whole change for a same-family arch. + +### What is auto-detected (no code needed) + +`DecoderConfig` infers structure from the presence of layer-0 weight tensors and from metadata: + +| Feature | Detected from | +|---|---| +| Per-head Q/K norm (qwen3, hunyuan) | `blk.0.attn_q_norm.weight` | +| Full-width Q/K norm (OLMoE) | `attn_q_norm.weight` width == `n_head*head_size` | +| Q/K/V projection biases (qwen2) | `blk.0.attn_q.bias` | +| Output-projection bias | `blk.0.attn_output.bias` | +| Fused QKV (phi-3, minicpm) | `blk.0.attn_qkv.weight` | +| Fused gate+up FFN (phi-3) | absence of `blk.0.ffn_gate.weight` | +| MoE routing (OLMoE, gpt-oss, qwen3moe) | `blk..ffn_gate_exps.weight` | +| Shared experts | `expert_shared_count` metadata + `ffn_*_shexp.weight` | +| Hybrid dense-lead MoE | `leading_dense_block_count` metadata | +| RoPE freq factors (llama-3, phi-3) | `rope_freqs.weight` | +| Scalar scales (minicpm) | `embedding_scale` / `residual_scale` / `logit_scale` metadata | +| Soft-caps (gemma2/3) | `attn_logit_softcapping` / `final_logit_softcapping` metadata | +| Sliding-window attention | `attention.sliding_window(_pattern)` metadata, or sinks | +| Per-layer KV heads | `attention.head_count_kv` as an array | + +All hyperparameters are read once in `decoder_config_from_meta()` +([`src/quant/gguf.cpp`](../src/quant/gguf.cpp)) and resolved once in `DecoderConfig`; the topology +builder never re-reads GGUF metadata. + +## Per-layer values: use the accessors, don't inline + +Architectures with per-layer variation (SWA layers, variable KV heads, per-layer head sizes) +are handled by the per-layer accessors on `DecoderConfig` — the single source of truth, +so the topology stays declarative: + +- `layer_is_swa(il)` — sliding-window layer? (per-layer flag array or period) +- `layer_head_size(il)` — head size (SWA layers may differ, e.g. gemma4) +- `layer_n_head_kv(il)` — KV head count (may vary per layer) +- `layer_kq_scale(il)` — attention softmax scale (`1/sqrt(layer_head_size(il))` unless overridden) +- `layer_rope_config(il)` — RoPE config (SWA layers may use a different freq_base / n_dims) +- `is_recurrent_layer(il)` — linear-attention (GDN) layer of a hybrid stack + +If a new arch adds a per-layer dimension, extend these accessors rather than adding a new +inline ternary in `build_layer()`. + +## The 10% case: structurally novel architectures need code + +The generic decoder builder assumes the standard block: `norm -> QKV -> RoPE -> attention -> +norm -> FFN/MoE -> residual`. Architectures that break this shape need a new detected flag on +`DecoderConfig` plus a branch in the relevant block. Examples of what required code in the past: + +- **MoE routing** — `blocks::moe_ffn()` (`MUL_MAT_ID` / `GatherMatmul`, top-k, gated activation). +- **gpt-oss** — attention sinks (5th `FLASH_ATTN_EXT` input), OAI gated activation + (`GGML_GLU_OP_SWIGLU_OAI`), softmax-after-topk gating. +- **gemma2/3** — post-attention / post-FFN norms, attention & final-logit soft-caps. +- **gemma4** — per-layer input embeddings, shared-KV layers, per-op RoPE (SWA vs global differ). +- **qwen35** — a hybrid stack where 3 of every 4 layers are a Gated DeltaNet block + (`blocks::gated_delta_net()`) instead of attention. It returns the same thing the attention + block returns — the sublayer output before the residual — so the shared FFN tail is reused. + +To add such a feature: +1. Add a detection line in `DecoderConfig`'s constructor (prefer weight-presence over an + arch-name check — it generalizes to future archs; only fall back to `arch == "..."` when the + tensor table is genuinely ambiguous, e.g. GeGLU-vs-SwiGLU). +2. Add the emission behind that flag in the relevant `blocks/` function, or in + `DecoderBuilder::build_layer()` when it is about the ORDER of sublayers rather than their + contents. +3. Add the op translator in `src/op/` if the feature needs a GGML op not yet handled, and + register it in `op_table.cpp`. + +## Adding a new model FAMILY (mmproj, audio, encoder-decoder) + +An architecture is data; a **family** is code. A family is a distinct graph shape with its own +inputs and its own notion of a layer — a vision/mmproj encoder and an audio encoder are each one, +and neither is a causal decoder. Do **not** add flags to `DecoderConfig` for them. + +Instead: + +1. Detect it in [`model_kind.cpp`](../src/builder/model_kind.cpp). mmproj files set + `general.architecture = "clip"` and carry `clip.has_vision_encoder` / `clip.has_audio_encoder` + (llama.cpp `tools/mtmd/clip-impl.h`), so `detect_model_kind()` already classifies them; the + check runs *before* any decoder hyperparameter is read, because those keys do not exist there. +2. Add a metadata reader next to `decoder_config_from_meta()` for that family's key layout, and a + config struct next to `DecoderConfig`. +3. Subclass [`ModelBuilder`](../src/builder/model_builder.hpp) in `arch/`, reusing `GraphEmitter` + and `blocks/common`. A ViT needs its own attention — non-causal, no KV cache, no RoPE — so it + will not reuse `blocks::attention`; this is the same split llama.cpp makes between + `llm_graph_context` and `clip_graph`. +4. Add a branch in `build_ggml_graph_from_gguf()`. + +Nothing in the decoder family changes. + +The downstream side already tolerates a non-decoder graph: `TranslateSession`'s LLM-specific +preprocessing is self-gating (`add_rope_sin_cos` only fires when `inp_pos` exists, +`add_sliced_mask` only when the mask and `token_len_per_seq` inputs exist), and the LLM-specific +passes (`MakeStateful`, `AdaptToGenAI`) are caller-registered rather than built in. + +## Verifying a new architecture + +1. **Converts + compiles**: convert through the frontend, then `core.compile_model(m, "CPU")`. + The frontend is not auto-selectable, so ask for it by name: + `fe = FrontEndManager().load_by_framework("gguf"); m = fe.convert(fe.load("model.gguf"))`. +2. **Graph is sane**: check the op-type histogram and that attention fused to + `ScaledDotProductAttention` and MoE to `GatherMatmul`. +3. **Numerics**: run generation through OpenVINO GenAI (`greedy_causal_lm model.gguf "..."`) and + compare to native llama.cpp (`build-ref/bin/llama-cli`) on the same prompt — the greedy tokens + should match (small drift after ~dozens of tokens is expected from kernel differences). +4. **No graph regression** for existing archs: `tests/test_arch_conversion.cpp` converts every + architecture fixture and asserts a pinned `(op count, input count)` fingerprint, so any + restructuring of a supported architecture shows up there. diff --git a/src/frontends/gguf/docs/frontend_design.md b/src/frontends/gguf/docs/frontend_design.md new file mode 100644 index 00000000000000..881805a431021e --- /dev/null +++ b/src/frontends/gguf/docs/frontend_design.md @@ -0,0 +1,396 @@ +# OpenVINO GGUF Frontend — design & developer guide + +The GGUF frontend converts GGUF model files into an `ov::Model`. It is a standard +`ov::frontend::FrontEnd` (registered as `"gguf"`, library `openvino_gguf_frontend`) and is the +path OpenVINO GenAI and the llama.cpp `ggml-openvino` backend take to run llama.cpp-style models +on OpenVINO devices. + +> [!IMPORTANT] +> The frontend is deliberately kept out of model auto-detection (`is_hidden_frontend` in +> `src/frontends/common/src/manager.cpp`): it is not listed by `available_front_ends()` and not +> selected by `load_by_model`, so **`core.read_model("model.gguf")` does not work**. Reach it +> either by linking `openvino::frontend::gguf` and constructing `FrontEnd` directly (what GenAI +> and the llama.cpp backend do), or by name: +> +> ```python +> fe = FrontEndManager().load_by_framework("gguf") +> model = fe.convert(fe.load("model.gguf")) +> ``` +> +> Nothing else is needed after `convert()`: the frontend runs its own normalization, and the only +> step `read_model` adds is `update_v10_model()`, which applies solely to legacy IR v10. Enabling +> `core.read_model` later is just a matter of dropping `"gguf"` from that list. + +For "how do I add a new model architecture", see [adding_an_architecture.md](adding_an_architecture.md). +This document explains the *design* — the layering, the two decoder paths, why the frontend loads +GGUF itself instead of depending on llama.cpp, and the memory-consumption model. + +## Layering + +``` + .gguf file / live ggml_cgraph + | + v + GgufDecoder (abstract) ── include/openvino/frontend/gguf/decoder.hpp + | \ + | \__ two concrete implementations (see "Two decoder paths") + v + TranslateSession ── src/translate_session.cpp + | walks the decoder's nodes, calls one op translator per node + v + op translators ── src/op/*.cpp (GGML_OP_MUL_MAT -> MatMul, GGML_OP_ROPE -> RoPE, ...) + | + normalization passes (SetRows lowering; caller extensions such as MakeStateful) + v + ov::Model (stateless; stateful when the caller registers MakeStateful) +``` + +The **`GgufDecoder` interface** is the single seam. It is node-scoped: `visit_subgraph` hands the +translator a decoder bound to one node, and per-node accessors (`get_op_type`, `get_input_shape`, +`get_attribute`, `get_op_case`, ...) refer to that node; model-scope accessors (`get_model_inputs`, +`get_model_output_names`, and the optional `get_model_extra_inputs` / `get_tokenizer_config`) answer +whole-model questions. The optional ones have do-nothing defaults, so a decoder implements only what +it actually knows — which is what lets two very different decoders satisfy one interface (the native +builder answers all of them, the cgraph decoder none). Note there is no weight accessor: weights are +nodes, see "Weights: the GGML_OP_NONE convention". + +Note what the interface deliberately does *not* have: anything describing the **execution mode**. +A decoder describes ggml *operations*, not a deployment, so there is no `is_stateful` / `is_static`. +Conversion always produces a stateless graph; see "Statefulness". + +Everything downstream of the decoder (translators, passes, the produced `ov::Model`) is shared by +both decoder paths — so a new op or a graph fix benefits both at once. + +## Two decoder paths (one frontend) + +The frontend is fed by **two** `GgufDecoder` implementations, deliberately: + +1. **`GgufBuilderDecoder`** (`src/builder/`) — the *native* path. Loading a `.gguf` path + parses the container (`src/quant/gguf.cpp`), and a `ModelBuilder` for the detected model family + (today `DecoderBuilder`, `src/builder/arch/decoder_builder.cpp`) builds a `GgufGraph` (a flat, + topologically-ordered node list in the GGML op vocabulary), which `GgufBuilderDecoder` exposes. + **No llama.cpp in the process.** This is the default and the path GenAI uses. + +2. **`GgmlOvDecoder`** (lives in llama.cpp's `ggml/src/ggml-openvino` backend, not in this repo) — + wraps a live `ggml_cgraph` that llama.cpp already built. Used when OpenVINO is a *backend inside + llama.cpp*; it links `openvino::frontend::gguf` and calls `FrontEnd::convert()` directly. + +Both produce the exact same op vocabulary, so they share the translators and passes verbatim. + +### Why the native path does not use llama.cpp to load the model + +The obvious alternative — have the native path call into llama.cpp to build the cgraph (reusing path +2's `GgmlOvDecoder`) — was considered and rejected for the default path. llama.cpp's graph builder +is not a standalone library: it is entangled with `llama_model` loading and the `libllama`/`libggml` +runtime, so using it means linking substantial llama.cpp + ggml into the OpenVINO/GenAI process +(whether via submodule, FetchContent, or a prebuilt lib — the mechanism is not the issue, the +dependency is). The drawbacks that made the native builder the default: + +- **A second model-loading runtime + ~2x transient memory.** llama.cpp would allocate the full + model into ggml tensors just to *build* the cgraph, and OpenVINO would then re-materialize the + weights as `Constant`s — roughly double the host memory during load. The native builder mmaps the + file and zero-copies weight bytes straight into OpenVINO `Constant`s (see "Memory model"). +- **Version coupling.** llama.cpp has no stable ABI and GGUF/architecture/tensor-naming conventions + change quickly; pinning a version means chasing upstream, not pinning means breakage. +- **Binary size, build matrix, supply-chain.** Shipping libllama+libggml (CPU kernels, quant + kernels, tokenizer, sampling) inside OpenVINO enlarges the binary, adds a CMake/SIMD/backend + matrix to build on every target, and adds a CVE/provenance surface. +- **Loss of self-containment.** Converting a `.gguf` works in any OpenVINO deployment with + nothing extra; a llama.cpp dependency would break that. + +The upside of llama.cpp — instant coverage of ~130 architectures — is real. The intended way to get +it *without* burdening the default path is to keep `GgmlOvDecoder` as an **optional, build-gated +alternative decoder** feeding the same frontend, never as the default. The native builder stays the +self-contained default. + +## Weights: the GGML_OP_NONE convention + +Weights are surfaced uniformly as `GGML_OP_NONE` leaf nodes (there is no `get_model_weights` +special path in the graph walk). `translate_weight` (`src/op/weight.cpp`) turns each leaf into a +compressed **decompression subgraph** — `Constant(low-bit) -> Convert -> Subtract(zp) -> +Multiply(scale) -> Reshape` — never a fully-materialized f32 constant. Two payload shapes are +accepted by the same translator: + +- Native builder: the parser already extracted `weight`/`scales`/`zp` tensors; the leaf carries + them as attributes and `translate_weight` calls `make_weight_node(base, weights, qtypes)`. +- cgraph path: the leaf carries the raw ggml bytes; `translate_weight` extracts them itself. + +Both build the *identical* compressed OpenVINO subgraph, so **inference speed and compile memory +are independent of which path/payload was used** (verified: OLMoE compile peak 8673 MB via +GGML_OP_NONE vs 8672 MB via the older eager path). + +## op_case: one numbering for both ingest paths + +Several ggml ops cover structurally different uses that need different OpenVINO subgraphs — a +`GGML_OP_RESHAPE` splitting a projection into heads is not the reshape that merges them back. The +decoder disambiguates with an `op_case` attribute, read via `NodeContext::get_op_case()`. + +**`op_case` describes the tensor operation, not which decoder produced it.** The cgraph decoder +derives it by inspecting the ggml node (`ggml-decoder.cpp::compute_op_case`); the native builder +sets it when it emits a node, choosing the case that matches what it is doing. Both therefore land +on the *same* case for the same operation, which is what keeps one translator body serving both +paths — and what makes the two graphs comparable node-for-node. + +A case that exists only to mean "this came from the builder" is a defect: it splits a shared +translator into two bodies that then drift apart. Three cases are legitimately builder-only, each +because the operation itself genuinely differs rather than for numbering reasons: + +| case | op | why | +| --- | --- | --- | +| 100 | `FLASH_ATTN_EXT` | The builder keeps q/k/v ggml-natural so the order is Concat -> GQA tile -> one Transpose -> SDPA, which is what the CPU plugin's `stateful_sdpa_fusion` matches; permuting first blocks the fuse into `ScaledDotProductAttentionWithKVCache`. A deliberately *better* graph, not an equivalent one. | +| 104 | `VIEW` | Takes a second (shape-reference) input the cgraph path does not supply, so it has a different arity than the shared cases. | +| 10 | `GET_ROWS` | llama.cpp reshapes `probs` before the MoE gating gather, so the cgraph decoder sees a different input shape that the generic path already handles. | + +Anything else should reuse a shared case, which usually means the builder describing its node the +way the shared translator expects (e.g. supplying `view_slice` for a plain single-axis shrink rather +than a bespoke attribute) or emitting the same node *count* ggml does instead of fusing steps. + +## Memory model + +Understanding where memory goes matters for large models (MoE especially). + +**Load / convert:** +- The GGUF file is memory-mapped (`ov::load_mmap_object`); non-quantized tensors are zero-copy views + into the mmap. Quantized tensors are *repacked* once into a single `AlignedBuffer` (u4/u8 weights + + f16 scales + integer zero-points in OpenVINO's compressed layout) and wrapped as `Constant`s + via a `SharedBuffer` — so there is no second full-model host allocation and weights are never + expanded to f32 at load. Measured: OLMoE-1B-7B q4_0 (3.9 GB file) read peak ≈ 3.9 GB. +- Because the graph keeps weights *compressed*, conversion memory is roughly the file size, not + the dequantized (f32) model size. + +**Compile / `compile_model`:** +- Weights stay compressed through compilation. Dense models fold the decompression into + `FullyConnected`; MoE experts fold into `GatherMatmulCompressed`. Measured: OLMoE compile peak + ≈ 8.7 GB (vs a ~53 GB blow-up if the expert decompression is *not* kept compressed). +- **Critical dependency for MoE:** the CPU plugin must recognize the expert decompression chain so + it is not expanded to f32. Two changes make this work and both are required: + `SnippetsMarkSkipped` must skip the `GatherMatmul` weight chain, and `is_decompression_multiply` + must accept `GatherMatmul` consumers. Without them, `ConstantFolding` expands u4/u8 experts to + f32 (the 53 GB case). See the `[CPU]` commits. +- KV cache precision: the stateful KV cache is f16 (set in `translate_session`); leaving it at the + CPU default of u8 causes NaNs in some decode paths. + +**Inference:** the compressed weights are decompressed on the fly by the plugin (dense: fused into +the FC kernel; MoE: `GatherMatmulCompressed`). No persistent f32 weight copy. + +**Rule of thumb:** peak host memory ≈ `max(file_size_for_read, compressed_graph + plugin_scratch)`, +NOT the dequantized model size — *provided* the CPU decompression-recognition changes above are in +place. If a future change makes a model's weights expand to f32 at compile, that is the regression +to look for (compile peak jumping toward the dequantized size). + +## Statefulness & the attention backends + +**The frontend is universal: conversion always produces a STATELESS graph.** Every KV cache is an +explicit model `Parameter`, written by a `SetRows` placeholder op and returned as a `Result`. That is +the same shape optimum-intel exports before applying its own +`apply_make_stateful_transformation` — and it is why the decoder interface carries no execution-mode +flag. + +**Statefulness is a caller concern**, chosen by registering a transformation extension. Extensions +run in the frontend's normalization stage *ahead of* the built-in `LowerSetRowsStateless`, so a +registered pass consumes the KV-cache `SetRows` ops and the default lowering only ever sees the ones +left over (e.g. MoE routing writes, which stay stateless either way): + +```cpp +// Stateless (the default): caches are inputs/outputs, SetRows -> ScatterUpdate. +auto stateless = fe.convert(fe.load("model.gguf")); // plain FrontEnd, no extension + +// Stateful: caches become Variables (ReadValue / Concat / Assign). +ov::frontend::gguf::FrontEnd fe; +fe.add_extension(std::make_shared( + ov::frontend::gguf::pass::MakeStateful())); +auto stateful = fe.convert(fe.load("model.gguf")); +``` + +`ov::Core::add_extension` works too when going through a `Core` (it forwards its +extensions to the frontend before `load`), but it is global; driving the frontend directly scopes the +choice to one conversion, which is what GenAI does. + +Three consumers, three combinations, one frontend: + +| Consumer | Extension registered | Result | +| --- | --- | --- | +| plain `FrontEnd::convert` | none | stateless graph, gguf-native IO | +| OpenVINO GenAI | `MakeStateful` + then `AdaptToGenAI` | stateful graph, GenAI IO | +| llama.cpp `ggml-openvino` | its own `LlamaCppToStateful` | stateful graph, its own cache layout & mask re-slicing | + +`MakeStateful` (`include/openvino/frontend/gguf/make_stateful.hpp`) is decoder-agnostic: it infers +each cache's append axis from the cache `Parameter`'s single dynamic axis (or takes it explicitly for +a preallocated cache), and re-splits the placeholder's flattened rows against the cache layout. It +scopes itself to cache growth only and does **not** touch the attention mask — a dynamically-sized +mask (what the native builder emits) needs no change, while a preallocated fixed mask window must be +re-sliced by the caller, which is what llama.cpp's own extension does. + +`AdaptToGenAI` (`src/pass/adapt_to_genai.cpp`, run by GenAI after conversion) then rewrites the +gguf-native IO (`inp_tokens`/`inp_pos`/`self_kq_mask`/...) into GenAI's contract +(`input_ids`/`attention_mask`/`position_ids`/`beam_idx` -> `logits`), so a GGUF model behaves like an +optimum-intel export. The two concerns are separate passes precisely because they are independent: +cache form vs IO contract. + +Either way the stateful graph is shaped so the CPU plugin's `stateful_sdpa_fusion` folds attention +into `ScaledDotProductAttentionWithKVCache`. + +The result is valid under **both** attention backends: plain stateful SDPA inference, and +`ov::pass::SDPAToPagedAttention` (the transform GenAI's ContinuousBatching adapter applies for +`ATTENTION_BACKEND=PA`). There is no mode flag — one graph serves both. + +That works because the two backends disagree only about *where the token count lives*. Plain +inference feeds `input_ids` as `[1, tokens]`; `SDPAToPagedAttention` rewrites the `Parameter` to +rank-1 `[tokens]` and splices an `Unsqueeze(axis=1)` in front of its consumers, so the body sees +`[tokens, 1]` and PA's hardcoded flattens read the count out of dim 0. Since ggml activations are +`[batch, tokens, heads, head_size]` with `batch == 1`, `[1, T, H, D]` and `[T, 1, H, D]` are +element-for-element the same buffer. + +**The invariant to preserve: no node may pin the leading two dims to constants.** `AdaptToGenAI` +derives them from the live `input_ids`, and the op translators reshape with `special_zero=true` so a +`0` copies dim 0 through instead of writing a literal `1` (`reshape` cases 1/2, `rope`'s +bhsd/paired/to_bhls targets, `set_rows`, `MakeStateful`'s row re-split). Two traps when touching +these: + +- **OV broadcasts elementwise operands from the right.** A rank-4 activation against a rank-3 one + *appears* to work while dim 0 is a literal batch 1 (`[1,1,T,E]` right-aligns onto `[1,T,E]`), then + silently forms a `T x T` outer product once tokens move to dim 0. **Activations are uniformly + rank-4** (ggml's `[batch, tokens, heads, head_size]`) — there is no second convention to pick + between, and a translator that emits rank 3 reintroduces exactly this bug. +- **`get_rows` lowers to `Gather(act, ids, axis=1, batch_dims=1)`** = `act[i, ids[i,j]]`, so an + identity selection is `ids[i,j] == j` — an index along axis 1 replicated over axis 0, *not* a + `0..tokens` range. A range only coincides with the identity when axis 0 is a batch of 1. + +A `Convert` between the KV-cache `Concat` and SDPA also silently disables PA: `StateManagementPattern` +admits none, so `TranslateSession` runs `EliminateConvert` after `ConvertConvertLike` to drop the +no-op ones. If PA conversion regresses to 0, check for a reintroduced `Convert` there first. + +Verifying PA is actually in use requires looking at the **compiled runtime graph**, not the +`ov::Model` — see "Measuring performance correctly" in +[`supported_models.md`](supported_models.md). + +## Tokenizer metadata + +A GGUF file embeds not just the weights but the full tokenizer (vocab, merges, scores, token +types, special-token ids, pre-tokenizer regex, chat template) under its `tokenizer.*` metadata +keys. The frontend carries that metadata out on the converted model so a consumer can build a +matching OpenVINO tokenizer/detokenizer **without re-opening the `.gguf` and without any GGUF +parser of its own** — the model object is self-describing. + +Mechanism (native path): +1. The builder scrapes every `tokenizer.*` key into an `ov::AnyMap` keyed by the sub-key after the + last dot (`model`, `tokens`, `merges`, `scores`, `token_type`, `pre`, `bos_token_id`, + `chat_template`, ...), each value being a `std::string` / `std::vector` / + `ov::Tensor` — `extract_tokenizer_config` in `gguf_builder.cpp`, surfaced through the decoder's + `get_tokenizer_config()`. +2. `TranslateSession` attaches it to the converted model's **runtime info** as a + `GGUFTokenizerMetadata` attribute under `gguf_tokenizer_metadata_key()` + (`include/openvino/frontend/gguf/tokenizer_metadata.hpp`). +3. The attribute is deliberately **non-serializable** (`is_copyable() == false`, empty + `to_string()`): the vocab+merges are large and only meaningful in-memory between conversion and + tokenizer construction, so it is dropped on clone and emitted as an empty placeholder if the IR + is serialized (it never bloats the XML). It is an in-process handoff, not part of the saved model. +4. A consumer (OpenVINO GenAI) reads `model->get_rt_info()[gguf_tokenizer_metadata_key()]` and + builds the OpenVINO tokenizer/detokenizer from it — see GenAI's `create_tokenizer_from_model` + (`gguf_utils/gguf_tokenizer.cpp`), which turns the map into BPE/Unigram tokenizer models via + `openvino_tokenizers`. So converting the `.gguf` + this rt_info is enough for GenAI to produce + both the inference model and its tokenizer. + +### With and without a llama.cpp dependency + +The tokenizer path is exactly where the frontend's llama.cpp-independence pays off, and it behaves +correctly on both decoder paths: + +- **Without llama.cpp (native `.gguf` path, the default).** The frontend parses the `tokenizer.*` + keys itself and emits the `GGUFTokenizerMetadata` rt_info. The whole tokenizer round-trip + (`.gguf` file -> OpenVINO tokenizer) happens with **no llama.cpp and no separate GGUF/tokenizer + library in the consumer** — the OpenVINO model is the single source of truth. This is what lets + the frontend + GenAI stand alone. + +- **With llama.cpp (the cgraph / `GgmlOvDecoder` path).** When OpenVINO runs as a backend *inside* + llama.cpp, llama.cpp already owns the tokenizer natively (it parsed the same `tokenizer.*` keys + to build its own `llama_vocab`). There is nothing for the frontend to hand off, so `GgmlOvDecoder` + leaves `get_tokenizer_config()` empty and **no rt_info is attached** — tokenization is done by + llama.cpp, the frontend only produces the compute graph. `TranslateSession` attaches the + metadata only when `get_tokenizer_config()` is non-empty, so the same code serves both paths + with no branching in the consumer. + +In other words: the tokenizer metadata is populated by whichever side *owns* GGUF parsing. On the +native path that is the frontend (so it exports the metadata for a llama.cpp-free consumer); on the +cgraph path that is llama.cpp (so the frontend stays out of the tokenizer's way). Either way the +`GgufDecoder::get_tokenizer_config()` seam is the single contract, and no consumer needs to link +llama.cpp to obtain a tokenizer. + +### How the tokenizer is constructed (consumer side) + +The frontend only *exports* the metadata; turning it into a runnable tokenizer is the consumer's +job. This is described here because it defines the contract the frontend must satisfy (which keys, +which value types). The reference consumer is OpenVINO GenAI +(`src/cpp/src/gguf_utils/gguf_tokenizer.cpp`); the tokenizer is itself built as a pair of +**`ov::Model`s** (tokenizer + detokenizer) out of nodes from the **`openvino_tokenizers`** runtime +library — there is no bespoke tokenizer engine and, again, no llama.cpp. + +Flow (native path): conversion attaches the rt_info, then +`create_tokenizer_from_model(model)`: + +1. **Fetch** `model->get_rt_info()[gguf_tokenizer_metadata_key()]`, cast to + `GGUFTokenizerMetadata`, and read its `.config` `AnyMap`. If the key is absent (model not from + the frontend, or metadata stripped by serialization) it asserts — see the serialization caveat + below. +2. **Normalize** the `AnyMap` into the same `map` that the from-file path + (`tokenizer_config_from_meta`) produces, so a single builder serves both. (`tokenizer_config_from_rt_info`.) +3. **Build the two models** (`build_tokenizer_models`), loading `openvino_tokenizers`' factory + entry point `create_tokenizer_node` at runtime (`get_symbol(..., "create_tokenizer_node")`). + +The tokenizer `ov::Model` is assembled from these `openvino_tokenizers` operations (a `string` +`Parameter` in, token-id tensor out): + +- `StringTensorUnpack` — unpack the input string tensor into (begins, ends, chars). +- `RegexNormalization` — text normalization, e.g. gemma4's SPM whitespace→metaspace (`U+2581 ▁`). +- `SpecialTokensSplit` — split out special tokens; the special-token set is derived from the GGUF + `tokens` + `token_type` keys (entries whose type is CONTROL/USER_DEFINED via `is_special_token`). +- `RegexSplit` — the pre-tokenizer regex (BPE families). +- The core tokenizer node, dispatched on the GGUF `tokenizer.ggml.model` key: + - `model == "llama"` (SPM/`plamo2`): `parse_spm_config` → **`SentencepieceTokenizer`**, fed the + `tokens` + `scores` + `token_type` arrays. + - `model == "gpt2"` / `"gemma4"` (byte-level BPE): `parse_bbpe_config` → **`BPETokenizer`**, fed + the `tokens` vocab + `merges`. +- `RaggedToDense` — produce the dense `input_ids` output. + +The detokenizer `ov::Model` is the inverse: **`VocabDecoder`** + `RegexNormalization` + +`StringTensorPack` for BPE, or **`SentencepieceDetokenizer`** for the llama/SPM family. + +The GGUF metadata keys the builder relies on (so the frontend must preserve them verbatim): +`model`, `tokens`, `merges`, `scores`, `token_type`, `pre` (pre-tokenizer id), the special-token +ids (`bos_token_id`/`eos_token_id`/`unknown_token_id`/`padding_token_id`), `add_bos_token` / +`add_space_prefix` flags, and `chat_template` (the chat template is carried through for GenAI to +apply; GenAI additionally patches a few known-malformed templates, e.g. qwen2.5, in +`patch_gguf_chat_template`). + +The resulting tokenizer/detokenizer `ov::Model`s are what `ov::genai::Tokenizer` wraps and compiles +like any other model — so tokenization runs on an OpenVINO device, consistent with the inference +model. + +**Serialization caveat.** Because `GGUFTokenizerMetadata` is non-serializable, it exists only on the +in-memory model straight out of the frontend. A model that was serialized to IR and reloaded has no +such rt_info; GenAI then falls back to re-reading the `.gguf` +(`create_tokenizer_from_config` → `tokenizer_config_from_meta`), which needs the file but still no +llama.cpp. The rt_info path is the fast in-process handoff; the file path is the durable fallback. + +## Source map + +| Path | Contents | +|---|---| +| `include/openvino/frontend/gguf/` | public headers: `decoder.hpp`, `frontend.hpp`, `make_stateful.hpp`, `adapt_to_genai.hpp`, `tokenizer_metadata.hpp`, `set_rows_op.hpp` | +| `src/frontend.cpp` | FrontEnd: `.gguf` magic sniff + native load path; live-decoder path; extensions | +| `src/translate_session.cpp` | graph walk, weight seeding, normalization passes (caller extensions then built-ins), tokenizer rt_info | +| `src/op/*.cpp` | one op translator per GGML op | +| `src/builder/` | native `.gguf` graph builder + `GgufBuilderDecoder`; layered as `graph_emitter` (arch-agnostic node emission), `blocks/` (reusable fragments), `decoder_config` (architecture detection), `arch/decoder_builder` (topology), `arch_registry` / `model_kind` (what is accepted, and which family) | +| `src/quant/` | GGUF container parser (`gguf.cpp`), dequant fill fns (`gguf_quants.cpp`), weight-node construction (`weights.cpp`) | +| `src/pass/` | `LowerSetRowsStateless` (built-in), `MakeStateful` + `AdaptToGenAI` (caller-registered) | +| `src/helper_ops/` | internal `SetRows` placeholder op | +| `tests/` | C++ op/dequant tests (in CI); standalone python dev/bench scripts | + +## Testing + +- C++ unit tests (`tests/*.cpp`, target `ov_gguf_frontend_tests`) cover op translators and weight + dequant against real-ggml reference `.npy` fixtures, and run in CI. +- A graph-fingerprint check (sha256 over sorted `(op_type, output_shape)` pairs of the converted + model, per architecture) is the recommended cheap regression gate for any builder change — it + proves the produced graph is unchanged. Accuracy is validated opt-in against llama.cpp (WWB-style) + and by comparing greedy tokens on the same prompt. + diff --git a/src/frontends/gguf/docs/how_to_add_op.md b/src/frontends/gguf/docs/how_to_add_op.md new file mode 100644 index 00000000000000..61d4756dc2c9f8 --- /dev/null +++ b/src/frontends/gguf/docs/how_to_add_op.md @@ -0,0 +1,175 @@ +# Adding an op translator to the GGUF frontend + +Procedure for enabling a ggml operation. For the *concepts* behind it — the two decoder paths, the +`GGML_OP_NONE` weight convention, `op_case` numbering, the memory model — read +[frontend_design.md](frontend_design.md) first; this document does not repeat them. + +Related: [adding_an_architecture.md](adding_an_architecture.md) (enabling a model family, which +usually needs *no* new op), [debugging_accuracy.md](debugging_accuracy.md) (when a translator +converts but produces wrong numbers). + +## Before writing a translator + +Check that an op translator is actually what is missing: + +- **A new architecture** normally needs only an entry in `arch_registry.cpp` — see + [adding_an_architecture.md](adding_an_architecture.md). Reach for a translator only when the graph + genuinely contains a ggml op the table does not have. +- **A structurally different use of an existing op** is an `op_case`, not a new translator. Read the + `op_case` section of [frontend_design.md](frontend_design.md) before adding a case — a case that + exists only to mean "this came from the builder" is a defect. +- Both decoder paths (native builder and llama.cpp cgraph) share translator bodies, so a change here + affects both. Keep the body path-agnostic; branch on `op_case`, never on "which decoder made this". + +## Checklist + +| # | File | Change | +|---|------|--------| +| 1 | `src/op/.cpp` | New translator function | +| 2 | [`src/op_table.hpp`](../src/op_table.hpp) | `GGUF_OP_CONVERTER(translate_);` declaration | +| 3 | [`src/op_table.cpp`](../src/op_table.cpp) | `{"GGML_OP_", op::translate_},` (list is alphabetical) | +| 4 | [`tests/CMakeLists.txt`](../tests/CMakeLists.txt) | Add `"${FE_SRC_DIR}/op/.cpp"` to `FRONTEND_SRCS` | +| 5 | [`tests/test_ops.cpp`](../tests/test_ops.cpp) | `TEST(GGUFOps, )` — **mandatory**, see the coverage gate below | + +> **Step 4 is mandatory and easy to miss.** `src/CMakeLists.txt` builds the library via +> `ov_add_frontend`, which picks up new sources automatically — but the test binary compiles the +> frontend sources from an **explicit list with no GLOB**. Omitting this yields an undefined symbol +> at link time, after a full compile. + +If the op maps 1:1 onto a single OpenVINO op with the same operand order, skip steps 1, 2 and 4 and +register a template from [`src/utils.hpp`](../src/utils.hpp) directly in `op_table.cpp`: + +```cpp +{"GGML_OP_SUB", op::translate_1to1_match_2_inputs}, +{"GGML_UNARY_OP_TANH", op::translate_1to1_match_1_input}, +``` + +## Translator shape + +```cpp +OutputVector translate_(const NodeContext& context) { + num_inputs_check(context, 1, 2); // min / max operand count + + float eps = context.get_attribute("eps"); + int op_case = context.get_op_case(); + + std::shared_ptr res = ...; + + return rename_outputs_with_suffix({res}, context.get_name()); +} +``` + +Always finish with `rename_outputs_with_suffix(..., context.get_name())`: the walk stores results in +the `TensorMap` under the decoder's output names, and stable friendly names are what the passes and +the graph-fingerprint gate rely on. + +`NodeContext` ([`src/node_context.hpp`](../src/node_context.hpp)): + +| Call | Purpose | +|------|---------| +| `get_input(idx)` / `get_input(name)` | Operand as `Output` | +| `has_input(name)` | Test an optional operand first | +| `get_input_size()` | Actual operand count | +| `get_input_shape(idx)` / `get_output_shape()` | **Static ggml** shape — use when the live OV shape is dynamic (KV-cache path) | +| `get_input_view_element_offset(idx)` | Element (not byte) offset for a ggml VIEW operand | +| `get_op_case()` | Structural variant (convenience wrapper, defaults to 0) | +| `get_output_type()` | Declared output element type | +| `get_attribute(name[, default])` | Any other typed op parameter | + +Helpers in [`src/utils.hpp`](../src/utils.hpp): `num_inputs_check`, `get_dimensions`, +`rename_outputs_with_suffix`, `make_sin_cos` (RoPE), `process_view_input`. + +Insert a `Convert` to `get_output_type()` when the op may change element type (`CONCAT`, `CPY`, +`SET_ROWS`, `GET_ROWS`) rather than assuming the input type. Prefer `ov::op::vX::OpName` over +`opsetX::OpName`, per the repository convention. + +## Test, and the coverage gate + +[`tests/test_op_coverage.cpp`](../tests/test_op_coverage.cpp) asserts that **every op registered in +`op_table.cpp` is exercised by some test**. Registering a translator without a test fails the suite; +the gate exists because `GGML_UNARY_OP_GELU_QUICK` once shipped with the wrong formula precisely +because nothing converted it. The exemption list is for ops whose *nature* makes a single-op test +meaningless — not for ops that are merely awkward to test. + +The gate only asserts when the full suite runs, so a narrowing `--gtest_filter` silently skips it. +**Run the binary unfiltered before pushing.** + +```cpp +TEST(GGUFOps, Scale) { + auto model = SingleOpBuilder() + .op("GGML_OP_SCALE") + .input("x", ov::element::f32, {2, 4}) + .output("out", ov::element::f32, {2, 4}) + .attr("scale", 2.5f) + .attr("bias", 1.0f) + .build(); + + auto out = run_on_cpu(model, {{"x", make_f32_tensor({2, 4}, x)}}); + expect_near(out, expected); +} +``` + +`SingleOpBuilder` drives the real `FrontEnd::convert` through an in-memory `SingleOpDecoder`, so no +`.gguf` file is involved. Helpers are in [`op_test_utils.hpp`](../tests/op_test_utils.hpp). + +**Where the expected values come from matters more than the test's shape.** Per the one rule in +[debugging_accuracy.md](debugging_accuracy.md), the reference must come from real ggml, not from +your own reading of the op's math: + +- Simple elementwise ops with an unambiguous closed form — compute inline in the test. +- Anything with layout, geometry or head structure (rope, conv, attention, views) — generate the + reference from ggml-CPU: an `.npy` fixture via [`gen_ggml_reference.c`](../tests/gen_ggml_reference.c), + or a standalone oracle such as `ssm_conv_oracle.c` / `imrope_oracle.c`, and paste its output with a + comment naming the oracle. + +Test at realistic dimensions. With one head many layout orders coincide, so a single-head test can +pass against a wrong reference. + +`expect_near(actual, expected, atol = 1e-4f, rtol = 2e-3f)` combines absolute and relative +tolerance. **Do not tighten `rtol`**: ARM CPU runs the graph in fp16 by default, and a tolerance +tuned only on fp32 x86 will fail there. + +To find the closest existing example without reading the whole (large) file: + +```bash +grep -n "^TEST(" src/frontends/gguf/tests/test_ops.cpp +``` + +## Build and run + +The frontend is **off by default** (`ENABLE_OV_GGUF_FRONTEND` in +[`cmake/features.cmake`](../../../../cmake/features.cmake)); without it the test target does not +exist. + +```bash +cmake -B build -DENABLE_OV_GGUF_FRONTEND=ON -DENABLE_TESTS=ON +cmake --build build --target ov_gguf_frontend_tests -j$(nproc) + +# iterate on one op ... +./build/bin/*/*/ov_gguf_frontend_tests --gtest_filter='GGUFOps.*' +# ... then unfiltered, so the coverage gate actually runs +./build/bin/*/*/ov_gguf_frontend_tests +``` + +CI runs the same binary in the "GGUF frontend tests" step of +[`job_cxx_unit_tests.yml`](../../../../.github/workflows/job_cxx_unit_tests.yml), ungated by Smart CI. + +For a change that touches a shared translator or a VIEW/`op_case` predicate, also re-run the +graph-fingerprint check ([`tests/graph_fingerprint.py`](../tests/graph_fingerprint.py)) across the +supported architectures: a guard that fixes one arch can reject another's legitimately-contiguous +view. + +## Bringing up a model that hits a missing op + +Conversion aborts on the **first** unsupported op with: + +``` +Translation for operation type GGML_OP_ is not implemented. +``` + +Enabling one op per run is the slow path. Instead diff the model's op vocabulary against the keys of +`get_supported_ops()` to get the full set at once, group the ops that collapse into +`translate_1to1_match_*` one-liners, and implement only the remainder as real translators. + +A different error, `Number of outputs greater than number of converted outputs`, means the +translator returned the wrong number of outputs — not that the op is unsupported. diff --git a/src/frontends/gguf/docs/supported_models.md b/src/frontends/gguf/docs/supported_models.md index 6e9bddc14c3b56..b897a48d709418 100644 --- a/src/frontends/gguf/docs/supported_models.md +++ b/src/frontends/gguf/docs/supported_models.md @@ -44,6 +44,391 @@ A run counts as verification only when the output is coherent (e.g. completes "...is Paris") and consistent with the pure-ggml CPU backend on the same prompt. A model that loads but emits garbage (e.g. `hunyuan`) is **not** counted as supported. +## Architectures accepted by the native `.gguf` builder + +Everything above is about the **llama.cpp cgraph** path. This section covers the *other* +decoder — the native `.gguf` builder (`DecoderBuilder` in +[`src/builder/arch/decoder_builder.cpp`](../src/builder/arch/decoder_builder.cpp)), which is what +OpenVINO GenAI uses. The two paths share all op +translators but have separate architecture lists. + +The builder's accept-list is the union of two sets, both defined in +[`src/builder/arch_registry.cpp`](../src/builder/arch_registry.cpp): + +- **`verified_archs()`** — convert + compile + generation checked against a reference on a + real checkpoint. +- **`experimental_archs()`** — expected to work via the builder's GGUF-tensor-table + auto-detection, but not end-to-end verified. These convert and emit a one-time + `OPENVINO_WARN` so callers know they are best-effort. + +Anything not in either set is rejected with an explicit `OPENVINO_ASSERT` at load time +rather than converting into a silently wrong graph. + +### `verified_archs()` — 14 architectures + +| Architecture | Notes | +|---|---| +| `llama` | llama-2 / llama-3 | +| `qwen2` | qwen2 / qwen2.5 | +| `qwen3` | QK-norm | +| `phi3` | fused QKV | +| `minicpm` | NORMAL rope + scalar embedding/residual/logit scales | +| `hunyuan-dense` | | +| `olmoe` | OLMoE 1B-7B (MoE) | +| `qwen3moe` | Qwen3 MoE; same topology as `olmoe` | +| `qwen35` | Qwen3.5/3.6 (and the Ternary-Bonsai backbone): hybrid Gated-DeltaNet + full attention, M-RoPE, interleaved query+gate projection. Greedy / batch 1 only | +| `gpt-oss` | MoE + attention sinks + SWA + OAI gated activation | +| `gemma` | Gemma 2B / 7B | +| `gemma2` | post-norms + attention soft-cap | +| `gemma3` | post-norms + final logit soft-cap | +| `gemma4` | SWA, per-layer embeddings, shared KV | + +### `experimental_archs()` — 16 architectures + +| Architecture | Notes | +|---|---| +| `llama-embed` | Bidirectional LLaMA (embedding model, no causal mask) | +| `exaone4` | EXAONE 4.0: NEOX rope, post-norms (attn + ffn) | +| `plamo3` | PLaMo-3: NEOX rope, post-norms (attn + ffn) | +| `smollm3` | SmolLM3: NORMAL rope + SWA | +| `hunyuan-moe` | NEOX rope, MoE routing, QK-norm | +| `glm4moe` | GLM 4.5 MoE: 1 dense lead layer, MoE + attn post-norm | +| `exaone-moe` | EXAONE MoE: SWA + MoE, shared expert | +| `minimax-m2` | Minimax M2: pure MoE | +| `ernie4_5-moe` | Ernie 4.5 MoE: NORMAL rope, dense lead layers + MoE stride | +| `bailingmoe2` | BailingMoe V2: MoE + shared expert + QK-norm | +| `maincoder` | Maincoder-1B: NORMAL rope, QK-norm (auto-detected) | +| `mistral3` | Ministral-3B: NORMAL rope, dense | +| `muse-glimmer` | Muse Glimmer (Meta Onyx): NORMAL rope on SWA layers only (global layers are NoPE), sigmoid attention output gate, QK-norm, pre+post norms, final logit soft-cap | +| `mellum` | JetBrains Mellum: pure MoE | +| `deepseek2-ocr` | DeepSeekOCR: dense lead layers + MoE | +| `jais2` | JAIS-2: dense (biases auto-detected) | + +RoPE flavor is **not** in these tables because it is a separate switch: archs listed in +`arch_uses_neox_rope()` use NEOX (rotate-halves), everything else uses NORMAL (rotate +consecutive pairs). Adding an arch to the accept-list without also classifying its RoPE is +the most common way to get a model that loads and produces garbage. + +### Measured status through OpenVINO GenAI + +Every architecture above was run through GenAI on CPU (`gguf_arch_check`, greedy, SDPA +backend) on the checkpoint named below. "Generates" means the model answered *"The capital +of France is"* correctly and coherently; **`llama.cpp` ref** is the same `.gguf` through +`llama-cli` on the default ggml CPU backend, which distinguishes a frontend bug from a +model/checkpoint that is simply weak on the prompt. + +| Arch | Set | Model used | GenAI | llama.cpp ref | +|---|---|---|---|---| +| `llama` | verified | Llama-3.2-1B-Instruct Q4_K_M | generates | generates | +| `qwen2` | verified | Qwen2.5-0.5B-Instruct Q4_K_M | generates | generates | +| `qwen3` | verified | Qwen3-0.6B Q8_0 | generates (reasoning preamble) | same | +| `phi3` | verified | Phi-3-mini-4k-instruct Q4 | generates | generates | +| `minicpm` | verified | MiniCPM-2B-dpo Q4_K_M | generates | generates | +| `hunyuan-dense` | verified | Hunyuan-0.5B-Instruct Q4_K_M | **degenerate** | generates | +| `olmoe` | verified | OLMoE-1B-7B-Instruct Q4_K_M | generates | generates | +| `qwen3moe` | verified | Qwen3-0.9B-A0.6B Q4_K_M | **degenerate** | generates | +| `gpt-oss` | verified | gpt-oss-20b MXFP4 | generates (harmony format) | same | +| `gemma` | verified | gemma-2b Q4_K_M | **throws** (SDPA shape mismatch) | degenerate too | +| `gemma2` | verified | gemma-2-2b-it Q4_K_M | **degenerate** | generates | +| `gemma3` | verified | gemma-3-1b-it Q4_K_M | generates | generates | +| `gemma4` | verified | gemma-4-E4B-it Q4_K_M | generates | generates | +| `llama-embed` | experimental | llama-nemotron-embed-1b-v2 Q4_K_M | repeats (embedding model) | degenerate too | +| `exaone4` | experimental | EXAONE-4.0-1.2B Q4_K_M | **degenerate** | generates | +| `plamo3` | experimental | plamo-3-nict-2b-base Q4_K_M | **degenerate** | degenerate too | +| `smollm3` | experimental | SmolLM3-3B Q4_K_M | generates (reasoning preamble) | same | +| `maincoder` | experimental | Maincoder-1B Q4_K_M | generates | generates | +| `mistral3` | experimental | Ministral-3-3B-Instruct-2512 Q4_K_M | generates | generates | +| `muse-glimmer` | experimental | Muse-Glimmer-30B Q4_0 | generates | generates | +| `qwen35` | verified | Qwen3.5-0.8B Q8_0 | generates | generates | +| `qwen35` (Bonsai) | verified | Ternary-Bonsai-27B Q2_g64 | generates | generates | +| `deepseek2-ocr` | experimental | deepseek-ocr-2 Q4_K_M | generates | generates | +| `ernie4_5-moe` | experimental | ERNIE-4.5-21B-A3B Q4_K_M | **degenerate** (blank) | generates | +| `bailingmoe2` | experimental | Ling-mini-2.0 Q2_K | generates | generates | +| `mellum` | experimental | Mellum2-12B-A2.5B-Instruct Q4_K_M | generates | generates | +| `hunyuan-moe` | experimental | — | not tested (no checkpoint) | — | +| `glm4moe` | experimental | — | not tested (smallest GLM-4.5-Air ≈ 40 GiB) | — | +| `exaone-moe` | experimental | — | not tested (smallest ≈ 9 GiB, 32B) | — | +| `minimax-m2` | experimental | — | not tested (smallest ≈ 78 GiB) | — | +| `jais2` | experimental | — | not tested (no checkpoint) | — | + +Two caveats on reading this table. `llama-embed` is an *embedding* model, so degenerate +greedy completion is expected of it, not a defect. `gemma` (v1 base) and `plamo3` (base, not +instruct) are degenerate on the reference too, so those rows are checkpoint/prompt artifacts +rather than frontend bugs. + +That leaves **5 architectures that generate correctly under llama.cpp but not through the +builder** — `hunyuan-dense`, `qwen3moe`, `gemma2`, `exaone4` and `ernie4_5-moe` (blank output) — +i.e. real conversion defects, plus `gemma`, which throws instead of converting cleanly. Four of +those (`hunyuan-dense`, `qwen3moe`, `gemma2`, `gemma`) are in `verified_archs()`, so that set is +currently **optimistic** and should be re-validated before it is relied on. + +`qwen35` was validated the same way muse-glimmer was, feeding llama.cpp's own token ids and +comparing greedy output. On Qwen3.5-0.8B-Q8_0 and on Ternary-Bonsai-27B-Q2_g64 the frontend +reproduces llama.cpp **token for token** (`" Paris.\nThe capital of France is Paris. ..."` and +`" Paris. Paris is the largest city in France. Paris is the most popular"` respectively). Final +logits agree to 1.0% on the 0.8B (sum -771776 vs -779763) and 0.12% on Bonsai (-812702 vs +-813701); the 0.8B figure is in line with the *verified* `qwen3` arch measured through the same +harness, so it is dequant/driver noise rather than an arch defect. + +**`qwen35` is greedy / batch-1 only.** The recurrent conv and delta states are a single +static-shaped block with no batch axis, and `MakeStateful` does not reorder them by `beam_idx` +the way it reorders a KV cache. Beam search or batch > 1 therefore **fails at inference** with a +shape mismatch on the conv window's `Concat` -- it does not silently mix state across beams, so +no wrong output can be produced. Prefix caching and PagedAttention are unavailable for the same +reason: a recurrent state cannot be re-derived from a cached prefix, and cannot be paged. + +What "verified" rests on for this arch, since a hybrid stack is easy to get subtly wrong: + +* Token-exact greedy agreement with llama.cpp on **two real checkpoints** of different size and + quantization (Qwen3.5-0.8B Q8_0, Ternary-Bonsai-27B Q2_g64), through GenAI's own harness. +* A **five-prompt** raw-completion sweep on the 0.8B. Three match llama.cpp exactly; two diverge + mid-continuation. Both divergences are near-ties, and the *already verified* `qwen3` arch + diverges on the same prompts in the same way ("20 years old" vs "22 years old"), so this is the + frontend's numerical baseline rather than a qwen35 defect. +* A **prefill-vs-decode consistency** check (generate N tokens, then re-prefill each own-output + prefix and compare the next token). `qwen35` scores 1 mismatch in 16 -- identical to `qwen3` + (1/16) at the identical position. `llama` scores 0/16. The GDN recurrence is computed chunk-wise + during prefill and step-wise during decode, which are mathematically equal but not bit-equal, so + a near-tie can flip; llama.cpp has the same two code paths. +* Final logits within 1.0% (0.8B) and 0.12% (Bonsai) of llama.cpp on the sum over the vocabulary. + +A caveat on the Bonsai artifacts: **`Ternary-Bonsai-27B-Q2_0.gguf` is not upstream `Q2_0`.** +It does not load in llama.cpp either (`tensor 'output_norm.weight' has offset 337715200, +expected 357580800`; the ratio 0.94444 is exactly `(34/128)/(18/64)`). The file is packed +**g128** -- one f16 scale per 128 weights, 2.125 bits/weight -- while `GGML_TYPE_Q2_0` is +**g64**, 18 bytes per 64 weights, 2.25 bits/weight. The model card says as much, calling the +deployed format "Q2_0_g128" and publishing a separate group-64 pack "matching the 64-value-group +Q2_0 packing in llama.cpp". Use `Ternary-Bonsai-27B-Q2_g64.gguf`; the g128 pack needs PrismML's +fork. The frontend rejects it safely (`tensor 'blk.63.ffn_up.weight' data runs past EOF`) rather +than dequantizing garbage. + +`muse-glimmer` needs a footnote of its own, because it is the one arch whose row was decided +by the *tokenizer*, not the graph. Fed llama.cpp's own token ids, the converted graph +reproduces llama.cpp token-for-token: for `<|begin_of_text|>The capital of France is` all 32 +greedy tokens are identical, and the final logits agree to within the frontend's ordinary +dequantization noise (sum -821515 vs -824015, 0.3%, *tighter* than the `qwen3` Q4_0 control +at 0.17% on values ~3x smaller). Through GenAI it initially looked degenerate, because +GenAI's GGUF tokenizer honored `tokenizer.ggml.add_bos_token` only on the SentencePiece +(`tokenizer.ggml.model = llama`) path; on the BPE (`gpt2`) path it built no CombineSegments +node, so the leading BOS was silently dropped. Muse Glimmer is `gpt2` + `add_bos_token = true` +and is BOS-sensitive, so it looped on `The capital of France is`. Without the BOS the +converted graph picks `" The"` at that position (top-8 `589=12.42 5422=10.97 1573=10.69`); +with it, `" It"` (`1573=17.34`), which is what llama.cpp emits. The gap was arch-independent +(`llama3` and `mistral3` lost their BOS the same way) and lived in GenAI, not in this +frontend; it is fixed in `gguf_tokenizer.cpp` by emitting BOS/EOS as a CombineSegments +segment on every tokenizer path. + +### Measured performance and memory (OpenVINO GenAI, CPU) + +Same runs as the table above. i9-12900K (16C/24T), OV defaults. Prefill = prompt tokens / +TTFT on a ~90-340-token prompt; decode = 1/TPOT over 32 greedy tokens, steady-state +iteration. `peak RSS` and `peak anon` are the maxima of `Rss:`/`Anonymous:` from +`/proc/self/smaps_rollup`, sampled every 20 ms in-process; `anon` is the part that +genuinely requires RAM (see [`frontend_design.md`](frontend_design.md) on the memory model). +`load` is `.gguf` → OV graph → `compile_model`. + +| Arch | Model MiB | load s | prefill t/s | decode t/s | peak RSS MiB | peak anon MiB | +|---|---|---|---|---|---|---| +| `qwen2` | 468 | 4.7 | 1196.3 | 78.68 | 1492 | 1418 | +| `qwen3` | 609 | 4.9 | 865.1 | 64.94 | 1612 | 1538 | +| `hunyuan-dense` | 338 | 4.2 | 630.3 | 62.93 | 1545 | 1473 | +| `gemma3` | 768 | 1.9 | 662.6 | 43.76 | 2610 | 2536 | +| `qwen3moe` | 531 | 5.7 | 158.0 | 44.96 | 1987 | 1912 | +| `maincoder` | 640 | 5.1 | 324.3 | 36.28 | 2321 | 2245 | +| `llama-embed` | 770 | 6.0 | 266.9 | 35.75 | 2601 | 2529 | +| `exaone4` | 774 | 4.3 | 236.5 | 35.54 | 2594 | 2520 | +| `olmoe` | 4018 | 12.9 | 86.6 | 35.17 | 11760 | 11684 | +| `llama` | 770 | 6.0 | 323.8 | 35.08 | 2628 | 2556 | +| `bailingmoe2` | 5573 | 44.9 | 108.1 | 26.97 | 36956 | 36237 | +| `deepseek2-ocr` | 1859 | 7.4 | 299.2 | 71.38 | 5392 | 5318 | +| `mellum` | 7697 | 27.7 | 70.8 | 21.48 | 21282 | 21194 | +| `gemma` | 1425 | 3.8 | 149.6 | 18.47 | 4570 | 4501 | +| `plamo3` | 1574 | 3.3 | 154.9 | 16.87 | 5661 | 5594 | +| `smollm3` | 1826 | 8.4 | 121.9 | 14.88 | 5972 | 5895 | +| `minicpm` | 1649 | 3.3 | 126.0 | 14.82 | 5041 | 4968 | +| `ernie4_5-moe` | 12873 | 46.6 | 45.8 | 14.67 | 36551 | 36460 | +| `gemma2` | 1629 | 3.4 | 132.8 | 14.35 | 5692 | 5617 | +| `mistral3` | 2047 | 8.8 | 103.6 | 12.96 | 6871 | 6796 | +| `phi3` | 2282 | 4.4 | 108.3 | 11.84 | 8197 | 8130 | +| `gemma4` | 4746 | 9.6 | 63.6 | 9.24 | 12309 | 11953 | +| `gpt-oss` | 11548 | 89.3 | 18.7 | 5.48 | 123720 | 123493 | +| `qwen35` (Qwen3.5-0.8B Q8_0) | 795 | 1.5 | 537.4 | 41.38 | 2292 | 2207 | +| `qwen35` (Bonsai-27B Q2_g64) | 7234 | 28.7 | 21.4 | 3.75 | 24009 | 23903 | +| `muse-glimmer` | 15512 | 28.3 | 28.9 | 2.66 | 37457 | 37357 | + +Numbers from architectures marked degenerate above still describe real compute cost (the +graph runs, it is just numerically wrong), so they are kept for completeness. + +The two `qwen35` rows come from the same `gguf_arch_check` harness as every other row: GenAI +runs this architecture now that `MakeStateful` also rewrites the recurrent conv/delta states and +`AdaptToGenAI` expands `position_ids` into M-RoPE's four sections. Both reproduce llama.cpp +token-for-token; see the numerical notes below. + +Against llama.cpp on the same host, `qwen35` decodes at **0.74x** (41.4 vs 56.1 tok/s) -- the +usual SDPA-path ratio -- while prefill is not directly comparable here because the two harnesses +use different prompt lengths (95 vs 5 tokens); on the matched 5-token prompt the frontend +prefills 1.10x faster (191.4 vs 174.5 tok/s). Bonsai inverts the decode picture dramatically: +**6.5x faster** (3.75 vs 0.58 tok/s). That is not an OpenVINO win so much +as an upstream gap -- ggml ships no x86 SIMD kernel for `Q2_0`, so `ggml_vec_dot_q2_0_q8_0` +falls back to the generic scalar reference, while the frontend lowers Q2_0 into the ordinary +u2 compressed-weights MatMul the CPU plugin already optimizes. (PrismML's own fork ships +tuned CUDA/Metal kernels; this comparison is upstream-CPU vs OpenVINO-CPU.) Memory is the +other side of that trade: llama.cpp mmaps the weights and peaks at 7.5 GiB for Bonsai, the +frontend materializes decompression constants and peaks at 22.7 GiB (3.1x the file). + +`muse-glimmer` is the largest checkpoint in the table (30B, 15.5 GiB) and is memory-bandwidth +bound at 2.66 tok/s decode; llama.cpp on the same file and host does 3.46 tok/s, so the +frontend lands at **0.77x llama.cpp**, in line with the 0.51-0.64x SDPA ratios measured on +the smaller models below. Peak anon is 2.4x the file, better than the 3-4x typical elsewhere, +because Q4_0 stays 4-bit and only the Q6_K tensors are requantized to Q8_0_C. + +One outlier remains: `gpt-oss` peaks at **124 GiB from an 11.5 GiB file (11x)**, versus a +typical 3-4x elsewhere. On a smaller-RAM host it would OOM. The cause is the compressed-weights +type gate on the MoE expert matmul described below — for gpt-oss the expert type is MXFP4 +(`f4e2m1`), which the frontend dequantizes on-graph in `MUL_MAT_ID` rather than routing through +`GatherMatmul` at all, so the plugin-side widening does not reach it. + +### Measuring performance correctly + +Three traps have each produced a wrong published number at least once. Read this before +benchmarking, especially when comparing against llama.cpp. + +**1. Disable prefix caching when measuring prefill under PagedAttention.** `ATTENTION_BACKEND=PA` +routes through GenAI's ContinuousBatching adapter, and `get_latency_oriented_scheduler_config()` +(GenAI `src/cpp/src/utils.cpp`) sets `enable_prefix_caching = true` by default. Benchmarks +typically repeat one fixed prompt for N iterations to amortize the first-request dynamic-shape +compile — with prefix caching on, **every iteration after the first is a cache hit**, so the +reported TTFT is not prefill work at all. On Llama-3.2-1B this reads 125 ms cached vs 300 ms +uncached (SDPA measures 304 ms): the cache made PA look 2.4x faster at prefill than an identical +computation. Pass an explicit scheduler config with it off: + +```cpp +ov::genai::SchedulerConfig sched; +sched.max_num_batched_tokens = std::numeric_limits::max(); // as the latency default +sched.enable_prefix_caching = false; +props[ov::genai::scheduler_config.name()] = sched; +``` + +Note the asymmetry that makes this specifically a *comparison* hazard: SDPA ignores this knob +entirely, and neither llama.cpp reference path caches across runs. `llama-bench` calls +`llama_memory_clear()` inside the rep loop before the timer starts (its state-reuse path is gated on +`-d/--n-depth > 0`, which defaults to 0); `llama-cli` is single-shot and its `--prompt-cache` +defaults to empty. So a cached PA number is being compared against two uncached ones. Sanity check: +run llama-bench with `-r 5` and confirm variance stays under ~1% — a cache hit shows as a large drop +after rep 0, not as noise. Prefix caching is a real PA capability worth reporting *separately*; it +just is not prefill throughput. + +**2. Confirm PA is actually in use — the fallback is silent.** GenAI catches a PA initialization +failure and falls back to SDPA with only a `GENAI_WARN` (`src/cpp/src/llm/pipeline.cpp`), and the +default log level is `ERR` (`src/cpp/src/logger.cpp`), so **the warning is invisible unless you set +`OPENVINO_LOG_LEVEL=4`**. Correct output and plausible timings therefore prove nothing. Counting +`PagedAttention` in the `ov::Model` is also insufficient — that is the graph handed *to* the plugin. +Check the compiled **runtime** graph: + +```cpp +auto rt = compiled_model.get_runtime_model(); +for (const auto& op : rt->get_ops()) + hist[op->get_rt_info().at("layerType").as()]++; +``` + +For Llama-3.2-1B (16 layers) the two backends must look like this — note `MemoryInput`/`MemoryOutput` +disappearing, since PA replaces the stateful KV cache with the plugin's block-table cache. A rename +alone would not do that: + +| runtime `layerType` | SDPA | PA | +|---|---|---| +| `PagedAttention` | 0 | 16 | +| `ScaledDotProductAttention` | 16 | 0 | +| `MemoryInput` / `MemoryOutput` | 32 / 32 | 0 / 0 | + +**3. Drop iteration 0 and pin the comparison.** Iteration 0 carries the first-request dynamic-shape +compile (several hundred ms to seconds); average iterations 1..N-1 for steady state. Compare on the +same `.gguf` file, the same prompt text, and the same `n_ctx` — llama.cpp preallocates the whole +`n_ctx` KV cache up front while OV's stateful cache grows on demand, so a mismatched context length +makes the memory figures incomparable. Also record the thread counts: llama.cpp auto-selects +P-cores only (8 on an i9-12900K) where OV uses all 24 by default, which is not a like-for-like +core budget unless equalized. + +Putting it together — the three commands behind the table below. llama.cpp is measured twice because +`llama-bench` gives steady-state kernel throughput with no process/load overhead, while `llama-cli` +walks the same end-to-end path as the GenAI sample and so is the fair peak-RSS comparison: + +```sh +# steady-state kernel throughput (cache cleared per rep; -r 5 to confirm low variance) +llama-bench -m "$MODEL" -p 128 -n 128 -r 5 + +# end-to-end, for max-RSS parity with the GenAI sample +/usr/bin/time -v llama-cli -m "$MODEL" -p "$PROMPT" -n 128 -c 1024 \ + -no-cnv -st --temp 0 --seed 1 --no-warmup --ignore-eos + +# GenAI, once per backend; the sample turns prefix caching off for PA (trap 1) and +# reports per-iteration TTFT/TPOT so iteration 0 can be dropped (trap 3) +/usr/bin/time -v bench_gguf_perf "$MODEL" "$PROMPT" 128 4 {SDPA|PA} +``` + +`bench_gguf_perf` is the GenAI sample at `samples/cpp/text_generation/bench_gguf_perf.cpp`; keep the +same `-c/n_ctx` on both sides and the same prompt text everywhere. + +#### PagedAttention vs SDPA vs llama.cpp (measured under the rules above) + +i9-12900K, Q4_K_M, 128 generated tokens, 4 iterations with iteration 0 dropped, `n_ctx=1024`, +prefix caching **off**, PA presence confirmed in the runtime graph for every row. llama.cpp is its +default ggml CPU backend (`llama-bench pp128/tg128`), 8 threads by its own auto-selection. + +| Model | prompt tok | prefill t/s (lcpp / SDPA / PA) | decode t/s (lcpp / SDPA / PA) | PA/SDPA | PA/lcpp | peak RSS GB (lcpp / SDPA / PA) | +|---|---|---|---|---|---|---| +| Llama-3.2-1B | 87 | 528 / 291 / 291 | 73.7 / 39.7 / 41.3 | 1.04 | 0.56 | 1.30 / 2.51 / 2.51 | +| Maincoder-1B | 68 | 591 / 405 / 409 | 84.1 / 43.9 / 45.7 | 1.04 | 0.54 | 1.10 / 2.24 / 2.24 | +| gemma-3-1b | 75 | 411 / 467 / 429 | 75.5 / 47.1 / 47.1 | 1.00 | 0.62 | 0.92 / 2.49 / 2.49 | +| Ministral-3-3B | 631 | 158 / 111 / 120 | 27.1 / 14.8 / 15.0 | 1.01 | 0.55 | 3.60 / 6.67 / 6.65 | +| SmolLM3-3B | 302 | 173 / 135 / 136 | 30.2 / 16.8 / 17.2 | 1.02 | 0.57 | 3.21 / 5.71 / 5.70 | +| mistral-7b-v0.1 | 55 | 68 / 52 / 56 | 13.7 / 6.94 / 6.99 | 1.01 | 0.51 | 7.37 / 11.81 / 11.82 | +| Ministral-8B | 53 | 69 / 40 / 39 | 12.8 / 7.01 / 7.05 | 1.01 | 0.55 | 7.92 / 13.10 / 13.11 | +| gemma-4-E4B | 58 | 111 / 55 / 56 | 18.2 / 11.1 / 11.6 | 1.04 | 0.64 | 6.96 / 12.36 / 11.92 | + +**PA vs SDPA: parity.** Decode 1.00-1.04x (PA marginally ahead on all 8), prefill within +-8%, peak +RSS within 0.2% except gemma-4 where PA is 0.44 GB lower. Enabling PA costs nothing; the reason to +use it is that continuous batching, prefix caching and multi-sequence serving become available at +all, which the SDPA-only graph could not do. + +**PA vs llama.cpp: decode 0.51-0.64x**, prefill 0.55-1.04x, peak RSS 1.7-2.4x. These ratios match +what the SDPA path already measured, so PA neither introduces nor closes that gap — see +[`frontend_design.md`](frontend_design.md) on the memory model for the RSS side. + +### MoE expert weights and the compressed-weights type gate + +Worth knowing when picking a quantization for a MoE model, though the handling is entirely +plugin-side. MoE expert weights do not go through `FullyConnected`: `MUL_MAT_ID` lowers to the +CPU plugin's `GatherMatmul` (equally, to `GroupedMatMul` on the public-op side — on CPU +`ConvertGroupedMatMulToGatherMatmul` rewrites it into the same node *before* the compression +pass, so the two are indistinguishable here). That node accepts a **narrower set of compressed +weight types than `FullyConnected` does**: + +| | accepted compressed weight types | +|---|---| +| `FullyConnected` | `u8, i8, u4, i4, nf4, f4e2m1, u2` | +| `GatherMatmul` / GPU grouped-matmul | `u8, i8, u4, i4` | + +If an expert weight's element type is outside the second set, `ConvertGatherMatmulToGather +MatmulCompressed` does not fire, the `Convert -> Subtract -> Multiply` dequantization block stays +in the graph, and constant folding materializes the experts **in f32** — a 16x expansion off a +2-bit type, i.e. far more than the quantization was saving. + +Q2_K is the case this affects: its weights map to `u2`. The CPU plugin's +`WidenGatherMatmulWeights` pass handles it by re-emitting *expert* weight constants as `u4` +(lossless — raw Q2_K values are `[0..3]`, which fit a nibble) at 2x the weight bytes, which is +much cheaper than falling off the compressed path. Dense `u2` weights are left alone. This is a +plugin-side workaround for a missing `u2` expert-matmul executor and needs nothing from the +frontend, which emits plain `u2` either way. Measured on Q2_K models, peak anonymous memory: + +| Model | file MiB | before | after | +|---|---|---|---| +| Qwen3-0.9B-A0.6B (`qwen3moe`) | 373 | 4251 | 2071 | +| Ling-mini-2.0 (`bailingmoe2`) | 5573 | 117245 | 36237 | + +Decode also improves (bailingmoe2: 12.7 → 27.0 t/s) because the experts are no longer read from +f32. + ## Adding a new architecture Support for a new architecture is a combination of: @@ -53,3 +438,9 @@ Support for a new architecture is a combination of: the weight path (`src/quant/weights.cpp`). 3. **Real-model verification** — run a real `.gguf` end-to-end as above before adding the architecture to the Supported table. + +For the native builder specifically, see +[`adding_an_architecture.md`](adding_an_architecture.md) — for a same-family arch the change +is usually just adding the name to `experimental_archs()` plus the `arch_uses_neox_rope()` +classification, and promotion to `verified_archs()` should require the GenAI-vs-llama.cpp +comparison above. diff --git a/src/frontends/gguf/docs/testing_architecture.md b/src/frontends/gguf/docs/testing_architecture.md new file mode 100644 index 00000000000000..f7c8aa4692958c --- /dev/null +++ b/src/frontends/gguf/docs/testing_architecture.md @@ -0,0 +1,446 @@ +# GGUF Frontend — Testing Architecture + +Design proposal, partially implemented. **T0 and T1 are now built** — see §11 for exactly what +landed and what its measured cost is. Everything above T1 is still a proposal. Markers below: +**[exists]** was already there, **[done]** landed with the T0/T1 work, and anything still marked +**gap** is not implemented. + +## 1. What is actually being tested + +Three repositories are involved, but only **one** of them holds shared logic. Everything else is a +consumer of it: + +``` + ┌──────────────── OpenVINO ─────────────────┐ + │ op translators + normalization passes │ ← the only shared code + │ GgufDecoder (contract, PUBLISHED header) │ + └────┬─────────────────────────────┬─────────┘ + implements ──────┘ └────── implements + GgufBuilderDecoder (in OpenVINO) GgmlOvDecoder (in llama.cpp) + src/frontends/gguf/src/builder/ ggml/src/ggml-openvino/ + │ │ + FrontEnd::convert(".gguf") ggml-openvino backend + │ │ + openvino.genai llama-completion / llama-bench + MakeStateful + AdaptToGenAI LlamaCppToStateful + + tokenizer from rt_info +``` + +That shape gives **four seams**, and a test that does not name its seam is not a test of anything in +particular: + +| | Seam | Fails as | +|---|---|---| +| **S1** | ggml op semantics ↔ OV translator | wrong numbers for one op, every arch that uses it | +| **S2** | `.gguf` file ↔ builder graph | one arch converts wrong / not at all | +| **S3** | `GgufDecoder` contract ↔ its two implementations | the two decoders disagree; one path silently differs | +| **S4** | converted model ↔ a runtime's IO contract | model is correct but nothing can drive it | + +Crossed with two axes that multiply: **architecture** (28 accepted by the builder, 101 known to +llama.cpp) × **execution mode** (stateless / stateful / genai-adapted / static). + +**S3 is the only seam that spans repositories, and it is the only one with no gate at all today.** +The `beam_idx` bug lived exactly there: the builder declared an input the cgraph decoder did not, so +the two decoders produced different stateless IO. It was caught by design review, not by a test, and +a gate costing milliseconds would have caught it. + +## 2. The governing constraint: no dependency cycle + +OpenVINO must not depend on llama.cpp at build or test time — that is the documented reason the +native builder exists at all (see [frontend_design.md](frontend_design.md), "Why the native path +does not use llama.cpp"). The testing architecture must not smuggle that dependency back in. + +The rule that follows: **no llama.cpp dependency in the product, or in anything that has to build or +run for the frontend to be usable.** + +- ggml op oracles → pregenerated `.npy` committed under `tests/test_data/`. **[exists]** + ([gen_ggml_reference.c](../tests/gen_ggml_reference.c) is run by hand and its output committed.) +- per-arch model fixtures → a committed *manifest* of reviewed expectations; the fixture files + themselves are generated in the Linux CI job from a pinned llama.cpp (§4a). +- the real `GgmlOvDecoder` pairing → tested **in llama.cpp**, against a contract suite OpenVINO + *publishes* (§6). + +Two places deliberately do build llama.cpp, both test-only and both fenced: + +- the **arch-fixture generation step** in the Linux build job, at a pinned commit (§4a). It produces + test data, is skipped when the GGUF component is unaffected, and its absence degrades to a skipped + test suite rather than a broken build — which is what keeps it out of the product's dependency graph. +- a **nightly canary job** that builds llama.cpp against OpenVINO master (§8). + +## 3. Tiers, and the cost principle + +Each defect class should be caught by the cheapest tier capable of catching it. Today most classes +are caught by the most expensive one — a human running a manual sweep over a local model zoo — which +is why the seven defects recorded in the 2026-07-28 arch sweep were found all at once, late, by hand. + +| Defect class | Cheapest catcher | Cost | Today | +|---|---|---|---| +| wrong op formula | T0 op unit | ms | **[done]** all 58 registered ops, gated | +| wrong dequant for a quant type | T0 dequant unit | ms | **[exists]** `test_dequant_vs_ggml.cpp` | +| conversion/pass contract broken | T1 graph unit | ms | **[exists]** `test_extensions.cpp` | +| arch X stops converting | T1 synthetic fixture | 2.5 ms | **[done]** `test_arch_conversion.cpp`, 101 archs | +| graph for arch X changed unintentionally | T1 fingerprint | 2.5 ms | **[done]** pinned per converting arch | +| the two decoders disagree | T2 contract | ms | **gap** — nothing | +| wrong numerics for arch X | T3 logits vs ggml CPU | seconds | **gap** — manual | +| tokenizer / E2E text | T4 | minutes | **[exists]** partial, opt-in | +| perf / memory regression | T5 | minutes | **gap** — ad-hoc shell scripts | + +### T0 — op and kernel units (hermetic, OpenVINO) +Single-op models through `SingleOpDecoder`, checked against committed ggml outputs. Already the +strongest tier, and now complete with respect to the op table: +- **[done] Every op registered in `op_table.cpp` is converted by some test**, enforced by + `test_op_coverage.cpp`. The "tested" side of that comparison is collected at run time from + `SingleOpDecoder`'s constructor, so it cannot drift from what the tests actually do. Closing the + last 9 ops found a real defect: `GGML_UNARY_OP_GELU_QUICK` was implemented as tanh-GELU instead of + ggml's `x*sigmoid(1.702x)` — off by 2.2e-2, and by ~7 orders of magnitude in the negative tail. + Nothing had converted the op, and the correct reference data was already committed but unread. +- **[done]** The real-ggml `.npy` pairs are wired in as a second, independent check + (`GGUFUnaryVsGgml`): the parameterized cases encode the ggml formula by hand in C++, these run the + actual kernel's captured output, so a misreading cannot be baked into both sides. The tolerances + are set by ggml's own fp16 GELU lookup table (~2e-3 / ~3.3e-3), not by OV. +- **Gap:** reference generation is still a manual two-step (`gen_ggml_reference.c` → `.py` → + `.npy`). Make it one scripted target with a recorded llama.cpp commit hash, the way + `gen_arch_fixtures.py` now does for T1(b). +- The oracle must stay *real ggml*, never a numpy reimplementation of the formula — a numpy oracle + encodes the same misreading the translator might have (this is exactly the trap `GELU_QUICK` fell + into). Op tests therefore either compute their reference inline, where the semantics are + unambiguous, or compare against captured ggml output; there is deliberately no numpy formula + oracle in the tree. + +### T1 — graph and contract units (hermetic, OpenVINO) +Two kinds, both millisecond-cheap and both precommit: + +**(a) Pass/mode contracts** — what `test_extensions.cpp` does now: stateless is the default, +`MakeStateful` as a `DecoderTransformationExtension` swaps the mode, `skip_caches` works, the +stateless graph's inputs are exactly the decoder's inputs, `MakeStateful` adds exactly `beam_idx`. +Keep and extend; this is the tier that should own every IO-contract invariant. + +**(b) Per-arch conversion over synthetic fixtures** — **[done]**, and it was the largest single +coverage win available. `test_arch_conversion.cpp`, §4a. + +### T2 — cross-decoder equivalence (S3) +The invariant: *for one model, both decoders produce the same stateless graph.* Decomposes into two +claims that need different homes: + +- **Neither decoder invents or omits an input.** Testable in OpenVINO with a test-double second + decoder — `SplitIoDecoder` in `test_extensions.cpp` covers the + `get_model_inputs`/`get_model_extra_inputs` split half. **[exists]** +- **The real `GgmlOvDecoder` agrees with `GgufBuilderDecoder` on a real file.** Not expressible in + OpenVINO without the forbidden dependency. Belongs in llama.cpp: it has both decoders in-process + (it links `openvino::frontend::gguf` and can drive it on the same path). Compare + graph fingerprints, not text. §6. + +### T3 — numerics per architecture (real models, nightly) +Oracle is **llama.cpp's default plain ggml CPU backend** on the same `.gguf`. Compare logits, not +generated text: text comparison is a lossy proxy that only fails after drift has already compounded, +and greedy streams legitimately diverge late. NMSE on the first-position logits vector is a sharp, +threshold-able signal — this is exactly what `test-llama-archs` already computes (§6). + +Generated-text checks stay useful as a *coarse* smoke gate (gibberish is unmistakable) but must not +be the primary numerical assertion. + +### T4 — end-to-end product (real models, nightly) +GenAI `LLMPipeline` on a `.gguf`: tokenizer built from `rt_info`, `MakeStateful` + `AdaptToGenAI`, +sampling, chat template. **[exists]** — `test_gguf_reader.py` in precommit, +`test_cli_text_gguf.py` (WWB similarity vs llama-cpp-python) opt-in behind `WWB_GGUF_TESTS=1`. +Gap: the WWB suite is not wired into any scheduled job, so in practice it runs when someone +remembers to run it. + +### T5 — performance and memory (real models, nightly) +Fixed small model set; record TTFT / TPOT / peak *anonymous* memory (not RSS — the file-backed mmap +of the weights dominates RSS and is not the interesting number) against a tracked baseline, with the +llama.cpp default ggml CPU backend as the control on the same file. Measured noise floor on this +machine is ±2 %, so a threshold tighter than ~5 % will flap. + +## 4. Fixtures — the load-bearing decision + +Every tier above T1(a) needs models, and that is the reason none of them are automated: real GGUFs +are gigabyte-scale, network- and license-encumbered, and cannot be committed. So the arch matrix is +checked by hand, on one developer's local zoo, and regressions surface in batches. + +**Two fixture classes, split by what they can actually prove.** + +### 4a. Synthetic tiny GGUF, in-repo, precommit — **[done]** + +llama.cpp can already emit a minimal valid `.gguf` for every architecture it knows, via +`llama_model_saver`, exposed as `test-llama-archs --out `. Measured: + +- **101 architectures**, 5–7 MB each, all written in **1.8 s**. +- Converted through the frontend in **~2.5 ms** each. +- **23** archs convert cleanly; each one's `{op count, input count}` is now pinned as a fingerprint. +- **`jais2` fails outright** — `MatMul` dimension mismatch, `ffn_down` fed a 192-wide operand against + a `[256,384]` weight. A real defect in a currently-`experimental` arch, surfaced in the first + minutes of running this. (Adds to the seven from the recorded arch sweep.) Recorded in the manifest + as `broken`, which asserts it *still fails* — fixing it makes the test demand promotion, so the + known-broken list cannot rot into a permanent excuse list. +- The other **77** archs are cleanly rejected as outside the accept-list — which is not a failure but + a free, exact, machine-checkable statement of *what is not supported yet* (§5). The test asserts + the rejection comes from the accept list specifically, so a crash or a silent wrong-graph success + is distinguishable from "not supported". + +Two things had to be handled: + +- **The frontend hard-required `general.file_type`, which llama.cpp's own writer does not emit** — + every fixture failed to load on it. That the frontend required a key llama.cpp omits was itself the + finding; the value is *written and never read anywhere* in the frontend (every weight carries its + own type in the tensor info, which is what the dequant path uses), so it is now optional in + `quant/gguf.cpp` rather than injected by the generator. +- **101 × 5 MB is not shippable, and not necessary.** The whole structural input to architecture + detection and graph construction is the GGUF *header*: the bytes before `min(tensor.data_offset)`, + i.e. magic + KV metadata + tensor table. That is what a fixture is + (`test_data/arch_fixtures/*.gguf.hdr`, ~25 KB each); the C++ test appends the manifest's recorded + number of zero bytes to rebuild a loadable file. Zero-filling is exact here rather than approximate + — every tensor the model saver writes is F32, so no dequantization is involved and no block-scale + field can be invalidated by a zero. Verified: zero-filled, seeded-RNG and sparse reconstructions all + convert to the identical graph as the original. + + The headers are **seed-independent** — verified byte-identical for `-s 1` and `-s 999`, since the + seed only feeds weight values. So the fixture set is reproducible regardless of the generator's + `std::random_device` default seed, though `gen_arch_fixtures.py` pins one anyway. + +- **The headers are generated in CI, not committed.** They started out committed (2547 KB raw → 120 KB + packed) and that works, but it puts 101 opaque binaries in the tree that no reviewer can read and + that have to be regenerated by hand whenever llama.cpp adds an architecture. Instead, the Linux + build job clones llama.cpp at a pinned commit, builds only `test-llama-archs`, and runs + `gen_arch_fixtures.py --fetch` into the tests artifact — **37.6 s** end to end, verified to + reproduce all 101 committed headers byte-identically. What stays committed is `manifest.txt`: the + reviewed expectation per architecture, which is the part that is source rather than output. + + **The pin is load-bearing, not hygiene.** `test-llama-archs` writes whatever KVs llama.cpp currently + defines, so an upstream KV addition shifts every fixture's bytes at once. Measured: between the pin + and a head 13 days later, all 101 headers changed (three new `attention.indexer.*` KVs) and two new + architectures appeared — while the op counts stayed identical. Following upstream would therefore + turn an unrelated llama.cpp commit into a red OpenVINO precommit; bumping the pin is its own + reviewed change. + + Consequences, stated rather than discovered later: the arch suite runs on **Linux only** (the step + is in `job_build_linux.yml`; mac/Windows/arm64 get the manifest without fixtures and + `GTEST_SKIP()`), and the generation step costs one llama.cpp build per affected Linux build job. + `test_arch_conversion.cpp` counts present fixtures rather than testing a boolean, so *no* fixtures + is a skip and a *partial* set is a failure — an incomplete generation cannot quietly shrink + coverage to whatever happened to be written. + +What synthetic fixtures **can** prove: the arch converts; the graph is structurally what it was +(fingerprint); the IO contract holds; the accept-list is honest; both decoders agree structurally. + +What they **cannot** prove, verified rather than assumed: +- **No vocabulary** — `tokenizer.ggml.model` is `no_vocab`, so the GenAI path rejects them at the + tokenizer and T4 is out of reach. Confirmed by running one through `bench_gguf_perf`. +- Weights are random small normals in F32/F16, so **quantization-kernel accuracy is untouched** and + any coherent-text assertion is meaningless. +- I could not drive one to a successful inference by hand-feeding the stateless inputs (a + `ScatterUpdate` index bound, then a mask-width broadcast). That is a finding in its own right — + **the stateless IO contract is undocumented and unexercised at runtime** — but it means T3-style + numerics on synthetic fixtures needs a documented, tested input-feeding helper before it will work. + +### 4b. Real small models, cached, nightly + +Unavoidable for T3/T4/T5. Rules: +- One *smallest available* real model per **verified** arch, pinned by repo + filename + revision. +- Fetched into a CI-persistent HF cache, never per-run. The existing model-hub nightly jobs already + mount `/mount/caches/huggingface`; reuse that mechanism rather than inventing one. +- Many of the models in the local zoo are symlinks into evicted HF blobs — a dangling-fixture check + must run first and report *skipped* distinctly from *failed*, or the suite reports 7 phantom + failures (observed). + +## 5. One architecture registry + +There are currently **three** independent lists of what works, and they can drift apart silently: +`verified_archs()` / `experimental_archs()` in `builder/arch_registry.cpp`, the tables in +[supported_models.md](supported_models.md), and the model list in genai's `test_cli_text_gguf.py`. + +Replace with one machine-readable registry in OpenVINO, consumed by everything else: + +```yaml +llama: {builder: verified, converts: yes, numerics: yes, fixture: } +gemma4: {builder: verified, converts: yes, numerics: yes, kv_precision: f16} +jais2: {builder: experimental, converts: NO, ticket: XXXXX} # found 2026-07-30 +exaone4: {builder: experimental, converts: yes, numerics: NO, ticket: XXXXX} +qwen3next:{builder: unsupported} +``` + +Three properties make this worth doing: + +1. **Known-broken does not block, but newly-broken does.** `converts: NO` is an xfail. +2. **XPASS is a failure.** An entry that starts passing fails the test, forcing the ledger to be + updated. Without this, a ledger rots into a permanent list of excuses — the mechanism that lets + `test-llama-archs`'s eleven `// FIXME` skips persist. +3. **`supported_models.md` is generated from it**, so the documentation cannot drift from the code, + and the promotion path experimental → verified is a single reviewable diff. + +## 6. llama.cpp's side, and the one big unexploited gate + +llama.cpp already has precisely the harness T3 wants, and it is currently switched off. + +`tests/test-llama-archs.cpp` enumerates every architecture, builds a tiny model, runs it on **every +registered ggml backend**, and reports **NMSE against the CPU backend** with a `1e-4` threshold plus +a GGUF round-trip check. `ggml-openvino` registers as a device and is already enumerated — it shows +up as `OpenVINO Runtime` in the device column. So the matrix "all archs × OV-vs-CPU logits" is +already written; it just does not run: + +- `build-openvino.yml` runs `ctest -L main -E "test-llama-archs"` on both CPU and GPU, with + `# TODO: fix and re-enable the test-llama-archs test below`. +- Verified why: with `GGML_OPENVINO_DEVICE=CPU` it starts the OpenVINO device row and then core-dumps. + Cause not yet isolated. + +**Re-enabling `test-llama-archs` for the OpenVINO device is the highest-value change available on the +llama.cpp side** — it converts the entire per-arch numerical sweep from a manual activity into a +precommit gate, using upstream's own harness and threshold, with no new infrastructure. It needs the +crash root-caused first, and it will need an expectations mechanism (§5) since not every arch is +expected to pass. + +llama.cpp should also host: +- **The real cross-decoder equivalence test (T2/S3).** It is the only process with both decoders. Same + file, converted both ways, graph fingerprints compared. +- **A `GgufDecoder` contract suite instantiated against `GgmlOvDecoder`.** OpenVINO publishes the + suite as headers and llama.cpp instantiates it — the pattern already used by + `src/frontends/tests/frontend/shared/include/*.hpp` for the other frontends. This is what makes the + contract testable from the implementer's side without OpenVINO depending on llama.cpp. + +## 7. openvino.genai's side + +Owns T4 and the contract at S4 — and only that. Its GGUF surface is the *consumer* wiring: +`MakeStateful` registration, `AdaptToGenAI`, tokenizer-from-`rt_info`, KV precision. + +- **[exists]** `test_gguf_reader.py` in precommit, guarded on the `GGUF` smart-CI component. +- **[exists]** `test_cli_text_gguf.py` — WWB similarity against llama-cpp-python on the same file, one + model per arch family. Wire it into a scheduled nightly instead of leaving it behind an env var. +- **Gap:** no test asserts the *IO contract* `AdaptToGenAI` depends on. It asserts `beam_idx` exists + and throws a clear message otherwise, which is good, but nothing pins the rest of the contract + (`input_ids`/`attention_mask`/`position_ids` names, dtypes, ranks, the `[b,seq,vocab]` logits + reshape). Those are graph-level facts, cheap to assert, and belong in genai's C++ unit tests — + ideally as the *same* assertions the frontend's `adapt_to_genai.hpp` doc block states in prose. +- **Gap:** GenAI must not silently accept a stateless model. The assert exists; it needs a test. + +Anti-pattern to avoid: genai's WWB similarity score becoming the de-facto detector for op-level +bugs. It is a slow, noisy, threshold-y signal at the far end of the pipeline. If a translator bug is +first noticed as a similarity drop in a nightly WWB run, the T0/T1 gates have failed. + +## 8. CI placement + +**OpenVINO** +- *Precommit* — T0 + T1 over synthetic fixtures. Target **< 60 s**; **[done]** the whole suite is + **231 tests in 1.35 s**, so the budget is not a constraint at this size. +- **[done]** *Plumbing.* There was **no `GGUF_FE` component** (genai has `GGUF`; OpenVINO had + nothing), so nothing could be scoped to frontend changes, and the `GGUF frontend tests` step in + `job_cxx_unit_tests.yml` ran unconditionally — the only frontend step with no `if:` guard. Now: + `GGUF_FE` in `.github/components.yml` (`revalidate: [CPU]`, `build: [CPU]` — the op tests infer on + the CPU plugin), `'category: GGUF FE'` in `.github/labeler.yml` (which is what makes the component + name resolve, via CI's `component_pattern: "category: (.*)"`), the `if:` guard on the test step, + `ov_gguf_frontend_tests` in `.github/coverage/tests_cpp.yml`, and a step in + `linux_sanitizers.yml`. Verified in both directions with smart_ci itself: a GGUF-only change + affects 12 components including CPU build+test; a PDPD-only change yields `GGUF_FE: None`. +- **[done]** *Fixture generation (§4a)* sits in `job_build_linux.yml`, not in the test job, and that + placement is forced: `job_cxx_unit_tests.yml` has **no source checkout and no cmake** (the `ov_test` + image ships `clang` only). The build job has both, so it generates the fixtures into + `$INSTALL_TEST_DIR/tests/test_data/arch_fixtures/` and they travel to the test job inside the + existing tests artifact, like any other test data. Consequence: the arch suite is **Linux-only**, and + the four other platform workflows calling `job_cxx_unit_tests.yml` (`ubuntu_24` shares the Linux + build, then `linux_arm64`, `mac_arm64`, `windows_vs2022_release`) skip it. +- *Nightly* — T3 numerics on cached real small models; T5 perf/memory; the **llama.cpp canary**: clone + a pinned llama.cpp, build `ggml-openvino` against freshly-built OpenVINO, run its OV tests. Today + llama.cpp CI pins OpenVINO **2026.2.1 release archives**, so a breaking change to + `decoder.hpp` — a *published* header two repos compile against — is invisible until someone bumps + the pin. The signal belongs where the breaking change lands. + +**llama.cpp** — `build-openvino.yml` exists and runs `ctest -L main` on CPU and GPU. Add: +re-enabled `test-llama-archs` for the OV device (§6), the cross-decoder equivalence test, the +instantiated contract suite. + +**openvino.genai** — existing precommit; promote the WWB gguf suite to a scheduled nightly; add the +S4 IO-contract units. + +## 9. Techniques worth productizing + +Two ad-hoc methods used during recent debugging are more valuable as standing gates: + +- **Graph-neutrality proof.** A change meant to be graph-neutral (refactor, relocation, renaming) + must leave a byte-identical `.bin` and a structurally identical `.xml`. This is how the `beam_idx` + relocation was verified, by hand. [`graph_fingerprint.py`](../tests/graph_fingerprint.py) already + computes the right thing but is gated behind `GGUF_FINGERPRINT_MODELS` because it needs real + models. **With synthetic fixtures it can run unconditionally in precommit**, and every refactor + proves its own neutrality for free. Note that raw `.xml` diffing is useless without normalizing + auto-generated node counters — removing one early `Parameter` renumbers everything after it and + produced an 11,352-line semantically-empty diff. +- **Differential seams.** The `DISABLE_OPS` / `DISABLE_TYPES` / `DEBUG_OUTPUT` / eval-callback + first-divergence machinery documented in + [debugging_accuracy.md](debugging_accuracy.md) is a debugging aid today. The + first-divergence comparison in particular is a *test* shape: when a T3 NMSE check fails, the suite + should automatically report the first diverging node rather than only the final logits delta. + +## 10. Sequencing + +Ordered by value per unit of work, not by tier number. + +1. **Arch registry (§5)** + generate `supported_models.md` from it. Unblocks everything else and + immediately records `jais2` and the seven known-broken archs as data instead of prose. The + `manifest.txt` from item 2 is now a partial, machine-checked stand-in for the `converts:` column, + so the registry's remaining job is the `numerics:`/`fixture:` columns and doc generation. +2. **[done] Synthetic fixture generator + manifests (§4a)**, and the per-arch conversion test over + them. Generated in the Linux build job from a pinned llama.cpp (37.6 s), 255 ms to run 101 archs in + precommit, nothing binary in the tree. `general.file_type` made optional. +3. **[done] `GGUF_FE` smart-CI component (§8)**, plus labeler entry, test-step guard, coverage + config, and sanitizers step. +4. **Root-cause and re-enable `test-llama-archs` for the OV device (§6).** Highest single-item value; + turns the whole per-arch numerical sweep into a gate. **Now the top remaining item.** +5. **[done] Close the untested ops (§T0)** and add the completeness check — found the + `GELU_QUICK` defect. +6. **Cross-decoder equivalence in llama.cpp (§6)** — closes the seam the `beam_idx` bug came through. +7. **Real-model nightly (T3/T4/T5)** with the dangling-fixture pre-check and tracked baselines. +8. **Published `GgufDecoder` contract suite (§6)**; llama.cpp instantiates it. +9. **llama.cpp canary in OpenVINO nightly (§8)** — closes the version-skew hole. + +## 11. What landed, and what it cost + +| | Where | Tests | Time | +|---|---|---|---| +| op completeness gate | `tests/test_op_coverage.cpp` | 2 + 1 global gate over 58 ops | ~0 ms | +| new unary op units | `tests/test_ops.cpp` (`GGUFUnary`, +6 cases) | 10 | 4 ms | +| real-ggml unary oracle | `tests/test_ops.cpp` (`GGUFUnaryVsGgml`) | 3 | 2 ms | +| permute / view / topk units | `tests/test_ops.cpp` | 7 | 9 ms | +| per-arch conversion | `tests/test_arch_conversion.cpp` | 101 + 1 manifest guard | 255 ms | +| **whole binary** | `ov_gguf_frontend_tests` | **231** | **1353 ms** | + +Plus one CI step: fixture generation in the Linux build job, **37.6 s**, gated on `GGUF_FE.test`. +Without it the 101 arch tests skip and the other 130 are unaffected. + +Two product fixes came out of it: `GGML_UNARY_OP_GELU_QUICK` had the wrong formula +(`src/op/unary_math.cpp`), and `general.file_type` was a hard requirement for a value the frontend +never reads (`src/quant/gguf.cpp`). + +### The arch fixtures + +CI generates them: the "Generate GGUF arch fixtures" step in +[job_build_linux.yml](../../../../.github/workflows/job_build_linux.yml) runs + +``` +python3 src/frontends/gguf/tests/gen_arch_fixtures.py --fetch --out-dir /test_data/arch_fixtures +``` + +which shallow-fetches the pinned llama.cpp commit, builds only `test-llama-archs`, emits all archs, +truncates each to its header, and rewrites `manifest.txt`. To make the suite live in-tree locally, +run the same thing with `--llama-src ` (or `--fetch`) and no `--out-dir`; the output is +gitignored. Four deliberate behaviours: + +- It **preserves existing expectations** and defaults every *new* fixture to `reject`. The manifest + therefore never asserts "whatever the frontend currently happens to do" — promoting a new arch to + `convert` is a one-word, reviewable edit, and its fingerprint must be added by hand. This is why the + generated fixtures can be trusted without review: the *expectations* were reviewed once, in the + manifest, and regeneration cannot change them. +- `LLAMA_CPP_COMMIT` pins upstream and must be a commit on `ggml-org/llama.cpp` master, since + `--fetch` fetches from there. Bumping it is a reviewed change of its own — see §4a for why. +- It builds with `LLAMA_BUILD_TESTS=ON` and everything else off. That flag is mandatory: + `test-llama-archs` is a test target, and upstream's release builds set it `OFF`, which is why no + prebuilt llama.cpp package ships the binary and `apt install` is not an option. +- The step is gated on `GGUF_FE.test`, so PRs that do not touch the frontend pay nothing for it. + +**Seven of the 28 builder-supported archs have no fixture**, because llama.cpp will not emit them: +`llama_model_saver_supports_arch` (a deny-list in `src/llama-model-saver.cpp`) or +`test-llama-archs`'s own `arch_supported` skips deepseek2-ocr, exaone-moe, gemma3, gemma4, +llama-embed, mellum and plamo3. Investigated: the saver has **no per-arch code at all** — it writes +one flat KV list for every arch — so "unsupported" never means "cannot be serialized". The deny-list +means "does not survive a bit-exact save→load→infer round-trip", which is a stricter property than +this suite needs. Un-gating them locally, **6 of the 7 save and convert fine**; only deepseek2-ocr +genuinely fails, on a missing `expert_feed_forward_length` KV in the fixture builder. So the real +coverage ceiling is 23 of 24 convertible archs, and reaching it needs an upstream fix rather than a +local patch. `gemma4` being among the seven is the notable gap: it is a *verified* arch. diff --git a/src/frontends/gguf/include/openvino/frontend/gguf/adapt_to_genai.hpp b/src/frontends/gguf/include/openvino/frontend/gguf/adapt_to_genai.hpp new file mode 100644 index 00000000000000..ca53c581678a5e --- /dev/null +++ b/src/frontends/gguf/include/openvino/frontend/gguf/adapt_to_genai.hpp @@ -0,0 +1,65 @@ +// Copyright (C) 2018-2026 Intel Corporation +// SPDX-License-Identifier: Apache-2.0 +// + +#pragma once + +#include "openvino/frontend/gguf/visibility.hpp" +#include "openvino/pass/pass.hpp" + +namespace ov { +namespace frontend { +namespace gguf { +namespace pass { + +/// \brief Rewrite a GGUF-frontend model's llama.cpp-style IO into the OpenVINO GenAI +/// LLMPipeline IO contract, so the model can be driven by genai's stateful pipeline. +/// +/// The GGUF frontend emits a stateful decoder with the gguf IO contract: +/// inputs : inp_tokens [1,1,1,D] i32, inp_pos [1,1,1,D] i32, inp_out_ids [1,1,1,D] i32, +/// self_kq_mask [1,1,D,D] f32 (+ self_kq_mask_swa for gpt-oss SWA), +/// token_len_per_seq [1] i64, plus beam_idx [D] i32 from the make-stateful pass +/// output : logits [1,1,seq,vocab] +/// +/// genai's StatefulLLMPipeline instead feeds: +/// inputs : input_ids [b,seq] i64, attention_mask [b,kv_len] i64, +/// position_ids [b,seq] i64, beam_idx [b] i32 +/// output : logits [b,seq,vocab] +/// +/// This pass prepends a subgraph deriving the gguf inputs from the genai ones, rewires the gguf +/// Parameters to it, and reshapes the logits to [b,seq,vocab]. The model must already be stateful: +/// the KV-cache sinks are preserved, and beam_idx (created by the make-stateful pass) passes +/// through unchanged since genai sets that tensor itself. +/// +/// If the required gguf inputs are absent (e.g. the model is already in genai form), the +/// pass is a no-op and returns false. +/// +/// LAYOUT POLYMORPHISM (why the leading dims are derived, never pinned). +/// The result must be valid under both attention backends, which disagree about where the token +/// count lives: plain SDPA feeds input_ids as [1, tokens], while ov::pass::SDPAToPagedAttention +/// rewrites it to rank-1 [tokens] and unsqueezes, so the body sees [tokens, 1]. Both are the same +/// buffer -- ggml's layout is [batch, tokens, heads, head_size] with batch == 1 -- so one graph +/// serves both PROVIDED no node pins the leading two dims. Hence deriving them from the live +/// input_ids here, and reshaping with special_zero in the translators. No backend flag needed. +class GGUF_FRONTEND_API AdaptToGenAI : public ov::pass::ModelPass { +public: + OPENVINO_MODEL_PASS_RTTI("ov::frontend::gguf::pass::AdaptToGenAI"); + + /// \brief Which genai input contract to expose. + /// IdsToLogits : input_ids -> logits (text LLMPipeline). The only mode implemented today. + /// EmbedsToLogits: inputs_embeds -> logits (reserved for the VLM language model, where + /// image+text embeddings are merged outside the graph). Not yet implemented. + enum class InputMode { IdsToLogits, EmbedsToLogits }; + + explicit AdaptToGenAI(InputMode mode = InputMode::IdsToLogits) : m_mode(mode) {} + + bool run_on_model(const std::shared_ptr& model) override; + +private: + InputMode m_mode; +}; + +} // namespace pass +} // namespace gguf +} // namespace frontend +} // namespace ov diff --git a/src/frontends/gguf/include/openvino/frontend/gguf/decoder.hpp b/src/frontends/gguf/include/openvino/frontend/gguf/decoder.hpp index d54fea9eac187a..4fc32f3b31e17b 100644 --- a/src/frontends/gguf/include/openvino/frontend/gguf/decoder.hpp +++ b/src/frontends/gguf/include/openvino/frontend/gguf/decoder.hpp @@ -11,6 +11,7 @@ #include #include +#include "openvino/core/any.hpp" #include "openvino/core/node.hpp" #include "openvino/frontend/decoder.hpp" #include "openvino/frontend/gguf/visibility.hpp" @@ -47,17 +48,17 @@ struct RopeConfig { // Following the established OpenVINO frontend pattern (cf. the PyTorch TorchDecoder + InputModel), // the translators see a GgufDecoder as a NODE decoder: visit_subgraph hands the visitor a fresh // decoder bound to a single node, and every per-node accessor (get_attribute, get_input_*, -// get_output_*, get_op_*) refers to that node -- no node index is threaded through. The -// MODEL-level questions (the graph's Parameter inputs, its output names, the shared RoPE config, -// and node iteration) are asked through ov::frontend::gguf::InputModel, not by treating a decoder -// instance as a "model decoder". The InputModel forwards those to the model-scope accessors below; -// a concrete decoder answers them when queried before visit_subgraph binds it to a node. +// get_output_*) refers to that node -- no node index is threaded through. The MODEL-level +// questions (the graph's Parameter inputs, its output names, the shared RoPE config, and node +// iteration) are asked through ov::frontend::gguf::InputModel, not by treating a decoder instance +// as a "model decoder". The InputModel forwards those to the model-scope accessors below; a +// concrete decoder answers them when queried before visit_subgraph binds it to a node. // // This is a typed, ggml-free interface: operation parameters are exposed through // get_attribute(name) / get_input_view_element_offset / get_output_shape / RopeConfig rather than -// raw ggml `op_params` int32 arrays. A concrete decoder (e.g. the llama.cpp cgraph decoder) only -// has to translate ggml's layout into these typed accessors -- the op translators never touch -// ggml memory. +// raw ggml `op_params` int32 arrays. A concrete decoder (e.g. the llama.cpp cgraph decoder, or the +// native .gguf builder decoder) only has to translate ggml's layout into these typed accessors -- +// the op translators never touch ggml memory. class GGUF_FRONTEND_API GgufDecoder : public DecoderBase { public: // ── Node scope (the bound node; used by the op translators) ────────────────────────────── @@ -104,26 +105,77 @@ class GGUF_FRONTEND_API GgufDecoder : public DecoderBase { // each node. This is the bridge from model scope to node scope. virtual void visit_subgraph(std::function)> node_visitor) const = 0; - // All model-scope input nodes: both primary inputs (Parameters) and auxiliary inputs - // (position IDs, KV-cache lengths, masks, etc.). Parameters are distinguished from auxiliary - // nodes by the caller via dynamic_pointer_cast. + // All model-scope primary input nodes (Parameters): the token/embedding input plus, on the + // stateful path, the KV-cache Parameters. Distinguished from auxiliary nodes by the caller via + // dynamic_pointer_cast. virtual const std::map>& get_model_inputs() const = 0; + virtual std::vector get_model_output_names() const = 0; - // NOTE: there is no get_model_weights(). A GGUF weight is surfaced as a regular node in - // visit_subgraph with the genuine ggml leaf op type "GGML_OP_NONE": the decoder marks it as a - // weight by exposing the raw weight bytes via get_attribute("data"), the ggml - // quant type name via get_attribute("quant_type") (e.g. "Q4_K", "F16") and the - // logical [rows, cols] shape via get_output_shape(). The frontend's translate_weight does the - // dequant / repacking / requantization, so the decoder never builds OV nodes itself. (Model - // inputs are also GGML_OP_NONE leaves, but they are returned via get_model_inputs() and - // resolved to Parameters before the walk, so they carry no "data".) + // ── Optional model scope ─────────────────────────────────────────────────────────────────── + // + // The accessors below are how a decoder OPTIONALLY enriches the graph; each has a + // do-nothing default so a decoder only implements what it actually knows. That is what lets + // two very different decoders satisfy one interface: the native .gguf builder answers all of + // them, while the llama.cpp cgraph decoder (which is handed an already-built ggml graph and no + // GGUF metadata) answers none and is not forced to write empty stubs. + // + // Note what is NOT here: nothing describes the execution mode. There is no is_stateful / + // is_static, because a decoder describes ggml OPERATIONS, not a deployment. Conversion always + // yields a stateless graph; a caller that wants an OpenVINO KV cache registers + // ov::frontend::gguf::pass::MakeStateful as a DecoderTransformationExtension. + + // Auxiliary model-scope inputs (position IDs, KV-cache lengths, attention masks). A decoder that + // folds these into get_model_inputs() leaves this empty. Note that beam_idx is not among them: + // it is a beam-search index into an OpenVINO state, which ggml has no counterpart for, so + // MakeStateful creates it rather than any decoder declaring it. + virtual const std::map>& get_model_extra_inputs() const { + return empty_node_map(); + } + + // GGUF tokenizer metadata (the file's `tokenizer.*` keys), attached to the converted model's + // rt_info so a downstream consumer (OpenVINO GenAI) can build the tokenizer without reopening + // the .gguf. Empty when the decoder carries no tokenizer metadata. + virtual const ov::AnyMap& get_tokenizer_config() const { + static const ov::AnyMap empty; + return empty; + } + + // Recurrent states, as {input name, output name} pairs: a linear-attention architecture + // (qwen35's Gated DeltaNet) carries a conv window and a delta matrix per recurrent layer, + // which the stateless graph exposes as a Parameter read at the start of a step and a Result + // holding its value at the end. + // + // These are NOT KV caches. A cache grows along a token axis and is written by SET_ROWS, so + // MakeStateful can find it by walking those writes and appending with a Concat; a recurrent + // state has no token axis and is overwritten wholesale, so nothing in the graph marks it. + // Hence this explicit pairing rather than a name convention: the decoder is the only thing + // that knows which Result feeds which Parameter back. + virtual const std::vector>& get_recurrent_states() const { + static const std::vector> empty; + return empty; + } // RoPE configuration, exposed through get_attribute("rope_config"): // - at model scope (via InputModel::get_rope_config), used by TranslateSession::preprocess // to pre-build the shared rope sin/cos table (skipped when RopeConfig::n_dims == 0, i.e. // no RoPE, or per_op == true); // - at node scope, the ROPE translator reads the same key for the op's own config. + // + // NOTE: weights are surfaced as GGML_OP_NONE leaves, by every decoder -- there is no separate + // weight accessor. A decoder marks such a leaf either with the raw ggml bytes + // (get_attribute("data") + get_attribute("quant_type") + + // get_output_shape(), the llama.cpp cgraph path) or with already-extracted weight/scales/zp + // tensors (get_attribute("gguf_weight") + "gguf.blob." + "gguf_qtype", the native + // .gguf builder path). translate_weight accepts both payloads and builds the same compressed + // decompression subgraph from either. + +protected: + // Shared empty map backing the optional accessors above, which return by const reference. + static const std::map>& empty_node_map() { + static const std::map> empty; + return empty; + } }; } // namespace ov::frontend::gguf diff --git a/src/frontends/gguf/include/openvino/frontend/gguf/frontend.hpp b/src/frontends/gguf/include/openvino/frontend/gguf/frontend.hpp index 6a302edb7ea1f6..1c750bd5b8875a 100644 --- a/src/frontends/gguf/include/openvino/frontend/gguf/frontend.hpp +++ b/src/frontends/gguf/include/openvino/frontend/gguf/frontend.hpp @@ -34,6 +34,10 @@ class GGUF_FRONTEND_API FrontEnd : public ov::frontend::FrontEnd { /// - `ov::frontend::ConversionExtension` — registers a custom op translator for the /// ggml op name given by `get_op_type()`. The converter receives an /// `ov::frontend::gguf::NodeContext` and returns an `ov::OutputVector`. + /// - `ov::frontend::DecoderTransformationExtension` — registers a normalization pass, run + /// AHEAD of the frontend's built-in lowerings. This is how the execution mode is chosen: the + /// frontend always converts to a stateless graph, and a caller that wants an OpenVINO KV + /// cache registers `ov::frontend::gguf::pass::MakeStateful` (or its own variant) here. /// - `ov::frontend::TelemetryExtension` — receives error / event callbacks. /// - `ov::detail::SOExtension` — shared-library extension; its inner extension is /// recursively registered. @@ -44,17 +48,19 @@ class GGUF_FRONTEND_API FrontEnd : public ov::frontend::FrontEnd { void add_extension(const std::shared_ptr& extension) override; protected: - /// \brief Check if FrontEnd can recognize model from given parts. - /// \note Always returns false: this frontend is hidden from FrontEndManager and is never - /// auto-selected. It is used only via direct linkage, by constructing FrontEnd and - /// calling convert() on an InputModel built from a GgufDecoder. - /// \param variants Unused. - /// \return Always false. + /// \brief Check if FrontEnd can recognize the model from the given parts. + /// \param variants Either a `std::shared_ptr`, or a path to a file whose extension + /// is `.gguf` and whose first four bytes are the GGUF magic. + /// \return True for either of those; false otherwise. bool supported_impl(const std::vector& variants) const override; - /// \brief Load the input model from a GgufDecoder. - /// \param variants A single GgufDecoder (a .gguf file path is not accepted; the caller supplies - /// the decoder). variants[0] must hold a std::shared_ptr. + /// \brief Load the input model, from either of the frontend's two ingest paths. + /// \param variants A single element, holding either: + /// - a `std::shared_ptr` — a decoder supplied by a direct linker, wrapping + /// an already-built ggml graph (the llama.cpp cgraph path); or + /// - a path to a `.gguf` file — parsed here, with the transformer graph built + /// per-architecture by the native builder. + /// Both yield a GgufDecoder, so conversion past this point is identical. /// \return InputModel::Ptr InputModel::Ptr load_impl(const std::vector& variants) const override; diff --git a/src/frontends/gguf/include/openvino/frontend/gguf/make_stateful.hpp b/src/frontends/gguf/include/openvino/frontend/gguf/make_stateful.hpp new file mode 100644 index 00000000000000..cbde65a6ed8e19 --- /dev/null +++ b/src/frontends/gguf/include/openvino/frontend/gguf/make_stateful.hpp @@ -0,0 +1,93 @@ +// Copyright (C) 2018-2026 Intel Corporation +// SPDX-License-Identifier: Apache-2.0 +// + +#pragma once + +#include +#include +#include + +#include "openvino/frontend/gguf/visibility.hpp" +#include "openvino/pass/pass.hpp" + +namespace ov::frontend::gguf::pass { + +/// \brief Turn the frontend's stateless GGUF model into an OpenVINO stateful one. +/// +/// The frontend always converts to a STATELESS graph -- every KV cache is a Parameter written by a +/// SetRows placeholder and read back as a Result -- mirroring how optimum-intel exports. Being +/// stateful is a caller concern, so register this as a DecoderTransformationExtension: +/// +/// ov::frontend::gguf::FrontEnd fe; +/// fe.add_extension(std::make_shared( +/// ov::frontend::gguf::pass::MakeStateful())); +/// auto model = fe.convert(fe.load(decoder_or_gguf_path)); +/// +/// Extensions run ahead of the built-in LowerSetRowsStateless, so this pass claims the KV-cache +/// SetRows ops and the default stateless lowering only sees the rest (e.g. MoE routing writes). +/// +/// Per KV cache it replaces the Parameter/Result pair with a Variable + ReadValue(empty init) + +/// Gather(beam_idx) + Concat(past, this step's rows) + Assign. Only a SetRows writing to a model +/// Parameter is converted. +/// +/// Two details that are load-bearing rather than cosmetic: +/// - `beam_idx` is ADDED here, not taken from the decoder: ggml has no counterpart, so declaring +/// it in a decoder would leave a consumer-less input on the stateless graph. +/// - the ReadValue init must be empty: CPU's stateful_sdpa_fusion folds the cache into +/// ScaledDotProductAttentionWithKVCache, whose MemoryInputSDPA aborts on zero parent edges. +/// +/// Scope: this grows the cache and deliberately does not touch the attention mask. The native +/// builder emits a dynamically sized mask, which needs no change; a graph that preallocates a +/// fixed mask window must be re-sliced by the caller, as the llama.cpp backend does. +/// Key under which the frontend records the model's recurrent (overwritten, non-appending) states +/// in rt_info, as a flat list of alternating {input name, output name}. Linear-attention +/// architectures (qwen35's Gated DeltaNet) carry a conv window and a delta matrix per recurrent +/// layer. Unlike a KV cache these have no token axis and no SetRows write marking them in the +/// graph, so the pairing has to be carried explicitly; see GgufDecoder::get_recurrent_states. +GGUF_FRONTEND_API const std::string& gguf_recurrent_states_key(); + +/// LIMITATION -- recurrent states are batch-1 and are NOT reordered by beam_idx. Unlike a KV +/// cache, which this pass gathers by beam_idx before appending, a recurrent state is a single +/// static-shaped block with no batch axis to reorder. Beam search or batch > 1 therefore fails at +/// inference with a shape mismatch on the state's Concat rather than silently mixing state across +/// beams; greedy, batch-1 generation is the supported mode for a linear-attention architecture. +/// +/// Key under which the frontend records that the model uses interleaved M-RoPE (qwen35 / +/// qwen3vl). Such a model expects inp_pos to carry FOUR position sections per token, so a consumer +/// feeding it plain per-token positions (as OpenVINO GenAI does) has to expand them first. +GGUF_FRONTEND_API const std::string& gguf_imrope_key(); + +class GGUF_FRONTEND_API MakeStateful : public ov::pass::ModelPass { +public: + OPENVINO_MODEL_PASS_RTTI("gguf::MakeStateful"); + + /// \param skip_caches Friendly names of cache Parameters to leave stateless. A sliding-window + /// cache is evicted from the front rather than only appended to, so an append-grown + /// Variable would not reproduce it; such caches keep the stateless form. + /// \param append_axis Cache axis the new rows are appended along (the token axis). -1 infers it + /// as the cache Parameter's single dynamic axis, which is how a graph that does not + /// preallocate the cache states its token axis. Pass an explicit axis for a fully static + /// (preallocated) cache, where there is nothing to infer from. + /// \param beam_idx_name Name of the beam-reorder input, which this pass ADDS to the model (it + /// belongs to the state, so no decoder declares it; see the note above). The past cache is + /// gathered by it along the batch axis before the append Concat. With batch 1 / + /// beam_idx [0] that Gather is an identity, but emitting it is what lets CPU's + /// stateful_sdpa_fusion match, and it is what makes beam search work. A model that + /// already carries a Parameter of this name has it reused instead. + explicit MakeStateful(std::set skip_caches = {}, + int64_t append_axis = -1, + std::string beam_idx_name = "beam_idx") + : m_skip_caches(std::move(skip_caches)), + m_append_axis(append_axis), + m_beam_idx_name(std::move(beam_idx_name)) {} + + bool run_on_model(const std::shared_ptr& model) override; + +private: + std::set m_skip_caches; + int64_t m_append_axis; + std::string m_beam_idx_name; +}; + +} // namespace ov::frontend::gguf::pass diff --git a/src/frontends/gguf/include/openvino/frontend/gguf/tokenizer_metadata.hpp b/src/frontends/gguf/include/openvino/frontend/gguf/tokenizer_metadata.hpp new file mode 100644 index 00000000000000..7c1b9c7af8ddb4 --- /dev/null +++ b/src/frontends/gguf/include/openvino/frontend/gguf/tokenizer_metadata.hpp @@ -0,0 +1,58 @@ +// Copyright (C) 2018-2026 Intel Corporation +// SPDX-License-Identifier: Apache-2.0 +// + +#pragma once + +#include "openvino/core/any.hpp" +#include "openvino/core/runtime_attribute.hpp" +#include "openvino/frontend/gguf/visibility.hpp" + +namespace ov { +namespace frontend { +namespace gguf { + +/// \brief Carries the GGUF tokenizer metadata (the `tokenizer.*` ggml keys) on a converted +/// model's runtime info, so a downstream consumer (e.g. OpenVINO GenAI) can build the +/// OpenVINO tokenizer/detokenizer without re-opening the .gguf file. +/// +/// The payload is an ov::AnyMap keyed by the tokenizer metadata sub-key (the part after the +/// last dot of `tokenizer.ggml.*` / `tokenizer.chat_template`, e.g. "model", "tokens", +/// "merges", "scores", "token_type", "pre", "bos_token_id", "eos_token_id", "chat_template"). +/// Each value holds exactly one of: +/// - std::string (e.g. "model" = "gpt2"/"llama"/"gemma4", "pre", "chat_template") +/// - std::vector (e.g. "tokens", "merges") +/// - ov::Tensor (arrays like "scores"/"token_type", and scalars such as +/// "*_token_id" stored as a shape-{} tensor) +/// which mirrors the GGUF metadata variant both the frontend and GenAI already use. +/// +/// This attribute is intentionally **non-serializable**: it is heavy (full vocab + merges) and +/// only meaningful in-memory between conversion and tokenizer construction. `is_copyable()` +/// returns false so it is dropped on clone, and `to_string()` is empty so that if the model is +/// serialized the IR writer emits an empty placeholder the deserializer ignores rather than +/// dumping the vocab into the XML. +class GGUF_FRONTEND_API GGUFTokenizerMetadata : public ov::RuntimeAttribute { +public: + OPENVINO_RTTI("gguf_tokenizer_metadata", "0", ov::RuntimeAttribute); + + GGUFTokenizerMetadata() = default; + explicit GGUFTokenizerMetadata(ov::AnyMap config) : config(std::move(config)) {} + + bool is_copyable() const override { + return false; + } + + std::string to_string() const override { + return {}; + } + + /// tokenizer sub-key -> value (string / vector / Tensor) + ov::AnyMap config; +}; + +/// Runtime-info key under which GGUFTokenizerMetadata is stored on the model. +GGUF_FRONTEND_API const std::string& gguf_tokenizer_metadata_key(); + +} // namespace gguf +} // namespace frontend +} // namespace ov diff --git a/src/frontends/gguf/src/CMakeLists.txt b/src/frontends/gguf/src/CMakeLists.txt index 79b50f3e7e9a17..fd8c994233e99f 100644 --- a/src/frontends/gguf/src/CMakeLists.txt +++ b/src/frontends/gguf/src/CMakeLists.txt @@ -2,10 +2,13 @@ # SPDX-License-Identifier: Apache-2.0 # -# LINKABLE_FRONTEND: installed alongside the other frontends so callers (the llama.cpp -# ggml-openvino backend, OpenVINO GenAI) can link openvino::frontend::gguf directly. It is kept -# out of the generic FrontEndManager loading API (FrontEndManager treats "gguf" as hidden); see -# the discoverability note in frontend.cpp. +# The GGUF frontend converts GGUF models to OpenVINO. It installs the library + headers alongside +# the other frontends (LINKABLE_FRONTEND, no SKIP_INSTALL) so direct linkers -- the llama.cpp +# ggml-openvino backend and OpenVINO GenAI -- can link openvino::frontend::gguf, and it is +# discoverable via FrontEndManager so core.read_model("model.gguf") selects it by the .gguf +# extension + GGUF magic (see supported_impl / get_front_end_data in frontend.cpp). Two ingest +# paths converge on the same op translators: a live GgufDecoder (cgraph path) and a native +# .gguf-file builder. ov_add_frontend(NAME gguf LINKABLE_FRONTEND FILEDESCRIPTION "FrontEnd to convert GGUF models" diff --git a/src/frontends/gguf/src/builder/arch/decoder_builder.cpp b/src/frontends/gguf/src/builder/arch/decoder_builder.cpp new file mode 100644 index 00000000000000..84578a847fd08a --- /dev/null +++ b/src/frontends/gguf/src/builder/arch/decoder_builder.cpp @@ -0,0 +1,404 @@ +// Copyright (C) 2018-2026 Intel Corporation +// SPDX-License-Identifier: Apache-2.0 +// +// Topology of the causal decoder family, in the GGML op vocabulary. +// +// Ground truth for the shape of the graph: llama.cpp src/models/{llama,qwen2,qwen3,phi3,minicpm, +// hunyuan-*,gemma*,gpt-oss,olmoe,qwen35}.cpp and the build_norm / build_qkv / build_attn / +// build_attn_mha / build_ffn expansions in src/llama-graph.cpp, plus the KV-cache cpy_k/get_k +// (SET_ROWS + VIEW) in src/llama-kv-cache.cpp. Op-case values follow +// ggml-decoder.cpp::compute_op_case. +// +// Everything architecture-specific has already been resolved into DecoderConfig, and every +// repeated fragment lives in blocks/, so this file is only the ORDER in which a decoder is +// assembled: inputs -> embeddings -> N x (norm, attention|GDN, norm, FFN|MoE) -> norm -> lm_head. + +#include "builder/arch/decoder_builder.hpp" + +#include + +#include "builder/blocks/common.hpp" +#include "builder/blocks/ffn.hpp" +#include "builder/blocks/gated_delta_net.hpp" +#include "openvino/op/parameter.hpp" + +namespace ov { +namespace frontend { +namespace gguf { + +using ov::element::f32; +using ov::element::i32; +using ov::element::i64; + +namespace { +// Dynamic token length, used for model-input Parameters only. +constexpr int64_t D = -1; +} // namespace + +DecoderBuilder::DecoderBuilder(const std::map& config, + std::unordered_map& weights, + std::unordered_map& qtypes) + : m_cfg(config, weights), + m_emit(weights, qtypes, m_cfg.arch), + m_kv(blocks::KvCachePlan::build(m_cfg)) { + auto& graph = *m_emit.graph(); + graph.has_rope = true; + graph.rope_config = m_cfg.rope_config; + graph.use_per_op_rope = m_cfg.use_per_op_rope; +} + +void DecoderBuilder::build_inputs() { + // Names match the cgraph decoder's get_graph_input_ov_name; gguf uses i32 for token/position/ + // index inputs. + m_emit.add_input("inp_tokens", i32, ps({1, 1, 1, D})); + m_emit.add_input("inp_pos", i32, ps({1, 1, 1, D})); + m_emit.add_input("inp_out_ids", i32, ps({1, 1, 1, D})); + m_emit.add_input("self_kq_mask", f32, ps({1, 1, D, D})); + // gpt-oss alternates sliding-window / full attention; the windowed mask is a separate + // input. Only added when the model uses SWA. + if (m_cfg.has_swa) { + m_emit.add_input("self_kq_mask_swa", f32, ps({1, 1, D, D})); + } + // No beam_idx here. It is a beam-search cache-reorder index for an OpenVINO STATEFUL cache, + // which ggml has no equivalent of -- so the cgraph decoder does not produce one either, and a + // builder that declared it would give the two decoders different stateless IO. The pass that + // creates the state creates it (MakeStateful), because that is where its only consumer, the + // Gather on the past, is emitted. + + // KV-cache update index (consumed by SET_ROWS; unused in the stateful Concat branch). + m_emit.add_input("inp_kv_idx", i32, ps({1, 1, 1, D})); + m_emit.set_tensor_meta("inp_kv_idx", ps({1, 1, 1, T}), i32); + + // token_len_per_seq: number of new tokens per sequence; used by TranslateSession's mask + // slicing (add_sliced_mask) to build KQ_mask_sliced. An extra (Parameter) input. + auto p = std::make_shared(i64, ps({1})); + p->set_friendly_name("token_len_per_seq"); + p->output(0).set_names({"token_len_per_seq"}); + m_emit.add_extra_input_node("token_len_per_seq", p); +} + +std::string DecoderBuilder::build_embeddings() { + // GET_ROWS(token_embd.weight, inp_tokens) -> "embd" + m_emit.add_weight("token_embd.weight"); + m_emit.set_tensor_meta("inp_tokens", ps({1, 1, 1, T}), i32); + std::string cur = m_emit.add_op("GGML_OP_GET_ROWS", + "embd", + {"token_embd.weight", "inp_tokens"}, + ps({1, 1, T, m_cfg.n_embd}), + f32); + // MiniCPM scales the embeddings by a constant. + if (m_cfg.embedding_scale != 1.0f) { + cur = blocks::scale(m_emit, cur, m_cfg.embedding_scale, "embd_scaled"); + } + // muse-glimmer normalizes the token embeddings with a WEIGHTLESS RMSNorm before layer 0 + // (build_norm with a null weight -> plain ggml_rms_norm, no multiplicative term). + if (m_cfg.scaleless_embd_norm) { + cur = m_emit.add_op("GGML_OP_RMS_NORM", + "embd_normed", + {cur}, + m_emit.shape_of_tensor(cur), + f32, + 0, + {{"eps", m_cfg.rms_eps}}); + } + return cur; +} + +void DecoderBuilder::build_per_layer_embeddings(const std::string& embd) { + // Gemma4: per-layer token embeddings are built before the layer loop and stored as a + // 4D tensor "per_layer_embd" of shape [n_layer, T, n_embd_per_layer] (ggml logical order). + // Inside each layer the VIEW op extracts the il-th slice [1, T, n_embd_per_layer]. + // The projection norm is applied per-slice (over the n_embd_per_layer dim). + // + // Topology (mirrors llama.cpp build_inp_per_layer + project_per_layer_inputs): + // pe_tok = GET_ROWS(per_layer_token_embd, inp_tokens) -> [1, n_layer, T, n_embd_per_layer] + // scaled by sqrt(n_embd_per_layer) + // pe_proj = MUL_MAT(per_layer_model_proj, cur_embd) -> [1, n_layer, T, n_embd_per_layer] + // scaled by 1/sqrt(n_embd) + RMS_NORM(per_layer_proj_norm) + // per_layer_embd = (pe_proj + pe_tok) * 1/sqrt(2) -> [1, n_layer, T, n_embd_per_layer] + const int n_layer = m_cfg.n_layer; + const int pe = m_cfg.n_embd_per_layer; + + // Turn a flat per-token per-layer embedding [1, 1, T, n_layer * pe] into the layer-major + // [1, n_layer, T, pe] the per-layer VIEW slices from. + // + // The data is contiguous as [T, n_layer, pe] (one row per token), so this is a reshape that + // splits the last axis followed by a transpose of the two middle axes -- a single reshape + // straight to the target would be wrong for T > 1. Emitted as the same two steps ggml uses + // (ggml_reshape_3d, then ggml_cont(ggml_permute)), so both reach the shared op cases: RESHAPE + // case 1 splits the last dim into [.., n_layer, pe] and PERMUTE case 1 is the {0,2,1,3} + // transpose. Fusing them into one node would need a builder-only case for the transpose. No + // CONT node: translate_cont's PERMUTE case is a pass-through, since translate_permute already + // emitted a real Transpose, so ggml's ggml_cont would convert to nothing here. + auto reshape_to_layer_major = [&](const std::string& flat, const std::string& name) { + const auto& flat_shape = m_emit.shape_of_tensor(flat); + const int64_t t = flat_shape[2].is_static() ? flat_shape[2].get_length() : 1; + auto split = m_emit.add_op("GGML_OP_RESHAPE", name + "_split", {flat}, ps({1, t, n_layer, pe}), f32, 1); + return m_emit.add_op("GGML_OP_PERMUTE", name, {split}, ps({1, n_layer, t, pe}), f32, 1); + }; + + m_emit.add_weight("per_layer_token_embd.weight"); + m_emit.add_weight("per_layer_model_proj.weight"); + m_emit.add_weight("per_layer_proj_norm.weight"); + const int pe_total = pe * n_layer; + + // Token embedding lookup: [1,1,T, pe_total] -> reshape to [1, n_layer, T, pe] + auto pe_flat = m_emit.add_op("GGML_OP_GET_ROWS", + "pe_tok_flat", + {"per_layer_token_embd.weight", "inp_tokens"}, + ps({1, 1, T, pe_total}), + f32); + const float pe_scale = std::sqrt(static_cast(pe)); + pe_flat = blocks::scale(m_emit, pe_flat, pe_scale, "pe_tok_flat_scaled"); + auto pe_tok = reshape_to_layer_major(pe_flat, "pe_tok"); + + // Model projection: MUL_MAT(per_layer_model_proj, embd) -> [1,1,T, pe_total] + // per_layer_model_proj is [n_embd, pe_total] -> output [pe_total] per token + auto proj_flat = m_emit.add_op("GGML_OP_MUL_MAT", + "pe_proj_flat", + {"per_layer_model_proj.weight", embd}, + ps({1, 1, T, pe_total}), + f32); + const float proj_scale = 1.0f / std::sqrt(static_cast(m_cfg.n_embd)); + proj_flat = blocks::scale(m_emit, proj_flat, proj_scale, "pe_proj_flat_scaled"); + + // Reshape to [1, n_layer, T, pe] for per-slice RMS_NORM + auto proj_4d = reshape_to_layer_major(proj_flat, "pe_proj_4d"); + + // RMS_NORM + per_layer_proj_norm weight (applied over last dim = pe) + auto proj_norm = m_emit.add_op("GGML_OP_RMS_NORM", + "pe_proj_rms", + {proj_4d}, + ps({1, n_layer, T, pe}), + f32, + 0, + {{"eps", m_cfg.rms_eps}}); + m_emit.set_tensor_meta("per_layer_proj_norm.weight", ps({1, 1, 1, pe}), f32); + auto proj_normed = m_emit.add_op("GGML_OP_MUL", + "pe_proj_normed", + {proj_norm, "per_layer_proj_norm.weight"}, + ps({1, n_layer, T, pe}), + f32); + + // Sum token embd + projection, scale by 1/sqrt(2) + auto pe_sum = m_emit.add_op("GGML_OP_ADD", "pe_sum", {proj_normed, pe_tok}, ps({1, n_layer, T, pe}), f32); + const float inv_sqrt2 = 1.0f / std::sqrt(2.0f); + m_emit.add_op("GGML_OP_SCALE", + "per_layer_embd", + {pe_sum}, + ps({1, n_layer, T, pe}), + f32, + 0, + {{"scale", inv_sqrt2}, {"bias", 0.0f}}); +} + +std::string DecoderBuilder::inject_per_layer_embedding(int il, const std::string& cur, const std::string& inpSA) { + // Gemma4: each layer takes a slice of the pre-projected per-layer embedding (shape + // [1,1,T,n_embd_per_layer]), gates it through inp_gate.weight (GELU), multiplies by the + // per-layer slice, projects back to n_embd, post-norms, then adds residual. + // per_layer_token_embd (global) is pre-projected before the loop. + const std::string p = "blk." + std::to_string(il) + "."; + const int pe = m_cfg.n_embd_per_layer; + + // Slice out this layer's [1,1,T,pe] chunk (dim 1 at index il). The second input is passed + // purely as a shape reference: per_layer_embd is stored layer-major, so the slice's token axis + // does not necessarily sit where the layer activation keeps its own, and the ops below combine + // the two elementwise. See the op_case 104 note in op/view.cpp. + // + // It has to be a tensor that still carries every token. At the last layer `cur` has already + // been filtered down to the output rows by the GET_ROWS in the attention tail, while + // per_layer_embd always holds all T of them, so using `cur` there would ask op_case 104 to + // reinterpret T*D values into 1*D. Use the layer's unfiltered input instead and let the + // GET_ROWS below do the filtering. + const std::string& pl_shape_ref = (il == m_cfg.n_layer - 1) ? inpSA : cur; + const std::string pl_slice = p + "per_layer_slice"; + m_emit.add_op("GGML_OP_VIEW", + pl_slice, + {"per_layer_embd", pl_shape_ref}, + ps({1, 1, T, pe}), + f32, + 104, // layer-index slice using the "layer_idx" attribute + {{"layer_idx", int64_t(il)}}); + + // At the last layer, cur is already filtered to 1 token via inp_out_ids. Mirror llama.cpp + // gemma4.cpp:347-349: also filter pl_slice so the MUL doesn't broadcast it back to the full + // sequence length. + std::string pl_slice_used = pl_slice; + if (il == m_cfg.n_layer - 1) { + pl_slice_used = p + "per_layer_slice_sel"; + m_emit.add_op("GGML_OP_GET_ROWS", pl_slice_used, {pl_slice, "inp_out_ids"}, ps({1, 1, T, pe}), f32); + } + + // gate: cur -> inp_gate.weight -> GELU -> [1,1,T,pe] + m_emit.add_weight(p + "inp_gate.weight"); + auto gated = + m_emit.add_op("GGML_OP_MUL_MAT", p + "inp_gate_mm", {p + "inp_gate.weight", cur}, ps({1, 1, T, pe}), f32); + gated = m_emit.add_op("GGML_UNARY_OP_GELU", p + "inp_gate_gelu", {gated}, ps({1, 1, T, pe}), f32); + + // elementwise multiply by per-layer slice + auto mul_pe = m_emit.add_op("GGML_OP_MUL", p + "pe_mul", {gated, pl_slice_used}, ps({1, 1, T, pe}), f32); + + // project back to n_embd + m_emit.add_weight(p + "proj.weight"); + auto pe_proj = + m_emit.add_op("GGML_OP_MUL_MAT", p + "pe_proj", {p + "proj.weight", mul_pe}, ps({1, 1, T, m_cfg.n_embd}), f32); + + // post-norm + residual add + pe_proj = blocks::rms_norm(m_emit, pe_proj, p + "post_norm.weight", p + "pe_post_norm", m_cfg.rms_eps); + return m_emit.add_op("GGML_OP_ADD", p + "pe_out", {cur, pe_proj}, ps({1, 1, T, m_cfg.n_embd}), f32); +} + +std::string DecoderBuilder::build_layer(int il, const std::string& layer_in) { + const std::string p = "blk." + std::to_string(il) + "."; + const std::string inpSA = layer_in; + std::string cur = layer_in; + + // eps for the post-attention / post-FFN norms; 0 in the config means "reuse rms_eps" + // (muse-glimmer's post-norms use a tighter 1e-8 than its pre-norms). + const float post_eps = m_cfg.post_norm_eps > 0.0f ? m_cfg.post_norm_eps : m_cfg.rms_eps; + + // attn_norm (key varies by arch: "attn_norm.weight" for most, "post_attention_norm.weight" + // for exaone4) + const std::string attn_norm = + blocks::rms_norm(m_emit, cur, p + m_cfg.attn_norm_key, p + "attn_norm", m_cfg.rms_eps); + + if (m_cfg.is_qwen35 && m_cfg.is_recurrent_layer(il)) { + // Hybrid stack: 3 of every 4 layers replace attention with a Gated DeltaNet block. + // It produces the same thing the attention path does -- the sublayer output before + // the residual -- so the shared FFN tail below is reused verbatim. + const std::string gdn_out = blocks::gated_delta_net(m_emit, m_cfg, il, attn_norm, T); + std::string sa = inpSA; + std::string ao = gdn_out; + if (il == m_cfg.n_layer - 1) { + ao = m_emit.add_op("GGML_OP_GET_ROWS", + p + "attn_out_g", + {gdn_out, "inp_out_ids"}, + ps({1, 1, T, m_cfg.n_embd}), + f32); + sa = m_emit.add_op("GGML_OP_GET_ROWS", + p + "inpSA_g", + {inpSA, "inp_out_ids"}, + ps({1, 1, T, m_cfg.n_embd}), + f32); + } + auto ffn_inp_r = m_emit.add_op("GGML_OP_ADD", p + "ffn_inp", {ao, sa}, ps({1, 1, T, m_cfg.n_embd}), f32); + auto ffn_norm_r = blocks::rms_norm(m_emit, ffn_inp_r, p + m_cfg.ffn_norm_key, p + "ffn_norm", m_cfg.rms_eps); + auto down_r = blocks::dense_ffn(m_emit, m_cfg, p, ffn_norm_r, T); + return m_emit.add_op("GGML_OP_ADD", p + "l_out", {down_r, ffn_inp_r}, ps({1, 1, T, m_cfg.n_embd}), f32); + } + + std::string attn_out = blocks::attention(m_emit, m_cfg, m_kv, il, attn_norm, T); + + // MiniCPM scales the attention sublayer output before the residual add. + if (m_cfg.residual_scale != 1.0f) { + attn_out = blocks::scale(m_emit, attn_out, m_cfg.residual_scale, p + "attn_out_scaled"); + } + std::string sa = inpSA; + std::string ao = attn_out; + if (il == m_cfg.n_layer - 1) { + ao = m_emit.add_op("GGML_OP_GET_ROWS", + p + "attn_out_g", + {attn_out, "inp_out_ids"}, + ps({1, 1, T, m_cfg.n_embd}), + f32); + sa = m_emit.add_op("GGML_OP_GET_ROWS", p + "inpSA_g", {inpSA, "inp_out_ids"}, ps({1, 1, T, m_cfg.n_embd}), f32); + } + // Gemma2: post-attention RMSNorm applied to the sublayer output before residual add. + // Applied after GET_ROWS so the selected-token path matches gemma2.cpp's order. + if (m_cfg.has_attn_post_norm) { + ao = blocks::rms_norm(m_emit, ao, p + "post_attention_norm.weight", p + "attn_post_norm", post_eps); + } + + auto ffn_inp = m_emit.add_op("GGML_OP_ADD", p + "ffn_inp", {ao, sa}, ps({1, 1, T, m_cfg.n_embd}), f32); + + // Pre-FFN/MoE norm. Key varies by arch (ffn_norm_key is resolved in DecoderConfig). + auto ffn_norm = blocks::rms_norm(m_emit, ffn_inp, p + m_cfg.ffn_norm_key, p + "ffn_norm", m_cfg.rms_eps); + + // Hybrid MoE: lead layers (il < n_dense_lead) are always dense regardless of is_moe. + const bool is_moe_layer = m_cfg.is_moe && (il >= m_cfg.n_dense_lead); + std::string down = is_moe_layer ? blocks::moe_ffn(m_emit, m_cfg, p, ffn_norm, T) + : m_cfg.is_geglu ? blocks::geglu_ffn(m_emit, m_cfg, p, ffn_norm, T) + : blocks::dense_ffn(m_emit, m_cfg, p, ffn_norm, T); + // MiniCPM scales the FFN sublayer output before the residual add. + if (m_cfg.residual_scale != 1.0f) { + down = blocks::scale(m_emit, down, m_cfg.residual_scale, p + "ffn_out_scaled"); + } + // Gemma2: post-FFN RMSNorm applied to the FFN output before residual add. + if (m_cfg.has_ffn_post_norm) { + down = blocks::rms_norm(m_emit, down, p + "post_ffw_norm.weight", p + "ffn_post_norm", post_eps); + } + + cur = m_emit.add_op("GGML_OP_ADD", p + "l_out", {down, ffn_inp}, ps({1, 1, T, m_cfg.n_embd}), f32); + + if (m_cfg.n_embd_per_layer > 0) { + cur = inject_per_layer_embedding(il, cur, inpSA); + } + + // Gemma4: per-layer output scale (layer_output_scale.weight is a scalar [1]). + if (m_emit.has_weight(p + "layer_output_scale.weight")) { + m_emit.add_named_weight(p + "layer_output_scale.weight"); + cur = m_emit.add_op("GGML_OP_MUL", + p + "scaled_out", + {cur, p + "layer_output_scale.weight"}, + m_emit.shape_of_tensor(cur), + f32); + } + return cur; +} + +std::string DecoderBuilder::build_head(const std::string& in) { + // final norm + lm_head + std::string cur = blocks::rms_norm(m_emit, in, "output_norm.weight", "result_norm", m_cfg.rms_eps); + + const std::string lm_head_w = m_emit.has_weight("output.weight") ? "output.weight" : "token_embd.weight"; + m_emit.add_weight(lm_head_w); + const int n_vocab = static_cast(m_emit.weight_tensor(lm_head_w).get_shape()[0]); // rows = vocab + auto logits = m_emit.add_op("GGML_OP_MUL_MAT", "result_output", {lm_head_w, cur}, ps({1, 1, T, n_vocab}), f32); + // MiniCPM scales the logits (1/(n_embd/dim_model_base)). + if (m_cfg.logit_scale != 1.0f) { + logits = blocks::scale(m_emit, logits, m_cfg.logit_scale, "result_output_scaled"); + } + // Gemma2/Gemma3 final logit soft-cap: tanh(logits / cap) * cap. + if (m_cfg.final_logit_soft_cap != 0.0f) { + logits = blocks::scale(m_emit, logits, 1.0f / m_cfg.final_logit_soft_cap, "logits_softcap_scaled"); + logits = + m_emit.add_op("GGML_UNARY_OP_TANH", "logits_softcap_tanh", {logits}, m_emit.shape_of_tensor(logits), f32); + logits = blocks::scale(m_emit, logits, m_cfg.final_logit_soft_cap, "result_output_softcapped"); + } + return logits; +} + +std::shared_ptr DecoderBuilder::build() { + build_inputs(); + + std::string cur = build_embeddings(); + + // rope freq factors (llama-3 long context, phi-3): an optional 3rd ROPE input. + if (m_cfg.has_rope_freqs) { + m_emit.add_weight("rope_freqs.weight"); + } + + // kq_scale, head size, KV heads and rope config are all computed per-layer inside the loop + // via the DecoderConfig accessors, so SWA and global layers each get the correct + // head-size-dependent values (e.g. gemma4 SWA layers use head_size_swa). + + if (m_cfg.n_embd_per_layer > 0) { + build_per_layer_embeddings(cur); + } + + for (int il = 0; il < m_cfg.n_layer; ++il) { + cur = build_layer(il, cur); + } + + const std::string logits = build_head(cur); + + // logits is the primary output; the cache_k/v_l* outputs (appended per layer above) + // become Assign sinks via MakeStateful. + auto graph = m_emit.graph(); + graph->model_output_names.insert(graph->model_output_names.begin(), logits); + return graph; +} + +} // namespace gguf +} // namespace frontend +} // namespace ov diff --git a/src/frontends/gguf/src/builder/arch/decoder_builder.hpp b/src/frontends/gguf/src/builder/arch/decoder_builder.hpp new file mode 100644 index 00000000000000..018c8a96b95f0d --- /dev/null +++ b/src/frontends/gguf/src/builder/arch/decoder_builder.hpp @@ -0,0 +1,75 @@ +// Copyright (C) 2018-2026 Intel Corporation +// SPDX-License-Identifier: Apache-2.0 +// + +#pragma once + +#include +#include +#include +#include + +#include "builder/blocks/attention.hpp" +#include "builder/decoder_config.hpp" +#include "builder/graph_emitter.hpp" +#include "builder/model_builder.hpp" +#include "quant/gguf.hpp" + +namespace ov { +namespace frontend { +namespace gguf { + +// Whole-model builder for the causal decoder family: the "llama family" of dense and MoE +// decoder-only transformers (llama-3, qwen2/2.5/3, phi-3, minicpm, gemma 1-4, gpt-oss, OLMoE, +// qwen3moe, the qwen35 hybrid Gated-DeltaNet stack, ...). +// +// ONE builder covers all of them. Per-architecture differences are resolved into DecoderConfig by +// structural detection on the GGUF tensor table, so enabling a same-family architecture is a name +// in arch_registry.cpp and no code here. This is a deliberate departure from llama.cpp's +// one-file-per-architecture layout, which exists there because each architecture must enumerate +// its tensors by hand; the GGUF frontend derives them instead. +// +// Emits the STATELESS graph: KV caches are plain model Parameters written by GGML_OP_SET_ROWS and +// read back as Results, and every activation keeps ggml's rank-4 shape. A caller that wants an +// OpenVINO KV cache registers ov::frontend::gguf::pass::MakeStateful as a transformation extension. +class DecoderBuilder : public ModelBuilder { +public: + DecoderBuilder(const std::map& config, + std::unordered_map& weights, + std::unordered_map& qtypes); + + std::shared_ptr build() override; + +private: + // Model inputs shared by every layer (tokens, positions, masks, KV update index). + void build_inputs(); + + // Token embedding lookup plus the optional embedding scale / weightless norm. + std::string build_embeddings(); + + // gemma4: the per-layer input embeddings, pre-projected once before the layer loop. + void build_per_layer_embeddings(const std::string& embd); + + // One decoder layer; `cur` is the layer input, returns the layer output. + std::string build_layer(int il, const std::string& cur); + + // gemma4: per-layer embedding injection appended after the FFN residual. + std::string inject_per_layer_embedding(int il, const std::string& cur, const std::string& inpSA); + + // Final norm, lm_head, and the optional logit scale / soft-cap. + std::string build_head(const std::string& cur); + + DecoderConfig m_cfg; + GraphEmitter m_emit; + blocks::KvCachePlan m_kv; + + // Per-node output shapes are STATIC, like the cgraph decoder (which builds the graph for a + // concrete token length). We use a representative token length T; the translators emit dynamic + // reshapes (-1 / 0) where needed, and MakeStateful + the dynamic input Parameters carry the + // real dynamic-ness. T affects only the per-node shape metadata. + static constexpr int64_t T = 1; +}; + +} // namespace gguf +} // namespace frontend +} // namespace ov diff --git a/src/frontends/gguf/src/builder/arch_registry.cpp b/src/frontends/gguf/src/builder/arch_registry.cpp new file mode 100644 index 00000000000000..bec01e5df0368f --- /dev/null +++ b/src/frontends/gguf/src/builder/arch_registry.cpp @@ -0,0 +1,94 @@ +// Copyright (C) 2018-2026 Intel Corporation +// SPDX-License-Identifier: Apache-2.0 +// +// The native builder's architecture accept list, and the one per-architecture property that +// cannot be derived from the GGUF tensor table (the RoPE mode). +// +// Adding a same-family architecture is a one-line change here and nothing else: the decoder +// builder derives everything else -- QK-norm, biases, fused QKV, MoE routing, SWA, soft-caps -- +// from the tensor table and metadata. See docs/adding_an_architecture.md. + +#include "arch_registry.hpp" + +namespace ov { +namespace frontend { +namespace gguf { + +bool arch_uses_neox_rope(const std::string& arch) { + return arch == "qwen2" || arch == "qwen3" || arch == "phi3" || arch == "hunyuan-dense" || arch == "gpt-oss" || + arch == "gemma" || arch == "gemma2" || arch == "gemma3" || arch == "gemma4" || arch == "olmoe" || + // H2 2025 dense additions + arch == "exaone4" || arch == "plamo3" || arch == "mellum" || + // H2 2025 MoE additions + arch == "hunyuan-moe" || arch == "glm4moe" || arch == "bailingmoe2" || arch == "exaone-moe" || + arch == "minimax-m2" || + // H2 2025 VL backbone / other + arch == "jais2" || arch == "deepseek2-ocr"; +} + +const std::set& verified_archs() { + static const std::set archs = { + "llama", // llama-2 / llama-3 + "qwen2", // qwen2 / qwen2.5 + "qwen3", + "phi3", // phi-3 (fused QKV) + "minicpm", // NORMAL rope + scalar scales + "hunyuan-dense", + "olmoe", // OLMoE 1B-7B (MoE) + "qwen3moe", // Qwen3 MoE: same topology as olmoe + // Qwen3.5/3.6 hybrid: Gated-DeltaNet linear attention on 3 of every 4 layers, full + // attention with M-RoPE and an interleaved query+gate projection on the rest. Verified + // token-exact against llama.cpp on Qwen3.5-0.8B-Q8_0 and Ternary-Bonsai-27B-Q2_g64. + // GREEDY / BATCH 1 ONLY: the recurrent conv and delta states are not reordered by + // beam_idx and have a static batch of 1, so beam search or batch > 1 fails at inference + // (a Concat shape mismatch on the conv window) rather than producing wrong output. + "qwen35", + "gpt-oss", // MoE + sinks + SWA + "gemma", // Gemma 2B / 7B + "gemma2", // Gemma 2: post-norms + attention soft-cap + "gemma3", // Gemma 3: post-norms + final logit soft-cap + "gemma4", // Gemma 4: SWA, per-layer embeddings, shared KV + }; + return archs; +} + +const std::set& experimental_archs() { + static const std::set archs = { + // H2 2025: dense + "llama-embed", // Bidirectional LLaMA (embedding, no causal mask) + "exaone4", // EXAONE 4.0: NEOX rope, post-norms (attn+ffn) + "plamo3", // PLaMo-3: NEOX rope, post-norms (attn+ffn) + "smollm3", // SmolLM3: NORMAL rope + SWA + // H2 2025: MoE + "hunyuan-moe", // Hunyuan MoE: NEOX rope, MoE routing, QK-norm + "glm4moe", // GLM 4.5 MoE: NEOX rope, 1 dense lead layer, MoE + attn post-norm + "exaone-moe", // EXAONE MoE: NEOX rope, SWA + MoE, shared expert + "minimax-m2", // Minimax M2: NEOX rope, pure MoE + "ernie4_5-moe", // Ernie 4.5 MoE: NORMAL rope, dense lead layers + MoE stride + "bailingmoe2", // BailingMoe V2: NEOX rope, MoE + shared expert + QK-norm + // 2026: dense + "maincoder", // Maincoder-1B: NORMAL rope, QK-norm (auto-detected) + "mistral3", // Ministral-3B: NORMAL rope, dense + "muse-glimmer", // Muse Glimmer (Meta Onyx): NORMAL rope on SWA layers only (global + // layers are NoPE), sigmoid attention output gate, QK-norm, + // pre+post norms, final logit soft-cap + // 2026: MoE + "mellum", // JetBrains Mellum: NEOX rope, pure MoE + "deepseek2-ocr", // DeepSeekOCR: NEOX rope, dense lead layers + MoE + "jais2", // JAIS-2: NEOX rope, dense (biases auto-detected) + }; + return archs; +} + +const std::set& supported_archs() { + static const std::set archs = [] { + std::set a = verified_archs(); + a.insert(experimental_archs().begin(), experimental_archs().end()); + return a; + }(); + return archs; +} + +} // namespace gguf +} // namespace frontend +} // namespace ov diff --git a/src/frontends/gguf/src/builder/arch_registry.hpp b/src/frontends/gguf/src/builder/arch_registry.hpp new file mode 100644 index 00000000000000..59edd3ef23103f --- /dev/null +++ b/src/frontends/gguf/src/builder/arch_registry.hpp @@ -0,0 +1,41 @@ +// Copyright (C) 2018-2026 Intel Corporation +// SPDX-License-Identifier: Apache-2.0 +// + +#pragma once + +#include +#include + +namespace ov { +namespace frontend { +namespace gguf { + +// gguf ROPE op-case (see ggml-decoder.cpp::compute_op_case). The high 16 bits encode the +// mode: NORMAL=0 (llama/minicpm: rotate consecutive pairs), NEOX=1 (qwen/phi/hunyuan: +// rotate halves). The input is not a VIEW here so the low bits stay 0. +constexpr int ROPE_OP_CASE_NORMAL = 0x00000000; +constexpr int ROPE_OP_CASE_NEOX = 0x00010000; +// IMROPE=2: interleaved multimodal rope (qwen35 / qwen3vl). inp_pos carries 4 mrope sections. +constexpr int ROPE_OP_CASE_IMROPE = 0x00020000; + +// Architectures whose rope is NEOX (rotate-halves); everything else in the supported set +// uses NORMAL (rotate consecutive pairs). Mirrors llama_model_rope_type. +bool arch_uses_neox_rope(const std::string& arch); + +// Architectures END-TO-END VERIFIED on the generic decoder builder: convert + compile + generation +// checked against the reference (native llama.cpp / HF) on a real checkpoint. Safe to rely on. +const std::set& verified_archs(); + +// Architectures EXPECTED to work via the generic builder's GGUF-tensor-table auto-detection, but +// NOT yet end-to-end verified on a real checkpoint. Enabled (they convert), but conversion emits a +// one-time warning so callers know they are best-effort. Promote to verified_archs() once a model +// of the family has been checked against the reference. See docs/adding_an_architecture.md. +const std::set& experimental_archs(); + +// All architectures the native builder accepts = verified + experimental. +const std::set& supported_archs(); + +} // namespace gguf +} // namespace frontend +} // namespace ov diff --git a/src/frontends/gguf/src/builder/blocks/attention.cpp b/src/frontends/gguf/src/builder/blocks/attention.cpp new file mode 100644 index 00000000000000..d567f7fd53a163 --- /dev/null +++ b/src/frontends/gguf/src/builder/blocks/attention.cpp @@ -0,0 +1,301 @@ +// Copyright (C) 2018-2026 Intel Corporation +// SPDX-License-Identifier: Apache-2.0 +// + +#include "builder/blocks/attention.hpp" + +#include +#include + +#include "builder/blocks/common.hpp" +#include "builder/blocks/qkv_repack.hpp" + +namespace ov { +namespace frontend { +namespace gguf { +namespace blocks { + +using ov::element::f32; + +namespace { +// Dynamic token length, used for model-input Parameters only. +constexpr int64_t D = -1; +} // namespace + +KvCachePlan KvCachePlan::build(const DecoderConfig& cfg) { + KvCachePlan plan; + plan.n_own_kv = cfg.n_layer - cfg.shared_kv_layers; + if (cfg.shared_kv_layers > 0) { + for (int i = plan.n_own_kv - 1; i >= 0; --i) { + const bool i_is_swa = cfg.layer_is_swa(i); + if (plan.anchor_swa < 0 && i_is_swa) + plan.anchor_swa = i; + if (plan.anchor_global < 0 && !i_is_swa) + plan.anchor_global = i; + if (plan.anchor_swa >= 0 && plan.anchor_global >= 0) + break; + } + } + return plan; +} + +std::string attention(GraphEmitter& e, + const DecoderConfig& cfg, + const KvCachePlan& kv, + int il, + const std::string& attn_norm, + int64_t T) { + const std::string p = "blk." + std::to_string(il) + "."; + auto& graph = *e.graph(); + + // Per-layer hyperparameters (SWA vs global head size / rope / scale; per-layer KV heads). + // See the DecoderConfig accessors for the arch-specific derivation rules. + const bool is_swa_layer = cfg.layer_is_swa(il); + const int head_size_l = cfg.layer_head_size(il); + const int n_head_kv_l = cfg.layer_n_head_kv(il); + const float kq_scale = cfg.layer_kq_scale(il); + const RopeConfig rope_config_l = cfg.layer_rope_config(il); + + // Q/K/V projections: MUL_MAT(w, attn_norm), then conceptual reshape to heads. + // Fused-QKV archs (phi-3, minicpm) carry a single attn_qkv weight; split it into + // separate q/k/v weights so the rest of the layer is architecture-agnostic. + if (cfg.is_qwen35) { + register_qwen35_q_gate(e, cfg, il); + e.add_weight(p + "attn_k.weight"); + e.add_weight(p + "attn_v.weight"); + } else if (cfg.has_fused_qkv) { + register_fused_qkv(e, cfg, il); + } else { + e.add_weight(p + "attn_q.weight"); + e.add_weight(p + "attn_k.weight"); + // MQA tie-V: some global attention layers (e.g. gemma4-12B every 6th layer) have + // head_count_kv==1 and no separate attn_v.weight; V shares K's weight tensor. + if (e.has_weight(p + "attn_v.weight")) { + e.add_weight(p + "attn_v.weight"); + } else { + // MQA tie-V: emit a separate attn_v.weight GGML_OP_NONE leaf that reuses K's + // extracted tensors, so attn_v.weight resolves in the tensor map like any weight. + e.add_weight_from(p + "attn_v.weight", p + "attn_k"); + } + } + auto q = e.add_op("GGML_OP_MUL_MAT", + p + "Qcur", + {p + "attn_q.weight", attn_norm}, + ps({1, 1, T, head_size_l * cfg.n_head}), + f32); + auto k = e.add_op("GGML_OP_MUL_MAT", + p + "Kcur", + {p + "attn_k.weight", attn_norm}, + ps({1, 1, T, head_size_l * n_head_kv_l}), + f32); + auto v = e.add_op("GGML_OP_MUL_MAT", + p + "Vcur", + {p + "attn_v.weight", attn_norm}, + ps({1, 1, T, head_size_l * n_head_kv_l}), + f32); + + // Q/K/V projection biases (qwen2 / qwen2.5: separate attn_{q,k,v}.bias). + if (cfg.has_qkv_bias && !cfg.has_fused_qkv) { + q = add_bias(e, q, p + "attn_q.bias", p + "Qcur_b"); + k = add_bias(e, k, p + "attn_k.bias", p + "Kcur_b"); + v = add_bias(e, v, p + "attn_v.bias", p + "Vcur_b"); + } + + // Full-width q/k norm (OLMoE): normalize the whole projection before splitting heads. + if (cfg.has_qk_norm && cfg.qk_norm_full) { + q = rms_norm(e, q, p + "attn_q_norm.weight", p + "Qcur_normed", cfg.rms_eps); + k = rms_norm(e, k, p + "attn_k_norm.weight", p + "Kcur_normed", cfg.rms_eps); + } + + // reshape Q/K/V to [1, n_tokens, n_head(_kv), head_size] + q = e.add_op("GGML_OP_RESHAPE", p + "Qcur_r", {q}, ps({1, T, cfg.n_head, head_size_l}), f32, 1); + k = e.add_op("GGML_OP_RESHAPE", p + "Kcur_r", {k}, ps({1, T, n_head_kv_l, head_size_l}), f32, 1); + v = e.add_op("GGML_OP_RESHAPE", p + "Vcur_r", {v}, ps({1, T, n_head_kv_l, head_size_l}), f32, 1); + + // per-head q_norm / k_norm (qwen3, hunyuan, gemma4) + if (cfg.has_qk_norm && !cfg.qk_norm_full) { + q = rms_norm(e, q, p + "attn_q_norm.weight", p + "Qcur_normed", cfg.rms_eps); + k = rms_norm(e, k, p + "attn_k_norm.weight", p + "Kcur_normed", cfg.rms_eps); + } + // gemma4: V gets a plain RMSNorm (no multiplicative weight, just normalize). + if (cfg.has_v_norm) { + v = e.add_op("GGML_OP_RMS_NORM", + p + "Vcur_normed", + {v}, + ps({1, T, n_head_kv_l, head_size_l}), + f32, + 0, + {{"eps", cfg.rms_eps}}); + } + + // RoPE (NEOX). rope_freqs.weight (per-dim frequency factor) is an optional 3rd input. + // For gemma4: global layers use rope_freqs (proportional/NTK scaling), SWA layers don't. + // muse-glimmer ropes ONLY its sliding-window layers; its global layers are NoPE + // (llama.cpp muse-glimmer.cpp: `const bool use_rope = hparams.is_swa(il)`). + const bool use_rope = !cfg.rope_on_swa_only || is_swa_layer; + const bool use_rope_freqs = cfg.has_rope_freqs && !is_swa_layer; + const std::vector q_rope_in = use_rope_freqs + ? std::vector{q, "inp_pos", "rope_freqs.weight"} + : std::vector{q, "inp_pos"}; + const std::vector k_rope_in = use_rope_freqs + ? std::vector{k, "inp_pos", "rope_freqs.weight"} + : std::vector{k, "inp_pos"}; + if (use_rope) { + q = e.add_op("GGML_OP_ROPE", + p + "Qcur_rope", + q_rope_in, + ps({1, T, cfg.n_head, head_size_l}), + f32, + cfg.rope_op_case, + {{"rope_config", rope_config_l}}); + k = e.add_op("GGML_OP_ROPE", + p + "Kcur_rope", + k_rope_in, + ps({1, T, n_head_kv_l, head_size_l}), + f32, + cfg.rope_op_case, + {{"rope_config", rope_config_l}}); + } + + // ---- KV cache store ---- + // Gemma4: layers with shared_kv_layers have no K/V of their own; they reuse the KV + // from the last layer of the same SWA type that has its own KV cache. SWA layers + // reuse the last own-KV SWA layer; global layers reuse the last own-KV global layer. + const bool has_own_kv = (cfg.shared_kv_layers == 0) || (il < kv.n_own_kv); + int anchor_il = il; + if (!has_own_kv) { + anchor_il = is_swa_layer ? kv.anchor_swa : kv.anchor_global; + if (anchor_il < 0) + anchor_il = kv.n_own_kv - 1; // fallback + } + const std::string kc = "cache_k_l" + std::to_string(anchor_il); + const std::string vc = "cache_v_l" + std::to_string(anchor_il); + + if (has_own_kv) { + // Per-layer f16 KV cache Parameters. The K/V read back out of a cache are f16, and so is + // Q after its convert in the translator, so the FLASH_ATTN inputs agree. + const ov::PartialShape cache_shape = ps({1, D, n_head_kv_l, head_size_l}); + if (!e.has_model_input(kc)) { + e.add_input(kc, ov::element::f16, cache_shape); + e.add_input(vc, ov::element::f16, cache_shape); + } + e.set_tensor_meta(kc, ps({1, T, n_head_kv_l, head_size_l}), ov::element::f16); + e.set_tensor_meta(vc, ps({1, T, n_head_kv_l, head_size_l}), ov::element::f16); + + // SET_ROWS(cur, idx, cache) -> the cache with this step's rows written in. Lowered by + // the frontend to a stateless ScatterUpdate, or by the caller-registered MakeStateful + // extension to a ReadValue/Concat/Assign OpenVINO state. + k = e.add_op("GGML_OP_SET_ROWS", + kc, + {k, "inp_kv_idx", kc}, + ps({1, T, n_head_kv_l, head_size_l}), + ov::element::f16); + v = e.add_op("GGML_OP_SET_ROWS", + vc, + {v, "inp_kv_idx", vc}, + ps({1, T, n_head_kv_l, head_size_l}), + ov::element::f16); + // The written-through caches are model outputs, so the stateless graph returns each + // updated cache as a Result (which MakeStateful, when registered, turns into an Assign + // sink paired with the cache's ReadValue). + graph.model_output_names.push_back(kc); + graph.model_output_names.push_back(vc); + } else { + // Shared-KV layer: K/V have already been set in the anchor layer's SET_ROWS. + // Use the anchor's combined cache. If the current layer has a smaller head size + // (SWA shared layer vs a global anchor), slice K/V to the layer's head_size along + // the last dim, mirroring llama.cpp's ggml_view_4d with n_embd_head_k(il). + const ov::PartialShape& anchor_kc_shape = e.shape_of_tensor(kc); // [1, T, n_kv, anchor_head] + const int64_t anchor_hs = anchor_kc_shape[3].is_static() ? anchor_kc_shape[3].get_length() : cfg.head_size; + if (head_size_l < static_cast(anchor_hs)) { + // Shrink the last (head-size) axis to head_size_l. This is a plain single-axis shrink + // at offset 0, which is exactly what the shared VIEW op_case 3 does -- and what the + // cgraph decoder assigns to llama.cpp's corresponding ggml_view_4d with + // n_embd_head_k(il) -- so describe it the way that case expects rather than with a + // builder-only case: "view_slice" = {ov_axis, start, len} plus the input's own shape. + // No "view_reshape": the sliced shape already is this node's output shape. + const ov::PartialShape slice_shape = ps({1, T, n_head_kv_l, head_size_l}); + const std::vector hs_slice{3, 0, int64_t(head_size_l)}; + k = e.add_op("GGML_OP_VIEW", + p + "k_hslice", + {kc}, + slice_shape, + ov::element::f16, + 3, + {{"view_slice", hs_slice}, {"input_ggml_shape", e.static_shape_of(kc)}}); + v = e.add_op("GGML_OP_VIEW", + p + "v_hslice", + {vc}, + slice_shape, + ov::element::f16, + 3, + {{"view_slice", hs_slice}, {"input_ggml_shape", e.static_shape_of(vc)}}); + } else { + k = kc; + v = vc; + } + } + + // NOTE: q/k/v stay in the ggml-natural [1, n_tokens, n_head(_kv), head_size] layout + // here. The PERMUTE to [1, n_head, n_tokens, head_size] is done INSIDE the FLASH_ATTN + // translator, AFTER the GQA broadcast of K/V. That ordering -- Concat -> GQA tile -> + // single Transpose -> SDPA -- is the exact shape the CPU plugin's stateful_sdpa_fusion + // matches (its multi-query-bcst sits on the concat output, before one transpose), so + // the attention fuses into ScaledDotProductAttentionWithKVCache. Permuting before the + // tile (the old order) put the transpose between concat and tile and blocked the fuse. + + // FLASH_ATTN_EXT(q, k, v, mask[, sinks]) -> [1, n_tokens, n_head, head_size]. + // gpt-oss: SWA layers use the sliding-window mask; plus a per-head sink logit. + const std::string mask_name = is_swa_layer ? "self_kq_mask_swa" : "self_kq_mask"; + std::vector attn_in = {q, k, v, mask_name}; + if (cfg.has_sinks) { + e.add_named_weight(p + "attn_sinks.weight"); + attn_in.push_back(p + "attn_sinks.weight"); + } + std::map attn_attrs = {{"scale", kq_scale}}; + if (cfg.attn_soft_cap != 0.0f) { + attn_attrs["kq_soft_cap"] = cfg.attn_soft_cap; + } + auto attn = e.add_op("GGML_OP_FLASH_ATTN_EXT", + p + "kqv", + attn_in, + ps({1, T, cfg.n_head, head_size_l}), + f32, + 100, // builder layout: q/k/v are ggml-natural [1, T, n_head, head_size] + std::move(attn_attrs)); + + // reshape back to [1, 1, n_tokens, n_head*head_size] + auto attn_2d = + e.add_op("GGML_OP_RESHAPE", p + "kqv_merged", {attn}, ps({1, 1, T, cfg.n_head * head_size_l}), f32, 2); + + // muse-glimmer: sigmoid output gate. The gate is a projection of the PRE-attention + // normed hidden (the same `attn_norm` tensor Q/K/V come from), squashed by sigmoid and + // multiplied elementwise into the merged attention output before the wo projection. + if (cfg.has_attn_gate) { + e.add_weight(p + "attn_gate.weight"); + auto gate = e.add_op("GGML_OP_MUL_MAT", + p + "attn_gate", + {p + "attn_gate.weight", attn_norm}, + ps({1, 1, T, cfg.n_head * head_size_l}), + f32); + gate = e.add_op("GGML_UNARY_OP_SIGMOID", p + "attn_gate_sig", {gate}, e.shape_of_tensor(gate), f32); + attn_2d = e.add_op("GGML_OP_MUL", p + "kqv_gated", {attn_2d, gate}, e.shape_of_tensor(attn_2d), f32); + } + + // output projection (+ optional bias) + e.add_weight(p + "attn_output.weight"); + auto attn_out = e.add_op("GGML_OP_MUL_MAT", + p + "attn_out", + {p + "attn_output.weight", attn_2d}, + ps({1, 1, T, cfg.n_embd}), + f32); + if (cfg.has_attn_out_bias) { + attn_out = add_bias(e, attn_out, p + "attn_output.bias", p + "attn_out_b"); + } + return attn_out; +} + +} // namespace blocks +} // namespace gguf +} // namespace frontend +} // namespace ov diff --git a/src/frontends/gguf/src/builder/blocks/attention.hpp b/src/frontends/gguf/src/builder/blocks/attention.hpp new file mode 100644 index 00000000000000..1724d116e0e852 --- /dev/null +++ b/src/frontends/gguf/src/builder/blocks/attention.hpp @@ -0,0 +1,48 @@ +// Copyright (C) 2018-2026 Intel Corporation +// SPDX-License-Identifier: Apache-2.0 +// + +#pragma once + +#include + +#include "builder/decoder_config.hpp" +#include "builder/graph_emitter.hpp" + +namespace ov { +namespace frontend { +namespace gguf { +namespace blocks { + +// Per-layer KV-cache routing, precomputed once for the whole stack. +// +// gemma4 shares KV across layers: the trailing `shared_kv_layers` layers have no K/V of their own +// and reuse the cache of the last own-KV layer of the SAME sliding-window type. Everything else +// leaves this at its defaults, where every layer owns its cache. +struct KvCachePlan { + int n_own_kv = 0; // first n_own_kv layers have their own KV cache + int anchor_swa = -1; // last own-KV sliding-window layer, or -1 + int anchor_global = -1; // last own-KV global layer, or -1 + + // Build the plan for `cfg`; mirrors llama.cpp's layer_reuse_cb. + static KvCachePlan build(const DecoderConfig& cfg); +}; + +// The attention sublayer of one decoder layer: Q/K/V projections (+ optional biases and Q/K/V +// norms), RoPE, the KV-cache store, FLASH_ATTN_EXT, the optional sigmoid output gate, and the +// output projection (+ optional bias). +// +// `attn_norm` is the pre-attention normed hidden. Returns the sublayer output tensor name BEFORE +// the residual add, matching what gated_delta_net() returns for the recurrent layers, so the +// decoder's shared FFN tail consumes either interchangeably. +std::string attention(GraphEmitter& e, + const DecoderConfig& cfg, + const KvCachePlan& kv, + int il, + const std::string& attn_norm, + int64_t T); + +} // namespace blocks +} // namespace gguf +} // namespace frontend +} // namespace ov diff --git a/src/frontends/gguf/src/builder/blocks/common.cpp b/src/frontends/gguf/src/builder/blocks/common.cpp new file mode 100644 index 00000000000000..517a752084bfbb --- /dev/null +++ b/src/frontends/gguf/src/builder/blocks/common.cpp @@ -0,0 +1,46 @@ +// Copyright (C) 2018-2026 Intel Corporation +// SPDX-License-Identifier: Apache-2.0 +// + +#include "builder/blocks/common.hpp" + +namespace ov { +namespace frontend { +namespace gguf { +namespace blocks { + +std::string rms_norm(GraphEmitter& e, + const std::string& in, + const std::string& weight, + const std::string& out_prefix, + float eps) { + e.add_weight(weight); + auto norm = e.add_op("GGML_OP_RMS_NORM", + out_prefix + ".rms", + {in}, + e.shape_of_tensor(in), + ov::element::f32, + 0, + {{"eps", eps}}); + return e.add_op("GGML_OP_MUL", out_prefix, {norm, weight}, e.shape_of_tensor(in), ov::element::f32); +} + +std::string scale(GraphEmitter& e, const std::string& x, float factor, const std::string& name) { + return e.add_op("GGML_OP_SCALE", + name, + {x}, + e.shape_of_tensor(x), + ov::element::f32, + 0, + {{"scale", factor}, {"bias", 0.0f}}); +} + +std::string add_bias(GraphEmitter& e, const std::string& x, const std::string& bias_weight, const std::string& name) { + e.add_named_weight(bias_weight); + return e.add_op("GGML_OP_ADD", name, {x, bias_weight}, e.shape_of_tensor(x), ov::element::f32); +} + +} // namespace blocks +} // namespace gguf +} // namespace frontend +} // namespace ov diff --git a/src/frontends/gguf/src/builder/blocks/common.hpp b/src/frontends/gguf/src/builder/blocks/common.hpp new file mode 100644 index 00000000000000..b79961fb29fa8a --- /dev/null +++ b/src/frontends/gguf/src/builder/blocks/common.hpp @@ -0,0 +1,40 @@ +// Copyright (C) 2018-2026 Intel Corporation +// SPDX-License-Identifier: Apache-2.0 +// + +#pragma once + +#include + +#include "builder/graph_emitter.hpp" + +namespace ov { +namespace frontend { +namespace gguf { +namespace blocks { + +// Small, architecture-agnostic graph fragments shared by every model family. +// +// They are free functions over GraphEmitter rather than methods on a builder so a future +// non-decoder family (mmproj/vision, audio) can reuse them without inheriting any LLM +// hyperparameters. Anything that needs to know about heads, layers or KV caches belongs in a +// decoder-specific block instead (see ffn.hpp / attention.hpp). + +// RMS_NORM followed by elementwise MUL with the norm weight (llama.cpp build_norm, LLM_NORM_RMS). +// `eps` is the epsilon fed to the RMS_NORM op. +std::string rms_norm(GraphEmitter& e, + const std::string& in, + const std::string& weight, + const std::string& out_prefix, + float eps); + +// Scale a tensor by a constant: GGML_OP_SCALE with attr "scale" (and bias 0). +std::string scale(GraphEmitter& e, const std::string& x, float factor, const std::string& name); + +// Elementwise add of a (broadcast) bias weight: GGML_OP_ADD(x, bias_weight). +std::string add_bias(GraphEmitter& e, const std::string& x, const std::string& bias_weight, const std::string& name); + +} // namespace blocks +} // namespace gguf +} // namespace frontend +} // namespace ov diff --git a/src/frontends/gguf/src/builder/blocks/ffn.cpp b/src/frontends/gguf/src/builder/blocks/ffn.cpp new file mode 100644 index 00000000000000..197614e07764df --- /dev/null +++ b/src/frontends/gguf/src/builder/blocks/ffn.cpp @@ -0,0 +1,247 @@ +// Copyright (C) 2018-2026 Intel Corporation +// SPDX-License-Identifier: Apache-2.0 +// + +#include "builder/blocks/ffn.hpp" + +#include "builder/blocks/common.hpp" + +namespace ov { +namespace frontend { +namespace gguf { +namespace blocks { + +using ov::element::f32; + +std::string geglu_ffn(GraphEmitter& e, + const DecoderConfig& cfg, + const std::string& p, + const std::string& ffn_norm, + int64_t T) { + e.add_weight(p + "ffn_gate.weight"); + e.add_weight(p + "ffn_up.weight"); + e.add_weight(p + "ffn_down.weight"); + const int n_ff = static_cast(e.weight_tensor(p + "ffn_gate.weight").get_shape()[0]); + auto gate = + e.add_op("GGML_OP_MUL_MAT", p + "ffn_gate", {p + "ffn_gate.weight", ffn_norm}, ps({1, 1, T, n_ff}), f32); + auto up = e.add_op("GGML_OP_MUL_MAT", p + "ffn_up", {p + "ffn_up.weight", ffn_norm}, ps({1, 1, T, n_ff}), f32); + auto glu = + e.add_op("GGML_GLU_OP_GEGLU", p + "ffn_geglu", {gate, up}, ps({1, 1, T, n_ff}), f32, 0, {{"swapped", false}}); + return e.add_op("GGML_OP_MUL_MAT", p + "ffn_out", {p + "ffn_down.weight", glu}, ps({1, 1, T, cfg.n_embd}), f32); +} + +std::string dense_ffn(GraphEmitter& e, + const DecoderConfig& cfg, + const std::string& p, + const std::string& ffn_norm, + int64_t T) { + e.add_weight(p + "ffn_up.weight"); + e.add_weight(p + "ffn_down.weight"); + const bool fused_ffn = !e.has_weight(p + "ffn_gate.weight"); // phi-3: fused gate+up + const bool has_ffn_bias = e.has_weight(p + "ffn_up.bias"); + std::string glu; + if (fused_ffn) { + const int up2 = static_cast(e.weight_tensor(p + "ffn_up.weight").get_shape()[0]); // 2*n_ff + auto up = e.add_op("GGML_OP_MUL_MAT", p + "ffn_up", {p + "ffn_up.weight", ffn_norm}, ps({1, 1, T, up2}), f32); + glu = e.add_op("GGML_GLU_OP_SWIGLU", + p + "ffn_swiglu", + {up}, + ps({1, 1, T, up2 / 2}), + f32, + 0, + {{"swapped", false}}); + } else { + e.add_weight(p + "ffn_gate.weight"); + const int n_ff = static_cast(e.weight_tensor(p + "ffn_gate.weight").get_shape()[0]); + auto gate = + e.add_op("GGML_OP_MUL_MAT", p + "ffn_gate", {p + "ffn_gate.weight", ffn_norm}, ps({1, 1, T, n_ff}), f32); + if (has_ffn_bias) { + gate = add_bias(e, gate, p + "ffn_gate.bias", p + "ffn_gate_b"); + } + auto up = e.add_op("GGML_OP_MUL_MAT", p + "ffn_up", {p + "ffn_up.weight", ffn_norm}, ps({1, 1, T, n_ff}), f32); + if (has_ffn_bias) { + up = add_bias(e, up, p + "ffn_up.bias", p + "ffn_up_b"); + } + glu = e.add_op("GGML_GLU_OP_SWIGLU", + p + "ffn_swiglu", + {gate, up}, + ps({1, 1, T, n_ff}), + f32, + 0, + {{"swapped", false}}); + } + auto out = e.add_op("GGML_OP_MUL_MAT", p + "ffn_out", {p + "ffn_down.weight", glu}, ps({1, 1, T, cfg.n_embd}), f32); + if (has_ffn_bias) { + out = add_bias(e, out, p + "ffn_down.bias", p + "ffn_out_b"); + } + return out; +} + +std::string moe_ffn(GraphEmitter& e, + const DecoderConfig& cfg, + const std::string& p, + const std::string& ffn_norm, + int64_t T) { + const int E = cfg.n_expert, K = cfg.n_expert_used; + const int n_ff = static_cast(e.weight_tensor(p + "ffn_gate_exps.weight").get_shape()[1]); + + // --- router: logits [1,1,T,E] = gate_inp · x --- + e.add_weight(p + "ffn_gate_inp.weight"); + auto logits = + e.add_op("GGML_OP_MUL_MAT", p + "moe_logits", {p + "ffn_gate_inp.weight", ffn_norm}, ps({1, 1, T, E}), f32); + if (cfg.has_moe_gate_bias) { + logits = add_bias(e, logits, p + "ffn_gate_inp.bias", p + "moe_logits_b"); + } + + // gating: softmax (OLMoE) or softmax-after-topk (gpt-oss "softmax_weight"). + std::string probs = logits; + if (!cfg.moe_softmax_weight) { + probs = e.add_op("GGML_OP_SOFT_MAX", + p + "moe_probs", + {logits}, + ps({1, 1, T, E}), + f32, + 0, + {{"softmax_axis", int64_t(-1)}}); + } + + // top-k expert selection -> indices [1,1,T,K] (i32). + auto selected = e.add_op("GGML_OP_TOP_K", p + "moe_topk", {probs}, ps({1, 1, T, K}), ov::element::i32); + + // weights = gather probs by selected; op_case 10 returns a per-expert column + // [1,T,K,1] (robust to dynamic T). gpt-oss softmaxes over the K (expert) axis. + auto weights = e.add_op("GGML_OP_GET_ROWS", p + "moe_w", {probs, selected}, ps({1, T, K, 1}), f32, 10); + if (cfg.moe_softmax_weight) { + weights = e.add_op("GGML_OP_SOFT_MAX", + p + "moe_w_sm", + {weights}, + ps({1, T, K, 1}), + f32, + 0, + {{"softmax_axis", int64_t(2)}}); + } + // gpt-oss expert_weights_scale: optional constant multiplier applied after softmax + // (mirrors llama.cpp build_moe_ffn w_scale != 0 && w_scale != 1.0 path). + if (cfg.expert_weights_scale != 0.0f && cfg.expert_weights_scale != 1.0f) { + weights = scale(e, weights, cfg.expert_weights_scale, p + "moe_w_scaled"); + } + + // expert FFN via MUL_MAT_ID. The routed input x is broadcast to K slots; the + // translator gathers each token's selected expert matrices. gpt-oss adds per-expert + // biases (ADD_ID gathers the selected experts' bias rows). + e.add_weight(p + "ffn_gate_exps.weight"); + e.add_weight(p + "ffn_up_exps.weight"); + e.add_weight(p + "ffn_down_exps.weight"); + const bool eb = cfg.has_moe_expert_bias; + auto up = e.add_op("GGML_OP_MUL_MAT_ID", + p + "moe_up", + {p + "ffn_up_exps.weight", ffn_norm, selected}, + ps({1, T, K, n_ff}), + f32); + if (eb) { + e.add_named_weight(p + "ffn_up_exps.bias"); + up = e.add_op("GGML_OP_ADD_ID", + p + "moe_up_b", + {up, p + "ffn_up_exps.bias", selected}, + ps({1, T, K, n_ff}), + f32); + } + auto gate = e.add_op("GGML_OP_MUL_MAT_ID", + p + "moe_gate", + {p + "ffn_gate_exps.weight", ffn_norm, selected}, + ps({1, T, K, n_ff}), + f32); + if (eb) { + e.add_named_weight(p + "ffn_gate_exps.bias"); + gate = e.add_op("GGML_OP_ADD_ID", + p + "moe_gate_b", + {gate, p + "ffn_gate_exps.bias", selected}, + ps({1, T, K, n_ff}), + f32); + } + std::string act; + if (cfg.moe_swiglu_oai) { + act = e.add_op("GGML_GLU_OP_SWIGLU_OAI", + p + "moe_act", + {gate, up}, + ps({1, T, K, n_ff}), + f32, + 0, + // Attribute names must match the cgraph decoder's (ggml-decoder.cpp), which + // is the external contract the shared translators read: glu_alpha/glu_limit. + {{"swapped", false}, {"glu_alpha", 1.702f}, {"glu_limit", 7.0f}}); + } else { + act = e.add_op("GGML_GLU_OP_SWIGLU", + p + "moe_act", + {gate, up}, + ps({1, T, K, n_ff}), + f32, + 0, + {{"swapped", false}}); + } + auto experts = e.add_op("GGML_OP_MUL_MAT_ID", + p + "moe_down", + {p + "ffn_down_exps.weight", act, selected}, + ps({1, T, K, cfg.n_embd}), + f32); + if (eb) { + e.add_named_weight(p + "ffn_down_exps.bias"); + experts = e.add_op("GGML_OP_ADD_ID", + p + "moe_down_b", + {experts, p + "ffn_down_exps.bias", selected}, + ps({1, T, K, cfg.n_embd}), + f32); + } + + // Weighted sum over the K selected experts. weights is [1,T,K,1] (per-expert col). + // experts [1,T,K,n_embd] * weights -> [1,T,K,n_embd] + // TRANSPOSE last two axes -> [1,T,n_embd,K]; SUM_ROWS over K -> [1,T,n_embd,1] + // RESHAPE (dynamic) -> [1,1,T,n_embd]. + // The reshape is op_case 5: collapse to [1, 1, -1, n_embd], the token axis staying dynamic. + // That is the case the cgraph decoder would assign to this same reshape too -- ggml ne + // [1,n_embd,T,1] -> [n_embd,T,1,1] satisfies both of its case-5 predicates -- so the shared + // case applies as-is and needs no builder-specific numbering. + auto weighted = e.add_op("GGML_OP_MUL", p + "moe_weighted", {experts, weights}, ps({1, T, K, cfg.n_embd}), f32); + auto tr = e.add_op("GGML_OP_TRANSPOSE", p + "moe_tr", {weighted}, ps({1, T, cfg.n_embd, K}), f32); + auto summed = e.add_op("GGML_OP_SUM_ROWS", p + "moe_sum", {tr}, ps({1, T, cfg.n_embd, 1}), f32); + auto moe_out = e.add_op("GGML_OP_RESHAPE", p + "moe_out", {summed}, ps({1, 1, T, cfg.n_embd}), f32, 5); + + // Shared experts (deepseek2-ocr, bailingmoe2, exaone-moe): always-active experts whose + // output is added to the routed experts' weighted sum. Uses plain SwiGLU dense FFN with + // ffn_{gate,up,down}_shexp.weight (n_ff_shared = shexp rows). Output added to moe_out. + if (cfg.n_expert_shared > 0 && e.has_weight(p + "ffn_gate_shexp.weight")) { + e.add_weight(p + "ffn_gate_shexp.weight"); + e.add_weight(p + "ffn_up_shexp.weight"); + e.add_weight(p + "ffn_down_shexp.weight"); + const int n_ff_s = static_cast(e.weight_tensor(p + "ffn_gate_shexp.weight").get_shape()[0]); + auto s_gate = e.add_op("GGML_OP_MUL_MAT", + p + "shexp_gate", + {p + "ffn_gate_shexp.weight", ffn_norm}, + ps({1, 1, T, n_ff_s}), + f32); + auto s_up = e.add_op("GGML_OP_MUL_MAT", + p + "shexp_up", + {p + "ffn_up_shexp.weight", ffn_norm}, + ps({1, 1, T, n_ff_s}), + f32); + auto s_act = e.add_op("GGML_GLU_OP_SWIGLU", + p + "shexp_act", + {s_gate, s_up}, + ps({1, 1, T, n_ff_s}), + f32, + 0, + {{"swapped", false}}); + auto s_down = e.add_op("GGML_OP_MUL_MAT", + p + "shexp_out", + {p + "ffn_down_shexp.weight", s_act}, + ps({1, 1, T, cfg.n_embd}), + f32); + moe_out = e.add_op("GGML_OP_ADD", p + "moe_shared_out", {moe_out, s_down}, ps({1, 1, T, cfg.n_embd}), f32); + } + return moe_out; +} + +} // namespace blocks +} // namespace gguf +} // namespace frontend +} // namespace ov diff --git a/src/frontends/gguf/src/builder/blocks/ffn.hpp b/src/frontends/gguf/src/builder/blocks/ffn.hpp new file mode 100644 index 00000000000000..1f9b1a671aec0c --- /dev/null +++ b/src/frontends/gguf/src/builder/blocks/ffn.hpp @@ -0,0 +1,51 @@ +// Copyright (C) 2018-2026 Intel Corporation +// SPDX-License-Identifier: Apache-2.0 +// + +#pragma once + +#include + +#include "builder/decoder_config.hpp" +#include "builder/graph_emitter.hpp" + +namespace ov { +namespace frontend { +namespace gguf { +namespace blocks { + +// Feed-forward sublayers of the decoder block, mirroring llm_graph_context::build_ffn / +// build_moe_ffn. `prefix` is the layer prefix ("blk.."), `ffn_norm` the pre-FFN normed hidden, +// and `T` the representative static token length. Each returns the sublayer output tensor name, +// before the residual add. + +// Dense SwiGLU FFN (llama/qwen/phi3/minicpm). Supports the fused gate+up projection (phi-3, no +// ffn_gate.weight) and optional per-projection biases (pangu-embedded), both auto-detected from +// the weight table. +std::string dense_ffn(GraphEmitter& e, + const DecoderConfig& cfg, + const std::string& prefix, + const std::string& ffn_norm, + int64_t T); + +// Dense GeGLU FFN (Gemma/Gemma2). Same layout as SwiGLU but uses GELU activation. +std::string geglu_ffn(GraphEmitter& e, + const DecoderConfig& cfg, + const std::string& prefix, + const std::string& ffn_norm, + int64_t T); + +// Mixture-of-experts FFN (OLMoE / gpt-oss / qwen3moe), mirroring llm_graph_context::build_moe_ffn. +// Routing: logits = gate_inp·x; probs = softmax/identity; pick top-k experts; per-token expert +// matmuls via MUL_MAT_ID; gated activation; weighted sum over the used experts; plus the optional +// always-active shared experts. +std::string moe_ffn(GraphEmitter& e, + const DecoderConfig& cfg, + const std::string& prefix, + const std::string& ffn_norm, + int64_t T); + +} // namespace blocks +} // namespace gguf +} // namespace frontend +} // namespace ov diff --git a/src/frontends/gguf/src/builder/blocks/gated_delta_net.cpp b/src/frontends/gguf/src/builder/blocks/gated_delta_net.cpp new file mode 100644 index 00000000000000..b61283427917ed --- /dev/null +++ b/src/frontends/gguf/src/builder/blocks/gated_delta_net.cpp @@ -0,0 +1,169 @@ +// Copyright (C) 2018-2026 Intel Corporation +// SPDX-License-Identifier: Apache-2.0 +// + +#include "builder/blocks/gated_delta_net.hpp" + +#include + +#include "builder/blocks/common.hpp" + +namespace ov { +namespace frontend { +namespace gguf { +namespace blocks { + +std::string gated_delta_net(GraphEmitter& e, + const DecoderConfig& cfg, + int il, + const std::string& attn_norm, + int64_t T) { + using ov::element::f32; + const std::string p = "blk." + std::to_string(il) + "."; + auto& graph = *e.graph(); + + const int64_t d_conv = cfg.ssm_conv_kernel; + const int64_t S = cfg.ssm_state_size; // head_k_dim == head_v_dim + const int64_t H_k = cfg.ssm_group_count; // num_k_heads + const int64_t H_v = cfg.ssm_dt_rank; // num_v_heads + const int64_t head_v = cfg.ssm_inner_size / H_v; // head_v_dim + const int64_t key_dim = S * H_k; + const int64_t value_dim = head_v * H_v; + const int64_t conv_dim = 2 * key_dim + value_dim; + + // ---- input projections ---- + e.add_weight(p + "attn_qkv.weight"); + auto qkv = + e.add_op("GGML_OP_MUL_MAT", p + "qkv_mixed", {p + "attn_qkv.weight", attn_norm}, ps({1, 1, T, conv_dim}), f32); + e.add_weight(p + "attn_gate.weight"); + auto z = e.add_op("GGML_OP_MUL_MAT", p + "z", {p + "attn_gate.weight", attn_norm}, ps({1, 1, T, value_dim}), f32); + + // beta = sigmoid(ssm_beta @ x), one scalar per v-head + e.add_weight(p + "ssm_beta.weight"); + auto beta = e.add_op("GGML_OP_MUL_MAT", p + "beta", {p + "ssm_beta.weight", attn_norm}, ps({1, 1, T, H_v}), f32); + beta = e.add_op("GGML_UNARY_OP_SIGMOID", p + "beta_sig", {beta}, ps({1, 1, T, H_v}), f32); + beta = e.add_op("GGML_OP_RESHAPE", p + "beta_4d", {beta}, ps({1, T, H_v, 1}), f32, 1); + + // g = softplus(ssm_alpha @ x + ssm_dt.bias) * ssm_a (ggml: -A_log.exp() * softplus) + e.add_weight(p + "ssm_alpha.weight"); + auto alpha = e.add_op("GGML_OP_MUL_MAT", p + "alpha", {p + "ssm_alpha.weight", attn_norm}, ps({1, 1, T, H_v}), f32); + e.add_named_weight(p + "ssm_dt.bias"); + alpha = e.add_op("GGML_OP_ADD", p + "alpha_biased", {alpha, p + "ssm_dt.bias"}, ps({1, 1, T, H_v}), f32); + alpha = e.add_op("GGML_UNARY_OP_SOFTPLUS", p + "alpha_sp", {alpha}, ps({1, 1, T, H_v}), f32); + e.add_named_weight(p + "ssm_a"); + auto g = e.add_op("GGML_OP_MUL", p + "gate", {alpha, p + "ssm_a"}, ps({1, 1, T, H_v}), f32); + g = e.add_op("GGML_OP_RESHAPE", p + "gate_4d", {g}, ps({1, T, H_v, 1}), f32, 1); + + // ---- causal depthwise conv over [conv state | this step's tokens] ---- + // conv_state holds the trailing d_conv-1 columns of the previous step's conv input. + const std::string cs = "conv_state_l" + std::to_string(il); + if (!e.has_model_input(cs)) { + e.add_input(cs, f32, ps({1, 1, conv_dim, d_conv - 1})); + } + e.set_tensor_meta(cs, ps({1, 1, conv_dim, d_conv - 1}), f32); + + // [1,1,T,conv_dim] -> [1,1,conv_dim,T] so the conv window grows along the last axis. + auto qkv_t = e.add_op("GGML_OP_TRANSPOSE", p + "qkv_t", {qkv}, ps({1, 1, conv_dim, T}), f32); + auto conv_in = e.add_op("GGML_OP_CONCAT", + p + "conv_in", + {cs, qkv_t}, + ps({1, 1, conv_dim, d_conv - 1 + T}), + f32, + 0, + {{"concat_axis", int{0}}}); + + // Next step's state is the trailing d_conv-1 columns of this window. + const std::vector tail_slice{3, -(d_conv - 1), d_conv - 1}; + auto cs_out = e.add_op("GGML_OP_VIEW", + cs + "_out", + {conv_in}, + ps({1, 1, conv_dim, d_conv - 1}), + f32, + 3, + {{"view_slice", tail_slice}, {"input_ggml_shape", e.static_shape_of(conv_in)}}); + graph.model_output_names.push_back(cs_out); + graph.recurrent_states.emplace_back(cs, cs_out); + + e.add_named_weight(p + "ssm_conv1d.weight"); + auto conv = + e.add_op("GGML_OP_SSM_CONV", p + "conv_out", {conv_in, p + "ssm_conv1d.weight"}, ps({1, 1, T, conv_dim}), f32); + conv = e.add_op("GGML_UNARY_OP_SILU", p + "conv_silu", {conv}, ps({1, 1, T, conv_dim}), f32); + + // ---- split the conv output into q | k | v and normalize q/k ---- + auto slice_heads = [&](const std::string& name, int64_t off, int64_t width, int64_t heads, int64_t dim) { + const std::vector sl{3, off, width}; + auto s = e.add_op("GGML_OP_VIEW", + p + name + "_s", + {conv}, + ps({1, 1, T, width}), + f32, + 3, + {{"view_slice", sl}, {"input_ggml_shape", e.static_shape_of(conv)}}); + return e.add_op("GGML_OP_RESHAPE", p + name, {s}, ps({1, T, heads, dim}), f32, 1); + }; + auto q = slice_heads("q_conv", 0, key_dim, H_k, S); + auto k = slice_heads("k_conv", key_dim, key_dim, H_k, S); + auto v = slice_heads("v_conv", 2 * key_dim, value_dim, H_v, head_v); + q = e.add_op("GGML_OP_L2_NORM", p + "q_l2", {q}, ps({1, T, H_k, S}), f32, 0, {{"eps", cfg.rms_eps}}); + k = e.add_op("GGML_OP_L2_NORM", p + "k_l2", {k}, ps({1, T, H_k, S}), f32, 0, {{"eps", cfg.rms_eps}}); + + // ---- recurrent delta rule ---- + // ggml state layout is [B, H_v, value_dim, key_dim]; the translator transposes it for + // the fused op and transposes the new state back, so the Parameter keeps ggml's layout. + const std::string ss = "ssm_state_l" + std::to_string(il); + if (!e.has_model_input(ss)) { + e.add_input(ss, f32, ps({1, H_v, head_v, S})); + } + e.set_tensor_meta(ss, ps({1, H_v, head_v, S}), f32); + + auto gdn = e.add_op("GGML_OP_GATED_DELTA_NET", + p + "gdn", + {q, k, v, g, beta, ss}, + ps({1, 1, T + head_v, head_v * H_v}), + f32, + 0, + {{"gdn_state_slots", int64_t{1}}}); + + // The op packs [attn rows | new-state rows]; op_case 4 slices them apart. The attn view's + // token axis is marked -1 explicitly rather than left as the representative T: the consumer + // replaces the FIRST dim equal to the token count, and with T == 1 that would match the + // leading batch dim and move the tokens into dim 0 for the rest of the model. + const std::vector attn_view{0, head_v}; + const std::vector state_view{1, head_v}; + auto attn = e.add_op("GGML_OP_VIEW", + p + "gdn_attn", + {gdn}, + ps({1, T, H_v, head_v}), + f32, + 4, + {{"gdn_view", attn_view}, {"view_reshape", std::vector{1, -1, H_v, head_v}}}); + auto new_state = e.add_op("GGML_OP_VIEW", + ss + "_out", + {gdn}, + ps({1, H_v, head_v, S}), + f32, + 4, + {{"gdn_view", state_view}, {"view_reshape", std::vector{1, H_v, head_v, S}}}); + graph.model_output_names.push_back(new_state); + graph.recurrent_states.emplace_back(ss, new_state); + + // ---- gated output norm + projection ---- + // build_norm_gated: rms_norm(attn, ssm_norm) * silu(z), normalizing the head_v axis. + auto out = rms_norm(e, attn, p + "ssm_norm.weight", p + "gdn_norm", cfg.rms_eps); + auto z_4d = e.add_op("GGML_OP_RESHAPE", p + "z_4d", {z}, ps({1, T, H_v, head_v}), f32, 1); + auto z_silu = e.add_op("GGML_UNARY_OP_SILU", p + "z_silu", {z_4d}, ps({1, T, H_v, head_v}), f32); + out = e.add_op("GGML_OP_MUL", p + "gdn_gated", {out, z_silu}, ps({1, T, H_v, head_v}), f32); + out = e.add_op("GGML_OP_RESHAPE", p + "gdn_merged", {out}, ps({1, 1, T, value_dim}), f32, 2); + + e.add_weight(p + "ssm_out.weight"); + return e.add_op("GGML_OP_MUL_MAT", + p + "linear_attn_out", + {p + "ssm_out.weight", out}, + ps({1, 1, T, cfg.n_embd}), + f32); +} + +} // namespace blocks +} // namespace gguf +} // namespace frontend +} // namespace ov diff --git a/src/frontends/gguf/src/builder/blocks/gated_delta_net.hpp b/src/frontends/gguf/src/builder/blocks/gated_delta_net.hpp new file mode 100644 index 00000000000000..bd5d45d45c2815 --- /dev/null +++ b/src/frontends/gguf/src/builder/blocks/gated_delta_net.hpp @@ -0,0 +1,31 @@ +// Copyright (C) 2018-2026 Intel Corporation +// SPDX-License-Identifier: Apache-2.0 +// + +#pragma once + +#include + +#include "builder/decoder_config.hpp" +#include "builder/graph_emitter.hpp" + +namespace ov { +namespace frontend { +namespace gguf { +namespace blocks { + +// qwen35 linear-attention (Gated DeltaNet) layer. +// Mirrors llama.cpp src/models/qwen35.cpp build_layer_attn_linear + delta-net-base.cpp. +// `attn_norm` is the pre-attention normed hidden; returns the sublayer output BEFORE the residual +// add, i.e. exactly what the shared decoder tail expects in place of attn_out. +// +// The two recurrent states are plain model Parameters written through to Results, matching how the +// builder treats KV caches: the frontend always emits a STATELESS graph and leaves statefulness to +// the consumer. Unlike a KV cache these are OVERWRITTEN, not appended, so they carry no token axis +// and MakeStateful's Concat path does not apply to them. +std::string gated_delta_net(GraphEmitter& e, const DecoderConfig& cfg, int il, const std::string& attn_norm, int64_t T); + +} // namespace blocks +} // namespace gguf +} // namespace frontend +} // namespace ov diff --git a/src/frontends/gguf/src/builder/blocks/qkv_repack.cpp b/src/frontends/gguf/src/builder/blocks/qkv_repack.cpp new file mode 100644 index 00000000000000..0743ef5c094c80 --- /dev/null +++ b/src/frontends/gguf/src/builder/blocks/qkv_repack.cpp @@ -0,0 +1,62 @@ +// Copyright (C) 2018-2026 Intel Corporation +// SPDX-License-Identifier: Apache-2.0 +// + +#include "builder/blocks/qkv_repack.hpp" + +#include +#include + +#include "quant/weights.hpp" + +namespace ov { +namespace frontend { +namespace gguf { +namespace blocks { + +namespace { + +// Re-key the sliced tensors from "..weight" to ".weight" so +// emit_weight_op's sub-key extraction (weight/scales/zp) and translate_weight's make_weight_node +// see the node's own base name. +std::unordered_map rekey(const std::unordered_map& extracted, + const std::string& dst_base) { + std::unordered_map out; + for (const auto& kv : extracted) { + auto dot = kv.first.rfind('.'); + out[dst_base + "." + kv.first.substr(dot + 1)] = kv.second; + } + return out; +} + +} // namespace + +void register_fused_qkv(GraphEmitter& e, const DecoderConfig& cfg, int il) { + const std::string p = "blk." + std::to_string(il) + "."; + const size_t n_q = static_cast(cfg.n_head) * cfg.head_size; + const size_t n_kv = static_cast(cfg.n_head_kv) * cfg.head_size; + auto parts = split_fused_qkv_extracted(p + "attn_qkv", e.weights(), e.qtypes(), n_q, n_kv, n_kv); + const std::array names = {p + "attn_q.weight", p + "attn_k.weight", p + "attn_v.weight"}; + const std::array rows = {(int64_t)n_q, (int64_t)n_kv, (int64_t)n_kv}; + for (size_t i = 0; i < 3; ++i) { + const std::string base = p + "attn_" + std::string(1, "qkv"[i]); // blk.N.attn_q etc. + e.emit_weight_op(names[i], rekey(parts[i].extracted, base), parts[i].qtype, ps({1, 1, rows[i], cfg.n_embd})); + } +} + +void register_qwen35_q_gate(GraphEmitter& e, const DecoderConfig& cfg, int il) { + const std::string p = "blk." + std::to_string(il) + "."; + auto parts = split_interleaved_q_gate(p + "attn_q", e.weights(), e.qtypes(), static_cast(cfg.head_size)); + const std::array names = {p + "attn_q.weight", p + "attn_gate.weight"}; + const std::array suffix = {"q", "gate"}; + const int64_t rows = static_cast(cfg.n_head) * cfg.head_size; + for (size_t i = 0; i < 2; ++i) { + const std::string base = p + "attn_" + suffix[i]; + e.emit_weight_op(names[i], rekey(parts[i].extracted, base), parts[i].qtype, ps({1, 1, rows, cfg.n_embd})); + } +} + +} // namespace blocks +} // namespace gguf +} // namespace frontend +} // namespace ov diff --git a/src/frontends/gguf/src/builder/blocks/qkv_repack.hpp b/src/frontends/gguf/src/builder/blocks/qkv_repack.hpp new file mode 100644 index 00000000000000..c2aece9e6ece19 --- /dev/null +++ b/src/frontends/gguf/src/builder/blocks/qkv_repack.hpp @@ -0,0 +1,35 @@ +// Copyright (C) 2018-2026 Intel Corporation +// SPDX-License-Identifier: Apache-2.0 +// + +#pragma once + +#include + +#include "builder/decoder_config.hpp" +#include "builder/graph_emitter.hpp" + +namespace ov { +namespace frontend { +namespace gguf { +namespace blocks { + +// Weight-level repacking that happens before any op is emitted for a layer, so the rest of the +// attention path only ever sees the plain blk..attn_{q,k,v}.weight / attn_gate.weight nodes and +// stays architecture-agnostic. + +// Split a fused blk..attn_qkv weight into separate q/k/v weight nodes registered under +// blk..attn_{q,k,v}.weight. Rows split as [n_head*head_size | n_head_kv*head_size x2]. +void register_fused_qkv(GraphEmitter& e, const DecoderConfig& cfg, int il); + +// qwen35 full-attention layers: attn_q packs query and attention-output gate interleaved per head +// ([q_h0 | gate_h0 | q_h1 | ...], stride 2*head_size). De-interleave into the plain +// blk.N.attn_q.weight / blk.N.attn_gate.weight the shared attention path expects, so the graph +// never sees the interleaving. llama.cpp does the same split with two strided views +// (src/models/qwen35.cpp build_layer_attn). +void register_qwen35_q_gate(GraphEmitter& e, const DecoderConfig& cfg, int il); + +} // namespace blocks +} // namespace gguf +} // namespace frontend +} // namespace ov diff --git a/src/frontends/gguf/src/builder/decoder_config.cpp b/src/frontends/gguf/src/builder/decoder_config.cpp new file mode 100644 index 00000000000000..175dcb06fd1caf --- /dev/null +++ b/src/frontends/gguf/src/builder/decoder_config.cpp @@ -0,0 +1,301 @@ +// Copyright (C) 2018-2026 Intel Corporation +// SPDX-License-Identifier: Apache-2.0 +// +// Architecture detection for the dense/MoE decoder family: turn the parsed GGUF metadata and +// tensor table into the flat DecoderConfig the topology builder reads. +// +// Ground truth for each rule is llama.cpp's per-architecture hparam loading +// (src/models/*.cpp::load_arch_hparams) and llm_graph_context. + +#include "decoder_config.hpp" + +#include +#include +#include + +#include "arch_registry.hpp" +#include "openvino/core/except.hpp" + +namespace ov { +namespace frontend { +namespace gguf { + +namespace { + +int cfg_int(const std::map& config, const std::string& k) { + auto it = config.find(k); + OPENVINO_ASSERT(it != config.end(), "[GGUF] internal: missing config key '", k, "'"); + const auto* v = std::get_if(&it->second); + OPENVINO_ASSERT(v, "[GGUF] internal: config key '", k, "' is not an int"); + return *v; +} + +float cfg_float(const std::map& config, const std::string& k) { + auto it = config.find(k); + OPENVINO_ASSERT(it != config.end(), "[GGUF] internal: missing config key '", k, "'"); + const auto* v = std::get_if(&it->second); + OPENVINO_ASSERT(v, "[GGUF] internal: config key '", k, "' is not a float"); + return *v; +} + +} // namespace + +DecoderConfig::DecoderConfig(const std::map& config, + const std::unordered_map& weights) { + const auto has = [&weights](const std::string& name) { + return weights.count(name) > 0; + }; + const auto cfg_i = [&config](const std::string& k) { + return cfg_int(config, k); + }; + const auto cfg_f = [&config](const std::string& k) { + return cfg_float(config, k); + }; + + n_layer = cfg_i("layer_num"); + n_head = cfg_i("head_num"); + n_head_kv = cfg_i("head_num_kv"); + head_size = cfg_i("head_size"); + n_embd = cfg_i("hidden_size"); + rms_eps = cfg_f("rms_norm_eps"); + if (config.count("head_num_kv_per_layer")) { + if (auto* t = std::get_if(&config.at("head_num_kv_per_layer"))) { + const auto* data = t->data(); + n_head_kv_per_layer.assign(data, data + t->get_size()); + } + } + + arch = std::get(config.at("architecture")); + + // Per-architecture structure, auto-detected from the GGUF tensor table (layer 0). + has_qk_norm = has("blk.0.attn_q_norm.weight"); + // q/k norm width: per-head (head_size, applied after reshape -> qwen3/hunyuan/gemma4) + // vs full projection width (n_head*head_size, applied before reshape -> OLMoE). + // For gemma4, norm width equals per-layer head_size (may differ by SWA/global); always + // per-head. For OLMoE, norm width equals n_head*head_size. Detect by checking against + // the global head_size; but for gemma4 override to per-head since it has mixed head sizes. + if (has_qk_norm) { + const size_t qn = weights.at("blk.0.attn_q_norm.weight").get_shape()[0]; + qk_norm_full = (arch != "gemma4") && (qn != static_cast(head_size)); + } + n_dense_lead = cfg_i("n_layer_dense_lead"); + // Detect MoE from the first non-dense-lead layer (layer 0 may be dense even in MoE models). + { + const int probe = std::max(0, n_dense_lead); + const std::string pp = "blk." + std::to_string(probe) + "."; + is_moe = has(pp + "ffn_gate_exps.weight"); + } + // Gemma/Gemma2 use GeGLU (GELU-gated FFN). Detected by arch name since other archs + // in the supported set (llama, qwen2, qwen3, phi3) all use SwiGLU. + is_geglu = arch == "gemma" || arch == "gemma2" || arch == "gemma3" || arch == "gemma4"; + // gemma4 only: V projection is also RMSNorm'd (no weight, just normalize). gemma3 has + // Q/K norm but NO V norm -- llama.cpp src/models/gemma3.cpp norms Qcur/Kcur only and + // sends Vcur straight to the cache (gemma4.cpp:227 adds the extra ggml_rms_norm on V). + has_v_norm = (arch == "gemma4"); + n_expert = cfg_i("expert_count"); + n_expert_used = cfg_i("expert_used_count"); + n_expert_shared = cfg_i("expert_shared_count"); + has_moe_gate_bias = has("blk.0.ffn_gate_inp.bias"); // gpt-oss + has_moe_expert_bias = has("blk.0.ffn_gate_exps.bias"); // gpt-oss + has_sinks = has("blk.0.attn_sinks.weight"); // gpt-oss + // Gemma2/Gemma4: per-layer post-attention and post-FFN RMSNorm applied after the sublayer + // output and before the residual add. Detected from the tensor table at layer 0. + // + // Key naming ambiguity across architectures: + // Gemma2/Gemma4: attn_norm (pre-attn) + ffn_norm (pre-FFN) + // + post_attention_norm (post-attn) + post_ffw_norm (post-FFN) + // gpt-oss: attn_norm (pre-attn) + post_attention_norm (pre-FFN, no ffn_norm!) + // exaone4: post_attention_norm (pre-attn!) + post_ffw_norm (pre-FFN!) — no attn_norm + // + // Rule: post_attention_norm is a true POST-attn norm only when both attn_norm.weight + // AND ffn_norm.weight also exist. Same for post_ffw_norm as a true POST-FFN norm. + { + const bool has_attn_norm_w = has("blk.0.attn_norm.weight"); + const bool has_ffn_norm_w = has("blk.0.ffn_norm.weight"); + has_attn_post_norm = has("blk.0.post_attention_norm.weight") && has_attn_norm_w && has_ffn_norm_w; + has_ffn_post_norm = has("blk.0.post_ffw_norm.weight") && has_ffn_norm_w; + + // Compute effective pre-attn and pre-FFN norm key suffixes. + // Standard: "attn_norm.weight" / "ffn_norm.weight". + // exaone4: "post_attention_norm.weight" / "post_ffw_norm.weight". + // gpt-oss: "attn_norm.weight" / "post_attention_norm.weight". + if (!has_attn_norm_w && has("blk.0.post_attention_norm.weight")) { + attn_norm_key = "post_attention_norm.weight"; // exaone4 + } + if (!has_ffn_norm_w) { + if (has("blk.0.post_ffw_norm.weight")) { + ffn_norm_key = "post_ffw_norm.weight"; // exaone4 + } else if (has("blk.0.post_attention_norm.weight")) { + ffn_norm_key = "post_attention_norm.weight"; // gpt-oss + } + } + } + // gpt-oss uses "softmax-after-topk" gating + the OAI gated activation; OLMoE uses + // softmax-before-topk + plain SwiGLU. Detect by weight-tensor presence so the logic + // extends to future architectures without touching this file. + moe_swiglu_oai = has_moe_gate_bias; // gate_inp bias is OAI-gating-specific + moe_softmax_weight = has_moe_gate_bias; // same tensor signals softmax-after-topk + // SWA is present if sinks (gpt-oss), per-layer flags (gemma4), or has_swa was set + // from the GGUF metadata (smollm3, exaone-moe, gemma3, and future archs that set + // attention.sliding_window_pattern / attention.sliding_window). + has_swa = has_sinks || (arch == "gemma3") || (arch == "gemma4") || cfg_i("has_swa") != 0; + has_fused_qkv = has("blk.0.attn_qkv.weight"); // phi-3, minicpm + has_qkv_bias = has("blk.0.attn_q.bias") || has("blk.0.attn_qkv.bias"); + has_attn_out_bias = has("blk.0.attn_output.bias"); + has_rope_freqs = has("rope_freqs.weight"); + // muse-glimmer: sigmoid output gate on the attention sublayer. The gate is projected + // from the SAME pre-attention normed hidden the Q/K/V projections consume, so it is + // wired inside the layer right before the output projection. + has_attn_gate = has("blk.0.attn_gate.weight"); + // muse-glimmer applies a weightless RMSNorm to the token embeddings before layer 0 + // (llama.cpp src/models/muse-glimmer.cpp: build_norm(inpL, nullptr, nullptr, RMS, -1)), + // ropes only its sliding-window layers (global layers are NoPE), and post-norms with a + // tighter eps (post_norm_eps = 1e-8) than the pre-norms use. + const bool is_muse_glimmer = arch == "muse-glimmer"; + scaleless_embd_norm = is_muse_glimmer; + rope_on_swa_only = is_muse_glimmer; + post_norm_eps = is_muse_glimmer ? 1e-8f : 0.0f; // 0 -> reuse rms_eps + + // ---- qwen35 (Qwen3.5/3.6): hybrid Gated-DeltaNet + full attention ---- + // Layers alternate: every full_attention_interval-th layer is full attention, the rest + // run a linear-attention (GDN) block. llama.cpp src/models/qwen35.cpp. + is_qwen35 = arch == "qwen35"; + ssm_conv_kernel = cfg_i("ssm_conv_kernel"); + ssm_state_size = cfg_i("ssm_state_size"); + ssm_group_count = cfg_i("ssm_group_count"); + ssm_dt_rank = cfg_i("ssm_time_step_rank"); + ssm_inner_size = cfg_i("ssm_inner_size"); + full_attn_interval = cfg_i("full_attention_interval"); + // NextN/MTP blocks live past the main stack and are not executed in a normal forward + // pass, so the layer loop must stop before them (llama.cpp runs 0..n_layer(), where + // n_layer() already excludes them; the GGUF block_count counts them in). + n_layer_nextn = cfg_i("nextn_predict_layers"); + if (config.count("recurrent_layer_flags")) { + if (const auto* v = std::get_if>(&config.at("recurrent_layer_flags"))) + recurrent_layer_flags = *v; + } + if (config.count("rope_sections")) { + if (const auto* v = std::get_if>(&config.at("rope_sections"))) + rope_sections = *v; + } + if (is_qwen35) { + // blk.0 is a RECURRENT layer whose attn_qkv.weight is the GDN q/k/v/z projection, not + // a fused attention QKV -- the generic probe above would mistake it for phi-3-style + // fused QKV and try to split it by attention head counts. Likewise blk.0's + // attn_gate.weight is the GDN output gate (z), not an attention gate. + has_fused_qkv = false; + // qwen35's full-attention layers DO have a sigmoid attention output gate, and it is + // the same construction muse-glimmer uses -- projected from the pre-attention normed + // hidden, sigmoid'd, multiplied into the merged attention output before wo. The only + // difference is where the weight lives: qwen35 interleaves it per head inside attn_q + // rather than storing a separate tensor, so register_qwen35_q_gate() de-interleaves it + // into blk.N.attn_q.weight + blk.N.attn_gate.weight and the shared path handles it. + has_attn_gate = true; + // Per-head QK-norm on the full-attention layers. Auto-detection probes blk.0, which is + // recurrent and carries no attn_q_norm, so set it explicitly. + has_qk_norm = true; + qk_norm_full = false; + // qwen35's pre-FFN norm key ("post_attention_norm.weight", HF's + // post_attention_layernorm) already falls out of the generic rule above: attn_norm + // exists, ffn_norm does not, which is the gpt-oss pattern. + // MTP/NextN blocks are stored past the main stack and are not part of a normal + // forward pass, so the layer loop must not walk into them. + n_layer -= n_layer_nextn; + OPENVINO_ASSERT(n_layer > 0, "[GGUF] qwen35: no trunk layers left after excluding NextN blocks"); + } + + // Per-architecture scalars from metadata (1.0 / 0.0 when absent -> no-op). + embedding_scale = cfg_f("embedding_scale"); + residual_scale = cfg_f("residual_scale"); + logit_scale = cfg_f("logit_scale"); + attention_scale = cfg_f("attention_scale"); // 0 -> 1/sqrt(head_size) + expert_weights_scale = cfg_f("expert_weights_scale"); // 0 -> 1.0 no-op + rope_freq_base_swa = cfg_f("rope_freq_base_swa"); + swa_layer_pattern = cfg_i("swa_layer_pattern"); + // Gemma4: per-layer SWA boolean flags (non-empty when swa_layer_pattern==0). + if (config.count("swa_layer_flags")) + swa_layer_flags = std::get>(config.at("swa_layer_flags")); + attn_soft_cap = cfg_f("attn_logit_softcapping"); // 0 -> no soft-cap + final_logit_soft_cap = cfg_f("final_logit_softcapping"); // 0 -> no soft-cap + // Gemma4: per-layer input embeddings and shared KV layers. + n_embd_per_layer = cfg_i("n_embd_per_layer"); + shared_kv_layers = cfg_i("shared_kv_layers"); + // Gemma4: SWA layers use a smaller head size than global attention layers. + head_size_swa = cfg_i("head_size_swa"); + rope_dim_swa = cfg_i("rope_dimension_count_swa"); + + // Interleaved M-RoPE (qwen35 / qwen3vl) is its own mode, so it must be decided here rather + // than in the per-arch block above -- this assignment runs later and would overwrite it. + rope_op_case = is_qwen35 ? ROPE_OP_CASE_IMROPE + : arch_uses_neox_rope(arch) ? ROPE_OP_CASE_NEOX + : ROPE_OP_CASE_NORMAL; + + rope_config.n_dims = cfg_i("rope_dimension_count"); + rope_config.n_ctx_orig = cfg_i("rope_n_ctx_orig"); + rope_config.freq_base = cfg_f("rope_freq_base"); + rope_config.freq_scale = cfg_f("rope_freq_scale"); + rope_config.ext_factor = cfg_f("rope_ext_factor"); + rope_config.attn_factor = 1.0f; + rope_config.beta_fast = 32.0f; + rope_config.beta_slow = 1.0f; + // Gemma4: separate rope config for SWA layers (different freq_base and n_dims). + rope_config_swa = rope_config; + rope_config_swa.freq_base = cfg_f("rope_freq_base_swa"); + rope_config_swa.n_dims = cfg_i("rope_dimension_count_swa"); + // Per-op sin/cos is required whenever SWA and global layers differ in any rope + // parameter that feeds the shared sin/cos table: n_dims (gemma4) OR freq_base (gemma3, + // whose SWA layers rope at 10000 vs the global 1000000). Without it every layer would + // share the global table and SWA layers would be roped wrong. + const bool swa_dims_differ = rope_dim_swa > 0 && rope_dim_swa != rope_config.n_dims; + const bool swa_freq_differs = rope_config_swa.freq_base != rope_config.freq_base; + if (has_swa && (swa_dims_differ || swa_freq_differs)) { + use_per_op_rope = true; + } + // M-RoPE: inp_pos carries 4 sections per token and only the first n_dims of each head + // are rotated (qwen35: head_size 256, rope.dimension_count 64). Build sin/cos per ROPE op + // rather than from the shared table, whose layout assumes the single-section full-head + // case (see RopeConfig::is_imrope / use_per_op_rope). + if (rope_op_case == ROPE_OP_CASE_IMROPE) { + rope_config.is_imrope = true; + use_per_op_rope = true; + } +} + +bool DecoderConfig::layer_is_swa(int il) const { + if (!swa_layer_flags.empty()) { + return il < static_cast(swa_layer_flags.size()) && swa_layer_flags[il] != 0; + } + return has_swa && swa_layer_pattern > 0 && (il % swa_layer_pattern < (swa_layer_pattern - 1)); +} + +int DecoderConfig::layer_head_size(int il) const { + return (layer_is_swa(il) && head_size_swa > 0) ? head_size_swa : head_size; +} + +int DecoderConfig::layer_n_head_kv(int il) const { + return (!n_head_kv_per_layer.empty() && il < static_cast(n_head_kv_per_layer.size())) + ? static_cast(n_head_kv_per_layer[il]) + : n_head_kv; +} + +float DecoderConfig::layer_kq_scale(int il) const { + return attention_scale != 0.0f ? attention_scale : 1.0f / std::sqrt(static_cast(layer_head_size(il))); +} + +RopeConfig DecoderConfig::layer_rope_config(int il) const { + return layer_is_swa(il) ? rope_config_swa : rope_config; +} + +bool DecoderConfig::is_recurrent_layer(int il) const { + if (!is_qwen35) { + return false; + } + if (il < static_cast(recurrent_layer_flags.size())) { + return recurrent_layer_flags[il] != 0; + } + return full_attn_interval > 0 && ((il + 1) % full_attn_interval != 0); +} + +} // namespace gguf +} // namespace frontend +} // namespace ov diff --git a/src/frontends/gguf/src/builder/decoder_config.hpp b/src/frontends/gguf/src/builder/decoder_config.hpp new file mode 100644 index 00000000000000..e50569b0a68a09 --- /dev/null +++ b/src/frontends/gguf/src/builder/decoder_config.hpp @@ -0,0 +1,148 @@ +// Copyright (C) 2018-2026 Intel Corporation +// SPDX-License-Identifier: Apache-2.0 +// + +#pragma once + +#include +#include +#include +#include + +#include "openvino/frontend/gguf/decoder.hpp" +#include "openvino/runtime/tensor.hpp" +#include "quant/gguf.hpp" + +namespace ov { +namespace frontend { +namespace gguf { + +// Everything the decoder topology needs to know about ONE model, resolved once up front. +// +// This is the GGUF frontend's equivalent of llama.cpp's llama_hparams plus its per-arch +// load_arch_hparams: a flat, already-decided description of the model, so the topology builder in +// arch/decoder_builder.cpp never re-derives anything and reads declaratively. +// +// Almost every field is DETECTED, not hard-coded per architecture: the presence of a layer-0 +// weight tensor decides whether the model has QK-norm, projection biases, fused QKV, MoE routing, +// attention sinks, post-norms and so on. That is what lets a new same-family architecture be +// enabled by adding its name to arch_registry.cpp and writing no code at all. Only a handful of +// genuinely tensor-table-ambiguous properties (GeGLU vs SwiGLU, gemma4's V-norm) fall back to an +// architecture-name check; prefer weight presence when adding a new one. +struct DecoderConfig { + // Resolve the whole description from the parsed metadata (already normalized by + // config_from_meta) and the parser's tensor table. + DecoderConfig(const std::map& config, + const std::unordered_map& weights); + + std::string arch; + + // ---- core dimensions ---- + int n_layer = 0; + int n_head = 0; + int n_head_kv = 0; + int head_size = 0; + int n_embd = 0; + float rms_eps = 0.0f; + std::vector n_head_kv_per_layer; // non-empty when head_count_kv varies by layer + + // ---- auto-detected per-architecture structure ---- + bool has_qk_norm = false; + bool qk_norm_full = false; // norm width is n_head*head_size (OLMoE) rather than per-head + bool has_qkv_bias = false; + bool has_attn_out_bias = false; + bool has_rope_freqs = false; + bool has_fused_qkv = false; + bool is_moe = false; + bool has_moe_gate_bias = false; + bool moe_softmax_weight = false; // softmax AFTER top-k (gpt-oss) rather than before + bool moe_swiglu_oai = false; // OAI gated activation (gpt-oss) + bool has_moe_expert_bias = false; + bool has_sinks = false; + bool has_swa = false; + bool has_attn_post_norm = false; + bool has_ffn_post_norm = false; + bool is_geglu = false; + bool has_v_norm = false; // gemma4: V is also RMSNorm'd like K + + // muse-glimmer specifics + bool has_attn_gate = false; // sigmoid gate multiplied into the attention output + bool scaleless_embd_norm = false; // weightless RMSNorm on the token embeddings + bool rope_on_swa_only = false; // global (non-SWA) layers are NoPE + float post_norm_eps = 0.0f; // eps for post-attn/post-FFN norms (0 -> rms_eps) + + // ---- qwen35 (hybrid Gated-DeltaNet + full attention) ---- + bool is_qwen35 = false; + int ssm_conv_kernel = 0; // d_conv + int ssm_state_size = 0; // head_k_dim / head_v_dim + int ssm_group_count = 0; // num_k_heads + int ssm_dt_rank = 0; // num_v_heads + int ssm_inner_size = 0; // d_inner = num_v_heads * head_v_dim + int full_attn_interval = 0; // full attention when (il + 1) % interval == 0 + int n_layer_nextn = 0; // trailing MTP blocks, not executed + std::vector rope_sections; // M-RoPE per-axis section widths + std::vector recurrent_layer_flags; // explicit per-layer recurrent flags + + // Norm key suffixes; overridden for archs that use non-standard naming (exaone4, gpt-oss). + std::string attn_norm_key{"attn_norm.weight"}; + std::string ffn_norm_key{"ffn_norm.weight"}; + + // ---- MoE / scalar metadata ---- + int n_expert = 0; + int n_expert_used = 0; + int n_dense_lead = 0; + int n_expert_shared = 0; + float embedding_scale = 1.0f; + float residual_scale = 1.0f; + float logit_scale = 1.0f; + float attention_scale = 0.0f; // 0 -> 1/sqrt(head_size) + float expert_weights_scale = 0.0f; // 0 -> 1.0 no-op + float attn_soft_cap = 0.0f; + float final_logit_soft_cap = 0.0f; + + // ---- SWA / gemma4 ---- + float rope_freq_base_swa = 0.0f; + int swa_layer_pattern = 2; + std::vector swa_layer_flags; // gemma4: per-layer SWA flags (1=SWA, 0=global) + int n_embd_per_layer = 0; // gemma4: per-layer embedding projection dimension + int shared_kv_layers = 0; // gemma4: N trailing layers that share KV from earlier layers + int head_size_swa = 0; // gemma4: head size for SWA layers (differs from global) + int rope_dim_swa = 0; // gemma4: rope dims for SWA layers + + // ---- RoPE ---- + int rope_op_case = 0; + RopeConfig rope_config{}; + RopeConfig rope_config_swa{}; + bool use_per_op_rope = false; + + // ---- Per-layer hyperparameter accessors ---- + // Centralize the per-layer derivations that vary by architecture (SWA layers, variable head + // counts). They are the single source of truth, so the topology loop stays declarative. + // If a new arch adds a per-layer dimension, extend these rather than inlining a ternary. + + // Whether attention layer `il` uses a sliding window. gemma4 carries an explicit per-layer + // boolean array; gpt-oss/gemma3/smollm3 use a period (il is SWA unless it is the last in each + // period, matching llama.cpp set_swa_pattern(period, dense_first=false)). + bool layer_is_swa(int il) const; + + // Head size for layer `il`. gemma4 SWA layers use a smaller head size than global layers. + int layer_head_size(int il) const; + + // KV head count for layer `il` (some archs, e.g. Deci-style, vary head_count_kv per layer). + int layer_n_head_kv(int il) const; + + // Attention (softmax) scale for layer `il`. An explicit metadata scale wins; otherwise use + // 1/sqrt(layer_head_size(il)) so SWA and global layers each get the head-size-correct scale. + float layer_kq_scale(int il) const; + + // RoPE config for layer `il` (gemma4 SWA layers rope with a different freq_base / n_dims). + RopeConfig layer_rope_config(int il) const; + + // True when layer `il` is a linear-attention (recurrent) layer. An explicit per-layer flag + // array wins over the interval rule, matching llama.cpp's load order. + bool is_recurrent_layer(int il) const; +}; + +} // namespace gguf +} // namespace frontend +} // namespace ov diff --git a/src/frontends/gguf/src/builder/gguf_builder.cpp b/src/frontends/gguf/src/builder/gguf_builder.cpp new file mode 100644 index 00000000000000..8a2392304e7fee --- /dev/null +++ b/src/frontends/gguf/src/builder/gguf_builder.cpp @@ -0,0 +1,105 @@ +// Copyright (C) 2018-2026 Intel Corporation +// SPDX-License-Identifier: Apache-2.0 +// +// Entry point of the native GGUF -> GgufGraph path: parse the container, decide which model FAMILY +// the file holds, and hand it to that family's ModelBuilder. No llama.cpp / gguf dependency. +// +// The emitted nodes use the GGML op vocabulary and reproduce llama.cpp's cgraph topology, so the +// resulting GgufGraph drives the same op translators as the llama.cpp cgraph path. +// +// Layering (see docs/adding_an_architecture.md): +// graph_emitter.hpp arch-agnostic "how do I write a node" +// blocks/ reusable graph fragments (norm, ffn, attention, gated delta net) +// decoder_config.hpp all per-architecture detection for the decoder family +// arch/decoder_builder the decoder family's topology +// arch_registry.hpp which architectures are accepted, and their RoPE mode +// model_kind.hpp which family a file belongs to +// +// Adding an ARCHITECTURE of an existing family is a name in arch_registry.cpp. +// Adding a FAMILY (mmproj vision/audio, encoder-decoder) is a new ModelBuilder subclass plus a +// branch in build_ggml_graph_from_gguf below; nothing existing changes. + +#include "gguf_builder.hpp" + +#include +#include + +#include "builder/arch/decoder_builder.hpp" +#include "builder/arch_registry.hpp" +#include "builder/model_builder.hpp" +#include "builder/model_kind.hpp" +#include "gguf_graph.hpp" +#include "openvino/core/except.hpp" +#include "openvino/util/log.hpp" +#include "quant/gguf.hpp" + +namespace ov { +namespace frontend { +namespace gguf { + +namespace { + +// Pull the `tokenizer.*` ggml metadata into an ov::AnyMap keyed by the sub-key after the last +// dot (e.g. "tokenizer.ggml.tokens" -> "tokens", "tokenizer.chat_template" -> "chat_template"). +// Each GGUF metadata variant is mapped to the ov::Any types a downstream tokenizer builder +// consumes: std::string / std::vector / ov::Tensor (arrays and shape-{} scalars). +ov::AnyMap extract_tokenizer_config(const std::unordered_map& metadata) { + const std::string prefix = "tokenizer."; + ov::AnyMap cfg; + for (const auto& [key, value] : metadata) { + if (key.compare(0, prefix.size(), prefix) != 0) { + continue; + } + const auto sub_key = key.substr(key.find_last_of('.') + 1); + std::visit( + [&](const auto& v) { + using T = std::decay_t; + if constexpr (std::is_same_v) { + // skip empty + } else { + cfg[sub_key] = v; + } + }, + value); + } + return cfg; +} + +} // namespace + +std::shared_ptr build_ggml_graph_from_gguf(const std::string& file) { + auto [metadata, weights, qtypes, mmap, quant_buf] = get_gguf_data(file); + + // Decide the family FIRST: the metadata key layout differs per family, so reading any + // decoder hyperparameter before this point would misreport an mmproj file as a broken LLM. + const ModelKind kind = detect_model_kind(metadata); + OPENVINO_ASSERT(kind == ModelKind::Decoder, + "[GGUF] this file holds a ", + model_kind_name(kind), + " model; the native GGUF builder currently implements the decoder family only. " + "Support is added by implementing a ModelBuilder subclass for that family (see " + "builder/model_builder.hpp)."); + + auto config = decoder_config_from_meta(metadata); + + const std::string arch = std::get(config.at("architecture")); + OPENVINO_ASSERT(supported_archs().count(arch), + "[GGUF] native GGUF builder does not support architecture '", + arch, + "'. See supported_archs() in builder/arch_registry.cpp for the full list."); + if (experimental_archs().count(arch)) { + OPENVINO_WARN("[GGUF] architecture '", + arch, + "' is experimental: it is built by structural auto-detection but has not been " + "end-to-end verified against a reference. Validate accuracy before relying on it."); + } + + std::unique_ptr builder = std::make_unique(config, weights, qtypes); + auto graph = builder->build(); + graph->tokenizer_config = extract_tokenizer_config(metadata); + return graph; +} + +} // namespace gguf +} // namespace frontend +} // namespace ov diff --git a/src/frontends/gguf/src/builder/gguf_builder.hpp b/src/frontends/gguf/src/builder/gguf_builder.hpp new file mode 100644 index 00000000000000..eb5a62209fe27a --- /dev/null +++ b/src/frontends/gguf/src/builder/gguf_builder.hpp @@ -0,0 +1,24 @@ +// Copyright (C) 2018-2026 Intel Corporation +// SPDX-License-Identifier: Apache-2.0 +// + +#pragma once + +#include +#include + +namespace ov { +namespace frontend { +namespace gguf { + +struct GgufGraph; // defined in gguf_graph.hpp; only used here as a shared_ptr return type + +// Build a GgufGraph natively from a .gguf file (no llama.cpp / gguf dependency). +// Parses the container, then dispatches to a per-architecture builder that emits nodes in +// the GGML op vocabulary reproducing llama.cpp's cgraph topology for that architecture. +// Throws if the architecture is not supported natively. +std::shared_ptr build_ggml_graph_from_gguf(const std::string& file); + +} // namespace gguf +} // namespace frontend +} // namespace ov diff --git a/src/frontends/gguf/src/builder/gguf_builder_decoder.cpp b/src/frontends/gguf/src/builder/gguf_builder_decoder.cpp new file mode 100644 index 00000000000000..a8c02394d30e2b --- /dev/null +++ b/src/frontends/gguf/src/builder/gguf_builder_decoder.cpp @@ -0,0 +1,198 @@ +// Copyright (C) 2018-2026 Intel Corporation +// SPDX-License-Identifier: Apache-2.0 +// + +#include "gguf_builder_decoder.hpp" + +#include + +#include "openvino/core/except.hpp" + +namespace ov { +namespace frontend { +namespace gguf { + +GgufBuilderDecoder::GgufBuilderDecoder(std::shared_ptr graph) : m_graph(std::move(graph)) {} + +const GgufOp& GgufBuilderDecoder::node() const { + OPENVINO_ASSERT(m_node_idx >= 0 && static_cast(m_node_idx) < m_graph->nodes.size(), + "[gguf] node index out of range: ", + m_node_idx); + return m_graph->nodes[m_node_idx]; +} + +// ---- Per-node typed attribute ---- +// +// In addition to keys stored in GgufOp::attributes, the following reserved keys are +// served so external converters can access per-input/output metadata through the public +// base NodeContext::get_attribute() interface without including internal headers: +// +// "input_shape[N]" -> ov::PartialShape for input N (0-based) +// "input_type[N]" -> ov::element::Type for input N +// "input_stride[N]" -> std::vector for input N +// "input_view_offset[N]" -> int64_t for input N +// "output_shape" -> ov::PartialShape of the node output +// "output_type" -> ov::element::Type of the node output +// "rope_config" -> RopeConfig (model-scope RoPE config; see get_attribute below) + +static bool parse_indexed_key(const std::string& name, const std::string& prefix, size_t& out_idx) { + if (name.size() <= prefix.size() + 2) + return false; + if (name.compare(0, prefix.size(), prefix) != 0) + return false; + if (name[prefix.size()] != '[' || name.back() != ']') + return false; + try { + out_idx = static_cast(std::stoul(name.substr(prefix.size() + 1, name.size() - prefix.size() - 2))); + return true; + } catch (...) { + return false; + } +} + +ov::Any GgufBuilderDecoder::get_attribute(const std::string& name) const { + // RoPE config is queried at model scope (TranslateSession::preprocess, to build the shared + // sin/cos table) and at node scope (each ROPE op's own config). At MODEL scope (no bound node) + // expose the graph's config with per_op / n_dims==0 encoding "no shared table". At NODE scope + // fall through to the node's own "rope_config" attribute -- the builder stores a per-node + // config on each ROPE op (e.g. gemma4 SWA layers use a different freq_base / n_dims), so the + // node value must win over the graph default. + if (name == "rope_config" && m_node_idx < 0) { + RopeConfig cfg = m_graph->rope_config; + cfg.per_op = m_graph->use_per_op_rope; + if (!m_graph->has_rope) { + cfg.n_dims = 0; + } + return cfg; + } + + const auto& n = node(); + + // Reserved keys for per-input metadata + size_t idx = 0; + if (parse_indexed_key(name, "input_shape", idx)) { + if (idx < n.input_names.size()) { + auto it = n.input_shapes.find(n.input_names[idx]); + if (it != n.input_shapes.end()) + return it->second; + } + return {}; + } + if (parse_indexed_key(name, "input_type", idx)) { + if (idx < n.input_names.size()) { + auto it = n.input_types.find(n.input_names[idx]); + if (it != n.input_types.end()) + return it->second; + } + return {}; + } + if (parse_indexed_key(name, "input_stride", idx)) { + if (idx < n.input_names.size()) { + auto it = n.input_strides.find(n.input_names[idx]); + if (it != n.input_strides.end()) + return it->second; + } + return {}; + } + if (parse_indexed_key(name, "input_view_offset", idx)) { + if (idx < n.input_names.size()) { + auto it = n.input_view_offsets.find(n.input_names[idx]); + return it != n.input_view_offsets.end() ? ov::Any(it->second) : ov::Any(int64_t{0}); + } + return {}; + } + + // Reserved keys for per-output metadata + if (name == "output_shape") + return n.output_shape; + if (name == "output_type") + return n.output_type; + + // Per-node op case (the op translators read it via get_attribute("op_case", 0)). + if (name == "op_case") + return n.op_case; + + // Named op attributes + auto it = n.attributes.find(name); + return it != n.attributes.end() ? it->second : ov::Any{}; +} + +// ---- Per-input metadata ---- + +PartialShape GgufBuilderDecoder::get_input_shape(const std::string& name) const { + const auto& m = node().input_shapes; + auto it = m.find(name); + OPENVINO_ASSERT(it != m.end(), "[gguf] no input shape for '", name, "'"); + return it->second; +} + +int64_t GgufBuilderDecoder::get_input_view_element_offset(const std::string& name) const { + // The builder does not emit strided VIEW inputs (it materializes slices as explicit ops), so + // there is no view offset to convert; the stored offsets, when present, are already in + // elements. Return 0 when the input is not a view. + const auto& m = node().input_view_offsets; + auto it = m.find(name); + return it == m.end() ? 0 : it->second; +} + +size_t GgufBuilderDecoder::get_input_size() const { + return node().input_names.size(); +} + +std::vector GgufBuilderDecoder::get_input_names() const { + return node().input_names; +} + +// ---- Per-node output metadata ---- + +PartialShape GgufBuilderDecoder::get_output_shape() const { + return node().output_shape; +} + +std::vector GgufBuilderDecoder::get_output_names() const { + return {node().output_name}; +} + +// ---- Op type / name ---- + +const std::string& GgufBuilderDecoder::get_op_type() const { + return node().op_type; +} + +const std::string& GgufBuilderDecoder::get_op_name() const { + return node().name; +} + +void GgufBuilderDecoder::visit_subgraph(std::function)> node_visitor) const { + for (size_t i = 0; i < m_graph->nodes.size(); i++) { + auto per_node = std::make_shared(*this); + per_node->m_node_idx = static_cast(i); + node_visitor(per_node); + } +} + +// ---- Model-level I/O ---- + +const std::map>& GgufBuilderDecoder::get_model_inputs() const { + return m_graph->model_inputs; +} + +const std::map>& GgufBuilderDecoder::get_model_extra_inputs() const { + return m_graph->model_extra_inputs; +} + +std::vector GgufBuilderDecoder::get_model_output_names() const { + return m_graph->model_output_names; +} + +const std::vector>& GgufBuilderDecoder::get_recurrent_states() const { + return m_graph->recurrent_states; +} + +const ov::AnyMap& GgufBuilderDecoder::get_tokenizer_config() const { + return m_graph->tokenizer_config; +} + +} // namespace gguf +} // namespace frontend +} // namespace ov diff --git a/src/frontends/gguf/src/builder/gguf_builder_decoder.hpp b/src/frontends/gguf/src/builder/gguf_builder_decoder.hpp new file mode 100644 index 00000000000000..8252236412eede --- /dev/null +++ b/src/frontends/gguf/src/builder/gguf_builder_decoder.hpp @@ -0,0 +1,59 @@ +// Copyright (C) 2018-2026 Intel Corporation +// SPDX-License-Identifier: Apache-2.0 +// + +#pragma once + +#include + +#include "gguf_graph.hpp" +#include "openvino/frontend/gguf/decoder.hpp" + +namespace ov { +namespace frontend { +namespace gguf { + +// GgufDecoder implementation over a GgufGraph built natively from a .gguf file (no +// llama.cpp / gguf dependency). It is the OpenVINO-side counterpart of llama.cpp's +// cgraph-backed GgmlOvDecoder: both feed the same op translators / TranslateSession. +// +// The whole-graph constructor wraps a complete GgufGraph. Following the node-scoped +// GgufDecoder contract, visit_subgraph hands the translator a per-node view by cloning +// this decoder with a fixed node index (m_node_idx); every per-node accessor then reads +// the node it is bound to, so no node index is threaded through the interface. +class GgufBuilderDecoder : public GgufDecoder { +public: + explicit GgufBuilderDecoder(std::shared_ptr graph); + + // Per-node accessors (bound to the node this decoder instance was cloned for). + ov::Any get_attribute(const std::string& name) const override; + PartialShape get_input_shape(const std::string& name) const override; + int64_t get_input_view_element_offset(const std::string& name) const override; + size_t get_input_size() const override; + std::vector get_input_names() const override; + PartialShape get_output_shape() const override; + std::vector get_output_names() const override; + const std::string& get_op_type() const override; + const std::string& get_op_name() const override; + + void visit_subgraph(std::function)> node_visitor) const override; + + // Model-level I/O. + const std::map>& get_model_inputs() const override; + const std::map>& get_model_extra_inputs() const override; + std::vector get_model_output_names() const override; + const std::vector>& get_recurrent_states() const override; + const ov::AnyMap& get_tokenizer_config() const override; + +private: + std::shared_ptr m_graph; + // Index of the node this decoder instance is bound to. -1 for the whole-graph decoder + // (only model-scope queries are valid on it). + int m_node_idx = -1; + + const GgufOp& node() const; +}; + +} // namespace gguf +} // namespace frontend +} // namespace ov diff --git a/src/frontends/gguf/src/builder/gguf_graph.hpp b/src/frontends/gguf/src/builder/gguf_graph.hpp new file mode 100644 index 00000000000000..d6cfa51b490684 --- /dev/null +++ b/src/frontends/gguf/src/builder/gguf_graph.hpp @@ -0,0 +1,84 @@ +// Copyright (C) 2018-2026 Intel Corporation +// SPDX-License-Identifier: Apache-2.0 +// + +#pragma once + +#include +#include +#include +#include + +#include "openvino/core/any.hpp" +#include "openvino/core/node.hpp" +#include "openvino/core/partial_shape.hpp" +#include "openvino/core/type/element_type.hpp" +#include "openvino/frontend/gguf/decoder.hpp" + +namespace ov { +namespace frontend { +namespace gguf { + +// One operation node in the GGUF-built graph, expressed in the GGML op vocabulary +// ("GGML_OP_MUL_MAT", "GGML_OP_ROPE", ...). It mirrors exactly what the GgufDecoder +// interface exposes per node, so GgufBuilderDecoder is a thin accessor over it. The arch +// builder fills these by construction, so all per-op parameters are typed attributes (no +// raw gguf op_params layout). +struct GgufOp { + std::string op_type; // e.g. "GGML_OP_MUL_MAT" + std::string name; // unique node/op name + std::vector input_names; // producer tensor names (weights / inputs / other nodes) + std::string output_name; // this node's output tensor name + ov::PartialShape output_shape; + ov::element::Type output_type = ov::element::dynamic; + int op_case = 0; + + // Per-input shape/stride/type and view-offset, keyed by input name. Populated for the + // inputs that translators query (shapes for MUL_MAT/RESHAPE, view offsets for VIEW). + std::map input_shapes; + std::map> input_strides; + std::map input_types; + std::map input_view_offsets; + + // Typed scalar/struct op attributes consumed by translators via get_attribute + // (e.g. "eps", "scale", "bias", "max_bias", "swapped", "rope_config"). + std::map attributes; +}; + +// The whole model as a flat, topologically-ordered list of GgufOp nodes plus the +// model-level I/O the decoder reports. Built by an architecture builder (e.g. qwen3) from +// a parsed GGUF file; consumed by GgufBuilderDecoder. +struct GgufGraph { + std::vector nodes; + + // Model inputs (Parameters) and extra inputs (e.g. attention_size; Parameter or Constant). + // Same semantics as the corresponding GgufDecoder getters. Weights are not here: they are + // emitted into `nodes` as GGML_OP_NONE leaves (see emit_weight_op). + std::map> model_inputs; + std::map> model_extra_inputs; + std::vector model_output_names; + + // Recurrent (overwritten, non-appending) states as {input name, output name} pairs; see + // GgufDecoder::get_recurrent_states. Empty for every architecture without linear attention. + std::vector> recurrent_states; + + bool has_rope = false; + RopeConfig rope_config; + + // When true, each ROPE op builds its own sin/cos table from its per-op rope_config + // (useful when different layers need different n_dims, e.g. gemma4 SWA vs global). + // TranslateSession::add_rope_sin_cos skips the shared table when this is set. + bool use_per_op_rope = false; + + // GGUF tokenizer metadata (the `tokenizer.*` keys), keyed by the sub-key after the last + // dot (e.g. "model", "tokens", "merges", "scores", "token_type", "pre", "bos_token_id", + // "chat_template"). Values are std::string / std::vector / ov::Tensor, + // mirroring the GGUF metadata variant. Attached to the model's (non-serializable) rt_info + // by TranslateSession so a downstream consumer can build the tokenizer without re-reading + // the .gguf. Empty if the file carries no tokenizer metadata. + ov::AnyMap tokenizer_config; +}; + +} // namespace gguf +} // namespace frontend +} // namespace ov diff --git a/src/frontends/gguf/src/builder/graph_emitter.cpp b/src/frontends/gguf/src/builder/graph_emitter.cpp new file mode 100644 index 00000000000000..333422c95603cb --- /dev/null +++ b/src/frontends/gguf/src/builder/graph_emitter.cpp @@ -0,0 +1,240 @@ +// Copyright (C) 2018-2026 Intel Corporation +// SPDX-License-Identifier: Apache-2.0 +// + +#include "graph_emitter.hpp" + +#include "openvino/core/except.hpp" +#include "openvino/op/constant.hpp" + +namespace ov { +namespace frontend { +namespace gguf { + +namespace { + +// Split ".weight" into ""; return the name unchanged when it does not end +// in ".weight" (biases and other plain tensors keep their full name as the base). +std::string strip_weight_suffix(const std::string& name) { + static const std::string suffix = ".weight"; + const bool ends_with_weight = + name.size() > suffix.size() && name.compare(name.size() - suffix.size(), suffix.size(), suffix) == 0; + return ends_with_weight ? name.substr(0, name.size() - suffix.size()) : name; +} + +} // namespace + +GraphEmitter::GraphEmitter(std::unordered_map& weights, + std::unordered_map& qtypes, + std::string arch) + : m_weights(weights), + m_qtypes(qtypes), + m_arch(std::move(arch)), + m_graph(std::make_shared()) {} + +const ov::Tensor& GraphEmitter::weight_tensor(const std::string& name) const { + auto it = m_weights.find(name); + OPENVINO_ASSERT(it != m_weights.end(), + "[GGUF] model is missing expected weight tensor '", + name, + "' for architecture '", + m_arch, + "'"); + return it->second; +} + +int64_t GraphEmitter::weight_rows(const std::string& name) const { + auto it = m_weights.find(name); + if (it == m_weights.end()) { + return 1; + } + const auto& s = it->second.get_shape(); + return s.empty() ? 1 : static_cast(s[0]); +} + +const ov::PartialShape& GraphEmitter::shape_of_tensor(const std::string& name) const { + auto it = m_tensor_shapes.find(name); + OPENVINO_ASSERT(it != m_tensor_shapes.end(), "[GGUF] internal: no shape recorded for '", name, "'"); + return it->second; +} + +ov::Shape GraphEmitter::static_shape_of(const std::string& tensor_name) const { + const auto& shape = shape_of_tensor(tensor_name); + OPENVINO_ASSERT(shape.is_static(), + "[GGUF] internal: shape of '", + tensor_name, + "' is dynamic (", + shape, + "), cannot be used as a static input shape"); + return shape.to_shape(); +} + +void GraphEmitter::set_tensor_meta(const std::string& name, const ov::PartialShape& shape, ov::element::Type type) { + m_tensor_shapes[name] = shape; + m_tensor_types[name] = type; +} + +std::string GraphEmitter::add_op(const std::string& op_type, + const std::string& name, + const std::vector& inputs, + const ov::PartialShape& out_shape, + ov::element::Type out_type, + int op_case, + std::map attrs) { + GgufOp op; + op.op_type = op_type; + op.name = name; + op.input_names = inputs; + op.output_name = name; + op.output_shape = out_shape; + op.output_type = out_type; + op.op_case = op_case; + op.attributes = std::move(attrs); + // Fill per-input shape/type from known producers so translators that query them + // (MUL_MAT, RESHAPE) get sane values. + for (const auto& in : inputs) { + if (auto it = m_tensor_shapes.find(in); it != m_tensor_shapes.end()) { + op.input_shapes[in] = it->second; + } + if (auto it = m_tensor_types.find(in); it != m_tensor_types.end()) { + op.input_types[in] = it->second; + } + } + m_tensor_shapes[name] = out_shape; + m_tensor_types[name] = out_type; + m_graph->nodes.push_back(std::move(op)); + return name; +} + +std::shared_ptr GraphEmitter::add_input(const std::string& name, + ov::element::Type type, + const ov::PartialShape& shape) { + auto p = std::make_shared(type, shape); + p->set_friendly_name(name); + p->output(0).set_names({name}); + m_graph->model_inputs[name] = p; + return p; +} + +void GraphEmitter::add_extra_input(const std::string& name, int64_t value) { + auto c = ov::op::v0::Constant::create(ov::element::i64, ov::Shape{1}, {value}); + c->set_friendly_name(name); + m_graph->model_extra_inputs[name] = c; +} + +void GraphEmitter::add_extra_input_node(const std::string& name, const std::shared_ptr& node) { + m_graph->model_extra_inputs[name] = node; +} + +void GraphEmitter::emit_weight_op(const std::string& node_name, + const std::unordered_map& extracted, + gguf_tensor_type qtype, + const ov::PartialShape& shape_4d) { + if (m_emitted_weights.count(node_name)) { + return; + } + m_emitted_weights.insert(node_name); + + GgufOp op; + op.op_type = "GGML_OP_NONE"; + op.name = node_name; + op.output_name = node_name; + op.output_shape = shape_4d; + op.output_type = ov::element::f32; + // translate_weight rebuilds the make_weight_node(base, weights, qtypes) inputs from these: + // "gguf.blob." -> extracted tensor ( in {weight, scales, zp}), and the qtype id. + op.attributes["gguf_weight"] = true; // marks this GGML_OP_NONE leaf as a weight + op.attributes["gguf_qtype"] = static_cast(qtype); + for (const auto& kv : extracted) { + const std::string& full = kv.first; // ".weight" / ".scales" / ".zp" + auto dot = full.rfind('.'); + std::string sub = (dot == std::string::npos) ? full : full.substr(dot + 1); + op.attributes["gguf.blob." + sub] = kv.second; + } + m_graph->nodes.push_back(std::move(op)); + + m_tensor_shapes[node_name] = shape_4d; + m_tensor_types[node_name] = ov::element::f32; +} + +void GraphEmitter::add_weight(const std::string& ggml_name) { + if (m_emitted_weights.count(ggml_name)) { + return; + } + const std::string base = strip_weight_suffix(ggml_name); + + // Collect the parser's extracted tensors for this weight (weight [+ scales [+ zp]]). + std::unordered_map extracted; + for (const char* sub : {".weight", ".scales", ".zp"}) { + auto it = m_weights.find(base + sub); + if (it != m_weights.end()) { + extracted[base + sub] = it->second; + } + } + gguf_tensor_type qtype = GGUF_TYPE_F16; + if (auto it = m_qtypes.find(base + ".qtype"); it != m_qtypes.end()) { + qtype = it->second; + } + + // Shape padded to the decoder's rank-4 convention; the extents matter only as the + // per-input shape/type for translators (MUL_MAT) that index dims [1] and [3]. + int64_t rows = 1, cols = 1; + if (auto it = m_weights.find(ggml_name); it != m_weights.end()) { + const auto& s = it->second.get_shape(); // [rows, cols(packed)] + rows = s.size() >= 1 ? static_cast(s[0]) : 1; + cols = s.size() >= 2 ? static_cast(s[1]) : 1; + } + emit_weight_op(ggml_name, extracted, qtype, ov::PartialShape({1, 1, rows, cols})); +} + +void GraphEmitter::add_weight_from(const std::string& node_name, const std::string& src_base) { + if (m_emitted_weights.count(node_name)) { + return; + } + const std::string dst_base = strip_weight_suffix(node_name); + std::unordered_map extracted; + for (const char* sub : {".weight", ".scales", ".zp"}) { + auto it = m_weights.find(src_base + sub); + if (it != m_weights.end()) { + extracted[dst_base + sub] = it->second; + } + } + gguf_tensor_type qtype = GGUF_TYPE_F16; + if (auto it = m_qtypes.find(src_base + ".qtype"); it != m_qtypes.end()) { + qtype = it->second; + } + int64_t rows = 1, cols = 1; + if (auto it = m_weights.find(src_base + ".weight"); it != m_weights.end()) { + const auto& s = it->second.get_shape(); + rows = s.size() >= 1 ? static_cast(s[0]) : 1; + cols = s.size() >= 2 ? static_cast(s[1]) : 1; + } + emit_weight_op(node_name, extracted, qtype, ov::PartialShape({1, 1, rows, cols})); +} + +void GraphEmitter::add_named_weight(const std::string& ggml_name) { + if (m_emitted_weights.count(ggml_name)) { + return; + } + auto it = m_weights.find(ggml_name); + OPENVINO_ASSERT(it != m_weights.end(), "[GGUF] weight not found in gguf: ", ggml_name); + const ov::Tensor& w = it->second; + // Map the OV element type back to the ggml float qtype so translate_weight rebuilds a plain + // Constant of the right precision. + gguf_tensor_type qtype = w.get_element_type() == ov::element::f32 ? GGUF_TYPE_F32 + : w.get_element_type() == ov::element::bf16 ? GGUF_TYPE_BF16 + : GGUF_TYPE_F16; + // emit_weight_op re-keys by the last '.'; a bias ends in ".bias", so key it as ".weight" + // explicitly (make_weight_node's plain-Constant path reads ".weight"). + std::unordered_map extracted{{ggml_name + ".weight", w}}; + const auto& s = w.get_shape(); + int64_t n = s.empty() ? 1 : static_cast(s[0]); + // 2-D plain weights (qwen35's ssm_conv1d, OV [conv_dim, d_conv]) keep both extents; + // 1-D ones (biases, norm scales) are the common case and stay a trailing vector. + const ov::PartialShape shape = s.size() == 2 ? ps({1, 1, n, static_cast(s[1])}) : ps({1, 1, 1, n}); + emit_weight_op(ggml_name, extracted, qtype, shape); +} + +} // namespace gguf +} // namespace frontend +} // namespace ov diff --git a/src/frontends/gguf/src/builder/graph_emitter.hpp b/src/frontends/gguf/src/builder/graph_emitter.hpp new file mode 100644 index 00000000000000..ea18ed0b9754f7 --- /dev/null +++ b/src/frontends/gguf/src/builder/graph_emitter.hpp @@ -0,0 +1,164 @@ +// Copyright (C) 2018-2026 Intel Corporation +// SPDX-License-Identifier: Apache-2.0 +// + +#pragma once + +#include +#include +#include +#include +#include +#include + +#include "gguf_graph.hpp" +#include "openvino/core/any.hpp" +#include "openvino/core/partial_shape.hpp" +#include "openvino/core/type/element_type.hpp" +#include "openvino/op/parameter.hpp" +#include "openvino/runtime/tensor.hpp" +#include "quant/gguf.hpp" + +namespace ov { +namespace frontend { +namespace gguf { + +// Appends nodes to a GgufGraph and tracks the shape/type of every tensor emitted so far. +// +// This is the whole "how do I write a node" layer, and it knows nothing about transformers, +// attention or any architecture: it is the GGUF-frontend counterpart of ggml's ggml_context + +// ggml_build_forward_expand. Everything above it (the block library in blocks/, the per-family +// topology builders in arch/) is expressed purely in terms of these calls, which is what lets a +// non-decoder family (mmproj/vision, audio) reuse it verbatim. +// +// Owns the emit-time bookkeeping the graph itself does not carry: +// - tensor shapes/types, so add_op can fill each node's per-input shape/type metadata for the +// translators that query it (MUL_MAT, RESHAPE, VIEW); +// - the set of already-emitted weight leaves, so a weight referenced by several ops (or tied, +// e.g. MQA tie-V) is emitted exactly once. +class GraphEmitter { +public: + // `weights` / `qtypes` are the parser's tensor tables; they are referenced, not copied, and + // must outlive the emitter. + GraphEmitter(std::unordered_map& weights, + std::unordered_map& qtypes, + std::string arch); + + // ---- graph under construction ---- + const std::shared_ptr& graph() const { + return m_graph; + } + + // ---- weight-table queries ---- + bool has_weight(const std::string& name) const { + return m_weights.count(name) > 0; + } + + // Look up a weight tensor by GGUF name, failing with the tensor NAME if the GGUF is missing it + // (a bare .at(name) throws std::out_of_range with no context). Used wherever a builder reads a + // weight's shape to size an op; a missing expected tensor means the file does not match the + // detected architecture. + const ov::Tensor& weight_tensor(const std::string& name) const; + + // First extent of a weight's OV shape (its row count), 1 when the weight is absent/scalar. + int64_t weight_rows(const std::string& name) const; + + std::unordered_map& weights() { + return m_weights; + } + std::unordered_map& qtypes() { + return m_qtypes; + } + + // ---- emitted-tensor queries ---- + const ov::PartialShape& shape_of_tensor(const std::string& name) const; + + // Static shape of an already-emitted tensor, for ops whose translator needs its input's own + // layout (VIEW op_case 3's "input_ggml_shape", which it uses to restore that layout before + // slicing). Per-node shapes are static except for the model-input Parameters, so a dynamic dim + // here means the caller asked about a tensor this cannot describe. + ov::Shape static_shape_of(const std::string& tensor_name) const; + + // Record a shape/type for a tensor the emitter did not create itself (a model input reused as + // an op input at a representative token length). + void set_tensor_meta(const std::string& name, const ov::PartialShape& shape, ov::element::Type type); + + bool has_model_input(const std::string& name) const { + return m_graph->model_inputs.count(name) > 0; + } + + // ---- emission ---- + + // Append one op node. `inputs` are producer tensor names (weights / model inputs / earlier node + // outputs). Returns the output tensor name (== node name). + std::string add_op(const std::string& op_type, + const std::string& name, + const std::vector& inputs, + const ov::PartialShape& out_shape, + ov::element::Type out_type, + int op_case = 0, + std::map attrs = {}); + + std::shared_ptr add_input(const std::string& name, + ov::element::Type type, + const ov::PartialShape& shape); + + void add_extra_input(const std::string& name, int64_t value); + + void add_extra_input_node(const std::string& name, const std::shared_ptr& node); + + // Emit a weight as a GGML_OP_NONE leaf node carrying the parser's already-extracted tensors + // (`.weight` [+ `.scales` [+ `.zp`]] + qtype) as node attributes. translate_weight + // rebuilds the weights/qtypes maps from these attributes and calls make_weight_node(base, + // weights, qtypes). Routing weights through GGML_OP_NONE makes that the single weight-loading + // API, shared with the llama.cpp cgraph decoder (which marks the same leaf with raw ggml bytes + // instead), and keeps the compressed decompression subgraph -- built lazily in translate_weight + // during the walk -- rather than materializing an ov::Node eagerly here. + // `node_name` is the tensor name translators reference (the GGML_OP_NONE output); + // `extracted` maps ".weight"/".scales"/".zp" -> tensor; `qtype` is the ggml type. + void emit_weight_op(const std::string& node_name, + const std::unordered_map& extracted, + gguf_tensor_type qtype, + const ov::PartialShape& shape_4d); + + // `ggml_name` is the full tensor name ending in ".weight" (the name translators reference). + void add_weight(const std::string& ggml_name); + + // Emit a weight node `node_name` (ends in ".weight") reusing the parser's extracted tensors of + // another weight `src_base` (base without ".weight"). Used for MQA tie-V, where V shares K's + // weight tensor; the two GGML_OP_NONE leaves reference the same underlying ov::Tensor blobs + // (cheap: SharedBuffer views into the parser's single quant buffer). + void add_weight_from(const std::string& node_name, const std::string& src_base); + + // Emit a plain (non-quantized) weight stored under its full GGUF name, e.g. a bias tensor + // "blk.N.attn_q.bias" (no ".weight" suffix). It flows through the same GGML_OP_NONE + + // translate_weight path; make_weight_node treats an F16/F32/BF16 blob as a plain Constant. + void add_named_weight(const std::string& ggml_name); + + bool weight_emitted(const std::string& name) const { + return m_emitted_weights.count(name) > 0; + } + +private: + std::unordered_map& m_weights; + std::unordered_map& m_qtypes; + // Architecture name, used only to make a missing-tensor diagnostic actionable. + std::string m_arch; + + std::shared_ptr m_graph; + std::map m_tensor_shapes; + std::map m_tensor_types; + // Names of weights already emitted as GGML_OP_NONE leaves, so a weight referenced by several + // ops (or tied, e.g. MQA tie-V) is emitted once. + std::set m_emitted_weights; +}; + +// Shapes are kept in the OpenVINO/GGML logical order [ne3, ne2, ne1, ne0] (reverse of GGUF +// on-disk order), matching the decoder's get_shape(). The translators consume them as-is. +inline ov::PartialShape ps(std::vector dims) { + return ov::PartialShape(std::move(dims)); +} + +} // namespace gguf +} // namespace frontend +} // namespace ov diff --git a/src/frontends/gguf/src/builder/model_builder.hpp b/src/frontends/gguf/src/builder/model_builder.hpp new file mode 100644 index 00000000000000..1c1dd9e83e5533 --- /dev/null +++ b/src/frontends/gguf/src/builder/model_builder.hpp @@ -0,0 +1,40 @@ +// Copyright (C) 2018-2026 Intel Corporation +// SPDX-License-Identifier: Apache-2.0 +// + +#pragma once + +#include + +#include "builder/gguf_graph.hpp" + +namespace ov { +namespace frontend { +namespace gguf { + +// Base class for a whole-model graph builder: one subclass per MODEL FAMILY, not per architecture. +// +// A family is a distinct graph SHAPE with its own inputs, its own block vocabulary and its own +// notion of a "layer": the causal decoder stack (DecoderBuilder, arch/decoder_builder.hpp) is one; +// a vision/mmproj encoder or an audio encoder would each be another. Within a family, individual +// architectures are data, not code -- they are detected from the GGUF tensor table and metadata +// (see DecoderConfig) and enabled by name in arch_registry.cpp. +// +// This mirrors llama.cpp's split between llm_graph_context (LLMs) and clip_graph (mmproj), where +// each family has its own base and its own build_norm/build_ffn/build_attn vocabulary, rather than +// one builder growing flags for structurally unrelated models. +// +// To add a family: subclass this, implement build(), and dispatch to it from +// build_ggml_graph_from_gguf() on the detected ModelKind. Nothing in the existing families needs +// to change, and the arch-agnostic pieces (GraphEmitter, blocks/common) are reused as-is. +class ModelBuilder { +public: + virtual ~ModelBuilder() = default; + + // Emit the whole model and return the finished graph. + virtual std::shared_ptr build() = 0; +}; + +} // namespace gguf +} // namespace frontend +} // namespace ov diff --git a/src/frontends/gguf/src/builder/model_kind.cpp b/src/frontends/gguf/src/builder/model_kind.cpp new file mode 100644 index 00000000000000..8233c295ff5449 --- /dev/null +++ b/src/frontends/gguf/src/builder/model_kind.cpp @@ -0,0 +1,68 @@ +// Copyright (C) 2018-2026 Intel Corporation +// SPDX-License-Identifier: Apache-2.0 +// + +#include "builder/model_kind.hpp" + +#include + +namespace ov { +namespace frontend { +namespace gguf { + +namespace { + +// Read a GGUF bool-ish metadata key. The parser stores every scalar as a shape-{} ov::Tensor, so a +// GGUF_VALUE_TYPE_BOOL arrives as an element::boolean tensor; accept an integer tensor too, since +// some writers emit these flags as u8/u32. Absent or non-scalar means false. +bool meta_flag(const std::unordered_map& metadata, const std::string& key) { + auto it = metadata.find(key); + if (it == metadata.end()) { + return false; + } + const auto* t = std::get_if(&it->second); + if (!t || t->get_size() != 1) { + return false; + } + if (t->get_element_type() == ov::element::boolean) { + return *t->data() != 0; + } + if (t->get_element_type() == ov::element::u8) { + return *t->data() != 0; + } + if (t->get_element_type() == ov::element::u32) { + return *t->data() != 0; + } + return false; +} + +} // namespace + +ModelKind detect_model_kind(const std::unordered_map& metadata) { + // Check the clip.* flags before the architecture: an mmproj file's general.architecture is + // "clip", which is not an LLM architecture name, and reading any ".block_count"-style + // key on it would fail with a misleading "missing config key" error. + if (meta_flag(metadata, "clip.has_vision_encoder")) { + return ModelKind::Vision; + } + if (meta_flag(metadata, "clip.has_audio_encoder")) { + return ModelKind::Audio; + } + return ModelKind::Decoder; +} + +const char* model_kind_name(ModelKind kind) { + switch (kind) { + case ModelKind::Vision: + return "vision (mmproj)"; + case ModelKind::Audio: + return "audio (mmproj)"; + case ModelKind::Decoder: + default: + return "decoder"; + } +} + +} // namespace gguf +} // namespace frontend +} // namespace ov diff --git a/src/frontends/gguf/src/builder/model_kind.hpp b/src/frontends/gguf/src/builder/model_kind.hpp new file mode 100644 index 00000000000000..f543aa4b0aa179 --- /dev/null +++ b/src/frontends/gguf/src/builder/model_kind.hpp @@ -0,0 +1,42 @@ +// Copyright (C) 2018-2026 Intel Corporation +// SPDX-License-Identifier: Apache-2.0 +// + +#pragma once + +#include +#include + +#include "quant/gguf.hpp" + +namespace ov { +namespace frontend { +namespace gguf { + +// Which FAMILY of model a .gguf file holds. This is the first thing the builder decides, before +// any hyperparameter is read, because the metadata key layout itself differs per family: a causal +// decoder carries ".block_count" / ".attention.head_count", while an mmproj file +// carries "clip.*" keys and none of those. +// +// It is deliberately NOT the architecture. Architectures within a family (llama, qwen3, gemma4, +// ...) are data, detected from the tensor table; families are code, one ModelBuilder subclass +// each. +enum class ModelKind { + Decoder, // causal decoder-only LLM: the "llama family" (dense + MoE + hybrid recurrent) + Vision, // mmproj vision encoder + projector (clip.has_vision_encoder) + Audio, // mmproj audio encoder + projector (clip.has_audio_encoder) +}; + +// Classify a parsed GGUF file from its raw metadata. +// +// mmproj files written by llama.cpp set general.architecture = "clip" and carry +// clip.has_vision_encoder / clip.has_audio_encoder (see llama.cpp tools/mtmd/clip-impl.h +// KEY_HAS_VISION_ENC / KEY_HAS_AUDIO_ENC); every LLM GGUF names its own architecture instead. +ModelKind detect_model_kind(const std::unordered_map& metadata); + +// Human-readable name for a kind, used in diagnostics. +const char* model_kind_name(ModelKind kind); + +} // namespace gguf +} // namespace frontend +} // namespace ov diff --git a/src/frontends/gguf/src/frontend.cpp b/src/frontends/gguf/src/frontend.cpp index d501c01173610b..4089ea91455e3d 100644 --- a/src/frontends/gguf/src/frontend.cpp +++ b/src/frontends/gguf/src/frontend.cpp @@ -4,9 +4,15 @@ #include "openvino/frontend/gguf/frontend.hpp" +#include +#include + +#include "builder/gguf_builder.hpp" +#include "builder/gguf_builder_decoder.hpp" #include "input_model.hpp" #include "op_table.hpp" #include "openvino/core/so_extension.hpp" +#include "openvino/frontend/common/path_util.hpp" #include "openvino/frontend/extension/conversion.hpp" #include "openvino/frontend/extension/decoder_transformation.hpp" #include "openvino/frontend/extension/telemetry.hpp" @@ -18,12 +24,19 @@ namespace ov { namespace frontend { namespace gguf { -// Discoverability (intentional): consumed only by direct linkage -- a caller (the llama.cpp -// ggml-openvino backend, OpenVINO GenAI) links openvino::frontend::gguf and feeds FrontEnd a live -// GgufDecoder (no .gguf-path reader; see supported_impl / load_impl). It exports the standard -// plugin entry points so FrontEndManager can scan the frontend dir without error, but "gguf" is -// treated as hidden there (manager.cpp is_hidden_frontend), so it is never listed or auto-selected. -// Drop it from that list once this frontend gains file-based loading and passes production review. +// This frontend has two ingest paths, both converging on the same GgufDecoder + op translators: +// 1. a live GgufDecoder passed in by a direct linker (the llama.cpp ggml-openvino cgraph path); +// 2. a .gguf file path (the OpenVINO-native path): the frontend parses the container and builds +// the transformer graph per-architecture via the native builder (see load_impl Path 2). +// +// Discoverability: "gguf" is in manager.cpp's is_hidden_frontend list, so it is not advertised by +// available_front_ends() and not auto-selected by load_by_model -- core.read_model(".gguf") does +// not resolve to it. It is still reachable explicitly, either by direct linkage (what GenAI and +// the llama.cpp backend do) or by name via load_by_framework("gguf"). supported_impl below stays +// implemented, so enabling core.read_model later is just dropping the name from that list. +// +// Driving the frontend directly needs no follow-up pass: normalization runs inside convert(), and +// the only step read_model adds, update_v10_model(), fires solely for legacy IR v10. struct FrontEnd::Impl { std::unordered_map op_extension_translators; @@ -47,6 +60,17 @@ std::unordered_map merged_ops( return ops; } +// True if the file at `path` begins with the GGUF magic ("GGUF"). +bool has_gguf_magic(const std::filesystem::path& path) { + std::ifstream f(path, std::ios::binary); + if (!f) { + return false; + } + char magic[4] = {}; + f.read(magic, sizeof(magic)); + return f.gcount() == 4 && magic[0] == 'G' && magic[1] == 'G' && magic[2] == 'U' && magic[3] == 'F'; +} + } // namespace FrontEnd::FrontEnd() : m_impl(std::make_shared()) {} @@ -91,32 +115,56 @@ void FrontEnd::add_extension(const std::shared_ptr& extension) { } } -bool FrontEnd::supported_impl(const std::vector&) const { - // Always false: this frontend is never selected by FrontEndManager (load_by_model). It is used - // only via direct linkage -- a caller constructs FrontEnd and calls convert() with an - // InputModel built from a GgufDecoder -- which does not go through supported(). See the - // discoverability note at the top of this file. +bool FrontEnd::supported_impl(const std::vector& variants) const { + // Two accepted inputs: + // 1. a GgufDecoder (the llama.cpp cgraph path passes one in directly), or + // 2. a path to a .gguf file (the OpenVINO-native path; sniff the GGUF magic). + if (variants.empty()) { + return false; + } + if (variants[0].is>()) { + return true; + } + if (auto path = ov::frontend::get_path_from_any(variants[0])) { + std::filesystem::path model_path = std::move(*path); + return model_path.extension() == ".gguf" && has_gguf_magic(model_path); + } return false; } InputModel::Ptr FrontEnd::load_impl(const std::vector& variants) const { FRONT_END_GENERAL_CHECK(!variants.empty(), "GGUF Frontend requires at least one parameter in model representation."); - FRONT_END_GENERAL_CHECK(variants[0].is>(), - "GGUF Frontend supports loading from a GgufDecoder only."); - auto decoder = variants[0].as>(); - FRONT_END_GENERAL_CHECK(decoder, "Couldn't cast ov::Any to std::shared_ptr"); - return std::make_shared(decoder); + + // Path 1: a GgufDecoder passed in directly (e.g. llama.cpp cgraph decoder). + if (variants[0].is>()) { + auto decoder = variants[0].as>(); + FRONT_END_GENERAL_CHECK(decoder, "Couldn't cast ov::Any to std::shared_ptr"); + return std::make_shared(decoder); + } + + // Path 2: a .gguf file path -> native builder -> GgufBuilderDecoder. + if (auto path = ov::frontend::get_path_from_any(variants[0])) { + std::filesystem::path model_path = std::move(*path); + FRONT_END_GENERAL_CHECK(model_path.extension() == ".gguf", + "GGUF Frontend file loading expects a .gguf file, got: ", + model_path.string()); + auto graph = build_ggml_graph_from_gguf(model_path.string()); + auto decoder = std::make_shared(graph); + return std::make_shared(decoder); + } + + FRONT_END_GENERAL_CHECK(false, + "GGUF Frontend doesn't support the provided model representation. Provide a GgufDecoder " + "or a path to a .gguf file."); } } // namespace gguf } // namespace frontend } // namespace ov -// Plugin registration. The frontend is installed in the frontend directory, so it must export -// these entry points or FrontEndManager throws while scanning it. It registers as hidden (see the -// discoverability note above): FrontEndManager loads it without error but never lists or -// auto-selects it; only direct linkers use it. +// Plugin registration. Exports the standard entry points so FrontEndManager can load the library; +// selection is covered by the discoverability note at the top of this file. GGUF_FRONTEND_C_API ov::frontend::FrontEndVersion get_api_version() { return OV_FRONTEND_API_VERSION; } diff --git a/src/frontends/gguf/src/input_model.cpp b/src/frontends/gguf/src/input_model.cpp index 7969bd860dc8ce..f6a65289b0900b 100644 --- a/src/frontends/gguf/src/input_model.cpp +++ b/src/frontends/gguf/src/input_model.cpp @@ -20,6 +20,10 @@ std::vector InputModel::get_model_output_names() const { return m_decoder->get_model_output_names(); } +const std::vector>& InputModel::get_recurrent_states() const { + return m_decoder->get_recurrent_states(); +} + RopeConfig InputModel::get_rope_config() const { // A decoder bound to a full LLM graph exposes "rope_config"; a decoder wrapping a bare op / // small cgraph (the former "naive" path) has no such attribute. Return a default config @@ -34,6 +38,10 @@ void InputModel::visit_subgraph(const std::functionvisit_subgraph(node_visitor); } +const std::shared_ptr& InputModel::get_model_decoder() const { + return m_decoder; +} + } // namespace gguf } // namespace frontend } // namespace ov diff --git a/src/frontends/gguf/src/input_model.hpp b/src/frontends/gguf/src/input_model.hpp index 7a3aec81823a32..c5224c43b54da1 100644 --- a/src/frontends/gguf/src/input_model.hpp +++ b/src/frontends/gguf/src/input_model.hpp @@ -7,12 +7,12 @@ #include #include #include -#include "openvino/frontend/input_model.hpp" #include #include #include "openvino/frontend/gguf/decoder.hpp" #include "openvino/frontend/gguf/visibility.hpp" +#include "openvino/frontend/input_model.hpp" namespace ov::frontend::gguf { @@ -33,9 +33,15 @@ class GGUF_FRONTEND_API InputModel : public ov::frontend::InputModel { // Model-scope topology (forwarded to the underlying decoder's model-scope accessors). const std::map>& get_model_inputs() const; std::vector get_model_output_names() const; + const std::vector>& get_recurrent_states() const; RopeConfig get_rope_config() const; void visit_subgraph(const std::function)>& node_visitor) const; + // The underlying node-scoped decoder. TranslateSession uses it for the remaining model-scope + // questions that are only relevant on the native .gguf builder / stateful path (weights, + // extra inputs, KV param/result pairs, is_stateful / is_static, tokenizer metadata). + const std::shared_ptr& get_model_decoder() const; + private: std::shared_ptr m_decoder; }; diff --git a/src/frontends/gguf/src/node_context.hpp b/src/frontends/gguf/src/node_context.hpp index dc15ee70b79baf..5663fff6f20d1b 100644 --- a/src/frontends/gguf/src/node_context.hpp +++ b/src/frontends/gguf/src/node_context.hpp @@ -5,10 +5,10 @@ #pragma once #include -#include "openvino/frontend/node_context.hpp" #include #include "openvino/frontend/gguf/decoder.hpp" +#include "openvino/frontend/node_context.hpp" namespace ov::frontend::gguf { @@ -24,22 +24,38 @@ class NodeContext : public frontend::NodeContext { m_output_names = decoder->get_output_names(); } - size_t get_input_size() const override { - return m_decoder->get_input_size(); + const std::vector& get_input_names() const { + return m_input_names; } - int64_t get_input_view_element_offset(size_t index) const { - return m_decoder->get_input_view_element_offset(m_input_names[index]); + size_t get_input_size() const override { + return m_decoder->get_input_size(); } PartialShape get_input_shape(size_t input_index) const { return m_decoder->get_input_shape(m_input_names[input_index]); } + // Element offset of a VIEW input into a larger tensor (0 when not a view). The decoder + // already divides ggml's raw byte offset by element size, so translators work in elements. + int64_t get_input_view_element_offset(size_t index) const { + return m_decoder->get_input_view_element_offset(m_input_names[index]); + } + PartialShape get_output_shape() const { return m_decoder->get_output_shape(); } + // Convenience typed reads over get_attribute, kept so both the attribute-style op bodies and + // the accessor-style (op_case / output_type) op bodies compile against one NodeContext. + int get_op_case() const { + return get_attribute("op_case", 0); + } + + ov::element::Type get_output_type() const { + return get_attribute("output_type"); + } + Output get_input(int idx) const override { return m_tensor_map->at(m_input_names[idx]); } diff --git a/src/frontends/gguf/src/op/argsort.cpp b/src/frontends/gguf/src/op/argsort.cpp index c601238ffc5cdd..8470ae009cfbc6 100644 --- a/src/frontends/gguf/src/op/argsort.cpp +++ b/src/frontends/gguf/src/op/argsort.cpp @@ -44,15 +44,9 @@ OutputVector translate_argsort(const NodeContext& context) { const int64_t axis = in_ps.rank().is_static() ? in_ps.rank().get_length() - 1 : 3; auto k = std::make_shared(get_dimensions(input, {(int)axis}), ov::op::v0::Constant::create(ov::element::i64, {1}, {0})); - auto topk = std::make_shared(input, - k, - axis, - mode, - ov::op::v11::TopK::SortType::SORT_VALUES, - index_type, - false); - - return rename_outputs_with_suffix({topk->output(1)}, context.get_name()); + auto indices = make_topk_indices(input, k, axis, mode, index_type); + + return rename_outputs_with_suffix({indices}, context.get_name()); } } // namespace op diff --git a/src/frontends/gguf/src/op/flash_attn_ext.cpp b/src/frontends/gguf/src/op/flash_attn_ext.cpp index 18e32fb039bb4a..1dea26183a7781 100644 --- a/src/frontends/gguf/src/op/flash_attn_ext.cpp +++ b/src/frontends/gguf/src/op/flash_attn_ext.cpp @@ -4,20 +4,31 @@ #include #include +#include + +#include "node_context.hpp" +#include "op_table.hpp" +#include "openvino/op/add.hpp" #include "openvino/op/broadcast.hpp" #include "openvino/op/concat.hpp" #include "openvino/op/constant.hpp" #include "openvino/op/convert.hpp" #include "openvino/op/convert_like.hpp" +#include "openvino/op/divide.hpp" +#include "openvino/op/exp.hpp" +#include "openvino/op/matmul.hpp" +#include "openvino/op/maximum.hpp" +#include "openvino/op/multiply.hpp" +#include "openvino/op/reduce_max.hpp" +#include "openvino/op/reduce_sum.hpp" #include "openvino/op/reshape.hpp" #include "openvino/op/scaled_dot_product_attention.hpp" #include "openvino/op/slice.hpp" +#include "openvino/op/softmax.hpp" +#include "openvino/op/subtract.hpp" +#include "openvino/op/tanh.hpp" #include "openvino/op/transpose.hpp" #include "openvino/op/unsqueeze.hpp" -#include - -#include "node_context.hpp" -#include "op_table.hpp" #include "utils.hpp" namespace ov { @@ -26,20 +37,27 @@ namespace gguf { namespace op { OutputVector translate_flash_attn_ext(const NodeContext& context) { - num_inputs_check(context, 4, 4); + num_inputs_check(context, 4, 5); auto q_f32 = context.get_input(0); auto k = context.get_input(1); auto v = context.get_input(2); auto mask = context.get_input(3); + // gpt-oss: optional 5th input is the per-head attention sink logit [n_head]. + const bool has_sinks = context.get_input_size() == 5; float scale = context.get_attribute("scale"); + float kq_soft_cap = context.get_attribute("kq_soft_cap", 0.0f); const auto sdpa_type = ov::element::f16; auto q = std::make_shared(q_f32, sdpa_type); auto scale_node = std::make_shared(sdpa_type, ov::Shape{}, std::vector{scale}); ov::Output mask_sliced, res; - const std::string mask_name = context.get_attribute("is_swa", false) ? "KQ_mask_swa_sliced" : "KQ_mask_sliced"; + // Pick the layer flavor's mask. The cgraph decoder answers the "is_swa" attribute directly; the + // builder identifies it by the mask input's name (self_kq_mask_swa). + const bool is_swa = + context.get_attribute("is_swa", false) || context.get_input_names()[3].find("swa") != std::string::npos; + const std::string mask_name = is_swa ? "KQ_mask_swa_sliced" : "KQ_mask_sliced"; if (context.has_input(mask_name)) { mask_sliced = context.get_input(mask_name); } else { @@ -54,19 +72,35 @@ OutputVector translate_flash_attn_ext(const NodeContext& context) { mask_sliced = std::make_shared(mask_sliced, sdpa_type); } + // The two decoders hand q/k/v over in different layouts, so the head axis and the need for a + // transpose depend on the op_case: + // op_case 0 (llama.cpp cgraph decoder): already PERMUTEd to [B, n_head, n_tokens, head_size], + // the canonical SDPA layout -- tile K/V on axis 1, feed SDPA directly. + // op_case 100 (native .gguf builder): ggml-natural [B, n_tokens, n_head(_kv), head_size] -- tile + // K/V on axis 2 FIRST, then transpose all three. That ordering (concat -> GQA tile -> + // single Transpose -> SDPA) is what the CPU plugin's stateful_sdpa_fusion matches + // (its multi-query-broadcast pattern sits on the KV-cache concat output, ahead of + // exactly one transpose), so the attention fuses into + // ScaledDotProductAttentionWithKVCache. + const int op_case = context.get_op_case(); + FRONT_END_CHECK_IMPLEMENTED(op_case == 0 || op_case == 100, "Unsupported FLASH_ATTN_EXT case"); + const bool ggml_natural = op_case == 100; + const size_t head_axis = ggml_natural ? 2 : 1; + auto tile_kv = [&](int64_t num_heads, int64_t num_heads_kv, int64_t head_size, ov::Output kv) { int64_t factor = num_heads / num_heads_kv; if (factor > 1 && num_heads_kv > 1) { - ov::Output kv_broadcast_shape, kv_unsqueezed, new_kv_shape; - auto unsqueeze_axes = ov::op::v0::Constant::create(ov::element::i64, Shape{}, {2}); - kv_unsqueezed = std::make_shared(kv, unsqueeze_axes); - - kv_broadcast_shape = ov::op::v0::Constant::create(ov::element::i64, - {5}, - {(int64_t)1, (int64_t)1, factor, (int64_t)1, (int64_t)1}); - new_kv_shape = - ov::op::v0::Constant::create(ov::element::i64, {4}, {(int64_t)0, num_heads, (int64_t)-1, head_size}); - + // Insert the repeat axis right after the head axis, broadcast it to `factor`, then fold + // it back into the head axis: [.., n_head_kv, ..] -> [.., n_head_kv * factor, ..]. + auto unsqueeze_axes = ov::op::v0::Constant::create(ov::element::i64, Shape{}, {(int64_t)head_axis + 1}); + auto kv_unsqueezed = std::make_shared(kv, unsqueeze_axes); + std::vector bcast(5, 1); + bcast[head_axis + 1] = factor; + auto kv_broadcast_shape = ov::op::v0::Constant::create(ov::element::i64, {5}, bcast); + // special_zero keeps the leading dims (incl. the dynamic token axis) as-is. + std::vector new_shape = ggml_natural ? std::vector{0, 0, num_heads, head_size} + : std::vector{0, num_heads, -1, head_size}; + auto new_kv_shape = ov::op::v0::Constant::create(ov::element::i64, {4}, new_shape); kv = std::make_shared(kv_unsqueezed, kv_broadcast_shape, ov::op::BroadcastType::BIDIRECTIONAL); @@ -80,17 +114,95 @@ OutputVector translate_flash_attn_ext(const NodeContext& context) { // concat), but the head-count / head-size dims are static ggml facts the decoder knows. auto q_shape = context.get_input_shape(0).to_shape(); auto k_shape = context.get_input_shape(1).to_shape(); - k = tile_kv(q_shape[1], k_shape[1], q_shape[3], k); - v = tile_kv(q_shape[1], k_shape[1], q_shape[3], v); + k = tile_kv(q_shape[head_axis], k_shape[head_axis], q_shape[3], k); + v = tile_kv(q_shape[head_axis], k_shape[head_axis], q_shape[3], v); // SDPA requires q/k/v to share an element type; match k/v to q (ConvertConvertLike lowers these). k = std::make_shared(k, q); v = std::make_shared(v, q); - auto sdpa = std::make_shared(q, k, v, mask_sliced, scale_node, false); + ov::Output q_t = q, k_t = k, v_t = v; + if (ggml_natural) { + // [B, L, H, S] -> [B, H, L, S] (canonical SDPA layout). Each transpose gets its OWN order + // constant: the GPU plugin's TransposeSDPAMatcher requires consumers_count(1) on it, and a + // shared one leaves the permutes in the decode path and blocks the broadcast-into-SDPA fusion. + auto to_bhls = [] { + return ov::op::v0::Constant::create(ov::element::i64, {4}, {0, 2, 1, 3}); + }; + q_t = std::make_shared(q, to_bhls()); + k_t = std::make_shared(k, to_bhls()); + v_t = std::make_shared(v, to_bhls()); + } + + ov::Output sdpa; + if (kq_soft_cap != 0.0f) { + // Gemma2 attention soft-cap: tanh(QK^T * scale * (1/cap)) * cap + mask -> softmax -> *V. + // OV SDPA v13 has no native softcap parameter, so we decompose the attention manually. + // Operates in f32 (q already converted to f16 for normal path; here stay f32). + // q_t / k_t / v_t are already [B, H, L, S] from the transpose above but in f16; + // convert to f32 for the manual decomposition. + using namespace ov::op; + auto q_f32_t = std::make_shared(q_t, element::f32); + auto k_f32_t = std::make_shared(k_t, element::f32); + auto v_f32_t = std::make_shared(v_t, element::f32); + auto mask_f32 = mask_sliced.get_element_type() != element::f32 + ? std::make_shared(mask_sliced, element::f32)->output(0) + : mask_sliced; + + // QK^T: [B, H, L, S] x [B, H, S, Lk] -> [B, H, L, Lk] + auto kT = + std::make_shared(k_f32_t, + v0::Constant::create(element::i64, {4}, std::vector{0, 1, 3, 2})); + auto qk = std::make_shared(q_f32_t, kT, false, false); + + // Apply scale * (1/softcap), then tanh, then *softcap + auto pre_cap_scale = v0::Constant::create(element::f32, Shape{}, std::vector{scale / kq_soft_cap}); + auto qk_scaled = std::make_shared(qk, pre_cap_scale); + auto qk_tanh = std::make_shared(qk_scaled); + auto post_cap_scale = v0::Constant::create(element::f32, Shape{}, std::vector{kq_soft_cap}); + auto qk_capped = std::make_shared(qk_tanh, post_cap_scale); + + // Add mask (already sliced to [B, 1, L, Lk] or [B, 1, 1, Lk]) + auto qk_masked = std::make_shared(qk_capped, mask_f32); + + // Softmax over last axis (key dimension) + auto attn_weights = std::make_shared(qk_masked, -1); + + // Weighted sum over values: [B, H, L, Lk] x [B, H, Lk, S] -> [B, H, L, S] + auto attn_out_caps = std::make_shared(attn_weights, v_f32_t, false, false); + + sdpa = attn_out_caps; + } else if (!has_sinks) { + sdpa = std::make_shared(q_t, k_t, v_t, mask_sliced, scale_node, false); + } else { + // gpt-oss attention sinks: a learned per-head logit participates in the softmax + // denominator (so the attention weights do not sum to 1) but contributes no value. + // OpenVINO SDPA has a native 6-input form (q, k, v, mask, scale, sink) that the CPU + // plugin folds the sink straight into its online-softmax, so we no longer decompose + // attention by hand. The sink logit is per head: [n_head] -> [1, n_head, 1, 1] to + // broadcast over [B, n_head, q, 1] (rank must equal the query rank, last dim 1). + using namespace ov::op; + auto sink = context.get_input(4); + auto sink_f16 = sink.get_element_type() != element::f16 + ? std::make_shared(sink, element::f16)->output(0) + : sink; + auto sink_shape = v0::Constant::create(element::i64, {4}, std::vector{1, (int64_t)q_shape[2], 1, 1}); + auto sink_r = std::make_shared(sink_f16, sink_shape, false); + sdpa = std::make_shared(q_t, + k_t, + v_t, + mask_sliced, + scale_node, + sink_r, + false); + } + // [B, H, L, S] -> [B, L, H, S] (ggml-natural layout expected by caller). res = std::make_shared(sdpa, ov::op::v0::Constant::create(ov::element::i64, {4}, {0, 2, 1, 3})); - res = std::make_shared(res, ov::element::f32); + // SDPA paths produce f16; the soft-cap path produces f32 directly. + if (kq_soft_cap == 0.0f) { + res = std::make_shared(res, ov::element::f32); + } return rename_outputs_with_suffix({res}, context.get_name()); } diff --git a/src/frontends/gguf/src/op/get_rows.cpp b/src/frontends/gguf/src/op/get_rows.cpp index 13b00eb84f506a..d804c4940ff3af 100644 --- a/src/frontends/gguf/src/op/get_rows.cpp +++ b/src/frontends/gguf/src/op/get_rows.cpp @@ -4,30 +4,53 @@ #include "node_context.hpp" #include "op_table.hpp" -#include "utils.hpp" - #include "openvino/core/node.hpp" #include "openvino/core/node_output.hpp" #include "openvino/op/constant.hpp" #include "openvino/op/convert.hpp" #include "openvino/op/gather.hpp" +#include "openvino/op/gather_elements.hpp" +#include "openvino/op/reshape.hpp" #include "openvino/op/squeeze.hpp" #include "openvino/op/unsqueeze.hpp" +#include "utils.hpp" namespace ov { namespace frontend { namespace gguf { namespace op { -OutputVector translate_get_rows(const NodeContext & context) { +OutputVector translate_get_rows(const NodeContext& context) { num_inputs_check(context, 2, 2); - int op_case = context.get_attribute("op_case", 0); + int op_case = context.get_op_case(); Output res; auto data = context.get_input(0); auto indices = context.get_input(1); + // MoE gating-weight gather: data = probs [1,1,T,E], indices = selected experts + // [1,1,T,K]; pick, per token, the probs of its K selected experts -> [1,1,T,K]. + // This is a per-row (GatherElements) gather over the expert axis, distinct from the + // embedding-style row gather below. + if (op_case == 10) { + // probs [1,1,T,E], selected [1,1,T,K] -> per-row gather over the last (expert) + // axis -> [1,1,T,K], then reshape to [1,T,K,1] for the broadcast-multiply with + // experts [1,T,K,n_embd]. Use an explicit [1,-1,K,1] reshape (K is static; T is + // dynamic) instead of Squeeze+Unsqueeze, which the CPU plugin implements as a + // Reshape internally and mis-infers the static pattern when T=1 at graph-build time. + // K is static (n_expert_used); read from the declared output shape [1,T,K,1]. + // Use PartialShape index to avoid .to_shape() throwing when T is dynamic. + const int64_t K = context.get_output_shape()[2].get_length(); + auto idx = std::make_shared(indices, ov::element::i32); + auto ge = std::make_shared(data, idx, -1); // [1,1,T,K] + auto col = std::make_shared( + ge, + ov::op::v0::Constant::create(ov::element::i64, {4}, std::vector{1, -1, K, 1}), + false); // [1,T,K,1] + return rename_outputs_with_suffix({col}, context.get_name()); + } + if (op_case == 2) { // The input comes from a VIEW indices = process_view_input(context, 1); @@ -55,10 +78,10 @@ OutputVector translate_get_rows(const NodeContext & context) { res = std::make_shared(data, indices, axis); } - auto output_type = context.get_attribute("output_type"); - if (res.get_element_type() != output_type) { - res = std::make_shared(res, output_type); + if (res.get_element_type() != context.get_output_type()) { + res = std::make_shared(res, context.get_output_type()); } + // The two Squeezes above dropped the leading axes; restore ggml's rank-4 form. res = std::make_shared(res, ov::op::v0::Constant::create(ov::element::i64, {1}, {0})); return rename_outputs_with_suffix({res}, context.get_name()); } diff --git a/src/frontends/gguf/src/op/glu_geglu.cpp b/src/frontends/gguf/src/op/glu_geglu.cpp index c4868a2cbf6569..46494a4d0cdb15 100644 --- a/src/frontends/gguf/src/op/glu_geglu.cpp +++ b/src/frontends/gguf/src/op/glu_geglu.cpp @@ -3,15 +3,15 @@ // #include + +#include "node_context.hpp" +#include "op_table.hpp" #include "openvino/core/node_output.hpp" #include "openvino/op/constant.hpp" #include "openvino/op/gelu.hpp" #include "openvino/op/multiply.hpp" #include "openvino/op/sigmoid.hpp" #include "openvino/op/slice.hpp" - -#include "node_context.hpp" -#include "op_table.hpp" #include "utils.hpp" namespace ov { @@ -28,7 +28,7 @@ OutputVector translate_glu_geglu(const NodeContext& context) { src0 = context.get_input(0); src1 = context.get_input(1); } else { - // GGML splits along ne[0] (OV last axis) using floor division: nc = ne[0] / 2. + // GGUF splits along ne[0] (OV last axis) using floor division: nc = ne[0] / 2. // Both halves are nc elements; if the dimension is odd, the last element is dropped. // Use Slice instead of Split to handle odd dimensions correctly. auto combined = context.get_input(0); @@ -51,7 +51,10 @@ OutputVector translate_glu_geglu(const NodeContext& context) { std::swap(src0, src1); } - // ggml's GEGLU uses the tanh GELU approximation (ggml_gelu_f32); v7::Gelu defaults to ERF. + // ggml's GGML_GLU_OP_GEGLU uses the tanh GELU approximation (ggml_gelu -> + // GGML_UNARY_OP_GELU = 0.5x(1+tanh(sqrt(2/pi) x (1+0.044715 x^2)))), NOT the erf form. + // OV's Gelu defaults to ERF, which is close but diverges ~1-2% per call and compounds + // across layers into a wrong argmax on deep models (e.g. gemma3-1b). Match ggml with TANH. auto gelu = std::make_shared(src0, ov::op::GeluApproximationMode::TANH); auto res = std::make_shared(gelu, src1); diff --git a/src/frontends/gguf/src/op/mul_mat_id.cpp b/src/frontends/gguf/src/op/mul_mat_id.cpp index e5592be7830158..23ec998e5b3f4d 100644 --- a/src/frontends/gguf/src/op/mul_mat_id.cpp +++ b/src/frontends/gguf/src/op/mul_mat_id.cpp @@ -18,7 +18,10 @@ #include "openvino/op/reshape.hpp" #include "openvino/op/shape_of.hpp" #include "openvino/op/slice.hpp" +#include "openvino/op/transpose.hpp" #include "openvino/op/unsqueeze.hpp" +#include "ov_ops/gather_matmul.hpp" +#include "transformations/utils/utils.hpp" #include #include "node_context.hpp" @@ -182,38 +185,109 @@ ov::Output translate_mul_mat_id_mxfp4_packed(const NodeContext& contex return result; } -} // namespace - -// GGML_OP_MUL_MAT_ID: per-token MoE expert matmul. ids select which expert row of the weight -// tensor each token uses; activations are gathered/broadcast accordingly and matmul'd. -OutputVector translate_mul_mat_id(const NodeContext& context) { - num_inputs_check(context, 3, 3); - - auto expert_weights = context.get_input(0); - auto activations = context.get_input(1); - auto ids = context.get_input(2); +// Lower to the internal ov::op::internal::GatherMatmul, which the CPU/GPU plugins execute as one +// optimized batched expert-matmul (and, when the expert weights are a compressed +// Constant->Convert->[Subtract]->Multiply block, fold into GatherMatmulCompressed so the weights +// stay compressed -- no host f32 expansion, which is what keeps MoE compile memory bounded). The +// CPU GatherMatmul node requires CONSTANT-backed weights, so this path is used only when the +// expert weights come from a Constant (the real .gguf builder / cgraph weight leaf). GatherMatmul: +// A [n_activated, T, cols] (n_activated == 1 broadcasts the same input to every +// selected expert; == K gives a per-slot input) +// B [n_expert, rows, cols] (transpose_b=true -> A . Bᵀ) +// indices [T, K] i32 (the selected expert per (token, slot)) +// out [K, T, rows] +// which we reshape back to the builder's [1, T, K, rows] convention. +// +// Input layouts (OpenVINO reversed order): +// expert_weights (as) : [n_expert, rows, cols] (or reversed rank-4 [1, n_expert, rows, cols]) +// activations (b) : [.., T, cols] (gate/up, shared input) or [.., T, K, cols] (down) +// ids : [1, 1, T, K] +ov::Output translate_mul_mat_id_gathermatmul(const NodeContext& context, + ov::Output expert_weights, + ov::Output activations, + ov::Output ids) { + // Normalize the expert weights to the rank-3 [n_expert, rows, cols] GatherMatmul expects. The + // native builder surfaces them rank-3 already; the cgraph path surfaces them as a reversed + // rank-4 [1, n_expert, rows, cols], so drop the leading unit batch dim. + ov::Output as = expert_weights; + if (as.get_partial_shape().rank().is_static() && as.get_partial_shape().rank().get_length() == 4) { + auto as_shape = std::make_shared(as, ov::element::i64); + as = std::make_shared(as, get_dimensions(as_shape, {1, 2, 3}), false); + } + auto b = activations; + + // Canonicalize ids to 2D [T, K] (the builder carries leading 1-dims). + const auto ids_rank = static_cast(ids.get_partial_shape().size()); + ov::Output ids_2d = std::make_shared( + ids, + std::make_shared( + ov::OutputVector{ov::op::v0::Constant::create(ov::element::i64, ov::Shape{1}, {-1}), + get_dimensions(ids, {ids_rank - 1})}, + 0), + false); // [T, K] + const int64_t K = ids.get_partial_shape()[ids_rank - 1].get_length(); + + // Build the GatherMatmul activation A = [n_activated, T, cols]. + // gate/up: b is [.., T, cols] (one shared input fanned out to all experts) -> A = [1, T, cols] + // down : b is [.., T, K, cols] (already per-slot) -> A = [K, T, cols] + const auto& bps = b.get_partial_shape(); + const int64_t cols = bps[bps.size() - 1].get_length(); + const bool has_k = + K > 1 && bps.size() >= 2 && bps[bps.size() - 2].is_static() && bps[bps.size() - 2].get_length() == K; + ov::Output a; + if (has_k) { + auto b_tkc = std::make_shared( + b, + ov::op::v0::Constant::create(ov::element::i64, ov::Shape{3}, std::vector{-1, K, cols}), + false); + a = std::make_shared( + b_tkc, + ov::op::v0::Constant::create(ov::element::i64, ov::Shape{3}, std::vector{1, 0, 2})); + } else { + a = std::make_shared( + b, + ov::op::v0::Constant::create(ov::element::i64, ov::Shape{3}, std::vector{1, -1, cols}), + false); + } - if (expert_weights.get_element_type() == ov::element::u8 && expert_weights.get_partial_shape().rank().is_static() && - expert_weights.get_partial_shape().rank().get_length() == 5) { - return rename_outputs_with_suffix({translate_mul_mat_id_mxfp4_packed(context, expert_weights, activations, ids)}, - context.get_name()); + if (ids_2d.get_element_type() != ov::element::i32) { + ids_2d = std::make_shared(ids_2d, ov::element::i32); } - // OpenVINO sees GGML tensors in reversed dimension order: - // weights: [1, n_expert, m, k] - // activations: [1, n_tokens, n_used_or_1, k] - // ids: [1, 1, n_tokens, n_used] + // B stays in its (possibly compressed) precision so ConvertGatherMatmulToGatherMatmulCompressed + // can pick up the decompression subgraph and keep the weights compressed. + auto gmm = std::make_shared(a, as, ids_2d); // [K, T, rows] + + // [K, T, rows] -> [1, T, K, rows] (builder convention). + const int64_t rows = as.get_partial_shape()[as.get_partial_shape().size() - 2].get_length(); + auto kt2tk = std::make_shared( + gmm, + ov::op::v0::Constant::create(ov::element::i64, ov::Shape{3}, std::vector{1, 0, 2})); // [T, K, rows] + return std::make_shared( + kt2tk, + ov::op::v0::Constant::create(ov::element::i64, ov::Shape{4}, std::vector{1, -1, K, rows}), + false); +} + +// Portable fallback: per-token expert matmul via Gather (select each token's expert rows) + a +// batched MatMul. Handles non-constant expert weights (e.g. the single-op unit tests) that the CPU +// GatherMatmul node does not support. Expects reversed rank-4 inputs +// (weights [1, n_expert, m, k], activations [1, T, 1_or_K, k], ids [1, 1, T, K]). +ov::Output translate_mul_mat_id_generic(const NodeContext& context, + ov::Output expert_weights, + ov::Output activations, + ov::Output ids) { auto expert_weights_shape_4d = std::make_shared(expert_weights, ov::element::i64); auto activations_shape_4d = std::make_shared(activations, ov::element::i64); auto ids_shape_4d = std::make_shared(ids, ov::element::i64); - auto expert_weights_shape_3d = get_dimensions(expert_weights_shape_4d, {1, 2, 3}); - auto activations_shape_3d = get_dimensions(activations_shape_4d, {1, 2, 3}); - auto ids_shape_2d = get_dimensions(ids_shape_4d, {2, 3}); - - expert_weights = std::make_shared(expert_weights, expert_weights_shape_3d, false); - activations = std::make_shared(activations, activations_shape_3d, false); - ids = std::make_shared(ids, ids_shape_2d, false); + expert_weights = std::make_shared(expert_weights, + get_dimensions(expert_weights_shape_4d, {1, 2, 3}), + false); + activations = std::make_shared(activations, + get_dimensions(activations_shape_4d, {1, 2, 3}), + false); + ids = std::make_shared(ids, get_dimensions(ids_shape_4d, {2, 3}), false); if (ids.get_element_type() != ov::element::i32 && ids.get_element_type() != ov::element::i64) { ids = std::make_shared(ids, ov::element::i32); @@ -242,25 +316,51 @@ OutputVector translate_mul_mat_id(const NodeContext& context) { FRONT_END_OP_CONVERSION_CHECK(output_shape.rank().is_static() && output_shape.rank().get_length() == 4, "Unexpected MUL_MAT_ID output rank"); FRONT_END_OP_CONVERSION_CHECK(output_shape[3].is_static(), "Expected static row dimension for MUL_MAT_ID output"); - const auto row_dim_value = output_shape[3].get_length(); - auto row_dim = ov::op::v0::Constant::create(ov::element::i64, {1}, {row_dim_value}); + auto row_dim = ov::op::v0::Constant::create(ov::element::i64, {1}, {output_shape[3].get_length()}); ov::Output result = std::make_shared(activations_expanded, selected_weights, false, true); - - auto result_target_dims = std::make_shared( - ov::OutputVector{ - batch_dim, - get_dimensions(ids_shape, {0, 1}), - row_dim, - }, - 0); + auto result_target_dims = + std::make_shared(ov::OutputVector{batch_dim, get_dimensions(ids_shape, {0, 1}), row_dim}, + 0); result = std::make_shared(result, result_target_dims, false); - if (result.get_element_type() != output_type) { result = std::make_shared(result, output_type); } + return result; +} + +} // namespace +// GGML_OP_MUL_MAT_ID: per-token MoE expert matmul. ids select which expert row of the weight +// tensor each token uses; activations are gathered/broadcast accordingly and matmul'd. Dispatch: +// - packed MXFP4 experts -> on-graph dequant + per-expert matmul; +// - constant-backed experts (the real .gguf models) -> internal GatherMatmul (compressed, +// memory-bounded, one fused batched matmul); +// - non-constant experts (single-op tests / dynamic) -> portable Gather + MatMul fallback. +OutputVector translate_mul_mat_id(const NodeContext& context) { + num_inputs_check(context, 3, 3); + + auto expert_weights = context.get_input(0); + auto activations = context.get_input(1); + auto ids = context.get_input(2); + + if (expert_weights.get_element_type() == ov::element::u8 && expert_weights.get_partial_shape().rank().is_static() && + expert_weights.get_partial_shape().rank().get_length() == 5) { + return rename_outputs_with_suffix({translate_mul_mat_id_mxfp4_packed(context, expert_weights, activations, ids)}, + context.get_name()); + } + + // The CPU GatherMatmul node requires constant-backed weights. Real .gguf models feed a + // (possibly compressed) Constant weight leaf -> use the fused GatherMatmul path (compressed, + // memory-bounded). A non-constant weights input (single-op tests, dynamic producers) can't use + // GatherMatmul on CPU, so fall back to the portable Gather + MatMul lowering. + ov::Output result; + if (ov::op::util::is_on_path(expert_weights)) { + result = translate_mul_mat_id_gathermatmul(context, expert_weights, activations, ids); + } else { + result = translate_mul_mat_id_generic(context, expert_weights, activations, ids); + } return rename_outputs_with_suffix({result}, context.get_name()); } diff --git a/src/frontends/gguf/src/op/permute.cpp b/src/frontends/gguf/src/op/permute.cpp index e55506025aaa5f..e7253f1990e545 100644 --- a/src/frontends/gguf/src/op/permute.cpp +++ b/src/frontends/gguf/src/op/permute.cpp @@ -5,6 +5,9 @@ #include #include #include + +#include "node_context.hpp" +#include "op_table.hpp" #include "openvino/core/node.hpp" #include "openvino/op/add.hpp" #include "openvino/op/concat.hpp" @@ -12,9 +15,6 @@ #include "openvino/op/reshape.hpp" #include "openvino/op/slice.hpp" #include "openvino/op/transpose.hpp" - -#include "node_context.hpp" -#include "op_table.hpp" #include "utils.hpp" namespace ov { @@ -25,7 +25,7 @@ namespace op { OutputVector translate_permute(const NodeContext& context) { num_inputs_check(context, 1, 1); - int op_case = context.get_attribute("op_case", 0); + int op_case = context.get_op_case(); FRONT_END_CHECK_IMPLEMENTED(op_case == 1 || op_case == 2 || op_case == 3 || op_case == 4, "Unsupported PERMUTE case"); @@ -37,17 +37,33 @@ OutputVector translate_permute(const NodeContext& context) { res = std::make_shared(src, perm); } else if (op_case == 4) { auto output_shape = context.get_output_shape().to_shape(); - auto n_heads = ov::op::v0::Constant::create(ov::element::i64, {1}, {output_shape[1]}); - auto head_size = ov::op::v0::Constant::create(ov::element::i64, {1}, {output_shape[3]}); - auto n_seq_active = context.has_input("n_seq_active") - ? context.get_input("n_seq_active") - : ov::op::v0::Constant::create(ov::element::i64, {1}, {output_shape[0]}); - auto neg_one = ov::op::v0::Constant::create(ov::element::i64, {1}, {-1}); + const auto n_heads = static_cast(output_shape[1]); + const auto head_size = static_cast(output_shape[3]); - auto new_shape = - std::make_shared(ov::OutputVector{n_seq_active, neg_one, n_heads, head_size}, 0); + ov::Output reshaped; + if (context.has_input("n_seq_active")) { + // Reshape shape inference can only use a pattern whose value bounds are known, and + // `n_seq_active` is a Parameter, so it has none. Building the whole pattern with a single + // Concat therefore discards the statically known n_heads/head_size as well, and Q reaches + // SDPA with a dynamic head size, which makes the GPU plugin decompose SDPA into + // Gemm+SoftMax. Splitting the reshape keeps the head layout in an all-constant pattern, so + // it survives shape inference. Both reshapes are metadata-only, so this costs no extra + // data movement. + auto neg_one = ov::op::v0::Constant::create(ov::element::i64, {1}, {-1}); + auto seq_pattern = + std::make_shared(ov::OutputVector{context.get_input("n_seq_active"), neg_one}, 0); + auto by_seq = std::make_shared(src, seq_pattern, false); - auto reshaped = std::make_shared(src, new_shape, true); + auto head_pattern = + ov::op::v0::Constant::create(ov::element::i64, {4}, std::vector{0, -1, n_heads, head_size}); + reshaped = std::make_shared(by_seq, head_pattern, true); + } else { + auto new_shape = ov::op::v0::Constant::create( + ov::element::i64, + {4}, + std::vector{static_cast(output_shape[0]), -1, n_heads, head_size}); + reshaped = std::make_shared(src, new_shape, true); + } res = std::make_shared(reshaped, perm); } else { auto cache_shape = src.get_partial_shape(); @@ -73,6 +89,8 @@ OutputVector translate_permute(const NodeContext& context) { seq_active_end = context.get_input("seq_active_end"); } else { int64_t n_seq_active = output_shape[0]; + // The decoder exposes the view's sequence-axis start as a typed attribute (already in + // elements); the op translators never touch raw ggml strides/offsets. int64_t seq_active_start_val = context.get_attribute("view_seq_offset", 0); int64_t seq_active_end_val = seq_active_start_val + n_seq_active; seq_active_start = ov::op::v0::Constant::create(ov::element::i64, {1}, {seq_active_start_val}); diff --git a/src/frontends/gguf/src/op/reshape.cpp b/src/frontends/gguf/src/op/reshape.cpp index b5039fa80b9407..b086722f1d0b11 100644 --- a/src/frontends/gguf/src/op/reshape.cpp +++ b/src/frontends/gguf/src/op/reshape.cpp @@ -14,6 +14,7 @@ #include "openvino/op/concat.hpp" #include "openvino/op/constant.hpp" #include "openvino/op/reshape.hpp" +#include "openvino/op/transpose.hpp" #include #include @@ -22,17 +23,19 @@ namespace frontend { namespace gguf { namespace op { -OutputVector translate_reshape(const NodeContext & context) { +OutputVector translate_reshape(const NodeContext& context) { num_inputs_check(context, 1, 1); - if (context.get_input(0).get_partial_shape() == context.get_output_shape()) { + if (context.get_input_shape(0) == context.get_output_shape()) { return {context.get_input(0)}; } - int op_case = context.get_attribute("op_case", 0); - FRONT_END_CHECK_IMPLEMENTED( - op_case == 1 || op_case == 2 || op_case == 3 || op_case == 4 || op_case == 5 || op_case == 6 || - op_case == 7 || op_case == 8, - "Unsupported RESHAPE case"); + // One numbering for both ingest paths: every case below is reachable from the llama.cpp cgraph + // decoder (see ggml-decoder.cpp::compute_op_case) and from the native .gguf builder, which + // describes its reshapes so that the same case applies. + int op_case = context.get_op_case(); + FRONT_END_CHECK_IMPLEMENTED(op_case == 1 || op_case == 2 || op_case == 3 || op_case == 4 || op_case == 5 || + op_case == 6 || op_case == 7 || op_case == 8, + "Unsupported RESHAPE case"); if (op_case == 8) { // Identity reshape (ggml src ne == node ne): a no-op. Pass the input through so any dynamic @@ -43,27 +46,68 @@ OutputVector translate_reshape(const NodeContext & context) { auto output_shape = context.get_output_shape().to_shape(); std::shared_ptr new_shape_node; if (op_case == 1) { + // [B, 1, T, n_head*head_size] -> [B, T, n_head, head_size]: split the last dim into heads and + // flatten whatever leads it into dim 1. Same shape in both stateful and non-stateful paths; + // the 3D form was causing RoPE broadcasting to T×T when the trailing dimensions are 1 (MQA, + // n_head_kv=1). + // + // The leading dim is COPIED from the input via special_zero rather than written as + // output_shape[0] (a literal 1). That is what makes the attention block layout-polymorphic: + // ov::pass::SDPAToPagedAttention moves the token count into dim 0 by rewriting input_ids, and + // a literal here would discard that and leave PA deriving [1, T*H*S] operands where the + // plugin wants [T, H*S]. With the 0 the same constant serves both: + // SDPA inference: in [1, 1, T, H*S] -> [1, T, H, S] + // PagedAttention: in [T, 1, 1, H*S] -> [T, 1, H, S] (identical buffer, tokens in dim 0) new_shape_node = ov::op::v0::Constant::create( - ov::element::i64, {4}, - std::vector{(int64_t) output_shape[0], -1, (int64_t) output_shape[2], (int64_t) output_shape[3]}); + ov::element::i64, + {4}, + std::vector{0, -1, (int64_t)output_shape[2], (int64_t)output_shape[3]}); + return rename_outputs_with_suffix( + {std::make_shared(context.get_input(0), new_shape_node, /*special_zero=*/true)}, + context.get_name()); } else if (op_case == 2) { + // Merge the heads back after attention. Like op_case 1, the leading dim is copied from the input + // (special_zero) rather than pinned to output_shape[0], so the token axis stays wherever the + // active attention backend put it. + // + // The rank stays 4 because the very next op is the residual Add against the layer input and OV + // broadcasts elementwise operands from the RIGHT: every activation in the graph is rank 4 (ggml's + // own convention), so mixing in a rank-3 result would right-align and silently form a + // token x token outer product once the token count is not on the axis one happens to expect. + // in [1, T, H, S] -> [1, 1, T, H*S] + // The last dim is the static n_head*head_size and the -1 absorbs the remaining axis, so the + // following MatMul against [n_embd, n_embd] is unaffected. new_shape_node = ov::op::v0::Constant::create( - ov::element::i64, {4}, - std::vector{(int64_t) output_shape[0], (int64_t) output_shape[1], -1, (int64_t) output_shape[3]}); + ov::element::i64, + {4}, + std::vector{0, (int64_t)output_shape[1], -1, (int64_t)output_shape[3]}); + return rename_outputs_with_suffix( + {std::make_shared(context.get_input(0), new_shape_node, /*special_zero=*/true)}, + context.get_name()); } else if (op_case == 3) { // Flatten-for-SET_ROWS: [F, tok, 1, 1] -> [1, F*tok, -1, 1] (the KV-cache write path, e.g. // gpt-oss cache_v). Token count stays on the dynamic axis via -1. new_shape_node = ov::op::v0::Constant::create( - ov::element::i64, {4}, std::vector{(int64_t) output_shape[0], (int64_t) output_shape[1], -1, 1}); + ov::element::i64, + {4}, + std::vector{(int64_t)output_shape[0], (int64_t)output_shape[1], -1, 1}); } else if (op_case == 4) { return {context.get_input(0).get_node_shared_ptr()->input_value(0)}; } else if (op_case == 5) { - std::vector shape_vec = {1, 1, -1, (int64_t) output_shape[3]}; + std::vector shape_vec = {1, 1, -1, (int64_t)context.get_output_shape().to_shape()[3]}; new_shape_node = ov::op::v0::Constant::create(ov::element::i64, {4}, shape_vec); + // // Alternative + // auto token_len = context.get_input("token_len"); + // auto emb_size = + // ov::op::v0::Constant::create(ov::element::i64, {1}, {(int64_t) + // context.get_output_shape().to_shape()[3]}); + // auto one = ov::op::v0::Constant::create(ov::element::i64, {1}, {1}); + // new_shape_node = std::make_shared(ov::OutputVector{one, one, token_len, emb_size}, 0); + } else if (op_case == 6) { // The output layout rearranges dims relative to the input (e.g. qwen3-next q/k_conv_predelta: // [128,2,8,T] -> [128,16,T,1]). The decoder supplies the OV-order target with -1 on the dynamic @@ -81,6 +125,7 @@ OutputVector translate_reshape(const NodeContext & context) { new_shape_node = ov::op::v0::Constant::create( ov::element::i64, {output_shape.size()}, std::vector(output_shape.begin(), output_shape.end())); + } auto res = std::make_shared(context.get_input(0), new_shape_node, false); return rename_outputs_with_suffix({res}, context.get_name()); diff --git a/src/frontends/gguf/src/op/rms_norm.cpp b/src/frontends/gguf/src/op/rms_norm.cpp index 9cfbdefe502ead..0813215f58705e 100644 --- a/src/frontends/gguf/src/op/rms_norm.cpp +++ b/src/frontends/gguf/src/op/rms_norm.cpp @@ -2,15 +2,15 @@ // SPDX-License-Identifier: Apache-2.0 // +#include "openvino/decompositions/rms_norm.hpp" + #include +#include "node_context.hpp" +#include "op_table.hpp" #include "openvino/core/node_output.hpp" -#include "openvino/decompositions/rms_norm.hpp" #include "openvino/op/constant.hpp" #include "openvino/pass/node_registry.hpp" - -#include "node_context.hpp" -#include "op_table.hpp" #include "utils.hpp" namespace ov { diff --git a/src/frontends/gguf/src/op/rope.cpp b/src/frontends/gguf/src/op/rope.cpp index ec5cabf6ed848f..8f2e57a5e9bbcf 100644 --- a/src/frontends/gguf/src/op/rope.cpp +++ b/src/frontends/gguf/src/op/rope.cpp @@ -24,7 +24,9 @@ #include "openvino/op/slice.hpp" #include "openvino/op/split.hpp" #include "openvino/op/subtract.hpp" +#include "openvino/op/transpose.hpp" #include "openvino/op/unsqueeze.hpp" +#include "openvino/pass/node_registry.hpp" #include "node_context.hpp" #include "op_table.hpp" @@ -38,7 +40,7 @@ namespace op { OutputVector translate_rope(const NodeContext& context) { num_inputs_check(context, 2, 3); - int op_case = context.get_attribute("op_case", 0); + int op_case = context.get_op_case(); ov::Output res; @@ -68,37 +70,40 @@ OutputVector translate_rope(const NodeContext& context) { cos_theta_node = sin_cos.second; } - // The canonical [1, -1, n_head, head_size] reshape target (token count on the dynamic axis), - // used by the VIEW prologue and the TYPE_NORMAL stack below. + // The canonical [B, -1, n_head, head_size] reshape target (token count on the dynamic axis), used + // by the VIEW prologue and the TYPE_NORMAL stack below. The leading 0 is a special_zero marker + // that COPIES the input's dim 0 rather than pinning a literal 1, so a token-major activation + // (the layout ov::pass::SDPAToPagedAttention establishes) keeps its tokens in dim 0. Every use + // must therefore pass special_zero=true. auto make_bhsd_shape = [&]() { return ov::op::v0::Constant::create( ov::element::i64, {4}, - std::vector{1, -1, (int64_t)output_shape[2], (int64_t)output_shape[3]}); + std::vector{0, -1, (int64_t)output_shape[2], (int64_t)output_shape[3]}); }; if (op_case == 2) { // The input comes from a VIEW int slice_len = static_cast(output_shape[2] * output_shape[3]); data = process_view_input(context, 0, slice_len); - data = std::make_shared(data, make_bhsd_shape(), false); + data = std::make_shared(data, make_bhsd_shape(), true); } if (mode == TYPE_NORMAL) { // Emit the Flux-style interleaved RoPE pattern so ov::pass::RoPEFusionFlux // folds this subgraph into ov::op::internal::RoPE → GPU ocl::rope::opt kernel. // RoPEFusionFlux requires rank-4 x with static last two dims [n_heads, head_size]. - // After the VIEW prologue the data is already [1,L,n_heads,head_size] (non-stateful) - // or [L,n_heads,head_size] (stateful, lifted to rank-4 below). + // After the VIEW prologue the data is already [B,L,n_heads,head_size]. const int64_t n_heads = static_cast(output_shape[2]); const int64_t head_size = static_cast(output_shape[3]); const int64_t half = head_size / 2; - // Reshape to [1, L, n_heads, half, 2] to expose interleaved pairs (reinterprets any - // incoming rank in element order, so no separate rank lift is needed). - auto paired_shape = ov::op::v0::Constant::create( - ov::element::i64, {5}, std::vector{1, -1, n_heads, half, 2}); - auto x_paired = std::make_shared(data, paired_shape, false); + // Reshape to [B, L, n_heads, half, 2] to expose interleaved pairs (reinterprets any + // incoming rank in element order, so no separate rank lift is needed). The leading 0 copies + // the input's batch dim (special_zero) so the token axis stays where the caller had it. + auto paired_shape = + ov::op::v0::Constant::create(ov::element::i64, {5}, std::vector{0, -1, n_heads, half, 2}); + auto x_paired = std::make_shared(data, paired_shape, true); auto split_axis = ov::op::v0::Constant::create(ov::element::i64, ov::Shape{}, {-1LL}); auto data_split = std::make_shared(x_paired, split_axis, 2); @@ -109,7 +114,7 @@ OutputVector translate_rope(const NodeContext& context) { auto x1_neg = std::make_shared(x1, neg_one_f); auto x_rotated_paired = std::make_shared(ov::OutputVector{x1_neg, x0}, -1); - auto x_rotated = std::make_shared(x_rotated_paired, make_bhsd_shape(), false); + auto x_rotated = std::make_shared(x_rotated_paired, make_bhsd_shape(), true); // Expand cos/sin from [B, L, 1, half] to [B, L, 1, head_size]. auto expand_cos_sin = [&](ov::Output cs) -> ov::Output { @@ -136,27 +141,81 @@ OutputVector translate_rope(const NodeContext& context) { const int64_t head_dim = static_cast(output_shape[3]); const int64_t n_rot = rope_config.n_dims > 0 ? rope_config.n_dims : head_dim; - // Rotate only the first n_rot elements of every head and concatenate the untouched tail. - Output rotary_in = data; - Output pass_through; + // Split the head into the rotated block [0, n_rot) and the untouched tail [n_rot, head_dim) + // on the innermost axis. Both branches below rotate `rotary_in` and re-concatenate the tail. + auto split_rotary = [&](ov::Output x, + ov::Output& rotary_in, + ov::Output& pass_through) { + rotary_in = x; + if (n_rot < head_dim) { + auto neg_one = ov::op::v0::Constant::create(ov::element::i64, {1}, {-1}); + auto zero = ov::op::v0::Constant::create(ov::element::i64, {1}, {0}); + auto one = ov::op::v0::Constant::create(ov::element::i64, {1}, {1}); + auto n_rot_c = ov::op::v0::Constant::create(ov::element::i64, {1}, {n_rot}); + auto head_c = ov::op::v0::Constant::create(ov::element::i64, {1}, {head_dim}); + rotary_in = std::make_shared(x, zero, n_rot_c, one, neg_one); + pass_through = std::make_shared(x, n_rot_c, head_c, one, neg_one); + } + }; + + // Build the canonical NEOX RoPE via the shared decomposition helper, which emits the exact + // split-halves + Multiply(-1)+Add + Concat pattern that ov::pass::RoPEFusion (specifically + // the RoPEFusionGPTOSS matcher) folds into the fused ov::op::internal::RoPE primitive on + // CPU/GPU. + // + // That matcher only fires when the rotated tensor is laid out as [B, H, L, S] and the + // cos/sin caches are [?, 1, ?, head/2]. Our tensors are ggml-natural: data is [B, L, H, S] + // and cos/sin are [B, L, 1, head/2]. So we transpose every operand into the canonical + // [B, H, L, S] layout (heads on axis 1), run the decomposition there, and transpose the + // result back to the gguf layout. The math is unchanged; the wrapping Transposes are sunk / + // cancelled against the adjacent PERMUTE during TransposeSinking. + const int64_t n_head_rope = static_cast(output_shape[2]); + const int64_t head_size_rope = static_cast(output_shape[3]); + const auto perm_bhls = ov::op::v0::Constant::create(ov::element::i64, {4}, {0, 2, 1, 3}); + + // The DATA reaches this op in inconsistent shapes depending on the layer's upstream rank: + // rank-3 [B, L, H*S] (e.g. n_head_kv=1 layers fed by a rank-3 producer), or rank-4 that may + // be [B, L, H, S] OR [B, 1, L, S]. A single fixed Transpose cannot normalize all of these. + // Instead, always Reshape the data to the canonical ggml-natural [B, L, H, S] using the op's + // output_shape (element order is preserved, so this correctly reinterprets every incoming + // layout), then Transpose {0,2,1,3}. The leading dim is copied through (special_zero) instead + // of written as a literal 1, so a token-major activation ([L,1,H,S], the layout + // ov::pass::SDPAToPagedAttention establishes) keeps its tokens in dim 0 here; cos/sin below + // broadcast against either arrangement. + auto data_to_bhls = [&](ov::Output x) -> ov::Output { + auto shape4d = ov::op::v0::Constant::create(ov::element::i64, + {4}, + std::vector{0, -1, n_head_rope, head_size_rope}); + x = std::make_shared(x, shape4d, true); // [B, L, H, S] + return std::make_shared(x, perm_bhls); // [B, H, L, S] + }; + // cos/sin always arrive rank-4 [B, L, 1, head/2]; just transpose to [B, 1, L, head/2]. + auto cossin_to_bhls = [&](ov::Output x) -> ov::Output { + return std::make_shared(x, perm_bhls); + }; + + auto x_bhls = data_to_bhls(data); // [B, H, L, S] + auto cos_bhls = cossin_to_bhls(cos_theta_node); // [B, 1, L, n_rot/2] + auto sin_bhls = cossin_to_bhls(sin_theta_node); // [B, 1, L, n_rot/2] + + // Slice the rotated block AFTER the layout change so the reshape above still sees the full + // head; the innermost axis is the head axis in both layouts. + ov::Output rotary_in; + ov::Output pass_through; + split_rotary(x_bhls, rotary_in, pass_through); + + ov::pass::NodeRegistry reg; + ov::Output roped = + ov::decomposition::rope(reg, rotary_in, cos_bhls, sin_bhls, n_rot / 2); // [B, H, L, n_rot] if (n_rot < head_dim) { - auto neg_one = ov::op::v0::Constant::create(ov::element::i64, {1}, {-1}); - auto zero = ov::op::v0::Constant::create(ov::element::i64, {1}, {0}); - auto one = ov::op::v0::Constant::create(ov::element::i64, {1}, {1}); - auto n_rot_c = ov::op::v0::Constant::create(ov::element::i64, {1}, {n_rot}); - auto head_c = ov::op::v0::Constant::create(ov::element::i64, {1}, {head_dim}); - rotary_in = std::make_shared(data, zero, n_rot_c, one, neg_one); - pass_through = std::make_shared(data, n_rot_c, head_c, one, neg_one); + roped = std::make_shared(ov::OutputVector{roped, pass_through}, -1); } - // Core split-halves RoPE via the shared decomposition helper: it emits the exact - // split + Multiply(-1)+Add + Concat pattern that ov::pass::RoPEFusion folds into - // ov::op::internal::RoPE (the previous hand-built Subtract form did not match and so - // was never fused). cos/sin already carry the n_rot/2 width. - ov::pass::NodeRegistry reg; - Output rotated = ov::decomposition::rope(reg, rotary_in, cos_theta_node, sin_theta_node, n_rot / 2); - res = (n_rot < head_dim) ? std::make_shared(ov::OutputVector{rotated, pass_through}, -1) - : rotated; + // Back to the gguf layout the rest of the graph expects (the downstream PERMUTE consumes + // rank-4 [B, L, H, S]). + res = std::make_shared( + roped, + ov::op::v0::Constant::create(ov::element::i64, {4}, {0, 2, 1, 3})); // [B, L, H, S] } else if (mode == TYPE_IMROPE) { // Partial rotary (ggml n_dims < head_dim): only the first n_rot dims of every head are // rotated, the tail is passed through unchanged -- e.g. qwen3.5 has head_dim 256 but diff --git a/src/frontends/gguf/src/op/set_rows.cpp b/src/frontends/gguf/src/op/set_rows.cpp index cbb92052c9248b..48bf85f1985ec2 100644 --- a/src/frontends/gguf/src/op/set_rows.cpp +++ b/src/frontends/gguf/src/op/set_rows.cpp @@ -33,7 +33,7 @@ OutputVector translate_set_rows(const NodeContext & context) { auto indices = context.get_input(1); auto dst = context.get_input(2); - data = std::make_shared(data, context.get_attribute("output_type")); + data = std::make_shared(data, context.get_output_type()); // Row size = the destination cache's innermost dim. Using the dst input (not the SET_ROWS // output shape) matters for the flattened KV-cache write (gpt-oss cache_v is stored as @@ -46,10 +46,13 @@ OutputVector translate_set_rows(const NodeContext & context) { auto ind_squeezed = std::make_shared(indices, ov::op::v0::Constant::create(ov::element::i64, {3}, {0, 1, 2})); + // Flatten the new rows to [.., 1, tokens, row_size]. The leading dim is copied from the incoming + // data (special_zero) instead of pinned to 1, so the KV write stays in whichever layout the + // attention block is running in; the stateful lowering re-splits it against the cache shape. auto data_reshaped = std::make_shared( data, - ov::op::v0::Constant::create(ov::element::i64, {4}, {(int64_t) 1, (int64_t) 1, (int64_t) -1, row_size}), - false); + ov::op::v0::Constant::create(ov::element::i64, {4}, {(int64_t)0, (int64_t)1, (int64_t)-1, row_size}), + true); auto set_rows = std::make_shared(data_reshaped, ind_squeezed, dst); return rename_outputs_with_suffix({set_rows}, context.get_name()); diff --git a/src/frontends/gguf/src/op/top_k.cpp b/src/frontends/gguf/src/op/top_k.cpp index 3df1fdce5aef27..19b38fb82f1ae2 100644 --- a/src/frontends/gguf/src/op/top_k.cpp +++ b/src/frontends/gguf/src/op/top_k.cpp @@ -6,6 +6,7 @@ #include "op_table.hpp" #include "openvino/core/node_output.hpp" #include "openvino/op/constant.hpp" +#include "openvino/op/squeeze.hpp" #include "openvino/op/topk.hpp" #include "utils.hpp" @@ -20,16 +21,31 @@ OutputVector translate_top_k(const NodeContext& context) { num_inputs_check(context, 1, 1); auto input = context.get_input(0); - const int64_t k = context.get_output_shape()[context.get_output_shape().size() - 1].get_length(); - auto k_node = ov::op::v0::Constant::create(ov::element::i64, ov::Shape{}, {k}); - auto topk = std::make_shared(input, - k_node, - -1, - ov::op::v11::TopK::Mode::MAX, - ov::op::v11::TopK::SortType::SORT_VALUES, - context.get_attribute("output_type")); - - return rename_outputs_with_suffix({topk->output(1)}, context.get_name()); + + // k is the output's last-axis extent. Prefer the static value, but fall back to reading it off + // the output shape at runtime so a dynamic extent converts instead of throwing (ARGSORT derives + // its k dynamically for the same reason). + const auto& out_ps = context.get_output_shape(); + const auto rank = out_ps.rank(); + const int64_t axis = rank.is_static() ? rank.get_length() - 1 : -1; + ov::Output k_node; + if (rank.is_static() && out_ps[rank.get_length() - 1].is_static()) { + k_node = ov::op::v0::Constant::create(ov::element::i64, + ov::Shape{}, + {out_ps[rank.get_length() - 1].get_length()}); + } else { + k_node = std::make_shared( + get_dimensions(input, {static_cast(rank.is_static() ? rank.get_length() - 1 : 3)}), + ov::op::v0::Constant::create(ov::element::i64, {1}, {0})); + } + + auto indices = make_topk_indices(input, + k_node, + axis, + ov::op::v11::TopK::Mode::MAX, + context.get_attribute("output_type")); + + return rename_outputs_with_suffix({indices}, context.get_name()); } } // namespace op diff --git a/src/frontends/gguf/src/op/view.cpp b/src/frontends/gguf/src/op/view.cpp index c9c3ac3f92cfc9..62a0c021c77f09 100644 --- a/src/frontends/gguf/src/op/view.cpp +++ b/src/frontends/gguf/src/op/view.cpp @@ -2,13 +2,19 @@ // SPDX-License-Identifier: Apache-2.0 // +#include +#include + #include "op_table.hpp" -#include "utils.hpp" +#include "openvino/frontend/exception.hpp" +#include "openvino/op/concat.hpp" #include "openvino/op/constant.hpp" +#include "openvino/op/gather.hpp" #include "openvino/op/reshape.hpp" +#include "openvino/op/shape_of.hpp" #include "openvino/op/slice.hpp" -#include -#include +#include "utils.hpp" + namespace ov { namespace frontend { namespace gguf { @@ -51,10 +57,16 @@ void place_dynamic_token_axis(std::vector & tgt, const ov::PartialShape } } // namespace +// Cases 2-5 are shared by both ingest paths: the llama.cpp cgraph decoder classifies a ggml view +// into them (see ggml-decoder.cpp::compute_op_case) and the native .gguf builder describes its own +// views the same way. Case 104 is the only VIEW case that is builder-only, and not for numbering +// reasons: it takes a second (shape-reference) input the cgraph path does not supply, so it has a +// different arity than the shared cases. See its comment below, and docs/frontend_design.md for the +// other two builder-only cases in the frontend. OutputVector translate_view(const NodeContext & context) { - num_inputs_check(context, 1, 1); + num_inputs_check(context, 1, 2); - if (context.get_attribute("op_case", 0) == 2) { + if (context.get_op_case() == 2) { auto dst_shape = context.get_output_shape().to_shape(); return rename_outputs_with_suffix( {process_view_input(context, 0, static_cast(dst_shape[2] * dst_shape[3]))}, @@ -207,6 +219,52 @@ OutputVector translate_view(const NodeContext & context) { } return rename_outputs_with_suffix({result}, context.get_name()); } + // op_case 104 (builder): layer-index slice for per-layer embedding. + // Input [1, n_layer, T, D] -> slice the layer axis (1) -> [1, 1, T, D]. + if (context.get_op_case() == 104) { + const int64_t layer_idx = context.get_attribute("layer_idx"); + auto input = context.get_input(0); + auto start = ov::op::v0::Constant::create(ov::element::i64, {1}, {layer_idx}); + auto stop = ov::op::v0::Constant::create(ov::element::i64, {1}, {layer_idx + 1}); + auto step = ov::op::v0::Constant::create(ov::element::i64, {1}, {1}); + auto axes = ov::op::v0::Constant::create(ov::element::i64, {1}, {1}); + ov::Output sliced = std::make_shared(input, start, stop, step, axes); + + // The slice comes out with the token count on the axis the per-layer tensor happens to keep it + // on (dim 1 here), because per_layer_embd is stored layer-major so the layer index can be + // sliced off axis 0. Every consumer, though, is an elementwise op against the layer's own + // activation, and OV broadcasts elementwise operands positionally -- so the two operands must + // agree on WHICH leading axis holds the tokens. That is not a fixed choice: it is [1, T, ..] + // under plain SDPA inference and [T, 1, ..] once ov::pass::SDPAToPagedAttention moves the token + // count into dim 0. Both hold the same T*D values contiguously, so when the builder supplies + // the activation as a second (shape-reference) input, reinterpret the slice into that operand's + // leading dims. Without this the multiply below broadcasts to a T x T outer product under PA. + if (context.get_input_size() > 1) { + const auto& ref = context.get_input(1); + const auto ref_rank = ref.get_partial_shape().rank(); + FRONT_END_OP_CONVERSION_CHECK(ref_rank.is_static(), + "VIEW case 104 shape reference must have a static rank"); + const auto d_ps = context.get_output_shape(); + const int64_t rank = ref_rank.get_length(); + FRONT_END_OP_CONVERSION_CHECK(d_ps.rank().is_static() && d_ps[d_ps.rank().get_length() - 1].is_static(), + "VIEW case 104 requires a static per-layer embedding width"); + const int64_t d = d_ps[d_ps.rank().get_length() - 1].get_length(); + + std::vector lead(rank - 1); + for (int64_t i = 0; i < rank - 1; ++i) { + lead[i] = i; + } + auto lead_dims = std::make_shared( + std::make_shared(ref, ov::element::i64), + ov::op::v0::Constant::create(ov::element::i64, {lead.size()}, lead), + ov::op::v0::Constant::create(ov::element::i64, {}, {0})); + auto target = std::make_shared( + ov::OutputVector{lead_dims, ov::op::v0::Constant::create(ov::element::i64, {1}, {d})}, + 0); + sliced = std::make_shared(sliced, target, false); + } + return rename_outputs_with_suffix({sliced.get_node_shared_ptr()}, context.get_name()); + } return {context.get_input(0)}; } diff --git a/src/frontends/gguf/src/op/weight.cpp b/src/frontends/gguf/src/op/weight.cpp index a1106c7ef6b981..8c5e7f6f2dc3b7 100644 --- a/src/frontends/gguf/src/op/weight.cpp +++ b/src/frontends/gguf/src/op/weight.cpp @@ -5,12 +5,14 @@ #include #include #include -#include "openvino/op/constant.hpp" -#include "openvino/op/reshape.hpp" +#include #include #include "node_context.hpp" #include "op_table.hpp" +#include "openvino/op/constant.hpp" +#include "openvino/op/reshape.hpp" +#include "openvino/op/unsqueeze.hpp" #include "quant/weights.hpp" #include "utils.hpp" @@ -20,13 +22,45 @@ namespace gguf { namespace op { // A GGUF weight surfaced as a node. A weight is a ggml leaf (op type "GGML_OP_NONE") that the -// decoder marks by exposing a "data" attribute (the raw weight bytes), alongside the ggml quant -// type name and the logical shape. The frontend does all dequant / repacking here, so the -// decoder never builds OV nodes itself. (Model-input leaves are also GGML_OP_NONE, but they are -// resolved to Parameters before the graph walk and never reach this translator.) +// decoder marks as a weight. Two payload shapes are supported, both dequantized here so the +// decoder never builds OV nodes itself: +// +// 1. Native .gguf builder path: the parser already extracted the weight into OpenVINO tensors +// (weight [+ scales [+ zp]]); the node carries them as attributes "gguf.blob." plus the +// quant type id "gguf_qtype" (and marker "gguf_weight"). We rebuild the make_weight_node( +// base, weights, qtypes) inputs and call that overload -- the exact dequant path the builder +// used before, unchanged numerics, and it handles fused-QKV parts / MoE experts uniformly. +// +// 2. llama.cpp cgraph path: the node carries the raw ggml bytes in "data" plus the ggml type +// name "quant_type"; make_weight_node(data, quant_type, shape) re-extracts and builds. This +// path also handles the MoE MXFP4 packed / rank>2 expert-weight layouts. +// +// (Model-input leaves are also GGML_OP_NONE, but they are resolved to Parameters before the graph +// walk and never reach this translator.) OutputVector translate_weight(const NodeContext& context) { + // Path 1: pre-extracted tensors from the native builder. + if (context.get_attribute("gguf_weight", false)) { + const std::string base = "weight"; + std::unordered_map weights; + for (const char* sub : {"weight", "scales", "zp"}) { + // scales/zp are absent for plain/symmetric types -> defaulted get_attribute (empty + // tensor) so the missing-key Any doesn't throw. + auto blob = context.get_attribute(std::string("gguf.blob.") + sub, ov::Tensor()); + if (blob) { + weights[base + "." + sub] = blob; + } + } + FRONT_END_OP_CONVERSION_CHECK(weights.count(base + ".weight"), + "GGML_OP_NONE weight leaf has no 'gguf.blob.weight' attribute"); + auto qtype = static_cast(context.get_attribute("gguf_qtype")); + std::unordered_map qtypes{{base + ".qtype", qtype}}; + auto node = make_weight_node(base, weights, qtypes); + return rename_outputs_with_suffix({node}, context.get_name()); + } + + // Path 2: raw ggml bytes from a live cgraph decoder. auto data = context.get_attribute("data"); - FRONT_END_OP_CONVERSION_CHECK(data, "GGML_OP_NONE node has no 'data' attribute; not a weight"); + FRONT_END_OP_CONVERSION_CHECK(data, "GGML_OP_NONE node has no weight payload; not a weight"); auto quant_type = context.get_attribute("quant_type"); auto shape = context.get_output_shape().to_shape(); diff --git a/src/frontends/gguf/src/op_table.cpp b/src/frontends/gguf/src/op_table.cpp index f3e6cc3f040acf..b6afba509ee16d 100644 --- a/src/frontends/gguf/src/op_table.cpp +++ b/src/frontends/gguf/src/op_table.cpp @@ -5,6 +5,7 @@ #include "op_table.hpp" #include "openvino/op/add.hpp" +#include "openvino/op/concat.hpp" #include "openvino/op/cos.hpp" #include "openvino/op/divide.hpp" #include "openvino/op/exp.hpp" @@ -54,13 +55,12 @@ std::unordered_map get_supported_ops() { {"GGML_OP_MUL", op::translate_1to1_match_2_inputs}, {"GGML_OP_MUL_MAT", op::translate_mulmat}, {"GGML_OP_MUL_MAT_ID", op::translate_mul_mat_id}, - // A GGML_OP_NONE leaf carrying a "data" attribute is a weight (see translate_weight). {"GGML_OP_NONE", op::translate_weight}, {"GGML_OP_NORM", op::translate_norm}, {"GGML_OP_PAD", op::translate_pad}, {"GGML_OP_PERMUTE", op::translate_permute}, - {"GGML_OP_RESHAPE", op::translate_reshape}, {"GGML_OP_REPEAT", op::translate_repeat}, + {"GGML_OP_RESHAPE", op::translate_reshape}, {"GGML_OP_RMS_NORM", op::translate_rms_norm}, {"GGML_OP_ROPE", op::translate_rope}, {"GGML_OP_SCALE", op::translate_scale}, @@ -74,8 +74,8 @@ std::unordered_map get_supported_ops() { {"GGML_OP_SUB", op::translate_1to1_match_2_inputs}, {"GGML_OP_SUM_ROWS", op::translate_sum_rows}, {"GGML_OP_TOP_K", op::translate_top_k}, - {"GGML_OP_TRI", op::translate_tri}, {"GGML_OP_TRANSPOSE", op::translate_transpose}, + {"GGML_OP_TRI", op::translate_tri}, {"GGML_OP_VIEW", op::translate_view}, {"GGML_UNARY_OP_ELU", op::translate_unary_elu}, {"GGML_UNARY_OP_EXP", op::translate_1to1_match_1_input}, diff --git a/src/frontends/gguf/src/op_table.hpp b/src/frontends/gguf/src/op_table.hpp index 39ceb369046891..6c17d3bfb033c3 100644 --- a/src/frontends/gguf/src/op_table.hpp +++ b/src/frontends/gguf/src/op_table.hpp @@ -14,49 +14,76 @@ namespace op { #define GGUF_OP_CONVERTER(op) OutputVector op(const NodeContext& context) -GGUF_OP_CONVERTER(translate_add_id); -GGUF_OP_CONVERTER(translate_argsort); -GGUF_OP_CONVERTER(translate_clamp); +// Structural / memory ops. GGUF_OP_CONVERTER(translate_concat); GGUF_OP_CONVERTER(translate_cont); -GGUF_OP_CONVERTER(translate_cumsum); -GGUF_OP_CONVERTER(translate_diag); -GGUF_OP_CONVERTER(translate_div); -GGUF_OP_CONVERTER(translate_fill); -GGUF_OP_CONVERTER(translate_gated_delta_net); +GGUF_OP_CONVERTER(translate_cpy); GGUF_OP_CONVERTER(translate_get_rows); -GGUF_OP_CONVERTER(translate_im2col); -GGUF_OP_CONVERTER(translate_l2_norm); -GGUF_OP_CONVERTER(translate_norm); -GGUF_OP_CONVERTER(translate_pad); -GGUF_OP_CONVERTER(translate_repeat); -GGUF_OP_CONVERTER(translate_ssm_conv); -GGUF_OP_CONVERTER(translate_mulmat); -GGUF_OP_CONVERTER(translate_mul_mat_id); GGUF_OP_CONVERTER(translate_permute); +GGUF_OP_CONVERTER(translate_repeat); GGUF_OP_CONVERTER(translate_reshape); +GGUF_OP_CONVERTER(translate_set); +GGUF_OP_CONVERTER(translate_set_rows); +GGUF_OP_CONVERTER(translate_transpose); +GGUF_OP_CONVERTER(translate_view); + +// Normalization. +GGUF_OP_CONVERTER(translate_norm); GGUF_OP_CONVERTER(translate_rms_norm); +GGUF_OP_CONVERTER(translate_l2_norm); + +// Matmul / attention. +GGUF_OP_CONVERTER(translate_mulmat); +GGUF_OP_CONVERTER(translate_flash_attn_ext); +GGUF_OP_CONVERTER(translate_soft_max); GGUF_OP_CONVERTER(translate_rope); GGUF_OP_CONVERTER(translate_scale); -GGUF_OP_CONVERTER(translate_set); -GGUF_OP_CONVERTER(translate_sqr); -GGUF_OP_CONVERTER(translate_sqrt); + +// Gated linear units. +GGUF_OP_CONVERTER(translate_glu_geglu); +GGUF_OP_CONVERTER(translate_glu_swiglu); +GGUF_OP_CONVERTER(translate_glu_swiglu_oai); + +// MoE (mixture-of-experts) routing ops. +GGUF_OP_CONVERTER(translate_mul_mat_id); +GGUF_OP_CONVERTER(translate_add_id); +GGUF_OP_CONVERTER(translate_argsort); GGUF_OP_CONVERTER(translate_top_k); -GGUF_OP_CONVERTER(translate_tri); GGUF_OP_CONVERTER(translate_sum_rows); + +// Elementwise clamp and division. +GGUF_OP_CONVERTER(translate_clamp); +GGUF_OP_CONVERTER(translate_div); + +// Unary activations. GGUF_OP_CONVERTER(translate_unary_silu); GGUF_OP_CONVERTER(translate_unary_gelu); GGUF_OP_CONVERTER(translate_unary_gelu_quick); +GGUF_OP_CONVERTER(translate_unary_relu); +GGUF_OP_CONVERTER(translate_unary_tanh); +GGUF_OP_CONVERTER(translate_unary_sigmoid); GGUF_OP_CONVERTER(translate_unary_elu); -GGUF_OP_CONVERTER(translate_soft_max); -GGUF_OP_CONVERTER(translate_transpose); -GGUF_OP_CONVERTER(translate_view); -GGUF_OP_CONVERTER(translate_glu_swiglu); -GGUF_OP_CONVERTER(translate_glu_swiglu_oai); -GGUF_OP_CONVERTER(translate_glu_geglu); -GGUF_OP_CONVERTER(translate_set_rows); -GGUF_OP_CONVERTER(translate_cpy); -GGUF_OP_CONVERTER(translate_flash_attn_ext); + +// Unary element-wise math. +GGUF_OP_CONVERTER(translate_sqr); +GGUF_OP_CONVERTER(translate_sqrt); +GGUF_OP_CONVERTER(translate_log); +GGUF_OP_CONVERTER(translate_sin); +GGUF_OP_CONVERTER(translate_cos); +GGUF_OP_CONVERTER(translate_cumsum); + +// Matrix-shaped helpers. +GGUF_OP_CONVERTER(translate_diag); +GGUF_OP_CONVERTER(translate_tri); +GGUF_OP_CONVERTER(translate_fill); + +// Convolution-family / sequence ops. +GGUF_OP_CONVERTER(translate_im2col); +GGUF_OP_CONVERTER(translate_pad); +GGUF_OP_CONVERTER(translate_ssm_conv); +GGUF_OP_CONVERTER(translate_gated_delta_net); + +// A GGML_OP_NONE leaf carrying a "data" attribute -> dequantized weight node (cgraph path). GGUF_OP_CONVERTER(translate_weight); } // namespace op diff --git a/src/frontends/gguf/src/pass/adapt_to_genai.cpp b/src/frontends/gguf/src/pass/adapt_to_genai.cpp new file mode 100644 index 00000000000000..925f24c9e9f4a1 --- /dev/null +++ b/src/frontends/gguf/src/pass/adapt_to_genai.cpp @@ -0,0 +1,283 @@ +// Copyright (C) 2018-2026 Intel Corporation +// SPDX-License-Identifier: Apache-2.0 +// + +#include "openvino/frontend/gguf/adapt_to_genai.hpp" +#include "openvino/frontend/gguf/make_stateful.hpp" + +#include +#include + +#include "openvino/op/broadcast.hpp" +#include "openvino/op/concat.hpp" +#include "openvino/op/constant.hpp" +#include "openvino/op/convert.hpp" +#include "openvino/op/gather.hpp" +#include "openvino/op/less_eq.hpp" +#include "openvino/op/parameter.hpp" +#include "openvino/op/range.hpp" +#include "openvino/op/read_value.hpp" +#include "openvino/op/reduce_prod.hpp" +#include "openvino/op/reshape.hpp" +#include "openvino/op/tile.hpp" +#include "openvino/op/result.hpp" +#include "openvino/op/scaled_dot_product_attention.hpp" +#include "openvino/op/select.hpp" +#include "openvino/op/shape_of.hpp" +#include "openvino/op/squeeze.hpp" +#include "openvino/op/subtract.hpp" +#include "openvino/op/transpose.hpp" +#include "openvino/runtime/properties.hpp" + +namespace ov { +namespace frontend { +namespace gguf { +namespace pass { + +namespace { + +using std::make_shared; + +// f16 lowest, matches genai's causal-mask "-inf" fill. +constexpr float NEG_INF = -65504.0f; + +std::shared_ptr const_i64(const std::vector& values) { + return ov::op::v0::Constant::create(ov::element::i64, ov::Shape{values.size()}, values); +} + +// Find a Parameter whose output tensor names (or friendly name) match `name`. +std::shared_ptr find_param(const std::shared_ptr& model, + const std::string& name) { + for (const auto& p : model->get_parameters()) { + const auto& names = p->output(0).get_names(); + if (names.count(name) || p->get_friendly_name() == name) { + return p; + } + } + return nullptr; +} + +void name_output(const ov::Output& out, const std::string& name) { + out.get_node_shared_ptr()->set_friendly_name(name); + out.get_node_shared_ptr()->output(0).set_names({name}); +} + +// Largest attention head size across the stateful KV caches (the ReadValue last dim). The +// frontend emits f16 KV caches mirroring llama.cpp, but the CPU plugin defaults +// KV_CACHE_PRECISION to u8 (dynamic-quantized) -- faster and accurate enough for the common +// head sizes used by llama/qwen/phi3/gpt-oss (64-128). For large head sizes the u8 +// quantization injects enough per-step error to compound across autoregressive decode into +// divergence and eventually NaN (observed on gemma4, global-attention head_size=512). +int64_t max_kv_cache_head_size(const std::shared_ptr& model) { + int64_t max_hs = 0; + for (const auto& op : model->get_ops()) { + if (!ov::as_type_ptr(op)) { + continue; + } + const auto& ps = op->get_output_partial_shape(0); + if (ps.rank().is_static() && ps[ps.rank().get_length() - 1].is_static()) { + max_hs = std::max(max_hs, ps[ps.rank().get_length() - 1].get_length()); + } + } + return max_hs; +} + +} // namespace + +bool AdaptToGenAI::run_on_model(const std::shared_ptr& model) { + OPENVINO_ASSERT(m_mode == InputMode::IdsToLogits, + "[gguf] AdaptToGenAI: only InputMode::IdsToLogits is implemented; " + "EmbedsToLogits (VLM language model) is reserved for future work."); + + // The gguf inputs we rewire. inp_tokens/inp_pos/self_kq_mask/token_len_per_seq are + // required; if they are absent the model is not a gguf-IO model (e.g. already adapted), + // so this pass is a no-op. + auto inp_tokens = find_param(model, "inp_tokens"); + auto inp_pos = find_param(model, "inp_pos"); + auto self_kq_mask = find_param(model, "self_kq_mask"); + auto token_len_per_seq = find_param(model, "token_len_per_seq"); + if (!inp_tokens || !inp_pos || !self_kq_mask || !token_len_per_seq) { + return false; + } + + // ---- new genai inputs: input_ids / attention_mask / position_ids [b, seq] i64 ---- + auto input_ids = make_shared(ov::element::i64, ov::PartialShape{-1, -1}); + name_output(input_ids, "input_ids"); + auto attention_mask = make_shared(ov::element::i64, ov::PartialShape{-1, -1}); + name_output(attention_mask, "attention_mask"); + auto position_ids = make_shared(ov::element::i64, ov::PartialShape{-1, -1}); + name_output(position_ids, "position_ids"); + + // beam_idx (i32 [D]) is added by the make-stateful pass, next to the Gather that reads it; genai + // sets it via set_tensor("beam_idx"). Keep that Parameter so its wiring is preserved. Its absence + // means the model is not stateful, which the genai contract requires. + auto beam_idx = find_param(model, "beam_idx"); + OPENVINO_ASSERT(beam_idx, + "[gguf] AdaptToGenAI: model has no 'beam_idx' input, so it is not stateful. " + "Register a make-stateful transformation extension (e.g. " + "ov::frontend::gguf::pass::MakeStateful) before converting."); + + // ---- token_len_per_seq = number of tokens in input_ids -> [1] ---- + // The token count is the ELEMENT COUNT of input_ids, not any single dimension of it. genai feeds + // [batch, seq] (batch == 1), but SDPAToPagedAttention rewrites this Parameter to rank-1 [tokens] + // and splices an Unsqueeze(axis=1) in front of its consumers, making it [tokens, 1]. Reading + // dim 1 would then yield 1 for every prompt, collapsing the causal mask and the logits to a + // single token; reading dim 0 breaks the un-rewritten case. ReduceProd is correct under both. + auto ids_shape = make_shared(input_ids, ov::element::i64); + auto seq_len = make_shared(ids_shape, const_i64({0}), true); // [1] + token_len_per_seq->output(0).replace(seq_len->output(0)); + + // The two gguf rank-4 input kinds carry the (batch, tokens) pair on different axes, so they get + // different lifts. Both are written so the genai Parameter's own leading dims flow through + // instead of being replaced by literals -- that is the whole mechanism by which one graph serves + // both attention backends (see the layout note on the class). + // + // INDEX vectors (inp_tokens, inp_out_ids): consumed by get_rows, which squeezes the two leading + // axes and gathers rows. The gathered result inherits the indices' trailing 2D shape, so the + // indices must present exactly the Parameter's own [batch, seq]: prepend two 1s. + // SDPA: [1,1,1,tokens] -> squeeze -> [1,tokens] -> embd [1,tokens,n_embd] + // PA : [1,1,tokens,1] -> squeeze -> [tokens,1] -> embd [tokens,1,n_embd] + const auto shape_1_1_batch_seq = make_shared(ov::OutputVector{const_i64({1, 1}), ids_shape}, 0); + + // ACTIVATION-like inputs (inp_pos): consumed by make_sin_cos, which transposes {0,3,1,2} to put + // the token axis at 1, yielding cos/sin [batch, tokens, 1, n_rot/2] that broadcast against the + // roped [batch, heads, tokens, head_size]. special_zero's 0 copies dim 0 and -1 absorbs the rest. + // SDPA: [1,1,1,tokens] -> cos/sin [1,tokens,1,half] + // PA : [tokens,1,1,1] -> cos/sin [tokens,1,1,half], which broadcasts against [tokens,H,1,S] + const auto shape_keep0_1_1_rest = const_i64({0, 1, 1, -1}); + + auto tokens_i32 = make_shared(input_ids, ov::element::i32); + auto tokens_4d = make_shared(tokens_i32, shape_1_1_batch_seq, false); + inp_tokens->output(0).replace(tokens_4d->output(0)); + + ov::Output pos_i32 = make_shared(position_ids, ov::element::i32); + // M-RoPE (qwen35): inp_pos carries FOUR position sections per token, laid out section-major -- + // make_sin_cos reshapes it to {..,4,tokens} and transposes. GenAI supplies one position per + // token, so tile it 4x along the token axis. All four sections hold the same value here: the + // per-section split only differs for image/video input, and a text-only prompt has no spatial + // axes to differ on (llama.cpp fills all sections with the text position likewise). + if (model->get_rt_info().count(gguf_imrope_key())) { + auto tile_repeats = const_i64({1, 4}); + pos_i32 = make_shared(pos_i32, tile_repeats); + } + auto pos_4d = make_shared(pos_i32, shape_keep0_1_1_rest, true); + inp_pos->output(0).replace(pos_4d->output(0)); + + // ---- self_kq_mask [1,1,seq,kv_len] f32: 0 where attended, -inf above causal ---- + // kv_len = attention_mask length (= past + seq). query absolute positions = position_ids[0]. + auto am_shape = make_shared(attention_mask, ov::element::i64); + auto kv_len = make_shared(am_shape, const_i64({1}), const_i64({0})); // [1] + + // Flatten position_ids to [seq] via a shape-independent Reshape({-1}) rather than Squeeze(axis=0): + // PA also rewrites position_ids to rank-1 and Unsqueezes it to [seq,1], where squeezing axis 0 + // would fail (or drop the wrong axis). + auto q_pos = + make_shared(make_shared(position_ids, const_i64({-1}), false), + ov::element::i32); // [seq] + auto q_pos_col = + make_shared(q_pos, + make_shared(ov::OutputVector{seq_len, const_i64({1})}, 0), + false); // [seq, 1] + + auto zero_i32 = ov::op::v0::Constant::create(ov::element::i32, ov::Shape{}, {0}); + auto one_i32 = ov::op::v0::Constant::create(ov::element::i32, ov::Shape{}, {1}); + auto kv_len_i32 = make_shared(make_shared(kv_len, ov::element::i32), + const_i64({0})); // scalar + auto k_range = make_shared(zero_i32, kv_len_i32, one_i32, ov::element::i32); // [kv_len] + auto k_row = + make_shared(k_range, + make_shared(ov::OutputVector{const_i64({1}), kv_len}, 0), + false); // [1, kv_len] + + auto allowed = make_shared(k_row, q_pos_col); // [seq, kv_len] bool + auto zero_f = ov::op::v0::Constant::create(ov::element::f32, ov::Shape{}, {0.0f}); + auto neg_f = ov::op::v0::Constant::create(ov::element::f32, ov::Shape{}, {NEG_INF}); + auto mask2d = make_shared(allowed, zero_f, neg_f); // [seq, kv_len] f32 + auto mask_4d = make_shared( + mask2d, + make_shared(ov::OutputVector{const_i64({1, 1}), seq_len, kv_len}, 0), + false); // [1, 1, seq, kv_len] + self_kq_mask->output(0).replace(mask_4d->output(0)); + + // gpt-oss sliding-window mask: for prompts within the window it equals the full causal + // mask, so the same value is correct here. + if (auto self_kq_mask_swa = find_param(model, "self_kq_mask_swa")) { + self_kq_mask_swa->output(0).replace(mask_4d->output(0)); + } + + // inp_out_ids selects which rows the output head runs on. Emit the LAST row only: genai reads + // just the final token's logits, so projecting every prompt position to vocab costs an extra + // (tokens - 1) x hidden x vocab matmul per prefill. + // + // get_rows lowers to Gather(activation, ids, axis=1, batch_dims=1), i.e. act[i, ids[i, j]], so + // the last row is ids == ids_shape[1] - 1 in an [ids_shape[0], 1] vector. That is correct in + // both layouts: default ids_shape is [1, tokens], giving [1,1] holding tokens - 1; under + // SDPAToPagedAttention it is [tokens, 1], so this evaluates to 0 -- the identity that layout + // needs, since it already carries one token per row. + if (auto inp_out_ids = find_param(model, "inp_out_ids")) { + auto batch_dim = + make_shared(ids_shape, const_i64({0}), const_i64({0})); // [1]: ids_shape[0] + auto seq_dim = + make_shared(ids_shape, const_i64({1}), const_i64({0})); // [1]: ids_shape[1] + auto last_index = make_shared( + make_shared(seq_dim, const_i64({1})), + ov::element::i32); // [1]: ids_shape[1] - 1 + auto out_grid = make_shared( + last_index, + make_shared(ov::OutputVector{batch_dim, const_i64({1})}, 0)); // [batch, 1] + auto out_ids = make_shared( + out_grid, + make_shared(ov::OutputVector{const_i64({1, 1}), batch_dim, const_i64({1})}, 0), + false); + inp_out_ids->output(0).replace(out_ids->output(0)); + } + + // ---- logits: rank-4 [.., .., .., vocab] -> [b, seq, vocab] ---- + // genai always wants [batch, seq, vocab] regardless of which axis the body kept the tokens on, + // and both layouts hold seq*vocab contiguous values, so collapse everything ahead of vocab into + // the sequence axis with a fixed batch of 1. (batch > 1 is not part of the genai stateful + // contract this pass targets; token_len_per_seq above is likewise a whole-input token count.) + auto old_result = model->get_results()[0]; + auto logits_src = old_result->input_value(0); + auto vocab = make_shared(make_shared(logits_src, ov::element::i64), + const_i64({-1}), + const_i64({0})); // [1] + auto logits_3d = make_shared( + logits_src, + make_shared(ov::OutputVector{const_i64({1, -1}), vocab}, 0), + false); // [1, seq, vocab] + name_output(logits_3d, "logits"); + auto new_result = make_shared(logits_3d); + new_result->set_friendly_name("logits"); + + model->add_results({new_result}); + model->remove_result(old_result); + + // Swap the input list to the genai contract. beam_idx is kept as-is; every other old + // gguf Parameter has had its output rewired (consumers now read the derived subgraph), + // so removing it is safe. + model->add_parameters({input_ids, attention_mask, position_ids}); + const auto params_snapshot = model->get_parameters(); // copy: remove_parameter mutates the list + for (const auto& p : params_snapshot) { + if (p == input_ids || p == attention_mask || p == position_ids || p == beam_idx) { + continue; + } + model->remove_parameter(p); + } + + // Pin the runtime KV-cache precision to f16 for large-head models so decode matches both + // prefill and llama.cpp; mainstream small-head models keep the faster u8 default. This is a + // consumer-side optimization policy (genai), not a property the frontend bakes into the model. + constexpr int64_t kU8SafeHeadSize = 128; + if (max_kv_cache_head_size(model) > kU8SafeHeadSize) { + model->set_rt_info(ov::element::f16, {"runtime_options", ov::hint::kv_cache_precision.name()}); + } + + model->validate_nodes_and_infer_types(); + return true; +} + +} // namespace pass +} // namespace gguf +} // namespace frontend +} // namespace ov diff --git a/src/frontends/gguf/src/pass/make_stateful.cpp b/src/frontends/gguf/src/pass/make_stateful.cpp new file mode 100644 index 00000000000000..c4ce359b3e64fc --- /dev/null +++ b/src/frontends/gguf/src/pass/make_stateful.cpp @@ -0,0 +1,309 @@ +// Copyright (C) 2018-2026 Intel Corporation +// SPDX-License-Identifier: Apache-2.0 +// + +#include "openvino/frontend/gguf/make_stateful.hpp" + +#include +#include +#include + +#include "openvino/core/graph_util.hpp" +#include "openvino/frontend/gguf/set_rows_op.hpp" +#include "openvino/op/assign.hpp" +#include "openvino/op/concat.hpp" +#include "openvino/op/constant.hpp" +#include "openvino/op/gather.hpp" +#include "openvino/op/parameter.hpp" +#include "openvino/op/read_value.hpp" +#include "openvino/op/reshape.hpp" +#include "openvino/op/result.hpp" +#include "openvino/op/util/variable.hpp" + +namespace ov::frontend::gguf::pass { + +namespace { + +std::shared_ptr find_param(const std::shared_ptr& model, const std::string& name) { + for (const auto& p : model->get_parameters()) { + if (p->get_friendly_name() == name || p->output(0).get_names().count(name)) { + return p; + } + } + return nullptr; +} + +// The axis the cache grows along. An un-preallocated cache Parameter states it by construction: it +// is the one dynamic axis (the token count), every other being a static batch / head / head-size. +// A caller that preallocates the cache must say which axis it is, since none is dynamic then. +int64_t resolve_append_axis(const ov::PartialShape& ps, const std::string& cache_name, int64_t requested) { + const int64_t rank = ps.rank().get_length(); + if (requested >= 0) { + OPENVINO_ASSERT(requested < rank, + "[gguf] MakeStateful: append axis ", + requested, + " is out of range for cache '", + cache_name, + "' of shape ", + ps); + return requested; + } + int64_t axis = -1; + for (int64_t i = 0; i < rank; ++i) { + if (ps[i].is_dynamic()) { + OPENVINO_ASSERT(axis < 0, + "[gguf] MakeStateful: cache '", + cache_name, + "' has shape ", + ps, + " with more than one dynamic axis, so its token axis cannot be inferred; " + "construct the pass with an explicit append_axis"); + axis = i; + } + } + OPENVINO_ASSERT(axis >= 0, + "[gguf] MakeStateful: cache '", + cache_name, + "' has the fully static shape ", + ps, + ", so its token axis cannot be inferred; construct the pass with an explicit append_axis"); + return axis; +} + +} // namespace + +const std::string& gguf_recurrent_states_key() { + static const std::string key = "gguf_recurrent_states"; + return key; +} + +const std::string& gguf_imrope_key() { + static const std::string key = "gguf_is_imrope"; + return key; +} + +// Rewrite the recurrent (overwritten, non-appending) states into OpenVINO Variables. +// +// A KV cache is found by walking the SetRows writes and is grown with a Concat along its token +// axis. A recurrent state has neither: it is read whole at the start of a step and replaced whole +// at the end, so the graph carries nothing that marks it and the pairing arrives via rt_info (see +// gguf_recurrent_states_key). The rewrite is correspondingly simpler -- ReadValue -> ... -> Assign +// with no Concat, and no beam Gather, since there is no past to reorder. +// +// Their shapes are fully static, so the init is a real zeros Constant. Zero is also the correct +// initial value: ggml starts a sequence with a zeroed conv window and delta matrix. +static bool make_recurrent_states_stateful(const std::shared_ptr& model) { + auto it = model->get_rt_info().find(gguf_recurrent_states_key()); + if (it == model->get_rt_info().end()) { + return false; + } + const auto flat = it->second.as>(); + if (flat.empty() || flat.size() % 2 != 0) { + return false; + } + + ov::ParameterVector params_to_remove; + ov::ResultVector results_to_remove; + ov::SinkVector new_sinks; + + for (size_t i = 0; i + 1 < flat.size(); i += 2) { + const std::string& in_name = flat[i]; + const std::string& out_name = flat[i + 1]; + + auto param = find_param(model, in_name); + if (!param) { + continue; // already rewritten (a second run of this pass) + } + const auto& ps = param->get_partial_shape(); + OPENVINO_ASSERT(ps.is_static(), + "[gguf] MakeStateful: recurrent state '", + in_name, + "' must have a fully static shape, got ", + ps); + + // The Result holding this state's new value: the one whose producing node carries the + // state's output name. The builder names that node after the state (see the VIEW cases), + // which is why those names have to survive translation. + std::shared_ptr state_result; + for (const auto& r : model->get_results()) { + const auto producer = r->get_input_node_shared_ptr(0); + if (producer->get_friendly_name().find(out_name) != std::string::npos) { + state_result = r; + break; + } + } + OPENVINO_ASSERT(state_result, "[gguf] MakeStateful: no Result produces recurrent state '", out_name, "'"); + + const auto et = param->get_element_type(); + auto var = std::make_shared(ov::op::util::VariableInfo{ps, et, in_name}); + auto init = ov::op::v0::Constant::create(et, ps.to_shape(), std::vector(1, 0.0f)); + auto read_value = std::make_shared(init, var); + read_value->set_friendly_name(in_name); + ov::replace_node(param, read_value); + + new_sinks.push_back(std::make_shared(state_result->input_value(0), var)); + model->add_variables({var}); + results_to_remove.push_back(state_result); + params_to_remove.push_back(param); + } + + if (params_to_remove.empty()) { + return false; + } + for (const auto& r : results_to_remove) { + model->remove_result(r); + } + model->add_sinks(new_sinks); + for (const auto& p : params_to_remove) { + model->remove_parameter(p); + } + return true; +} + +bool MakeStateful::run_on_model(const std::shared_ptr& model) { + // beam_idx reorders the past cache along the batch axis for beam search. With batch 1 / + // beam_idx [0] the Gather is an identity, but emitting it is what lets CPU's + // stateful_sdpa_fusion match. + // + // It belongs to the STATE, so this pass owns it: it is a beam-search index into an OpenVINO + // cache, which ggml has no equivalent of, so no decoder should declare it -- a decoder that did + // would give the stateless graph an input with no consumer, and the two decoders different + // stateless IO. Created here, next to its only consumer (the Gather below). A model that + // already has one (a caller that declared it, or a second run of this pass) keeps it. + auto beam_idx = find_param(model, m_beam_idx_name); + const bool created_beam_idx = beam_idx == nullptr; + if (created_beam_idx) { + beam_idx = std::make_shared(ov::element::i32, ov::PartialShape{ov::Dimension()}); + beam_idx->set_friendly_name(m_beam_idx_name); + beam_idx->output(0).set_names({m_beam_idx_name}); + } + + // Only a SetRows writing into a model Parameter is a cache write; the rest (e.g. MoE routing + // writes) are left to the default stateless lowering that runs after this pass. Collect first, + // then rewrite, so the graph is not mutated while being walked. + std::vector> cache_writes; + for (const auto& node : model->get_ops()) { + auto set_rows = ov::as_type_ptr(node); + if (!set_rows) { + continue; + } + auto dst = ov::as_type_ptr(set_rows->input_value(2).get_node_shared_ptr()); + if (dst && !m_skip_caches.count(dst->get_friendly_name())) { + cache_writes.push_back(set_rows); + } + } + // A model can have recurrent states and no KV cache at all (an all-linear-attention stack), + // so the recurrent rewrite must not sit behind this early return. + if (cache_writes.empty()) { + return make_recurrent_states_stateful(model); + } + + ov::ParameterVector params_to_remove; + ov::ResultVector results_to_remove; + ov::SinkVector new_sinks; + + for (const auto& set_rows : cache_writes) { + auto new_rows = set_rows->input_value(0); + auto cache_param = ov::as_type_ptr(set_rows->input_value(2).get_node_shared_ptr()); + const auto& cache_name = cache_param->get_friendly_name(); + const auto& ps = cache_param->get_partial_shape(); + const auto et = cache_param->get_element_type(); + OPENVINO_ASSERT(ps.rank().is_static(), + "[gguf] MakeStateful requires a static cache rank, got ", + ps, + " for '", + cache_name, + "'"); + const int64_t axis = resolve_append_axis(ps, cache_name, m_append_axis); + + // The state holds however many tokens have accumulated, so the append axis is dynamic on the + // Variable and its initial extent is 0 (no past on the first inference). Every other axis + // keeps the Parameter's declared dimension and so must be static to build the init constant. + ov::PartialShape var_shape = ps; + var_shape[axis] = ov::Dimension::dynamic(); + auto var = std::make_shared(ov::op::util::VariableInfo{var_shape, et, cache_name}); + + ov::Shape init_shape; + for (int64_t i = 0; i < ps.rank().get_length(); ++i) { + if (i == axis) { + init_shape.push_back(0); + continue; + } + OPENVINO_ASSERT(ps[i].is_static(), + "[gguf] MakeStateful requires static non-token cache dims, got ", + ps, + " for '", + cache_name, + "'"); + init_shape.push_back(static_cast(ps[i].get_length())); + } + // Empty init: required, not cosmetic -- CPU's MemoryInputSDPA aborts on a MemoryInput with + // zero parent edges (see the header note). + auto init = ov::op::v0::Constant::create(et, init_shape, std::vector{}); + auto read_value = std::make_shared(init, var); + + // The SetRows placeholder presents the new rows flattened to [.., 1, tokens, row_size] (see + // translate_set_rows), which need not be the cache's own split of those same elements -- e.g. + // a [1, tokens, n_head_kv, head_size] cache receives [1, 1, tokens, n_head_kv*head_size]. So + // re-split them against the cache layout before the Concat: the token axis is -1, the axes + // after it take the cache's static dims, and the axes before it are copied from the incoming + // data (special_zero's 0) rather than pinned to literals, which is what keeps this valid in + // the token-major layout ov::pass::SDPAToPagedAttention establishes. + std::vector split_pattern; + for (int64_t i = 0; i < ps.rank().get_length(); ++i) { + split_pattern.push_back(i < axis ? 0 : (i == axis ? -1 : ps[i].get_length())); + } + new_rows = std::make_shared( + new_rows, + ov::op::v0::Constant::create(ov::element::i64, {split_pattern.size()}, split_pattern), + true); + + // Reorder the past by beam_idx before appending, so each beam continues its own history. + auto axis0 = ov::op::v0::Constant::create(ov::element::i64, ov::Shape{}, {0}); + auto past = std::make_shared(read_value, beam_idx, axis0); + auto concat = std::make_shared(ov::OutputVector{past, new_rows}, axis); + concat->set_friendly_name(set_rows->get_friendly_name()); + new_sinks.push_back(std::make_shared(concat, var)); + + // The stateless graph returns each updated cache as a Result; in the stateful form the Assign + // sink above takes that role, so those Results go. Identify them as the Results reading THIS + // write -- not by matching the cache's name, which only happens to work while the builder + // names a cache's write after the cache itself. Collect them before replace_node, while the + // SetRows is still the node they read. + for (const auto& consumer : set_rows->output(0).get_target_inputs()) { + if (auto r = ov::as_type_ptr(consumer.get_node()->shared_from_this())) { + results_to_remove.push_back(r); + } + } + + // Every remaining consumer of the write (attention's read of the cache) now reads the grown + // Concat instead. + ov::replace_node(set_rows, concat); + model->add_variables({var}); + + params_to_remove.push_back(cache_param); + } + + // The cache is no longer part of the model's IO: its Parameter is now a ReadValue and its Result + // an Assign sink. Remove the Results first so the Parameters have no consumers left. + for (const auto& r : results_to_remove) { + model->remove_result(r); + } + model->add_sinks(new_sinks); + // Only now, having actually built the Gathers that read it -- so a pass that converted nothing + // adds no input. + if (created_beam_idx) { + model->add_parameters({beam_idx}); + } + for (const auto& p : params_to_remove) { + model->remove_parameter(p); + } + + // Recurrent states are independent of the KV caches; a hybrid stack (qwen35) has both. + make_recurrent_states_stateful(model); + + model->validate_nodes_and_infer_types(); + return true; +} + +} // namespace ov::frontend::gguf::pass diff --git a/src/frontends/gguf/src/quant/gguf.cpp b/src/frontends/gguf/src/quant/gguf.cpp new file mode 100644 index 00000000000000..7b690608789b7b --- /dev/null +++ b/src/frontends/gguf/src/quant/gguf.cpp @@ -0,0 +1,1032 @@ +// Copyright (C) 2018-2026 Intel Corporation +// SPDX-License-Identifier: Apache-2.0 +// + +// Native GGUF container parser. Replaces the third-party gguflib dependency: the file is +// memory-mapped via ov::load_mmap_object and parsed directly per the GGUF v2/v3 format +// (https://github.com/ggml-org/ggml/blob/master/docs/gguf.md). No llama.cpp / ggml +// dependency. + +#include "gguf.hpp" +#include "weights.hpp" + +#include +#include +#include +#include + +#include "openvino/core/except.hpp" +#include "openvino/core/type/element_type_traits.hpp" +#include "openvino/runtime/aligned_buffer.hpp" +#include "openvino/runtime/shared_buffer.hpp" +#include "openvino/util/mmap_object.hpp" + +namespace ov { +namespace frontend { +namespace gguf { + +namespace { + +constexpr uint32_t GGUF_MAGIC = 0x46554747; // "GGUF" little-endian +constexpr uint64_t GGUF_DEFAULT_ALIGNMENT = 32; + +// (items_per_block, bytes_per_block) per gguf_tensor_type, indexed by the type id. +struct TypeTraits { + uint32_t items_per_block; + uint32_t bytes_per_block; +}; + +TypeTraits type_traits(uint32_t type) { + switch (type) { + case GGUF_TYPE_F32: + return {1, 4}; + case GGUF_TYPE_F16: + return {1, 2}; + case GGUF_TYPE_Q4_0: + return {32, 18}; + case GGUF_TYPE_Q4_1: + return {32, 20}; + case GGUF_TYPE_Q5_0: + return {32, 22}; + case GGUF_TYPE_Q5_1: + return {32, 24}; + case GGUF_TYPE_Q8_0: + return {32, 34}; + case GGUF_TYPE_Q8_1: + return {32, 40}; + case GGUF_TYPE_Q2_K: + return {256, 84}; + case GGUF_TYPE_Q2_0: + return {64, 18}; // f16 scale + 64x 2-bit codes (16 bytes) + case GGUF_TYPE_Q3_K: + return {256, 110}; + case GGUF_TYPE_Q4_K: + return {256, 144}; + case GGUF_TYPE_Q5_K: + return {256, 176}; + case GGUF_TYPE_Q6_K: + return {256, 210}; + case GGUF_TYPE_Q8_K: + return {256, 292}; + case GGUF_TYPE_I8: + return {1, 1}; + case GGUF_TYPE_I16: + return {1, 2}; + case GGUF_TYPE_I32: + return {1, 4}; + case GGUF_TYPE_I64: + return {1, 8}; + case GGUF_TYPE_F64: + return {1, 8}; + case GGUF_TYPE_BF16: + return {1, 2}; + case GGUF_TYPE_MXFP4: + return {32, 17}; // 1-byte E8M0 scale + 16 bytes (32x 4-bit) + default: + return {0, 0}; + } +} + +std::optional gguf_type_to_dtype(uint32_t gguf_type) { + switch (gguf_type) { + case GGUF_TYPE_F64: + return ov::element::f64; + case GGUF_TYPE_F32: + return ov::element::f32; + case GGUF_TYPE_F16: + return ov::element::f16; + case GGUF_TYPE_BF16: + return ov::element::bf16; + case GGUF_TYPE_I8: + return ov::element::i8; + case GGUF_TYPE_I16: + return ov::element::i16; + case GGUF_TYPE_I32: + return ov::element::i32; + case GGUF_TYPE_I64: + return ov::element::i64; + default: + return std::nullopt; + } +} + +// Sequential little-endian reader over the mmaped file. Bounds-checked on every read. +class Cursor { +public: + Cursor(const uint8_t* data, uint64_t size) : m_data(data), m_size(size) {} + + uint64_t offset() const { + return m_off; + } + const uint8_t* ptr() const { + return m_data + m_off; + } + + template + T read() { + OPENVINO_ASSERT(m_off + sizeof(T) <= m_size, "[load_gguf] unexpected end of file"); + T v; + std::memcpy(&v, m_data + m_off, sizeof(T)); + m_off += sizeof(T); + return v; + } + + std::string read_string() { + uint64_t len = read(); + OPENVINO_ASSERT(m_off + len <= m_size, "[load_gguf] string runs past end of file"); + std::string s(reinterpret_cast(m_data + m_off), static_cast(len)); + m_off += len; + return s; + } + + void skip(uint64_t n) { + OPENVINO_ASSERT(m_off + n <= m_size, "[load_gguf] skip past end of file"); + m_off += n; + } + +private: + const uint8_t* m_data; + uint64_t m_size; + uint64_t m_off = 0; +}; + +size_t value_type_size(uint32_t type) { + switch (type) { + case GGUF_VALUE_TYPE_UINT8: + case GGUF_VALUE_TYPE_INT8: + case GGUF_VALUE_TYPE_BOOL: + return 1; + case GGUF_VALUE_TYPE_UINT16: + case GGUF_VALUE_TYPE_INT16: + return 2; + case GGUF_VALUE_TYPE_UINT32: + case GGUF_VALUE_TYPE_INT32: + case GGUF_VALUE_TYPE_FLOAT32: + return 4; + case GGUF_VALUE_TYPE_UINT64: + case GGUF_VALUE_TYPE_INT64: + case GGUF_VALUE_TYPE_FLOAT64: + return 8; + default: + return 0; // string / array handled separately + } +} + +ov::element::Type value_type_to_dtype(uint32_t type) { + switch (type) { + case GGUF_VALUE_TYPE_UINT8: + return ov::element::u8; + case GGUF_VALUE_TYPE_INT8: + return ov::element::i8; + case GGUF_VALUE_TYPE_UINT16: + return ov::element::u16; + case GGUF_VALUE_TYPE_INT16: + return ov::element::i16; + case GGUF_VALUE_TYPE_UINT32: + return ov::element::u32; + case GGUF_VALUE_TYPE_INT32: + return ov::element::i32; + case GGUF_VALUE_TYPE_FLOAT32: + return ov::element::f32; + case GGUF_VALUE_TYPE_BOOL: + return ov::element::boolean; + case GGUF_VALUE_TYPE_UINT64: + return ov::element::u64; + case GGUF_VALUE_TYPE_INT64: + return ov::element::i64; + case GGUF_VALUE_TYPE_FLOAT64: + return ov::element::f64; + default: + OPENVINO_THROW("[load_gguf] unexpected scalar metadata type ", type); + } +} + +// Read a single metadata value of `type` at the cursor into `out`. +void read_metadata_value(Cursor& cur, uint32_t type, GGUFMetaData& out) { + if (type == GGUF_VALUE_TYPE_STRING) { + out = cur.read_string(); + return; + } + if (type == GGUF_VALUE_TYPE_ARRAY) { + uint32_t elem_type = cur.read(); + uint64_t len = cur.read(); + OPENVINO_ASSERT(elem_type != GGUF_VALUE_TYPE_ARRAY, "[load_gguf] nested arrays are not supported."); + if (elem_type == GGUF_VALUE_TYPE_STRING) { + std::vector strs(len); + for (auto& s : strs) { + s = cur.read_string(); + } + out = std::move(strs); + return; + } + auto dtype = value_type_to_dtype(elem_type); + ov::Tensor t(dtype, ov::Shape{static_cast(len)}); + const size_t nbytes = static_cast(len) * value_type_size(elem_type); + std::memcpy(t.data(), cur.ptr(), nbytes); + cur.skip(nbytes); + out = std::move(t); + return; + } + // Scalar: store as a shape-{} ov::Tensor of the right element type. + auto dtype = value_type_to_dtype(type); + ov::Tensor t(dtype, ov::Shape(0)); + const size_t nbytes = value_type_size(type); + std::memcpy(t.data(), cur.ptr(), nbytes); + cur.skip(nbytes); + out = std::move(t); +} + +// Zero-copy view into the mmap for a non-quantized tensor. Uses the dev-API +// Tensor(view, so) constructor so that the mmap shared_ptr is stored in _so and keeps +// the mapping alive for the full lifetime of the returned tensor (and any Constant that +// wraps it via Constant(const Tensor&) -> SharedBuffer). +ov::Tensor extract_tensor_data(const gguf_tensor& tensor, const std::shared_ptr& mmap) { + auto dtype = gguf_type_to_dtype(tensor.type); + OPENVINO_ASSERT(dtype.has_value(), + "[load_gguf] tensor '", + std::string(tensor.name, tensor.namelen), + "' has unsupported non-quantized type ", + tensor.type); + auto shape = get_shape(tensor); + ov::Tensor view(dtype.value(), shape, const_cast(static_cast(tensor.weights_data))); + // Attach the mmap shared_ptr as _so so the mapping stays alive for the tensor's lifetime. + return ov::Tensor(view, mmap); +} + +// Fetch a metadata value as an ov::Tensor, failing with the KEY NAME on either failure mode: +// the key is absent, or it is present but not stored as a scalar tensor. A bare metadata.at(key) +// / std::get would throw std::out_of_range / std::bad_variant_access with no context. +static const ov::Tensor& metadata_scalar_tensor(const std::unordered_map& metadata, + const std::string& key) { + auto it = metadata.find(key); + OPENVINO_ASSERT(it != metadata.end(), "[GGUF] required metadata key is missing: '", key, "'"); + const auto* tensor = std::get_if(&it->second); + OPENVINO_ASSERT(tensor && tensor->data(), + "[GGUF] metadata key '", + key, + "' is not a scalar numeric value as expected"); + return *tensor; +} + +float metadata_to_float(const std::unordered_map& metadata, const std::string& key) { + const auto& tensor = metadata_scalar_tensor(metadata, key); + return *(tensor.data::value_type>()); +} + +int metadata_to_int(const std::unordered_map& metadata, const std::string& key) { + const auto& tensor = metadata_scalar_tensor(metadata, key); + // GGUF stores these counts as u32; reinterpret as i32 (values fit comfortably). + return static_cast(*(tensor.data::value_type>())); +} + +} // namespace + +ov::Shape get_shape(const gguf_tensor& tensor) { + ov::Shape shape; + // GGUF stores dimensions fastest-varying first; the logical (GGML) order is reversed. + for (int i = static_cast(tensor.ndim) - 1; i >= 0; i--) { + shape.push_back(tensor.dim[i]); + } + return shape; +} + +GGUFLoad get_gguf_data(const std::string& file) { + std::unordered_map metadata; + std::unordered_map arrays; + std::unordered_map qtype; + + auto mapped = ov::load_mmap_object(file); + OPENVINO_ASSERT(mapped && mapped->data(), "[load_gguf] failed to mmap '", file, "'"); + const auto* base = reinterpret_cast(mapped->data()); + const uint64_t fsize = mapped->size(); + + Cursor cur(base, fsize); + + // ---- Header ---- + uint32_t magic = cur.read(); + OPENVINO_ASSERT(magic == GGUF_MAGIC, "[load_gguf] '", file, "' is not a GGUF file (bad magic)"); + uint32_t version = cur.read(); + OPENVINO_ASSERT(version == 2 || version == 3, "[load_gguf] unsupported GGUF version ", version); + uint64_t tensor_count = cur.read(); + uint64_t kv_count = cur.read(); + + // ---- Metadata kv pairs ---- + for (uint64_t i = 0; i < kv_count; i++) { + std::string key = cur.read_string(); + uint32_t vtype = cur.read(); + auto& slot = metadata.insert({key, GGUFMetaData{}}).first->second; + read_metadata_value(cur, vtype, slot); + } + + uint64_t alignment = GGUF_DEFAULT_ALIGNMENT; + if (auto it = metadata.find("general.alignment"); it != metadata.end()) { + if (auto* t = std::get_if(&it->second)) { + alignment = *(t->data::value_type>()); + } + } + + // ---- Tensor info section ---- + struct TensorInfo { + std::string name; + uint32_t type = 0; + uint32_t ndim = 0; + uint64_t dim[4] = {1, 1, 1, 1}; + uint64_t offset = 0; + }; + std::vector infos(tensor_count); + for (uint64_t i = 0; i < tensor_count; i++) { + TensorInfo& ti = infos[i]; + ti.name = cur.read_string(); + ti.ndim = cur.read(); + OPENVINO_ASSERT(ti.ndim <= 4, "[load_gguf] tensor '", ti.name, "' has unsupported ndim ", ti.ndim); + for (uint32_t d = 0; d < ti.ndim; d++) { + ti.dim[d] = cur.read(); + } + ti.type = cur.read(); + ti.offset = cur.read(); + } + + // Tensor data starts at the next `alignment`-aligned offset after the info section. + uint64_t data_off = cur.offset(); + if (uint64_t rem = data_off % alignment) { + data_off += alignment - rem; + } + + // Helper: for a quantized tensor, compute (weights_bytes, scale_bytes, zp_bytes). + // Symmetric types (Q4_0, Q8_0, Q5_0, Q6_K): zp_bytes = 0. + // Asymmetric types (Q4_1, Q4_K): zp u4 packed (same count as scales, half the bytes). + // Asymmetric Q5_K: zp u8 (one byte per sub-block, same count as scales). + auto quant_sizes = [](const TensorInfo& ti) -> std::tuple { + auto shape = [&]() { + ov::Shape s; + for (int i = static_cast(ti.ndim) - 1; i >= 0; --i) + s.push_back(ti.dim[i]); + return s; + }(); + + if (ti.type == GGUF_TYPE_Q8_K) { + // Q8_K: 256 i8 weights + f32 scale + 16 i16 bsums (ignored) per block. + size_t nelems = 1; + for (auto d : shape) nelems *= d; + const size_t n_blocks = nelems / 256; + return {nelems, n_blocks * sizeof(float), 0}; // w_bytes, s_bytes(f32), no zp + } + + if (ti.type == GGUF_TYPE_MXFP4) { + size_t nelems = 1; + for (auto d : shape) + nelems *= d; + const size_t cols = shape.back(); + const size_t groups = cols / 32; + size_t rows = nelems / cols; + const size_t w_bytes = (nelems + 1) / 2; // f4e2m1: 4-bit + const size_t s_bytes = rows * groups; // f8e8m0: 1 byte/group + return {w_bytes, s_bytes, 0}; + } + + size_t w_nelems = 1; + for (auto d : shape) + w_nelems *= d; + + // Q2_K: u2 (4 per byte), 16 sub-blocks of 16 per super-block. + if (ti.type == GGUF_TYPE_Q2_K) { + const size_t w_bytes = (w_nelems + 3) / 4; // u2: 4 values per byte + auto scale_shape = shape; + scale_shape.back() /= 16; + size_t s_nelems = 1; + for (auto d : scale_shape) + s_nelems *= d; + return {w_bytes, s_nelems * sizeof(uint16_t), s_nelems}; // zp: u8 per sub-block + } + + // Q2_0: u2 (4 per byte), one f16 scale per 64 weights, constant zero-point of 1. + if (ti.type == GGUF_TYPE_Q2_0) { + const size_t w_bytes = (w_nelems + 3) / 4; // u2: 4 values per byte + auto scale_shape = shape; + scale_shape.back() /= 64; + size_t s_nelems = 1; + for (auto d : scale_shape) + s_nelems *= d; + return {w_bytes, s_nelems * sizeof(uint16_t), s_nelems}; // zp: u8 per block + } + + // Q3_K: i4 packed (2 per byte), 16 sub-blocks of 16 per super-block. + if (ti.type == GGUF_TYPE_Q3_K) { + const size_t w_bytes = (w_nelems + 1) / 2; // i4: 2 values per byte + auto scale_shape = shape; + scale_shape.back() /= 16; + size_t s_nelems = 1; + for (auto d : scale_shape) + s_nelems *= d; + return {w_bytes, s_nelems * sizeof(uint16_t), 0}; // symmetric: no zp + } + + // Weights: i8 or u8 stored in byte arrays (not u32-packed anymore for sym; u32 only for 4-bit). + // 4-bit types: Q4_0(i4 in u32), Q4_1(u4 in u32), Q4_K(u4 in u32). + // 8-bit types: Q8_0(i8), Q5_0(i8), Q5_1(i8), Q5_K(i8), Q6_K(i8). + const bool is_4bit = (ti.type == GGUF_TYPE_Q4_0 || ti.type == GGUF_TYPE_Q4_1 || ti.type == GGUF_TYPE_Q4_K); + uint64_t weights_per_byte = is_4bit ? 2 : 1; + // Q6_K: 16 weights per sub-block (16 sub-blocks × 16 = 256 per super-block). + uint64_t weights_per_block = (ti.type == GGUF_TYPE_Q6_K) ? 16 : 32; + + size_t w_bytes; + if (is_4bit) { + // u32-packed nibbles: floor((n+7)/8)*4 bytes + w_bytes = ((w_nelems / weights_per_byte) / 4) * sizeof(uint32_t); + } else { + // i8 byte per element + w_bytes = w_nelems; + } + + auto scale_shape = shape; + scale_shape.back() /= weights_per_block; + size_t s_nelems = 1; + for (auto d : scale_shape) + s_nelems *= d; + const size_t s_bytes = s_nelems * sizeof(uint16_t); + + // Zero-point bytes: + // Symmetric (Q4_0, Q8_0, Q5_0, Q6_K): no zp. + // Q4_1, Q4_K: u4 zp — same element count as scales, packed 2/byte. + // Q5_K, Q5_1: u8 zp — one byte per sub-block. + size_t z_bytes = 0; + if (ti.type == GGUF_TYPE_Q4_1 || ti.type == GGUF_TYPE_Q4_K) { + z_bytes = (s_nelems + 1) / 2; // u4 packed + } else if (ti.type == GGUF_TYPE_Q5_K || ti.type == GGUF_TYPE_Q5_1) { + z_bytes = s_nelems; // u8 + } + return {w_bytes, s_bytes, z_bytes}; + }; + + // ---- Pass 1: total bytes needed for all repacked quantized data ---- + size_t total_quant_bytes = 0; + for (const auto& ti : infos) { + const bool is_quant = ti.type == GGUF_TYPE_Q4_0 || ti.type == GGUF_TYPE_Q4_1 || ti.type == GGUF_TYPE_Q5_0 || + ti.type == GGUF_TYPE_Q5_1 || ti.type == GGUF_TYPE_Q8_0 || ti.type == GGUF_TYPE_Q2_K || + ti.type == GGUF_TYPE_Q3_K || ti.type == GGUF_TYPE_Q4_K || ti.type == GGUF_TYPE_Q5_K || + ti.type == GGUF_TYPE_Q6_K || ti.type == GGUF_TYPE_Q8_K || ti.type == GGUF_TYPE_MXFP4 || + ti.type == GGUF_TYPE_Q2_0; + if (!is_quant) + continue; + auto [wb, sb, bb] = quant_sizes(ti); + total_quant_bytes += wb + sb + bb; + } + + // Single allocation for all repacked quantized data (IR-frontend AlignedBuffer pattern). + auto quant_buf = std::make_shared(total_quant_bytes > 0 ? total_quant_bytes : 1); + + // ---- Pass 2: materialize tensors, slicing into quant_buf for quantized ones ---- + size_t quant_offset = 0; + for (const auto& ti : infos) { + gguf_tensor tensor; + tensor.name = ti.name.data(); + tensor.namelen = ti.name.size(); + tensor.type = ti.type; + tensor.ndim = ti.ndim; + uint64_t nelem = 1; + for (uint32_t d = 0; d < ti.ndim; d++) { + tensor.dim[d] = ti.dim[d]; + nelem *= ti.dim[d]; + } + tensor.num_weights = nelem; + tensor.offset = ti.offset; + + auto tr = type_traits(ti.type); + OPENVINO_ASSERT(tr.bytes_per_block != 0, "[load_gguf] tensor '", ti.name, "' has unsupported type ", ti.type); + tensor.bsize = (nelem / tr.items_per_block) * tr.bytes_per_block; + + uint64_t abs_off = data_off + ti.offset; + OPENVINO_ASSERT(abs_off + tensor.bsize <= fsize, "[load_gguf] tensor '", ti.name, "' data runs past EOF"); + tensor.weights_data = base + abs_off; + + const std::string& name = ti.name; + constexpr std::string_view weight_suffix = ".weight"; + const bool has_weight_suffix = + name.size() >= weight_suffix.size() && + name.compare(name.size() - weight_suffix.size(), weight_suffix.size(), weight_suffix) == 0; + // For tensors ending in ".weight" strip the suffix so make_weight_node keys are + // "blk.N.attn_q" (not "blk.N.attn_q.weight"). For quantized non-weight tensors + // (e.g. gpt-oss "blk.N.attn_k.bias") keep the full name as the prefix; the builder + // looks up scales/qtype as name + ".scales" / name + ".qtype". + const std::string name_prefix = + has_weight_suffix ? name.substr(0, name.length() - weight_suffix.length()) : name; + if (ti.type == GGUF_TYPE_Q4_0) { + // Symmetric: i4 weights (XORed u4) + f16 scales, no bias tensor. + auto [wb, sb, bb] = quant_sizes(ti); + char* buf_ptr = quant_buf->get_ptr(); + auto shape = get_shape(tensor); + auto weights_shape = shape; + weights_shape.back() /= 8; // u32 packs 8 i4 nibbles + auto scale_shape = shape; + scale_shape.back() /= 32; + + std::shared_ptr so_buf(quant_buf); + ov::Tensor w_view(ov::element::u32, weights_shape, static_cast(buf_ptr + quant_offset)); + ov::Tensor weights(w_view, so_buf); + quant_offset += wb; + ov::Tensor s_view(ov::element::f16, scale_shape, static_cast(buf_ptr + quant_offset)); + ov::Tensor scales(s_view, so_buf); + quant_offset += sb; + + gguf_fill_q4_0(tensor, weights, scales); + mapped->hint_evict(abs_off, tensor.bsize); + + arrays.emplace(name, std::move(weights)); + arrays.emplace(name_prefix + ".scales", std::move(scales)); + qtype.emplace(name_prefix + ".qtype", GGUF_TYPE_Q4_0); + } else if (ti.type == GGUF_TYPE_Q3_K) { + // Symmetric: i4 weights (2 per byte) + f16 scales. No zero-point. + auto [wb, sb, zb] = quant_sizes(ti); + (void)zb; + char* buf_ptr = quant_buf->get_ptr(); + + auto shape = get_shape(tensor); + auto scale_shape = shape; + scale_shape.back() /= 16; // 16 sub-blocks per super-block + + std::shared_ptr so_buf(quant_buf); + ov::Tensor w_view(ov::element::i4, shape, static_cast(buf_ptr + quant_offset)); + ov::Tensor weights(w_view, so_buf); + quant_offset += wb; + ov::Tensor s_view(ov::element::f16, scale_shape, static_cast(buf_ptr + quant_offset)); + ov::Tensor scales(s_view, so_buf); + quant_offset += sb; + + gguf_fill_sym(tensor, weights, scales); + mapped->hint_evict(abs_off, tensor.bsize); + + arrays.emplace(name, std::move(weights)); + arrays.emplace(name_prefix + ".scales", std::move(scales)); + qtype.emplace(name_prefix + ".qtype", GGUF_TYPE_Q3_K); + } else if (ti.type == GGUF_TYPE_Q2_K) { + // Asymmetric: u2 weights (4 per byte) + f16 scales + u8 zp per sub-block. + auto [wb, sb, zb] = quant_sizes(ti); + char* buf_ptr = quant_buf->get_ptr(); + + auto shape = get_shape(tensor); + auto scale_shape = shape; + scale_shape.back() /= 16; // 16 sub-blocks per super-block + + std::shared_ptr so_buf(quant_buf); + ov::Tensor w_view(ov::element::u2, shape, static_cast(buf_ptr + quant_offset)); + ov::Tensor weights(w_view, so_buf); + quant_offset += wb; + ov::Tensor s_view(ov::element::f16, scale_shape, static_cast(buf_ptr + quant_offset)); + ov::Tensor scales(s_view, so_buf); + quant_offset += sb; + // Fractional zp (= min/scale) in the scale's element type -- see the Q4_K/Q5_K branch. + ov::Tensor zp(scales.get_element_type(), scale_shape); + quant_offset += zb; + + gguf_fill_asym(tensor, weights, scales, zp); + mapped->hint_evict(abs_off, tensor.bsize); + + arrays.emplace(name, std::move(weights)); + arrays.emplace(name_prefix + ".scales", std::move(scales)); + arrays.emplace(name_prefix + ".zp", std::move(zp)); + qtype.emplace(name_prefix + ".qtype", GGUF_TYPE_Q2_K); + } else if (ti.type == GGUF_TYPE_Q2_0) { + // Ternary: u2 weights (4 per byte) + one f16 scale per 64 + integer zp of exactly 1. + // ggml packs Q2_0 codes LSB-first, 4 per byte -- the same order OpenVINO's u2 Constant + // expects -- so the code bytes are copied verbatim, no repacking. + auto [wb, sb, zb] = quant_sizes(ti); + char* buf_ptr = quant_buf->get_ptr(); + + auto shape = get_shape(tensor); + auto scale_shape = shape; + scale_shape.back() /= 64; // one scale per 64-weight block + + std::shared_ptr so_buf(quant_buf); + ov::Tensor w_view(ov::element::u2, shape, static_cast(buf_ptr + quant_offset)); + ov::Tensor weights(w_view, so_buf); + quant_offset += wb; + ov::Tensor s_view(ov::element::f16, scale_shape, static_cast(buf_ptr + quant_offset)); + ov::Tensor scales(s_view, so_buf); + quant_offset += sb; + // Integer zero-point: (code - 1) * scale, so zp is the constant 1 for every block. + ov::Tensor zp(ov::element::u8, scale_shape); + quant_offset += zb; + + gguf_fill_q2_0(tensor, weights, scales, zp); + mapped->hint_evict(abs_off, tensor.bsize); + + arrays.emplace(name, std::move(weights)); + arrays.emplace(name_prefix + ".scales", std::move(scales)); + arrays.emplace(name_prefix + ".zp", std::move(zp)); + qtype.emplace(name_prefix + ".qtype", GGUF_TYPE_Q2_0); + } else if (ti.type == GGUF_TYPE_Q8_K) { + // Q8_K: 256 weights/block, f32 scale (NOT f16), 16 i16 bsums (ignored). + // block_q8_K: [f32 d][i8 qs[256]][i16 bsums[16]] = 4+256+32 = 292 bytes. + auto [wb, sb, zb] = quant_sizes(ti); + (void)zb; + char* buf_ptr = quant_buf->get_ptr(); + + auto shape = get_shape(tensor); + auto scale_shape = shape; + scale_shape.back() /= 256; // one f32 scale per 256-weight block + + std::shared_ptr so_buf(quant_buf); + ov::Tensor w_view(ov::element::i8, shape, static_cast(buf_ptr + quant_offset)); + ov::Tensor weights_t(w_view, so_buf); + quant_offset += wb; + ov::Tensor s_view(ov::element::f32, scale_shape, static_cast(buf_ptr + quant_offset)); + ov::Tensor scales_t(s_view, so_buf); + quant_offset += sb; + + gguf_fill_sym(tensor, weights_t, scales_t); + mapped->hint_evict(abs_off, tensor.bsize); + + arrays.emplace(name, std::move(weights_t)); + arrays.emplace(name_prefix + ".scales", std::move(scales_t)); + qtype.emplace(name_prefix + ".qtype", GGUF_TYPE_Q8_K); + } else if (ti.type == GGUF_TYPE_Q8_0 || ti.type == GGUF_TYPE_Q5_0 || ti.type == GGUF_TYPE_Q6_K) { + // Symmetric: i8 weights (no u32 packing) + f16 scales. No zero-point. + auto [wb, sb, zb] = quant_sizes(ti); + (void)zb; + char* buf_ptr = quant_buf->get_ptr(); + + auto shape = get_shape(tensor); + const uint64_t weights_per_block = (ti.type == GGUF_TYPE_Q6_K) ? 16 : 32; + auto scale_shape = shape; + scale_shape.back() /= weights_per_block; + + std::shared_ptr so_buf(quant_buf); + ov::Tensor w_view(ov::element::i8, shape, static_cast(buf_ptr + quant_offset)); + ov::Tensor weights(w_view, so_buf); + quant_offset += wb; + ov::Tensor s_view(ov::element::f16, scale_shape, static_cast(buf_ptr + quant_offset)); + ov::Tensor scales(s_view, so_buf); + quant_offset += sb; + + gguf_fill_sym(tensor, weights, scales); + mapped->hint_evict(abs_off, tensor.bsize); + + arrays.emplace(name, std::move(weights)); + arrays.emplace(name_prefix + ".scales", std::move(scales)); + qtype.emplace(name_prefix + ".qtype", static_cast(ti.type)); + } else if (ti.type == GGUF_TYPE_Q4_1 || ti.type == GGUF_TYPE_Q4_K || ti.type == GGUF_TYPE_Q5_K || + ti.type == GGUF_TYPE_Q5_1) { + // Asymmetric: weights + f16 scales + integer zp. + // 4-bit (Q4_1, Q4_K): u32-packed u4 weights, u4 zp. + // 8-bit (Q5_K, Q5_1): i8 weights, u8 zp. + auto [wb, sb, zb] = quant_sizes(ti); + char* buf_ptr = quant_buf->get_ptr(); + + auto shape = get_shape(tensor); + const bool is_4bit = (ti.type == GGUF_TYPE_Q4_1 || ti.type == GGUF_TYPE_Q4_K); + const uint64_t weights_per_block = 32; + + auto weights_shape = shape; + if (is_4bit) + weights_shape.back() /= 8; // u32 packs 8 u4 + auto scale_shape = shape; + scale_shape.back() /= weights_per_block; + + std::shared_ptr so_buf(quant_buf); + ov::element::Type w_elem = is_4bit ? ov::element::u32 : ov::element::i8; + ov::Tensor w_view(w_elem, is_4bit ? weights_shape : shape, static_cast(buf_ptr + quant_offset)); + ov::Tensor weights(w_view, so_buf); + quant_offset += wb; + ov::Tensor s_view(ov::element::f16, scale_shape, static_cast(buf_ptr + quant_offset)); + ov::Tensor scales(s_view, so_buf); + quant_offset += sb; + // Both ingest paths must agree on the zero-point representation; see + // gguf_zero_point_type in quant/weights.hpp for why it matters. + const auto zp_elem = gguf_zero_point_type(name, static_cast(ti.type)); + // Only Q4_K's integer zp actually rounds: Q2_0's zero-point is the exact integer 1. + if (zp_elem == ov::element::u8 && ti.type == GGUF_TYPE_Q4_K) { + notify_lossy_weight_approximation(LossyWeightApproximation::IntegerZeroPoint); + } + ov::Tensor zp(zp_elem, scale_shape); + quant_offset += zb; + + gguf_fill_asym(tensor, weights, scales, zp); + mapped->hint_evict(abs_off, tensor.bsize); + + arrays.emplace(name, std::move(weights)); + arrays.emplace(name_prefix + ".scales", std::move(scales)); + arrays.emplace(name_prefix + ".zp", std::move(zp)); + qtype.emplace(name_prefix + ".qtype", static_cast(ti.type)); + } else if (ti.type == GGUF_TYPE_MXFP4) { + // MXFP4: slice weight (f4e2m1) + scale (f8e8m0) out of quant_buf. + auto [wb, sb, dummy_bb] = quant_sizes(ti); + (void)dummy_bb; + char* buf_ptr = quant_buf->get_ptr(); + + auto shape = get_shape(tensor); + const size_t cols = shape.back(); + size_t nelems = 1; + for (auto d : shape) + nelems *= d; + const size_t groups = cols / 32; + ov::Shape scale_shape = shape; + scale_shape.back() = groups; + + std::shared_ptr so_buf(quant_buf); + ov::Tensor w_view(ov::element::f4e2m1, shape, static_cast(buf_ptr + quant_offset)); + ov::Tensor weights(w_view, so_buf); + quant_offset += wb; + + ov::Tensor s_view(ov::element::f8e8m0, scale_shape, static_cast(buf_ptr + quant_offset)); + ov::Tensor scales(s_view, so_buf); + quant_offset += sb; + + gguf_fill_mxfp4(tensor, weights, scales); + mapped->hint_evict(abs_off, tensor.bsize); + + constexpr std::string_view weight_suffix = ".weight"; + const std::string prefix = name.substr(0, name.length() - weight_suffix.length()); + arrays.emplace(name, std::move(weights)); + arrays.emplace(prefix + ".scales", std::move(scales)); + qtype.emplace(prefix + ".qtype", GGUF_TYPE_MXFP4); + } else { + ov::Tensor loaded = extract_tensor_data(tensor, mapped); // zero-copy mmap view + OPENVINO_ASSERT(arrays.emplace(name, loaded).second, "[load_gguf] duplicate tensor name '", name, "'"); + constexpr std::string_view weight_suffix = ".weight"; + if (name.size() >= weight_suffix.size()) { + const std::string name_prefix = name.substr(0, name.length() - weight_suffix.length()); + qtype.emplace(name_prefix + ".qtype", static_cast(ti.type)); + } + } + } + + return {metadata, arrays, qtype, mapped, quant_buf}; +} + +std::map decoder_config_from_meta( + const std::unordered_map& metadata) { + std::map config; + // The architecture key drives every other key lookup; fail with a clear message (not a bare + // std::out_of_range / std::bad_variant_access) if the file carries no / a non-string one. + auto arch_it = metadata.find("general.architecture"); + OPENVINO_ASSERT(arch_it != metadata.end(), + "[GGUF] file has no 'general.architecture' metadata key; not a valid GGUF model file"); + const auto* arch_ptr = std::get_if(&arch_it->second); + OPENVINO_ASSERT(arch_ptr, "[GGUF] 'general.architecture' metadata is not a string"); + const std::string arch = *arch_ptr; + config["architecture"] = arch; + config["layer_num"] = metadata_to_int(metadata, arch + ".block_count"); + config["head_num"] = metadata_to_int(metadata, arch + ".attention.head_count"); + config["head_size"] = metadata.count(arch + ".attention.key_length") + ? metadata_to_int(metadata, arch + ".attention.key_length") + : (metadata_to_int(metadata, arch + ".embedding_length") / + metadata_to_int(metadata, arch + ".attention.head_count")); + { + const std::string kv_key = arch + ".attention.head_count_kv"; + if (metadata.count(kv_key)) { + const auto& kv_val = metadata.at(kv_key); + if (auto* t = std::get_if(&kv_val)) { + if (t->get_shape().size() == 1 && t->get_shape()[0] > 1) { + // Per-layer array: store as a config tensor for use in the builder. + config["head_num_kv_per_layer"] = *t; + // Global default = the most common value (max over the array). + const auto* data = t->data(); + int global_kv = static_cast(*std::max_element(data, data + t->get_size())); + config["head_num_kv"] = global_kv; + } else { + config["head_num_kv"] = metadata_to_int(metadata, kv_key); + } + } + } else { + config["head_num_kv"] = metadata_to_int(metadata, arch + ".attention.head_count"); + } + } + config["hidden_size"] = metadata_to_int(metadata, arch + ".embedding_length"); + config["max_position_embeddings"] = + metadata.count(arch + ".context_length") ? metadata_to_int(metadata, arch + ".context_length") : 2048; + config["rms_norm_eps"] = metadata_to_float(metadata, arch + ".attention.layer_norm_rms_epsilon"); + config["rope_freq_base"] = + metadata.count(arch + ".rope.freq_base") ? metadata_to_float(metadata, arch + ".rope.freq_base") : 10000.0f; + // Advisory: the dominant quant type of the file. Purely informational -- every weight carries + // its own type in the tensor info, which is what the dequant path uses. Optional because + // llama.cpp's own model writer (llama_model_saver, the source of the per-arch test fixtures) + // does not emit it; requiring it would reject files llama.cpp itself considers valid. + config["file_type"] = + metadata.count("general.file_type") ? metadata_to_int(metadata, "general.file_type") : 0; + + // RoPE YaRN scaling: freq_scale = 1/factor (default 1.0 = no scaling), ext_factor = 1.0 + // for YARN type (0.0 otherwise), n_ctx_orig from rope.scaling.original_context_length. + // Mirrors llama.cpp: hparams.rope_freq_scale_train = 1/ropescale; ext_factor = (yarn?1:0). + { + const float ropescale = metadata.count(arch + ".rope.scaling.factor") + ? metadata_to_float(metadata, arch + ".rope.scaling.factor") + : 0.0f; + config["rope_freq_scale"] = ropescale == 0.0f ? 1.0f : 1.0f / ropescale; + + const bool is_yarn = metadata.count(arch + ".rope.scaling.type") && + std::get(metadata.at(arch + ".rope.scaling.type")) == "yarn"; + config["rope_ext_factor"] = is_yarn ? 1.0f : 0.0f; + + // n_ctx_orig: use rope.scaling.original_context_length when present; fall back to + // context_length (the training context, which is also n_ctx_train in llama.cpp). + config["rope_n_ctx_orig"] = metadata.count(arch + ".rope.scaling.original_context_length") + ? metadata_to_int(metadata, arch + ".rope.scaling.original_context_length") + : std::get(config["max_position_embeddings"]); + } + + // Number of rope dimensions (n_rot); defaults to head_size. Some archs (e.g. partial- + // rotary models) set it smaller. A value of 0 means "no RoPE" or full rotation — + // treat as head_size to avoid division-by-zero and empty-vector crashes downstream. + { + const int rope_dims = metadata.count(arch + ".rope.dimension_count") + ? metadata_to_int(metadata, arch + ".rope.dimension_count") + : 0; + config["rope_dimension_count"] = (rope_dims > 0) ? rope_dims : std::get(config["head_size"]); + } + // Gemma4: SWA layers use a smaller head size (and fewer rope dims) than global layers. + // key_length_swa / value_length_swa / rope.dimension_count_swa default to key_length. + config["head_size_swa"] = metadata.count(arch + ".attention.key_length_swa") + ? metadata_to_int(metadata, arch + ".attention.key_length_swa") + : std::get(config["head_size"]); + config["rope_dimension_count_swa"] = metadata.count(arch + ".rope.dimension_count_swa") + ? metadata_to_int(metadata, arch + ".rope.dimension_count_swa") + : std::get(config["rope_dimension_count"]); + // Gemma4: per-layer embedding dimension (0 = absent / not used). + config["n_embd_per_layer"] = + metadata.count(arch + ".embedding_length_per_layer_input") + ? metadata_to_int(metadata, arch + ".embedding_length_per_layer_input") + : 0; + // Gemma4: number of layers that have their own KV cache (from the start). + // shared_kv_layers trailing layers reuse KV from the preceding full-KV layer. + // 0 = all layers have KV (default for non-Gemma4 architectures). + config["shared_kv_layers"] = metadata.count(arch + ".attention.shared_kv_layers") + ? metadata_to_int(metadata, arch + ".attention.shared_kv_layers") + : 0; + + // Mixture-of-experts config (0 when dense). + config["expert_count"] = + metadata.count(arch + ".expert_count") ? metadata_to_int(metadata, arch + ".expert_count") : 0; + config["expert_used_count"] = + metadata.count(arch + ".expert_used_count") ? metadata_to_int(metadata, arch + ".expert_used_count") : 0; + config["expert_feed_forward_length"] = metadata.count(arch + ".expert_feed_forward_length") + ? metadata_to_int(metadata, arch + ".expert_feed_forward_length") + : 0; + // Hybrid MoE: first N layers are dense, remainder use MoE routing. + // Mirrors llama.cpp hparams.n_layer_dense_lead (deepseek2-ocr, ernie4_5-moe, glm4moe). + config["n_layer_dense_lead"] = metadata.count(arch + ".leading_dense_block_count") + ? metadata_to_int(metadata, arch + ".leading_dense_block_count") + : 0; + // Shared (always-active) experts: run in parallel with routed experts, outputs summed. + // Mirrors llama.cpp hparams.n_expert_shared (deepseek2-ocr, bailingmoe2, exaone-moe). + config["expert_shared_count"] = metadata.count(arch + ".expert_shared_count") + ? metadata_to_int(metadata, arch + ".expert_shared_count") + : 0; + + // Per-architecture scalars (MiniCPM family). MiniCPM bakes these into hparams with + // backward-compatible defaults when the GGUF lacks the keys (older exports); newer + // exports carry the keys and override. Other archs default to 1.0 (no-op). + const bool is_minicpm = arch.rfind("minicpm", 0) == 0; + // Gemma/Gemma2/Gemma3/Gemma4 scale embeddings by sqrt(n_embd) before the first layer. + const bool is_gemma = arch == "gemma" || arch == "gemma2" || arch == "gemma3" || arch == "gemma4"; + const float def_embedding_scale = + is_minicpm ? 12.0f : (is_gemma ? std::sqrt(static_cast(std::get(config["hidden_size"]))) : 1.0f); + const float def_residual_scale = + is_minicpm ? 1.4f / std::sqrt(static_cast(std::get(config["layer_num"]))) : 1.0f; + const float def_logit_scale = is_minicpm ? 256.0f / static_cast(std::get(config["hidden_size"])) : 1.0f; + config["embedding_scale"] = metadata.count(arch + ".embedding_scale") + ? metadata_to_float(metadata, arch + ".embedding_scale") + : def_embedding_scale; + config["residual_scale"] = metadata.count(arch + ".residual_scale") + ? metadata_to_float(metadata, arch + ".residual_scale") + : def_residual_scale; + config["logit_scale"] = + metadata.count(arch + ".logit_scale") ? metadata_to_float(metadata, arch + ".logit_scale") : def_logit_scale; + // Hybrid linear-attention (Gated DeltaNet) parameters: qwen35 / qwen3next / kimi-linear. + // 0 for every non-SSM architecture, which is what the builder tests against. + auto ssm_key = [&](const std::string& k) { + return metadata.count(arch + "." + k) ? metadata_to_int(metadata, arch + "." + k) : 0; + }; + config["ssm_conv_kernel"] = ssm_key("ssm.conv_kernel"); + config["ssm_state_size"] = ssm_key("ssm.state_size"); + config["ssm_group_count"] = ssm_key("ssm.group_count"); + config["ssm_time_step_rank"] = ssm_key("ssm.time_step_rank"); + config["ssm_inner_size"] = ssm_key("ssm.inner_size"); + // Every full_attention_interval-th layer is full attention, the rest are recurrent: + // llama.cpp qwen35 is_recr(il) = (il < n_layer) && ((il + 1) % interval != 0). + // llama.cpp defaults the interval to 4 when the GGUF omits the key, so match that rather + // than rejecting the model (llama.cpp src/models/qwen35.cpp load_arch_hparams). + config["full_attention_interval"] = + metadata.count(arch + ".full_attention_interval") ? ssm_key("full_attention_interval") : 4; + // NextN / MTP: extra decoder blocks stored past the main stack and NOT executed in a normal + // forward pass. The builder must stop its layer loop before them. + config["nextn_predict_layers"] = ssm_key("nextn_predict_layers"); + // Explicit per-layer recurrent flags. llama.cpp consults this FIRST and only falls back to + // full_attention_interval when it is absent (src/models/qwen35.cpp load_arch_hparams). + { + std::vector recr; + const std::string key = arch + ".attention.recurrent_layers"; + if (metadata.count(key)) { + const auto& t = std::get(metadata.at(key)); + const auto* p = t.data(); + for (size_t i = 0; i < t.get_size(); ++i) + recr.push_back(static_cast(p[i])); + } + config["recurrent_layer_flags"] = recr; + } + + // M-RoPE section widths (qwen35 / qwen3vl): 4 per-axis rotary section sizes. + { + std::vector sections; + const std::string key = arch + ".rope.dimension_sections"; + if (metadata.count(key)) { + const auto& t = std::get(metadata.at(key)); + const auto* p = t.data(); + for (size_t i = 0; i < t.get_size() && i < 4; ++i) + sections.push_back(static_cast(p[i])); + } + config["rope_sections"] = sections; + } + + // Optional explicit attention (softmax) scale; 0 -> use 1/sqrt(head_size). + // Gemma4 uses scale=1.0 (no pre-attn scaling), per llama.cpp hparams.f_attention_scale=1.0. + // Gemma3 (like gemma/gemma2) uses 1/sqrt(n_embd_head_k); llama.cpp applies it as a + // Qcur pre-scale with build_attn(scale=1.0), which is numerically 1/sqrt(head_size) -- + // exactly the default branch here, so gemma3 must NOT force scale=1.0. + const float def_attention_scale = (arch == "gemma4") ? 1.0f : 0.0f; + config["attention_scale"] = metadata.count(arch + ".attention.scale") + ? metadata_to_float(metadata, arch + ".attention.scale") + : def_attention_scale; + + // gpt-oss SWA: separate RoPE frequency base for sliding-window attention layers. + // Defaults to the global rope_freq_base when the key is absent (non-SWA or legacy models), + // EXCEPT gemma3: llama.cpp's gemma3 loader keeps the struct default 10000.0 when the key is + // absent (it does not reset to the global base first, unlike gemma2/gemma4/cohere2/etc.), so + // gemma3 SWA layers rope at freq_base=10000 while global layers use 1000000. See + // llama.cpp src/models/gemma3.cpp load_arch_hparams. + const float def_freq_base_swa = (arch == "gemma3") ? 10000.0f : std::get(config["rope_freq_base"]); + config["rope_freq_base_swa"] = metadata.count(arch + ".rope.freq_base_swa") + ? metadata_to_float(metadata, arch + ".rope.freq_base_swa") + : def_freq_base_swa; + + // has_swa: true when the GGUF carries either a sliding_window_pattern or a + // sliding_window (a finite window length), indicating SWA is active. The builder uses + // this to add the self_kq_mask_swa input and route SWA layers to the windowed mask. + // Architectures that always use sinks (gpt-oss) or per-layer flags (gemma4) are handled + // separately inside the builder; has_swa catches newly-added archs like smollm3. + config["has_swa"] = + (metadata.count(arch + ".attention.sliding_window_pattern") || + (metadata.count(arch + ".attention.sliding_window") && + // A value of 0 or UINT32_MAX typically means "no SWA"; treat only positive finite + // values as real SWA. We check the tensor value directly. + [&]() { + const auto& t = std::get(metadata.at(arch + ".attention.sliding_window")); + const uint32_t v = *t.data(); + return v > 0 && v < 0xFFFFFFFFu; + }())) + ? 1 : 0; + + // gpt-oss SWA: alternation period (default 2: even layers are SWA). Matches llama.cpp's + // set_swa_pattern(swa_period, dense_first=false): il is SWA if il % period < period - 1. + // Gemma4: sliding_window_pattern is a boolean array (one entry per layer); stored as + // an ov::Tensor of element::boolean. Detect by checking the variant type. + if (metadata.count(arch + ".attention.sliding_window_pattern")) { + const auto& v = metadata.at(arch + ".attention.sliding_window_pattern"); + if (std::holds_alternative(v)) { + const auto& t = std::get(v); + if (t.get_element_type() == ov::element::boolean) { + // Boolean array: convert to vector (1=SWA, 0=global) for the builder. + const size_t n = t.get_size(); + std::vector swa_flags(n); + const auto* bdata = t.data(); + for (size_t i = 0; i < n; ++i) + swa_flags[i] = bdata[i] ? 1 : 0; + config["swa_layer_flags"] = swa_flags; + config["swa_layer_pattern"] = 0; // 0 = use per-layer flags, not period + } else { + config["swa_layer_pattern"] = metadata_to_int(metadata, arch + ".attention.sliding_window_pattern"); + config["swa_layer_flags"] = std::vector{}; + } + } else { + config["swa_layer_pattern"] = metadata_to_int(metadata, arch + ".attention.sliding_window_pattern"); + config["swa_layer_flags"] = std::vector{}; + } + } else { + // No explicit pattern key. gemma3 defaults to period 6 (llama.cpp gemma3 load_arch_hparams + // passes swa_period=6 to get_key_or_arr); gpt-oss and others default to 2. + config["swa_layer_pattern"] = (arch == "gemma3") ? 6 : 2; + config["swa_layer_flags"] = std::vector{}; + } + + // gpt-oss MoE: optional per-expert routing weight scale applied after softmax (0 = 1.0 no-op). + config["expert_weights_scale"] = metadata.count(arch + ".expert_weights_scale") + ? metadata_to_float(metadata, arch + ".expert_weights_scale") + : 0.0f; + + // Gemma2 attention soft-cap: tanh(QK^T * (1/cap)) * cap applied inside the attention. + // 0.0 means no soft-cap (default for all non-Gemma2 architectures). + config["attn_logit_softcapping"] = metadata.count(arch + ".attn_logit_softcapping") + ? metadata_to_float(metadata, arch + ".attn_logit_softcapping") + : 0.0f; + + // Gemma2/Gemma3 final logit soft-cap applied after lm_head: tanh(x/cap)*cap. + // 0.0 means no soft-cap. + config["final_logit_softcapping"] = metadata.count(arch + ".final_logit_softcapping") + ? metadata_to_float(metadata, arch + ".final_logit_softcapping") + : 0.0f; + + return config; +} + +} // namespace gguf +} // namespace frontend +} // namespace ov diff --git a/src/frontends/gguf/src/quant/gguf.hpp b/src/frontends/gguf/src/quant/gguf.hpp index e0b0a6b03360cf..12b6992ae2afa4 100644 --- a/src/frontends/gguf/src/quant/gguf.hpp +++ b/src/frontends/gguf/src/quant/gguf.hpp @@ -112,8 +112,8 @@ void gguf_fill_asym(const gguf_tensor& tensor, ov::Tensor& weights, ov::Tensor& // Fill pre-allocated f4e2m1 weights and f8e8m0 scales from an MXFP4 GGUF tensor. void gguf_fill_mxfp4(const gguf_tensor& tensor, ov::Tensor& weights, ov::Tensor& scales); -// Fill pre-allocated u2 weights, f16 scales and zero-points from a Q2_0 (ternary) tensor. -// The zero-point is the constant 1 for every block. +// Fill pre-allocated u2 weights, f16 scales and u8 zero-points from a Q2_0 (ternary) tensor. +// The zero-point is the constant 1 for every block: value = (code - 1) * scale. void gguf_fill_q2_0(const gguf_tensor& tensor, ov::Tensor& weights, ov::Tensor& scales, ov::Tensor& zp); // Fused bit-exact ggml dequant + channel-wise Q8_0_C requant for the token_embd/output/Q6_K/Q5_K @@ -138,9 +138,19 @@ void dequant_row_q6_k_f32_for_test(const uint8_t* row, size_t cols, float* y); // repacked weight/scale/bias data lives in one allocation (IR-frontend pattern). GGUFLoad get_gguf_data(const std::string& file); -// Extract the architecture config (architecture, layer_num, head_num, head_size, -// head_num_kv, hidden_size, max_position_embeddings, rms_norm_eps, rope_freq_base, -// file_type) from parsed metadata. -std::map config_from_meta(const std::unordered_map& metadata); +// Extract the DECODER-family architecture config (architecture, layer_num, head_num, head_size, +// head_num_kv, hidden_size, max_position_embeddings, rms_norm_eps, rope_freq_base, file_type, ...) +// from parsed metadata. +// +// Every key it reads is prefixed with the LLM architecture name (".block_count", +// ".attention.head_count", ...), so it is only meaningful for a causal-decoder GGUF. Call +// detect_model_kind() first: an mmproj file names its architecture "clip" and carries "clip.*" +// keys instead, and would fail here on a missing block_count. A future non-decoder family gets its +// own reader next to this one rather than extending it. +std::map decoder_config_from_meta( + const std::unordered_map& metadata); + +// Reverse of the GGML dimension order (GGUF stores dims fastest-first). +ov::Shape get_shape(const gguf_tensor& tensor); } // namespace ov::frontend::gguf diff --git a/src/frontends/gguf/src/quant/gguf_quants.cpp b/src/frontends/gguf/src/quant/gguf_quants.cpp index c89b8692f9d7b1..d86aaa5c1739e4 100644 --- a/src/frontends/gguf/src/quant/gguf_quants.cpp +++ b/src/frontends/gguf/src/quant/gguf_quants.cpp @@ -640,8 +640,10 @@ void fill_q8_k(const gguf_tensor& tensor, ov::Tensor& weights_arr, ov::Tensor& s }); } -// Block = |f16 d|u2 qs[64]| (18 bytes / 64 weights). ggml packs the codes 4 per byte LSB-first, -// the same order an OpenVINO u2 Constant reads, so the code bytes are copied verbatim. +// Q2_0 ternary: block = |f16 d|u2 qs[64]| (18 bytes / 64 weights), value = (code - 1) * d with +// code in [0..3] -> {-1, 0, +1, +2}. ggml packs the codes 4 per byte LSB-first +// (dequantize_row_q2_0: `(qs[j/4] >> ((j%4)*2)) & 3`), which is exactly the order OpenVINO's u2 +// Constant reads, so the 16 code bytes are copied verbatim. The zero-point is the constant 1. void gguf_fill_q2_0(const gguf_tensor& tensor, ov::Tensor& weights_arr, ov::Tensor& scales_arr, ov::Tensor& zp_arr) { const uint64_t bytes_per_block = 18; const uint64_t bytes_per_block_codes = 16; diff --git a/src/frontends/gguf/src/quant/weights.cpp b/src/frontends/gguf/src/quant/weights.cpp index 1599c03c43aa38..aa781545825305 100644 --- a/src/frontends/gguf/src/quant/weights.cpp +++ b/src/frontends/gguf/src/quant/weights.cpp @@ -10,9 +10,12 @@ #include "weights.hpp" +#include #include #include #include +#include +#include #include #include "openvino/core/except.hpp" @@ -35,12 +38,45 @@ namespace { const ov::Tensor& get(const std::unordered_map& weights, const std::string& key) { auto it = weights.find(key); - OPENVINO_ASSERT(it != weights.end(), "[ggml] missing weight tensor: ", key); + OPENVINO_ASSERT(it != weights.end(), "[GGUF] missing weight tensor: ", key); return it->second; } +// Copy rows [r0, r1) out of a 2D tensor. Rows are block-independent in every GGUF quant layout +// (a full row's worth of blocks is contiguous), so a fused attn_qkv weight can be split into +// q/k/v by a plain byte-range row copy without touching the quant blocks. +ov::Tensor slice_rows(const ov::Tensor& t, size_t r0, size_t r1) { + const auto& s = t.get_shape(); + OPENVINO_ASSERT(s.size() == 2 && r1 <= s[0] && r0 <= r1, "[GGUF] bad row slice"); + ov::Shape out_shape{r1 - r0, s[1]}; + ov::Tensor out(t.get_element_type(), out_shape); + const size_t row_bytes = t.get_byte_size() / s[0]; + std::memcpy(out.data(), static_cast(t.data()) + r0 * row_bytes, (r1 - r0) * row_bytes); + return out; +} + +// Gather rows in a repeating per-block pattern: for every `block` consecutive rows, take +// [0, take) into the result. qwen35's attn_q interleaves query and gate per head as +// [q_h0 | gate_h0 | q_h1 | gate_h1 | ...], so the query is gather(block=2*head_dim, +// take=head_dim, offset=0) and the gate the same with offset=head_dim. Like slice_rows this +// works on raw row bytes, which is safe for the packed types because a quantization block +// never spans two rows. +ov::Tensor gather_rows_strided(const ov::Tensor& t, size_t block, size_t take, size_t offset) { + const auto& s = t.get_shape(); + OPENVINO_ASSERT(s.size() == 2 && block > 0 && offset + take <= block && s[0] % block == 0, + "[GGUF] bad strided row gather"); + const size_t n_blocks = s[0] / block; + ov::Tensor out(t.get_element_type(), ov::Shape{n_blocks * take, s[1]}); + const size_t row_bytes = t.get_byte_size() / s[0]; + const auto* src = static_cast(t.data()); + auto* dst = static_cast(out.data()); + for (size_t b = 0; b < n_blocks; ++b) { + std::memcpy(dst + b * take * row_bytes, src + (b * block + offset) * row_bytes, take * row_bytes); + } + return out; +} + -// Shared shape helpers for grouped weight layouts. See make_int8 comment for why we keep // all leading dims separate rather than flattening: the trailing Reshape must be // (orig_rank+1)D -> orig_rank for the CompressedWeightsBlock matcher to fire. ov::Shape grouped_weight_shape(const ov::Shape& orig, size_t num_groups, size_t group_size) { @@ -399,6 +435,50 @@ bool needs_q8_0_c_requant(const std::string& name, gguf_tensor_type qtype) { } // namespace +void notify_lossy_weight_approximation(LossyWeightApproximation kind) { + // Written to std::cerr, NOT OPENVINO_WARN: the latter expands to a no-op unless the build sets + // ENABLE_OPENVINO_DEBUG (cmake/features.cmake defaults it OFF), so in every shipped build it + // reaches nobody -- and this notice exists precisely to reach the user. It is a deliberate, + // permanent, at-most-once-per-process diagnostic, not leftover tracing. + // + // One flag per kind, so a model that hits both approximations reports both -- but each at most + // once, however many thousands of weights are affected. + static std::once_flag requant_once; + static std::once_flag zero_point_once; + + switch (kind) { + case LossyWeightApproximation::Q8_0_C_Requant: + std::call_once(requant_once, [] { + std::cerr << "[GGUF] accuracy notice: the token embedding / output / Q6_K / Q5_K weights are " + "requantized channel-wise to Q8_0_C (one int8 scale per row). This is lossy, so results " + "may differ slightly from the original GGUF weights. It reproduces the llama.cpp " + "ggml-openvino backend's weight pipeline. Reported once per process." + << std::endl; + }); + break; + case LossyWeightApproximation::IntegerZeroPoint: + std::call_once(zero_point_once, [] { + std::cerr << "[GGUF] accuracy notice: Q4_K weights use an integer (u8) zero-point, which rounds each " + "sub-block's minimum to a multiple of its scale. This is lossy, so results may differ " + "slightly from the original GGUF weights. It keeps the dequantization foldable into an " + "int8 MatMul, which is roughly twice as fast at prefill. Reported once per process." + << std::endl; + }); + break; + } +} + +ov::element::Type gguf_zero_point_type(const std::string& name, gguf_tensor_type qtype) { + // The CPU compressed-FullyConnected fast path only folds the dequant when the zero-point is an + // INTEGER constant; a fractional f16 one leaves a ~2x slower kernel. Q4_K carries the matmul + // weights of modern models and Q2_0's zp is the exact integer 1, so both use u8. The others + // keep a faithful f16 zp: their zp = min/scale can exceed u8 range, and rounding it injects + // error into every weight. Tensors that are requantized to Q8_0_C are excluded -- their dequant + // feeds the channel-wise path, not a compressed FC. + const bool integer_zp = (qtype == GGUF_TYPE_Q4_K || qtype == GGUF_TYPE_Q2_0); + return (integer_zp && !needs_q8_0_c_requant(name, qtype)) ? ov::element::u8 : ov::element::f16; +} + std::shared_ptr make_weight_node(const std::string& base, const std::unordered_map& weights, const std::unordered_map& qtypes) { @@ -478,16 +558,99 @@ gguf_tensor_type gguf_type_from_name(const std::string& quant_type) { ch = static_cast(std::toupper(static_cast(ch))); } auto it = names.find(key); - OPENVINO_ASSERT(it != names.end(), "[ggml] unsupported weight quant type: ", quant_type); + OPENVINO_ASSERT(it != names.end(), "[GGUF] unsupported weight quant type: ", quant_type); return it->second; } +std::array split_fused_qkv_extracted( + const std::string& base, + const std::unordered_map& weights, + const std::unordered_map& qtypes, + size_t n_q, + size_t n_k, + size_t n_v) { + gguf_tensor_type qtype = GGUF_TYPE_F16; + if (auto it = qtypes.find(base + ".qtype"); it != qtypes.end()) { + qtype = it->second; + } + const bool has_scales = qtype == GGUF_TYPE_Q4_0 || qtype == GGUF_TYPE_Q4_1 || qtype == GGUF_TYPE_Q4_K || + qtype == GGUF_TYPE_Q5_0 || qtype == GGUF_TYPE_Q5_1 || qtype == GGUF_TYPE_Q8_0 || + qtype == GGUF_TYPE_Q2_K || qtype == GGUF_TYPE_Q3_K || qtype == GGUF_TYPE_Q5_K || + qtype == GGUF_TYPE_Q6_K || qtype == GGUF_TYPE_Q2_0; + const bool has_zp = qtype == GGUF_TYPE_Q4_1 || qtype == GGUF_TYPE_Q4_K || qtype == GGUF_TYPE_Q5_K || + qtype == GGUF_TYPE_Q5_1 || qtype == GGUF_TYPE_Q2_K || qtype == GGUF_TYPE_Q2_0; + + const ov::Tensor& w = get(weights, base + ".weight"); + const size_t total_rows = w.get_shape()[0]; + OPENVINO_ASSERT(n_q + n_k + n_v == total_rows, "[GGUF] fused qkv row mismatch for ", base); + + const std::array, 3> ranges = {std::make_pair(size_t(0), n_q), + std::make_pair(n_q, n_q + n_k), + std::make_pair(n_q + n_k, total_rows)}; + const std::array parts = {base + ".q", base + ".k", base + ".v"}; + + std::array out; + for (size_t i = 0; i < 3; ++i) { + const auto [r0, r1] = ranges[i]; + out[i].qtype = qtype; + out[i].extracted[parts[i] + ".weight"] = slice_rows(w, r0, r1); + if (has_scales) { + out[i].extracted[parts[i] + ".scales"] = slice_rows(get(weights, base + ".scales"), r0, r1); + } + if (has_zp) { + out[i].extracted[parts[i] + ".zp"] = slice_rows(get(weights, base + ".zp"), r0, r1); + } + } + return out; +} + +// qwen35: attn_q packs the query and the attention output gate interleaved per head, as +// [q_h0 | gate_h0 | q_h1 | gate_h1 | ...] with a stride of 2*head_dim rows. De-interleave it +// into two plain weights so the graph sees ordinary projections. Returns {query, gate}. +std::array split_interleaved_q_gate(const std::string& base, + const std::unordered_map& weights, + const std::unordered_map& qtypes, + size_t head_dim) { + gguf_tensor_type qtype = GGUF_TYPE_F16; + if (auto it = qtypes.find(base + ".qtype"); it != qtypes.end()) { + qtype = it->second; + } + const bool has_scales = qtype == GGUF_TYPE_Q4_0 || qtype == GGUF_TYPE_Q4_1 || qtype == GGUF_TYPE_Q4_K || + qtype == GGUF_TYPE_Q5_0 || qtype == GGUF_TYPE_Q5_1 || qtype == GGUF_TYPE_Q8_0 || + qtype == GGUF_TYPE_Q2_K || qtype == GGUF_TYPE_Q3_K || qtype == GGUF_TYPE_Q5_K || + qtype == GGUF_TYPE_Q6_K || qtype == GGUF_TYPE_Q2_0; + const bool has_zp = qtype == GGUF_TYPE_Q4_1 || qtype == GGUF_TYPE_Q4_K || qtype == GGUF_TYPE_Q5_K || + qtype == GGUF_TYPE_Q5_1 || qtype == GGUF_TYPE_Q2_K || qtype == GGUF_TYPE_Q2_0; + + const ov::Tensor& w = get(weights, base + ".weight"); + const size_t block = 2 * head_dim; + OPENVINO_ASSERT(w.get_shape()[0] % block == 0, "[GGUF] interleaved q/gate row mismatch for ", base); + + const std::array parts = {base + ".q", base + ".gate"}; + const std::array offsets = {0, head_dim}; + + std::array out; + for (size_t i = 0; i < 2; ++i) { + out[i].qtype = qtype; + out[i].extracted[parts[i] + ".weight"] = gather_rows_strided(w, block, head_dim, offsets[i]); + if (has_scales) { + out[i].extracted[parts[i] + ".scales"] = + gather_rows_strided(get(weights, base + ".scales"), block, head_dim, offsets[i]); + } + if (has_zp) { + out[i].extracted[parts[i] + ".zp"] = + gather_rows_strided(get(weights, base + ".zp"), block, head_dim, offsets[i]); + } + } + return out; +} + std::shared_ptr make_weight_node(const ov::Tensor& data, const std::string& quant_type, const ov::Shape& logical_shape, const std::string& name) { OPENVINO_ASSERT(logical_shape.size() == 2, - "[ggml] weight logical shape must be 2D [rows, cols], got rank ", + "[GGUF] weight logical shape must be 2D [rows, cols], got rank ", logical_shape.size()); const uint64_t rows = logical_shape[0]; const uint64_t cols = logical_shape[1]; @@ -533,8 +696,14 @@ std::shared_ptr make_weight_node(const ov::Tensor& data, // they are not perf-critical here, and their zp = -min/scale can fall outside u8 range. The // requant path (token_embd/output) also keeps f16 -- its dequant feeds channel-wise Q8_0_C. const bool requant = needs_q8_0_c_requant(name, qtype); - const ov::element::Type zp_type = - (!requant && (qtype == GGUF_TYPE_Q4_K || qtype == GGUF_TYPE_Q2_0)) ? ov::element::u8 : ov::element::f16; + const ov::element::Type zp_type = gguf_zero_point_type(name, qtype); + if (requant) { + notify_lossy_weight_approximation(LossyWeightApproximation::Q8_0_C_Requant); + } + // Only Q4_K's integer zp actually rounds: Q2_0's zero-point is the exact integer 1. + if (zp_type == ov::element::u8 && qtype == GGUF_TYPE_Q4_K) { + notify_lossy_weight_approximation(LossyWeightApproximation::IntegerZeroPoint); + } // K-quant requant sources: the fused dequant -> Q8_0_C streams from the raw bytes, so skip the // full-tensor gguf_fill_* extraction below (it would be discarded) and return before the switch. @@ -631,7 +800,7 @@ std::shared_ptr make_weight_node(const ov::Tensor& data, break; } default: - OPENVINO_THROW("[ggml] unsupported weight quant type: ", quant_type); + OPENVINO_THROW("[GGUF] unsupported weight quant type: ", quant_type); } // Non-K requant sources (e.g. an F16 / Q4_0 / Q8_0 token_embd or output): the K-quant fast path diff --git a/src/frontends/gguf/src/quant/weights.hpp b/src/frontends/gguf/src/quant/weights.hpp index fa7cac374c6a82..3a745898da4e0e 100644 --- a/src/frontends/gguf/src/quant/weights.hpp +++ b/src/frontends/gguf/src/quant/weights.hpp @@ -4,6 +4,7 @@ #pragma once +#include #include #include #include @@ -16,6 +17,29 @@ class Node; namespace ov::frontend::gguf { +// Element type of the zero-point constant for an asymmetric quantized weight. Both ingest +// paths must agree on this: it decides whether the CPU folds the dequant into the MatMul. +ov::element::Type gguf_zero_point_type(const std::string& name, gguf_tensor_type qtype); + +// A lossy weight approximation the frontend deliberately makes, reported to the user once so a +// later accuracy investigation starts from the right place. +enum class LossyWeightApproximation { + // token_embd / output / Q6_K / Q5_K tensors requantized channel-wise to Q8_0_C. + Q8_0_C_Requant, + // Q4_K asymmetric weights expressed with an INTEGER (u8) zero-point, which forces each + // sub-block's min to a multiple of its scale. + IntegerZeroPoint, +}; + +// Warn -- ONCE per process and per approximation kind -- that weights are being converted with a +// lossy approximation, so generated text may differ slightly from the original GGUF weights. +// +// Once per process rather than once per model: the message describes a fixed property of the +// conversion strategy, not of one tensor or one file, and a model has thousands of affected +// weights (every Q4_K matmul), so anything finer-grained would bury the log. Call it at the point +// the approximation is actually PERFORMED, not where its parameters are queried. +void notify_lossy_weight_approximation(LossyWeightApproximation kind); + // Build the OpenVINO node for a GGUF weight with base name `base` (the tensor name without // the trailing ".weight", e.g. "blk.0.attn_q" or "token_embd"). Quantized weights become a // low-bitness compressed subgraph (u4/u8 weights + zero-point + f16 scale, Convert -> @@ -48,4 +72,32 @@ std::shared_ptr make_weight_node(const ov::Tensor& data, // Map a ggml quant type name (e.g. "Q4_K") to its gguf_tensor_type id. Throws if unknown. gguf_tensor_type gguf_type_from_name(const std::string& quant_type); +// One split part of a fused attn_qkv weight: the extracted tensors keyed as ".weight" +// [+ ".scales" [+ ".zp"]] plus the shared quant type. Used by the GGUF builder to emit a +// GGML_OP_NONE weight leaf per q/k/v part (routing them through translate_weight like any other +// weight) instead of building the decompression nodes eagerly. +struct FusedQkvPart { + std::unordered_map extracted; + gguf_tensor_type qtype = GGUF_TYPE_F16; +}; + +// Row-slice a fused `` attn_qkv weight into q/k/v extracted-tensor sub-maps (no OV nodes). +// The returned parts' tensors are keyed ".q.weight"/".scales"/".zp" etc. Same slicing as +// make_fused_qkv_weights, but returns the extracted payload for GGML_OP_NONE emission. +std::array split_fused_qkv_extracted( + const std::string& base, + const std::unordered_map& weights, + const std::unordered_map& qtypes, + size_t n_q, + size_t n_k, + size_t n_v); + +// De-interleave a qwen35 `` attn_q weight, which packs the query and the attention output +// gate per head as [q_h0 | gate_h0 | q_h1 | gate_h1 | ...], into two plain projections. +// Returns {query, gate}, keyed ".q.*" and ".gate.*". +std::array split_interleaved_q_gate(const std::string& base, + const std::unordered_map& weights, + const std::unordered_map& qtypes, + size_t head_dim); + } // namespace ov::frontend::gguf diff --git a/src/frontends/gguf/src/tokenizer_metadata.cpp b/src/frontends/gguf/src/tokenizer_metadata.cpp new file mode 100644 index 00000000000000..b596c51b002982 --- /dev/null +++ b/src/frontends/gguf/src/tokenizer_metadata.cpp @@ -0,0 +1,18 @@ +// Copyright (C) 2018-2026 Intel Corporation +// SPDX-License-Identifier: Apache-2.0 +// + +#include "openvino/frontend/gguf/tokenizer_metadata.hpp" + +namespace ov { +namespace frontend { +namespace gguf { + +const std::string& gguf_tokenizer_metadata_key() { + static const std::string key = "gguf_tokenizer_metadata"; + return key; +} + +} // namespace gguf +} // namespace frontend +} // namespace ov diff --git a/src/frontends/gguf/src/translate_session.cpp b/src/frontends/gguf/src/translate_session.cpp index 3e37dd6e987173..ae0c239d66e68e 100644 --- a/src/frontends/gguf/src/translate_session.cpp +++ b/src/frontends/gguf/src/translate_session.cpp @@ -4,15 +4,26 @@ #include "translate_session.hpp" +#include #include #include #include #include +#include + +#include "input_model.hpp" +#include "node_context.hpp" +#include "openvino/core/graph_util.hpp" #include "openvino/core/node.hpp" +#include "openvino/core/rt_info/weightless_caching_attributes.hpp" +#include "openvino/frontend/gguf/make_stateful.hpp" +#include "openvino/frontend/gguf/tokenizer_metadata.hpp" #include "openvino/op/add.hpp" #include "openvino/op/broadcast.hpp" #include "openvino/op/concat.hpp" +#include "openvino/op/constant.hpp" #include "openvino/op/convert.hpp" +#include "openvino/op/convert_like.hpp" #include "openvino/op/cos.hpp" #include "openvino/op/divide.hpp" #include "openvino/op/gather.hpp" @@ -27,11 +38,8 @@ #include "openvino/op/strided_slice.hpp" #include "openvino/op/transpose.hpp" #include "openvino/pass/constant_folding.hpp" - -#include "input_model.hpp" -#include "node_context.hpp" -#include "openvino/core/rt_info/weightless_caching_attributes.hpp" #include "pass/lower_set_rows_stateless.hpp" +#include "transformations/common_optimizations/nop_elimination.hpp" #include "transformations/fp16_compression/mark_decompression_convert_constant_folding.hpp" #include "transformations/op_conversions/convert_convertlike.hpp" #include "utils.hpp" @@ -68,7 +76,12 @@ void add_sliced_mask(TensorMap& tensor_map) { create_sliced_mask("self_kq_mask_swa", "KQ_mask_swa_sliced"); } -void add_rope_sin_cos(TensorMap& tensor_map, const RopeConfig& rope_config) { +void add_rope_sin_cos(TensorMap& tensor_map, GgufDecoder& gguf_model_decoder) { + // A decoder bound to a full LLM graph exposes "rope_config"; a decoder wrapping a bare op / + // small cgraph (a single-op test) has no such attribute -> default RopeConfig (n_dims == 0, + // "no RoPE") so the shared table is skipped and the op falls back to its own sin/cos. + const auto rope_config_any = gguf_model_decoder.get_attribute("rope_config"); + const auto rope_config = rope_config_any.empty() ? RopeConfig{} : rope_config_any.as(); // n_dims == 0 means the model uses no RoPE; per_op means each ROPE op builds its own sin/cos // (e.g. gemma4 where SWA and global layers differ), so skip the shared table entirely. if (tensor_map.find("inp_pos") == tensor_map.end() || rope_config.n_dims == 0 || rope_config.per_op) { @@ -91,9 +104,9 @@ void add_rope_sin_cos(TensorMap& tensor_map, const RopeConfig& rope_config) { } // Create common patterns -void preprocess(TensorMap& tensor_map, const RopeConfig& rope_config) { +void preprocess(TensorMap& tensor_map, GgufDecoder& gguf_model_decoder) { add_sliced_mask(tensor_map); - add_rope_sin_cos(tensor_map, rope_config); + add_rope_sin_cos(tensor_map, gguf_model_decoder); } } // namespace @@ -121,26 +134,42 @@ std::shared_ptr TranslateSession::translate_graph(const frontend::InputMo std::shared_ptr resulting_model; const auto& gguf_model = std::dynamic_pointer_cast(input_model); + std::shared_ptr gguf_model_decoder = gguf_model->get_model_decoder(); + + // An auxiliary input Parameter whose only consumer may be created by a later normalization pass, + // after the unused-Parameter pruning below. Track them so pruning never drops one for lack of a + // consumer at translate time. A pass that ends up not consuming one leaves it as a dangling + // input, which later constant folding removes. + std::set deferred_use_params; + + for (const auto& it : gguf_model_decoder->get_model_inputs()) { + params.push_back(std::dynamic_pointer_cast(it.second)); + (*tensor_map)[it.first] = it.second; + } - for (const auto& it : gguf_model->get_model_inputs()) { - if (auto param = std::dynamic_pointer_cast(it.second)) { - params.push_back(param); + for (const auto& it : gguf_model_decoder->get_model_extra_inputs()) { + if (auto p = std::dynamic_pointer_cast(it.second)) { + params.push_back(p); + deferred_use_params.insert(p.get()); } (*tensor_map)[it.first] = it.second; } - // Weights are not seeded here: a weight is visited as a regular "GGML_OP_NONE" node (a ggml - // leaf carrying a "data" attribute) in visit_subgraph, and translate_weight writes its - // dequantized node into the tensor map under the weight name, before the consuming op is - // visited (the cgraph is topologically ordered). + // Weights are not seeded here: every decoder surfaces them as "GGML_OP_NONE" leaves that + // translate_weight turns into a compressed subgraph during the walk below, which keeps them + // lazy (never materialized to f32) and keeps one weight-loading path for both ingest paths. auto node_visitor = [&](std::shared_ptr decoder) { auto operation_type = decoder->get_op_type(); if (operation_type == "GGML_OP_NONE") { - // A GGML_OP_NONE leaf is a weight only if the decoder exposes its raw bytes via the - // "data" attribute; otherwise it is a model-input leaf (already seeded as a Parameter + // A GGML_OP_NONE leaf is a weight if the decoder marks it as one: either the native + // builder's pre-extracted payload (bool "gguf_weight") or the cgraph decoder's raw + // bytes ("data"). Otherwise it is a model-input leaf (already seeded as a Parameter // above) and there is nothing to translate. - if (!decoder->get_attribute("data").is()) { + const bool is_builder_weight = decoder->get_attribute("gguf_weight").is() && + decoder->get_attribute("gguf_weight").as(); + const bool is_cgraph_weight = decoder->get_attribute("data").is(); + if (!is_builder_weight && !is_cgraph_weight) { return; } } @@ -178,10 +207,10 @@ std::shared_ptr TranslateSession::translate_graph(const frontend::InputMo // the model uses a shared rope table (n_dims != 0, not per-op). For a bare op / small cgraph // (no rope_config -> default n_dims == 0, no mask/pos inputs) both no-op, and the ROPE/attention // translators fall back to building their own -- so there is no separate "naive" mode. - preprocess(*tensor_map, gguf_model->get_rope_config()); - gguf_model->visit_subgraph(node_visitor); + preprocess(*tensor_map, *gguf_model_decoder); + gguf_model_decoder->visit_subgraph(node_visitor); - for (const auto& name : gguf_model->get_model_output_names()) { + for (const auto& name : gguf_model_decoder->get_model_output_names()) { FRONT_END_GENERAL_CHECK(tensor_map->find(name) != tensor_map->end(), "Output name not found in tensor map: ", name); @@ -192,13 +221,51 @@ std::shared_ptr TranslateSession::translate_graph(const frontend::InputMo ov::ParameterVector used_params; for (const auto& param : params) { - if (!param->output(0).get_target_inputs().empty()) { + // Keep a Parameter if it currently feeds something, OR if its consumer is created by a + // later normalization pass. + if (!param->output(0).get_target_inputs().empty() || deferred_use_params.count(param.get())) { used_params.push_back(param); } } resulting_model = std::make_shared(results, used_params); - apply_transformations(resulting_model); + // M-RoPE models need 4 position sections per token, which changes the position_ids contract a + // GenAI-facing consumer has to satisfy; record it so AdaptToGenAI can adapt (it runs on the + // model alone and cannot ask the decoder). + { + const auto rc_any = gguf_model_decoder->get_attribute("rope_config"); + if (!rc_any.empty() && rc_any.as().is_imrope) { + resulting_model->get_rt_info()[pass::gguf_imrope_key()] = true; + } + } + + // Record the recurrent (overwritten, non-appending) states BEFORE transformations: a caller + // that registered MakeStateful runs it inside apply_transformations, and unlike a KV cache + // these carry nothing in the graph that identifies them (see gguf_recurrent_states_key). + // Flattened to alternating {input, output} because rt_info takes an ov::Any. + { + const auto& rs = gguf_model_decoder->get_recurrent_states(); + if (!rs.empty()) { + std::vector flat; + flat.reserve(rs.size() * 2); + for (const auto& kv : rs) { + flat.push_back(kv.first); + flat.push_back(kv.second); + } + resulting_model->get_rt_info()[pass::gguf_recurrent_states_key()] = flat; + } + } + + resulting_model = apply_transformations(resulting_model); + + // Attach GGUF tokenizer metadata (the file's tokenizer.* keys) to the model's rt_info as a + // non-serializable attribute, so a downstream consumer (OpenVINO GenAI) can build the + // tokenizer without reopening the .gguf. Empty when the decoder carries no tokenizer config. + const auto& tok_cfg = gguf_model_decoder->get_tokenizer_config(); + if (!tok_cfg.empty()) { + resulting_model->get_rt_info()[gguf_tokenizer_metadata_key()] = + std::make_shared(tok_cfg); + } // Set WeightlessCacheAttribute on large constants to avoid unnecessary memory copies // in the NPUW plugin. Without this attribute, NPUW's LazyTensor constructor @@ -216,7 +283,8 @@ std::shared_ptr TranslateSession::translate_graph(const frontend::InputMo size_t offset = 0; for (auto& node : resulting_model->get_ordered_ops()) { if (auto cnst = ov::as_type_ptr(node); - cnst && cnst->get_byte_size() / cnst->get_element_type().size() >= 16) { + cnst && cnst->get_element_type().size() > 0 && + cnst->get_byte_size() / cnst->get_element_type().size() >= 16) { auto& rt_info = cnst->get_rt_info(); if (rt_info.find(ov::WeightlessCacheAttribute::get_type_info_static()) == rt_info.end()) { rt_info[ov::WeightlessCacheAttribute::get_type_info_static()] = @@ -232,17 +300,23 @@ std::shared_ptr TranslateSession::apply_transformations(std::shared_ptr(); - // Caller-registered transformation extensions run first. A SetRows-lowering extension (e.g. - // the backend's stateful lowering) consumes the KV-cache SetRows ops here; the built-in stateless - // lowering below then only fires on the ops left untouched. With no extension registered, the - // stateless lowering handles every SetRows op -- so a plain convert() yields the - // llama.cpp-faithful stateless model. + // Caller-registered transformation extensions run first, which is what makes execution mode a + // caller concern rather than a frontend one: an extension that lowers SetRows itself -- e.g. + // ov::frontend::gguf::pass::MakeStateful, or a backend's own variant -- consumes the KV-cache + // SetRows ops here, and the built-in stateless lowering below then only fires on the ops left + // untouched (MoE routing writes and the like). With no extension registered, the stateless + // lowering handles every SetRows op, so a plain convert() yields the stateless model. for (const auto& ext : m_transformation_extensions) { ext->register_pass(manager); } manager.register_pass(); - manager.register_pass(); + manager.register_pass(); + // The lowered ConvertLikes are frequently no-ops (k/v already share q's precision). Drop them: + // a same-type Convert on an SDPA k/v input is invisible to the plugins but breaks + // StateManagementPattern, which admits no Convert between the KV-cache Concat and SDPA, and so + // silently disables the PagedAttention backend for every GGUF model. + manager.register_pass(); manager.run_passes(model); return model; } diff --git a/src/frontends/gguf/src/utils.cpp b/src/frontends/gguf/src/utils.cpp index 32f99fd37b144b..5387e48a593bb0 100644 --- a/src/frontends/gguf/src/utils.cpp +++ b/src/frontends/gguf/src/utils.cpp @@ -34,6 +34,18 @@ void num_inputs_check(const NodeContext& context, size_t min_inputs, size_t max_ FRONT_END_OP_CONVERSION_CHECK(input_size <= max_inputs, "Got more inputs than expected"); } +int non_cont_dim(std::vector ne, std::vector nb) { + int dim = nb.size() - 1; + size_t bytes = nb[dim]; + for (int i = dim; i > 0; i--) { + bytes *= ne[i]; + if (bytes != nb[i - 1]) { + return i; + } + } + return 0; +} + std::shared_ptr get_dimensions(const std::shared_ptr& shape, const std::vector& dims) { using namespace ov::op; @@ -57,6 +69,22 @@ OutputVector rename_outputs_with_suffix(const OutputVector& outputs, const std:: return outputs; } +ov::Output make_topk_indices(const ov::Output& input, + const ov::Output& k, + int64_t axis, + ov::op::v11::TopK::Mode mode, + const ov::element::Type& index_type, + bool stable) { + auto topk = std::make_shared(input, + k, + axis, + mode, + ov::op::v11::TopK::SortType::SORT_VALUES, + index_type, + stable); + return topk->output(1); // indices +} + namespace { ov::Output rope_yarn_ramp_mix(int n_dims, const float corr_dims[2], float ext_factor) { int half_n_dims = n_dims / 2; @@ -106,8 +134,16 @@ void gguf_rope_yarn_corr_dims(int n_dims, std::pair, ov::Output> make_sin_cos(const RopeConfig& rope_config, std::shared_ptr inp_pos, std::shared_ptr rope_freqs_weight, - bool imrope) { - if (imrope) { + bool imrope, + bool stateful) { + if (stateful) { + inp_pos = + std::make_shared(inp_pos, ov::op::v0::Constant::create(ov::element::i64, {1}, {0})); + inp_pos = std::make_shared(inp_pos, ov::element::f32); + auto pos_perm = + std::make_shared(ov::element::i64, ov::Shape{3}, std::vector{2, 1, 0}); + inp_pos = std::make_shared(inp_pos, pos_perm); + } else if (imrope) { inp_pos = std::make_shared(inp_pos, ov::element::f32); auto pos_shape = ov::op::v0::Constant::create(ov::element::i64, ov::Shape{5}, {0, 0, 0, 4, -1}); inp_pos = std::make_shared(inp_pos, pos_shape, true); @@ -158,9 +194,27 @@ std::pair, ov::Output> make_sin_cos(const RopeConfig& rop for (size_t i = 1; i < factor.size(); i++) { factor[i] = theta_scale * factor[i - 1]; } - freq_factors = - std::make_shared(ov::element::f32, ov::Shape{1, 1, 1, factor.size()}, factor); + if (stateful) { + freq_factors = + std::make_shared(ov::element::f32, ov::Shape{1, 1, factor.size()}, factor); + } else { + freq_factors = + std::make_shared(ov::element::f32, ov::Shape{1, 1, 1, factor.size()}, factor); + } if (rope_freqs_weight) { + // rope_freqs_weight has shape [N] for the model's maximum n_dims/2. When this + // ROPE op uses a smaller n_dims (e.g. gemma4 SWA layers), slice to n_dims_half. + auto rfw_shape = rope_freqs_weight->get_output_partial_shape(0); + if (rfw_shape.is_static() && rfw_shape.size() == 1 && + rfw_shape[0].get_length() > static_cast(n_dims_half)) { + auto start = ov::op::v0::Constant::create(ov::element::i64, {1}, {0}); + auto stop = ov::op::v0::Constant::create(ov::element::i64, {1}, {static_cast(n_dims_half)}); + auto step = ov::op::v0::Constant::create(ov::element::i64, {1}, {1}); + auto axes = ov::op::v0::Constant::create(ov::element::i64, {1}, {0}); + rope_freqs_weight = + std::make_shared(rope_freqs_weight, start, stop, step, axes)->output(0) + .get_node_shared_ptr(); + } freq_factors = std::make_shared(freq_factors, rope_freqs_weight); } @@ -173,7 +227,12 @@ std::pair, ov::Output> make_sin_cos(const RopeConfig& rop theta = theta_interp; } else { auto ramp_mix = rope_yarn_ramp_mix(n_dims, corr_dims, ext_factor); - Output one = ov::op::v0::Constant::create(ov::element::f32, Shape{1, 1, 1, 1}, {1.0f}); + Output one; + if (stateful) { + one = ov::op::v0::Constant::create(ov::element::f32, Shape{1, 1, 1}, {1.0f}); + } else { + one = ov::op::v0::Constant::create(ov::element::f32, Shape{1, 1, 1, 1}, {1.0f}); + } auto one_minus_ramp = std::make_shared(one, ramp_mix); theta = @@ -200,6 +259,8 @@ ov::Output process_view_input(const NodeContext& context, int input_in // Only works for VIEW operations that slice at the lowest dimension // If the VIEW also reshape the result, `slice_len` should be provided auto input = context.get_input(input_index); + // The decoder already returns the view start offset in ELEMENTS (it divides ggml's raw byte + // offset by the element size), so no stride division is needed here. int64_t split_addr = context.get_input_view_element_offset(input_index); auto begin = ov::op::v0::Constant::create(ov::element::i64, {1}, {split_addr}); diff --git a/src/frontends/gguf/src/utils.hpp b/src/frontends/gguf/src/utils.hpp index 8ce6c9b76ddc6c..82c40ad33c828c 100644 --- a/src/frontends/gguf/src/utils.hpp +++ b/src/frontends/gguf/src/utils.hpp @@ -7,6 +7,10 @@ #include #include +#include "openvino/core/node.hpp" +#include "openvino/op/shape_of.hpp" +#include "openvino/op/topk.hpp" + #include "openvino/core/node_vector.hpp" #include "node_context.hpp" @@ -23,6 +27,18 @@ namespace gguf { void num_inputs_check(const NodeContext& context, size_t min_inputs, size_t max_inputs); +int non_cont_dim(std::vector ne, std::vector nb); + +template +std::vector permute(const std::vector& x, const std::vector& perm) { + std::vector result; + result.reserve(perm.size()); + for (size_t i : perm) { + result.push_back(x[i]); + } + return result; +} + std::shared_ptr get_dimensions(const std::shared_ptr& shape, const std::vector& dims); // Takes the Output rather than the node so a producer with several outputs keeps the right port. @@ -30,10 +46,31 @@ std::shared_ptr get_dimensions(const ov::Output& output, con OutputVector rename_outputs_with_suffix(const OutputVector& outputs, const std::string& suffix); +/// \brief Build a TopK over `axis` and return its INDICES port. +/// +/// Shared by the ARGSORT and TOP_K translators. Both want ggml's "indices that sort/select along +/// ne[0]" semantics, which in OpenVINO is output(1) of a TopK whose index element type follows the +/// decoder's "output_type" attribute. Keeping that contract in one place stops the two call sites +/// from drifting apart. +/// +/// \param input tensor to sort/select over +/// \param k number of elements to keep along `axis` (may be a dynamic value) +/// \param axis axis to operate on +/// \param mode MAX for descending, MIN for ascending +/// \param index_type element type of the returned indices +/// \param stable whether ties keep their input order +ov::Output make_topk_indices(const ov::Output& input, + const ov::Output& k, + int64_t axis, + ov::op::v11::TopK::Mode mode, + const ov::element::Type& index_type, + bool stable = false); + std::pair, ov::Output> make_sin_cos(const RopeConfig& rope_config, std::shared_ptr inp_pos, std::shared_ptr rope_freqs_weight = nullptr, - bool imrope = false); + bool imrope = false, + bool stateful = false); ov::Output process_view_input(const NodeContext& context, int input_index, int slice_len = 0); diff --git a/src/frontends/gguf/tests/CMakeLists.txt b/src/frontends/gguf/tests/CMakeLists.txt index 4eb970883a1806..5cfd575662dfd7 100644 --- a/src/frontends/gguf/tests/CMakeLists.txt +++ b/src/frontends/gguf/tests/CMakeLists.txt @@ -14,9 +14,25 @@ set(FRONTEND_SRCS "${FE_SRC_DIR}/translate_session.cpp" "${FE_SRC_DIR}/utils.cpp" "${FE_SRC_DIR}/helper_ops/set_rows_op.cpp" + "${FE_SRC_DIR}/pass/adapt_to_genai.cpp" "${FE_SRC_DIR}/pass/lower_set_rows_stateless.cpp" + "${FE_SRC_DIR}/pass/make_stateful.cpp" + "${FE_SRC_DIR}/tokenizer_metadata.cpp" + "${FE_SRC_DIR}/quant/gguf.cpp" "${FE_SRC_DIR}/quant/gguf_quants.cpp" "${FE_SRC_DIR}/quant/weights.cpp" + "${FE_SRC_DIR}/builder/gguf_builder.cpp" + "${FE_SRC_DIR}/builder/gguf_builder_decoder.cpp" + "${FE_SRC_DIR}/builder/graph_emitter.cpp" + "${FE_SRC_DIR}/builder/arch_registry.cpp" + "${FE_SRC_DIR}/builder/decoder_config.cpp" + "${FE_SRC_DIR}/builder/model_kind.cpp" + "${FE_SRC_DIR}/builder/arch/decoder_builder.cpp" + "${FE_SRC_DIR}/builder/blocks/attention.cpp" + "${FE_SRC_DIR}/builder/blocks/common.cpp" + "${FE_SRC_DIR}/builder/blocks/ffn.cpp" + "${FE_SRC_DIR}/builder/blocks/gated_delta_net.cpp" + "${FE_SRC_DIR}/builder/blocks/qkv_repack.cpp" "${FE_SRC_DIR}/op/add_id.cpp" "${FE_SRC_DIR}/op/argsort.cpp" "${FE_SRC_DIR}/op/clamp.cpp" @@ -52,6 +68,7 @@ set(FRONTEND_SRCS "${FE_SRC_DIR}/op/top_k.cpp" "${FE_SRC_DIR}/op/tri.cpp" "${FE_SRC_DIR}/op/sum_rows.cpp" + "${FE_SRC_DIR}/op/top_k.cpp" "${FE_SRC_DIR}/op/transpose.cpp" "${FE_SRC_DIR}/op/unary_elu.cpp" "${FE_SRC_DIR}/op/unary_gelu.cpp" @@ -61,8 +78,10 @@ set(FRONTEND_SRCS ) set(TEST_SRCS + "${CMAKE_CURRENT_SOURCE_DIR}/test_arch_conversion.cpp" "${CMAKE_CURRENT_SOURCE_DIR}/test_dequant_vs_ggml.cpp" "${CMAKE_CURRENT_SOURCE_DIR}/test_extensions.cpp" + "${CMAKE_CURRENT_SOURCE_DIR}/test_op_coverage.cpp" "${CMAKE_CURRENT_SOURCE_DIR}/test_ops.cpp" "${CMAKE_CURRENT_SOURCE_DIR}/test_weights.cpp" ) @@ -83,17 +102,31 @@ ov_add_test_target( INCLUDES "${CMAKE_CURRENT_SOURCE_DIR}/../src" "${CMAKE_CURRENT_SOURCE_DIR}/../include" + # frontend.cpp (compiled into the self-contained test binary) includes the native + # .gguf load path (get_path_from_any in openvino/frontend/common/path_util.hpp). + "${OpenVINO_SOURCE_DIR}/src/frontends/common/include" + "${OpenVINO_SOURCE_DIR}/src/frontends/common/dev_api" ADD_CLANG_FORMAT LABELS OV UNIT GGUF_FE ) -# Install the .npy reference data next to the test binary (found via getExecutableDirectory()). +# Install the reference data next to the test binary (found via getExecutableDirectory()): +# *.npy op/dequant references, plus the per-architecture fixture manifest. +# +# The manifest's companion *.gguf.hdr fixtures are deliberately NOT here: they are generated from +# llama.cpp by tests/gen_arch_fixtures.py into this same directory (see the "Generate GGUF arch +# fixtures" step in .github/workflows/job_build_linux.yml). The manifest itself IS installed -- +# it carries the reviewed per-architecture expectation for each fixture, which is source, not +# generated output. Where no fixtures were generated the arch suite skips itself; see +# test_arch_conversion.cpp. install(DIRECTORY "${CMAKE_CURRENT_SOURCE_DIR}/test_data" DESTINATION tests COMPONENT tests EXCLUDE_FROM_ALL - FILES_MATCHING PATTERN "*.npy") + FILES_MATCHING + PATTERN "*.npy" + PATTERN "manifest.txt") # Sources compiled into the binary: resolve the visibility macros to local definitions, not dllimport. target_compile_definitions(${TARGET_NAME} PRIVATE openvino_gguf_frontend_EXPORTS) diff --git a/src/frontends/gguf/tests/bench_gguf.py b/src/frontends/gguf/tests/bench_gguf.py new file mode 100644 index 00000000000000..98cdbbce08130f --- /dev/null +++ b/src/frontends/gguf/tests/bench_gguf.py @@ -0,0 +1,254 @@ +# Copyright (C) 2018-2026 Intel Corporation +# SPDX-License-Identifier: Apache-2.0 +# +# Benchmark a GGUF model on two backends: +# 1. OpenVINO GGUF frontend (frontend convert + compile on CPU) +# 2. llama.cpp (llama-simple binary, CPU only, no GPU layers) +# +# Metrics reported per backend: +# - load (convert / llama load) time [s] +# - compile time [s] (OV only) +# - prefill throughput [tok/s] +# - decode throughput [tok/s] +# +# Usage: +# PYTHONPATH=/bin/intel64/Release/python \ +# LD_LIBRARY_PATH=/bin/intel64/Release \ +# python3 bench_gguf.py --gguf model.gguf --tokenizer \ +# [--llama-simple ] [--prompt "..."] [--prefill-tokens 128] [--gen-tokens 64] +# +# llama-simple is expected to print on stderr: +# "prompt eval time = ... / N tokens (... ms per token, PP tok/s tokens per second)" +# "eval time = ... / N runs (... ms per token, TG tok/s tokens per second)" + +import argparse +import re +import subprocess +import sys +import time + +import numpy as np + + +# --------------------------------------------------------------------------- +# OpenVINO helpers +# --------------------------------------------------------------------------- + +def build_inputs(tokens, past_len): + n = len(tokens) + inp_tokens = np.array(tokens, dtype=np.int32).reshape(1, 1, 1, n) + inp_pos = np.arange(past_len, past_len + n, dtype=np.int32).reshape(1, 1, 1, n) + inp_out_ids = np.array([n - 1], dtype=np.int32).reshape(1, 1, 1, 1) + mask = np.zeros((1, 1, n, past_len + n), dtype=np.float32) + for i in range(n): + mask[0, 0, i, past_len + i + 1:] = -np.inf + token_len = np.array([n], dtype=np.int64) + beam_idx = np.zeros((1,), dtype=np.int32) + return { + "inp_tokens": inp_tokens, + "inp_pos": inp_pos, + "inp_out_ids": inp_out_ids, + "self_kq_mask": mask, + "self_kq_mask_swa": mask.copy(), + "token_len_per_seq": token_len, + "beam_idx": beam_idx, + } + + +def bench_openvino(gguf_path, prompt_ids, gen_tokens, device="CPU"): + import openvino as ov +from openvino.frontend import FrontEndManager + + +def convert_gguf(model_path: str): + """Convert a .gguf through the GGUF frontend. + + The frontend is not auto-selectable (see is_hidden_frontend in + src/frontends/common/src/manager.cpp), so core.read_model(".gguf") does not reach it and it + has to be requested by name. + """ + fe = FrontEndManager().load_by_framework("gguf") + return fe.convert(fe.load(model_path)) + + core = ov.Core() + + t0 = time.perf_counter() + model = convert_gguf(gguf_path) + t_read = time.perf_counter() - t0 + + t0 = time.perf_counter() + compiled = core.compile_model(model, device) + t_compile = time.perf_counter() - t0 + + req = compiled.create_infer_request() + model_inputs = {n for p in compiled.inputs for n in p.get_names()} + + def run(tokens, past): + raw = build_inputs(tokens, past) + feed = {} + for k, v in raw.items(): + if k in model_inputs: + feed[k] = ov.Tensor(v) + out = req.infer(feed) + return list(out.values())[0] + + # warm-up: 1-token prefill + 1 decode + run(prompt_ids[:1], 0) + run([0], 1) + # reset states + for state in req.query_state(): + state.reset() + + # prefill + t0 = time.perf_counter() + next_id = int(run(prompt_ids, 0).reshape(-1).argmax()) + t_prefill = time.perf_counter() - t0 + pp_tps = len(prompt_ids) / t_prefill + + # decode + past = len(prompt_ids) + t0 = time.perf_counter() + for _ in range(gen_tokens - 1): + next_id = int(run([next_id], past).reshape(-1).argmax()) + past += 1 + t_decode = time.perf_counter() - t0 + tg_tps = (gen_tokens - 1) / t_decode + + return { + "load_s": t_read, + "compile_s": t_compile, + "pp_tps": pp_tps, + "tg_tps": tg_tps, + "prefill_tokens": len(prompt_ids), + "gen_tokens": gen_tokens, + } + + +# --------------------------------------------------------------------------- +# llama.cpp helpers +# --------------------------------------------------------------------------- + +def bench_llamacpp(llama_simple, gguf_path, prompt_text, gen_tokens): + """Run llama-simple and parse its timing lines from stderr.""" + import os + env = dict(os.environ) + # ensure the llama.cpp libs are found + lib_dir = str(__import__("pathlib").Path(llama_simple).parent) + old = env.get("LD_LIBRARY_PATH", "") + env["LD_LIBRARY_PATH"] = f"{lib_dir}:{old}" if old else lib_dir + + cmd = [ + llama_simple, + "-m", gguf_path, + "-n", str(gen_tokens), + prompt_text, + ] + + result = subprocess.run(cmd, capture_output=True, text=True, env=env) + combined = result.stdout + result.stderr + + # llama_perf output lines: + # "llama_perf_context_print: load time = 167.53 ms" + # "llama_perf_context_print: prompt eval time = 37.42 ms / N tokens (... PP tok/s tokens per second)" + # "llama_perf_context_print: eval time = 251.15 ms / N runs (... TG tok/s tokens per second)" + load_ms = _parse_perf_line(combined, "load time") + pp_tps = _parse_tps(combined, "prompt eval time") + tg_tps = _parse_tps(combined, r" eval time") + + return { + "load_s": (load_ms / 1000) if load_ms else None, + "compile_s": None, + "pp_tps": pp_tps, + "tg_tps": tg_tps, + "gen_tokens": gen_tokens, + "returncode": result.returncode, + "stderr": combined, + } + + +def _parse_perf_line(text, label): + m = re.search(rf"{re.escape(label)}\s*=\s*([\d.]+)\s*ms", text) + return float(m.group(1)) if m else None + + +def _parse_tps(text, label_pattern): + m = re.search(rf"{label_pattern}.*?([\d.]+)\s+tokens per second", text) + return float(m.group(1)) if m else None + + +# --------------------------------------------------------------------------- +# Main +# --------------------------------------------------------------------------- + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("--gguf", required=True) + ap.add_argument("--tokenizer", required=True, help="HF tokenizer dir") + ap.add_argument("--prompt", default="The capital of France is Paris. Tell me about this city.") + ap.add_argument("--prefill-tokens", type=int, default=128, + help="Pad/truncate prompt to this many tokens") + ap.add_argument("--gen-tokens", type=int, default=64) + ap.add_argument("--llama-simple", + default="/home/vmaxim/llama.cpp/build-ref/bin/llama-simple") + ap.add_argument("--device", default="CPU") + ap.add_argument("--skip-ov", action="store_true") + ap.add_argument("--skip-llama", action="store_true") + args = ap.parse_args() + + from transformers import AutoTokenizer + tok = AutoTokenizer.from_pretrained(args.tokenizer) + prompt_ids = tok(args.prompt, return_tensors="np")["input_ids"][0].tolist() + + # pad / trim to prefill_tokens + if len(prompt_ids) < args.prefill_tokens: + # repeat prompt until we hit target + while len(prompt_ids) < args.prefill_tokens: + prompt_ids = prompt_ids + prompt_ids + prompt_ids = prompt_ids[:args.prefill_tokens] + + print(f"Model : {args.gguf}") + print(f"Prompt: {args.prefill_tokens} tokens (prefill), {args.gen_tokens} tokens (gen)") + print() + + ov_res = None + lc_res = None + + if not args.skip_ov: + print("=== OpenVINO GGUF frontend ===") + ov_res = bench_openvino(args.gguf, prompt_ids, args.gen_tokens, args.device) + print(f" convert : {ov_res['load_s']:.2f}s") + print(f" compile : {ov_res['compile_s']:.2f}s") + print(f" prefill : {ov_res['pp_tps']:.1f} tok/s ({ov_res['prefill_tokens']} tokens)") + print(f" decode : {ov_res['tg_tps']:.1f} tok/s ({ov_res['gen_tokens']} tokens)") + print() + + if not args.skip_llama: + print("=== llama.cpp (CPU) ===") + lc_res = bench_llamacpp( + args.llama_simple, + args.gguf, + args.prompt, + args.gen_tokens, + ) + if lc_res["load_s"]: + print(f" load : {lc_res['load_s']:.2f}s") + if lc_res["pp_tps"]: + print(f" prefill : {lc_res['pp_tps']:.1f} tok/s") + else: + print(f" prefill : N/A (1-token prompt in llama-simple)") + if lc_res["tg_tps"]: + print(f" decode : {lc_res['tg_tps']:.1f} tok/s ({lc_res['gen_tokens']} tokens)") + if lc_res["returncode"] != 0: + print(f" [WARN] llama-simple returned {lc_res['returncode']}") + print() + + if ov_res and lc_res and lc_res["tg_tps"]: + ratio = ov_res["tg_tps"] / lc_res["tg_tps"] + print(f"=== Summary ===") + print(f" OV decode : {ov_res['tg_tps']:.1f} tok/s") + print(f" llama decode: {lc_res['tg_tps']:.1f} tok/s") + print(f" OV / llama : {ratio:.2f}x") + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/src/frontends/gguf/tests/compare_with_llama.py b/src/frontends/gguf/tests/compare_with_llama.py new file mode 100644 index 00000000000000..474265a260f189 --- /dev/null +++ b/src/frontends/gguf/tests/compare_with_llama.py @@ -0,0 +1,121 @@ +# Copyright (C) 2018-2026 Intel Corporation +# SPDX-License-Identifier: Apache-2.0 +# +# Compare the OpenVINO GGUF frontend against native llama.cpp on a GGUF model. +# +# Greedy-decodes a prompt through the model produced by the GGUF frontend +# (the GGUF frontend) and prints the generated text / token ids, so it can be diffed +# against `llama-simple -m model.gguf ` (which also greedy-decodes). +# +# The frontend's model uses the gguf IO contract (inp_tokens / inp_pos / self_kq_mask / +# token_len_per_seq + a stateful KV cache). This script builds those inputs from a token +# sequence -- i.e. it is a standalone version of the genai IO adapter. The same logic moved +# into the graph (or a genai-side wrapper) lets the model run under genai's LLMPipeline. +# +# Usage: +# PYTHONPATH=/bin/intel64/Release/python \ +# LD_LIBRARY_PATH=/bin/intel64/Release \ +# python3 compare_with_llama.py --gguf model.gguf --tokenizer \ +# --prompt "The capital of France is" --n 16 + +import argparse +import sys + +import numpy as np +import openvino as ov +from openvino.frontend import FrontEndManager + + +def convert_gguf(model_path: str): + """Convert a .gguf through the GGUF frontend. + + The frontend is not auto-selectable (see is_hidden_frontend in + src/frontends/common/src/manager.cpp), so core.read_model(".gguf") does not reach it and it + has to be requested by name. + """ + fe = FrontEndManager().load_by_framework("gguf") + return fe.convert(fe.load(model_path)) + + +def build_inputs(tokens, past_len): + """Build the gguf-IO tensors for one decode step. + + tokens : list[int] of the new token ids for this step. + past_len : number of tokens already in the KV cache. + Returns a dict of input name -> ov.Tensor. + """ + n = len(tokens) + total = past_len + n + inp_tokens = np.array(tokens, dtype=np.int32).reshape(1, 1, 1, n) + inp_pos = np.arange(past_len, past_len + n, dtype=np.int32).reshape(1, 1, 1, n) + # last-token logits only (matches llama-simple's per-step argmax on the final token) + inp_out_ids = np.array([n - 1], dtype=np.int32).reshape(1, 1, 1, 1) + # causal mask [1, 1, n, total]: 0 where attended, -inf where masked. + mask = np.zeros((1, 1, n, total), dtype=np.float32) + for i in range(n): + # query token i (absolute position past_len + i) may attend to keys 0..past_len+i + allowed = past_len + i + 1 + mask[0, 0, i, allowed:] = -np.inf + token_len = np.array([n], dtype=np.int64) + # beam_idx: identity beam reorder for the (single-beam, batch-1) stateful KV cache. + beam_idx = np.zeros((1,), dtype=np.int32) + return { + "inp_tokens": ov.Tensor(inp_tokens), + "inp_pos": ov.Tensor(inp_pos), + "inp_out_ids": ov.Tensor(inp_out_ids), + "self_kq_mask": ov.Tensor(mask), + # gpt-oss sliding-window mask: for prompts shorter than the window it equals the + # full causal mask, so the same tensor is correct here. + "self_kq_mask_swa": ov.Tensor(mask.copy()), + "token_len_per_seq": ov.Tensor(token_len), + "beam_idx": ov.Tensor(beam_idx), + } + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("--gguf", required=True) + ap.add_argument("--tokenizer", required=True, help="HF tokenizer dir/json for the model") + ap.add_argument("--prompt", default="The capital of France is") + ap.add_argument("--n", type=int, default=16, help="tokens to generate") + ap.add_argument("--device", default="CPU") + args = ap.parse_args() + + from transformers import AutoTokenizer + + tok = AutoTokenizer.from_pretrained(args.tokenizer) + prompt_ids = tok(args.prompt, return_tensors="np")["input_ids"][0].tolist() + print(f"prompt: {args.prompt!r}") + print(f"prompt token ids: {prompt_ids}") + + core = ov.Core() + model = convert_gguf(args.gguf) + compiled = core.compile_model(model, args.device) + req = compiled.create_infer_request() + # only feed the inputs the (pruned) model actually exposes + model_inputs = {n for p in compiled.inputs for n in p.get_names()} + + def run(tokens, past): + feed = {k: v for k, v in build_inputs(tokens, past).items() if k in model_inputs} + out = req.infer(feed) + return list(out.values())[0] + + # ---- prefill ---- + next_id = int(run(prompt_ids, 0).reshape(-1).argmax()) + generated = [next_id] + past = len(prompt_ids) + + # ---- decode ---- + for _ in range(args.n - 1): + next_id = int(run([next_id], past).reshape(-1).argmax()) + generated.append(next_id) + past += 1 + + text = tok.decode(generated) + print(f"\ngenerated token ids: {generated}") + print(f"generated text: {text!r}") + print(f"\nfull: {args.prompt}{text}") + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/src/frontends/gguf/tests/gen_arch_fixtures.py b/src/frontends/gguf/tests/gen_arch_fixtures.py new file mode 100644 index 00000000000000..0150810b759f9d --- /dev/null +++ b/src/frontends/gguf/tests/gen_arch_fixtures.py @@ -0,0 +1,241 @@ +# Copyright (C) 2018-2026 Intel Corporation +# SPDX-License-Identifier: Apache-2.0 + +"""Produce the per-architecture GGUF conversion fixtures consumed by test_arch_conversion.cpp. + +A fixture is a GGUF *header*: everything before min(tensor.data_offset), i.e. the magic, the KV +metadata and the tensor table. That is the entire input to architecture detection and graph +construction -- the weight bytes that follow contribute nothing to the shape of the converted model, +so the C++ test rebuilds a loadable .gguf by appending the recorded number of zero bytes. Headers +are ~25 KB each; the full models they come from are 559 MB. + +The upstream source of the models is llama.cpp's `test-llama-archs`, which walks every architecture +it knows and saves a tiny random model for each. This script wraps it end to end: + + # from a checkout you already have (nothing is cloned): + python3 gen_arch_fixtures.py --llama-src + + # or let it fetch the pinned commit itself (this is what CI does): + python3 gen_arch_fixtures.py --fetch --out-dir + +llama.cpp is pinned to LLAMA_CPP_COMMIT below and stays there until a fixture refresh is +deliberately made. The pin is load-bearing, not hygiene: `test-llama-archs` writes whatever KVs +llama.cpp currently defines, so upstream adding one shifts the bytes of *every* fixture at once +(measured: between 86a9c79f8 and a head 13 days later, all 101 headers changed because three +`attention.indexer.*` KVs were added, and two new architectures appeared). Tracking a moving +upstream would therefore turn an unrelated llama.cpp commit into a red OpenVINO precommit. Bump the +pin as its own reviewed change, and say so in the commit message so the provenance stays traceable. + +The headers are seed-independent -- verified byte-identical for `-s 1` and `-s 999` -- because the +seed only feeds weight values. Pinning one anyway (below) costs nothing and keeps the run +deterministic. + +Requires the `gguf` Python package (llama.cpp's gguf-py) to read back the generated files. +""" + +import argparse +import glob +import os +import shutil +import subprocess +import sys +import tempfile + +# Pinned llama.cpp revision. MUST be a commit on ggml-org/llama.cpp master: --fetch clones from +# there, so a SHA that only exists in a fork makes CI fail with "couldn't find remote ref". +LLAMA_CPP_COMMIT = "86a9c79f866799eb0e7e89c03578ccfbcc5d808e" +LLAMA_CPP_REPO = "https://github.com/ggml-org/llama.cpp.git" +# The generator's own default seed is std::random_device, i.e. non-reproducible. Pin it: it does not +# affect the header bytes, but it keeps the whole run deterministic. +SEED = 1 + +HERE = os.path.dirname(os.path.abspath(__file__)) +FIXTURE_DIR = os.path.join(HERE, "test_data", "arch_fixtures") + +MANIFEST_HEADER = """\ +# GGUF per-architecture conversion fixtures -- see gen_arch_fixtures.py (generator) and +# test_arch_conversion.cpp (consumer). Generated from llama.cpp {commit} at seed {seed}. +# +# One line per fixture: +# +# +# header-only fixture in this directory; the loadable .gguf is followed by +# zero bytes (all fixture tensors are F32, so no dequant is involved). +# what conversion must do, and the test asserts exactly this: +# convert converts cleanly. The graph fingerprint is pinned in test_arch_conversion.cpp. +# reject the architecture is not on the builder's accept list, so conversion must fail with +# that specific diagnostic -- not a crash, and not a silent wrong-graph success. +# broken a supported architecture that currently fails to convert: a real defect, recorded so +# it cannot be forgotten. The test asserts it STILL FAILS; fixing the defect turns this +# line into a failure telling you to promote it to `convert`. +""" + + +def run(cmd, **kwargs): + """Run a command, echoing it first so a CI log shows exactly what happened.""" + print("+ " + " ".join(cmd), flush=True) + subprocess.run(cmd, check=True, **kwargs) + + +def fetch_llama_cpp(dest): + """Shallow-fetch the pinned commit into dest. ~35 MB, a few seconds. + + A bare `clone --depth 1` cannot take a SHA, and a full clone of llama.cpp is large and slow, so + fetch the one commit explicitly: init + fetch --depth 1 . This requires the SHA to be + reachable on the remote, which is why LLAMA_CPP_COMMIT must be an upstream commit. + """ + os.makedirs(dest, exist_ok=True) + run(["git", "init", "-q", dest]) + run(["git", "-C", dest, "remote", "add", "origin", LLAMA_CPP_REPO]) + run(["git", "-C", dest, "fetch", "-q", "--depth", "1", "origin", LLAMA_CPP_COMMIT]) + run(["git", "-C", dest, "checkout", "-q", "FETCH_HEAD"]) + return dest + + +def build_generator(src, build_dir, jobs): + """Configure and build only the test-llama-archs target. + + LLAMA_BUILD_TESTS=ON is required -- the target is a test, and upstream's own release builds turn + tests off, which is why no prebuilt llama.cpp package ships it. Everything else is off: the + examples, tools and server are not needed to save a model. + """ + run([ + "cmake", "-S", src, "-B", build_dir, + "-DCMAKE_BUILD_TYPE=Release", + "-DLLAMA_BUILD_TESTS=ON", + "-DLLAMA_BUILD_EXAMPLES=OFF", + "-DLLAMA_BUILD_TOOLS=OFF", + "-DLLAMA_BUILD_SERVER=OFF", + ]) + cmd = ["cmake", "--build", build_dir, "--target", "test-llama-archs"] + if jobs: + cmd += ["-j", str(jobs)] + run(cmd) + exe = os.path.join(build_dir, "bin", "test-llama-archs") + if not os.path.isfile(exe): + sys.exit(f"error: build succeeded but {exe} is missing") + return exe + + +def run_generator(exe, out_dir): + run([exe, "-s", str(SEED), "-o", out_dir], stdout=subprocess.DEVNULL) + files = sorted(glob.glob(os.path.join(out_dir, "*.gguf"))) + if not files: + sys.exit("error: generator produced no .gguf files") + return files + + +def strip_to_header(path, out_path): + """Write everything before the first tensor's data offset; return (header_len, data_len, arch).""" + import gguf + + reader = gguf.GGUFReader(path) + if not reader.tensors: + return None + header_len = min(t.data_offset for t in reader.tensors) + total = os.path.getsize(path) + with open(path, "rb") as src, open(out_path, "wb") as dst: + dst.write(src.read(header_len)) + arch_field = reader.fields["general.architecture"] + arch = str(bytes(arch_field.parts[-1]), "utf-8") + return header_len, total - header_len, arch + + +def read_expectations(manifest): + """Existing -> mapping, or {} if there is no manifest yet.""" + previous = {} + if not os.path.exists(manifest): + return previous + with open(manifest) as f: + for line in f: + line = line.strip() + if not line or line.startswith("#"): + continue + parts = line.split() + if len(parts) >= 3: + previous[parts[0]] = parts[2] + return previous + + +def main(): + ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) + src = ap.add_mutually_exclusive_group(required=True) + src.add_argument("--llama-src", help="existing llama.cpp checkout to build from") + src.add_argument("--fetch", action="store_true", + help=f"shallow-fetch the pinned commit ({LLAMA_CPP_COMMIT[:12]}) into a temp dir") + ap.add_argument("--out-dir", default=FIXTURE_DIR, + help="where to write the .gguf.hdr fixtures and manifest.txt (default: the " + "in-tree test_data/arch_fixtures)") + ap.add_argument("--llama-build", help="reuse this build directory instead of a temp one") + ap.add_argument("-j", "--jobs", type=int, default=os.cpu_count(), help="parallel build jobs") + ap.add_argument("--keep", action="store_true", help="keep the full generated .gguf files") + args = ap.parse_args() + + work = tempfile.mkdtemp(prefix="gguf_arch_fixtures_") + try: + src_dir = args.llama_src + if args.fetch: + src_dir = fetch_llama_cpp(os.path.join(work, "llama.cpp")) + else: + # An arbitrary checkout may sit at any revision, and the fixture bytes depend on it. + # Report what we actually built so a mismatch with the pin is visible in the log. + rev = subprocess.run(["git", "-C", src_dir, "rev-parse", "HEAD"], + capture_output=True, text=True).stdout.strip() + if rev and rev != LLAMA_CPP_COMMIT: + print(f"warning: {src_dir} is at {rev[:12]}, not the pinned {LLAMA_CPP_COMMIT[:12]}; " + f"the fixtures will differ from the committed ones", flush=True) + + exe = build_generator(src_dir, args.llama_build or os.path.join(work, "build"), args.jobs) + + models = os.path.join(work, "models") + os.makedirs(models, exist_ok=True) + files = run_generator(exe, models) + print(f"generated {len(files)} models in {models}") + + out_dir = os.path.abspath(args.out_dir) + os.makedirs(out_dir, exist_ok=True) + manifest = os.path.join(out_dir, "manifest.txt") + # Preserve reviewed expectations before the stale fixtures are removed. + previous = read_expectations(manifest) + for stale in glob.glob(os.path.join(out_dir, "*.gguf.hdr")): + os.remove(stale) + + entries = [] + for path in files: + name = os.path.basename(path) + ".hdr" + result = strip_to_header(path, os.path.join(out_dir, name)) + if result is None: + print(f" skip {os.path.basename(path)}: no tensors") + continue + header_len, data_len, arch = result + entries.append((name, data_len, arch)) + print(f" {name}: header {header_len} B, data {data_len} B, arch {arch}") + + # Expectations are NOT inferred from the frontend here -- that would make the fixtures assert + # whatever the frontend currently does, which is not a test. Preserve the reviewed + # expectation from the existing manifest and default new fixtures to `reject`, which is + # correct for any architecture the builder's accept list does not name. A new architecture + # that should convert is then a deliberate, reviewable one-word manifest edit. + with open(manifest, "w") as f: + f.write(MANIFEST_HEADER.format(commit=LLAMA_CPP_COMMIT, seed=SEED)) + for name, data_len, _arch in sorted(entries): + expectation = previous.get(name, "reject") + if name not in previous: + print(f" NEW fixture {name}: defaulted to `reject`; review it") + f.write(f"{name} {data_len} {expectation}\n") + + total = sum(os.path.getsize(os.path.join(out_dir, n)) for n, _, _ in entries) + print(f"\nwrote {len(entries)} headers ({total / 1024:.1f} KB raw) + manifest to {out_dir}") + + dropped = set(previous) - {n for n, _, _ in entries} + if dropped: + print(f"note: {len(dropped)} fixture(s) disappeared upstream and were dropped: {sorted(dropped)}") + finally: + if args.keep: + print(f"work directory kept: {work}") + else: + shutil.rmtree(work, ignore_errors=True) + + +if __name__ == "__main__": + main() diff --git a/src/frontends/gguf/tests/gen_ggml_reference.c b/src/frontends/gguf/tests/gen_ggml_reference.c new file mode 100644 index 00000000000000..1765bfd6462c41 --- /dev/null +++ b/src/frontends/gguf/tests/gen_ggml_reference.c @@ -0,0 +1,170 @@ +// Copyright (C) 2018-2026 Intel Corporation +// SPDX-License-Identifier: Apache-2.0 +// +// Generates per-op reference data for the GGUF frontend tests by running the ops +// through *real ggml* (the same code llama.cpp executes), NOT a numpy reimplementation +// of the formula. This is the authoritative oracle: it catches translator/formula +// mismatches (e.g. GELU erf-vs-tanh) that a hand-rolled numpy reference would encode +// and therefore hide. +// +// For each op it writes _input*.bin and _expected.bin as raw little-endian +// f32 (shape recorded in a sidecar .shape text file). A small python step +// (gen_ggml_reference.py) converts these to the .npy files the C++ tests load, so the +// committed test vectors are byte-for-byte ggml outputs. +// +// Build & run (see gen_ggml_reference.py which wraps this): +// cc gen_ggml_reference.c -I/ggml/include -L/build-ref/bin -lggml -lggml-base -lggml-cpu -o gen_ggml_reference +// LD_LIBRARY_PATH=/build-ref/bin ./gen_ggml_reference + +#include +#include +#include +#include + +#include "ggml.h" +#include "ggml-cpu.h" + +static void write_shape(const char* dir, const char* name, const int64_t* dims, int ndim) { + char path[1024]; + snprintf(path, sizeof(path), "%s/%s.shape", dir, name); + FILE* f = fopen(path, "w"); + for (int i = 0; i < ndim; ++i) fprintf(f, "%lld%s", (long long)dims[i], i + 1 < ndim ? " " : ""); + fprintf(f, "\n"); + fclose(f); +} + +static void write_bin(const char* dir, const char* name, const float* data, int64_t n, + const int64_t* dims, int ndim) { + char path[1024]; + snprintf(path, sizeof(path), "%s/%s.bin", dir, name); + FILE* f = fopen(path, "wb"); + if (!f) { fprintf(stderr, "cannot open %s\n", path); exit(1); } + fwrite(data, sizeof(float), (size_t)n, f); + fclose(f); + write_shape(dir, name, dims, ndim); +} + +// Raw byte dump (for quantized blocks). Shape file records the byte count. +static void write_raw(const char* dir, const char* name, const void* data, int64_t nbytes) { + char path[1024]; + snprintf(path, sizeof(path), "%s/%s.bin", dir, name); + FILE* f = fopen(path, "wb"); + if (!f) { fprintf(stderr, "cannot open %s\n", path); exit(1); } + fwrite(data, 1, (size_t)nbytes, f); + fclose(f); + int64_t d[1] = { nbytes }; + write_shape(dir, name, d, 1); +} + +// Build a 1-input unary graph, run it on the CPU backend, dump input + output. +typedef struct ggml_tensor* (*unary_fn)(struct ggml_context*, struct ggml_tensor*); + +static void run_unary(const char* dir, const char* name, unary_fn fn, + const float* in, const int64_t* dims, int ndim) { + int64_t n = 1; + for (int i = 0; i < ndim; ++i) n *= dims[i]; + + struct ggml_init_params params = { 64 * 1024 * 1024, NULL, false }; + struct ggml_context* ctx = ggml_init(params); + + int64_t ne[4] = {1, 1, 1, 1}; + for (int i = 0; i < ndim; ++i) ne[i] = dims[ndim - 1 - i]; // ggml is reversed vs row-major + struct ggml_tensor* x = ggml_new_tensor(ctx, GGML_TYPE_F32, ndim, ne); + memcpy(x->data, in, sizeof(float) * (size_t)n); + + struct ggml_tensor* y = fn(ctx, x); + struct ggml_cgraph* gf = ggml_new_graph(ctx); + ggml_build_forward_expand(gf, y); + ggml_graph_compute_with_ctx(ctx, gf, 1); + + char inn[256]; + snprintf(inn, sizeof(inn), "%s_input", name); + char outn[256]; + snprintf(outn, sizeof(outn), "%s_expected", name); + write_bin(dir, inn, in, n, dims, ndim); + write_bin(dir, outn, (const float*)y->data, n, dims, ndim); + printf(" %-16s in/out %lld elems\n", name, (long long)n); + + ggml_free(ctx); +} + +// Quantize synthetic float data with ggml, then dump (a) the raw quantized block bytes and +// (b) ggml's own dequantization of those exact bytes. The C++ test feeds the SAME bytes through +// the frontend's dequant subgraph and compares against ggml's to_float -- so ggml is the oracle +// for both the quantization AND the dequantization, exactly like llama.cpp test-quantize-fns. +// +// Emits: _qbytes.bin (u8 raw block) + _qbytes.shape ("nbytes") +// _deq.bin (f32 ggml dequant) + _deq.shape ("rows cols") +static void run_dequant(const char* dir, const char* name, enum ggml_type type, + int64_t rows, int64_t cols) { + const int64_t n = rows * cols; + const struct ggml_type_traits* tt = ggml_get_type_traits(type); + if (!tt || !tt->to_float) { fprintf(stderr, "no traits for %s\n", name); return; } + + float* in = malloc(sizeof(float) * n); + for (int64_t i = 0; i < n; ++i) { + // Smooth, asymmetric data so per-block scale AND min are non-trivial (fractional) -- + // the case that exposes an integer-zero-point dequant bug (Q4_K/Q5_K). + in[i] = 0.37f + 1.7f * sinf(0.013f * (float)i) + 0.4f * cosf(0.0007f * (float)i * (float)i); + } + + const size_t row_bytes = ggml_row_size(type, cols); + uint8_t* q = malloc(row_bytes * rows); + // quantize row by row (ggml_quantize_chunk works on whole rows) + ggml_quantize_chunk(type, in, q, 0, rows, cols, NULL); + + float* deq = malloc(sizeof(float) * n); + for (int64_t r = 0; r < rows; ++r) { + tt->to_float(q + r * row_bytes, deq + r * cols, cols); + } + + char nm[256]; + snprintf(nm, sizeof(nm), "%s_qbytes", name); + write_raw(dir, nm, q, (int64_t)(row_bytes * rows)); + + int64_t ddims[2] = { rows, cols }; + snprintf(nm, sizeof(nm), "%s_deq", name); + write_bin(dir, nm, deq, n, ddims, 2); + printf(" %-12s rows=%lld cols=%lld qbytes=%lld\n", name, (long long)rows, (long long)cols, + (long long)(row_bytes * rows)); + + free(in); free(q); free(deq); +} + +int main(int argc, char** argv) { + const char* out_dir = argc > 1 ? argv[1] : "."; + + // Deterministic input spanning the interesting activation range. + const int64_t dims[2] = {4, 32}; // 128 values + int64_t n = dims[0] * dims[1]; + float* in = malloc(sizeof(float) * n); + for (int64_t i = 0; i < n; ++i) { + // range roughly [-6, 6] + in[i] = -6.0f + 12.0f * (float)i / (float)(n - 1); + } + + printf("generating ggml reference data into %s\n", out_dir); + run_unary(out_dir, "gelu_ggml", ggml_gelu, in, dims, 2); + run_unary(out_dir, "gelu_erf_ggml", ggml_gelu_erf, in, dims, 2); + run_unary(out_dir, "gelu_quick_ggml", ggml_gelu_quick, in, dims, 2); + run_unary(out_dir, "silu_ggml", ggml_silu, in, dims, 2); + free(in); + + // Dequant references for every quant type the GGUF frontend supports. cols=256 satisfies + // every block/super-block size (32 and 256); 4 rows exercises multiple blocks. + printf("generating ggml dequant reference data\n"); + const int64_t R = 4, C = 256; + run_dequant(out_dir, "q4_0", GGML_TYPE_Q4_0, R, C); + run_dequant(out_dir, "q4_1", GGML_TYPE_Q4_1, R, C); + run_dequant(out_dir, "q5_0", GGML_TYPE_Q5_0, R, C); + run_dequant(out_dir, "q5_1", GGML_TYPE_Q5_1, R, C); + run_dequant(out_dir, "q8_0", GGML_TYPE_Q8_0, R, C); + run_dequant(out_dir, "q2_k", GGML_TYPE_Q2_K, R, C); + run_dequant(out_dir, "q3_k", GGML_TYPE_Q3_K, R, C); + run_dequant(out_dir, "q4_k", GGML_TYPE_Q4_K, R, C); + run_dequant(out_dir, "q5_k", GGML_TYPE_Q5_K, R, C); + run_dequant(out_dir, "q6_k", GGML_TYPE_Q6_K, R, C); + run_dequant(out_dir, "q2_0", GGML_TYPE_Q2_0, R, C); + + return 0; +} diff --git a/src/frontends/gguf/tests/gen_ggml_reference.py b/src/frontends/gguf/tests/gen_ggml_reference.py new file mode 100644 index 00000000000000..46e57dddb6098d --- /dev/null +++ b/src/frontends/gguf/tests/gen_ggml_reference.py @@ -0,0 +1,83 @@ +# Copyright (C) 2018-2026 Intel Corporation +# SPDX-License-Identifier: Apache-2.0 +# +# Builds and runs gen_ggml_reference.c (which links real ggml from llama.cpp) and converts +# its raw .bin dumps into the .npy files the C++ op tests load. Unlike a numpy reimplementation +# of each op's formula -- which can only confirm whatever formula the author guessed -- this +# produces *authoritative ggml outputs*, the exact values llama.cpp computes, so a translator +# that picks the wrong formula variant is caught. +# +# IMPORTANT (learned the hard way): ggml computes GELU/GELU_QUICK via an f16 lookup table +# (GGML_GELU_FP16 in ggml-cpu/vec.h): the input is rounded to f16, indexed into a 65536-entry +# f16 table, output in f16. That quantization is ~2e-3 -- LARGER than the erf-vs-tanh formula +# difference (~4e-4). So: +# * the reference values here carry ggml's f16-table quantization; +# * the matching C++ test tolerance must be ~3e-3, not 1e-5; +# * at that tolerance an erf/tanh swap in the translator is NOT distinguishable for GELU +# alone -- the real defense against formula bugs is the per-layer end-to-end diff vs +# llama-eval-callback on a deep model (see SKILL.md "Finding the bug"), where the +# per-call error compounds. Use these op-level vectors to catch GROSS errors (wrong op, +# wrong axis, wrong constant), and the end-to-end diff to catch subtle formula drift. +# +# Usage: +# python3 gen_ggml_reference.py --llama /home/vmaxim/llama.cpp --out-dir test_data + +import argparse +import os +import subprocess +import sys + +import numpy as np + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("--llama", default="/home/vmaxim/llama.cpp", + help="llama.cpp checkout with ggml headers and a built build-ref/bin") + ap.add_argument("--out-dir", default="test_data") + ap.add_argument("--build-dir", default="build-ref/bin", + help="path under --llama holding libggml*.so") + args = ap.parse_args() + + here = os.path.dirname(os.path.abspath(__file__)) + src = os.path.join(here, "gen_ggml_reference.c") + inc = os.path.join(args.llama, "ggml", "include") + libdir = os.path.join(args.llama, args.build_dir) + exe = "/tmp/gen_ggml_reference" + raw = "/tmp/ggml_ref_raw" + os.makedirs(raw, exist_ok=True) + os.makedirs(args.out_dir, exist_ok=True) + + cc = subprocess.run( + ["cc", src, "-I", inc, "-L", libdir, "-lggml", "-lggml-base", "-lggml-cpu", "-lm", + "-o", exe], + capture_output=True, text=True) + if cc.returncode != 0: + print(cc.stderr, file=sys.stderr) + sys.exit("compile failed") + + env = dict(os.environ, LD_LIBRARY_PATH=libdir) + run = subprocess.run([exe, raw], env=env, capture_output=True, text=True) + print(run.stdout) + if run.returncode != 0: + print(run.stderr, file=sys.stderr) + sys.exit("run failed") + + # Convert every .bin + .shape pair to .npy. + for fn in sorted(os.listdir(raw)): + if not fn.endswith(".bin"): + continue + base = fn[:-4] + with open(os.path.join(raw, base + ".shape")) as f: + shape = tuple(int(x) for x in f.read().split()) + # *_qbytes are raw quantized blocks (u8, 1-D byte count); everything else is f32. + if base.endswith("_qbytes"): + arr = np.fromfile(os.path.join(raw, fn), dtype=np.uint8).reshape(shape) + else: + arr = np.fromfile(os.path.join(raw, fn), dtype=np.float32).reshape(shape) + np.save(os.path.join(args.out_dir, base + ".npy"), arr) + print(f" wrote {base}.npy {arr.shape} {arr.dtype}") + + +if __name__ == "__main__": + main() diff --git a/src/frontends/gguf/tests/gguf_tokenize.py b/src/frontends/gguf/tests/gguf_tokenize.py new file mode 100644 index 00000000000000..7faab60b175b57 --- /dev/null +++ b/src/frontends/gguf/tests/gguf_tokenize.py @@ -0,0 +1,150 @@ +# Copyright (C) 2018-2026 Intel Corporation +# SPDX-License-Identifier: Apache-2.0 +# +# Tokenizer helper: reads tokenizer from a GGUF file via openvino_genai.Tokenizer. +# Falls back to a user-supplied HF tokenizer dir when genai can't parse the GGUF. +# +# Prints JSON {ids: [...], eos_id: int} or {text: str, eos_id: int} to stdout. +# +# Usage: +# python3 gguf_tokenize.py encode [--hf-tokenizer ] +# python3 gguf_tokenize.py decode ... [--hf-tokenizer ] + +import json +import sys + + +def _genai_tokenizer(gguf_path): + import openvino_genai as genai + return genai.Tokenizer(gguf_path) + + +def _encode_with_special_tokens(tok, text): + """ + Encode `text` respecting special tokens that tok.encode() would fragment. + + tok.encode() uses the BPE/SentencePiece subword model and treats special + tokens (e.g. <|hy_begin▁of▁sentence|>) as ordinary character sequences, + splitting them into subword pieces. We detect the special tokens by + scanning the vocabulary for entries that contain the fullwidth vertical bar + '｜' (U+FF5C) or start with '<|', then split the text on those tokens and + encode the non-special pieces normally. + """ + import re + + vocab = tok.get_vocab() # returns dict[bytes, int] (keys are raw bytes) + # Build special-token map: decoded_string -> token_id + special = {} + for raw_key, tid in vocab.items(): + if isinstance(raw_key, bytes): + try: + s = raw_key.decode("utf-8") + except UnicodeDecodeError: + continue + else: + s = raw_key + # Heuristic: contains fullwidth bar (Hunyuan) or looks like <|...|> + if "｜" in s or (s.startswith("<|") and s.endswith("|>")): + special[s] = tid + + if not special: + # No special tokens found — fall back to plain encode + return tok.encode(text).input_ids.data.flatten().tolist() + + # Sort longest first so longer tokens shadow shorter prefixes + pattern = re.compile( + "(" + "|".join(re.escape(k) for k in sorted(special, key=len, reverse=True)) + ")" + ) + parts = pattern.split(text) + + ids = [] + for part in parts: + if not part: + continue + if part in special: + ids.append(special[part]) + else: + ids.extend(tok.encode(part).input_ids.data.flatten().tolist()) + return ids + + +def main(): + args = sys.argv[1:] + + # Parse --hf-tokenizer from the tail + hf_tokenizer = None + clean = [] + i = 0 + while i < len(args): + if args[i] == "--hf-tokenizer" and i + 1 < len(args): + hf_tokenizer = args[i + 1] + i += 2 + else: + clean.append(args[i]) + i += 1 + args = clean + + if len(args) < 2: + print("Usage: gguf_tokenize.py encode [--hf-tokenizer ]", file=sys.stderr) + print(" gguf_tokenize.py decode ... [--hf-tokenizer ]", file=sys.stderr) + sys.exit(1) + + # Parse --chat flag + apply_chat = "--chat" in args + args = [a for a in args if a != "--chat"] + + mode = args[0] + gguf_path = args[1] + + if mode == "encode": + text = args[2] if len(args) > 2 else "" + try: + tok = _genai_tokenizer(gguf_path) + eos_id = tok.get_eos_token_id() + if apply_chat: + text = tok.apply_chat_template( + [{"role": "user", "content": text}], + add_generation_prompt=True, + ) + ids = _encode_with_special_tokens(tok, text) + print(json.dumps({"ids": ids, "eos_id": eos_id})) + return + except Exception as e: + if not hf_tokenizer: + raise RuntimeError(f"genai tokenizer failed and no --hf-tokenizer given: {e}") from e + from transformers import AutoTokenizer + hf = AutoTokenizer.from_pretrained(hf_tokenizer, trust_remote_code=True) + eos_id = hf.eos_token_id if hf.eos_token_id is not None else -1 + if apply_chat and hasattr(hf, "apply_chat_template") and hf.chat_template: + text = hf.apply_chat_template( + [{"role": "user", "content": text}], + add_generation_prompt=True, + tokenize=False, + ) + ids = hf(text, return_tensors="np")["input_ids"][0].tolist() + print(json.dumps({"ids": ids, "eos_id": eos_id})) + + elif mode == "decode": + ids = [int(x) for x in args[2:]] + try: + tok = _genai_tokenizer(gguf_path) + eos_id = tok.get_eos_token_id() + text = tok.decode(ids) + print(json.dumps({"text": text, "eos_id": eos_id})) + return + except Exception as e: + if not hf_tokenizer: + raise RuntimeError(f"genai tokenizer failed and no --hf-tokenizer given: {e}") from e + from transformers import AutoTokenizer + hf = AutoTokenizer.from_pretrained(hf_tokenizer, trust_remote_code=True) + eos_id = hf.eos_token_id if hf.eos_token_id is not None else -1 + text = hf.decode(ids) + print(json.dumps({"text": text, "eos_id": eos_id})) + + else: + print(f"Unknown mode: {mode!r}. Use 'encode' or 'decode'.", file=sys.stderr) + sys.exit(1) + + +if __name__ == "__main__": + main() diff --git a/src/frontends/gguf/tests/graph_fingerprint.py b/src/frontends/gguf/tests/graph_fingerprint.py new file mode 100644 index 00000000000000..0811b665fa99a7 --- /dev/null +++ b/src/frontends/gguf/tests/graph_fingerprint.py @@ -0,0 +1,61 @@ +# Copyright (C) 2018-2026 Intel Corporation +# SPDX-License-Identifier: Apache-2.0 +# +# Graph-fingerprint regression tool for the GGUF frontend's native builder. +# +# The fingerprint is a sha256 over the sorted (op_type, output_partial_shape) pairs of every op in +# the converted ov::Model, plus the sorted model input names. It is a cheap, exact regression gate: +# any builder change that alters the produced graph for a given model changes its fingerprint, and +# any change that is meant to be graph-neutral (refactor, robustness, renaming) must leave every +# fingerprint unchanged. +# +# Usage (baseline / manual): +# PYTHONPATH=/bin/intel64/Release/python LD_LIBRARY_PATH=/bin/intel64/Release \ +# python3 graph_fingerprint.py model_a.gguf model_b.gguf ... +# +# It prints " " per model. Save the output as the baseline; after a change, re-run +# and diff. The pytest wrapper (test_graph_fingerprint.py) automates this against a JSON baseline +# and is gated on GGUF_FINGERPRINT_MODELS being set (so CI skips when no local models are present). + +import hashlib +import sys + +import openvino as ov +from openvino.frontend import FrontEndManager + + +def convert_gguf(model_path: str): + """Convert a .gguf through the GGUF frontend. + + The frontend is not auto-selectable (see is_hidden_frontend in + src/frontends/common/src/manager.cpp), so core.read_model(".gguf") does not reach it and it + has to be requested by name. + """ + fe = FrontEndManager().load_by_framework("gguf") + return fe.convert(fe.load(model_path)) + + +def fingerprint(model_path: str) -> dict: + m = convert_gguf(model_path) + sig = [] + for op in m.get_ops(): + shape = str(op.get_output_partial_shape(0)) if op.get_output_size() > 0 else "" + sig.append(op.get_type_name() + "|" + shape) + inputs = sorted(list(i.get_names())[0] if i.get_names() else "?" for i in m.inputs) + outputs = sorted(list(o.get_names())[0] if o.get_names() else "?" for o in m.outputs) + h = hashlib.sha256("\n".join(sorted(sig)).encode()).hexdigest()[:16] + return {"sighash": h, "num_ops": len(m.get_ops()), "inputs": inputs, "outputs": outputs} + + +def main(argv): + if len(argv) < 2: + print("usage: graph_fingerprint.py [ ...]", file=sys.stderr) + return 2 + for path in argv[1:]: + fp = fingerprint(path) + print(f"{path}\t{fp['sighash']}\tops={fp['num_ops']}\tinputs={fp['inputs']}") + return 0 + + +if __name__ == "__main__": + sys.exit(main(sys.argv)) diff --git a/src/frontends/gguf/tests/op_test_utils.hpp b/src/frontends/gguf/tests/op_test_utils.hpp index 48c669aa478cb7..9b1020eac289a8 100644 --- a/src/frontends/gguf/tests/op_test_utils.hpp +++ b/src/frontends/gguf/tests/op_test_utils.hpp @@ -16,18 +16,18 @@ #pragma once -#include -#include - #include #include #include #include #include +#include #include #include +#include "cnpy.h" #include "common_test_utils/file_utils.hpp" +#include "gtest/gtest.h" #include "op_table.hpp" #include "openvino/core/model.hpp" #include "openvino/core/partial_shape.hpp" @@ -50,6 +50,16 @@ namespace ov_gguf_test { using namespace ov::frontend::gguf; +// Set of ggml op types that some test in this binary has actually converted. Every +// SingleOpDecoder construction records its op type here, so the record is a by-product of the tests +// running rather than a hand-maintained list that can drift. Checked against op_table.cpp by the +// coverage gate in test_op_coverage.cpp, which therefore fails when a new op is registered without +// a test. Populated at run time, so the gate has to run last -- see that file for how. +inline std::set& converted_op_types() { + static std::set ops; + return ops; +} + // Description of one tensor (graph input or op output) in the single-op model. struct TensorDesc { std::string name; @@ -72,6 +82,7 @@ class SingleOpDecoder : public GgufDecoder, public std::enable_shared_from_this< m_inputs(std::move(inputs)), m_output(std::move(output)), m_attributes(std::move(attributes)) { + converted_op_types().insert(m_op_type); for (const auto& in : m_inputs) { m_input_names.push_back(in.name); auto p = std::make_shared(in.type, in.shape); @@ -128,6 +139,10 @@ class SingleOpDecoder : public GgufDecoder, public std::enable_shared_from_this< return {m_output.name}; } + // The optional model-scope accessors (get_model_extra_inputs, get_tokenizer_config) both + // default to empty on GgufDecoder, which is exactly right for a single-op test decoder: no + // auxiliary inputs and no tokenizer metadata. So neither is overridden here. + private: const TensorDesc& find_input(const std::string& name) const { for (const auto& in : m_inputs) { diff --git a/src/frontends/gguf/tests/test_arch_conversion.cpp b/src/frontends/gguf/tests/test_arch_conversion.cpp new file mode 100644 index 00000000000000..947fd86e380eda --- /dev/null +++ b/src/frontends/gguf/tests/test_arch_conversion.cpp @@ -0,0 +1,349 @@ +// Copyright (C) 2018-2026 Intel Corporation +// SPDX-License-Identifier: Apache-2.0 +// +// Per-architecture conversion tests for the native .gguf builder path (FrontEnd::load(path) -> +// build_ggml_graph_from_gguf -> GgufBuilderDecoder -> convert). +// +// The single-op tests in test_ops.cpp cover translators in isolation; nothing there exercises +// architecture detection, the GGUF metadata reader, or the builder's whole-graph assembly. This +// file does, over every architecture llama.cpp can emit a model for -- 101 fixtures -- and asserts a +// definite outcome for each: +// +// convert the graph builds, and its fingerprint (op count / input count) matches the pinned +// value, so a refactor that quietly restructures a supported architecture is caught. +// reject the architecture is not on the builder's accept list, so conversion must fail with +// that specific diagnostic. This turns "unsupported" from a claim in a doc into a +// machine-checked fact, and distinguishes a clean rejection from a crash or -- worse -- +// a silent success producing a wrong graph. +// broken a supported architecture that currently does NOT convert. The test asserts it still +// fails, so the defect is recorded rather than forgotten, and FIXING it makes this test +// fail with an instruction to promote the fixture to `convert`. An xfail that silently +// passes is how a known-broken list rots into a permanent excuse list. +// +// == Fixtures == +// test_data/arch_fixtures/*.gguf.hdr are GGUF *headers* only: the bytes before the first tensor's +// data offset, i.e. magic + KV metadata + tensor table. That is the whole input to architecture +// detection and graph construction; the weight bytes after it do not affect the converted graph's +// structure. Each is turned back into a loadable .gguf here by appending the manifest's recorded +// number of zero bytes. Full models would be 559 MB; the headers are ~25 KB each. +// +// The headers are GENERATED, not committed: tests/gen_arch_fixtures.py builds llama.cpp's +// test-llama-archs at a pinned commit and strips its output. CI does this in the Linux build job +// (.github/workflows/job_build_linux.yml) and ships the result in the tests artifact. The +// manifest, by contrast, IS committed -- it carries the reviewed expectation per architecture. +// +// So on a platform where the generator did not run, the manifest is present but its fixtures are +// not. That is a legitimate configuration and the suite skips itself (see present_fixture_count()); +// a manifest entry missing its file while OTHER fixtures exist is a broken generation and fails. +// Running the generator locally into test_data/arch_fixtures makes the suite live in-tree too. + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "common_test_utils/common_utils.hpp" +#include "common_test_utils/file_utils.hpp" +#include "gtest/gtest.h" +#include "op_test_utils.hpp" +#include "openvino/frontend/gguf/frontend.hpp" +#include "openvino/util/file_util.hpp" + +using namespace ov_gguf_test; + +namespace { + +enum class Expectation { Convert, Reject, Broken }; + +struct ArchFixture { + std::string header_file; // e.g. "llama-dense.gguf.hdr" + size_t data_bytes = 0; // zero bytes to append to rebuild a loadable .gguf + Expectation expectation = Expectation::Reject; +}; + +std::string fixture_dir() { + return ov::util::path_join({test_data_dir(), "arch_fixtures"}).string(); +} + +// Parse test_data/arch_fixtures/manifest.txt: " ", '#' comments. +std::vector read_manifest() { + const std::string path = ov::util::path_join({fixture_dir(), "manifest.txt"}).string(); + std::ifstream in(path); + if (!in) { + // Returning empty would silently reduce this file to zero tests, so make it loud. The + // suite below turns this into a failure via ManifestIsPresentAndComplete. + return {}; + } + std::vector fixtures; + std::string line; + while (std::getline(in, line)) { + if (line.empty() || line[0] == '#') { + continue; + } + std::istringstream ls(line); + ArchFixture f; + std::string expectation; + if (!(ls >> f.header_file >> f.data_bytes >> expectation)) { + continue; + } + if (expectation == "convert") { + f.expectation = Expectation::Convert; + } else if (expectation == "reject") { + f.expectation = Expectation::Reject; + } else if (expectation == "broken") { + f.expectation = Expectation::Broken; + } else { + continue; // unknown keyword; ManifestIsPresentAndComplete catches the resulting gap + } + fixtures.push_back(f); + } + return fixtures; +} + +// Whether the generated *.gguf.hdr fixtures are present at all. +// +// Counted rather than checked as a boolean so a PARTIAL generation is distinguishable from no +// generation: none present means the generator did not run on this platform (skip), some present +// means it ran and produced an incomplete set (a failure, asserted in ManifestIsPresentAndComplete). +size_t count_present_fixtures(const std::vector& fixtures) { + size_t present = 0; + for (const auto& f : fixtures) { + if (ov::util::file_exists(ov::util::path_join({fixture_dir(), f.header_file}).string())) { + ++present; + } + } + return present; +} + +// Cached: every parameterized test consults this, and there are ~101 of them. +size_t present_fixture_count() { + static const size_t count = count_present_fixtures(read_manifest()); + return count; +} + +// Reconstruct a loadable .gguf from a header fixture: header bytes followed by data_bytes zeros. +// +// Zero-filling is exact for these fixtures, not an approximation: every tensor llama.cpp's model +// saver writes here is F32, so no dequantization is involved and no block-scale field can be made +// invalid by a zero. The values themselves are irrelevant -- this test asserts graph structure, and +// the converted graph is built from the metadata and tensor table alone. +std::string materialize(const ArchFixture& fixture, const std::string& dir) { + const std::string src = ov::util::path_join({fixture_dir(), fixture.header_file}).string(); + std::ifstream in(src, std::ios::binary); + if (!in) { + return {}; + } + std::vector header((std::istreambuf_iterator(in)), std::istreambuf_iterator()); + + // Drop the ".hdr" suffix so the path ends in .gguf, which the frontend requires. + std::string name = fixture.header_file; + const std::string suffix = ".hdr"; + if (name.size() > suffix.size() && name.compare(name.size() - suffix.size(), suffix.size(), suffix) == 0) { + name.resize(name.size() - suffix.size()); + } + const std::string dst = ov::util::path_join({dir, name}).string(); + + std::ofstream out(dst, std::ios::binary); + if (!out) { + return {}; + } + out.write(header.data(), static_cast(header.size())); + // Chunked zero-fill: a fixture's data section can be tens of MB. + const std::vector zeros(64 * 1024, 0); + size_t remaining = fixture.data_bytes; + while (remaining > 0) { + const size_t chunk = std::min(remaining, zeros.size()); + out.write(zeros.data(), static_cast(chunk)); + remaining -= chunk; + } + out.close(); + return out ? dst : std::string{}; +} + +// Graph fingerprints for the fixtures expected to convert: {op count, input count}. +// +// Pinned so that a refactor which restructures a supported architecture's graph has to be an +// explicit, reviewed change to these numbers rather than an invisible one. They are a structural +// signature, not a correctness claim -- numerical accuracy per architecture is a separate tier that +// needs real checkpoints (docs/testing_architecture.md). +struct Fingerprint { + size_t ops; + size_t inputs; +}; + +const std::map& fingerprints() { + static const std::map fp{ + {"bailingmoe2-moe.gguf.hdr", {457, 10}}, {"ernie4_5-moe-moe.gguf.hdr", {431, 10}}, + {"exaone4-dense.gguf.hdr", {406, 11}}, {"gemma-dense.gguf.hdr", {349, 10}}, + {"gemma2-dense.gguf.hdr", {403, 11}}, {"glm4moe-moe.gguf.hdr", {469, 10}}, + {"gpt-oss-moe.gguf.hdr", {562, 11}}, {"hunyuan-dense-dense.gguf.hdr", {400, 10}}, + {"hunyuan-moe-moe.gguf.hdr", {518, 10}}, {"llama-dense.gguf.hdr", {384, 10}}, + {"llama-moe.gguf.hdr", {490, 10}}, {"maincoder-dense.gguf.hdr", {416, 10}}, + {"minicpm-dense.gguf.hdr", {382, 10}}, {"minicpm-moe.gguf.hdr", {488, 10}}, + {"minimax-m2-moe.gguf.hdr", {518, 10}}, {"mistral3-dense.gguf.hdr", {384, 10}}, + {"mistral3-moe.gguf.hdr", {490, 10}}, {"olmoe-moe.gguf.hdr", {518, 10}}, + {"phi3-dense.gguf.hdr", {364, 10}}, {"qwen2-dense.gguf.hdr", {352, 10}}, + {"qwen3-dense.gguf.hdr", {400, 10}}, {"qwen3moe-moe.gguf.hdr", {534, 10}}, + {"qwen35-dense.gguf.hdr", {391, 10}}, {"smollm3-dense.gguf.hdr", {368, 10}}, + }; + return fp; +} + +// A scratch directory that outlives one test, so the reconstructed .gguf can be written once per +// test and removed afterwards without leaving multi-MB files behind on failure. +class ScratchDir { +public: + ScratchDir() : m_path(ov::test::utils::generateTestFilePrefix() + "_gguf_arch") { + ov::util::create_directory_recursive(std::filesystem::path(m_path)); + } + ~ScratchDir() { + ov::test::utils::removeDir(m_path); + } + const std::string& path() const { + return m_path; + } + +private: + std::string m_path; +}; + +class GGUFArchConversion : public ::testing::TestWithParam {}; + +TEST_P(GGUFArchConversion, MatchesManifestExpectation) { + const ArchFixture fixture = GetParam(); + if (present_fixture_count() == 0) { + GTEST_SKIP() << "no arch fixtures in " << fixture_dir() + << " -- generate them with tests/gen_arch_fixtures.py --fetch"; + } + ScratchDir scratch; + const std::string model_path = materialize(fixture, scratch.path()); + ASSERT_FALSE(model_path.empty()) << "could not materialize fixture " << fixture.header_file; + + ov::frontend::gguf::FrontEnd fe; + std::shared_ptr model; + std::string error; + try { + model = fe.convert(fe.load(model_path)); + } catch (const std::exception& e) { + error = e.what(); + } + ov::test::utils::removeFile(model_path); + + switch (fixture.expectation) { + case Expectation::Convert: { + ASSERT_TRUE(model) << "expected " << fixture.header_file << " to convert, but it failed:\n" << error; + const auto it = fingerprints().find(fixture.header_file); + ASSERT_NE(it, fingerprints().end()) + << fixture.header_file << " is marked `convert` in the manifest but has no fingerprint entry in " + << "test_arch_conversion.cpp; add one."; + EXPECT_EQ(model->get_ops().size(), it->second.ops) + << "graph op count for " << fixture.header_file << " changed. If the new structure is intended, " + << "update fingerprints() -- and say why in the commit message."; + EXPECT_EQ(model->inputs().size(), it->second.inputs) + << "graph input count for " << fixture.header_file << " changed; see fingerprints()."; + break; + } + case Expectation::Reject: { + // The accept list must be what rejects it: a different failure means the file broke the + // reader or the builder somewhere it should never have reached. + ASSERT_FALSE(model) << fixture.header_file << " is marked `reject` but converted successfully. If the " + << "builder now supports this architecture, add it to verified/experimental_archs() " + << "and change the manifest line to `convert` with a fingerprint."; + EXPECT_NE(error.find("native GGUF builder does not support architecture"), std::string::npos) + << fixture.header_file << " failed for a reason other than the architecture accept list, which means " + << "it got further into the builder than it should. Actual error:\n" + << error; + break; + } + case Expectation::Broken: { + // XPASS is a failure on purpose: a `broken` entry that starts converting must be promoted, + // not left to accumulate as a permanent excuse. + ASSERT_FALSE(model) << fixture.header_file << " is marked `broken` but now converts. The defect is fixed: " + << "change its manifest line to `convert` and add its fingerprint to fingerprints()."; + // Not an accept-list rejection -- that is what makes it a defect rather than an unsupported + // architecture. + EXPECT_EQ(error.find("native GGUF builder does not support architecture"), std::string::npos) + << fixture.header_file << " is marked `broken` but is actually rejected by the accept list; it should " + << "be `reject`."; + break; + } + } +} + +std::string fixture_test_name(const ::testing::TestParamInfo& info) { + // "llama-dense.gguf.hdr" -> "llama_dense"; gtest names allow only [A-Za-z0-9_]. + std::string name = info.param.header_file; + const auto dot = name.find(".gguf"); + if (dot != std::string::npos) { + name.resize(dot); + } + for (auto& c : name) { + if (!std::isalnum(static_cast(c))) { + c = '_'; + } + } + return name; +} + +INSTANTIATE_TEST_SUITE_P(GGUFArchs, GGUFArchConversion, ::testing::ValuesIn(read_manifest()), fixture_test_name); + +// The manifest is read at static-init time to instantiate the suite above, so a missing or truncated +// manifest would quietly produce zero tests instead of a failure. Assert its shape explicitly. +TEST(GGUFArchConversionManifest, IsPresentAndComplete) { + const auto fixtures = read_manifest(); + ASSERT_FALSE(fixtures.empty()) << "no fixtures parsed from " << fixture_dir() + << "/manifest.txt -- is the test_data directory installed?"; + // Guards against a manifest that parses but lost most of its lines. + EXPECT_GE(fixtures.size(), 90u) << "manifest has far fewer fixtures than the ~101 architectures llama.cpp " + << "emits; regenerate with gen_arch_fixtures.py"; + + // Fixture files are generated, so "none present" is a valid platform configuration and only + // means the suite above skips. "Some present" is not: it means the generator ran and produced + // an incomplete set, which would silently shrink coverage to whatever happened to be written. + const size_t present = present_fixture_count(); + if (present != 0) { + for (const auto& f : fixtures) { + const std::string path = ov::util::path_join({fixture_dir(), f.header_file}).string(); + EXPECT_TRUE(ov::util::file_exists(path)) + << "manifest lists " << f.header_file << " but the file is missing, while " << present << " of " + << fixtures.size() << " fixtures are present -- fixture generation was incomplete. Re-run " + << "tests/gen_arch_fixtures.py (CI: the 'Generate GGUF arch fixtures' build step)."; + } + } else { + GTEST_LOG_(INFO) << "no fixture files in " << fixture_dir() + << "; the GGUFArchs suite is skipped on this platform"; + } + + // Manifest/fingerprint consistency is checked whether or not the fixtures were generated: both + // are committed source, so a mismatch is a bug on every platform. + size_t convertible = 0; + for (const auto& f : fixtures) { + if (f.expectation == Expectation::Convert) { + ++convertible; + EXPECT_NE(fingerprints().count(f.header_file), 0u) + << f.header_file << " is `convert` but has no fingerprint entry"; + } + } + EXPECT_GT(convertible, 0u) << "no fixture is expected to convert, which would make this suite vacuous"; + + // Every fingerprint must correspond to a live manifest entry, so stale entries cannot linger + // after a fixture is dropped upstream. + for (const auto& entry : fingerprints()) { + bool found = false; + for (const auto& f : fixtures) { + if (f.header_file == entry.first && f.expectation == Expectation::Convert) { + found = true; + break; + } + } + EXPECT_TRUE(found) << "fingerprints() has a stale entry for " << entry.first + << " (no matching `convert` line in the manifest)"; + } +} + +} // namespace diff --git a/src/frontends/gguf/tests/test_data/arch_fixtures/.gitignore b/src/frontends/gguf/tests/test_data/arch_fixtures/.gitignore new file mode 100644 index 00000000000000..0810a35309679e --- /dev/null +++ b/src/frontends/gguf/tests/test_data/arch_fixtures/.gitignore @@ -0,0 +1,4 @@ +# The *.gguf.hdr fixtures are generated by ../../gen_arch_fixtures.py, not committed -- see the +# manifest header and test_arch_conversion.cpp. Running the generator in-tree is the supported way +# to make the arch suite live locally, so keep its output out of `git status`. +*.gguf.hdr diff --git a/src/frontends/gguf/tests/test_data/arch_fixtures/manifest.txt b/src/frontends/gguf/tests/test_data/arch_fixtures/manifest.txt new file mode 100644 index 00000000000000..7ba44736506634 --- /dev/null +++ b/src/frontends/gguf/tests/test_data/arch_fixtures/manifest.txt @@ -0,0 +1,116 @@ +# GGUF per-architecture conversion fixtures -- see gen_arch_fixtures.py (generator, run offline) and +# test_arch_conversion.cpp (consumer). Generated from llama.cpp 476c01efe88aad7880a8132d5d3a415f2ca75139 at seed 1. +# +# One line per fixture: +# +# +# header-only fixture in this directory; the loadable .gguf is followed by +# zero bytes (all fixture tensors are F32, so no dequant is involved). +# what conversion must do, and the test asserts exactly this: +# convert converts cleanly. The graph fingerprint is pinned in test_arch_conversion.cpp. +# reject the architecture is not on the builder's accept list, so conversion must fail with +# that specific diagnostic -- not a crash, and not a silent wrong-graph success. +# broken a supported architecture that currently fails to convert: a real defect, recorded so +# it cannot be forgotten. The test asserts it STILL FAILS; fixing the defect turns this +# line into a failure telling you to promote it to `convert`. +arcee-dense.gguf.hdr 3944448 reject +arctic-moe.gguf.hdr 8669440 reject +baichuan-dense.gguf.hdr 4730752 reject +bailingmoe-moe.gguf.hdr 9453824 reject +bailingmoe2-moe.gguf.hdr 7087968 convert +bloom-dense.gguf.hdr 3958272 reject +chatglm-dense.gguf.hdr 4730624 reject +codeshell-dense.gguf.hdr 3956480 reject +cogvlm-dense.gguf.hdr 9181056 reject +command-r-dense.gguf.hdr 4597632 reject +dbrx-moe.gguf.hdr 7087744 reject +deci-dense.gguf.hdr 4741248 reject +deepseek-moe.gguf.hdr 7092288 reject +deepseek2-moe.gguf.hdr 4403296 reject +deepseek32-moe.gguf.hdr 4733152 reject +dots1-moe.gguf.hdr 7094368 reject +dream-dense.gguf.hdr 4731264 reject +ernie4_5-moe-moe.gguf.hdr 5914528 convert +ernie4_5-moe.gguf.hdr 4732800 reject +exaone-dense.gguf.hdr 4731008 reject +exaone4-dense.gguf.hdr 4733056 convert +falcon-dense.gguf.hdr 3942912 reject +falcon-h1-dense.gguf.hdr 8985152 reject +gemma-dense.gguf.hdr 4599680 convert +gemma2-dense.gguf.hdr 4603776 convert +glm-dsa-moe.gguf.hdr 4733152 reject +glm4-dense.gguf.hdr 4734720 reject +glm4moe-moe.gguf.hdr 7094368 convert +gpt-oss-moe.gguf.hdr 7112704 convert +gpt2-dense.gguf.hdr 4087296 reject +gptneox-dense.gguf.hdr 3956224 reject +granite-dense.gguf.hdr 4741248 reject +granite-moe.gguf.hdr 7096448 reject +granitehybrid-dense.gguf.hdr 5810272 reject +granitemoe-dense.gguf.hdr 4741248 reject +granitemoe-moe.gguf.hdr 7096448 reject +grok-moe.gguf.hdr 9457920 reject +grovemoe-moe.gguf.hdr 8669056 reject +hunyuan-dense-dense.gguf.hdr 4732800 convert +hunyuan-moe-moe.gguf.hdr 9455872 convert +hunyuan_vl-dense.gguf.hdr 4732800 reject +hy_v3-moe.gguf.hdr 11815360 reject +internlm2-dense.gguf.hdr 4730752 reject +jais-dense.gguf.hdr 4745856 reject +jais2-dense.gguf.hdr 3956480 broken +jamba-dense.gguf.hdr 6061856 reject +kimi-linear-moe.gguf.hdr 2833984 reject +lfm2-dense.gguf.hdr 4731520 reject +lfm2moe-moe.gguf.hdr 5913248 reject +llada-dense.gguf.hdr 4735104 reject +llada-moe-moe.gguf.hdr 7096192 reject +llama-dense.gguf.hdr 4741248 convert +llama-moe.gguf.hdr 7096448 convert +llama4-moe.gguf.hdr 13921664 reject +maincoder-dense.gguf.hdr 4732800 convert +mamba-dense.gguf.hdr 5029120 reject +mamba2-dense.gguf.hdr 4509120 reject +minicpm-dense.gguf.hdr 4741248 convert +minicpm-moe.gguf.hdr 7096448 convert +minicpm3-dense.gguf.hdr 7616512 reject +minimax-m2-moe.gguf.hdr 7098304 convert +mistral3-dense.gguf.hdr 4741248 convert +mistral3-moe.gguf.hdr 7096448 convert +mistral4-moe.gguf.hdr 4403296 reject +mpt-dense.gguf.hdr 4094464 reject +nemotron-dense.gguf.hdr 3956480 reject +nemotron_h-dense.gguf.hdr 4230240 reject +nemotron_h_moe-dense.gguf.hdr 4230240 reject +olmo-dense.gguf.hdr 4725632 reject +olmoe-moe.gguf.hdr 7098240 convert +openelm-dense.gguf.hdr 4595328 reject +orion-dense.gguf.hdr 4735872 reject +paddleocr-moe.gguf.hdr 4732800 reject +pangu-embedded-dense.gguf.hdr 4733056 reject +phi2-dense.gguf.hdr 3952896 reject +phi3-dense.gguf.hdr 4731136 convert +phimoe-moe.gguf.hdr 7102336 reject +plamo-dense.gguf.hdr 4728704 reject +plamo2-dense.gguf.hdr 5920096 reject +qwen-dense.gguf.hdr 3550848 reject +qwen2-dense.gguf.hdr 4731264 convert +qwen2moe-moe.gguf.hdr 9455872 reject +qwen2vl-dense.gguf.hdr 4731264 reject +qwen3-dense.gguf.hdr 4733824 convert +qwen35-dense.gguf.hdr 5271040 convert +qwen35moe-moe.gguf.hdr 9995904 reject +qwen3moe-moe.gguf.hdr 7096192 convert +qwen3next-moe.gguf.hdr 11044416 reject +qwen3vl-dense.gguf.hdr 4733824 reject +qwen3vlmoe-moe.gguf.hdr 7096192 reject +refact-dense.gguf.hdr 4741248 reject +refact-moe.gguf.hdr 7096448 reject +rnd1-moe.gguf.hdr 7096192 reject +seed_oss-dense.gguf.hdr 4730752 reject +smallthinker-moe.gguf.hdr 7094144 reject +smollm3-dense.gguf.hdr 4730752 convert +stablelm-dense.gguf.hdr 4739968 reject +starcoder-dense.gguf.hdr 4087296 reject +starcoder2-dense.gguf.hdr 3956480 reject +talkie-dense.gguf.hdr 4725760 reject +xverse-dense.gguf.hdr 4730752 reject diff --git a/src/frontends/gguf/tests/test_data/gelu_ggml_expected.npy b/src/frontends/gguf/tests/test_data/gelu_ggml_expected.npy new file mode 100644 index 00000000000000..0813a3f6b4c5a6 Binary files /dev/null and b/src/frontends/gguf/tests/test_data/gelu_ggml_expected.npy differ diff --git a/src/frontends/gguf/tests/test_data/gelu_ggml_input.npy b/src/frontends/gguf/tests/test_data/gelu_ggml_input.npy new file mode 100644 index 00000000000000..790c02ef2163a6 Binary files /dev/null and b/src/frontends/gguf/tests/test_data/gelu_ggml_input.npy differ diff --git a/src/frontends/gguf/tests/test_data/gelu_quick_ggml_expected.npy b/src/frontends/gguf/tests/test_data/gelu_quick_ggml_expected.npy new file mode 100644 index 00000000000000..a0d9a49824ccff Binary files /dev/null and b/src/frontends/gguf/tests/test_data/gelu_quick_ggml_expected.npy differ diff --git a/src/frontends/gguf/tests/test_data/gelu_quick_ggml_input.npy b/src/frontends/gguf/tests/test_data/gelu_quick_ggml_input.npy new file mode 100644 index 00000000000000..790c02ef2163a6 Binary files /dev/null and b/src/frontends/gguf/tests/test_data/gelu_quick_ggml_input.npy differ diff --git a/src/frontends/gguf/tests/test_data/silu_ggml_expected.npy b/src/frontends/gguf/tests/test_data/silu_ggml_expected.npy new file mode 100644 index 00000000000000..30ada7c05310ba Binary files /dev/null and b/src/frontends/gguf/tests/test_data/silu_ggml_expected.npy differ diff --git a/src/frontends/gguf/tests/test_data/silu_ggml_input.npy b/src/frontends/gguf/tests/test_data/silu_ggml_input.npy new file mode 100644 index 00000000000000..790c02ef2163a6 Binary files /dev/null and b/src/frontends/gguf/tests/test_data/silu_ggml_input.npy differ diff --git a/src/frontends/gguf/tests/test_dequant_vs_ggml.cpp b/src/frontends/gguf/tests/test_dequant_vs_ggml.cpp index 77c1f473a12fd7..795592eb06dbb3 100644 --- a/src/frontends/gguf/tests/test_dequant_vs_ggml.cpp +++ b/src/frontends/gguf/tests/test_dequant_vs_ggml.cpp @@ -4,7 +4,7 @@ // Dequantization correctness tests with REAL ggml as the oracle. // // The reference data was produced offline by linking real ggml from llama.cpp -// (see tests/gen_ggml_reference.c): ggml quantizes smooth, asymmetric synthetic +// (captured from real ggml): ggml quantizes smooth, asymmetric synthetic // data into real GGUF-format blocks (_qbytes) and dequantizes those exact bytes // (_deq). The committed .npy files mean the tests need no ggml / llama.cpp at // build or run time. @@ -15,14 +15,13 @@ // Tolerance: ggml stores K-quant scales as f16 and the dequant subgraph runs in f16, // so allow ~3e-3 (matching llama.cpp's MAX_QUANTIZATION_TOTAL_ERROR-class thresholds). -#include - #include #include #include #include #include +#include "gtest/gtest.h" #include "op_test_utils.hpp" #include "openvino/op/constant.hpp" #include "openvino/op/result.hpp" @@ -160,16 +159,18 @@ TEST_P(FaithfulDequantVsGGML, MatchesGgmlToFloat) { for (size_t r = 0; r < kRows; ++r) { c.dq(qbytes.data() + r * bytes_per_row, kCols, ours.data() + r * kCols); } - EXPECT_LE(max_abs_diff(ours, ref), 3e-3f) - << c.stem << ": faithful per-row dequant diverges from ggml to_float"; + EXPECT_LE(max_abs_diff(ours, ref), 3e-3f) << c.stem << ": faithful per-row dequant diverges from ggml to_float"; } -INSTANTIATE_TEST_SUITE_P(FaithfulKQuant, - FaithfulDequantVsGGML, - ::testing::Values(FaithfulCase{"q4_k", GGUF_TYPE_Q4_K, ov::frontend::gguf::dequant_row_q4_k_f32_for_test}, - FaithfulCase{"q5_k", GGUF_TYPE_Q5_K, ov::frontend::gguf::dequant_row_q5_k_f32_for_test}, - FaithfulCase{"q6_k", GGUF_TYPE_Q6_K, ov::frontend::gguf::dequant_row_q6_k_f32_for_test}), - [](const ::testing::TestParamInfo& i) { return std::string(i.param.stem); }); +INSTANTIATE_TEST_SUITE_P( + FaithfulKQuant, + FaithfulDequantVsGGML, + ::testing::Values(FaithfulCase{"q4_k", GGUF_TYPE_Q4_K, ov::frontend::gguf::dequant_row_q4_k_f32_for_test}, + FaithfulCase{"q5_k", GGUF_TYPE_Q5_K, ov::frontend::gguf::dequant_row_q5_k_f32_for_test}, + FaithfulCase{"q6_k", GGUF_TYPE_Q6_K, ov::frontend::gguf::dequant_row_q6_k_f32_for_test}), + [](const ::testing::TestParamInfo& i) { + return std::string(i.param.stem); + }); INSTANTIATE_TEST_SUITE_P(AllQuantTypes, DequantVsGGML, @@ -183,6 +184,9 @@ INSTANTIATE_TEST_SUITE_P(AllQuantTypes, DeqCase{"q4_k", GGUF_TYPE_Q4_K, kTolIntZp}, DeqCase{"q5_k", GGUF_TYPE_Q5_K, kTolRequant}, DeqCase{"q6_k", GGUF_TYPE_Q6_K, kTolRequant}, + // Q2_0 is bit-exact: both sides compute (code - 1) * d + // from the same f16 scale, and the u8 zero-point of 1 is + // represented exactly, so no dequant noise is introduced. DeqCase{"q2_0", GGUF_TYPE_Q2_0, kTolExact}), [](const ::testing::TestParamInfo& i) { return std::string(i.param.stem); diff --git a/src/frontends/gguf/tests/test_extensions.cpp b/src/frontends/gguf/tests/test_extensions.cpp index 505311f09a124a..a0f79ce2822338 100644 --- a/src/frontends/gguf/tests/test_extensions.cpp +++ b/src/frontends/gguf/tests/test_extensions.cpp @@ -3,17 +3,37 @@ // // Tests for FrontEnd::add_extension (the extension-passing path in frontend.cpp). // -// A ConversionExtension registers a custom translator for a ggml op name; the frontend -// merges it into the op table (overriding a built-in translator on name collision, or -// adding a translator for an otherwise unsupported op). The converter receives an -// ov::frontend::NodeContext, which the gguf NodeContext derives from. +// Two extension kinds are covered: +// +// - ov::frontend::ConversionExtension registers a custom translator for a ggml op name; the +// frontend merges it into the op table (overriding a built-in translator on name collision, or +// adding a translator for an otherwise unsupported op). The converter receives an +// ov::frontend::NodeContext, which the gguf NodeContext derives from. +// +// - ov::frontend::DecoderTransformationExtension registers a normalization pass, run ahead of the +// frontend's built-in lowerings. This is how the EXECUTION MODE is chosen: conversion always +// yields a stateless graph (KV caches as Parameter/Result pairs written by a SetRows +// placeholder), and a caller that wants an OpenVINO KV cache registers +// ov::frontend::gguf::pass::MakeStateful here, which consumes those SetRows ops before the +// default stateless lowering ever sees them. -#include -#include -#include +#include +#include +#include #include "op_test_utils.hpp" #include "openvino/frontend/extension/conversion.hpp" +#include "openvino/frontend/extension/decoder_transformation.hpp" +#include "openvino/frontend/gguf/make_stateful.hpp" +#include "openvino/frontend/gguf/set_rows_op.hpp" +#include "openvino/op/abs.hpp" +#include "openvino/op/assign.hpp" +#include "openvino/op/concat.hpp" +#include "openvino/op/gather.hpp" +#include "openvino/op/multiply.hpp" +#include "openvino/op/negative.hpp" +#include "openvino/op/read_value.hpp" +#include "openvino/op/scatter_update.hpp" using namespace ov_gguf_test; @@ -93,3 +113,207 @@ TEST(GGUFExtensions, UnsupportedOpWithoutExtensionThrows) { .output("out", ov::element::f32, {3}); EXPECT_ANY_THROW(builder.build()); } + +// ── DecoderTransformationExtension: choosing the execution mode ───────────────────────────────── + +namespace { + +// One GGML_OP_SET_ROWS writing `data` rows at `idx` into the `cache` input -- the shape of a KV +// cache write, in the layout the native .gguf builder emits: [1, tokens, n_head_kv, head_size], +// whose one dynamic axis (1, the token axis) is what MakeStateful infers the append axis from. +SingleOpBuilder kv_cache_write_builder() { + return SingleOpBuilder() + .op("GGML_OP_SET_ROWS") + .input("data", ov::element::f32, {1, -1, 2, 4}) + .input("idx", ov::element::i64, {1, 1, 1, -1}) + .input("cache", ov::element::f16, {1, -1, 2, 4}) + .output("cache_out", ov::element::f16, {1, -1, 2, 4}); +} + +size_t count_ops_of_type(const std::shared_ptr& model, const ov::DiscreteTypeInfo& type) { + size_t n = 0; + for (const auto& op : model->get_ops()) { + if (op->get_type_info() == type) { + n++; + } + } + return n; +} + +} // namespace + +// The default: with no extension registered, conversion lowers every SetRows to the stateless +// ScatterUpdate form, and the cache stays an ordinary model input/output. This is the baseline the +// design rests on -- the frontend itself is stateless, like an optimum-intel export. +TEST(GGUFExtensions, NoExtensionYieldsStatelessCache) { + auto model = kv_cache_write_builder().build(); + + EXPECT_TRUE(model->get_variables().empty()); + EXPECT_TRUE(model->get_sinks().empty()); + EXPECT_EQ(count_ops_of_type(model, ov::op::v3::ScatterUpdate::get_type_info_static()), 1); + // The SetRows placeholder is an internal op and must never survive conversion. + EXPECT_EQ(count_ops_of_type(model, SetRows::get_type_info_static()), 0); + // cache is still an input, cache_out still an output. + EXPECT_EQ(model->get_parameters().size(), 3); + EXPECT_EQ(model->get_results().size(), 1); + + // No beam_idx: it is a stateful-cache concept, so the stateless graph must not carry one. This is + // what lets the native builder and a llama.cpp cgraph decoder agree on their stateless IO -- a + // decoder-declared beam_idx would be an input with no consumer here. + for (const auto& p : model->get_parameters()) { + EXPECT_NE(p->get_friendly_name(), "beam_idx"); + } +} + +// Registering MakeStateful as a DecoderTransformationExtension swaps the execution mode: the same +// conversion now yields an OpenVINO state. The cache Parameter/Result pair is gone, replaced by a +// Variable with a ReadValue/Concat/Assign, and no ScatterUpdate is emitted -- the extension ran +// ahead of the built-in stateless lowering and consumed the SetRows first. +TEST(GGUFExtensions, MakeStatefulExtensionYieldsStatefulCache) { + auto model = kv_cache_write_builder().build_with_extensions( + {std::make_shared(pass::MakeStateful())}); + + ASSERT_EQ(model->get_variables().size(), 1); + EXPECT_EQ(model->get_sinks().size(), 1); + EXPECT_EQ(count_ops_of_type(model, ov::op::v6::ReadValue::get_type_info_static()), 1); + EXPECT_EQ(count_ops_of_type(model, ov::op::v6::Assign::get_type_info_static()), 1); + EXPECT_EQ(count_ops_of_type(model, ov::op::v3::ScatterUpdate::get_type_info_static()), 0); + EXPECT_EQ(count_ops_of_type(model, SetRows::get_type_info_static()), 0); + + // The cache left the model's IO entirely: data + idx remain, and beam_idx was ADDED by the pass + // (see below). The cache Result became the Assign sink. + EXPECT_EQ(model->get_parameters().size(), 3); + EXPECT_EQ(model->get_results().size(), 0); + + // beam_idx belongs to the state, so the pass creates it -- no decoder declares it. Its Gather on + // the past is what CPU's stateful_sdpa_fusion matches. + auto beam_idx = std::find_if(model->get_parameters().begin(), + model->get_parameters().end(), + [](const std::shared_ptr& p) { + return p->get_friendly_name() == "beam_idx"; + }); + ASSERT_NE(beam_idx, model->get_parameters().end()); + EXPECT_EQ((*beam_idx)->get_element_type(), ov::element::i32); + EXPECT_EQ((*beam_idx)->get_partial_shape(), ov::PartialShape({-1})); + EXPECT_EQ(count_ops_of_type(model, ov::op::v8::Gather::get_type_info_static()), 1); + + // The Variable is named after the cache input and its append axis is dynamic (the state grows + // by this step's rows on every inference), the rest keeping the cache's declared dims. + const auto& info = model->get_variables()[0]->get_info(); + EXPECT_EQ(info.variable_id, "cache"); + EXPECT_EQ(info.data_type, ov::element::f16); + EXPECT_EQ(info.data_shape, ov::PartialShape({1, -1, 2, 4})); +} + +// skip_caches leaves a named cache stateless while other caches are converted. A sliding-window +// cache needs this: it is evicted from the front, not only appended to, so an append-grown Variable +// would not reproduce it. +TEST(GGUFExtensions, MakeStatefulSkipsNamedCache) { + auto model = kv_cache_write_builder().build_with_extensions( + {std::make_shared(pass::MakeStateful({"cache"}))}); + + // The only cache was skipped, so the pass made no change and the built-in stateless lowering + // handled the SetRows -- an identical result to registering no extension at all. + EXPECT_TRUE(model->get_variables().empty()); + EXPECT_EQ(count_ops_of_type(model, ov::op::v3::ScatterUpdate::get_type_info_static()), 1); + EXPECT_EQ(model->get_parameters().size(), 3); + EXPECT_EQ(model->get_results().size(), 1); +} + +// ── the stateless IO contract: two decoders of one model must agree ────────────────────────────── + +namespace { + +// A decoder that routes a named subset of its inputs through get_model_extra_inputs() instead of +// get_model_inputs(), which is the one structural difference between how the native .gguf builder +// and the llama.cpp cgraph decoder present a model's IO. Both halves land in the same graph, so +// converting either way must yield the same stateless inputs. +class SplitIoDecoder : public SingleOpDecoder { +public: + SplitIoDecoder(const SingleOpDecoder& base, const std::set& as_extra) : SingleOpDecoder(base) { + for (const auto& name : as_extra) { + auto it = m_split_main.find(name); + if (it == m_split_main.end()) { + throw std::runtime_error("SplitIoDecoder: no such input '" + name + "'"); + } + m_split_extra[name] = it->second; + m_split_main.erase(it); + } + } + + const std::map>& get_model_inputs() const override { + return m_split_main; + } + const std::map>& get_model_extra_inputs() const override { + return m_split_extra; + } + +private: + // Seeded from the base decoder's inputs (member initializers run before the constructor body), + // then partitioned by that body. + std::map> m_split_main = SingleOpDecoder::get_model_inputs(); + std::map> m_split_extra; +}; + +std::set input_names(const std::shared_ptr& model) { + std::set names; + for (const auto& p : model->get_parameters()) { + names.insert(p->get_friendly_name()); + } + return names; +} + +} // namespace + +// The frontend invents no inputs of its own: the stateless graph's inputs are exactly what the +// decoder declared, however the decoder chose to split them between get_model_inputs() and +// get_model_extra_inputs(). That split is the one structural difference between the native builder +// and the llama.cpp cgraph decoder, so pinning it down here is half of "the two decoders produce the +// same graph"; the other half -- that neither decoder declares an input the other cannot, beam_idx +// being the case that got this wrong -- needs a real .gguf and lives in the model-level checks. +TEST(GGUFExtensions, StatelessIoIsExactlyTheDecoderInputs) { + const std::set declared{"data", "idx", "cache"}; + + auto base = kv_cache_write_builder(); + auto via_main = base.build(); + EXPECT_EQ(input_names(via_main), declared); + + // The same op, with "cache" and "idx" presented as auxiliary inputs the way the cgraph decoder + // presents its extras. Same graph inputs -> the two decoders agree. + FrontEnd fe; + auto split = std::make_shared(*std::dynamic_pointer_cast(base.decoder()), + std::set{"cache", "idx"}); + auto via_extra = fe.convert(fe.load(std::static_pointer_cast(split))); + EXPECT_EQ(input_names(via_extra), declared); +} + +// And making the model stateful adds exactly one input, beam_idx, on top of that contract -- so the +// stateful IO is a function of the pass, not of which decoder produced the stateless graph. +TEST(GGUFExtensions, MakeStatefulAddsOnlyBeamIdx) { + auto stateless = input_names(kv_cache_write_builder().build()); + auto stateful = input_names(kv_cache_write_builder().build_with_extensions( + {std::make_shared(pass::MakeStateful())})); + + // The cache Parameter became a Variable, and beam_idx appeared. + stateless.erase("cache"); + stateless.insert("beam_idx"); + EXPECT_EQ(stateful, stateless); +} + +// A DecoderTransformationExtension can hold any pass, not only the ones the frontend ships: here a +// plain lambda pass, which must run during conversion (it renames the model, observable after). +TEST(GGUFExtensions, ArbitraryTransformationExtensionRuns) { + auto model = SingleOpBuilder() + .op("GGML_OP_SCALE") + .input("x", ov::element::f32, {2, 2}) + .output("out", ov::element::f32, {2, 2}) + .attr("scale", 2.0f) + .attr("bias", 0.0f) + .build_with_extensions({std::make_shared( + [](const std::shared_ptr& m) { + m->set_friendly_name("touched_by_extension"); + return true; + })}); + + EXPECT_EQ(model->get_friendly_name(), "touched_by_extension"); +} diff --git a/src/frontends/gguf/tests/test_graph_fingerprint.py b/src/frontends/gguf/tests/test_graph_fingerprint.py new file mode 100644 index 00000000000000..a7826c1b06b01c --- /dev/null +++ b/src/frontends/gguf/tests/test_graph_fingerprint.py @@ -0,0 +1,54 @@ +# Copyright (C) 2018-2026 Intel Corporation +# SPDX-License-Identifier: Apache-2.0 +# +# Gated graph-fingerprint regression test for the GGUF frontend native builder. +# +# The GGUF files are multi-GB and cannot ship in-repo, so this test is OPT-IN: it runs only when +# GGUF_FINGERPRINT_MODELS points at local models. It converts each model through the frontend +# (requested by name, since it is not auto-selectable) and asserts the graph fingerprint matches +# any unintended change to the produced graph for a supported architecture. +# +# GGUF_FINGERPRINT_MODELS format: semicolon-separated entries "name=path=expected_sighash" +# (the expected hash is optional; when omitted the test just asserts the model converts and prints +# the observed hash so it can be recorded as the baseline). +# +# Example: +# GGUF_FINGERPRINT_MODELS="qwen3=/models/Qwen3-0.6B-Q8_0.gguf=87cf7f3b0742a6cd" \ +# PYTHONPATH=/python LD_LIBRARY_PATH= pytest test_graph_fingerprint.py -v + +import os + +import pytest + +from graph_fingerprint import fingerprint + +_SPEC = os.environ.get("GGUF_FINGERPRINT_MODELS", "").strip() + + +def _parse_spec(spec): + cases = [] + for entry in spec.split(";"): + entry = entry.strip() + if not entry: + continue + parts = entry.split("=") + name, path = parts[0], parts[1] + expected = parts[2] if len(parts) > 2 else None + cases.append((name, path, expected)) + return cases + + +_CASES = _parse_spec(_SPEC) + + +@pytest.mark.skipif(not _CASES, reason="set GGUF_FINGERPRINT_MODELS=name=path[=hash];... to run") +@pytest.mark.parametrize("name,path,expected", _CASES, ids=[c[0] for c in _CASES]) +def test_graph_fingerprint(name, path, expected): + assert os.path.exists(path), f"model not found: {path}" + fp = fingerprint(path) + print(f"\n{name}: sighash={fp['sighash']} ops={fp['num_ops']} inputs={fp['inputs']}") + if expected: + assert fp["sighash"] == expected, ( + f"{name} graph fingerprint changed: got {fp['sighash']}, expected {expected}. " + f"The builder produced a different graph for this architecture." + ) diff --git a/src/frontends/gguf/tests/test_op_coverage.cpp b/src/frontends/gguf/tests/test_op_coverage.cpp new file mode 100644 index 00000000000000..b72524c410fd87 --- /dev/null +++ b/src/frontends/gguf/tests/test_op_coverage.cpp @@ -0,0 +1,105 @@ +// Copyright (C) 2018-2026 Intel Corporation +// SPDX-License-Identifier: Apache-2.0 +// +// Coverage gate: every ggml op registered in op_table.cpp must be exercised by some test in this +// binary. +// +// Without this, the per-op suite silently stops keeping pace with the op table -- adding a +// translator and forgetting its test is invisible, and a wrong-but-plausible formula ships. That is +// not hypothetical: GGML_UNARY_OP_GELU_QUICK was registered with the tanh-GELU formula instead of +// ggml's x*sigmoid(1.702x) and went unnoticed because nothing converted it. +// +// The "tested" side of the comparison is collected at run time: SingleOpDecoder's constructor +// records its op type in converted_op_types(), so the record cannot drift from what the tests +// actually do (a hand-written list would just be a second thing to forget). Consequently this check +// must run AFTER all other tests, which gtest guarantees for a global test environment's TearDown -- +// so the assertion lives there rather than in a TEST body. +// +// A gtest --gtest_filter that excludes op tests would leave the record incomplete and the gate would +// fire spuriously, so it only asserts when the full suite ran (no filter narrowing in effect). + +#include +#include +#include +#include + +#include "gtest/gtest.h" +#include "op_table.hpp" +#include "op_test_utils.hpp" + +using namespace ov_gguf_test; + +namespace { + +// Ops that are registered but intentionally not covered by a single-op test, each with the reason. +// Anything here needs a justification that is about the op's nature, not about effort -- an op that +// is merely awkward to test belongs in the suite, not on this list. +const std::set& coverage_exemptions() { + static const std::set exemptions{ + // Aliases of ops already covered under a different ggml name, translated by the very same + // function pointer, so a separate case would test the same code path twice. + // (none currently -- GGML_OP_ADD1 has its own test because its broadcast shape differs) + }; + return exemptions; +} + +class OpCoverageEnvironment : public ::testing::Environment { +public: + void TearDown() override { + // Only meaningful when the whole suite ran; a narrowing filter makes the record partial. + const std::string filter = ::testing::GTEST_FLAG(filter); + if (filter != "*" && filter != "*.*") { + GTEST_LOG_(INFO) << "op coverage gate skipped: --gtest_filter=" << filter << " is in effect"; + return; + } + + const auto& converted = converted_op_types(); + std::vector missing; + for (const auto& entry : ov::frontend::gguf::get_supported_ops()) { + const std::string& op = entry.first; + if (converted.count(op) == 0 && coverage_exemptions().count(op) == 0) { + missing.push_back(op); + } + } + std::sort(missing.begin(), missing.end()); + + if (!missing.empty()) { + std::string list; + for (const auto& op : missing) { + list += "\n " + op; + } + // Failures raised from an Environment's TearDown are counted separately from tests, so + // the run reports "0 FAILED TESTS" while still exiting non-zero. Name the check + // explicitly so the CI log is not misread as a spurious failure. + ADD_FAILURE() << "[GGUF op coverage gate] " << missing.size() + << " op(s) registered in op_table.cpp have no test in ov_gguf_frontend_tests:" << list + << "\nAdd a case to test_ops.cpp (or test_weights.cpp for weight leaves). If the op " + << "genuinely cannot be tested in isolation, add it to coverage_exemptions() in " + << "test_op_coverage.cpp with the reason."; + } + } +}; + +// Registered at static-init time; gtest runs environment TearDown after the last test. +const auto* const op_coverage_env = ::testing::AddGlobalTestEnvironment(new OpCoverageEnvironment()); + +} // namespace + +// Guard the guard: the coverage record must be non-empty and must contain ops the suite obviously +// converts. If SingleOpDecoder ever stops recording, the TearDown check above would pass vacuously +// for an empty op table and fail confusingly otherwise; this makes the wiring itself testable. +TEST(GGUFOpCoverage, RecordIsPopulated) { + // This test's own decoder construction guarantees at least one entry regardless of test order. + SingleOpBuilder() + .op("GGML_OP_ADD") + .input("a", ov::element::f32, {1}) + .output("out", ov::element::f32, {1}) + .decoder(); + EXPECT_NE(converted_op_types().count("GGML_OP_ADD"), 0u); +} + +// The op table itself must be non-degenerate: a build that dropped the registrations would make the +// coverage gate above pass trivially. +TEST(GGUFOpCoverage, OpTableIsNonEmpty) { + EXPECT_GT(ov::frontend::gguf::get_supported_ops().size(), 50u); +} diff --git a/src/frontends/gguf/tests/test_ops.cpp b/src/frontends/gguf/tests/test_ops.cpp index 2deae757bb0fea..e1408a95f513e7 100644 --- a/src/frontends/gguf/tests/test_ops.cpp +++ b/src/frontends/gguf/tests/test_ops.cpp @@ -61,13 +61,17 @@ INSTANTIATE_TEST_SUITE_P(GGUFOps, [](const ::testing::TestParamInfo& i) { return std::string(i.param.name); }); // ── Elementwise unary ops (single f32 input) ──────────────────────────────────── -// silu / gelu(tanh) / tanh / softplus share the same one-input graph and driver. +// Every registered GGML_UNARY_OP_* plus the elementwise GGML_OP_{LOG,SIN,COS} share the same +// one-input graph and driver, so they are parameterized over (op type, reference lambda). struct UnaryCase { const char* name; const char* op_type; std::function ref; float atol; + // Input values; empty means the default sign-spanning ramp below. Ops with a restricted + // domain (log) supply their own. + std::vector x{}; }; class GGUFUnaryElementwise : public ::testing::TestWithParam {}; @@ -80,7 +84,8 @@ TEST_P(GGUFUnaryElementwise, MatchesReference) { .output("out", ov::element::f32, {2, 4}) .build(); - std::vector x{-2, -1, -0.5f, 0, 0.5f, 1, 2, 3}; + std::vector x = c.x.empty() ? std::vector{-2, -1, -0.5f, 0, 0.5f, 1, 2, 3} : c.x; + ASSERT_EQ(x.size(), 8u) << "UnaryCase input must match the [2,4] graph shape"; auto out = run_on_cpu(model, {{"x", make_f32_tensor({2, 4}, x)}}); std::vector expected(x.size()); @@ -89,8 +94,10 @@ TEST_P(GGUFUnaryElementwise, MatchesReference) { expect_near(out, expected, c.atol); } -// ggml GELU is the tanh approximation, but the frontend maps GGML_UNARY_OP_GELU to v7::Gelu(TANH) -// which is close enough to the exact (erf) form to check against it at 1e-3. +// References are the scalar ggml kernels from ggml/src/ggml-cpu/vec.h, so a translator that picks +// a different-but-plausible formula for the same name is caught (GELU vs GELU_QUICK are distinct +// approximations, not interchangeable). GELU is checked against the exact erf form instead: the +// frontend maps it to v7::Gelu(TANH), which agrees with ggml's tanh kernel and with erf to 1e-3. // Softplus uses 1e-3 to cover ARM CPU fp16 execution (small outputs where fp16 spacing ~1e-3 // dominates); on x86 fp32 the exact reference still matches comfortably within that bound. INSTANTIATE_TEST_SUITE_P( @@ -102,19 +109,27 @@ INSTANTIATE_TEST_SUITE_P( "GGML_UNARY_OP_GELU", [](float x) { return 0.5f * x * (1.0f + std::erf(x / std::sqrt(2.0f))); }, 1e-3f}, - UnaryCase{"tanh", "GGML_UNARY_OP_TANH", [](float x) { return std::tanh(x); }, 1e-4f}, - UnaryCase{"relu", "GGML_UNARY_OP_RELU", [](float x) { return x > 0.0f ? x : 0.0f; }, 1e-4f}, - UnaryCase{"elu", "GGML_UNARY_OP_ELU", [](float x) { return x > 0.0f ? x : std::expm1(x); }, 1e-4f}, + // ggml_gelu_quick_f32: x*(1/(1+expf(-1.702f*x))) -- NOT the tanh GELU above. UnaryCase{"gelu_quick", "GGML_UNARY_OP_GELU_QUICK", - [](float x) { return x * (1.0f / (1.0f + std::exp(-1.702f * x))); }, + [](float x) { return x / (1.0f + std::exp(-1.702f * x)); }, 1e-4f}, + UnaryCase{"tanh", "GGML_UNARY_OP_TANH", [](float x) { return std::tanh(x); }, 1e-4f}, + UnaryCase{"relu", "GGML_UNARY_OP_RELU", [](float x) { return x > 0.0f ? x : 0.0f; }, 1e-4f}, + // ggml_vec_elu_f32: (x > 0) ? x : expm1f(x), i.e. ELU with alpha == 1. + UnaryCase{"elu", "GGML_UNARY_OP_ELU", [](float x) { return x > 0.0f ? x : std::expm1(x); }, 1e-4f}, UnaryCase{"sin", "GGML_OP_SIN", [](float x) { return std::sin(x); }, 1e-4f}, UnaryCase{"cos", "GGML_OP_COS", [](float x) { return std::cos(x); }, 1e-4f}, UnaryCase{"softplus", "GGML_UNARY_OP_SOFTPLUS", [](float x) { return std::log1p(std::exp(-std::abs(x))) + std::max(x, 0.0f); }, - 1e-3f}), + 1e-3f}, + // Log's domain is x > 0, so this case overrides the default ramp. + UnaryCase{"log", + "GGML_OP_LOG", + [](float x) { return std::log(x); }, + 1e-4f, + {0.25f, 0.5f, 1.0f, 2.0f, 3.0f, 10.0f, 100.0f, 1e-3f}}), [](const ::testing::TestParamInfo& i) { return std::string(i.param.name); }); // Log is only defined for x > 0, so it gets its own inputs rather than the shared range above. @@ -153,26 +168,51 @@ TEST(GGUFOps, GetDimensionsKeepsOutputPort) { EXPECT_EQ(shape_of->input_value(0).get_index(), 1u) << "get_dimensions measured the wrong port"; } -// ggml_top_k: indices of the k largest values along ne[0], ordered by descending value. -TEST(GGUFOps, TopK) { - auto model = SingleOpBuilder() - .op("GGML_OP_TOP_K") - .input("x", ov::element::f32, {1, 1, 2, 4}) - .output("out", ov::element::i32, {1, 1, 2, 2}) - .build(); +// ── Real-ggml reference data (test_data/*_ggml_{input,expected}.npy) ──────────── +// The parameterized cases above encode the ggml formula by hand in C++. These cases instead run +// the actual ggml kernel's output, captured offline from real ggml linked against +// libggml, over a [-6, 6] ramp -- so a misreading of the kernel cannot be baked into both sides. +// This is the check that distinguishes GELU_QUICK's sigmoid form from the tanh form: they differ +// by 2.2e-2 in the negative tail, far outside these tolerances. +// +// Tolerances are set by ggml's own arithmetic, not by OV's. ggml evaluates GELU and GELU_QUICK +// through an fp16 lookup table (GGML_GELU_FP16), which costs ~2e-3 / ~3.3e-3 against the exact +// fp32 form; SILU runs in fp32 in the reference build and needs only 1e-5. +struct GgmlRefCase { + const char* name; // also the test_data/_ggml_{input,expected}.npy stem prefix + const char* op_type; + float atol; +}; - std::vector x{4, 1, 3, 2, 10, 40, 20, 30}; - auto out = run_on_cpu(model, {{"x", make_f32_tensor({1, 1, 2, 4}, x)}}); +class GGUFUnaryVsGgml : public ::testing::TestWithParam {}; - ASSERT_EQ(out.get_element_type(), ov::element::i32); - ASSERT_EQ(out.get_size(), 4u); - const int32_t* a = out.data(); - // Row 0: 4,1,3,2 -> top2 are 4 (idx 0) then 3 (idx 2). Row 1: 10,40,20,30 -> 40 (1), 30 (3). - std::vector expected{0, 2, 1, 3}; - for (size_t i = 0; i < expected.size(); ++i) - EXPECT_EQ(a[i], expected[i]) << "mismatch at index " << i; +TEST_P(GGUFUnaryVsGgml, MatchesGgmlKernel) { + const GgmlRefCase c = GetParam(); + const auto x = load_npy(std::string(c.name) + "_ggml_input"); + const auto expected = load_npy(std::string(c.name) + "_ggml_expected"); + ASSERT_EQ(x.size(), expected.size()); + ASSERT_FALSE(x.empty()); + + // The captured reference is a [4, 32] ramp. + const ov::Shape shape{4, 32}; + ASSERT_EQ(ov::shape_size(shape), x.size()); + + auto model = SingleOpBuilder() + .op(c.op_type) + .input("x", ov::element::f32, shape) + .output("out", ov::element::f32, shape) + .build(); + auto out = run_on_cpu(model, {{"x", make_f32_tensor(shape, x)}}); + expect_near(out, expected, c.atol); } +INSTANTIATE_TEST_SUITE_P(GGUFOps, + GGUFUnaryVsGgml, + ::testing::Values(GgmlRefCase{"silu", "GGML_UNARY_OP_SILU", 1e-5f}, + GgmlRefCase{"gelu", "GGML_UNARY_OP_GELU", 2.5e-3f}, + GgmlRefCase{"gelu_quick", "GGML_UNARY_OP_GELU_QUICK", 4e-3f}), + [](const ::testing::TestParamInfo& i) { return std::string(i.param.name); }); + // Scale: out = in * scale + bias (scale/bias in op-params slots 0,1). TEST(GGUFOps, Scale) { const float scale = 2.5f; @@ -302,7 +342,7 @@ TEST(GGUFOps, SoftMaxAlibi) { const float m1 = std::pow(2.0f, -(max_bias / 2.0f) / n_head_log2); std::vector expected(x.size()); for (uint32_t h = 0; h < n_head; ++h) { - float slope = h < n_head_log2 ? std::pow(m0, static_cast(h + 1)) : std::pow(m1, static_cast(2 * (h - n_head_log2) + 1)); + float slope = h < n_head_log2 ? std::pow(m0, h + 1) : std::pow(m1, 2 * (h - n_head_log2) + 1); for (size_t t = 0; t < T; ++t) { float mx = -1e30f; std::vector z(Kd); @@ -562,6 +602,152 @@ TEST(GGUFOps, TransposePerm) { expect_near(out, expected); } +// Permute op_case 1: the plain head/token axis swap, perm {0,2,1,3}. +TEST(GGUFOps, PermuteCase1SwapsHeadAndTokenAxes) { + // [1, tok=2, heads=3, hs=2] -> [1, heads=3, tok=2, hs=2] + auto model = SingleOpBuilder() + .op("GGML_OP_PERMUTE") + .input("x", ov::element::f32, {1, 2, 3, 2}) + .output("out", ov::element::f32, {1, 3, 2, 2}) + .op_case(1) + .build(); + + std::vector x(12); + for (size_t i = 0; i < x.size(); ++i) + x[i] = static_cast(i); + auto out = run_on_cpu(model, {{"x", make_f32_tensor({1, 2, 3, 2}, x)}}); + + ASSERT_EQ(out.get_shape(), (ov::Shape{1, 3, 2, 2})); + // out[0,h,t,d] == x[0,t,h,d]; with x flat == index, x[0,t,h,d] = ((t*3)+h)*2+d. + std::vector expected; + for (int64_t h = 0; h < 3; ++h) + for (int64_t t = 0; t < 2; ++t) + for (int64_t d = 0; d < 2; ++d) + expected.push_back(static_cast((t * 3 + h) * 2 + d)); + expect_near(out, expected, 0.0f); +} + +// Permute op_case 4: reshape the flat projection to [n_seq, -1, n_heads, head_size] first, then +// apply the same axis swap. This is the Q/K projection path, where the head split and the permute +// are a single ggml op. +TEST(GGUFOps, PermuteCase4SplitsHeadsThenSwaps) { + // Flat [1, 1, tok=2, heads*hs=6] -> [1, heads=3, tok=2, hs=2] + auto model = SingleOpBuilder() + .op("GGML_OP_PERMUTE") + .input("x", ov::element::f32, {1, 1, 2, 6}) + .output("out", ov::element::f32, {1, 3, 2, 2}) + .op_case(4) + .build(); + + std::vector x(12); + for (size_t i = 0; i < x.size(); ++i) + x[i] = static_cast(i); + auto out = run_on_cpu(model, {{"x", make_f32_tensor({1, 1, 2, 6}, x)}}); + + ASSERT_EQ(out.get_shape(), (ov::Shape{1, 3, 2, 2})); + // The reshape splits the 6-wide row into 3 heads of 2, so out[0,h,t,d] == x_flat[t*6 + h*2 + d]. + std::vector expected; + for (int64_t h = 0; h < 3; ++h) + for (int64_t t = 0; t < 2; ++t) + for (int64_t d = 0; d < 2; ++d) + expected.push_back(static_cast(t * 6 + h * 2 + d)); + expect_near(out, expected, 0.0f); +} + +// Permute rejects op_cases it does not implement rather than silently emitting a wrong graph. +TEST(GGUFOps, PermuteUnsupportedCaseThrows) { + EXPECT_THROW(SingleOpBuilder() + .op("GGML_OP_PERMUTE") + .input("x", ov::element::f32, {1, 2, 3, 2}) + .output("out", ov::element::f32, {1, 3, 2, 2}) + .op_case(7) + .build(), + ov::Exception); +} + +// View with no op_case is a pure reinterpretation of the same buffer: a pass-through. +TEST(GGUFOps, ViewDefaultIsPassThrough) { + auto model = SingleOpBuilder() + .op("GGML_OP_VIEW") + .input("x", ov::element::f32, {1, 1, 2, 4}) + .output("out", ov::element::f32, {1, 1, 2, 4}) + .build(); + + std::vector x{1, 2, 3, 4, 5, 6, 7, 8}; + auto out = run_on_cpu(model, {{"x", make_f32_tensor({1, 1, 2, 4}, x)}}); + expect_near(out, x, 0.0f); +} + +// View op_case 3: the decoder resolves ggml's ne/nb/offset into a {axis, start, len} slice plus the +// view's own output layout. Here it selects columns [1, 3) of a [1,1,2,4] source. +TEST(GGUFOps, ViewCase3SlicesAndReshapes) { + auto model = SingleOpBuilder() + .op("GGML_OP_VIEW") + .input("x", ov::element::f32, {1, 1, 2, 4}) + .output("out", ov::element::f32, {1, 1, 2, 2}) + .op_case(3) + .attr("input_ggml_shape", ov::Shape{1, 1, 2, 4}) + .attr>("view_slice", {3, 1, 2}) + .attr>("view_reshape", {1, 1, 2, 2}) + .build(); + + std::vector x{1, 2, 3, 4, 5, 6, 7, 8}; + auto out = run_on_cpu(model, {{"x", make_f32_tensor({1, 1, 2, 4}, x)}}); + + ASSERT_EQ(out.get_shape(), (ov::Shape{1, 1, 2, 2})); + std::vector expected{2, 3, 6, 7}; + expect_near(out, expected, 0.0f); +} + +// View op_case 3 also has to restore the source's original ggml shape when the OV input arrives at +// a different rank (a preceding op already reshaped it), otherwise the slice lands on the wrong +// axis. Same slice as above, but the input is presented flat. +TEST(GGUFOps, ViewCase3RestoresGgmlShapeBeforeSlicing) { + auto model = SingleOpBuilder() + .op("GGML_OP_VIEW") + .input("x", ov::element::f32, {2, 4}) + .output("out", ov::element::f32, {1, 1, 2, 2}) + .op_case(3) + .attr("input_ggml_shape", ov::Shape{1, 1, 2, 4}) + .attr>("view_slice", {3, 1, 2}) + .attr>("view_reshape", {1, 1, 2, 2}) + .build(); + + std::vector x{1, 2, 3, 4, 5, 6, 7, 8}; + auto out = run_on_cpu(model, {{"x", make_f32_tensor({2, 4}, x)}}); + + ASSERT_EQ(out.get_shape(), (ov::Shape{1, 1, 2, 2})); + std::vector expected{2, 3, 6, 7}; + expect_near(out, expected, 0.0f); +} + +// TopK: indices of the k largest values per row, k taken from the output's last dim. +// +// ggml's own kernel deliberately swaps the first two result slots ("emphasize that the order is not +// important", ops.cpp), so only the SET of returned indices is contractual -- the test compares +// sorted index sets rather than positions. +TEST(GGUFOps, TopK) { + auto model = SingleOpBuilder() + .op("GGML_OP_TOP_K") + .input("x", ov::element::f32, {1, 1, 2, 5}) + .output("out", ov::element::i32, {1, 1, 2, 3}) + .build(); + + std::vector x{1, 9, 3, 7, 5, 50, 10, 40, 20, 30}; + auto out = run_on_cpu(model, {{"x", make_f32_tensor({1, 1, 2, 5}, x)}}); + + ASSERT_EQ(out.get_element_type(), ov::element::i32); + ASSERT_EQ(out.get_size(), 6u); + const int32_t* a = out.data(); + // Row 0: values 1,9,3,7,5 -> top 3 are 9,7,5 at indices 1,3,4. + // Row 1: values 50,10,40,20,30 -> top 3 are 50,40,30 at indices 0,2,4. + std::vector row0(a, a + 3), row1(a + 3, a + 6); + std::sort(row0.begin(), row0.end()); + std::sort(row1.begin(), row1.end()); + EXPECT_EQ(row0, (std::vector{1, 3, 4})); + EXPECT_EQ(row1, (std::vector{0, 2, 4})); +} + // Repeat: tile src to fill the output shape (integer multiples per axis). TEST(GGUFOps, Repeat) { auto model = SingleOpBuilder() @@ -947,6 +1133,74 @@ TEST(GGUFOps, ReshapeCase3) { expect_near(out, x, 0.0f); } +// RESHAPE op_cases 1 and 2 are the attention split/merge pair, and they must be LAYOUT-POLYMORPHIC: +// valid both for plain SDPA inference, which feeds a batch-major activation ([1, tokens, ..]), and +// after ov::pass::SDPAToPagedAttention, which moves the token count into dim 0 ([tokens, 1, ..]). +// Those two are the same buffer, so the op must copy the leading dim through rather than pin a +// literal 1 -- pinning silently rewrites a token-major activation into a batch-major one and the +// PagedAttention operands come out token-count-squared. +// +// Feeding the same values under both arrangements and requiring the same output element order is +// exactly that property, and it fails on a literal-1 reshape. +TEST(GGUFOps, ReshapeCase1SplitHeadsIsLayoutPolymorphic) { + const int64_t tokens = 2, heads = 2, head_size = 2; + const std::vector x{1, 2, 3, 4, 5, 6, 7, 8}; + + // Batch-major (what genai feeds a plain SDPA model): [1, 1, tokens, heads*head_size]. + auto sdpa = SingleOpBuilder() + .op("GGML_OP_RESHAPE") + .input("x", ov::element::f32, {1, 1, tokens, heads * head_size}) + .output("out", ov::element::f32, {1, tokens, heads, head_size}) + .op_case(1) + .build(); + auto sdpa_out = run_on_cpu(sdpa, {{"x", make_f32_tensor({1, 1, (size_t)tokens, (size_t)(heads * head_size)}, x)}}); + EXPECT_EQ(sdpa_out.get_shape(), (ov::Shape{1, (size_t)tokens, (size_t)heads, (size_t)head_size})); + + // Token-major (the layout SDPAToPagedAttention establishes): [tokens, 1, 1, heads*head_size]. + // The op must keep the tokens in dim 0 instead of moving them to dim 1. + auto pa = SingleOpBuilder() + .op("GGML_OP_RESHAPE") + .input("x", ov::element::f32, {tokens, 1, 1, heads * head_size}) + .output("out", ov::element::f32, {1, tokens, heads, head_size}) + .op_case(1) + .build(); + auto pa_out = run_on_cpu(pa, {{"x", make_f32_tensor({(size_t)tokens, 1, 1, (size_t)(heads * head_size)}, x)}}); + EXPECT_EQ(pa_out.get_shape(), (ov::Shape{(size_t)tokens, 1, (size_t)heads, (size_t)head_size})); + + // Same buffer in, same buffer out. + expect_near(sdpa_out, x, 0.0f); + expect_near(pa_out, x, 0.0f); +} + +TEST(GGUFOps, ReshapeCase2MergeHeadsIsLayoutPolymorphic) { + const int64_t tokens = 2, heads = 2, head_size = 2; + const std::vector x{1, 2, 3, 4, 5, 6, 7, 8}; + + // Non-stateful ggml keeps activations rank-4 throughout: [1, tokens, H, S] -> [1, 1, tokens, H*S]. + auto sdpa = SingleOpBuilder() + .op("GGML_OP_RESHAPE") + .input("x", ov::element::f32, {1, tokens, heads, head_size}) + .output("out", ov::element::f32, {1, 1, tokens, heads * head_size}) + .op_case(2) + .build(); + auto sdpa_out = + run_on_cpu(sdpa, {{"x", make_f32_tensor({1, (size_t)tokens, (size_t)heads, (size_t)head_size}, x)}}); + EXPECT_EQ(sdpa_out.get_shape(), (ov::Shape{1, 1, (size_t)tokens, (size_t)(heads * head_size)})); + + // Token-major: the tokens stay in dim 0. + auto pa = SingleOpBuilder() + .op("GGML_OP_RESHAPE") + .input("x", ov::element::f32, {tokens, 1, heads, head_size}) + .output("out", ov::element::f32, {1, 1, tokens, heads * head_size}) + .op_case(2) + .build(); + auto pa_out = run_on_cpu(pa, {{"x", make_f32_tensor({(size_t)tokens, 1, (size_t)heads, (size_t)head_size}, x)}}); + EXPECT_EQ(pa_out.get_shape(), (ov::Shape{(size_t)tokens, 1, 1, (size_t)(heads * head_size)})); + + expect_near(sdpa_out, x, 0.0f); + expect_near(pa_out, x, 0.0f); +} + // SET_ROWS into a flattened KV-cache row (row_size taken from the dst input, not the op output): // dst cache [1,1,ctx=3,row=2], data [1,1,n=2,row=2] written at indices {2,0}. TEST(GGUFOps, SetRowsFlattenedCache) { diff --git a/src/plugins/intel_cpu/src/transformations/cpu_opset/common/pass/widen_gather_matmul_weights.cpp b/src/plugins/intel_cpu/src/transformations/cpu_opset/common/pass/widen_gather_matmul_weights.cpp new file mode 100644 index 00000000000000..32778bc4b2debc --- /dev/null +++ b/src/plugins/intel_cpu/src/transformations/cpu_opset/common/pass/widen_gather_matmul_weights.cpp @@ -0,0 +1,165 @@ +// Copyright (C) 2018-2026 Intel Corporation +// SPDX-License-Identifier: Apache-2.0 +// + +#include "widen_gather_matmul_weights.hpp" + +#include +#include +#include +#include +#include + +#include "openvino/cc/pass/itt.hpp" +#include "openvino/core/graph_util.hpp" +#include "openvino/core/rt_info.hpp" +#include "openvino/core/type.hpp" +#include "openvino/core/type/element_iterator.hpp" +#include "openvino/op/constant.hpp" +#include "openvino/op/convert.hpp" +#include "openvino/op/multiply.hpp" +#include "openvino/op/reshape.hpp" +#include "openvino/op/subtract.hpp" +#include "openvino/op/transpose.hpp" +#include "openvino/pass/pattern/matcher.hpp" +#include "openvino/pass/pattern/op/pattern.hpp" +#include "openvino/pass/pattern/op/wrap_type.hpp" +#include "openvino/reference/convert.hpp" +#include "openvino/runtime/tensor.hpp" +#include "ov_ops/gather_matmul.hpp" + +namespace { + +struct Widening { + ov::element::Type_t from; + ov::element::Type_t to; +}; + +// Narrow types GatherMatmul has no executor for, paired with the narrowest type it does support +// that represents them losslessly. u2 holds [0..3] and u4 a nibble, so the values survive. +constexpr std::array kWidenings{{{ov::element::u2, ov::element::u4}}}; + +// Re-emit `src` with the same values stored as `to`. element::iterator handles the sub-byte +// (un)packing, so this is just a value copy between two integer types. +// One explicit branch per kWidenings entry: the iterator element types are template arguments. +std::shared_ptr widen_constant(const ov::op::v0::Constant& src, ov::element::Type to) { + ov::Tensor dst(to, src.get_shape()); + const size_t count = ov::shape_size(src.get_shape()); + if (src.get_element_type() == ov::element::u2 && to == ov::element::u4) { + ov::reference::convert(ov::element::iterator(static_cast(src.get_data_ptr())), + ov::element::iterator(static_cast(dst.data())), + count); + } else { + OPENVINO_THROW("WidenGatherMatmulWeights: no widening implemented for ", + src.get_element_type(), + " -> ", + to); + } + return std::make_shared(dst); +} + +// Is `out` consumed, through the rest of a CompressedWeightsBlock, as a GatherMatmul's weight +// input? Traversal is restricted to the block's own op types -- all of which preserve the element +// type -- so it cannot wander into the surrounding model. +bool feeds_gather_matmul_weights(const ov::Output& out) { + constexpr size_t kWeightsInput = 1; + std::vector> stack{out}; + std::unordered_set visited; + while (!stack.empty()) { + const auto cur = stack.back(); + stack.pop_back(); + for (const auto& target : cur.get_target_inputs()) { + auto* node = target.get_node(); + if (ov::is_type(node)) { + if (target.get_index() == kWeightsInput) { + return true; + } + continue; + } + if (!ov::is_type_any_of(node) || + !visited.insert(node).second) { + continue; + } + for (const auto& o : node->outputs()) { + stack.push_back(o); + } + } + } + return false; +} + +} // namespace + +ov::intel_cpu::WidenGatherMatmulWeights::WidenGatherMatmulWeights( + const std::vector& supported_weights_types) { + MATCHER_SCOPE(WidenGatherMatmulWeights); + + const auto supported = [&supported_weights_types](ov::element::Type t) { + return std::find(supported_weights_types.begin(), supported_weights_types.end(), t) != + supported_weights_types.end(); + }; + + // Only widenings this build actually needs: source unsupported, target supported. + std::vector widenings; + for (const auto& w : kWidenings) { + if (!supported(w.from) && supported(w.to)) { + widenings.push_back(w); + } + } + if (widenings.empty()) { + return; + } + + const auto is_widenable = [widenings](const ov::Output& out) { + return std::any_of(widenings.begin(), widenings.end(), [&](const Widening& w) { + return out.get_element_type() == w.from; + }); + }; + + // Match the head of the dequantization subgraph (Constant -> Convert), not the whole + // CompressedWeightsBlock: the block's shape between there and the GatherMatmul varies + // (Reshape / Transpose / extra Convert), and only the Constant's storage type changes here. + auto weights = ov::pass::pattern::wrap_type(is_widenable); + auto convert = ov::pass::pattern::wrap_type({weights}); + + ov::matcher_pass_callback callback = [OV_CAPTURE_CPY_AND_THIS](ov::pass::pattern::Matcher& m) { + const auto& pattern_map = m.get_pattern_value_map(); + auto weights_out = pattern_map.at(weights); + auto constant = ov::as_type_ptr(weights_out.get_node_shared_ptr()); + if (!constant) { + return false; + } + + // Widening costs real bytes, so do it only where it buys the compressed path: a weight + // input of a GatherMatmul. Walk forward from the Convert through the remaining + // CompressedWeightsBlock shapes only (Subtract / Multiply / Reshape / Transpose / Convert, + // all element-type-preserving), so the search stays inside the dequantization subgraph + // instead of escaping into the rest of the model. + if (!feeds_gather_matmul_weights(pattern_map.at(convert))) { + return false; + } + + ov::element::Type to; + for (const auto& w : kWidenings) { + if (constant->get_element_type() == w.from) { + to = w.to; + break; + } + } + if (to == ov::element::dynamic) { + return false; + } + + auto widened = widen_constant(*constant, to); + widened->set_friendly_name(constant->get_friendly_name()); + ov::copy_runtime_info(constant, widened); + ov::replace_node(constant, widened); + return true; + }; + + this->register_matcher(std::make_shared(convert, matcher_name), callback); +} diff --git a/src/plugins/intel_cpu/src/transformations/cpu_opset/common/pass/widen_gather_matmul_weights.hpp b/src/plugins/intel_cpu/src/transformations/cpu_opset/common/pass/widen_gather_matmul_weights.hpp new file mode 100644 index 00000000000000..018a905b1a7190 --- /dev/null +++ b/src/plugins/intel_cpu/src/transformations/cpu_opset/common/pass/widen_gather_matmul_weights.hpp @@ -0,0 +1,48 @@ +// Copyright (C) 2018-2026 Intel Corporation +// SPDX-License-Identifier: Apache-2.0 +// + +#pragma once + +#include + +#include "openvino/core/type/element_type.hpp" +#include "openvino/pass/matcher_pass.hpp" + +namespace ov::intel_cpu { + +// ============================================================================ +// WidenGatherMatmulWeights +// ============================================================================ +// +// Widens a GatherMatmul's compressed weight Constant to the narrowest element type the node +// actually supports, when its own type is not supported but can be represented losslessly by +// one that is. +// +// GatherMatmul accepts fewer compressed weight types than FullyConnected does (see +// GatherMatmul::getSupportedCompressedWeightsTypes) -- notably it has no u2 executor, while +// FullyConnected does. Without this pass a u2 weight tensor is simply not matched by +// ConvertGatherMatmulToGatherMatmulCompressed, so its Convert -> Subtract -> Multiply +// dequantization subgraph stays in the graph and constant folding materializes the weights in +// f32: a 16x expansion off a 2-bit type, which on a wide MoE model is many gigabytes. Paying +// 2x to reach a supported type is far cheaper than falling off the compressed path entirely. +// +// The widening is lossless (u2's [0..3] fits a nibble) and leaves the dequantization arithmetic +// untouched -- only the storage type of the weight Constant changes -- so the following +// compression pass matches and numerics are unaffected. Must run BEFORE +// ConvertGatherMatmulToGatherMatmulCompressed. +// +// This is a workaround for a missing executor, not a desirable state: when GatherMatmul gains +// native support for a type listed in kWidenings below, drop that entry. +class WidenGatherMatmulWeights : public ov::pass::MatcherPass { +public: + OPENVINO_MATCHER_PASS_RTTI("WidenGatherMatmulWeights"); + + // `supported_weights_types` is the node's own capability list, i.e. + // GatherMatmul::getSupportedCompressedWeightsTypes(). A widening is applied only if its + // source type is absent from that list and its target type is present, so the pass is a + // no-op on a build/architecture where the narrow type is already supported. + explicit WidenGatherMatmulWeights(const std::vector& supported_weights_types); +}; + +} // namespace ov::intel_cpu diff --git a/src/plugins/intel_cpu/tests/unit/transformations/widen_gather_matmul_weights_test.cpp b/src/plugins/intel_cpu/tests/unit/transformations/widen_gather_matmul_weights_test.cpp new file mode 100644 index 00000000000000..b90f9e846bef28 --- /dev/null +++ b/src/plugins/intel_cpu/tests/unit/transformations/widen_gather_matmul_weights_test.cpp @@ -0,0 +1,144 @@ +// Copyright (C) 2018-2026 Intel Corporation +// SPDX-License-Identifier: Apache-2.0 +// + +#include "transformations/cpu_opset/common/pass/widen_gather_matmul_weights.hpp" + +#include + +#include +#include + +#include "openvino/core/model.hpp" +#include "openvino/op/constant.hpp" +#include "openvino/op/convert.hpp" +#include "openvino/op/matmul.hpp" +#include "openvino/op/multiply.hpp" +#include "openvino/op/parameter.hpp" +#include "openvino/op/reshape.hpp" +#include "openvino/op/subtract.hpp" +#include "openvino/pass/manager.hpp" +#include "ov_ops/gather_matmul.hpp" + +using namespace ov::intel_cpu; + +namespace { + +// GatherMatmul's x86_64 capability list (see GatherMatmul::getSupportedCompressedWeightsTypes): +// no u2, hence the pass. +const std::vector kSupported{ov::element::u8, ov::element::i8, ov::element::u4, ov::element::i4}; + +constexpr size_t kExperts = 4; +constexpr size_t kOc = 8; +constexpr size_t kGroups = 2; +constexpr size_t kGroupSize = 16; +constexpr size_t kIc = kGroups * kGroupSize; + +std::vector weight_values() { + std::vector v(kExperts * kOc * kIc); + for (size_t i = 0; i < v.size(); ++i) { + v[i] = static_cast(i % 4); // covers the full u2 range + } + return v; +} + +// weights(`wt`, grouped) -> Convert -> Subtract(zp) -> Multiply(scale) -> Reshape: a +// CompressedWeightsBlock over per-expert weights. +std::shared_ptr make_weights_block(ov::element::Type wt, const std::vector& values) { + const ov::Shape grouped{kExperts, kOc, kGroups, kGroupSize}; + const ov::Shape per_group{kExperts, kOc, kGroups, 1}; + auto weights = std::make_shared(wt, grouped, values); + auto convert = std::make_shared(weights, ov::element::f32); + auto zp = ov::op::v0::Constant::create(ov::element::u8, per_group, {1}); + auto sub = + std::make_shared(convert, std::make_shared(zp, ov::element::f32)); + auto scale = ov::op::v0::Constant::create(ov::element::f32, per_group, {0.5f}); + auto mul = std::make_shared(sub, scale); + auto pattern = + ov::op::v0::Constant::create(ov::element::i32, ov::Shape{3}, std::vector{kExperts, kOc, kIc}); + return std::make_shared(mul, pattern, false); +} + +std::shared_ptr make_gather_matmul_model(ov::element::Type wt, const std::vector& values) { + auto input = std::make_shared(ov::element::f32, ov::PartialShape{2, 6, kIc}); + auto index = ov::op::v0::Constant::create(ov::element::i32, ov::Shape{6, 2}, {1}); + auto bgm = std::make_shared(input, make_weights_block(wt, values), index); + return std::make_shared(ov::OutputVector{bgm}, ov::ParameterVector{input}); +} + +void run_pass(const std::shared_ptr& model, const std::vector& supported) { + ov::pass::Manager manager; + manager.register_pass(supported); + manager.run_passes(model); +} + +// The weight Constant at the head of the decompression subgraph: the one feeding the Convert. +std::shared_ptr find_weights(const std::shared_ptr& model) { + std::shared_ptr found; + for (const auto& node : model->get_ordered_ops()) { + auto convert = ov::as_type_ptr(node); + if (!convert) { + continue; + } + auto c = ov::as_type_ptr(convert->get_input_node_shared_ptr(0)); + if (c && c->get_shape() == ov::Shape{kExperts, kOc, kGroups, kGroupSize}) { + EXPECT_EQ(found, nullptr) << "expected exactly one weight Constant"; + found = c; + } + } + return found; +} + +} // namespace + +// A u2 expert-weight Constant is re-emitted as u4, values preserved. Only the storage type +// changes: the rest of the decompression subgraph is untouched, so the compression pass that runs +// next can match it. +TEST(WidenGatherMatmulWeights, U2WeightsWidenedToU4) { + const auto values = weight_values(); + auto model = make_gather_matmul_model(ov::element::u2, values); + const size_t ops_before = model->get_ordered_ops().size(); + + run_pass(model, kSupported); + + auto weights = find_weights(model); + ASSERT_NE(weights, nullptr); + EXPECT_EQ(weights->get_element_type(), ov::element::u4); + EXPECT_EQ(weights->get_shape(), (ov::Shape{kExperts, kOc, kGroups, kGroupSize})); + EXPECT_EQ(weights->cast_vector(), values) << "widening must be lossless"; + EXPECT_EQ(model->get_ordered_ops().size(), ops_before) << "no node added or removed"; +} + +// u4 is natively supported: nothing to do. +TEST(WidenGatherMatmulWeights, SupportedTypeIsUntouched) { + auto model = make_gather_matmul_model(ov::element::u4, weight_values()); + run_pass(model, kSupported); + EXPECT_EQ(find_weights(model)->get_element_type(), ov::element::u4); +} + +// The pass is driven by the node's own capability list, so on a build where u2 is supported it is +// a no-op rather than a wasted 2x. +TEST(WidenGatherMatmulWeights, NoOpWhenNarrowTypeIsSupported) { + auto model = make_gather_matmul_model(ov::element::u2, weight_values()); + std::vector supported_with_u2{kSupported}; + supported_with_u2.push_back(ov::element::u2); + + run_pass(model, supported_with_u2); + + EXPECT_EQ(find_weights(model)->get_element_type(), ov::element::u2); +} + +// Widening costs 2x the weight bytes, so it must not touch weights that no GatherMatmul consumes: +// those reach FullyConnected, which supports u2 natively. +TEST(WidenGatherMatmulWeights, NonGatherMatmulConsumerIsUntouched) { + auto input = std::make_shared(ov::element::f32, ov::PartialShape{kExperts, 6, kIc}); + auto mm = std::make_shared(input, + make_weights_block(ov::element::u2, weight_values()), + false, + true); + auto model = std::make_shared(ov::OutputVector{mm}, ov::ParameterVector{input}); + + run_pass(model, kSupported); + + EXPECT_EQ(find_weights(model)->get_element_type(), ov::element::u2); +} diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index 6eccbd6bdcb7a6..b341fb62dee33a 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -8,5 +8,5 @@ add_subdirectory(samples_tests) add_subdirectory(e2e_tests) add_subdirectory(memory_tests) -install(FILES requirements_pytorch requirements_tensorflow requirements_onnx requirements_jax +install(FILES requirements_pytorch requirements_tensorflow requirements_onnx requirements_jax requirements_gguf DESTINATION tests COMPONENT tests EXCLUDE_FROM_ALL) diff --git a/tests/model_hub_tests/gguf/conftest.py b/tests/model_hub_tests/gguf/conftest.py new file mode 100644 index 00000000000000..70d8accf63b27e --- /dev/null +++ b/tests/model_hub_tests/gguf/conftest.py @@ -0,0 +1,12 @@ +# Copyright (C) 2018-2026 Intel Corporation +# SPDX-License-Identifier: Apache-2.0 + +import inspect + +from models_hub_common.utils import get_params + + +def pytest_generate_tests(metafunc): + test_gen_attrs_names = list(inspect.signature(get_params).parameters) + params = get_params() + metafunc.parametrize(test_gen_attrs_names, params, scope="function") diff --git a/tests/model_hub_tests/gguf/gguf_models_nightly b/tests/model_hub_tests/gguf/gguf_models_nightly new file mode 100644 index 00000000000000..3694dba5add745 --- /dev/null +++ b/tests/model_hub_tests/gguf/gguf_models_nightly @@ -0,0 +1,30 @@ +# List of larger models (roughly 7B-30B parameters) used to verify the GGUF frontend converts, +# compiles, and runs a single forward step correctly for each supported architecture that has +# no small (<=4B) checkpoint available. +# +# Format: arch,repo_id,filename[,mark,reason] +# arch - architecture name (matches src/frontends/gguf/docs/supported_models.md) +# repo_id - Hugging Face Hub repo hosting the .gguf file +# filename - the specific .gguf file to download from that repo +# mark - optional: "skip" or "xfail" +# reason - required if mark is set +# +# See src/frontends/gguf/docs/supported_models.md for the full architecture list and the +# checkpoints this project validates them against. +# +# Runner sizing: these convert + compile in-process, so the driving cost is the peak +# resident memory recorded in that document's memory table (the largest entry kept here, +# muse-glimmer, peaks around 37 GiB) -- hence the 16-core/64 GiB nightly runner. +olmoe,allenai/OLMoE-1B-7B-0924-Instruct-GGUF,olmoe-1b-7b-0924-instruct-q4_0.gguf +muse-glimmer,unsloth/Muse-Glimmer-30B-GGUF,Muse-Glimmer-30B-UD-Q4_K_XL.gguf +qwen35-bonsai,prism-ml/Ternary-Bonsai-27B-gguf,Ternary-Bonsai-27B-Q2_g64.gguf +ernie4_5-moe,bartowski/baidu_ERNIE-4.5-21B-A3B-PT-GGUF,baidu_ERNIE-4.5-21B-A3B-PT-Q4_K_M.gguf +bailingmoe2,bartowski/inclusionAI_Ling-mini-2.0-GGUF,inclusionAI_Ling-mini-2.0-Q4_K_M.gguf +mellum,bartowski/Mellum2-12B-A2.5B-Instruct-GGUF,Mellum2-12B-A2.5B-Instruct-Q4_K_M.gguf +deepseek2-ocr,aditya00196/DeepSeek-OCR-2-Q4_K_M.gguf,DeepSeek-OCR-2-Q4_K_M.gguf +gpt-oss,unsloth/gpt-oss-20b-GGUF,gpt-oss-20b-Q4_K_M.gguf,skip,"Needs ~120 GiB peak anon RAM to convert+compile (see the memory table in supported_models.md); the largest CI runner is 64 GiB" +hunyuan-moe,,,skip,"No small checkpoint publicly available (see supported_models.md)" +glm4moe,,,skip,"Smallest checkpoint (GLM-4.5-Air) is ~40 GiB, impractical even for nightly" +exaone-moe,,,skip,"Smallest checkpoint is ~9 GiB / 32B, not currently validated (see supported_models.md)" +minimax-m2,,,skip,"Smallest checkpoint is ~78 GiB, impractical even for nightly" +jais2,,,skip,"No small checkpoint publicly available (see supported_models.md)" diff --git a/tests/model_hub_tests/gguf/gguf_models_precommit b/tests/model_hub_tests/gguf/gguf_models_precommit new file mode 100644 index 00000000000000..540ce58288e4d6 --- /dev/null +++ b/tests/model_hub_tests/gguf/gguf_models_precommit @@ -0,0 +1,35 @@ +# List of small models (roughly <=4B parameters) used to verify the GGUF frontend converts, +# compiles, and runs a single forward step correctly for each supported architecture. +# +# Format: arch,repo_id,filename[,mark,reason] +# arch - architecture name (matches src/frontends/gguf/docs/supported_models.md) +# repo_id - Hugging Face Hub repo hosting the .gguf file +# filename - the specific .gguf file to download from that repo +# mark - optional: "skip" or "xfail" +# reason - required if mark is set +# +# See src/frontends/gguf/docs/supported_models.md for the full architecture list and the +# checkpoints this project validates them against; entries below prefer the smallest +# available quantization of the same (or an equivalently small) checkpoint. +# +# Runner sizing: these convert + compile in-process, so the driving cost is the peak +# resident memory recorded in that document's memory table (the largest entry here, +# gemma4, peaks around 12 GiB) -- hence the 8-core/32 GiB precommit runner. +llama,bartowski/Llama-3.2-1B-Instruct-GGUF,Llama-3.2-1B-Instruct-Q4_K_M.gguf +qwen2,Qwen/Qwen2.5-0.5B-Instruct-GGUF,qwen2.5-0.5b-instruct-q4_k_m.gguf +qwen3,Qwen/Qwen3-0.6B-GGUF,Qwen3-0.6B-Q8_0.gguf +phi3,bartowski/Phi-3-mini-4k-instruct-GGUF,Phi-3-mini-4k-instruct-Q4_K_M.gguf +minicpm,runfuture/MiniCPM-2B-dpo-q4km-gguf,MiniCPM-2B-dpo-q4km-gguf.gguf +hunyuan-dense,bartowski/tencent_Hunyuan-0.5B-Instruct-GGUF,tencent_Hunyuan-0.5B-Instruct-Q4_K_M.gguf +qwen3moe,mradermacher/Qwen3-0.9B-A0.6B-GGUF,Qwen3-0.9B-A0.6B.Q4_K_M.gguf +gemma,RichardErkhov/google_-_gemma-2b-it-gguf,gemma-2b-it.Q4_K_M.gguf +gemma2,bartowski/gemma-2-2b-it-GGUF,gemma-2-2b-it-Q4_K_M.gguf +gemma3,unsloth/gemma-3-1b-it-GGUF,gemma-3-1b-it-Q4_K_M.gguf +gemma4,unsloth/gemma-4-E4B-it-GGUF,gemma-4-E4B-it-Q4_K_M.gguf +llama-embed,sabafallah/llama-nemotron-embed-1b-v2-GGUF,llama-nemotron-embed-1b-v2-Q4_K_M.gguf +exaone4,LGAI-EXAONE/EXAONE-4.0-1.2B-GGUF,EXAONE-4.0-1.2B-Q4_K_M.gguf +plamo3,mmnga-o/plamo-3-nict-2b-base-gguf,plamo-3-nict-2b-base-Q4_K_M.gguf +smollm3,bartowski/HuggingFaceTB_SmolLM3-3B-GGUF,HuggingFaceTB_SmolLM3-3B-Q4_K_M.gguf +maincoder,mradermacher/Maincoder-1B-GGUF,Maincoder-1B.Q4_K_M.gguf +mistral3,mradermacher/Ministral-3b-instruct-GGUF,Ministral-3b-instruct.Q4_K_M.gguf +qwen35,ggml-org/Qwen3.5-0.8B-GGUF,Qwen3.5-0.8B-Q4_0.gguf diff --git a/tests/model_hub_tests/gguf/test_gguf.py b/tests/model_hub_tests/gguf/test_gguf.py new file mode 100644 index 00000000000000..9b7567bb7e1bd0 --- /dev/null +++ b/tests/model_hub_tests/gguf/test_gguf.py @@ -0,0 +1,56 @@ +# -*- coding: utf-8 -*- +# Copyright (C) 2018-2026 Intel Corporation +# SPDX-License-Identifier: Apache-2.0 +# +# Verifies the GGUF frontend against real .gguf checkpoints downloaded from the Hugging Face +# Hub, one per architecture listed in src/frontends/gguf/docs/supported_models.md. +# +# Each test downloads one .gguf file, converts it with core.read_model(), compiles it, and +# runs a single forward step (one new token, no prior context -- see utils.py for why that is +# enough to exercise the frontend's default stateless lowering without extra setup). The +# check is deliberately shallow: a finite, correctly shaped logits tensor. It does not build a +# tokenizer or run more than one step, since this suite is about frontend conversion/inference +# correctness, not generation quality or text coherence (which +# src/frontends/gguf/tests/compare_with_llama.py and the GenAI-based harness described in +# supported_models.md already cover for the architectures where that matters). +# +# gguf_models_precommit lists the small (<=~4B parameter) architectures, run on every commit. +# gguf_models_nightly lists the rest (up to ~30B), run nightly given their download size. + +import os + +import pytest + +from models_hub_common.constants import clean_hf_cache_dir, hf_cache_dir +from models_hub_common.utils import cleanup_dir +from utils import assert_valid_logits, parse_gguf_model_list, run_gguf_model + + +class TestGGUF: + def teardown_method(self): + if clean_hf_cache_dir: + cleanup_dir(hf_cache_dir) + + def run(self, arch, repo_id, filename, mark, reason, ie_device): + assert mark in (None, "skip", "xfail"), f"Unknown mark {mark!r} for {arch}" + if mark == "skip": + pytest.skip(reason) + if mark == "xfail": + pytest.xfail(reason) + + logits = run_gguf_model(repo_id, filename, device=ie_device) + assert_valid_logits(logits) + + @pytest.mark.parametrize( + "arch,repo_id,filename,mark,reason", + parse_gguf_model_list(os.path.join(os.path.dirname(__file__), "gguf_models_precommit"))) + @pytest.mark.precommit + def test_gguf_precommit(self, arch, repo_id, filename, mark, reason, ie_device): + self.run(arch, repo_id, filename, mark, reason, ie_device) + + @pytest.mark.parametrize( + "arch,repo_id,filename,mark,reason", + parse_gguf_model_list(os.path.join(os.path.dirname(__file__), "gguf_models_nightly"))) + @pytest.mark.nightly + def test_gguf_nightly(self, arch, repo_id, filename, mark, reason, ie_device): + self.run(arch, repo_id, filename, mark, reason, ie_device) diff --git a/tests/model_hub_tests/gguf/utils.py b/tests/model_hub_tests/gguf/utils.py new file mode 100644 index 00000000000000..b2d978af73846f --- /dev/null +++ b/tests/model_hub_tests/gguf/utils.py @@ -0,0 +1,159 @@ +# -*- coding: utf-8 -*- +# Copyright (C) 2018-2026 Intel Corporation +# SPDX-License-Identifier: Apache-2.0 + +import csv + +import numpy as np +import openvino as ov +from openvino.frontend import FrontEndManager + +from models_hub_common.constants import hf_cache_dir + + +def parse_gguf_model_list(file_name: str): + """Parse a gguf_models_* list file into (arch, repo_id, filename, mark, reason) tuples. + + Each non-comment, non-empty line is "arch,repo_id,filename[,mark,reason]" (mark/reason + may be present but empty). repo_id/filename are empty for a fully skipped entry that has + no runnable checkpoint at all (see gguf_models_nightly). + """ + models = [] + with open(file_name, "r") as f_in: + for row in csv.reader(f_in): + if not row or not row[0] or row[0].startswith("#"): + continue + row = [item.strip() for item in row] + row += [""] * (5 - len(row)) + arch, repo_id, filename, mark, reason = row[:5] + models.append((arch, repo_id, filename, mark or None, reason or None)) + return models + + +def gguf_hub_download(repo_id: str, filename: str) -> str: + """Download one .gguf file from the HF hub into the shared HF cache dir.""" + from huggingface_hub import hf_hub_download + + return hf_hub_download(repo_id=repo_id, filename=filename, cache_dir=hf_cache_dir) + + +def _kv_cache_head_count(compiled_model) -> int: + """Number of KV heads, read from any cache_k_* input port's static heads dim. + + All per-layer KV caches use the same [1, ctx, n_head_kv, head_size] layout (see + src/frontends/gguf/src/builder/blocks/attention.cpp), so any one of them gives the count + needed to size inp_kv_idx below. Falls back to 1 if the model has no KV cache at all + (e.g. a purely recurrent / linear-attention stack). + """ + for port in compiled_model.inputs: + name = next(iter(port.get_names()), "") + if name.startswith("cache_k_"): + heads_dim = port.get_partial_shape()[2] + if heads_dim.is_static: + return heads_dim.get_length() + return 1 + + +def build_single_token_inputs(compiled_model, token_id: int = 1, is_imrope: bool = False): + """Build a minimal, single-new-token input feed for the GGUF frontend's stateless IO + contract (inp_tokens / inp_pos / inp_out_ids / self_kq_mask[_swa] / token_len_per_seq / + inp_kv_idx), plus zero-filled tensors for every other input the model declares (per-layer + KV caches and, for hybrid/recurrent architectures such as qwen35, the fully-static conv / + delta state Parameters). + + This deliberately does not model real multi-token generation or attend to any real + prompt/tokenizer: with T=1 (one new token, no prior context) every KV cache write covers + the cache in full, so the frontend's default (caller-extension-free) SetRows lowering + -- a plain ScatterUpdate over the whole cache -- is well-defined without requiring the + caller to also register the MakeStateful extension. That keeps this check to exactly + what "verify basic correctness" needs: the model converts, compiles, and produces a + finite, correctly shaped output for one forward pass. + + is_imrope selects the interleaved M-RoPE position layout (qwen35 / qwen3vl), read from + the model's "gguf_is_imrope" rt_info by the caller (CompiledModel does not expose + rt_info, so this must be read from the ov.Model before compiling): such a model expects + inp_pos to carry 4 position sections per token instead of 1. + """ + n_head_kv = _kv_cache_head_count(compiled_model) + n_pos_sections = 4 if is_imrope else 1 + feed = {} + for port in compiled_model.inputs: + name = next(iter(port.get_names()), None) + assert name, "GGUF model input port has no name" + + if name == "inp_tokens": + feed[name] = ov.Tensor(np.array([[[[token_id]]]], dtype=np.int32)) + elif name == "inp_pos": + feed[name] = ov.Tensor(np.zeros((1, 1, 1, n_pos_sections), dtype=np.int32)) + elif name == "inp_out_ids": + feed[name] = ov.Tensor(np.array([[[[0]]]], dtype=np.int32)) + elif name in ("self_kq_mask", "self_kq_mask_swa"): + feed[name] = ov.Tensor(np.zeros((1, 1, 1, 1), dtype=np.float32)) + elif name == "token_len_per_seq": + feed[name] = ov.Tensor(np.array([1], dtype=np.int64)) + elif name == "beam_idx": + feed[name] = ov.Tensor(np.zeros((1,), dtype=np.int32)) + elif name == "inp_kv_idx": + # One write index per (new token, kv head): translate_set_rows flattens the new + # rows to [.., 1, tokens * n_head_kv, row_size] before the default ScatterUpdate + # lowering, so indices must cover that same flattened extent. + feed[name] = ov.Tensor(np.arange(n_head_kv, dtype=np.int32).reshape(1, 1, 1, n_head_kv)) + else: + # Any other input (a per-layer KV cache, or a recurrent/linear-attention state + # such as qwen35's Gated-DeltaNet conv window and delta matrix): zero-fill, + # resolving the one dynamic (token/context) axis, if any, to T=1. + partial_shape = port.get_partial_shape() + shape = [d.get_length() if d.is_static else 1 for d in partial_shape] + dtype = port.get_element_type().to_dtype() + feed[name] = ov.Tensor(np.zeros(shape, dtype=dtype)) + return feed + + +def convert_gguf_model(path: str) -> ov.Model: + """Convert a .gguf into an ov.Model through the GGUF frontend. + + The frontend is deliberately not auto-selectable (see is_hidden_frontend in + src/frontends/common/src/manager.cpp), so core.read_model(".gguf") does not resolve to it and + the frontend has to be asked for by name. Nothing else is needed: the frontend runs its own + normalization inside convert(), and the only step read_model would add on top, + update_v10_model, applies solely to legacy IR v10 models. + """ + fe = FrontEndManager().load_by_framework("gguf") + return fe.convert(fe.load(path)) + + +def run_gguf_model(repo_id: str, filename: str, device: str = "CPU"): + """Download, convert, compile, and run one forward step of a .gguf model. + + Returns the raw logits tensor as a numpy array. Raises on any conversion / compilation / + inference failure; the caller is expected to additionally sanity-check the result (shape, + finiteness) -- see assert_valid_logits below. + """ + path = gguf_hub_download(repo_id, filename) + core = ov.Core() + model = convert_gguf_model(path) + is_imrope = False + if "gguf_is_imrope" in model.get_rt_info(): + is_imrope = bool(model.get_rt_info(["gguf_is_imrope"]).get()) + compiled_model = core.compile_model(model, device) + request = compiled_model.create_infer_request() + feed = build_single_token_inputs(compiled_model, is_imrope=is_imrope) + outputs = request.infer(feed) + return next(iter(outputs.values())) + + +def assert_valid_logits(logits: np.ndarray): + """Basic correctness: a real (non-empty), finite tensor of logits over some vocabulary. + + Deliberately does not check the actual predicted token / text: this suite verifies the + frontend converts and runs each supported architecture, not that any given (quantized, + single-token, no-history) forward pass produces a meaningful continuation. + """ + assert logits.size > 0, "model produced an empty output tensor" + assert np.isfinite(logits).all(), "model output contains NaN/Inf" + vocab_size = logits.shape[-1] + assert vocab_size > 1, f"output's last dim ({vocab_size}) does not look like a vocab axis" + # argmax must be a valid vocabulary index -- mostly a sanity check on the shape/dtype + # plumbing above, since any finite tensor trivially satisfies this. + best = int(logits.reshape(-1, vocab_size)[-1].argmax()) + assert 0 <= best < vocab_size diff --git a/tests/requirements_gguf b/tests/requirements_gguf new file mode 100644 index 00000000000000..b40c5a7c188036 --- /dev/null +++ b/tests/requirements_gguf @@ -0,0 +1,11 @@ +# Requirements for the GGUF frontend model hub tests (tests/model_hub_tests/gguf). +# +# The tests only need numpy (to build input tensors), pytest, and huggingface_hub (to +# download the .gguf checkpoints). No tokenizer / transformers / framework dependency: +# the suite feeds synthetic token ids and checks conversion + a single forward step, +# not generated text. +numpy==1.26.4; python_version < "3.12" +numpy==2.2.1; python_version >= "3.12" +pytest==7.0.1 +pytest-html==4.2.0 +huggingface-hub==0.25.2