Skip to content

[GGUF FE] Native .gguf graph builder - #37421

Draft
mvafin wants to merge 3 commits into
openvinotoolkit:masterfrom
mvafin:mvafin/gguf/builder-and-moe
Draft

[GGUF FE] Native .gguf graph builder#37421
mvafin wants to merge 3 commits into
openvinotoolkit:masterfrom
mvafin:mvafin/gguf/builder-and-moe

Conversation

@mvafin

@mvafin mvafin commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

Details:

  • Part 2 of 2. Adds the native .gguf graph builder: instead of a live GgufDecoder handed over by llama.cpp, read the container directly and build the transformer graph per architecture, emitting nodes in the GGML op vocabulary so both ingest paths share the same op translators. No llama.cpp dependency.
  • One generic dense-transformer builder covers the llama family, with per-architecture differences derived from the file itself (q/k norms, qkv biases, rope frequency factors, scalar scales), so most new architectures of that family are just a name in the supported list. MoE routing, muse-glimmer and qwen35's hybrid Gated-DeltaNet stack need real code and have it.
  • Also here, because none of it exists until a model can be read from a file:
    • AdaptToGenAI and the tokenizer metadata attached to rt_info, so OpenVINO GenAI can drive the model and build a tokenizer without reopening the file. Neither is reachable from the llama.cpp integration, which owns its own tokenizer and applies its own stateful transformation.
    • MakeStateful's recurrent-state rewrite, for the linear-attention states qwen35 carries. Unlike a KV cache these have no token axis and no SetRows write, so the pairing arrives from the decoder via rt_info.
    • Q4_K integer zero-point: the CPU compressed-FullyConnected fast path is ~2x slower with a fractional f16 zp, which dominates prefill.
    • Per-architecture conversion tests over header-only fixtures generated in CI from llama.cpp, plus a model_hub suite that downloads real checkpoints.
  • The frontend stays out of model auto-detection, so core.read_model(".gguf") does not resolve to it; consumers either link it directly or ask for it by name via load_by_framework("gguf").
  • Validated on all 25 checkpoints in tests/model_hub_tests/gguf: 25/25 convert, and generation through GenAI matches the per-architecture expectations recorded in docs/supported_models.md. ov_gguf_frontend_tests: 138/138.

Important

Stacked on #37435 (part 1). GitHub only allows a base branch that lives in this repo, so this PR still targets master and its diff therefore includes #37435's commit. Review only the second commit, [GGUF FE] Native .gguf graph builder. It will shrink to that automatically once #37435 merges.

Note

The CPU-side WidenGatherMatmulWeights transformation that MoE expert weights need is intentionally not in this PR and will follow separately; MoE models therefore need that change to compile.

Tickets:

AI Assistance:

  • AI assistance used: yes
  • AI wrote the code and tests. Human-validated by building and running real models rather than trusting the suite: all 25 checkpoints convert, generation was compared against llama.cpp per architecture, and the prefill/zero-point changes were measured end to end.

@github-actions github-actions Bot added category: Core OpenVINO Core (aka ngraph) category: CPU OpenVINO CPU plugin category: build OpenVINO cmake script / infra category: CI OpenVINO public CI category: docs OpenVINO documentation category: TF FE OpenVINO TensorFlow FrontEnd github_actions Pull requests that update GitHub Actions code category: PyTorch FE OpenVINO PyTorch Frontend no-match-files category: JAX FE OpenVINO JAX FrontEnd labels Aug 13, 2026
@mvafin
mvafin force-pushed the mvafin/gguf/builder-and-moe branch 4 times, most recently from 6b69da6 to a2cc31b Compare August 13, 2026 21:11
@mvafin
mvafin force-pushed the mvafin/gguf/builder-and-moe branch from a2cc31b to c61e617 Compare August 13, 2026 22:31
@mvafin mvafin changed the title [GGUF FE] Native .gguf builder, MoE support and model_hub_tests coverage [GGUF FE] Native .gguf graph builder Aug 13, 2026
@mvafin
mvafin force-pushed the mvafin/gguf/builder-and-moe branch 2 times, most recently from b2adc14 to 80e761d Compare August 13, 2026 22:55
mvafin and others added 3 commits August 14, 2026 01:12
…cture

Groundwork on the existing frontend, all of it reachable through the
GgmlOvDecoder path that master already ships. The native .gguf builder is a
separate change and does not appear here.

MakeStateful. The frontend always converts to a stateless graph -- every KV
cache an explicit Parameter/Result pair, as optimum-intel exports -- and being
stateful is the consumer's choice, registered as a DecoderTransformationExtension
so it runs ahead of the built-in stateless lowering. The pass also takes over
beam_idx: it is a beam-search index into an OpenVINO state with no ggml
counterpart, so declaring it in a decoder would leave a consumer-less input on
the stateless graph.

Op translators. Keep the output port when handing a value between translators
(taking .get_node_shared_ptr() silently resolved to output 0, which throws for
multi-output ops such as TopK); keep the static head layout in permute op_case 4;
drop the builder-only op_case numbering from RESHAPE and VIEW; give each
attention Transpose its own order constant; make the graph valid under both the
SDPA and PagedAttention layouts by deriving the leading dims rather than pinning
them; share the TopK-indices construction between ARGSORT and TOP_K, and let
TOP_K tolerate a dynamic k instead of throwing.

Quantization. Support the Q2_0 (ternary) type used by the Bonsai family.

Decoder interface. Drop get_model_weights, which nothing calls.

Tests. Add an op-coverage gate so a newly registered op cannot ship without a
conversion test, and check the activation translators against captured output
from real ggml rather than a numpy reimplementation of the formula -- a numpy
oracle can only confirm the formula the author already guessed, which is how the
GELU_QUICK error survived.

CI. Add a GGUF_FE component so frontend changes scope their own jobs.

ov_gguf_frontend_tests: 137/137.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add the second ingest path: instead of a live GgufDecoder handed over by
llama.cpp, read the .gguf container directly and build the transformer graph
per-architecture, emitting nodes in the GGML op vocabulary so both paths share
the same op translators. No llama.cpp dependency.

One generic dense-transformer builder covers the llama family, with the
per-architecture differences derived from the file itself (q/k norms, qkv
biases, rope frequency factors, scalar scales), so most new architectures of
that family are just a name in the supported list. MoE routing, muse-glimmer
and qwen35's hybrid Gated-DeltaNet stack need real code and have it.

Also here, because they only exist once a model can be read from a file:

  * AdaptToGenAI, which rewrites the llama.cpp-style IO into the OpenVINO GenAI
    LLMPipeline contract, and the tokenizer metadata the frontend attaches to
    rt_info so GenAI can build a tokenizer without reopening the file. Neither
    is reachable from the llama.cpp integration, which owns its own tokenizer
    and applies its own stateful transformation.
  * MakeStateful's recurrent-state rewrite, for the linear-attention states
    qwen35 carries. Unlike a KV cache these have no token axis and no SetRows
    write, so the pairing arrives from the decoder via rt_info.
  * The Q4_K integer zero-point: the CPU compressed-FullyConnected fast path is
    ~2x slower with a fractional f16 zp, which dominates prefill.
  * Per-architecture conversion tests over header-only fixtures generated in CI
    from llama.cpp, and a model_hub suite that downloads real checkpoints.

The frontend stays out of model auto-detection, so core.read_model(".gguf")
does not resolve to it; consumers either link it directly or ask for it by name
via load_by_framework("gguf").

Validated on all 25 checkpoints in tests/model_hub_tests/gguf: 25/25 convert,
and generation through GenAI matches the per-architecture expectations recorded
in docs/supported_models.md. ov_gguf_frontend_tests: 138/138.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…acy debugging

The frontend docs already cover architecture bring-up, accuracy debugging and the
testing architecture, but nothing routes an agent to them, so that knowledge is
re-derived from sources on every session.

Add three thin skills that point at the existing documents:

  - ov-gguf-enable-op        -> docs/how_to_add_op.md (new)
  - ov-gguf-add-architecture -> docs/adding_an_architecture.md, supported_models.md
  - ov-gguf-debug-accuracy   -> docs/debugging_accuracy.md

Op enablement was the one procedural gap with no document, so add
docs/how_to_add_op.md for it. It covers only the procedure and defers the concepts
to frontend_design.md: the file checklist including the test CMake source list that
is not globbed, the NodeContext accessors, the op-coverage gate in
test_op_coverage.cpp and the fact that a narrowing --gtest_filter silences it, the
rule that non-trivial reference values come from a ggml oracle rather than
hand-derived math, and the build flag the target needs.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

category: build OpenVINO cmake script / infra category: CI OpenVINO public CI category: Core OpenVINO Core (aka ngraph) category: CPU OpenVINO CPU plugin category: docs OpenVINO documentation category: JAX FE OpenVINO JAX FrontEnd category: PyTorch FE OpenVINO PyTorch Frontend category: TF FE OpenVINO TensorFlow FrontEnd github_actions Pull requests that update GitHub Actions code no-match-files

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant