[GGUF FE] Native .gguf graph builder - #37421
Draft
mvafin wants to merge 3 commits into
Draft
Conversation
4 tasks
mvafin
force-pushed
the
mvafin/gguf/builder-and-moe
branch
4 times, most recently
from
August 13, 2026 21:11
6b69da6 to
a2cc31b
Compare
mvafin
force-pushed
the
mvafin/gguf/builder-and-moe
branch
from
August 13, 2026 22:31
a2cc31b to
c61e617
Compare
mvafin
force-pushed
the
mvafin/gguf/builder-and-moe
branch
2 times, most recently
from
August 13, 2026 22:55
b2adc14 to
80e761d
Compare
…cture Groundwork on the existing frontend, all of it reachable through the GgmlOvDecoder path that master already ships. The native .gguf builder is a separate change and does not appear here. MakeStateful. The frontend always converts to a stateless graph -- every KV cache an explicit Parameter/Result pair, as optimum-intel exports -- and being stateful is the consumer's choice, registered as a DecoderTransformationExtension so it runs ahead of the built-in stateless lowering. The pass also takes over beam_idx: it is a beam-search index into an OpenVINO state with no ggml counterpart, so declaring it in a decoder would leave a consumer-less input on the stateless graph. Op translators. Keep the output port when handing a value between translators (taking .get_node_shared_ptr() silently resolved to output 0, which throws for multi-output ops such as TopK); keep the static head layout in permute op_case 4; drop the builder-only op_case numbering from RESHAPE and VIEW; give each attention Transpose its own order constant; make the graph valid under both the SDPA and PagedAttention layouts by deriving the leading dims rather than pinning them; share the TopK-indices construction between ARGSORT and TOP_K, and let TOP_K tolerate a dynamic k instead of throwing. Quantization. Support the Q2_0 (ternary) type used by the Bonsai family. Decoder interface. Drop get_model_weights, which nothing calls. Tests. Add an op-coverage gate so a newly registered op cannot ship without a conversion test, and check the activation translators against captured output from real ggml rather than a numpy reimplementation of the formula -- a numpy oracle can only confirm the formula the author already guessed, which is how the GELU_QUICK error survived. CI. Add a GGUF_FE component so frontend changes scope their own jobs. ov_gguf_frontend_tests: 137/137. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add the second ingest path: instead of a live GgufDecoder handed over by
llama.cpp, read the .gguf container directly and build the transformer graph
per-architecture, emitting nodes in the GGML op vocabulary so both paths share
the same op translators. No llama.cpp dependency.
One generic dense-transformer builder covers the llama family, with the
per-architecture differences derived from the file itself (q/k norms, qkv
biases, rope frequency factors, scalar scales), so most new architectures of
that family are just a name in the supported list. MoE routing, muse-glimmer
and qwen35's hybrid Gated-DeltaNet stack need real code and have it.
Also here, because they only exist once a model can be read from a file:
* AdaptToGenAI, which rewrites the llama.cpp-style IO into the OpenVINO GenAI
LLMPipeline contract, and the tokenizer metadata the frontend attaches to
rt_info so GenAI can build a tokenizer without reopening the file. Neither
is reachable from the llama.cpp integration, which owns its own tokenizer
and applies its own stateful transformation.
* MakeStateful's recurrent-state rewrite, for the linear-attention states
qwen35 carries. Unlike a KV cache these have no token axis and no SetRows
write, so the pairing arrives from the decoder via rt_info.
* The Q4_K integer zero-point: the CPU compressed-FullyConnected fast path is
~2x slower with a fractional f16 zp, which dominates prefill.
* Per-architecture conversion tests over header-only fixtures generated in CI
from llama.cpp, and a model_hub suite that downloads real checkpoints.
The frontend stays out of model auto-detection, so core.read_model(".gguf")
does not resolve to it; consumers either link it directly or ask for it by name
via load_by_framework("gguf").
Validated on all 25 checkpoints in tests/model_hub_tests/gguf: 25/25 convert,
and generation through GenAI matches the per-architecture expectations recorded
in docs/supported_models.md. ov_gguf_frontend_tests: 138/138.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…acy debugging The frontend docs already cover architecture bring-up, accuracy debugging and the testing architecture, but nothing routes an agent to them, so that knowledge is re-derived from sources on every session. Add three thin skills that point at the existing documents: - ov-gguf-enable-op -> docs/how_to_add_op.md (new) - ov-gguf-add-architecture -> docs/adding_an_architecture.md, supported_models.md - ov-gguf-debug-accuracy -> docs/debugging_accuracy.md Op enablement was the one procedural gap with no document, so add docs/how_to_add_op.md for it. It covers only the procedure and defers the concepts to frontend_design.md: the file checklist including the test CMake source list that is not globbed, the NodeContext accessors, the op-coverage gate in test_op_coverage.cpp and the fact that a narrowing --gtest_filter silences it, the rule that non-trivial reference values come from a ggml oracle rather than hand-derived math, and the build flag the target needs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Details:
.ggufgraph builder: instead of a liveGgufDecoderhanded over by llama.cpp, read the container directly and build the transformer graph per architecture, emitting nodes in the GGML op vocabulary so both ingest paths share the same op translators. No llama.cpp dependency.muse-glimmerandqwen35's hybrid Gated-DeltaNet stack need real code and have it.AdaptToGenAIand the tokenizer metadata attached tort_info, so OpenVINO GenAI can drive the model and build a tokenizer without reopening the file. Neither is reachable from the llama.cpp integration, which owns its own tokenizer and applies its own stateful transformation.MakeStateful's recurrent-state rewrite, for the linear-attention statesqwen35carries. Unlike a KV cache these have no token axis and noSetRowswrite, so the pairing arrives from the decoder viart_info.FullyConnectedfast path is ~2x slower with a fractional f16 zp, which dominates prefill.model_hubsuite that downloads real checkpoints.core.read_model(".gguf")does not resolve to it; consumers either link it directly or ask for it by name viaload_by_framework("gguf").tests/model_hub_tests/gguf: 25/25 convert, and generation through GenAI matches the per-architecture expectations recorded indocs/supported_models.md.ov_gguf_frontend_tests: 138/138.Important
Stacked on #37435 (part 1). GitHub only allows a base branch that lives in this repo, so this PR still targets
masterand its diff therefore includes #37435's commit. Review only the second commit,[GGUF FE] Native .gguf graph builder. It will shrink to that automatically once #37435 merges.Note
The CPU-side
WidenGatherMatmulWeightstransformation that MoE expert weights need is intentionally not in this PR and will follow separately; MoE models therefore need that change to compile.Tickets:
AI Assistance: