feat(sdk): add MTP speculative-decoding coverage to bench matrix - #1348
Merged
Conversation
Wire the typical MTP user invocation (draft-mtp / gemma-4-26B target + RachidAR assistant draft / --draft-tokens 3) into the QDC bench matrix, gated to SC8480XP, and surface draft acceptance in the per-cell JSON so the aggregate step can pair each spec cell with its no-spec baseline and show accept% + decode-tps uplift in a standalone MTP table. Random-token prefill would collapse draft acceptance to ~0% on any real target/draft pair, so spec rows are forced through --prompt-file with a pre-existing per-ctx fixture (plus a new sample_prompt_8192.txt to cover the 8k ctx the MTP row exercises). Signed-off-by: Mengsheng Wu <mengshengwu@outlook.com>
matrix.* is not addressable in a job-level if:, so GHA rejected the inline spec-device gate. Move the same logic into load-models so the bench matrix drops spec×non-SC8480XP up front and catalog-only rows never reach the strategy matrix at all. Signed-off-by: Mengsheng Wu <mengshengwu@outlook.com>
When --model-name filters the model list to a single spec row, the draft entry it references gets dropped, so _resolve_draft_model_id fails. Pull the referenced draft back in from the full catalog after filtering. Signed-off-by: Mengsheng Wu <mengshengwu@outlook.com>
…-rate JSON Without model-manager resolution the plugin gets a raw id like org/repo:Q4_0 for --draft-model, fails to open it as a file, and silently falls back to non-speculative decode — draft_n_total stays 0 and the MTP bench measures nothing spec-specific. Also drop the accept-rate additions to per-cell JSON + the MTP table's Accept% column: the bench only needs to prove the spec code path runs at random-token throughput, matching the plain llama_cpp bucket, so the prompt-file requirement + accept-rate reporting are unnecessary. Delete sample_prompt_8192.txt (spec cells now share random-ids prefill). Signed-off-by: Mengsheng Wu <mengshengwu@outlook.com>
Signed-off-by: Mengsheng Wu <mengshengwu@outlook.com>
Mengsheng Wu (mengshengwu)
changed the title
Diagnostic aid: when download_cells returns empty, pull .log/.stdout/.txt members from the QDC log archive and print them so the failure cause (usually a device-side stderr) is visible in the GH Actions log instead of being locked inside QDC's archive. Signed-off-by: Mengsheng Wu <mengshengwu@outlook.com>
Two issues found from the diag dump: 1. Spec-decoding accept-rate is fine (~70%) but llama.cpp fails to alloc a KV memory slot for the 4-token (target+3 draft) batch on the 2nd measured run, and bench.c exit(1)s on any generate failure -- so 0 cell JSONs were written for the entire spec matrix. Work around by forcing '-r 1 --no-warmup' on spec invocations; single measured run avoids the KV re-use path entirely. 2. The per-row ctx override in main() checked 'len(models) == 1', which silently disabled the 8192 sweep the moment resolve_via_mm pulled the draft-dep entry into 'models'. Filter to entries with non-empty devices before the count check. Signed-off-by: Mengsheng Wu <mengshengwu@outlook.com>
Bench defaults pp+tg = ctx exactly, but spec-decoding needs draft_tokens+1 extra KV slots on the last decode step (target + N drafted tokens), or llama.cpp trips 'decode: failed to find a memory slot for batch of size N+1' and bench exits with no JSON. Trim tg by that margin for spec cells; ctx=512 stays a 384-prompt cell but decodes 124 tokens instead of 128. Signed-off-by: Mengsheng Wu <mengshengwu@outlook.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
gemma-4-26B-A4B-it-mtprow tosdk/benchmark/qdc/bench-models.jsonthat dogfoods the typical MTP invocation (--spec-type draft-mtp --draft-model RachidAR/... --draft-tokens 3). The paired draft is a sibling catalog-only entry (devices: []) soaggregatecan resolve its download URL without dispatching its own bench. Spec × non-SC8480XP combos are dropped via a dynamicstrategy.matrix.excludecomputed inload-models— spec targets only fit on Snapdragon X2 Elite for now.--draft-modelnow resolves through the model-manager (resolve_draft_via_mm). Without this the plugin got a raworg/repo:Q4_0id, failed to open it as a file, and silently fell back to non-speculative decode.draft_tokens+1KV headroom by trimming-nfor the bench call. Bench defaultspp+tg = ctx; without headroom, the last decode step'sbatch of size N+1(target + drafted tokens) fails to find a KV memory slot and the whole bench call exits with no JSON.-p N) with the plain llama_cpp bucket inrun_windows.ps1. Bench only needs to measure mechanical decode throughput with the spec path enabled; realistic draft acceptance isn't the goal, so no--prompt-filefixture and no per-run accept-rate accounting.render()emits a standalone## MTP (speculative decoding)table (target | draft | device | ctx | test | decode (mtp) | decode (baseline) | uplift). Main perf table filters out spec cells viaparams.spec_typeso the two views don't mix.ctxoverride inbench-models.jsonlets the MTP row sweep[512, 1024, 4096, 8192]while everything else keeps the existing 3-ctx default.download_cellsreturns empty, dump the QDC.log/.stdout/.txtmembers so the on-device stderr is visible in the GH Actions log instead of being locked inside QDC's archive.Test plan
Geniex Bench --device SC8480XP --model gemma-4-26B-A4B-it-mtp,gemma-4-26B-A4B-itrenders a paired MTP row on the standalone table with uplift on NPU (~1.2-1.5×) and no MTP row on the main perf table — verified on run 31829193361.org/repo:Q4_0no longer reaches the plugin) — verified in the diag-dump output of run 31820543255 where the resolved path is a local.gguf.load-models.excludes(no queued runner slot for it) — visible in matrix expansion of run 31810545986.