perf(gqh): port the multi-column matvec optimizations from the llama.cpp fork - #33
Open
DeanoC wants to merge 1 commit into
Open
perf(gqh): port the multi-column matvec optimizations from the llama.cpp fork#33DeanoC wants to merge 1 commit into
DeanoC wants to merge 1 commit into
Conversation
…cpp fork Ports the ncols>1 work from GeometricAGI/llama.cpp#1 into the vendored gqh.cu, verbatim (the two trees' gqh.cu were byte-identical before this). The defect it fixes: the generic <RUNG,8,1> instantiation re-read the activations per (row, superblock, column) -- 1024 B per warp per column against 137 B of wire. Weights stream once regardless of ncols, so N=3 should cost about N=1; instead each extra column cost ~+9 ms. Fixed with exact-width instantiations sharing one activation read, wave-uniform addressing, a shortened per-superblock dependency chain, and exact-width arms out to five columns. On the fork this took MTP n-max=2 decode from 33.4 to 50.3 tok/s. **In lucebox the gain is currently latent**: nothing here dispatches ncols 2..3. Greedy decode is ncols 1, chunked prefill is ncols 512 (GEMM), and the dflash/ddtree spec-decode verify is 16-23 wide, above the fused path's LUCE_MMVQ_MAX_NCOLS ceiling (default 3). It becomes live if lucebox gains an MTP/narrow-draft consumer, or with the cap change below. Greedy is unaffected and verified: hot_ms 33.56 ms, decode 28.5 tok/s, greedy sha 1269c41a... unchanged (ncols 1 dispatches the same arm as before). ctest -R gqh: 25/25 pass. Separately measured, not changed here: raising LUCE_MMVQ_MAX_NCOLS from its default 3 lets the fused kernel take the widths it now has exact arms for, and the fallback it replaces is pathological for GQH types -- cap=3: N=1 33.60 N=4 301.63 N=8 300.76 (ms/forward) cap=5: N=1 33.92 N=4 47.82 N=8 304.60 cap=8: N=1 33.69 N=4 47.94 N=8 112.85 N=4 is 6.3x faster at cap>=5. Note this also changes which kernel serves ncols 2..8, so batched-verify numerics move (the fused arm is bit-exact against the f32 reference; MMQ is not the same rounding) -- it needs its own frozen reference before shipping, which the ncols-1 greedy sha cannot provide. Left as an env override rather than a default change.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #32 — review that first; this branch adds one commit on top of it.
What
Ports the
ncols>1work from GeometricAGI/llama.cpp#1 into the vendoredgqh.cu, verbatim. The two trees'gqh.cuwere byte-identical before this, so it is a clean single-file transfer.The defect: the generic
<RUNG,8,1>instantiation re-read activations per (row, superblock, column) — 1024 B per warp per column against 137 B of wire, ~12 TB/s effective (L0-saturated). Weights stream once regardless ofncols, so N=3 should cost about N=1; instead each extra column cost ~+9 ms. Fixed with exact-width instantiations sharing one activation read, wave-uniform addressing, a shortened per-superblock dependency chain, and exact-width arms out to five columns.Honest scope: the gain is currently latent in lucebox
On the fork this took MTP n-max=2 decode from 33.4 → 50.3 tok/s. Nothing in lucebox dispatches ncols 2..3 today:
DFLASH27B_DRAFT_BLOCK_SIZE16, ddtree budget 22) — above the fused path'sLUCE_MMVQ_MAX_NCOLSceiling (default 3)So this lands as dead-but-correct code until lucebox gains an MTP or narrow-draft consumer, or the cap changes (below). I'm proposing it anyway because the trees were identical and keeping them in sync is cheaper than re-deriving this later — but it should be reviewed as a sync, not as a speedup.
Verification
Greedy is unaffected, as expected since ncols 1 dispatches the same arm as before:
hot_ms1269c41a…1269c41a…(unchanged)ctest -R gqh: 25/25 pass (backend + cpu-decode across gqh3/gqh2_h/gqh2_c/gqh4 × 1x256/4x512/64x1024, plus the header parser).A separate, larger finding — the ncols cap
Measured, not changed in this PR. Raising
LUCE_MMVQ_MAX_NCOLSfrom its default 3 lets the fused kernel take widths it now has exact arms for, and the fallback it replaces is pathological for GQH types:N=4 is 6.3× faster at cap ≥ 5. Worth noting this corrects an earlier claim of mine: the "N=4 = 304 ms cliff" was previously written off as a
--profile-scalingrig artifact. It is not — it is the fused-path cap falling back to a dequant→GEMM chain.I did not change the default because raising it also changes which kernel serves ncols 2..8, so batched-verify numerics move. The fused arm is bit-exact against the f32 reference and MMQ is not the same rounding, so this is arguably an accuracy improvement — but it needs its own frozen reference to demonstrate that, and the ncols-1 greedy sha structurally cannot provide one. Happy to do that as a follow-up if you want the default moved.
Measured on gfx1201 (R9700) only.