fix(ds4): surface the DSpark AR fallback and the fused-verify preconditions - #608
Open
geometric[bot] wants to merge 2 commits into
Open
fix(ds4): surface the DSpark AR fallback and the fused-verify preconditions#608geometric[bot] wants to merge 2 commits into
geometric[bot] wants to merge 2 commits into
Conversation
…itions A DSpark server prints 'spec-decode ENABLED' at startup and then decodes pure AR for any request the greedy-only verifier cannot serve, saying nothing. The sampler case is a live trap in this tree: when a request OMITS temperature the HTTP layer falls back to the model card's sampling defaults, and share/model_cards/deepseek-v4-flash-0731-rocmfpx.json sets temperature 1.0 — so a temperature-less benchmark measures autoregressive decode while the log still claims speculation. Measured, same binary and flags, same prompt: no temperature field gives 290 tokens in 289 steps (one step per token, i.e. pure AR); "temperature": 0.0 gives 268 tokens at accept_rate 0.69. This cost a day of chasing a spec-decode 'regression' between two builds that differed only in whether they shipped that card. The decode gate now logs one line naming which condition routed the request to AR — force_ar / budget stop tokens / sampling — with the effective sampler values. The fused-verify gate reports every precondition, including verify_hooks, out_logits and expert_runtime; the previous version gated its own warning on two of the pointers whose absence it was meant to report, so a null there stayed invisible. Both fire once. The fused-verify one has never fired on gfx1151 — it is preventive, and the comment says so rather than implying it caught something.
Three things a DSpark measurement needs and DS4.md did not say. The fallback conditions and the tell in the log (decode tokens=N steps=N-1 instead of [ds4-spec] gen=... steps=...), so 'spec decode is broken' can be checked before it is reported. The acceptance envelope, because speculative throughput is acceptance-bound and acceptance is workload-bound: on a gfx1151 host, code/math prompts (accept 0.83-0.90, ~2.4 committed tokens per verify step) measured 24-27 tok/s where open-ended prose (accept 0.64-0.69, ~1.1 committed) measured 16-18 tok/s from the SAME server under the same flags. A spec tok/s figure is not comparable across prompt mixes. And an idle-box warning: Strix Halo decode is unified-memory bandwidth bound, so an unrelated co-tenant compile moved an identical request from 18.2 to 3.8 tok/s with acceptance and step count UNCHANGED — which reads exactly like a model or kernel regression. Record loadavg beside any tok/s figure.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A DSpark server prints
DSpark spec-decode ENABLEDat startup and then decodesplain autoregressive for any request the greedy-only verifier cannot serve —
silently. Nothing in the log distinguishes "speculating" from "quietly not".
The sampler case is a live trap in this tree. When a request omits
temperature, the HTTP layer falls back to the model card's sampling defaults,and
share/model_cards/deepseek-v4-flash-0731-rocmfpx.jsonsetstemperature: 1.0. So a temperature-less benchmark measures AR while the logstill claims speculation.
Measured — same binary, same flags, same prompt:
temperaturefield: 290 tokens in 289 steps (one step per token, pure AR)"temperature": 0.0: 268 tokens at accept_rate 0.69 (speculating)This cost a day of chasing a spec-decode "regression" between two builds that
turned out to differ only in whether they shipped that model card. Anyone
benchmarking DSpark without an explicit
temperatureis measuring the wrongthing and cannot currently tell.
No behaviour changes — the fallback logic is untouched. This makes it visible,
and documents what a DSpark measurement needs to be comparable.
Impact
No benchmark impact reported.
Implementation
fix(ds4): the decode gate logs one line naming which condition routed the request to AR —force_ar/ budget stop tokens / sampling — with the effective sampler values (temp,rep_pen,freq_pen,pres_pen).fix(ds4): the fused-verify gate reports every precondition, includingverify_hooks,out_logitsandexpert_runtime. The previous version gated its own warning on two of the pointers whose absence it was meant to report, so a null there stayed invisible.docs(ds4): DS4.md gains the fallback conditions, the tell in the log, the acceptance envelope, and an idle-box warning.Runtime Flags / Configuration
No new runtime flags or configuration switches.
Validation
temperatureomitted vs0.0dflash_serverfrom this branch onmainNotes
The acceptance envelope, which is why this is a docs change too.
Speculative throughput is acceptance-bound and acceptance is workload-bound. On
a gfx1151 host, from the same server under the same flags: code/math prompts
(accept 0.83-0.90, ~2.4 committed tokens per verify step) measured 24-27 tok/s,
while open-ended prose (accept 0.64-0.69, ~1.1 committed) measured 16-18 tok/s.
A spec tok/s figure is therefore not comparable across prompt mixes, and DS4.md
now says so next to the numbers.
Measure on an idle box. Strix Halo decode is unified-memory bandwidth
bound: an unrelated co-tenant compile moved an identical request from 18.2 to
3.8 tok/s with acceptance rate and step count unchanged — which reads
exactly like a model or kernel regression. DS4.md now asks for loadavg beside
any tok/s figure.
Scope: no commits to the verifier, no change to when AR is chosen, no change to
the sampler contract. If you would rather the model card not default
temperatureto 1.0, that is a separate and arguably better fix — this PR onlymakes the current behaviour observable.
Mentioning @Luce-Org/maintainers per CONTRIBUTING now that this is ready for
review.