feat(dspark): expose forced-rejection diagnostics - #1170
Conversation
Allow operators to commit one target token while preserving the full proposal and verification round, so infrastructure cost can be measured independently of model acceptance. Signed-off-by: Jin Pan <jin.pan@amd.com>
3e31648 to
11ea9cb
Compare
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 11ea9cb74f
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| if row_count <= 0 or ( | ||
| decode_input_ids is None and not FORCE_SINGLE_TOKEN_VERIFY | ||
| ): |
There was a problem hiding this comment.
When TOKENSPEED_FORCE_SINGLE_TOKEN_VERIFY=1 is used with the triton_full or flashinfer_full sampling backend and repetition, frequency, or presence penalties, this override happens only after verify() has already updated its token-count state from the unmodified accept_index (triton_full.py:548-560, flashinfer_full.py:476-488). Tokens discarded by forcing the returned length to one therefore remain in the sampler's history and alter penalties in subsequent rounds, so the diagnostic no longer behaves as though it committed exactly one token. Pass the forced width into verification or otherwise limit/undo the count accumulation for discarded tokens.
Useful? React with 👍 / 👎.
1 participant
Summary
TOKENSPEED_FORCE_SINGLE_TOKEN_VERIFY=1to keep the full speculative proposal/target-verify round while committing exactly one target tokenThis supplies the acceptance-independent gate requested by #1132. On exact 131072/1024 TP8/EP8 it measured 36.80 ms per speculative round versus 17.06 ms ordinary decode (
2.157x). Forced DSpark remained token-identical to ordinary greedy for a 4K/1K repeated prompt and an exact 131K/128 natural prompt.Test plan
84 passedon gfx950isort --check-only,black --check,git diff --checkLocal
pre-commitcould not create its configured Python 3.12 hook environment because this ROCm container only ships Python 3.10; the equivalent Python formatting/syntax checks above passed.