feat(dspark): expose forced-rejection diagnostics

[jhinpan]

Summary

• add TOKENSPEED_FORCE_SINGLE_TOKEN_VERIFY=1 to keep the full proposal/target-verify round while committing exactly one target token
• reuse the existing per-row force mask, preserve remote-recovery semantics, and keep the default local decode path unchanged
• document the diagnostic and cover CPU, mixed-row, default-off, and CUDA-graph behavior

This supplies the acceptance-independent gate requested by #1132. On exact 131072/1024 TP8/EP8 it measured 36.80 ms per speculative round versus 17.06 ms ordinary decode (2.157x). Forced DSpark remained token-identical to ordinary greedy for a 4K/1K repeated prompt and an exact 131K/128 natural prompt.

Test plan

• focused DSpark/runtime suite: 84 passed on gfx950
• isort --check-only, black --check, git diff --check
• exact 131072/1024, B1/C1, TP8/EP8, FP8 KV, greedy, decode graphs, prefix caching off: three unprofiled runs per arm
• separate all-rank first-64 trace for attribution

Local pre-commit could not create its configured Python 3.12 hook environment because this ROCm container only ships Python 3.10; the equivalent Python formatting/syntax checks above passed.

[jhinpan]

I have read the DCO document and I hereby sign off on the commits contained in this pull request.

[chatgpt-codex-connector]

💡 Codex Review

tokenspeed/python/tokenspeed/runtime/execution/model_executor.py

Lines 742 to 744 in 458ec2c

	accept_lengths = self._apply_force_single_token_verify(
	accept_lengths, 0, num_decodes, ctx.decode_input_ids
	)

Limit penalty counts to the forced token

When this diagnostic is enabled with the flashinfer_full or triton_full sampling backend and a request uses presence, frequency, or repetition penalties, verify() updates its per-request count table for every naturally accepted token before the returned length is replaced with one here. Tokens discarded by the forced length therefore still influence later logits, so the resulting sequence no longer models committing exactly one token per round and can invalidate comparisons against ordinary decoding. Apply the force mask before the backend accumulates counts, or limit/roll back those updates for forced rows.

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[jhinpan]

Superseded by #1170 with an identical tree and a commit-level DCO sign-off. Closing this branch rather than rewriting its published history.