Skip to content

Fix Whisper ASR timestamp regression - #1742

Open
anshusaurav wants to merge 1 commit into
huggingface:mainfrom
anshusaurav:fix/whisper-timestamp-regression-1684
Open

Fix Whisper ASR timestamp regression#1742
anshusaurav wants to merge 1 commit into
huggingface:mainfrom
anshusaurav:fix/whisper-timestamp-regression-1684

Conversation

@anshusaurav

Copy link
Copy Markdown

Summary

Fixes #1684 — Whisper ASR timestamp regression between v3.8.1 and v4.2.0.

Three fixes targeting the seek loop and logits processor introduced in commit 43b2662 ("Overdue Whisper fixes #1594"):

  • Use actual audio length for seek loop bounds (modeling_whisper.js): The seek loop used input_features.dims[2] (always 3000 = padded 30s) as total_frames instead of generation_config.num_frames (set by the ASR pipeline to the real mel frame count). For clips shorter than 30 seconds, this caused the loop to process silence/padding as real audio, producing hallucinated text and incorrect timestamps.

  • Apply max_initial_timestamp_index constraint (logits_process.js): A continue statement in WhisperTimeStampLogitsProcessor._call() skipped the max_initial_timestamp_index check at line 326, making it dead code. This allowed the model to generate any timestamp as its first token instead of being constrained to the allowed range. The Python reference implementation has no such early exit — both blocks execute sequentially.

  • Guard against infinite seek loop (modeling_whisper.js): If segment_offset computes to zero (e.g., from a degenerate timestamp pair), the seek loop would run forever. Added a break when segment_offset <= 0.

Test plan

  • Verify with the reproduction case from ASR (Whisper) regression for detected timestamps #1684 (Steve Jobs keynote audio) — timestamps should match v3.8.1 output
  • Verify [applause] segment is no longer dropped
  • Existing Whisper prefix token tests (test_modeling_whisper.js) still pass — they use max_new_tokens which bypasses the seek loop
  • Test with audio shorter than 30s — seek loop should terminate after one iteration
  • Test with audio longer than 30s — multi-segment seek should still work correctly

Three fixes for the timestamp regression between v3.8.1 and v4.2.0:

1. Use actual audio length for seek loop bounds: The seek loop used
   input_features.dims[2] (always 3000 = padded 30s) instead of
   generation_config.num_frames (actual audio length). For clips
   shorter than 30s, this caused the loop to process silence as
   real audio, producing hallucinated text and wrong timestamps.

2. Apply max_initial_timestamp_index constraint: A `continue`
   statement in WhisperTimeStampLogitsProcessor skipped the
   max_initial_timestamp_index check, allowing the model to
   generate any timestamp as its first token. The Python reference
   implementation has no such early exit.

3. Guard against infinite seek loop: If segment_offset is zero
   (e.g., from a degenerate timestamp pair), the loop ran forever.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ASR (Whisper) regression for detected timestamps

1 participant