Skip to content

Fix Moonshine ASR tokenizer decoding - #1746

Open
jeevan6996 wants to merge 1 commit into
huggingface:mainfrom
jeevan6996:fix/moonshine-asr-tokenizer-decode
Open

Fix Moonshine ASR tokenizer decoding#1746
jeevan6996 wants to merge 1 commit into
huggingface:mainfrom
jeevan6996:fix/moonshine-asr-tokenizer-decode

Conversation

@jeevan6996

Copy link
Copy Markdown

Summary

Fix Moonshine automatic speech recognition pipelines that fail when decoding generated token IDs.

Moonshine processors provide audio feature extraction but do not necessarily include a tokenizer. The pipeline already owns the tokenizer loaded for the model, so decoding must use that pipeline-level tokenizer rather than calling batch_decode on the processor.

Changes

  • Decode Moonshine outputs with this.tokenizer.batch_decode.
  • Add a regression test using a processor without a tokenizer and a pipeline tokenizer.

Validation

  • Direct regression check passed with a minimal Moonshine pipeline.
  • Prettier check passed for the changed files.
  • TypeScript build passed.
  • The focused Jest command could not complete because the test harness encountered a network fetch failed during setup.

Fixes #1735

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Moonshine ASR pipeline throws 'Unable to decode without a tokenizer' — _call_moonshine calls processor.batch_decode but processor has no tokenizer

1 participant