Skip to content

gradium_phonon.md

Latest commit

 

History

History
110 lines (85 loc) · 5.4 KB

gradium_phonon.md

File metadata and controls

110 lines (85 loc) · 5.4 KB
Raw

Gradium Phonon

gradium_phonon runs Gradium's Phonon text-to-speech models. Phonon is a small flow-matching language model with a Mimi decoder, a descendant of Kyutai's PocketTTS, so this family reuses the pocket_tts graphs and session. Output is mono 24 kHz.

The packaged model is Phonon 31m English (gradium/phonon-31m-en, Apache-2.0), in two packages:

Package Source Weights
gradium_phonon_31m_en_q8_0 (default) gradium/phonon-31m-en-audiocpp GGUF: flow transformer linears Q8_0, the rest as published
gradium_phonon_31m_en_safetensors gradium/phonon-31m-en, as published Safetensors: BF16 flow model, F32 Mimi

The 40m checkpoints (gradium/phonon-40m-*, CC-BY-NC-4.0, gated) use the same code with nine layers and load the same way, but are not packaged.

Usage

python3 tools/model_manager_v2.py install gradium_phonon_31m_en_q8_0 --models-root models

audiocpp_cli --task tts --family gradium_phonon \
  --model models/phonon-31m-en --backend metal \
  --text "Hello from Phonon." --out out.wav

The package's default-voice.safetensors is the default voice. --mode streaming emits one audio event per generated frame. The safetensors package installs to models/phonon-31m-en-safetensors and runs the same way.

Voices

Phonon has no voice cloning from audio: the released checkpoints carry no Mimi encoder. Voices are speaker_wavs latent files (one tensor [1, 512, 125]), which the checkpoint projects into its voice prompt. Every Phonon checkpoint reads the same files.

More voices are in gradium/phonon-voices, listed with their names and languages in voices.jsonl. Pass one with --request-option voice_embedding_file=/path/to/voice.safetensors, or copy it into the model folder as embeddings/<name>.safetensors: --voice-id <name> and the server's "voice": "<name>" then select it, and GET /v1/audio/voices lists it.

Text

The text goes to the SentencePiece tokenizer after the PocketTTS clean-up: whitespace collapsed, first letter capitalized, a final period added. There is no number or symbol normalization, so write numbers, currencies and abbreviations out in words for the best results. Letters the tokenizer has no piece for, such as û and ï, reach the model as byte tokens.

Controls

Request options (--request-option name=value). An option not listed here is an error:

Option Default Meaning
temperature 0.3 Flow noise temperature. The reference runtime samples at 0.3 and does not read the config's temp.
seed random Noise seed.
eos_threshold 4 EOS logit threshold, from the checkpoint's config.json.
frames_after_eos -1 Frames kept after EOS, counting the EOS frame. -1 keeps 4 for texts of up to four words and 2 otherwise, as the reference does.
max_frames 0 Frame cap; zero estimates it from the text length.
text_chunk_tokens 50 Tokens per text chunk.
text_chunk_size unset Character chunk size instead of sentence/token budgeting.
voice_embedding_file unset A voice file outside the model folder.
export_voice_state_file unset Export the prepared voice state.

Session options are those of PocketTTS under the gradium_phonon. prefix, for example --session-option gradium_phonon.weight_type=f32, minus the Mimi encoder ones.

Conversion

The Q8_0 package is built from the safetensors checkpoint with the second command below; the first keeps the published types. audiocpp_gguf embeds config.json and tokenizer.model in the GGUF; put default-voice.safetensors (or embeddings/default.safetensors) beside it:

audiocpp_gguf --input models/phonon-31m-en-safetensors/model.safetensors \
  --root models/phonon-31m-en-safetensors \
  --family gradium_phonon --output phonon-31m-en-bf16.gguf --type orig \
  --exclude-prefix flow_lm.flow.w_s_t --overwrite

audiocpp_gguf --input models/phonon-31m-en-safetensors/model.safetensors \
  --root models/phonon-31m-en-safetensors \
  --family gradium_phonon --output phonon-31m-en-q8_0.gguf --type q8_0 \
  --exclude-prefix flow_lm.flow.w_s_t \
  --keep-type 'mimi.*=f32' --keep-type 'flow_lm.flow_net.*=bf16' \
  --keep-type 'flow_lm.condition_provider.*=bf16' \
  --keep-type 'flow_lm.input_linear.*=bf16' --keep-type 'flow_lm.out_eos.*=bf16' \
  --overwrite

orig keeps the checkpoint's types: BF16 for the flow model, F32 for Mimi. The Q8_0 recipe quantizes only the flow transformer's linear layers, as Gradium's own runtime does; Mimi stays F32. flow_lm.flow.w_s_t is used only in training.

The loader reads the model sizes from config.json, uses the plain RMS time embedding when it says "time_rms_norm": "rms", and folds the summed conditioners (num_speakers = 1, duration_delta = 0) into a bias on input_linear, which is what the reference adds to every generated frame.

Validation

Checked against Gradium's Rust runtime with its reference dumps, feeding the same noise through noise_file: with F32 weights on the CPU, the token ids, the first EOS frame and the frame count match exactly, and the per-frame latents agree to about 1e-5 relative. BF16 and Q8_0 weights and the Metal backend drift as usual for a lower precision, with the same Whisper transcripts.