[WebGPU] embeddinggemma-300m-ONNX q4/q8 produce silently wrong embeddings (cos ~0 vs WASM); fp32 correct

[jethac]

System Info

• transformers.js 3.8.1 (also reproduced on 3.7.0), loaded from jsDelivr
• Chrome 150.0.7871.125, Windows 11
• GPU: NVIDIA GeForce RTX 5060 Ti (Blackwell), driver 595.79; WebGPU adapter reports shader-f16: true
• Model: onnx-community/embeddinggemma-300m-ONNX

Description

With device: "webgpu" and dtype: "q4" or "q8", EmbeddingGemma returns embeddings that are essentially orthogonal to the correct ones — no error, no warning, normal-looking norms, full speed. The same code and weights on device: "wasm" are correct, and dtype: "fp32" on WebGPU is correct, so this isn't the model-card fp16-activation restriction — it points at the WebGPU quantized-matmul path (MatMulNBits / dequant kernels).

Cosine similarity between WASM and WebGPU outputs for identical inputs (should be ~1.0):

dtype	cos(wasm, webgpu) per input	verdict
q4	0.0687, 0.0254, 0.0562, 0.0367	wrong
q8	-0.0496, 0.0756, 0.0432, 0.0011	wrong
fp32	1.0000, 1.0000, 1.0000, 1.0000	correct

I also validated against the reference google/embeddinggemma-300m via sentence-transformers in Python: WASM q4 = mean cos 0.973 with 0.89 top-10 retrieval overlap on a real 32k-chunk corpus (i.e. healthy quantization loss); WebGPU q4/q8 = mean cos ≈ 0.00, 0.00 overlap; WebGPU fp32 = cos 1.0000, identical top-10s.

The failure mode is nasty because it's silent: the vectors are unit-norm and look plausible, so a latency-focused integration ships broken retrieval without noticing. #1418 (user reporting q8 + WebGPU-with-fallback producing embeddings that "don't match the example", unresolved) may be this same root cause. Likely related as a class: #1317 (q8 decoders produce gibberish on WebGPU, correct on WASM). Possibly-relevant external datapoint: EmbeddingGemma also returns all-zero vectors on LiteRT's Metal GPU backend unless forced to fp32 (google-ai-edge/LiteRT#7693) — this model seems to be an unusually good canary for reduced-precision GPU paths.

Reproduction

Self-contained page — loads the same dtype on both backends and compares outputs (open with ?dtype=q4, ?dtype=q8, ?dtype=fp32):

<script type="module">
import { AutoModel, AutoTokenizer } from
  "https://cdn.jsdelivr.net/npm/@huggingface/transformers@3.8.1";

const DTYPE = new URLSearchParams(location.search).get("dtype") || "q4";
const MODEL = "onnx-community/embeddinggemma-300m-ONNX";
const TEXTS = [
  "task: search result | query: why does the constable distrust the visiting scientist",
  "task: search result | query: rules of acquisition about profit",
  "task: search result | query: the captain is assimilated by the collective",
  "task: search result | query: a tailor who used to be a spy",
];

async function embed(device) {
  const tokenizer = await AutoTokenizer.from_pretrained(MODEL);
  const model = await AutoModel.from_pretrained(MODEL, { device, dtype: DTYPE });
  const { sentence_embedding } = await model(tokenizer(TEXTS, { padding: true }));
  const [b, d] = sentence_embedding.dims;
  const rows = [];
  for (let i = 0; i < b; i++) {
    const v = Array.from(sentence_embedding.data.slice(i * d, (i + 1) * d));
    const n = Math.hypot(...v) || 1;
    rows.push(v.map(x => x / n));
  }
  await model.dispose();
  return rows;
}

const cos = (a, b) => a.reduce((s, x, i) => s + x * b[i], 0);
const W = await embed("wasm");
const G = await embed("webgpu");
TEXTS.forEach((t, i) =>
  console.log(`dtype=${DTYPE} cos(wasm, webgpu) = ${cos(W[i], G[i]).toFixed(4)}`));
</script>

Observed output:

dtype=q4 cos(wasm, webgpu) = 0.0687
dtype=q4 cos(wasm, webgpu) = 0.0254
dtype=q4 cos(wasm, webgpu) = 0.0562
dtype=q4 cos(wasm, webgpu) = 0.0367

Expected: ~1.0 (or at least the ~0.97 the same weights achieve on WASM vs the reference encoder).

Happy to run diagnostics on this hardware if useful (single data point so far: RTX 5060 Ti / Blackwell / Chrome 150 / Windows 11).