Feature request
I've been building WebAI Bench, an open-source, in-browser benchmark for WebGPU/WASM AI inference. It runs entirely client-side (nothing leaves your device unless you choose to submit results), and the goal is a crowdsourced answer to a question I couldn't find good data on anywhere: for a given model + quantization + runtime, what actually works on real hardware, not just the machine the library maintainer tested on?
The reason I'm posting now rather than waiting for a "finished" product: while spiking this out, I ran Xenova/all-MiniLM-L6-v2 (a small, ~22M-parameter embedding model) through @huggingface/transformers on both WebGPU and WASM execution providers, same device, same input. WASM won by a lot — ~3.5x faster to init, ~9.6x faster to embed. That's the opposite of the "WebGPU is the fast path, WASM is the fallback" assumption I'd baked into my own project glossary before running the numbers. The likely cause: at this model size, WebGPU's fixed per-call overhead (shader/pipeline compile, kernel dispatch, host↔device transfer) never gets amortized — WASM's lower fixed cost wins outright. Full writeup: docs/spikes/sp3-findings.md.
That result is exactly why I think this needs to be measured, not assumed, and measured across a lot of real devices, not just mine.
What's actually built right now (being precise about this):
- A capability probe, a versioned model registry (currently 6 cells across WebLLM, Transformers.js, and wllama), and a full run flow — progress, a Stop button that actually works, a size warning before large downloads, local JSON export.
- All three runtimes have adapters passing a shared conformance test suite. Of those, only the WebLLM/WebGPU path has been run end-to-end in a real browser so far — TTFT and decode-tps brackets agree with WebLLM's own self-reported numbers within ~1-2%. Transformers.js and wllama are wired in but haven't had that same real-browser exercise yet.
- No backend yet — this phase is local-only. Opt-in submission/aggregation is next if this phase validates.
What I'd actually value from this community specifically:
- Does the methodology (fixed prompt/N=3 median/warmup-then-discard/etc.) look right to people who actually build these runtimes? Anything you'd measure differently?
- Any known runtime-specific gotchas I should know about before more people hit this in the wild — I've been logging every one I've found in
packages/harness/src/adapters/QUIRKS.md as I go and would rather find out about more of them from people who know the internals than from confused bug reports later.
- If you have five minutes and a spare device: https://main.webai-bench-web.pages.dev and see what it reports. Numbers export locally as JSON right now; nothing is collected without explicit opt-in later.
Repo: https://github.com/Sallamhamza/webai-bench. Apache-2.0 code, CC-BY-4.0 docs/dataset. Feedback, especially the kind that pokes holes in the approach, is genuinely what I'm here for at this stage.
Motivation
Not a feature request in the usual sense — flagging an open-source benchmark project (WebAI Bench) that measures in-browser AI inference across WebGPU/WASM, and an early finding specific to this library: on a small (~22M param) embedding model, WASM beat WebGPU by ~3.5x on init and ~9.6x on embed time (same device, same input, Xenova/all-MiniLM-L6-v2). That's the opposite of the "WebGPU is the fast path" assumption baked into a lot of tooling, including my own project's glossary before I ran the numbers. Full writeup: https://github.com/Sallamhamza/webai-bench/blob/main/docs/spikes/sp3-findings.md
I'm posting here because the maintainers and users of this library are exactly the people who'd know whether that result is expected, a fluke of my test device, or worth digging into further — and because the benchmark's methodology (fixed prompt, N=3 median, warmup-then-discard — https://github.com/Sallamhamza/webai-bench/blob/main/docs/04-benchmark-methodology.md) could use scrutiny from people who actually build this runtime before more people rely on it.
Your contribution
The benchmark itself is open source and live: https://main.webai-bench-web.pages.dev (results export locally as JSON, nothing collected without explicit opt-in). Repo: https://github.com/Sallamhamza/webai-bench (Apache-2.0 code, CC-BY-4.0 docs/dataset). I'm not proposing a PR to this repo — happy to log any Transformers.js-specific quirks I find into my project's own QUIRKS.md (https://github.com/Sallamhamza/webai-bench/blob/main/packages/harness/src/adapters/QUIRKS.md) instead, and to fold in whatever feedback comes back here.
Feature request
I've been building WebAI Bench, an open-source, in-browser benchmark for WebGPU/WASM AI inference. It runs entirely client-side (nothing leaves your device unless you choose to submit results), and the goal is a crowdsourced answer to a question I couldn't find good data on anywhere: for a given model + quantization + runtime, what actually works on real hardware, not just the machine the library maintainer tested on?
The reason I'm posting now rather than waiting for a "finished" product: while spiking this out, I ran
Xenova/all-MiniLM-L6-v2(a small, ~22M-parameter embedding model) through@huggingface/transformerson both WebGPU and WASM execution providers, same device, same input. WASM won by a lot — ~3.5x faster to init, ~9.6x faster to embed. That's the opposite of the "WebGPU is the fast path, WASM is the fallback" assumption I'd baked into my own project glossary before running the numbers. The likely cause: at this model size, WebGPU's fixed per-call overhead (shader/pipeline compile, kernel dispatch, host↔device transfer) never gets amortized — WASM's lower fixed cost wins outright. Full writeup: docs/spikes/sp3-findings.md.That result is exactly why I think this needs to be measured, not assumed, and measured across a lot of real devices, not just mine.
What's actually built right now (being precise about this):
What I'd actually value from this community specifically:
packages/harness/src/adapters/QUIRKS.mdas I go and would rather find out about more of them from people who know the internals than from confused bug reports later.Repo: https://github.com/Sallamhamza/webai-bench. Apache-2.0 code, CC-BY-4.0 docs/dataset. Feedback, especially the kind that pokes holes in the approach, is genuinely what I'm here for at this stage.
Motivation
Not a feature request in the usual sense — flagging an open-source benchmark project (WebAI Bench) that measures in-browser AI inference across WebGPU/WASM, and an early finding specific to this library: on a small (~22M param) embedding model, WASM beat WebGPU by ~3.5x on init and ~9.6x on embed time (same device, same input, Xenova/all-MiniLM-L6-v2). That's the opposite of the "WebGPU is the fast path" assumption baked into a lot of tooling, including my own project's glossary before I ran the numbers. Full writeup: https://github.com/Sallamhamza/webai-bench/blob/main/docs/spikes/sp3-findings.md
I'm posting here because the maintainers and users of this library are exactly the people who'd know whether that result is expected, a fluke of my test device, or worth digging into further — and because the benchmark's methodology (fixed prompt, N=3 median, warmup-then-discard — https://github.com/Sallamhamza/webai-bench/blob/main/docs/04-benchmark-methodology.md) could use scrutiny from people who actually build this runtime before more people rely on it.
Your contribution
The benchmark itself is open source and live: https://main.webai-bench-web.pages.dev (results export locally as JSON, nothing collected without explicit opt-in). Repo: https://github.com/Sallamhamza/webai-bench (Apache-2.0 code, CC-BY-4.0 docs/dataset). I'm not proposing a PR to this repo — happy to log any Transformers.js-specific quirks I find into my project's own QUIRKS.md (https://github.com/Sallamhamza/webai-bench/blob/main/packages/harness/src/adapters/QUIRKS.md) instead, and to fold in whatever feedback comes back here.