Load external data without copying it onto the JS heap - #1747
Open
astefanutti wants to merge 3 commits into
Open
Load external data without copying it onto the JS heap#1747astefanutti wants to merge 3 commits into
astefanutti wants to merge 3 commits into
Conversation
onnxruntime PR #29477, "[web] Support Blob-backed external data for on-demand loading in JSPI builds", merged on 2026-07-14. The pin here predates it (2026-04-16), so the runtime rejects a Blob handed to it as external data and the change in the next commit has nothing to hand it to. 1.29.0-dev.20260811-e415ef9afd is dated after that merge and is the build the change was developed and measured against.
Loading a model from Cache Storage currently materialises every external-data file into the JS heap before onnxruntime-web ever sees it: `loadResourceFile` reads the `Response` into a `Uint8Array`, and `getModelDataFiles` passes that buffer on. For a multi-gigabyte model that is a full second copy of the weights, live at the same moment the runtime is allocating its own. A Cache Storage `Response.blob()` is a file reference rather than a copy, and onnxruntime-web accepts one for external data as of the runtime bumped in the previous commit. So on a cache hit the bytes can go straight from disk to the runtime and never enter the heap at all. Measured on a 2.887 GB model, same runtime and same session: peak resident goes from 6,056 MB to 4,221 MB. The saving is the size of the weights, so it grows with the model. Two deliberate limits. It only applies on a cache HIT. Reading the stream to report download progress would defeat the purpose — the chunks would be resident twice, once as buffers and once in Blob storage — and while several gigabytes are arriving, progress is worth more than peak memory. Every load after the first takes the new path, which is where the peak actually matters, and is unchanged for Node, which keeps returning a path. And it is opt-in, threaded from the one caller that knows it is looking at external data. `loadResourceFile` is generic — it also serves config.json and tokenizer.json, whose callers parse the result as text — so returning a Blob unconditionally would break them.
…ing it The previous commit made a WARM load cheap: a cached file comes back as a Blob and its bytes never reach the JS heap. A cold one was untouched, and a cold one is the case that fails. `getModelDataFiles` starts every external-data chunk concurrently, and each is read into a `Uint8Array` so download progress can be reported — so a first load peaks at the SUM of the chunks rather than the largest. Measured on a 17 GB model from an empty cache: `Array buffer allocation failed` at 16.2 of 17.0 GB, 13 GB cached, load failed. It succeeds on a later attempt only because each file is cached as it completes, so the retry has less left to buffer. The body can go straight into Cache Storage instead. `cache.put` takes a `Response` and writes it to disk without materialising it, and reading it back with `cache.match` gives the same Blob a warm load gets — so the bytes travel network -> disk -> runtime and never sit on the heap. Progress survives, which is the only reason the buffer existed: a pass-through `TransformStream` counts bytes as they go past. It holds one chunk, not the file, and backpressure keeps it that way. If the cache refuses the write — QuotaExceededError being the expected one — it falls back to the buffered path. That has to re-fetch rather than reuse the response, whose body the failed attempt consumed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Loading a model's external data currently routes every byte through the JS heap. On a cache hit,
loadResourceFilereads theResponseinto aUint8ArrayandgetModelDataFilespasses that buffer to onnxruntime-web, which allocates its own — so a multi-gigabyte model is resident twice at the same moment. On a cache miss the same buffer is built in order to report download progress. AndgetModelDataFilesstarts every chunk concurrently, so the peak is the sum of the chunks, not the largest one.For a model with eight ~2 GB chunks that is ~16 GB of
Uint8Arraybefore the runtime has allocated anything, and it fails.A Cache Storage
Response.blob()is a file reference rather than a copy, and onnxruntime-web accepts one for external data as of #29477 (merged 2026-07-14).cache.put()likewise takes aResponseand streams it to disk without materialising it. Between them the bytes can go network → disk → runtime and never touch the heap.Measured
Warm load — same model, same runtime, same session, peak RSS sampled at 200 ms, external data passed as
Uint8ArrayvsBlob, on a 2.887 GB model:The 1,835 MB saved is one full copy of the weights, so it scales with model size. (Caveats: one session rather than the two an app may load concurrently, wasm EP so the GPU's own residency is not counted, and summed RSS double-counts pages shared between processes. The ratio is the finding; the absolutes are indicative.)
Cold load — a 17 GB model (8 external-data files, largest 1.996 GB), empty cache, first attempt:
Array buffer allocation failedBefore this, a first load of a model that size fails and only completes on a later attempt, because each file is cached as it completes so the retry has less left to buffer. Progress reporting still works throughout (1.6 → 4.2 → 6.8 → 9.2 → 11.8 → 14.3 → 16.1 → 17.0 GB).
That model has also run three full conversation suites through the new path — 84 generated answers, no failures.
Design notes
Progress is why the buffer existed, so the cold path keeps it with a pass-through
TransformStreamthat counts bytes as they go past. It holds one chunk, not the file, and backpressure keeps it that way.It is opt-in, threaded from the one caller that knows it is looking at external data.
loadResourceFileis generic — it also servesconfig.jsonandtokenizer.json, whose callers parse the result as text — so returning aBlobunconditionally would break them. Hence theas_blobparameter rather than a change in behaviour. Node is unaffected and keeps returning a path.If the cache refuses the write (
QuotaExceededErrorbeing the expected one) it falls back to the buffered path. That has to re-fetch rather than reuse the response, whose body the failed attempt consumed; the re-fetch goes throughgetFile()so a gated repo keeps itsAuthorizationheader.The onnxruntime-web bump is load-bearing, not incidental. The current pin (2026-04-16) predates #29477, so the runtime materialises any Blob handed to it — which is the peak this removes, and is also why it throws over the 2 GiB
ArrayBufferlimit. The new pin is dated after that merge and is what the numbers above were measured against. The package already pins a dev build, so this is a bump rather than a new kind of dependency.Not covered
getModelDataFilesstill starts every chunk concurrently. That no longer costs heap, but it does mean N simultaneous connections; serialising them is a separate question and not addressed here.