Groth16 prover: memory scoping + MSM acceleration + prover options - #628
Open
OBrezhniev wants to merge 113 commits into
Open
Groth16 prover: memory scoping + MSM acceleration + prover options#628OBrezhniev wants to merge 113 commits into
OBrezhniev wants to merge 113 commits into
Conversation
# Conflicts: # build/snarkjs.js # build/snarkjs.min.js # src/groth16_prove.js
…y needed. Changed a few anonymous functions to named for easier profiling. Smaller chunks in joinABC to split work in most cases, and pass buffer ownership to worker threads there.
…ly when they are needed (bfj, ejs). And don't use bfj module for small json files (proofs, public signals). Comment out vm module and manual garbage collection in zkey_new.js.
…Switch between different buildABC implementations through options. Debug & logging in groth16_prove. Rebuild.
Picks up the 2-phase termination and WorkerSlot identity model from ffjavascript so all buildABC modes (js/wasm/wasm1) run reliably without intermittent "Worker terminated unexpectedly" failures or hangs.
Picks up ffjavascript console.log cleanup (engine_fft, engine_multiexp, threadman) via rebuilt bundles. No logic changes in snarkjs src. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Picks up the worker-side in-place reversePermutation (no WASM memory growth, zero-copy) via the inlined ffjavascript in the IIFE bundle. No snarkjs src changes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…script Picks up the ffjavascript fix that stops pairingEq from detaching caller-owned G1.g/G2.g buffers. Restores the full prove/verify pipeline: snarkjs test suite now 49/49 passing. No snarkjs src changes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ding Picks up the fastfile node:fs->fs.constants fix and the ffjavascript bn128 prebuilt-wasm loader fix (atob/arrayBuffer) so the browser builds run in a real browser again. snarkjs browser test suite (browser_tests) now passes: full setup/prove/verify in headless Chrome on both the IIFE and ESM builds. No snarkjs src or rollup config changes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…pt; drop dead taskmanager
- misc.getRandomBytes / sha256digest: drop `process.browser`. Prefer the Node
crypto module (createHash, randomFillSync -- no per-call size limit), fall back
to Web Crypto (getRandomValues chunked to 65536 bytes; subtle.digest on the
view, not data.buffer, so subarray byteOffset/byteLength is respected).
- askEntropy: use the browser prompt only when a real DOM window exists
(typeof window / window.prompt), not !process.browser -- "not Node" is not the
same as "browser" (Bun/Deno/edge/SES have neither).
- Delete src/taskmanager.js: unused dead code, and the only place using
`new Worker(code, {eval:true})` + require() codegen, which trips CSP/eval
scanners.
Rebuilt browser bundles. (package.json fastfile dep left as-is; separate.)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…stubs Rebuilt IIFE bundles after the ffjavascript (os/crypto) and fastfile (fs/constants) "browser" field additions. No snarkjs src changes. Browser e2e (IIFE + ESM) passes; bundles have no Node-builtin leaks. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Picks up ffjavascript's vendored, statically-imported prebuilt wasm (no dynamic import of wasmcurves, no gzip decode). No snarkjs src changes. Browser e2e (IIFE + ESM) and tutorial e2e pass; node suite 49/49. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… from ffjavascript Picks up ffjavascript's lazy getWorkerSource (no Blob/btoa at import) and the base64 decoder that prefers Buffer/atob with a pure-JS SES fallback. No snarkjs src changes; browser e2e + node suite pass. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The five base-point sections (A/B1/B2/C/H) were each read whole into a buffer and then sliced per worker chunk -- two full copies of every section, and all five held in RAM concurrently (the calc promises run in parallel). Switch them to curve.G1/G2.multiExpAffineChunked with a per-section reader (mkSectionReader) that returns each chunk directly via fdZKey.readToBuffer. No full section buffer, no main-thread slice, and only a few chunks resident at a time. Using readToBuffer directly (rather than binFileUtils.readSection per chunk) also avoids amplifying readSection's per-call console.time logging by ~40x/section. Effect scales with circuit size: - authV3 (29 MB zkey, 19 MB bases): peak RSS ~603 -> ~586 MB, time neutral. - sha256 (1.1 GB zkey, 733 MB bases): peak RSS ~3.85 -> ~3.41 GB (~12%), and a bit faster + lower variance (less GC pressure from the big transient allocs). Proof identical / verifies OK in both cases. Validated: snarkjs 49, ffjavascript 63, tutorial e2e (groth16/plonk/fflonk), authV3 + sha256 prove+verify. Bundles rebuilt. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The default buildABC for circuits whose witness fits one pass was buildABCWASM1: a single worker that loads ALL coefficients (e.g. 357 MB) + witness + the three output buffers (~621 MB) and never shrinks (WASM memory can't), so that ~681 MB stayed resident for the whole prove, and the build ran single-threaded. Replace it with a streaming build: - buildABCStream: still a SINGLE full-witness pass (so each disjoint output range is computed completely -> no batchAdd/joinABC merge, unlike multi-threaded buildABC), but the domain is split into nChunks output ranges processed with bounded in-flight. Each task holds only witness + one coeff chunk + one output chunk, so a worker's high-water is ~witness+chunk instead of the whole thing. - pickStreamParams: adaptive. Each busy worker's WASM memory persists, so the floor buildABC leaves behind is ~maxInFlight x perWorker. It sizes nChunks so a worker holds ~2x witness, derives maxInFlight from a worker-memory floor budget (default 256 MB), and sets nChunks to a few per BUSY worker (not full concurrency -- that would re-copy the witness per chunk for nothing at low parallelism). Small circuits -> full parallelism; large -> bounded. Tunable via options.buildABCFloorBudget / buildABCnChunks / buildABCmaxInFlight. - Default path uses streaming whenever the witness fits a single pass (all normal circuits); multi-threaded buildABC stays as the witness-too-big fallback. buildABCWASM1 is kept for the explicit "wasm1" option. Measured (sha256, 1.1GB zkey, vs the old wasm1 default 9.11s / 3354 MB): - default 256 MB budget (n9/k2): ~8.5s / ~3.05 GB -> ~7% faster AND ~9% less peak - memory-first (<=192 MB budget, k1): ~9.3s / ~2.9 GB -> -13% peak, +2% time - raising the budget is counterproductive (more memory, eventually slower from worker oversubscription against the concurrent multiexps); the knob is useful only downward. authV3 (small) picks n45/k15 -> unchanged (~1.08s). All verify OK. Validated: snarkjs 49, tutorial e2e (groth16/plonk/fflonk), authV3 + sha256 prove+verify. Bundles rebuilt. ffjavascript untouched. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Picks up the binfileutils MAX_BUFFER_SIZE and ffjavascript BigBuffer PAGE_SIZE cleanup (dead `Buffer.constants` probe replaced with an explicit `1 << 30`). The IIFE and browser-ESM bundles inline those deps; main.cjs/cli.cjs import them externally and are unaffected. No behaviour change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The IFFT->applyKey->FFT pipeline drops each FFT/IFFT input immediately (buffX_T is nulled; buffXodd goes out of scope), so pass consume=true to ffjavascript's fft/ifft and skip its defensive full-input copy. The 3 FFT inputs are flat Uint8Arrays (from batchApplyKey) and are consumed; the 3 IFFT inputs are BigBuffers (from buildABC) and still flatten as before. sha256: ~100-300 MB lower peak RSS, time within noise (the FFT copies overlap the concurrent multiexps, so the win is modest). authV3 + sha256 verify OK; snarkjs 49, ffjavascript 64, tutorial e2e all pass. Bundles rebuilt. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
When the domain fits under BigBuffer's 1 GiB page, allocate the A/B/C outputs as flat Uint8Arrays instead of BigBuffers. The downstream IFFT (consume=true) can then take them in place and skip its defensive full-input copy -- previously a BigBuffer input forced a flatten-copy. Larger domains stay paged BigBuffers (the IFFT flattens those as before). ~2.5% faster end-to-end on sha256 (8.59s -> 8.37s, quiet machine, the whole run distribution shifts down); peak RSS unchanged (the consumed copies are early in the pipeline, not at the peak). authV3 + sha256 verify OK. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
buildABCWASM1 (single worker, one full-witness pass, all coeffs at once) is exactly buildABCStream with nChunks=1 -- and the streaming default strictly dominates it (lower, bounded worker memory + tunable parallelism). Delete the ~150-line function and route the explicit "wasm1" option to buildABCStream(..., 1, 1), byte-identical. The other variants stay: "wasm" (multi-threaded, the witness-too-big-for-one-pass fallback that splits the witness across passes) and "js" (pure-JS element-at-a-time, zero bulk wasm allocation -- the universal fallback for arrays beyond wasm's 32-bit limit). Behaviour-preserving; default/wasm1/js/wasm all verify OK. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
r1csInfo prints header counts (wires, constraints, inputs, labels, outputs) but
called readR1cs() with defaults, which loads the entire constraints section (and
the wire-to-label map) -- reading the whole, potentially many-GB, .r1cs file just
to print a few numbers. On a 12 GB r1cs this took minutes and gigabytes of RAM.
Pass {loadConstraints: false, loadMap: false}: the counts all come from section 1
(the header). Same output, now near-instant (12 GB r1cs: ~minutes -> 0.17 s,
~79 MB RSS). r1csInfo never touches cir.constraints, and no caller uses the
returned value's constraints.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
options.msmBatching selects the ffjavascript MSM batching mode and is threaded to all five multiexps (A, B1, B2, C, H): "auto" (default) batch-affine only for cache-friendly chunk sizes; "enabled" always batch (best for small/medium circuits); "disabled" plain multiexp (lowest memory; best for very large circuits). Invalid values throw.
- cli.js: drop the decade-old TODO command list and the commented-out zksnarkSetup (references identifiers that no longer exist). - groth16_prove.js: fold the explicit buildABC === 'stream' branch into the identical default (validation above still gates the accepted values); condense history-lesson comments (getCutPoint bisection, retired wasm/wasm1 options, gc() hang postmortem) while keeping the load-bearing Float64-NaN-canonicalization rationale in the gather loop; document the buildABCStream backpressure wait. - zkey_new.js: remove the local memUsage() helper and its 12 unconditional logger.info dumps -- memory logging is opt-in via the prover's memoryLogging option now. - Bundles rebuilt.
Picks up wasmcurves' partitioned batch MSM (zeros dropped, ones plain-added, small scalars clamp the window sweep) and ffjavascript's triviality-aware auto batching policy via re-pins; bundles rebuilt. Measured e2e (20 cores): sha256 (2M constraints, 100% trivial witness scalars) 5.98-6.66s -> 5.62-6.24s (~-5%); authV3 unchanged 0.71-0.74s (MSM already a small share). Microbench of the witness-MSM shape alone (2^17 points, 2/3 zero + 1/3 one scalars): 23ms -> 17ms (-26%). The e2e gap vs rapidsnark's -23% is expected: snarkjs overlaps its MSMs with FFT/IO phases, so Amdahl caps the win.
GLV/GLS now skipped on gathered big-partitions under 4096 full-width scalars, where decomposition overhead beats halved windows; all-big fast-path chunks (the dominant prove shape) keep the endomorphism. A/B confirmed endo stays a clear net win in this stack (authV3 451 vs 558ms, sha256 5.75 vs 6.03s with endo disabled) -- rapidsnark's contrary finding stems from its waves-of-chunks threading model, which doesn't apply to our one-MSM-per-worker-chunk design.
Apply the memory-scoping approach to newZKey: - header temporaries (alpha/beta points, generator encodings) wrapped in a scoped IIFE, dropped once section 2 is written; - tauG2 -- the largest ptau section (2x G1 size) -- is now read lazily right before the B2 phase instead of up front, so it is never resident alongside the three G1 sections plus the IC/C/A/B1 working set; - alphatauG1/betatauG1 released after their last consumer (C phase), tauG1 after B1, tauG2 and sR1cs after B2. Peak RSS (in-process sampling, sha256 2M-constraint setup): the early-phase plateau drops ~600 MB (all-sections-resident baseline 6.19 GB -> 5.88 GB peak, now dominated by the B2/contributions tail); authV3 unchanged (~1.0 GB, peak set by compose phases, not sections). Setup time unchanged (back-to-back A/B: 22.06 vs 22.07 s). Output zkeys byte-identical to baseline for both circuits.
Re-pin fastfile/binfileutils: readBinFile with an http(s) URL now opens a range-backed fd instead of buffering the whole body, so the prover's chunked section reads (multiExpAffineChunked) fetch the zkey point sections chunk-by-chunk -- same bounded residency as the Node file path. Browser callers can also pass a File/Blob to stream a local zkey. Servers without Range support keep the old buffer-it-all behavior. e2e test proves groth16 against a local server in both modes and asserts the streaming path never receives the zkey in one response. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Wo6AVSAwvL9mHTpvnREZPR
Pulls ffjavascript 21f0005: a deterministic worker boot failure now rejects queued tasks after 8 consecutive failures instead of respawning workers forever at full CPU with the prover hanging. Trigger example: running snarkjs under 'node --input-type=module -e' (workers inherit execArgv, which the web-worker shim's file-entry workers cannot accept). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Wo6AVSAwvL9mHTpvnREZPR
wasmcurves ea14742 (now type:module, Vite-built main.cjs, Vitest suite) through ffjavascript 4078b99 (re-vendored curve wasm; layout constants unchanged), binfileutils 372283f, r1csfile 6a315ea. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Wo6AVSAwvL9mHTpvnREZPR
The n=1 branch called fromEvaluations(polynomial.coef), which IFFTs the coefficient buffer; f(x^1) = f(x), so return a copy via fromPolynomial. Also drop a leftover nocommit debug comment in zkey_new. Bundles rebuilt. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… matrix - misc.test.js: getRandomBytes (incl. >64KiB Web Crypto window fill) and sha256digest (vector + subarray-view regression) - wtns_calculate.test.js: output-fd lifecycle -- no zero-byte wtns left behind when witness calculation throws - polynomial.test.js: expX n=1 identity copy and n<1 rejection - fullprocess.js: stream tuning knobs (floor budget clamp, serial chunk, non-divisor chunk count), memoryLogging timer smoke test, plonk verify wrong public-signal count returns false without a logger - optimization_levels.test.js: same circuit compiled at circom -O0/-O1/-O2 (committed fixtures); each level must set up, prove (streamed + js + forced chunk boundaries), and verify. Reproduces the getCutPoint gap bug asymmetry: with the old bisection -O0/-O1 fail while -O2 passes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…nnotate unreachable branches - zkey_utils.writeZKey never awaited its curve lookup and could not have worked; now covered by a read/write/prove round-trip test. - Polynomial.divByVanishing2 computed wrong results for most input sizes (its chunked recurrence reads propagated coefficients before they are written) and was unused, debug-instrumented dead code. Removed. - ~150 genuinely unreachable branches annotated with /* c8 ignore */ and a one-line reason each: internal sanity checks on self-computed data, progress logs beyond fixture size, BigBuffer paths behind the 1 GiB section threshold or a 2^28 domain, and checks reachable only with hand-forged ceremony files. Bundles rebuilt. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
….3%) New suites: misc, curves, mul_z, proof, bigarray, cpolynomial, ptau_tools, zkey_utils, wtns_calculate. Extended: polynomial, fullprocess, fflonk. - corruption matrices over ptau contributions/sections, zkey regions and bellman responses, asserting each verification check fires - cross-protocol zkey guards, crafted wrong-curve/length witnesses, per-commitment and per-evaluation verify tampers for plonk/fflonk - fullProve wrappers, CLI-facing exporters (calldata, Solidity verifiers, JSON), ptau convert/truncate/exportJson, wtns debug, BLS12-381 dispatch - BigBuffer-threshold polynomial ops, property-based division tests - mocha now runs with --expose-gc (package.json), covering the gc guards and matching the documented profiling setup Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Nine functions (newZKey, plonkSetup, fflonkSetup, plonk16Prove, fflonkProve, powersoftau verify, phase2verifyFromInit, phase2importMPCParams, wtnsCheck) opened their binary files and only closed them on the success path; every early error return and throw (wrong protocol, curve mismatch, unprepared/too-small ptau, corrupted sections, invalid witness) leaked the fds. The leaks became visible as Node fd-closed-on-garbage-collection warnings once the test suite started running with --expose-gc. Each function now hoists its fd handles and closes them in a finally, so no exit path can leak; the success-path closes stay where they are (freeing fds early) and the finally re-close is a guarded no-op -- fastfile's close() throws synchronously on a second close. Test suite: 229 passing with zero fd-on-gc warnings (previously dozens). Bundles rebuilt. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
fastfile's close() used to throw "Closing the file twice" synchronously; it is now idempotent like fs.promises.FileHandle.close() (repeat calls return the same promise, including the same rejection when the final flush fails). Re-pinned bottom-up: fastfile 6278879 via binfileutils 60dcebe. The fd-lifecycle finally blocks keep their try/catch -- not for double-close anymore, but so a close() failing during exception unwinding cannot mask the original error -- with comments updated. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
fastfile 547eb27: 99.5% lines / 100% functions (was 75.8% / 62.9%); fixed readString past-EOF RangeError (memfile) and an unterminated-readString infinite loop (bigmemfile). binfileutils 44c1fe8: 99.4% lines / 100% functions. r1csfile 6f1381a: 100% lines / branches / functions (was 67.9% / 68.4%). ffjavascript 16cb6c4: 98.4% lines / 96.8% functions (was 62.4% / 49.8%); fixed five dormant bugs (mixed-representation sub sign error, F2.e out-of-bounds write, F3.exp calling a nonexistent method, F3Field deserialization stride, EC zero-point serialization writing to a copy). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01833VaUEJmrFZ7bVprrWwpp
… mirror); re-pin deps Vectors for every hash primitive through the exact import paths the sources use: SHA-256 (FIPS 180-4 incl. the two-block message), BLAKE2b-512 (RFC 7693 appendix A), Keccak-256 (Ethereum classic vectors), and a byte-exact independent mirror of the Keccak256Transcript challenge derivation with an order-sensitivity check. ffjavascript's matching standards suite (RFC 8439 ChaCha, IETF BLS12-381 parameters, bn254 vs an independent BigInt reference) lands via the re-pin: ffjavascript c7337ca, binfileutils 705336c, r1csfile 54bb924. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01833VaUEJmrFZ7bVprrWwpp
wasmcurves b83021f: 99.5% lines / 100% functions (was 78.8% / 78.3%), 151 tests. Fixed isPrime never reaching Miller-Rabin (build_f1m silently skipped generating _sqrt/_isSquare for any prime outside the hardcoded bn254/bls12-381/mnt6753 list); removed three broken wasmsnark leftovers (build_mulacc, build_mem, build_testg1); revived the excluded 27-test mnt6753 suite. Chain: ffjavascript 3cc2f7a, binfileutils 824e9f1, r1csfile f0133fd. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01833VaUEJmrFZ7bVprrWwpp
…Node 20 dispatch) Fixes the failing browser CI job: in Chrome the Range probe for URL-read files is answered 206 from the browser's HTTP cache while the st static server ignores Range, so the reader aborted with 'file changed while reading' during verifyFromR1cs. fastfile dba4d82 degrades to full buffering when a mid-session 200 carries an unchanged strong validator. Also via ffjavascript 6ba342f: worker teardown is hard-terminated portably (a Bun process could never exit after proving -- the residue behind snarkjs#490/#533), and dispatching a detached transfer buffer rejects on every Node version (Node 20 posted silently and hung). Chain: binfileutils 9a54502, r1csfile 85827cd. Bundles rebuilt; browser test suite passes locally. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01833VaUEJmrFZ7bVprrWwpp
Local installs ran with --allow-git=root, which silently omits transitive git dependencies (ffjavascript -> wasmcurves) from the lock; CI's npm ci then fails with 'Missing: wasmcurves from lock file'. Regenerated with --allow-git=all. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01833VaUEJmrFZ7bVprrWwpp
…ory-scoping) Port of the feature/esm-tooling migration (41a8ca3) onto the current branch: Rollup -> Vite (node/cli/browser/iife builds), Mocha -> Vitest (node-esm + browser-chromium via Playwright), flat eslint.config.js. Everything since the original fork point is preserved and converted: - All 14 post-fork test suites (coverage program, standards vectors, optimization-levels matrix, http streaming, buildABC gap, ptau/zkey tools) converted: before/after -> beforeAll/afterAll, this.timeout dropped (config testTimeout/hookTimeout 600s). Node assert imports kept. - node-esm runs test files sequentially (fileParallelism: false): every suite builds a curve with a full worker pool, and parallel files oversubscribe the CPU badly enough that the large-domain FFT test starves past 600s. Sequential, the whole suite is 246 tests in ~45s. - CLI banner is the migration's portable "#!/usr/bin/env node"; all globalThis.gc() call sites are guarded, so dropping --expose-gc is safe (profile with `node --expose-gc build/cli.cjs` as before). - Indent autofix reindented the fd-leak `try { ... } finally` bodies that were deliberately left unindented to minimize those diffs (whitespace-only); eslint caughtErrors: none matches the absorb-close catch(e){} style. - CI keeps OUR three-job workflow (3-OS matrix, hardhat verifier contracts, browser_tests bundle harness) and adds a vitest browser-chromium job; actions bumped to checkout@v6 / setup-node@v6. - Re-pinned all four siblings to their migrated SHAs: binfileutils f71fb82, ffjavascript 4ac1cba, fastfile e7eb0f0, r1csfile 09b93c2. vitest family ^4.1.11 (GHSA-p63j-vcc4-9vmv); postcss/brace-expansion overrides (npm audit clean). Mocha config block removed. 246 node tests + 10 browser tests (Chromium) pass; lint clean; all bundles rebuilt with vite (snarkjs.js dropped, only the umd-referenced snarkjs.min.js is built); CLI r1cs-info and CJS require smoke-verified. (cherry picked from commit 41a8ca3) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01833VaUEJmrFZ7bVprrWwpp
- Re-pin ffjavascript 2c20f30 (sequential vitest files — CI starvation fix), fastfile 748772b, binfileutils f8aedac, r1csfile f5ae083. - browser.esm.js externalizes only ffjavascript again, bundling binfileutils/ r1csfile/fastfile, matching the old rollup build: the browser_tests harness (and any consumer with the documented importmap) only maps ffjavascript, so the wider externalization broke bare-specifier resolution. - browser_tests point at snarkjs.min.js (the unminified snarkjs.js IIFE is no longer built). 246 node tests, 10 vitest browser tests, and the puppeteer bundle harness (IIFE + ESM against a real ceremony ptau) all pass. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01833VaUEJmrFZ7bVprrWwpp
ffjavascript a6abbfe (lockdown harness explicit exit, execFileSync timeouts, CI timeout-minutes), fastfile e74f95d, binfileutils 9ee4f27, r1csfile 0a53eb0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01833VaUEJmrFZ7bVprrWwpp
…ade) wasmcurves c380d6c -> ffjavascript 5475db8, fastfile 38ffb38, binfileutils 7127862, r1csfile 0235e34. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01833VaUEJmrFZ7bVprrWwpp
wasmcurves 13d4acc -> ffjavascript 76fc09a; fastfile ad59e79; binfileutils c69440f; r1csfile bf6933b. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01833VaUEJmrFZ7bVprrWwpp
ffjavascript f16db3c (zombie worker threads can no longer hold the process open after terminate) -> fastfile 87d38de, binfileutils 6c2616f, r1csfile e0bb6f8. Browser bundles vendored from ffjavascript rebuilt. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01833VaUEJmrFZ7bVprrWwpp
…not main Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01833VaUEJmrFZ7bVprrWwpp
…urces)
groth16.prove / groth16.fullProve (proverOptions) and wtns.calculate
(options) accept persistentCache: true | {blockSize, maxBytes, dbName}.
When the zkey (or circuit wasm) is an absolute http(s) URL string or an
explicit {type: "http"} descriptor, the open is routed through fastfile's
IndexedDB block cache: the first proving session populates it, later
sessions read the key locally -- the warm-start e2e proves the second
prove touches the network for the open probe only. Local paths, mem/bigMem
descriptors and open fds pass through untouched, and in Node (no
IndexedDB) the option is a safe no-op.
Re-pinned fastfile to the persistentCache prototype (4ee05be, the
feature/direct_rw_optimization line) and binfileutils to a066c87 -- which
includes the essential rebuild of its browser bundle: that bundle INLINES
fastfile's browser build, so a fastfile re-pin without a binfileutils
rebuild ships the stale copy and the option is silently ignored (found the
hard way: the warm-start test saw 11 fetches instead of 1). snarkjs's own
browser bundles inline binfileutils in turn and are rebuilt here too.
Tests: browser warm-start e2e (cold prove populates the cache, warm prove
= 1 probe request, both proofs verify), Node no-op prove over the http
test server, and unit coverage of the source-mapping helper.
249 node + 11 browser tests pass; lint clean.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01833VaUEJmrFZ7bVprrWwpp
… bundles Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01833VaUEJmrFZ7bVprrWwpp
…uild bundles Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01833VaUEJmrFZ7bVprrWwpp
… rebuild bundles Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01833VaUEJmrFZ7bVprrWwpp
…; rebuild bundles Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01833VaUEJmrFZ7bVprrWwpp
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Groth16 prover: memory scoping + MSM acceleration + msmBatching option
Summary
Companion to the
ffjavascript/wasmcurvesPRs (file-linked dependencies —land all three together). Two arcs:
Memory scoping — chunked zkey bases reads (streamed to workers instead of
whole-section buffers), adaptive streaming buildABC with per-chunk witness
gathering (the witness never enters WASM, so there is no witness size limit;
replaces the retired wasm1/multi-threaded variants and the oversized-witness
fallback), fft
consumewiring in the abc pipeline, explicit 1 GiBBigBuffer/binfileutils paging, fast
r1cs info(header-only read).MSM acceleration — exposes
options.msmBatching = "auto"|"enabled"|"disabled"on
groth16Prove, threaded to all five multiexps (A, B1, B2, C, H). Defaultautopicks the batch/endomorphism path per chunk size.Results (all proofs verify; suites pass):
Also included
browser_tests/bench.mjs: instrumented in-browser proving benchmark(prove wall, main-thread heap, renderer-tree RSS scoped to the launched
Chrome, in-page verification).
only reachable via ffjavascript's unused custom-
pluginspath) stubbed outof the single-file browser builds —
snarkjs.js5.50 → 3.81 MB,snarkjs.min.js756 → 573 KB (−24%).--memlog[=ms]ongroth16 prove/fullprove(API:memoryLogging):opt-in periodic heap/RSS/external logging, Node-gated, cleared on
completion with a final sample. Debug console output removed from the
prover (timers now only via the optional logger).
mocha 11, ejs 6); eslint surfaced and fixed a latent ReferenceError in
Polynomial.expXplus assorted dead code.the manifest resolves standalone; local dev uses uncommitted
file:overrides. Land the sibling PRs first, then re-pin here before merge.
Validation
49 passing; groth16 e2e (prove+verify) on authV3 and sha256 at every step;
CLI smoke tests on the built bundles; browser runs verify in-page.
🤖 Generated with Claude Code