Repository navigation
Calibrated conversion, measured quantization, and release-integrity fixes - #1
Merged
Merged
Conversation
…ixes Adds a calibrated path to `rai convert` and brings the quantizer's quality under measurement for the first time, alongside a set of correctness and release-integrity repairs found while doing so. Calibrated conversion - `rai convert --calibration-text <file>` chooses each group's quantization parameters from the activations the projection actually receives, not from the range its weights span. On SmolLM2-1.7B this is 9.22 perplexity against round-to-nearest's 10.30 at a byte-for-byte identical file, no GPU and no Python. It reaches architectures the Python exporter refuses (projection biases among them), and refuses — with a named reason — the ones it cannot yet reproduce exactly (MoE routing, sandwich/post norms, per-head QK norms) rather than quantizing against statistics from a model that behaves differently. Measurement - `rai perplexity`, a llama.cpp-compatible perplexity with per-token top-K records and standard errors; `scripts/reference_ppl.py` and `scripts/compare_topk.py` for a paired comparison against the full model. - BENCHMARKS.md carries the sweep and the calibrated result, with generations shown so the behaviour is visible, not only the number. - A dependency-free kernel benchmark, `cargo bench --bench gemm`. Quantizer - `--group-size` is usable on 7B-class models: the kernel group cap is a validation bound rather than a fixed working size, and per-row scratch scales with the group size the model uses. Release integrity - The archives build the inference product alone; a gate fails the build if any shipped binary carries a TLS or HTTP stack. - Third-party attribution is generated from the shipped crate's own graph and travels in every archive; a license, advisory and source policy gates CI. - The Python exporters read the RoPE base correctly on transformers 5 rather than defaulting it — a silent wrong-encoding bug on SmolLM2, Mistral and Llama-3. - A conversion job can no longer wedge the single conversion slot; the container's capacity limits are single-sourced and tested; a yanked dependency is updated.
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| # throughout; see the TopKWriter doc comment in cli/perplexity.rs. | ||
| dump = None | ||
| if args.dump_topk: | ||
| dump = open(args.dump_path, "wb") |
| import json | ||
| import math | ||
| import struct | ||
| import sys |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Calibrated conversion, and the quantizer under measurement
rai convert --calibration-text <file>chooses each group's quantizationparameters from the activations a projection actually receives, not from the
range its weights span. Pure Rust, no GPU, no Python; same container and same
file size as the round-to-nearest path.
Measured on SmolLM2-1.7B (wikitext-2):
Greedy generations track the full model — same facts, coherent, no
degeneration; the round-to-nearest build's repetition and one wrong fact
(a city, a date) are gone. Details and the generations are in BENCHMARKS.md.
It reaches checkpoints the Python exporter refuses (projection biases among
them) and refuses, with a named reason, the ones it cannot yet reproduce
exactly (MoE routing, sandwich/post norms, per-head QK norms) rather than
quantizing against statistics from a model that behaves differently.
Also in this branch
rai perplexityand a paired top-K comparison against the full model; thefirst quantization-cost sweep in BENCHMARKS.md.
--group-sizeusable on 7B-class models (the kernel group cap became avalidation bound); a dependency-free
cargo bench --bench gemm.rejects any shipped binary carrying a TLS/HTTP stack; generated third-party
attribution travels in every archive; the transformers-5 RoPE-base bug is
fixed; the single conversion slot can no longer wedge; capacity limits are
single-sourced and tested.
412 tests, clippy clean, byte-identity with the reference exporter preserved on
the uncalibrated path.