Add llguidance-based constrained generation (@huggingface/transformers-llguidance)

[nico-martin]

Adds grammar-constrained generation to Transformers.js via a pure JavaScript implementation of llguidance, so model output can be forced to match a JSON schema, a regex, or generic JSON guaranteed at the token level, not by prompting.

The integration is split into a minimal core hook and a separate opt-in package, so the core stays free of any llguidance dependency:

1. Core: post-sampling hook (@huggingface/transformers)

• New optional LogitsProcessorList.onTokensSampled(token_ids, input_ids) hook, dispatched after each generation step to every processor that implements it. This lets a logits processor observe which token was actually sampled. Required for grammar engines that advance their state per committed token.
• The generation loop in modeling_utils.js appends the sampled tokens for the full batch, then fires the hook with the post-append state. Behavior is unchanged for existing users. Processors without the hook are unaffected.
• logger is now exported from the main entry point.
• New core tests: hook ordering/batching and custom StoppingCriteria support.

2. New package: @huggingface/transformers-llguidance

LlguidanceConstraint.fromResponseFormat(tokenizer, response_format) accepts json_object, json_schema, or regex formats and returns { logits_processor, stopping_criteria, stats, dispose } ready to pass to generate():

• Each step, the llguidance interpreter computes a packed token bitmask (zero-allocation via computeMaskInto into a reused Uint32Array), and disallowed tokens are banned by writing -Infinity into the logits. The mask application is optimized for skewed grammar masks: fully-allowed 32-token words are skipped, fully-banned words use fill, and only mixed words pay per-bit cost.
• Sampled tokens are committed back to the interpreter through the new hook. The stopping criteria ends generation once the grammar accepts.
• On acceptance reported by the mask, logits are constrained to the tokenizer's EOS token(s) so no unconstrained token can leak into the output.
• Fails closed everywhere: unsatisfiable grammars (dead_end), requested backtracking, missing vocab/EOS metadata, and batch sizes other than 1 all throw clear errors instead of silently producing invalid output.
• The llguidance interpreter is disposed automatically on terminal states and error paths; dispose() (idempotent) covers early exits such as max_new_tokens.
• stats exposes per-run timings (computeMaskMs, applyMaskMs, commitTokenMs), step count, stop reason, and llguidance mask-cache instrumentation.

Demo

import { pipeline } from "@huggingface/transformers";
import { LlguidanceConstraint } from "@huggingface/transformers-llguidance";

const pipe = await pipeline("text-generation", "onnx-community/gemma-4-E2B-it-ONNX", {
  device: "webgpu",
  dtype: "q4f16",
});

const { logits_processor, stopping_criteria } =
  await LlguidanceConstraint.fromResponseFormat(pipe.tokenizer, {
    type: "json_schema",
    json_schema: {
      type: "object",
      properties: {
        answer: { type: "string" },
      },
      required: ["answer"],
      additionalProperties: false,
    },
  });

const output = await pipe(
  [
    {
      role: "system",
      content: "You are a helpful assistant.",
    },
    { role: "user", content: "Answer with a short friendly greeting." },
  ],
  {
    max_new_tokens: 64,
    logits_processor,
    stopping_criteria,
  },
);

console.log("=== ANSWER ===");
console.log(JSON.parse(output[0].generated_text.pop().content));

await pipe.dispose();

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[Copilot]

Pull request overview

This PR adds an opt-in llguidance integration for grammar-constrained generation in Transformers.js, supported by a new core post-sampling hook so constraint engines can advance state based on the actually-sampled token.

Changes:

• Core: add LogitsProcessorList.onTokensSampled(token_ids, input_ids) and invoke it once per generation step after tokens are appended.
• Core: export logger from the main entry point and add generation tests for the new hook + custom stopping criteria.
• New package: introduce @huggingface/transformers-llguidance (TypeScript) implementing LlguidanceConstraint with masking, stopping criteria, stats, and unit tests; add llguidance@0.2.0 dependency.

Reviewed changes

Copilot reviewed 15 out of 17 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
pnpm-lock.yaml	Adds the packages/transformers-llguidance workspace importer and llguidance@0.2.0 dependency lock.
packages/transformers/tests/utils/generation.test.js	Adds tests validating the post-sample hook behavior and custom StoppingCriteria usage.
packages/transformers/src/transformers.js	Exports logger from the public entry point for external integrations.
packages/transformers/src/models/modeling_utils.js	Invokes the new post-sampling hook after appending sampled tokens each step.
packages/transformers/src/generation/logits_process.js	Adds LogitsProcessorList.onTokensSampled dispatcher to call processor hooks.
packages/transformers-llguidance/tsconfig.json	New TS build config to emit declaration files for the new package.
packages/transformers-llguidance/tests/llguidance-constraint.test.js	Adds unit tests for the constraint logic (masking, stopping, disposal, failure modes).
packages/transformers-llguidance/src/utils/types.ts	Defines shared state/stat types for the llguidance integration.
packages/transformers-llguidance/src/utils/mask.ts	Implements packed mask application and EOS-forcing utilities.
packages/transformers-llguidance/src/LlguidanceConstraint.ts	Implements LlguidanceConstraint with logits processor hook integration, stopping criteria, and stats.
packages/transformers-llguidance/src/index.ts	Exposes package public exports/types.
packages/transformers-llguidance/scripts/dev.mjs	Adds a watch-mode build script (esbuild + tsc watch) for local development.
packages/transformers-llguidance/scripts/build.mjs	Adds esbuild bundle script for ESM/CJS outputs.
packages/transformers-llguidance/README.md	Documents package usage and constraint modes (json_object/json_schema/regex).
packages/transformers-llguidance/package.json	Adds the new package manifest, build/test scripts, deps/peerDeps, and publish metadata.
packages/transformers-llguidance/jest.config.mjs	Adds Jest config for the new package tests and coverage settings.
.gitignore	Ignores *.local.* files repository-wide.

Files not reviewed (1)

• pnpm-lock.yaml: Generated file

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[nico-martin]

I tested two thing.

A rather long generation:

const messages = [
  {
    role: "system",
    content: "You are a helpful cooking assistant. Return practical, realistic recipes that match the requested format.",
  },
  {
    role: "user",
    content: "Create a weeknight vegetarian pasta recipe with pantry ingredients."
  }
];

And also a really short one:

const messages = [
  {
    role: "system",
    content: "You are a helpful AI assistant.",
  },
  {
    role: "user",
    content: "Answer with a short friendly greeting."
  }
];

Yes, the constrains make the geneeation slower, but I'd say it's within an expected and acceptable range

[nico-martin]

@xenova, I've completely rewritten the response-constraints.
It does not require any dependencies and it is much faster than before. I also ran benchmarks with different tasks and models (ONNX). And as it stands now the constraints add suuuper little overhead:

gemma-4-E2B

simple JSON schema

{"answer": "Hello! How can I assist you today?"}

regex

Hi there!

complex JSON schema

{"answer": {"text": "Hello! How can I assist you today?", "tone": "friendly", "language": "en"}, "alternatives": ["Hi there!", "Greetings!", "Hey!"], "metadata": {"confidence": 98, "safe": true, "tags": ["greeting", "welcome", "english"]}}

unconstrained

Hello! How can I help you today? 😊

LFM2.5-2.6B

simple JSON schema

{"answer": "string (minLength 1, maxLength 120)}"}

regex

Hello.

complex JSON schema

{"answer": {"text": "Hello! How can I help you today?", "tone": "friendly", "language": "en"}, "alternatives": ["Hi there!", "Greetings!", "Hey!"], "metadata": {"confidence": 95, "safe": true, "tags": ["greeting", "friendly"]}}

unconstrained

The user wants a short friendly greeting. I should respond with a simple, warm greeting.</think>Hello! How can I help you today? 😊

granite-4.0-h-350m-ONNX

simple JSON schema

{"answer": "Hello!"}

regex

Hello!

complex JSON schema

{"answer": {"text": "Hello", "tone": "formal", "language": "en"}, "alternatives": ["friendly", "playful", "serious"], "metadata": {"confidence": 95, "safe": true, "tags": ["formal", "playful", "serious"]}}

unconstrained

Hello!

Timing (median ms/token, steady state)

constraint	gemma-4-E2B ms/tok	gemma-4-E2B engine	gemma-4-E2B overhead	LFM2.5-2.6B ms/tok	LFM2.5-2.6B engine	LFM2.5-2.6B overhead	granite-4.0-h-350m-ONNX ms/tok	granite-4.0-h-350m-ONNX engine	granite-4.0-h-350m-ONNX overhead
simple JSON schema	26.86	0.07	0.23	9.48	0.06	0.16	20.18	0.1	-2.49
regex	26.82	0.215	0.19	9.77	0.115	0.45	21.28	0.09	-1.39
complex JSON schema	28.3	0.215	1.67	9.35	0.11	0.03	20.75	0.105	-1.92
unconstrained	26.63			9.32			22.67