Qwenium

The inference engine that shows its work.

A self-contained C++ engine for the Qwen and Gemma families on ggml, serving up to 10 concurrent users from a single 32 GB Mac. Two model families on one cross-family architecture. Vision. Speculative decoding with a trained MTP head. Grammar-leashed output. Portable session snapshots. Warm conversational serving. Metal throughout, with custom fused DeltaNet kernels.

Every answer comes with the working notes behind it: which parts of your input it used, which parts it never touched, and a byte-identical re-run whenever you want to check. No other local engine does this.

With --flash-attn --persistent-graph decode is within 21% of llama.cpp on the same ggml revision (see Performance for the history — this gap has moved with upstream ggml churn, not with anything in this engine); prefill and memory footprint match it, and the forward pass agrees token-for-token with HuggingFace transformers — the model's own definition — not merely with another engine.

Three things it is built to be:

Fast where it counts. Metal throughout, batched multi-slot decode, three tiers of KV reuse. See Performance for the numbers, ours and theirs.

Readable and changeable. One directory per layer type, one file per model recipe, tests co-located with their module. Adding a model is a recipe, not a refactor; adding a layer type is a file, not a factory. The architecture is a written contract (CLAUDE.md, docs/architecture.md), enforced by a cross-family rule: no forward-pass interface is done until it hosts both a Qwen and a Gemma recipe without bending.

Auditable. The part below.

Built for receipts

An engine that wants to show its work needs three things from the forward pass, and Qwenium exposes all three as engine primitives:

• Where it looked — attention. By default every decode step's attention is materialized in the graph (not fused away) and tappable by name (kq_soft.<il>) at zero extra compute, on every recipe, Qwen and Gemma alike. Taps can select single heads, read whole prefill blocks, and run on a truncated stack (truncate_after_layer) or a partially loaded model (max_blocks), so a reader pays only for the layers it reads. --flash-attn fuses the softmax into one kernel that never writes those rows down; the attention implementation is a per-phase setting, so a client can keep the tapped phase materialized and still run a flash prefill.
• What decided it — determinism. Greedy decode is byte-deterministic, with forkable state (warm prefix restore + recurrent-state checkpoints). Remove one line of the input, re-run, and the diff is fact, not sampling noise. On MoE models the expert selection can be captured and replayed, so a re-run routes exactly as the original did.
• Proof it happened — integrity. A weights hash computed at load, version-gated snapshots, kernel-path tags, and fail-loud replay refusal: a generation can carry a kilobyte witness and re-run byte-identical on demand.

What a reader does with these primitives — and whatever per-model calibration that takes — is the client's business, not the engine's. Clients build this engine as a library and read the forward pass through one narrow API, engine/readout: readout::prefill / readout::decode with a per-call spec (layer cut, attention taps, logits, MoE routing capture / replay), every override scoped to the call and restored even when it throws (src/engine/readout.h). The engine's side of the relationship is a written contract: the change classes that can move a client's results, and the rule that such changes land only when every registered client's gate passes (docs/architecture.md §11).

What it does

• Multi-family support — Qwen 2/2.5/3/3.5/3.6/3.8 and Gemma 1/2/3/4, sharing the same layer modules and serving stack.
• Hybrid architectures — Pure transformer (Qwen3, Gemma), attention + SSM/GatedDeltaNet (Qwen3.5/3.8, one recipe), DeltaNet + attention + MoE (Qwen3.6), and per-layer embeddings + sliding-window attention (Gemma 3/4).
• Attention taps — the post-softmax attention rows of any layer and head, read from the same forward pass at no extra compute, on every recipe; with layer truncation, partial model load and MoE routing capture/replay for readers that need only part of the stack; clients read them through the engine/readout API. See Built for receipts.
• Vision (multimodal) — Image input for Gemma 3 (SigLIP encoder), Gemma 4 (gemma4uv projector), and the Qwen 3.5/3.6/3.8 family (qwen3vl encoder + merger, in-ViT M-RoPE) via a --mmproj GGUF, in both the CLI (--image) and the server (OpenAI image_url content).
• Batched inference — Slot-based KV cache with batched decode, up to 10 concurrent slots; the Qwen DeltaNet hybrids enforce that ceiling with a fail-loud guard rather than a silent degradation (aborts at 11 slots on Qwen 3.6, 15 on Qwen 3.5/3.8). Throughput gain is strongly model-dependent — ~4× measured on Qwen 2.5 Coder 14B Q4 (dense transformer), only a ~1.6× ceiling on the Qwen 3.6 hybrid; see Performance.
• KV caching, three opt-in tiers — --prefix-cache (disk-backed warm KV for a recurring system prompt, survives restarts), --chat-prefix-cache (transparent reuse of a slot's KV when a chat history re-arrives as a strict prefix; measured 2.4× per-turn at 4K context), and --conversational (explicit conversation_id handle: the server keeps the conversation's KV warm and clients send only the new turn). Image variants: --image-embed-cache, --image-prefix-cache.
• OpenAI-compatible API — /v1/completions and /v1/chat/completions with SSE streaming, per-request sampling params, and a grammar field for constrained output. Drop-in for existing tools.
• Grammar-constrained generation — GBNF grammars with a precomputed token-trie for fast constrained decoding; the valid-token set also drives a sparse LM head (skip logits for illegal tokens).
• Session snapshots — Save a mid-generation session to a portable file and resume it (--save-session / --load-session), byte-faithful across processes.
• Speculative decoding — Two draft sources: prompt-lookup (--speculative / --speculative pld, no draft model needed) and a trained MTP/NextN head (--speculative mtp, --mtp-max-draft, Qwen 3.6 only). MTP is experimental: 74–92% acceptance, ~3.3 tokens/step, but end-to-end throughput is only ~baseline on M1 Pro until the per-head dispatch overhead is addressed.
• Flash attention (--flash-attn) — opt-in, on prefill and decode, every recipe. One fused kernel replaces the materialized kq → softmax → kqv chain and the V transpose. Worth 5–32% of a decode step depending on the model, and it grows with prompt length on prefill (~7% at 756 tokens, ~55% at 3000) because materialized attention is O(n²) and this is not. Token-stable, not byte-identical — so it stays opt-in; it never writes the attention rows the taps read.
• Metal acceleration — Apple Silicon via ggml's Metal backend, with custom fused DeltaNet kernels (patches/) for Qwen 3.5 / 3.6 decode.

Supported Models

Model	Architecture	Quant	Memory	Status
Qwen 2.5 Coder 14B Instruct	Transformer	Q4_0	~9 GB	✅ Primary target
Qwen 3 1.7B / 0.6B	Transformer	BF16 / Q8_0	~3.4 GB / ~0.6 GB	✅ Tested
Qwen 3.5 (9B / 4B / 0.8B)	Attention + SSM	Q4 / Q8 / BF16	~9 GB / ~4 GB / ~1.4 GB	✅
Qwen 3.8 (27B / 9B)	Attention + SSM, same qwen35 recipe as 3.5; 27B carries a NextN/MTP head (not yet used for drafting) and vision (qwen3vl mmproj)	Q3_K_M / Q8_0	~14 GB / ~9 GB	✅
Qwen 3.6 35B-A3B	DeltaNet + Attention + MoE; vision via qwen3vl mmproj	Q2 / Q3	~13–16 GB	✅
Gemma 1 (2B / 7B)	Transformer	Q4 / Q8	~1.5 GB / ~5 GB	✅
Gemma 2 (2B / 9B)	Transformer + logit soft-cap	Q4 / Q8	~1.5 GB / ~6 GB	✅
Gemma 3 (4B)	Transformer + sliding-window; vision via SigLIP mmproj	Q4 / Q8	~3 GB	✅
Gemma 4 (12B / 26B-A4B)	Transformer + PLE + pruned RoPE (dense / MoE); vision via gemma4uv mmproj	Q8 (12B) / Q2–Q4 (MoE)	~12 GB / ~10–16 GB	✅

Quick Start

# Build (macOS)
mkdir build && cd build
cmake -DGGML_USE_METAL=ON -DCMAKE_BUILD_TYPE=Release ..
make -j$(sysctl -n hw.ncpu)

# Build (Linux)
mkdir build && cd build
cmake -DCMAKE_BUILD_TYPE=Release ..
make -j$(nproc)

Start the server

./bin/qwenium-server --model path/to/model.gguf --port 8080
# optional: --mmproj vision.gguf (image input) · --slots N · --ctx N
#           --prefix-cache DIR | --chat-prefix-cache | --conversational
#           --flash-attn (faster)

# Generate
curl -s http://localhost:8080/v1/completions \
  -H "Content-Type: application/json" \
  -d '{"prompt": "Write hello world in Python:", "max_tokens": 100}'

# Stream
curl -N http://localhost:8080/v1/completions \
  -H "Content-Type: application/json" \
  -d '{"prompt": "Explain quicksort:", "max_tokens": 200, "stream": true}'

# Chat (OpenAI messages format; add "grammar" for GBNF-constrained output,
# or an image_url content part when the server was started with --mmproj)
curl -s http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"messages": [{"role": "user", "content": "Fix this bug: ..."}], "max_tokens": 200}'

# Warm conversation (server started with --conversational): pass
# "conversation_id": "new" on the first turn, then reuse the returned id and
# send only the new user turn. DELETE /v1/conversations/{id} to clear.

Or use the CLI

./bin/qwenium model.gguf --chat                  # interactive
./bin/qwenium model.gguf --chat --flash-attn     # fused attention (prefill + decode)
./bin/qwenium model.gguf --chat --speculative    # with speculative decoding (PLD; add "mtp" on Qwen 3.6)
./bin/qwenium model.gguf -p "Hello, world!"      # single prompt
./bin/qwenium model.gguf --chat \
    --mmproj vision.gguf --image photo.jpg       # ask about an image (Gemma 3/4, Qwen 3.5/3.6/3.8)

Architecture

Qwenium is organized by concept, not by model. Each layer type is an independent module that builds into a shared ggml_cgraph. Models are recipes that compose layer modules.

src/
├── layers/                      # Layer-type modules (build graphs, share one ggml_context)
│   ├── attention.*              #   Plain + gated attention (build_attention, build_gated_attention)
│   ├── deltanet.*               #   Gated DeltaNet (Qwen3.5/3.6 recurrent path)
│   ├── ffn.*                    #   SwiGLU FFN
│   ├── moe.*                    #   Mixture-of-Experts dispatch (3x mul_mat_id)
│   ├── norm.*                   #   Shared RMS-norm helper (+ Gemma (1+w) variant)
│   ├── ple.*                    #   Per-layer embeddings (Gemma 4)
│   ├── layer.h                  #   Shared vocabulary: Phase, AttentionKind, LayerSpec
│   └── transformer_block.*      #   Per-layer composer (norm → attn → FFN → residual)
├── state/                       # Per-sequence state (KV + recurrent)
│   ├── kv_cache_simple.*        #   ggml-backed slot KV cache (append semantics)
│   ├── deltanet_state.*         #   Recurrent state for GatedDeltaNet
│   └── recurrent_state.*        #   Base for checkpoint/restore semantics
├── models/                      # Model recipes (compose layer modules)
│   ├── forward_pass_base.*      #   Shared graph scaffolding + the recipe interface
│   ├── model_registry.*         #   GGUF arch string → factory + inventory validator
│   ├── qwen3.*                  #   Qwen2/2.5/3 (pure transformer)
│   ├── qwen35.*                 #   Qwen3.5 / 3.8 (attention + SSM)
│   ├── qwen36.*                 #   Qwen3.6 (DeltaNet + attention + MoE)
│   └── gemma1-4.*               #   Gemma family (logit soft-cap, sliding-window, PLE, pruned RoPE)
├── graph_inputs/                # Typed graph inputs (tokens, positions, masks, image embeddings)
├── vision/                      # Image → soft-token encoders (SigLIP, gemma4uv, qwen3vl)
├── engine/                      # The loaded model + orchestration of one step over it
├── session/                     # Session persistence: snapshot format + slot/prefix/image caches
├── loader/                      # GGUF parsing, mmap tensor loading, tokenizer, chat templates
├── sampling/                    # Sampling, grammar, token-trie, speculative decoding
├── server/                      # Inference server + OpenAI-compatible HTTP API
├── image/                       # Host-side image pipeline (decode/resample/normalize, marker expansion)
└── cli/                         # Chat + single-prompt CLI

One inference thread owns the model. HTTP threads push requests to a queue and block on per-request token queues. All active slots decode together in a single batched forward pass.

For the full as-built map (codemap, dataflows, invariants), see docs/architecture.md; for the blueprint and rationale, see docs/modular-layer-architecture.md.

Research & Experimentation

The small, modular codebase makes this useful for research that's painful in larger frameworks:

• Layer skipping — Comment out a build_* call in the model recipe, run the test suite, see what breaks.
• Logit lens — Project hidden states through the LM head after each layer to watch the model "decide".
• Attention head pruning — Zero out heads and measure impact on downstream tasks.
• Activation patching — Swap activations between runs to isolate what drives specific outputs.
• New layer types — Drop a file in src/layers/ and a unit test alongside it. No factory plumbing, no inheritance hierarchies.
• Custom sampling — The sampler is ~180 lines; easy to add new strategies.

Performance

Head-to-head against llama.cpp, both Release + Metal, same GGUF file, same token IDs. M1 Pro (32 GB), Qwen3.5-0.8B BF16, 756-token prompt, 128 decode steps. Each of our runs is immediately followed by llama's, so every ratio comes from one thermal moment — this machine's per-step cost drifts ~8% with temperature and ours drifts more than llama's, which makes un-paired numbers worthless.

Current, 2026-09-14, both engines on the same ggml lineage again (ours: vendored b10964; llama.cpp: 41abbfd, 4 commits ahead of our pin — negligible). We bumped our vendored ggml this session specifically to close a vintage gap that had opened up (see history below), then re-measured:

Metric	Qwenium	llama.cpp
Prefill, 756 tok	~2150 tok/s (~2340 --flash-attn)	~2170 tok/s
Decode (batch 1)	69 tok/s (--flash-attn --persistent-graph, range 67–72) · 64 without	85 tok/s

Decode is 1.21× behind with our best config, 1.34× without. The ggml bump closed the vintage confound but did not close the speed gap — see docs/decode-gap-status.md §21 for why (the short version: whatever upstream's Metal work bought llama.cpp's decode path this cycle, our graph shape didn't capture a comparable share of it).

Gap history, all measured the same way (756 tokens, 128 decode steps, M1 Pro, interleaved runs)

date	ggml pins	best config	ratio
2026-08-30	both e85caa81e	--flash-attn --persistent-graph	1.04–1.07×
2026-09-14 AM	ours e85caa81e, llama ~3wk newer	--flash-attn --persistent-graph	1.16×
2026-09-14, post-bump	both b10964/41abbfd	--flash-attn --persistent-graph	1.21× (current)

The 09-14-AM number moved because llama.cpp's own ggml had drifted ahead of ours, not because of anything in this engine — bumping our pin to match confirmed that (llama's decode throughput didn't change across the bump; ours barely did either). Full analysis, including the porting story for our two custom Metal kernels across an upstream per-op kernel-file split, is in docs/note-ggml-upgrade-b10964.md.

Prefill parity depends on prompt length. At 756 tokens the two engines match. At 3000 tokens Qwenium is 0.95× of llama with --flash-attn and 0.74× without — llama runs flash attention by default, and the materialized path pays an O(n²) kq that flash does not. Longer prompts widen the difference, which is what makes the flag worth setting inside the 10K-context envelope, where a single high-resolution document page costs 2K–8K soft tokens.

What --flash-attn is worth varies by an order of magnitude and does not sort by family or by attention-layer count — measure your model rather than extrapolating:

Model	Shape	Decode gain
Gemma 1-2B	transformer, MQA	+4.9%
Qwen 3.6-35B-A3B	DeltaNet + attention + MoE	+7.7%
Qwen 3.5-0.8B	attention + SSM	+9.0%
Gemma 2-2B	transformer + attn soft-cap	+16.5%
Gemma 3-1B	transformer, sliding-window	+18.6%
Gemma 4-12B	transformer + PLE	+28.8%
Qwen 3-1.7B	transformer	+32.4%

Sorted by payoff the families interleave: the best result is a Qwen and the worst is a Gemma. What predicts it is the share of the step the materialized attention core occupies — diluted by hybrid DeltaNet/MoE layers that flash never touches, and by weight-bandwidth-bound models with few KV heads.

Where this sits beside faster engines. The fastest published Apple Silicon engine today is BaseRT ("Best-in-Class LLM Inference on Apple Silicon via Native Metal", July 2026), and its own limitations section is the useful part: single-device inference only, no continuous batching, no parallel decoding across multiple requests, no tensor parallelism — confirmed by its future-work section, which lists continuous batching and speculative decoding as not yet present. Its margin over llama.cpp also narrows sharply on the model classes this engine targets: 1.04× on Qwen3-30B-A3B Q4, 1.07× on Gemma-4-26B-A4B Q4 (and 0.90× — a loss — against MLX on that same model); the larger margins, up to 1.56×, are on sub-1B dense models. MLX's own continuous batching exists only as a third-party vLLM port, not in its core runtime. Qwenium is not faster than llama.cpp (see above) or BaseRT. What it does that neither does natively, on the same hardware class: serve up to 10 concurrent requests as one batched forward pass, with per-slot sampling and per-slot grammar.

Engine-level numbers, M1 Pro (32 GB), Qwen 2.5 Coder 14B Q4:

Metric	Value
Throughput (10 concurrent)	~4× vs sequential — measured on this one dense model; strongly model-dependent, down to a ~1.6× amortization ceiling on the Qwen 3.6 hybrid (note-batch-scaling-cross-family.md)
Warm chat prefix hit (--chat-prefix-cache)	2.4× per turn at 4K context (flat vs quadratic re-prefill)
Persistent decode graph (--persistent-graph)	1.32× on Qwen 3.6 35B-A3B; ~parity on small models — the win is model-specific
Flash attention (--flash-attn)	+5–32% decode, +7–55% prefill (grows with prompt length); see above
The two together	Compose cleanly — token-identical to either alone, and the fastest configuration measured
DSL code generation	100% pass rate (10 concurrent)

Build Options

Option	Default	Description
GGML_USE_METAL	OFF	Metal GPU acceleration (macOS)
QWEN3_BUILD_TESTS	OFF	Build unit + integration tests
QWEN3_ENABLE_OPENMP	ON	OpenMP parallelism

Tests

cmake -DQWEN3_BUILD_TESTS=ON -DGGML_USE_METAL=ON -DCMAKE_BUILD_TYPE=Debug ..
make -j8

./bin/unit-tests                     # per-module unit tests (no model needed)
./bin/integration-tests              # e2e generation (needs model files)
./bin/parallel-order-mngmt-test      # 10-concurrent DSL benchmark

Each module under src/<dir>/<module>.cpp has its unit test at tests/unit/test_<module>.cpp. Python benchmarks in py/ for load testing and DSL validation.

Dependencies

Fetched automatically via CMake:

• ggml — tensor ops + Metal backend
• cpp-httplib — HTTP server
• nlohmann/json — JSON
• Google Test — testing

Acknowledgements

Inspired by and built on concepts from llama.cpp and ggml. This project wouldn't exist without the foundational work by Georgi Gerganov and the ggml community.

Contributing

Issues and PRs welcome. The codebase is intentionally small and modular — please keep it that way. Locality-of-reasoning, test co-location, and concept-named directories are load-bearing rules; see CLAUDE.md for the short list.

License

MIT