The initial measurements compared one token and 4,639 intermediate tensors with llama.cpp. Later sequence tests and source inspection corrected several initial conclusions. These notes describe the tested Bonsai model, not every model with the same architecture name. They do not establish that no other architecture documentation exists.
Use the native correctness guide for current reproduction commands and sequence tests.
| key | value |
|---|---|
general.architecture |
qwen35 (not bonsai2, not qwen3next) |
block_count |
64 |
embedding_length |
5120 |
feed_forward_length |
17408 |
attention.head_count / _kv |
24 / 4 |
attention.key_length / value_length |
256 / 256 |
full_attention_interval |
4 |
context_length |
262144 |
rope.dimension_count |
64 |
rope.dimension_sections |
[11, 11, 10, 0] |
rope.freq_base |
1e7 |
ssm.conv_kernel / state_size |
4 / 128 |
ssm.group_count / time_step_rank |
16 / 48 |
ssm.inner_size |
6144 |
| tensors / size | 851 / 6.71 GB, 26.90 B params |
Layer types. 16 layers (3, 7, 11 … 63) are full attention and carry
attn_q/k/v/output. The other 48 are gated-delta and carry attn_qkv plus
ssm_*. The naming inverts the obvious reading: attn_qkv belongs to the
SSM layers, as its own projection.
401 weights are Hadamard-folded (prism.hadamard.weight_names). The fold is
baked into the weights; the matching rotation must be applied to the
activation immediately before the matmul. Skip it and output is
uncorrelated with truth - measured cosine -0.07 on attn_qkv.
Order (llama-graph.cpp:1571):
[optional tiled->grouped head permutation, when perm_rep > 1]
x = x * signs # sign_mode = explicit
x = WHT(x) # normalized Sylvester-Walsh, block 1024, scale 1/32
y = W @ x
sign_values is 28,672 entries; sign_widths is [5120, 6144, 17408] -
one sign vector per distinct activation width: hidden, ssm-inner, ffn.
token_embd.weight is in prism.hadamard.inverse_weight_names. It stores
rotated rows, so the lookup restores the primal basis - and the order is
reversed: rotation first, then signs.
h = signs * WHT(z) # embedding lookup -> rel 0.000e+00, cos 1.00000000
h = WHT(signs * z) # forward order -> cos -0.0075, garbage
Getting this backwards poisons the residual stream from token 0.
| weight | in-width | convention | verified |
|---|---|---|---|
output.weight |
5120 | signs + WHT | argmax match, top-10 10/10 |
ffn_gate, ffn_up |
5120 | signs + WHT | cos 0.99999661 (block) |
ffn_down |
17408 | signs + WHT | same |
attn_qkv |
5120 | signs + WHT | cos 0.99999907 |
ssm_out |
6144 | permute + signs + WHT | cos 0.99999463 |
attn_output |
6144 | signs + WHT, no permute | cos 0.99999876 |
ssm_alpha, ssm_beta |
5120 | plain, NOT folded (BF16) | rel 1.6e-07 / 8.3e-08 |
The fold applies to PQ2_0 weights only. ssm_alpha and ssm_beta are BF16
and take the activation unrotated - applying the transform to them gives rel
1.03e+00 and 8.2e-01. That is consistent with weight_names having 401 entries
against 402 PQ2_0 tensors.
ssm_out needs the tiled [hd=128, nk=16, rep=3] -> grouped [hd, rep, nk]
permutation first. Without it: cos -0.11. With WHT but no permute: cos
-0.32. Both wrong, in different directions.
attn_output does NOT need the permutation, and adding it breaks the result
(cos 0.664 against 0.99999876 without). The reason is structural: attention
heads are laid out with h = kv*6 + r, which is already the grouped order, so
the permutation is an identity there. GDN's layout is genuinely tiled. Two
head-bearing activations of the same width, two different answers - which is
why each one has to be measured rather than inferred from the other.
This is the single most surprising result and the easiest way to get 48 or 16 layers silently wrong.
| path | mapping | evidence |
|---|---|---|
| full attention (24q / 4kv) | blocked, h / 6 |
4.6e-04 vs 2.3e+00 interleaved |
| gated-delta (48v / 16k) | interleaved, h % 16 |
1.3e-07 vs 1.8e+00 blocked |
Four orders of magnitude in both directions. Assuming one convention for the whole model injects a bug into one half while fixing the other.
Attention is checkable cheaply: with a single token the softmax is over one element, so each query head's output equals the V of its mapped KV head exactly.
State layout is S[dv][dk] - v-major, k-minor. Settled unambiguously: on a
fresh context the prior state is zero, so new_state must equal
beta * outer(v, k). It matches that to 1.342e-07 and outer(k, v) to
1.0e+00.
Gates. The alpha and beta tensors are RAW pre-activations
(ggml-cuda/gated_delta_net.cu:92-114):
beta = sigmoid(beta_raw)
g = exp( ssm_a[h] * softplus(alpha_raw[h] + ssm_dt_bias[h]) )
Using them raw gives rel 4.1e+00. Typical values: g ~ 0.9993 (near-unity decay), beta ~ 0.077.
Recurrence, validated at 9.169e-08:
kv = S k [dv]
delta = (v - g*kv) * beta [dv]
S_new = g*S + delta (x) k [dv][dk]
o = (S_new q) * scale [dv]
The delta term is what makes this not linear attention: it subtracts what the
state already predicts for k before writing v, so a repeated key overwrites
instead of accumulating.
State cost: 151.0 MB (48 layers x 48 heads x 128 x 128 f32) plus ~5.9 MB of conv state. Constant in context length - which is why 3/4 of this model costs nothing extra as context grows.
Pre-recurrence chain: attn_qkv -> depthwise conv1d (K=4, validated
2.6e-08) -> SiLU (6.9e-08) -> split q 2048 / k 2048 / v 6144.
q and k arrive already L2-normalised per head (norms exactly 1.0, and
q_conv_predelta is bit-identical to the first half of qk_conv_l2).
Conv window convention. conv_input is [K=4, channels] with the current
token in the LAST slot: at position 0 it is [0, 0, 0, x], cols 0-2 exactly
zero. Standard causal depthwise conv with an empty state.
Gate sources. alpha and beta are projections of the attn_norm output
through the BF16 ssm_alpha/ssm_beta weights, unrotated.
Post-recurrence: silu(z) * rmsnorm_per_head(o, ssm_norm) -> ssm_out.
Validated at 9.788e-08, cosine 1.00000000.
The ssm_norm epsilon is scaled by the head dimension. This is the single subtlest thing in the whole model:
eps_ssm = head_v_dim * rms_eps = 128 * 1e-6 = 1.28e-4
With plain rms_eps the cosine is 0.9279; with dv * rms_eps it is exactly
1.00000000. Solved independently on all 48 heads: mean 1.279996e-04, std
5.63e-09.
How it was found: the error ratio was ~1.0 for heads with large ||o|| and
diverged for small ones, which is the fingerprint of an epsilon mismatch rather
than a structural error. Solving 1/sc^2 - mean(o^2) per head gave the same
constant 48 times.
dimension_sections = [11, 11, 10, 0] sums to 32, or rope_dim / 2.
Only the first 64 dimensions of each 256-dimension head receive the rotation.
On the text path, every section uses the same position. The rotation therefore reduces to standard RoPE.
The inspected llama.cpp revision rejects K-shifts when n_pos_per_embd() > 1.
A text-only implementation can express the position rotation without that restriction.
This observation does not establish correct cache reuse after a context edit. The changed prefix also changes GDN recurrent state and later hidden states. Position shifts alone cannot restore those states. Correct reuse requires valid state checkpoints and recomputation of dependent tokens. The native Pulse decoder does not implement this cache reuse path. The earlier claim of an established engine capability advantage is withdrawn.
16 attention layers x 4 kv-heads x 256 dim = 134.2 MB per 1k tokens (f32).
| context | f32 | f16 |
|---|---|---|
| 32k | 4.3 GB | 2.1 GB |
| 256k | 34.4 GB | 17.2 GB |
The f16/256k figure independently confirms the 17 GB number this project quoted from an unrelated derivation.
| quantity | value |
|---|---|
| spec peak (256-bit LPDDR5X @ 8533) | 273 GB/s |
| achievable at a 6.70 GB working set | 216 GB/s |
| llama.cpp decode | 183 GB/s (85% of achievable) |
| best Pulse matvec (v4) | 176 GB/s (96% of llama.cpp) |
Four kernel iterations: 31.3 -> 144.6 -> 169.0 -> 176.0 GB/s. Gains +113, +24, +7 - asymptotic to llama.cpp, approached from below.
Splitting the qs/scale arrays measured 12% SLOWER (155.3 GB/s), bit-identical output. GGUF's 34-byte interleaving puts each scale in a cache line the warp is already fetching; separating them creates a second stream. The layout is a locality optimisation, not a compromise.
Conclusion: an engine is buildable and is not where throughput is. Build it for capability (RoPE/KV ownership, scheduling), not speed.
Five architectural facts above were invisible at op level and fatal in composition. Every one was caught the same way:
- Dump ground truth first.
cb_evalgives every named intermediate. A forward pass is 64 layers; without per-stage references a wrong answer tells you nothing about where. - Run candidates, print all of them. Every check here is an A/B/C, never an assertion. Costs a few lines; caught the Hadamard fold, the ffn width, the qkv fold, both head mappings, and the ssm_out permutation.
- Scale error by
sum|terms|, not by the result. These dot products have condition number ~6e3. Relative-to-result flagged a correct kernel as failing at 2.3e-05; scaled properly it was 1.2e-08, below fp32 epsilon. - Exploit degenerate cases. A zero initial state turns the recurrence into a pure outer product and pins the layout exactly. One token makes attention softmax trivial and exposes the head mapping directly.
- Validating against your own reference proves nothing about the model. Step 8's GDN kernel passed its own test while being wrong three ways.
- Read the error's SHAPE, not just its size. The ssm_norm epsilon showed up as an error that vanished for large-magnitude heads and grew for small ones. That pattern says "additive constant in a denominator", which turns a search over conventions into solving one equation.
Pulse runs this model end to end and predicts the same token llama.cpp does:
embedding -> 64 layers -> final norm -> output head
argmax pulse 198 (8.6358) llama.cpp 198 (8.6142) MATCH
top-10 ranking agreement : 10/10
logit cosine : 0.99999318
worst per-layer cosine : 0.99997288 (layer 62 of 64)
The sweep carries Pulse's own residual stream through all 64 layers and never re-seeds from the reference. Error does not accumulate - layer 32 (0.99999704) is tighter than layer 2 (0.99999283).
At a single token the softmax is over one element and Q, K, the q/k norms and
RoPE all cancel. A two-token reference exercises them. Validated on layer 3,
both tokens, all 24 heads, against attn_pregate-3:
| Q source | RoPE | worst rel |
|---|---|---|
Qcur |
no | 9.122e-02 |
Qcur |
yes | 1.613e-01 |
Qcur_normed |
no | 1.043e-01 |
Qcur_normed |
yes | 6.877e-04 |
Order is: q_norm / k_norm FIRST, then RoPE, then attention. Both operands
are normalised before rotation; using the pre-norm Qcur fails at 9e-02 whether
or not RoPE is applied. At token 1 the softmax weights are 0.0657 / 0.9343, so
both keys are genuinely live.
That also confirms the RoPE implementation in composition at a nonzero position, which the single-token run could not do.
The original two-token reconstruction treated strided Q/K/V views as contiguous arrays. That error produced a relative state error of 0.886557940360. The corrected view interpretation produced 0.0000000803380473226 with the same recurrence equations and weights. The earlier conclusion that sequential recurrence cannot reproduce this state is withdrawn.
The reference dumper now writes contiguous logical values and records the layout contract. Do not use the old packed-array interpretation to infer an architectural difference.
Source inspection at llama.cpp commit 999b0a9a6f2fb3e3b60bb3d6bbac4b6b0c0f7fd7 confirms two multi-token implementation routes.
delta-net-base.cpp selects the fused operation when fused_gdn_ch is enabled.
The CUDA implementation in gated_delta_net.cu iterates over tokens inside one kernel.
The unfused route uses the separate chunk algorithm.
The configuration name alone does not prove that the fused implementation uses a parallel scan.
Native Pulse can process prompt token IDs through its sequential decoder. It does not yet provide a complete batched prefill path. A multi-token recurrent kernel is one possible first step. Complete batched prefill also needs matrix operations, convolution, causal attention, and state scheduling. See the native correctness guide for the current validated execution scope.