Position-dependent attention bias parameterized by the query, unifying ALiBi-style decay with a per-token gating signal.
GRAPE (Group Representational Position Encoding, arXiv:2512.07805) provides a unified framework for position encodings via group actions. The Additive branch subsumes ALiBi and FoX as special cases. The query-gated variant adds a data-dependent gate: instead of a fixed decay slope per head, the penalty for attending to distant keys is modulated by the current query.
BlaGPT tested the query-gated Additive variant as a full replacement for RoPE. The hypothesis: a learned, query-dependent distance penalty could outperform fixed rotary rotations, especially when composed with the existing XSA + GOAT sink + composable gated attention stack.
For head
This bias is added to the attention scores before softmax:
scores[i, j] = q_i k_j / sqrt(d) + b_h(i, j)
attn = softmax(scores)
The components:
-
Distance
$(i-j)$ : causal, clamped to$\geq 0$ . Same triangular structure as ALiBi. -
Decay rate
$\omega_h$ : one scalar per head, parameterized throughsoftplusso it stays non-negative. Controls how aggressively distance penalizes attention. -
ALiBi slopes
$\text{slope}_h$ : the standard ALiBi geometric slope assignment per head. Provides the base multi-scale decay pattern. -
Query gate
$v_h^\top q_i / \sqrt{D}$ : a per-head projection vector$v_h$ (shape[D]) dotted with the query, then passed throughsoftplus. This makes the distance penalty content-dependent — some queries penalize distant keys more, others less. The gate is L2-normalized at initialization and runtime.
When pos_encoding="grape_a_qgate", RoPE is not applied at all. The positional signal comes entirely from the additive bias.
The implementation lives in bla_gpt/attentions.py:
-
GrapeQueryGatedBiasis annn.Modulethat precomputes distance and slope buffers atblock_size(non-persistent, compile-safe). - Parameters:
omega([n_head]) andv([n_head, head_dim]). Total overhead:$H + H \times D$ parameters. - The bias is injected into three attention paths:
- Base
Attention._flash_attention— builds a combined causal + GRAPE float mask, disables SDPA's built-inis_causal. - Base
Attention._manual_attention— adds the bias to the score matrix directly. -
GOATSinkAttentionoverrides — GRAPE composes with the key-0 sink prior in both flash and manual paths.
- Base
-
ComposableGatedAttentioninherits fromGOATSinkAttention, so the full stack (XSA + GOAT sink + gated output + GRAPE) works automatically.
Configuration:
pos_encoding = "grape_a_qgate"
attention = "xsa"
use_goat_sink_prior = True
use_composable_gated_attn = True
# optional hyperparameters (defaults shown):
grape_omega_init = 1.0 # initial softplus-inverse target for omega
grape_v_init_std = None # None => 1/sqrt(head_dim)
grape_v_l2_norm = True # L2-normalize v at runtimeFull 5100-step autoresearch runs:
| Run | Change | Validation loss | Delta | Step avg | Peak memory |
|---|---|---|---|---|---|
| F90r2 | lr ×1.4 confirmed (RoPE) | 3.1965 |
— | 491ms |
58632 MiB |
| F92 | (run artifacts lost; not in the ledger) | 3.1925 |
-0.0040 |
— | — |
| F94 | GRAPE-A query-gated replaces RoPE | 3.1897 |
-0.0068 |
580ms |
59643 MiB |
F94 is a clean keep and the new trusted best. The +18% step-time overhead (580ms vs 491ms) comes from replacing the flash causal fast-path with an explicit (B, H, T, T) float attention mask — the same cost as GOAT sink's masked path, just applied at every layer instead of only where the sink is active.
RoPE is not the only way to encode position. A learned additive bias that adapts to the query — "should this token care about distant context?" — outperformed fixed rotary rotations at the same training budget. The mechanism is cheap in parameters (a few hundred) and composes cleanly with the existing attention stack. The wall-clock cost is real but bounded: it is an attention-mask problem, not a parameter-count problem.
Paper: GRAPE: Generalizing Position Encoding via Group Representations
Reference implementation: model-architectures/GRAPE
Implementation: bla_gpt/attentions.py::GrapeQueryGatedBias