A playground to test language model architectures, layers and training tricks on a small budget. Each new idea goes on top of the best config. It stays only if the final val loss drops.
| Item | Value |
|---|---|
| Val loss | 3.1153 (5100 steps, 8x H200) |
| Model | 10 layers, d=768, 404.64M params (193M of them in the Engram table) |
| Main parts | Latent Lattice · RoPE + RoVE · Aurora optimizer · Engram (vocab ×5) |
| Speed | 732 ms/step, peak 70,569 MiB per GPU |
| Config | best_model_config.py · ar/best_config.json |
torchrun --standalone --nproc_per_node=8 bla_gpt/train.py --run_name best_model --model_name bestContents: Progression · Alternative architectures · Byte-level models · Optimizers · Adding a new model · Training · Techniques
Each row adds one change to the best model at that time. ✅ = kept, ❌ = discarded, 👑 = current best. — = not recorded in the original run.
| Technique | Val loss | Δ | Peak MiB | ms/step | Notes | |
|---|---|---|---|---|---|---|
| Setting base theta to 1_000_000 llama3 |
— | 3.3324 | — | — | — | increased the final validation loss |
| Z-loss regularization link |
— | 3.3527 | — | — | — | increased the final validation loss by 0.02 |
| KV-Shifting attention link |
— | 3.3138 | -0.0172 | 42858 | — | lower loss than its base (3.3310) |
| Dilated Attention (LongNet) link |
— | — | — | — | — | — |
| Multi-Head Latent Attention link |
— | 3.3479 | — | 42192 | — | — |
| Per token output bias | — | 3.3257 | — | 42120 | — | — |
| DyT Norm link |
— | — | — | — | — | loss stayed high; did not train well |
| Forgetting Transformer (Vanilla and Pro vers) link |
— | 3.3243 | — | — | — | vanilla version; the Pro version ran out of memory |
| Multi-Token Attention link |
— | 3.3357 | — | 42136 | — | — |
| Differential Attention link |
❌ | 3.2460 | +0.0049 | 41521 | — | — |
| Softpick link |
— | 3.3446 | — | 59417 | — | — |
| Canon Layer link |
— | 3.3217 | — | 43199 | — | — |
| Parallel Transformer Block link |
— | 3.3473 | — | 40302 | — | — |
| Per Layer Token Embedding link |
— | 3.2411 | — | 40916 | — | — |
| PolyNorm link |
❌ | 3.3017 | +0.0606 | 40895 | — | — |
| PolyReLU link |
❌ | 3.2642 | +0.0231 | 40890 | — | — |
| TOP loss paper · explanation |
❌ | 3.2636 | +0.0225 | 47816 | — | — |
| Simplified RoPe link |
❌ | 3.2620 | +0.0209 | 43585 | 388.54 | — |
| Gated Attention paper · explanation |
✅ | 3.2327 | -0.0084 | 45968 | 413.01 | — |
| ResFormer (Plus) paper · explanation |
❌ | 3.3538 | +0.1211 | 38223 | 326.32 | — |
| Engram (My Simple Variant) paper · explanation |
✅ | 3.2296 | -0.0031 | 50488 | 504.09 | — |
| Differential Attention v2 paper · explanation |
✅ | 3.2274 | -0.0022 | 52829 | 535.16 | — |
| Exclusive Self Attention paper |
✅ | 3.2303 | -0.0024 | 49859 | 424.54 | tested on the Gated Attention base (3.2327), in parallel with the two rows above |
| Hyper-Connections paper · explanation |
❌ | 3.2741 | +0.0414 | 68729 | 928.26 | tested on the Gated Attention base (3.2327) |
| All keeps combined (5100 steps) | ✅ | 3.2354 | — | — | — | new 5100-step base: all kept techniques above in one config (ar/full_train/combined_keeps.json); every row below uses this 5100-step setup |
| GOAT Sink Prior paper · explanation |
✅ | 3.2298 | -0.0056 | 52231 | 377.61 | — |
| GatedNorm paper · explanation |
✅ | 3.2230 | -0.0068 | 50023 | 410.68 | — |
| HybridNorm paper · explanation |
✅ | 3.2224 | -0.0006 | 50495 | 474.82 | — |
| Composable Gated Attention paper · explanation |
✅ | 3.2128 | -0.0096 | 58553 | 489.07 | — |
| Composable Value Residual paper · explanation |
✅ | 3.2011 | -0.0117 | 58988 | 497.55 | — |
| U-net Long Skips source · explanation |
✅ | 3.1979 | -0.0032 | 58632 | 494.31 | — |
| Learning rate 0.001 -> 0.0014 Muon source |
✅ | 3.1965 | -0.0014 | 58632 | 491.00 | — |
| GRAPE-A Positional Encoding paper · explanation |
✅ | 3.1897 | -0.0068 | 59643 | 580.32 | replaces RoPE; later replaced again by RoPE + RoVE |
| Aurora Optimizer paper · explanation |
✅ | 3.1603 | -0.0294 | 59643 | 721.25 | replaces Muon, learning_rate=0.03 |
| Weight EMA + LR floor | ✅ | 3.1487 | -0.0116 | 61239 | 731.65 | LR decays to 15% of peak, not to 0, and eval uses the average weights of the last 300 steps; removed in F107b_e5 |
| Token normalization for Engram PR #375 · explanation |
❌ | 3.1564 | -0.0039 | 59643 | 719.88 | tested on the Aurora base (3.1603), in parallel with Weight EMA + LR floor; tokens with the same first 2 bytes share one Engram entry; off by default |
| RoPE + RoVE + Latent Lattice (F107b_e5) RoPE · RoVE · Latent Lattice · explanation |
👑 | 3.1153 | -0.0334 | 70569 | 732.44 | 404.64M params; RoPE + RoVE replace GRAPE-A; one recurrent latent (width 256) on layers 2-7 modulates q/k/v and the MLP (FiLM after the QK norm, Lattice params in AdamW lr 1e-3, vstream); weight EMA and LR floor removed; second seed: 3.1230 |
Early techniques with no recorded loss: Multi-token prediction · Weight tying · Grouped query attention · Capping logits · QKV bias · Zero-init projection layer · Post and pre-RMSNorm.
Run logs: ar/full_results.tsv (full runs), ar/results.tsv (short runs), ar/best_config.json (best config). Technique notes: techniques/.
| Model | Val loss | Peak MiB | ms/step | Notes |
|---|---|---|---|---|
| MegaByte link |
3.810 | — | — | — |
| FTP (heavily modified) link |
3.901 | — | — | — |
| Rene link |
3.340 | — | — | — |
| Rwkv7 link |
4.450 | — | — | — |
| Zamba2 link |
— | — | — | no loss recorded; ranked Zamba2 > Rene > Rwkv7 |
| Hourglass Transformer (modified) link |
3.710 | — | — | token-level run; ranked Hourglass > MegaByte > FTP |
| Hymba link |
4.7505 | — | — | much slower per step than the Transformer |
| Tokenformer (in BlaGPT model) link |
3.390 | — | — | — |
| LLaDa (dLLM) link |
8.6930 | — | — | diffusion loss; cross-entropy 4.2891 compares with the other models (from bla_gpt/llada_validation_cross_entropy.py) |
| Avey link |
3.323 | 51962 | 2871 | batch size 8; very slow to train; more than 3x the memory of other models |
| LFM2 link |
— | — | — | TBD |
| Kimi Delta Attention (1:3 interleaved Full Attention) link |
3.2411→3.2532 | 47391 | 568.1 | — |
Val loss here is per byte, not per BPE token. Do not compare it with the tables above.
| Model | Val loss | Peak MiB | ms/step | Notes |
|---|---|---|---|---|
| Hourglass Transformer (modified) link |
1.0048 | — | 524.76 | byte-level run of the token-level model above |
| AUNet link |
1.1502 | — | 1423.60 | — |
| SpaceByte link |
1.6755 | 27781 | 423.36 | — |
| HNet link |
1.4554 | 23948 | 433.75 | — |
| Optimizer | Val loss | Peak MiB | ms/step | Notes |
|---|---|---|---|---|
| PaLMForeachSOAP link |
— | — | — | best loss of the early optimizers, but almost 2x slower than Adam |
| Ademamix link |
— | — | — | unstable at all tested learning rates |
| Adopt link |
— | — | — | loss became NaN |
| CAdamW link |
3.3517 | — | — | — |
| AdamW with independent weight decay link |
3.320 | — | — | — |
| Adam | 3.3224 | — | — | — |
| AdamW | 3.3310 | 42053 | 533 | — |
| DeMo link |
3.4676 | 41534 | 820 | saves 7 GB per GPU; higher loss and slower steps than Adam |
| Adam-Mini | 3.3324 | 41534 | 610 | higher loss and slower steps than Adam and AdamW; saves a little memory |
| MARS link |
3.3459 | 40953 | 628 | — |
| Muon link |
3.2923 | 40332 | 620.24 | — |
| AdaMuon paper · explanation |
— | — | — | Adaptive Muon with second-moment estimation |
| Aurora paper · tilde-research/aurora-release |
3.1897→3.1603 | 59643 | 721 | matrix weights use Aurora, other weights use AdamW; lr=0.03, weight_decay=0.025 |
| BiClip link |
7.2292 | 39751 | 510 | did not train well |
| NorMuon paper · explanation |
3.2411→3.4630 | 44154 | 387.46 | — |
| Cautious Weight Decay paper · explanation |
3.2327→3.2334 | 45971 | 434.2 | — |
- Implement the model. The forward function must return the loss.
- Add a function
register_<name>with the@register_modeldecorator inbla_gpt/model_registers.py. - Start training with
--model_name <name>.
See the existing functions in model_registers.py for examples.
Training scripts, quick start, datasets, tokenizers, LR finder
BlaGPT provides two training scripts:
| Script | Purpose | Data Sources |
|---|---|---|
train.py |
Quick experimentation with pre-tokenized data | Binary shards only |
train_flex.py |
Flexible training with custom datasets/tokenizers | Binary shards, HF streaming, byte-level |
For rapid architecture benchmarking with the default FineWeb10B dataset:
# Get the pre-tokenized data
python data/fineweb10B_cached.py
# Start training
torchrun --standalone --nproc_per_node=8 bla_gpt/train.py --run_name my_experiment --model_name blagptFor custom datasets, different tokenizers, or HuggingFace streaming:
torchrun --standalone --nproc_per_node=8 bla_gpt/train_flex.py \
--model_name blagpt \
--run_name my_experimentStream data directly from HuggingFace Hub - no pre-download required:
# FineWeb with GPT-2 tokenizer (tiktoken)
torchrun --standalone --nproc_per_node=8 bla_gpt/train_flex.py \
--model_name blagpt \
--run_name hf_streaming_test \
--use_hf_streaming \
--hf_dataset "HuggingFaceFW/fineweb" \
--hf_dataset_config "sample-10BT"
# C4 dataset with HuggingFace tokenizer
torchrun --standalone --nproc_per_node=8 bla_gpt/train_flex.py \
--model_name blagpt \
--run_name c4_experiment \
--use_hf_streaming \
--hf_dataset "allenai/c4" \
--hf_dataset_config "en" \
--tokenizer_backend huggingface \
--tokenizer_name "gpt2"
# Custom dataset with custom text column
torchrun --standalone --nproc_per_node=8 bla_gpt/train_flex.py \
--model_name blagpt \
--run_name custom_data \
--use_hf_streaming \
--hf_dataset "username/my-dataset" \
--hf_text_column "content"| Option | Default | Description |
|---|---|---|
--use_hf_streaming |
False |
Enable HuggingFace streaming mode |
--hf_dataset |
HuggingFaceFW/fineweb |
HuggingFace dataset name or path |
--hf_dataset_config |
sample-10BT |
Dataset configuration/subset |
--hf_split |
train |
Training split name |
--hf_text_column |
text |
Column containing text data |
--hf_val_dataset |
Same as train | Separate validation dataset (optional) |
--hf_val_dataset_config |
Same as train | Validation dataset config |
--hf_val_split |
train |
Validation split name |
--hf_val_samples |
1000 |
Number of samples for validation (first N of val split) |
--hf_shuffle_buffer |
10000 |
Shuffle buffer size for streaming |
Train/Validation Split: By default, the first 1000 samples are used for validation and training skips these samples (no overlap). When using a separate validation dataset or split, this behavior is automatically disabled.
| Option | Default | Description |
|---|---|---|
--tokenizer_backend |
tiktoken |
Backend: tiktoken or huggingface |
--tokenizer_name |
gpt2 |
Tokenizer name (encoding or model path) |
tiktoken encodings: gpt2, cl100k_base, o200k_base
HuggingFace tokenizers: Any model name, e.g., meta-llama/Llama-2-7b-hf, mistralai/Mistral-7B-v0.1
Run before training to find optimal learning rate:
torchrun --standalone --nproc_per_node=8 bla_gpt/find_lr.py --model_name blagpt
# Output
Results:
Steepest gradient learning rate: 3.31e-06
Elbow point learning rate: 1.20e-01
Plot saved to: logs/lr_finder_blagpt/lr_finder_plot.png
Results saved to: logs/lr_finder_blagpt/lr_finder_results.ptThe initial code is based on
Nano GPT - link
Modded NanoGPT - link
Thanks to @xumingyu2021 for memory friendly implementation of the Differential Attention