Shakespeare Text Generation in 120 Seconds on CPU
No GPU needed. Train a GPT language model on Shakespeare and generate text — all on CPU in 2 minutes.
| Metric | Karpathy's nanoGPT (GPU) | GPT_at_CPU (CPU) |
|---|---|---|
| BPB | 2.10 | 2.03 |
| Training Time | ~82s (GPU) | 120s (CPU) |
| Hardware | GPU | CPU only |
| Parameters | - | 2.19M |
CPU'da 120 saniyede Karpathy'nin GPU baseline'ını geçtik.
Combines three techniques that beat AdamW by 14.4%:
- Shampoo-lite: Lazy preconditioner updated every 10 steps
- Lookahead: Slow weight sync every 5 steps (α=0.5)
- P3: Layer-wise gradient normalization
Attention and MLP run in parallel, sharing LayerNorm:
x_out = x + Attention(LN(x)) + MLP(LN(x))
LazySmart + ParallelBlock + Compass — the winning combination.
# Install
pip install torch tiktoken numpy
# Train + Generate (120 seconds)
python gpt_at_cpu.py
# Custom training time
python gpt_at_cpu.py --time 300
# Custom prompt
python gpt_at_cpu.py --prompt "ROMEO:"KING HENRY:
Here, by the'd You will do you,
Or not to that And all your hands.
KING EDWARD:
The noble lord; I to us, as what I shall,
My lord'st you to the king,
My lord, I do, I know the;
KING RICHARD III:
That not: No, and thou thou, and so, nor this,
The world with a name,
Default (best quality):
- 3 layers, 192 dim, 6 heads = 2.19M params
- Block size: 64 tokens
- LR: 0.01, AdamW + LazySmart
50-layer variant (for research):
python gpt_at_cpu.py --n_layer 50 --n_embd 32 --n_head 2| File | Description |
|---|---|
gpt_at_cpu.py |
Complete training + generation script |
paper_predgrad_lazysmart.docx |
Research paper |
- BPB (Bits-Per-Byte): Tokenizer-independent compression quality. Lower = better.
- Loss: Cross-entropy loss on validation set.
- Accuracy: Next-token prediction accuracy.
@misc{gpt_at_cpu_2026,
title={GPT_at_CPU: Efficient Language Model Training on CPU},
author={umiteknoloji},
year={2026},
url={https://github.com/umiteknoloji/GPT_at_CPU}
}
MIT