Skip to content

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

4 Commits

Folders and files

Repository files navigation

GPT_at_CPU

Shakespeare Text Generation in 120 Seconds on CPU

No GPU needed. Train a GPT language model on Shakespeare and generate text — all on CPU in 2 minutes.

Results

Metric Karpathy's nanoGPT (GPU) GPT_at_CPU (CPU)
BPB 2.10 2.03
Training Time ~82s (GPU) 120s (CPU)
Hardware GPU CPU only
Parameters - 2.19M

CPU'da 120 saniyede Karpathy'nin GPU baseline'ını geçtik.

Key Innovations

1. LazySmart Optimizer

Combines three techniques that beat AdamW by 14.4%:

  • Shampoo-lite: Lazy preconditioner updated every 10 steps
  • Lookahead: Slow weight sync every 5 steps (α=0.5)
  • P3: Layer-wise gradient normalization

2. ParallelBlock (PaLM-style)

Attention and MLP run in parallel, sharing LayerNorm:

x_out = x + Attention(LN(x)) + MLP(LN(x))

3. Hybrid_Inno

LazySmart + ParallelBlock + Compass — the winning combination.

Quick Start

# Install
pip install torch tiktoken numpy

# Train + Generate (120 seconds)
python gpt_at_cpu.py

# Custom training time
python gpt_at_cpu.py --time 300

# Custom prompt
python gpt_at_cpu.py --prompt "ROMEO:"

Sample Output (120s CPU training)

KING HENRY:
Here, by the'd You will do you,
Or not to that And all your hands.
KING EDWARD:
The noble lord; I to us, as what I shall,
My lord'st you to the king,
My lord, I do, I know the;
KING RICHARD III:
That not: No, and thou thou, and so, nor this,
The world with a name,

Configuration

Default (best quality):

  • 3 layers, 192 dim, 6 heads = 2.19M params
  • Block size: 64 tokens
  • LR: 0.01, AdamW + LazySmart

50-layer variant (for research):

python gpt_at_cpu.py --n_layer 50 --n_embd 32 --n_head 2

Files

File Description
gpt_at_cpu.py Complete training + generation script
paper_predgrad_lazysmart.docx Research paper

Metrics Explained

  • BPB (Bits-Per-Byte): Tokenizer-independent compression quality. Lower = better.
  • Loss: Cross-entropy loss on validation set.
  • Accuracy: Next-token prediction accuracy.

Citation

@misc{gpt_at_cpu_2026,
  title={GPT_at_CPU: Efficient Language Model Training on CPU},
  author={umiteknoloji},
  year={2026},
  url={https://github.com/umiteknoloji/GPT_at_CPU}
}

License

MIT

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages