Skip to content

Add a Mojo GPU port (CUDA + Metal, Mojo 1.0) to notable forks - #858

Open
ulmentflam wants to merge 1 commit into
karpathy:masterfrom
ulmentflam:master
Open

Add a Mojo GPU port (CUDA + Metal, Mojo 1.0) to notable forks#858
ulmentflam wants to merge 1 commit into
karpathy:masterfrom
ulmentflam:master

Conversation

@ulmentflam

Copy link
Copy Markdown

This adds one line under the existing Mojo entry in notable forks, following
the pattern of #332.

llm.🔥 is my port of llm.c that
extends @dorjeduck's already-listed Mojo port (Mojo 25.5, CPU) with hand-written
GPU training kernels for Mojo 1.0.0b3: a CUDA/cuBLAS backend, an Apple Metal
backend, and a portable GPU path. What it adds beyond the existing entry: on a
GB10 (DGX Spark) the bf16 path is at parity with llm.c's CUDA trainer (135.97
vs 135.77 ms/step median, B=4, T=1024, GPT-2 124M), and on an M4 Max the Metal
path trains 1.72x faster than PyTorch MPS bf16 (llm.c has no Metal port, so
MPS is the baseline there). Correctness is gated by a ported test_gpt2.mojo:
16 gradient tensors plus a 10-step loss trajectory checked against PyTorch.

Benchmark methodology and reproduction: https://github.com/ulmentflam/llm.mojo#benchmarks

dorjeduck's entry is kept as is; his port is the lineage this builds on.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant