[ROCm] Fix HIP build on systems with partial GCC installs and off-PATH amdgpu-arch - #1
Merged
Conversation
…H amdgpu-arch Two Makefile fixes for the USE_HIP=1 build: 1. ROCm's clang selects the highest /usr/lib/gcc/<triple>/<ver> dir even when that GCC's libstdc++ headers are absent (e.g. Ubuntu installs libgcc-14-dev without libstdc++-14-dev), failing with "Could not find standard C++ header 'cmath'". Probe hipcc with a syntax-only compile and, when it fails, pin --gcc-install-dir to the newest GCC version that has matching headers under /usr/include/c++/<ver>. 2. amdgpu-arch often is not on PATH (it ships in <rocm>/llvm/bin), which made the arch auto-detection silently fall back to gfx90a and produce binaries that cannot run on the local GPUs. Locate the tool via hipconfig --rocmpath when it is not on PATH. Tested on 2x AMD Radeon AI PRO R9700 (gfx1201), ROCm 7.1.0: test_gpt2fp32cu and test_gpt2cu (default, -r 0/2, -w 0, -b 32) all pass; FP32 train_gpt2cu matches the PyTorch reference losses (loss_checker_ci max diff 0.18%); BF16 tinyshakespeare run reaches val loss 3.50 with no NaNs; CPU test_gpt2 and dev/test dataloader tests unaffected.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on the
moat-portbranch behind karpathy#854 — two small Makefile fixes for portability of theUSE_HIP=1build:Partial GCC installs break the build. ROCm's clang selects the highest
/usr/lib/gcc/<triple>/<ver>dir even when that GCC's libstdc++ headers are absent (e.g. Ubuntu 24.04 shipslibgcc-14-devwithoutlibstdc++-14-dev), failing withCould not find standard C++ header 'cmath'. The Makefile now probes hipcc with a syntax-only compile and, when it fails, pins--gcc-install-dirto the newest GCC version that has matching headers under/usr/include/c++/<ver>.Silent gfx90a fallback when
amdgpu-archis off PATH. The tool ships in<rocm>/llvm/bin, which is often not on PATH, so arch auto-detection silently fell back to gfx90a and produced binaries that cannot run on the local GPUs. Detection now locates the tool viahipconfig --rocmpathas a fallback.Testing
On 2x AMD Radeon AI PRO R9700 (gfx1201, RDNA4), ROCm 7.1.0, Ubuntu 24.04:
test_gpt2fp32cuandtest_gpt2cu(default,-r 0,-r 2,-w 0,-b 32): alloverall okay: 1— on gfx1201 all 16/16 grad tensors pass, including ln1b/lnfbtrain_gpt2cu10-step run vs PyTorch reference (dev/loss_checker_ci.py): success, max deviation 0.18%train_gpt2cuon tinyshakespeare: train loss 4.29 → 3.38, val loss 3.50, coherent samples, no NaNs, ~49k tok/stest_gpt2overall okay: 1;dev/testdataloader tests all OK