BUG: Fix missing bf16 Metal kernels on Apple Silicon

[codingl2k1]

Problem

Loading a model with BF16 tensors (e.g. gemma-4) on a bfloat-capable Apple GPU fails during warmup with:

E ggml_metal_library_compile_pipeline: Error Domain=MTLLibraryErrorDomain Code=5
  "Function kernel_mul_mv_ext_bf16_f32_r1_2 was not found in the library"

Root cause

The wheel builds with GGML_METAL_EMBED_LIBRARY=ON, so the Metal shader source is embedded and JIT-compiled at runtime. llama.cpp injects GGML_METAL_HAS_BF16=1 whenever the GPU reports bfloat support, but ggml-metal.metal re-checks that decision itself:

#if __METAL_VERSION__ < 310 && defined(GGML_METAL_HAS_BF16)
#undef GGML_METAL_HAS_BF16
#endif

llama.cpp never sets MTLCompileOptions.languageVersion, so Metal derives the default shading language version from the LC_BUILD_VERSION of the host executable — the user's python interpreter, not our extension. Interpreters linked against old SDKs (conda, python.org ≤ 3.12) yield MSL 2.x, where the guard above silently strips all 55 bf16 kernels from a library that still compiles successfully while the C++ side keeps has_bfloat = true. The model loads; the first BF16 matmul dies with "was not found in the library".

The same mechanism disables the M5 tensor-API path under old-SDK hosts: <metal_tensor> / MetalPerformancePrimitives require MSL 4.0, so the capability probes fail even on hardware that supports them.

Fix

patches/llama.cpp/0002-metal-pin-msl-language-version.patch — one file (ggml-metal-device.m), applied at build time by scripts/build.py:

1. ggml_metal_device_msl_version_min() maps device props to the MSL version its enabled features need: has_tensor → 4.0, has_bfloat → 3.1, else 0 (host default untouched). Raw major << 16 | minor values keep the patch buildable against older Xcode SDKs.
2. ggml_metal_compile_source() — a single shared helper used by both JIT call sites (ggml_metal_library_init, ggml_metal_library_init_from_source). It applies the pin and wraps the compile in @try/@catch: an MSL version the linked framework rejects can raise an uncaught NSException instead of populating NSError*, which would abort() the whole python process — now it is logged and treated as an ordinary compile failure.
3. bf16 probe at device init (mirrors upstream's own tensor-API probes): compiles a one-line bfloat kernel through the same helper. If the environment genuinely cannot compile what the device claims to support, has_bfloat is disabled with a warning and BF16 ops fall back to the CPU backend — instead of crashing at the first BF16 op.

Per-chip effect

Chip	has_tensor / has_bfloat	Pin	Scenario	Before	After
M1–M4	no / yes	MSL 3.1	modern host (recent-SDK python), macOS 14+	works (host default ≥ 3.1)	same — pinned to exactly 3.1, plus a one-time ~ms probe at init
M1–M4	no / yes	MSL 3.1	old-SDK host (conda, python.org ≤ 3.12), macOS 14+	crash at first BF16 op	bf16 kernels compile → works
M1/M2	no / yes	MSL 3.1	macOS 13 (MSL 3.1 does not exist there)	crash	probe fails → bfloat disabled (WARN) → BF16 on CPU, Metal keeps working
M5+	yes / yes	MSL 4.0	old-SDK host	tensor API silently disabled (probes fail)	probes request 4.0 → tensor API works; bf16 covered too
Intel / non-macOS	—	—	GGML_METAL=OFF or no Metal	—	patched file never compiled; output unchanged

Notes:

• The pin is exact (3.1/4.0), not "raise-only": deterministic shader builds regardless of host, matching what the test suite verifies.
• On macOS 13 there is no environment where bf16 can work (MSL 3.1 shipped with macOS 14), so CPU fallback loses nothing functional. GGML_METAL_BF16_DISABLE=1 remains as a manual escape hatch.
• Devices without bfloat support are never touched (ver_min == 0 → host default preserved).

Testing

tests/test_metal_bf16.py (runs on Apple Silicon CI):

1. Builds a tiny llama GGUF whose weight matrices are BF16 and loads it in-process with all layers on GPU.
2. Repeats the load under a python host whose LC_BUILD_VERSION is rewritten to SDK 11.0 (vtool), reproducing the conda/python.org failing scenario.
3. On M3+ it additionally asserts the absence of "disabling bfloat support" — a silent CPU fallback must fail the test, not pass it.

Notes for reviewers

• The patch is applied/reverted around the CMake build; the submodule checkout stays pristine. Once upstream lands an equivalent fix, delete the patch and bump the submodule — a patch that no longer applies fails the build loudly.
• Replaces the earlier 0001-metal-pin-msl-language-version.patch, which pinned 3.1 only for has_bfloat && !has_tensor (leaving the M5 path unprotected) and lacked the exception guard.

[gemini-code-assist]

Code Review

This pull request introduces a build-time patching mechanism for the vendored llama.cpp submodule, applying local hotfixes (such as pinning the Metal Shading Language version to ensure bf16 kernel support on Apple Silicon) and reverting them afterward to keep the working tree clean. It also adds comprehensive unit and regression tests. The review feedback suggests several robustness improvements: using raw MSL version values to avoid compilation failures on older macOS SDKs, wrapping the patch application in a try...except block to revert applied patches if one fails, guarding against None values for LIBDIR in tests, and asserting on subprocess return codes to improve test diagnostics.

[gemini-code-assist]

In some Python environments or virtual environments, sysconfig.get_config_var("LIBDIR") can return None. If libdir is None, compiling the host executable with clang will fail. Adding a guard to skip the test if LIBDIR is not available makes the test suite more robust.

    include = sysconfig.get_paths()["include"]
    libdir = sysconfig.get_config_var("LIBDIR")
    if not libdir:
        pytest.skip("LIBDIR is not available in sysconfig")

[iwr-redmond]

This PR is very similar to the fix used upstream. You may wish to compare the two patches for validity.

[codingl2k1]

This PR is very similar to the fix used upstream. You may wish to compare the two patches for validity.

Thanks, I’ll compare the two patches.