Skip to content

Expose explicit fused attention selection in the C API - #9

Draft
Gajesh2007 wants to merge 1 commit into
mainfrom
feat/kv-quant-fused-prefill
Draft

Expose explicit fused attention selection in the C API#9
Gajesh2007 wants to merge 1 commit into
mainfrom
feat/kv-quant-fused-prefill

Conversation

@Gajesh2007

Copy link
Copy Markdown
Member

Summary

Expose the existing C++ force_fused attention option through a new C entry point, mlx_fast_scaled_dot_product_attention_with_force_fused. Callers that must bound temporary allocation can request a fused kernel and receive an error for unsupported shapes instead of a composed score tensor.

The existing C entry point and ABI remain available and delegate with force_fused=false.

Before

flowchart LR
  A[C attention caller] --> B[mlx_fast_scaled_dot_product_attention]
  B --> C[C++ force_fused hardcoded false]
  C --> D[Default fused or composed routing]
Loading

After

flowchart LR
  A[Existing C caller] --> B[Existing entry point]
  B --> C[New entry point with false]
  N[Caller requiring fused allocation] --> T[New entry point with true]
  C --> D[Existing default routing]
  T --> E[C++ fused kernel or explicit refusal]
Loading

Validation

Five Swift/C API tests passed on M4 Max: explicit-false equality with the existing entry point, fused pitched views/masks/sinks against a dense reference, GQA8, D512 refusal, and a no-score-materialization allocation check. The wide-head cases use the companion MLX tile correction; the Swift package pins both dependencies together.

Used by the separately reserved prefill candidate in Layr-Labs/mlx-swift-lm#142 and Layr-Labs/d-inference#860.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant