Canonical KV cache allocation + offloading integration - #1

Open

Etelis wants to merge 17 commits into

orozery:kv-offload-hmafrom

Etelis:canonical-on-offload-hma

Commits on Apr 12, 2026

Add CanonicalKVCaches data classes for HMA KV cache representation
EtelisIBM
committed
26cbb1e
Add WorkerConnectorInitializationData and initialize_worker_connector
EtelisIBM
committed
544b689
Add canonical KV cache allocation for HMA models
EtelisIBM
committed
2268cf4
Wire up canonical KV cache allocation in gpu_model_runner
EtelisIBM
committed
7b6771b
Add unit tests for canonical KV cache allocation
EtelisIBM
committed
ac68b33
Fix mypy error: rename shadowed variable in use_canonical_kv_caches
EtelisIBM
committed
ccdeea4
Move canonical KV cache dataclasses to connector base
EtelisIBM
committed
77f75b6
Address CR: relax group count check and use per-group spec
EtelisIBM
committed
69d011a
Address CR: prioritize canonical path and scope initialize_worker_connector
EtelisIBM
committed
ab66101
Address CR: merge loops in allocate_canonical_kv_caches
EtelisIBM
committed
cc3f747
Address CR: always call initialize_worker_connector
EtelisIBM
committed
3cc7485
Validate tensor sizes in use_canonical_kv_caches
EtelisIBM
committed
6f84560
Refactor canonical KV cache allocation into single-pass loop
EtelisIBM
committed
0c05217
Simplify canonical KV cache allocation using physical buffer
EtelisIBM
committed
a5282d9
Move per-layer reshape logic into inner loop
EtelisIBM
committed
d257b23
Address CR: use single cross-layers int8 tensor for canonical KV caches
EtelisIBM
committed
4be3987
Integrate canonical KV caches with offloading connector
EtelisIBM
committed
a17dae7