[KV Offload] Support packed HMA KV cache layout by LucasWilkinson · Pull Request #46205 · vllm-project/vllm

LucasWilkinson · 2026-06-20T01:53:53Z

Summary

add an opt-in VLLM_USE_PACKED_HMA_KV_CACHE path for multi-group HMA KV cache packing
keep the existing DeepSeek V4 packed path unchanged
register packed HMA offload as one canonical backing tensor with one full-row ref per KV group, preserving the packed topology for CPU offload

Duplicate-work check

Checked open PRs for packed HMA KV cache, gpt-oss gemma packed kv cache, VLLM_USE_PACKED_HMA_KV_CACHE, packed KV cache offloading, and hybrid KV cache offload.
Related work exists, especially [KV Connector] Canonical KV Cache Allocation for HMA Models #37885 and [KV Offload] Reshape the transfer data model: per group specs and offloaded side alignment offset #44865, but this PR is intentionally narrower: it is a flagged alternative to [KV Connector] Canonical KV Cache Allocation for HMA Models #37885 for gpt-oss/Gemma-style HMA packing and preserves [DSv4] Pack KV caches into contiguous per-block allocations for DeepSeek V4 #44577/DeepSeek V4 behavior.
[DSv4] Pack KV caches into contiguous per-block allocations for DeepSeek V4 #44577 is merged and this PR builds on that packed layout support rather than duplicating the DSV4 change.

Benchmarks

openai/gpt-oss-20b, B300, 128K, OffloadingConnector, 2 CPU-hit iterations: packed HMA full-row refs used 1 CPU tensor and averaged ~124.95 ms vs per-slice registration with 12 CPU tensors at ~144.37 ms (~13.5% faster).
google/gemma-3-1b-it, 4K: packed HMA used 1 CPU tensor vs 4 and CPU-hit latency was effectively flat, ~12.23 ms vs ~12.32 ms.

Tests

.venv/bin/python -m pytest tests/v1/core/test_contiguous_kv_packing.py tests/v1/simple_kv_offload/test_scheduler.py tests/v1/kv_connector/unit/offloading_connector/test_worker.py tests/v1/kv_offload/cpu/test_gpu_worker.py -q
.venv/bin/pre-commit run ruff-check --files vllm/v1/kv_offload/base.py vllm/v1/kv_offload/cpu/gpu_worker.py vllm/distributed/kv_transfer/kv_connector/v1/offloading/worker.py
commit hook also ran ruff check, ruff format, typos, mypy py3.10, SPDX, config validation, and other repository hooks successfully.

AI Assistance

AI assistance was used to implement and iterate on this change. This PR has been reviewed by the author.

Add an opt-in packed KV cache layout for multi-group HMA models while preserving the existing DeepSeek V4 packed path. For HMA offloading, register the packed backing as one canonical tensor and use one full-row ref per KV group so CPU offload keeps the packed topology instead of allocating/copying per-slice tensors. Benchmark notes: - openai/gpt-oss-20b, B300, 128K, OffloadingConnector, 2 CPU-hit iterations: packed HMA full-row refs used 1 CPU tensor and averaged ~124.95 ms vs per-slice registration with 12 CPU tensors at ~144.37 ms (~13.5% faster). - google/gemma-3-1b-it, 4K: packed HMA used 1 CPU tensor vs 4 and CPU-hit latency was effectively flat, ~12.23 ms vs ~12.32 ms. Tests: - .venv/bin/python -m pytest tests/v1/core/test_contiguous_kv_packing.py tests/v1/simple_kv_offload/test_scheduler.py tests/v1/kv_connector/unit/offloading_connector/test_worker.py tests/v1/kv_offload/cpu/test_gpu_worker.py -q - .venv/bin/pre-commit run ruff-check --files vllm/v1/kv_offload/base.py vllm/v1/kv_offload/cpu/gpu_worker.py vllm/distributed/kv_transfer/kv_connector/v1/offloading/worker.py Co-authored-by: OpenAI Codex <codex@openai.com> Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>

Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com> Co-authored-by: OpenAI Codex <codex@openai.com> Co-authored-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>

Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com> Co-authored-by: OpenAI Codex <codex@openai.com> Co-authored-by: Tyler Michael Smith <tlrmchlsmth@gmail.com> Signed-off-by: Qiang Li <qiang.li2@amd.com>

### What this PR does / why we need it? This PR upgrades the verified vLLM main commit to ee0da84ab9e04ac7610e28580af62c365e898389 (v0.24.0 tag) and adapts vLLM Ascend to upstream API changes introduced after the previous verified commit. Changes # Changes ### `examples/offline_data_parallel.py` - Slice `ASCEND_RT_VISIBLE_DEVICES` according to the local DP rank before creating the `LLM` instance. - Fall back to `torch.npu.device_count()` when `ASCEND_RT_VISIBLE_DEVICES` is not set. - Keep application-level DP compatible with the upstream removal of automatic device isolation. - Upstream source: [vllm#45026](vllm-project/vllm#45026). --- ### `tests/e2e/conftest.py` - Slice `ASCEND_RT_VISIBLE_DEVICES` for each DP rank before creating `LLM()` instances or worker processes. - Fall back to `torch.npu.device_count()` when the environment variable is unavailable. - Ensure each test process uses an independent NPU subset after upstream removed automatic device isolation. - Upstream source: [vllm#45026](vllm-project/vllm#45026). --- ### `tests/ut/quantization/methods/test_w4a16_mxfp4.py` - Skip the test to match the upstream behavior. --- ### `tests/ut/spec_decode/test_speculators_vwn_eagle3.py` - Mock `vllm.v1.attention.selector._cached_get_attn_backend()` for non-NPU unit tests. - Avoid attention backend initialization failures introduced by the updated v0.24.0 initialization path. - Keep spec decode unit tests runnable without physical NPU devices. --- ### `vllm_ascend/_310p/worker_310p.py` - Use `MemorySnapshot(device=device)` for non-0.23.0 releases. - Align the 310P worker with the updated worker implementation. - Related upstream changes: - [vllm#30868](vllm-project/vllm#30868) --- ### `vllm_ascend/patch/platform/patch_dp_device_ids.py` - Add a patch for `get_physical_gpu_ids_for_local_dp_rank()`. - Support pre-sharded `ASCEND_RT_VISIBLE_DEVICES` by avoiding the upstream DP-rank offset when device isolation is already handled externally. - Prevent `IndexError` for application-level DP after the upstream device isolation changes. - Wire the patch into the platform initialization for non-0.23.0. - Upstream source: [vllm#45026](vllm-project/vllm#45026). --- ### `vllm_ascend/patch/platform/patch_kv_cache_utils.py` - Patch `_get_kv_cache_config_deepseek_v4()` on v0.23.0. - Patch `_get_kv_cache_config_packed()` for non-0.23.0 by reusing the existing non-packed implementation. - Avoid excessive KV-cache allocation introduced by the packed KV-cache layout and prevent NPU OOM. - Upstream source: [vllm#46205](vllm-project/vllm#46205). --- ### `vllm_ascend/patch/worker/__init__.py` - Monkey-patch `vllm.v1.utils.CpuGpuBuffer.__init__()` to remap `torch.uint64` to `torch.int64` on non-0.23.0. - Preserve the original behavior for all other data types. - Avoid the Ascend runtime failure caused by unsupported `DT_UINT64` in `aclnnInplaceZero`. - Upstream source: [vllm#44665](vllm-project/vllm#44665). --- ### `vllm_ascend/spec_decode/ngram_proposer.py` - Preserve the original initialization path on v0.23.0. - Delay the Ascend-specific `propose()` implementation during base-class initialization on newer releases. - Avoid eager GPU-only initialization introduced upstream while preserving the original behavior afterward. - Upstream source: [vllm#29184](vllm-project/vllm#29184). --- ### `vllm_ascend/worker/utils.py` - Correct the type annotation of `kernel_block_sizes` to match its actual nested-list usage. - Keep the implementation consistent with the upstream interface. --- ### `vllm_ascend/worker/worker.py` - Adjust the local device rank for single-node application-level DP on non-Ray backends. - Match the upstream device assignment behavior after automatic device isolation was removed. - Ensure each DP worker is mapped to the correct NPU subset. - Upstream source: [vllm#45026](vllm-project/vllm#45026). --- ### `vllm_ascend/__init__.py` - Reorganize imports to eliminate circular dependencies during package initialization. - vLLM version: v0.23.0 - vLLM main: vllm-project/vllm@b9a7cd4 --------- Signed-off-by: zhangxinyuehfad <starmoon_zhang@163.com> Signed-off-by: hfadzxy <starmoon_zhang@163.com>

Resolve the vllm/envs.py conflict per the reusable playbook (docs/superpowers/specs/2026-05-14-envs-merge-conflict-resolution-design.md): the legacy `if TYPE_CHECKING:` block and `environment_variables` dict were dropped wholesale (superseded by the pydantic-settings models on this branch), then main's semantic delta was ported as targeted `Field` edits. Merge of origin/main (39a1d32) into base a46abb7; 10 main-side commits touched envs.py. Additions (5 new vars, all native pydantic parsing — no validator needed): - VLLM_ROCM_USE_AITER_CUSTOM_AR (bool=True) — vllm-project#46065 - VLLM_MAX_IMAGE_PIXELS (int=178_956_970) — vllm-project#47010; also added to compile_factors ignored_factors, matching main's exclude-set - VLLM_GPU_SYNC_CHECK (Literal["warn","error"]|None=None) — vllm-project#44800; a Literal Field reproduces main's env_with_choices reject-on-invalid - VLLM_MOONCAKE_LOAD_RECV_THREADS (int=1) — vllm-project#45971 - VLLM_MOE_SKIP_PADDING (bool=False) — vllm-project#46428 Modification: - VLLM_ROCM_QUICK_REDUCE_QUANTIZATION gained INT3 choice — vllm-project#45666 Not ported (deliberate no-ops): - VLLM_ENFORCE_STRICT_TOOL_CALLING (vllm-project#45892) — main only reflowed the lambda; branch Field is already default=True - VLLM_PORT doc-URL (vllm-project#35530) — branch already has its own valid URL - VLLM_USE_PACKED_HMA_KV_CACHE — added (vllm-project#46205) then removed (vllm-project#46252) within this window; net-neutral Deletions: none. tests/test_envs.py: not conflicted this run. docs/configuration/env_vars.md is generated from the fields at build time. AI assistance (Claude) was used for this merge resolution per AGENTS.md. Co-authored-by: Claude <noreply@anthropic.com> Signed-off-by: Vinay Damodaran <vrdn@hey.com>

### What this PR does / why we need it? This PR upgrades the verified vLLM main commit to ee0da84ab9e04ac7610e28580af62c365e898389 (v0.24.0 tag) and adapts vLLM Ascend to upstream API changes introduced after the previous verified commit. Changes # Changes ### `examples/offline_data_parallel.py` - Slice `ASCEND_RT_VISIBLE_DEVICES` according to the local DP rank before creating the `LLM` instance. - Fall back to `torch.npu.device_count()` when `ASCEND_RT_VISIBLE_DEVICES` is not set. - Keep application-level DP compatible with the upstream removal of automatic device isolation. - Upstream source: [vllm#45026](vllm-project/vllm#45026). --- ### `tests/e2e/conftest.py` - Slice `ASCEND_RT_VISIBLE_DEVICES` for each DP rank before creating `LLM()` instances or worker processes. - Fall back to `torch.npu.device_count()` when the environment variable is unavailable. - Ensure each test process uses an independent NPU subset after upstream removed automatic device isolation. - Upstream source: [vllm#45026](vllm-project/vllm#45026). --- ### `tests/ut/quantization/methods/test_w4a16_mxfp4.py` - Skip the test to match the upstream behavior. --- ### `tests/ut/spec_decode/test_speculators_vwn_eagle3.py` - Mock `vllm.v1.attention.selector._cached_get_attn_backend()` for non-NPU unit tests. - Avoid attention backend initialization failures introduced by the updated v0.24.0 initialization path. - Keep spec decode unit tests runnable without physical NPU devices. --- ### `vllm_ascend/_310p/worker_310p.py` - Use `MemorySnapshot(device=device)` for non-0.23.0 releases. - Align the 310P worker with the updated worker implementation. - Related upstream changes: - [vllm#30868](vllm-project/vllm#30868) --- ### `vllm_ascend/patch/platform/patch_dp_device_ids.py` - Add a patch for `get_physical_gpu_ids_for_local_dp_rank()`. - Support pre-sharded `ASCEND_RT_VISIBLE_DEVICES` by avoiding the upstream DP-rank offset when device isolation is already handled externally. - Prevent `IndexError` for application-level DP after the upstream device isolation changes. - Wire the patch into the platform initialization for non-0.23.0. - Upstream source: [vllm#45026](vllm-project/vllm#45026). --- ### `vllm_ascend/patch/platform/patch_kv_cache_utils.py` - Patch `_get_kv_cache_config_deepseek_v4()` on v0.23.0. - Patch `_get_kv_cache_config_packed()` for non-0.23.0 by reusing the existing non-packed implementation. - Avoid excessive KV-cache allocation introduced by the packed KV-cache layout and prevent NPU OOM. - Upstream source: [vllm#46205](vllm-project/vllm#46205). --- ### `vllm_ascend/patch/worker/__init__.py` - Monkey-patch `vllm.v1.utils.CpuGpuBuffer.__init__()` to remap `torch.uint64` to `torch.int64` on non-0.23.0. - Preserve the original behavior for all other data types. - Avoid the Ascend runtime failure caused by unsupported `DT_UINT64` in `aclnnInplaceZero`. - Upstream source: [vllm#44665](vllm-project/vllm#44665). --- ### `vllm_ascend/spec_decode/ngram_proposer.py` - Preserve the original initialization path on v0.23.0. - Delay the Ascend-specific `propose()` implementation during base-class initialization on newer releases. - Avoid eager GPU-only initialization introduced upstream while preserving the original behavior afterward. - Upstream source: [vllm#29184](vllm-project/vllm#29184). --- ### `vllm_ascend/worker/utils.py` - Correct the type annotation of `kernel_block_sizes` to match its actual nested-list usage. - Keep the implementation consistent with the upstream interface. --- ### `vllm_ascend/worker/worker.py` - Adjust the local device rank for single-node application-level DP on non-Ray backends. - Match the upstream device assignment behavior after automatic device isolation was removed. - Ensure each DP worker is mapped to the correct NPU subset. - Upstream source: [vllm#45026](vllm-project/vllm#45026). --- ### `vllm_ascend/__init__.py` - Reorganize imports to eliminate circular dependencies during package initialization. - vLLM version: v0.23.0 - vLLM main: vllm-project/vllm@b9a7cd4 --------- Signed-off-by: zhangxinyuehfad <starmoon_zhang@163.com> Signed-off-by: hfadzxy <starmoon_zhang@163.com>

### What this PR does / why we need it? This PR upgrades the verified vLLM main commit to ee0da84ab9e04ac7610e28580af62c365e898389 (v0.24.0 tag) and adapts vLLM Ascend to upstream API changes introduced after the previous verified commit. Changes # Changes ### `examples/offline_data_parallel.py` - Slice `ASCEND_RT_VISIBLE_DEVICES` according to the local DP rank before creating the `LLM` instance. - Fall back to `torch.npu.device_count()` when `ASCEND_RT_VISIBLE_DEVICES` is not set. - Keep application-level DP compatible with the upstream removal of automatic device isolation. - Upstream source: [vllm#45026](vllm-project/vllm#45026). --- ### `tests/e2e/conftest.py` - Slice `ASCEND_RT_VISIBLE_DEVICES` for each DP rank before creating `LLM()` instances or worker processes. - Fall back to `torch.npu.device_count()` when the environment variable is unavailable. - Ensure each test process uses an independent NPU subset after upstream removed automatic device isolation. - Upstream source: [vllm#45026](vllm-project/vllm#45026). --- ### `tests/ut/quantization/methods/test_w4a16_mxfp4.py` - Skip the test to match the upstream behavior. --- ### `tests/ut/spec_decode/test_speculators_vwn_eagle3.py` - Mock `vllm.v1.attention.selector._cached_get_attn_backend()` for non-NPU unit tests. - Avoid attention backend initialization failures introduced by the updated v0.24.0 initialization path. - Keep spec decode unit tests runnable without physical NPU devices. --- ### `vllm_ascend/_310p/worker_310p.py` - Use `MemorySnapshot(device=device)` for non-0.23.0 releases. - Align the 310P worker with the updated worker implementation. - Related upstream changes: - [vllm#30868](vllm-project/vllm#30868) --- ### `vllm_ascend/patch/platform/patch_dp_device_ids.py` - Add a patch for `get_physical_gpu_ids_for_local_dp_rank()`. - Support pre-sharded `ASCEND_RT_VISIBLE_DEVICES` by avoiding the upstream DP-rank offset when device isolation is already handled externally. - Prevent `IndexError` for application-level DP after the upstream device isolation changes. - Wire the patch into the platform initialization for non-0.23.0. - Upstream source: [vllm#45026](vllm-project/vllm#45026). --- ### `vllm_ascend/patch/platform/patch_kv_cache_utils.py` - Patch `_get_kv_cache_config_deepseek_v4()` on v0.23.0. - Patch `_get_kv_cache_config_packed()` for non-0.23.0 by reusing the existing non-packed implementation. - Avoid excessive KV-cache allocation introduced by the packed KV-cache layout and prevent NPU OOM. - Upstream source: [vllm#46205](vllm-project/vllm#46205). --- ### `vllm_ascend/patch/worker/__init__.py` - Monkey-patch `vllm.v1.utils.CpuGpuBuffer.__init__()` to remap `torch.uint64` to `torch.int64` on non-0.23.0. - Preserve the original behavior for all other data types. - Avoid the Ascend runtime failure caused by unsupported `DT_UINT64` in `aclnnInplaceZero`. - Upstream source: [vllm#44665](vllm-project/vllm#44665). --- ### `vllm_ascend/spec_decode/ngram_proposer.py` - Preserve the original initialization path on v0.23.0. - Delay the Ascend-specific `propose()` implementation during base-class initialization on newer releases. - Avoid eager GPU-only initialization introduced upstream while preserving the original behavior afterward. - Upstream source: [vllm#29184](vllm-project/vllm#29184). --- ### `vllm_ascend/worker/utils.py` - Correct the type annotation of `kernel_block_sizes` to match its actual nested-list usage. - Keep the implementation consistent with the upstream interface. --- ### `vllm_ascend/worker/worker.py` - Adjust the local device rank for single-node application-level DP on non-Ray backends. - Match the upstream device assignment behavior after automatic device isolation was removed. - Ensure each DP worker is mapped to the correct NPU subset. - Upstream source: [vllm#45026](vllm-project/vllm#45026). --- ### `vllm_ascend/__init__.py` - Reorganize imports to eliminate circular dependencies during package initialization. - vLLM version: v0.23.0 - vLLM main: vllm-project/vllm@b9a7cd4 --------- Signed-off-by: zhangxinyuehfad <starmoon_zhang@163.com> Signed-off-by: hfadzxy <starmoon_zhang@163.com> Signed-off-by: xqchen7 <chenxueqing7@huawei.com>

Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com> Co-authored-by: OpenAI Codex <codex@openai.com> Co-authored-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>

### What this PR does / why we need it? This PR upgrades the verified vLLM main commit to ee0da84ab9e04ac7610e28580af62c365e898389 (v0.24.0 tag) and adapts vLLM Ascend to upstream API changes introduced after the previous verified commit. Changes # Changes ### `examples/offline_data_parallel.py` - Slice `ASCEND_RT_VISIBLE_DEVICES` according to the local DP rank before creating the `LLM` instance. - Fall back to `torch.npu.device_count()` when `ASCEND_RT_VISIBLE_DEVICES` is not set. - Keep application-level DP compatible with the upstream removal of automatic device isolation. - Upstream source: [vllm#45026](vllm-project/vllm#45026). --- ### `tests/e2e/conftest.py` - Slice `ASCEND_RT_VISIBLE_DEVICES` for each DP rank before creating `LLM()` instances or worker processes. - Fall back to `torch.npu.device_count()` when the environment variable is unavailable. - Ensure each test process uses an independent NPU subset after upstream removed automatic device isolation. - Upstream source: [vllm#45026](vllm-project/vllm#45026). --- ### `tests/ut/quantization/methods/test_w4a16_mxfp4.py` - Skip the test to match the upstream behavior. --- ### `tests/ut/spec_decode/test_speculators_vwn_eagle3.py` - Mock `vllm.v1.attention.selector._cached_get_attn_backend()` for non-NPU unit tests. - Avoid attention backend initialization failures introduced by the updated v0.24.0 initialization path. - Keep spec decode unit tests runnable without physical NPU devices. --- ### `vllm_ascend/_310p/worker_310p.py` - Use `MemorySnapshot(device=device)` for non-0.23.0 releases. - Align the 310P worker with the updated worker implementation. - Related upstream changes: - [vllm#30868](vllm-project/vllm#30868) --- ### `vllm_ascend/patch/platform/patch_dp_device_ids.py` - Add a patch for `get_physical_gpu_ids_for_local_dp_rank()`. - Support pre-sharded `ASCEND_RT_VISIBLE_DEVICES` by avoiding the upstream DP-rank offset when device isolation is already handled externally. - Prevent `IndexError` for application-level DP after the upstream device isolation changes. - Wire the patch into the platform initialization for non-0.23.0. - Upstream source: [vllm#45026](vllm-project/vllm#45026). --- ### `vllm_ascend/patch/platform/patch_kv_cache_utils.py` - Patch `_get_kv_cache_config_deepseek_v4()` on v0.23.0. - Patch `_get_kv_cache_config_packed()` for non-0.23.0 by reusing the existing non-packed implementation. - Avoid excessive KV-cache allocation introduced by the packed KV-cache layout and prevent NPU OOM. - Upstream source: [vllm#46205](vllm-project/vllm#46205). --- ### `vllm_ascend/patch/worker/__init__.py` - Monkey-patch `vllm.v1.utils.CpuGpuBuffer.__init__()` to remap `torch.uint64` to `torch.int64` on non-0.23.0. - Preserve the original behavior for all other data types. - Avoid the Ascend runtime failure caused by unsupported `DT_UINT64` in `aclnnInplaceZero`. - Upstream source: [vllm#44665](vllm-project/vllm#44665). --- ### `vllm_ascend/spec_decode/ngram_proposer.py` - Preserve the original initialization path on v0.23.0. - Delay the Ascend-specific `propose()` implementation during base-class initialization on newer releases. - Avoid eager GPU-only initialization introduced upstream while preserving the original behavior afterward. - Upstream source: [vllm#29184](vllm-project/vllm#29184). --- ### `vllm_ascend/worker/utils.py` - Correct the type annotation of `kernel_block_sizes` to match its actual nested-list usage. - Keep the implementation consistent with the upstream interface. --- ### `vllm_ascend/worker/worker.py` - Adjust the local device rank for single-node application-level DP on non-Ray backends. - Match the upstream device assignment behavior after automatic device isolation was removed. - Ensure each DP worker is mapped to the correct NPU subset. - Upstream source: [vllm#45026](vllm-project/vllm#45026). --- ### `vllm_ascend/__init__.py` - Reorganize imports to eliminate circular dependencies during package initialization. - vLLM version: v0.23.0 - vLLM main: vllm-project/vllm@b9a7cd4 --------- Signed-off-by: zhangxinyuehfad <starmoon_zhang@163.com> Signed-off-by: hfadzxy <starmoon_zhang@163.com>

mergify Bot added v1 kv-connector labels Jun 20, 2026

LucasWilkinson added the ready ONLY add when PR is ready to merge/full CI is needed label Jun 20, 2026

LucasWilkinson marked this pull request as ready for review June 20, 2026 02:40

LucasWilkinson requested review from ApostaC, NickLucche, WoosukKwon, alexm-redhat, heheda12345, njhill, orozery, robertgshaw2-redhat, xuechendi and ywang96 as code owners June 20, 2026 02:40

tlrmchlsmth approved these changes Jun 20, 2026

View reviewed changes

Merge branch 'main' into codex/packed-kv-hma

83127f6

tlrmchlsmth enabled auto-merge (squash) June 20, 2026 19:28

tlrmchlsmth merged commit cc22621 into vllm-project:main Jun 20, 2026
91 checks passed

zhangxinyuehfad mentioned this pull request Jul 4, 2026

[CI] main2mian v0.24.0 vllm-project/vllm-ascend#11227

Merged

ayaangazali mentioned this pull request Jul 18, 2026

[Doc] Document SimpleCPUOffloadConnector in the KV offloading guide #49048

Open

Navigation Menu

Uh oh!

Uh oh!

[KV Offload] Support packed HMA KV cache layout - #46205

[KV Offload] Support packed HMA KV cache layout#46205
tlrmchlsmth merged 2 commits into
vllm-project:mainfrom
neuralmagic:codex/packed-kv-hma

LucasWilkinson commented Jun 20, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Uh oh!

Conversation

LucasWilkinson commented Jun 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Duplicate-work check

Benchmarks

Tests

AI Assistance

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

LucasWilkinson commented Jun 20, 2026 •

edited

Loading