Skip to content

vmmem growth starves Windows-side GPU/NPU allocations (Level Zero ZE_RESULT_ERROR_OUT_OF_HOST_MEMORY, D3D12 OOM) — page cache is never returned, and neither side reports the cause #41285

Description

@Premshay

Summary

When vmmem grows toward the .wslconfig memory= cap, Windows-side device allocations
start failing: Level Zero cannot create an NPU command queue
(ZE_RESULT_ERROR_OUT_OF_HOST_MEMORY), and D3D12 shared-memory allocations for the iGPU
fail far below their nominal budgets. The cause is that WSL2 returns memory to Windows only
through free-page reporting, which covers memory the Linux kernel has freed — the page
cache and all anonymous memory count as in-use and stay in vmmem indefinitely. The failure
is invisible from both sides: Linux shows tens of GB "available" (reclaimable cache),
Windows shows almost nothing free, and the driver errors name neither. No Windows setting
fixes this; only releasing memory inside the guest does.

Environment

  • Windows 11, 128 GB RAM, WSL2 Ubuntu 26.04, networkingMode=mirrored
  • .wslconfig: memory=124GB, autoMemoryReclaim=disabled
  • Host-side consumers: Intel NPU via OpenVINO GenAI (Level Zero), Intel iGPU via D3D12
    (Core Ultra 7 270K)

Reproduction

  1. Set memory= close to the machine total (the default 50%–8GB rule avoids this, which
    is why the failure mostly hits tuned setups).
  2. Run a memory-heavy Linux workload — in my case an LLM inference server with large
    mmap-loaded model files, which fills the page cache by design, plus process RSS in the
    tens of GB.
  3. On the Windows side, start anything that needs device memory: an OpenVINO GenAI
    pipeline on the NPU, or a D3D12 app using iGPU shared memory.

Observed once vmmem neared the cap:

# OpenVINO GenAI, NPU pipeline creation:
Exception from src\plugins\intel_npu\...\zero_wrappers.cpp:323:
L0 zeCommandQueueCreate result: ZE_RESULT_ERROR_OUT_OF_HOST_MEMORY, code 0x70000002

# D3D12 (iGPU shared memory), allocations of 745 MB and 997 MB on a device
# reporting 67 GB visible: ErrorOutOfDeviceMemory

Diagnosis

The two sides disagree about the same physical memory:

# WSL — looks healthy, 74 GB "available":
$ free -g
              total   used   free   buff/cache   available
Mem:            121     46      6           78          74

# Windows — nearly exhausted:
PS> Get-CimInstance Win32_OperatingSystem | select FreePhysicalMemory
11.9 GB free of 127.3

Linux's 74 GB "available" is page cache it would drop under its own memory pressure —
but the host's pressure generates no signal inside the guest, so it never drops, and page
reporting has nothing to return.

Confirming the mechanism: releasing ~19 GB of anonymous memory inside the guest (stopping
the inference server) returned it to Windows within seconds, and the same
zeCommandQueueCreate then succeeded. No reboot, no wsl --shutdown.

Two things do not help, worth stating because they are the documented knobs:

  • echo 1 > /proc/sys/vm/drop_caches (the workaround in the memory-reclaim blog post)
    cannot evict pages actively mmapped by a running process — a resident model file is
    pinned regardless.
  • autoMemoryReclaim trades this problem for another on this workload class: dropping the
    cache means re-reading tens of GB of model files on the next inference, so mmap-heavy
    setups run with it disabled by design.

Suggestions

  1. Docs (cheap, high value): a paragraph in the .wslconfig reference stating that
    memory= should be sized to leave the host a working floor when the host itself runs
    GPU/NPU/driver workloads, because page cache and process memory are not returned while
    in use — and that host-side symptoms are device-allocation failures that never mention
    WSL. Today nothing connects those dots for the person debugging the driver error.
  2. Diagnosability: a wsl command (or perf counter) exposing "guest memory returnable
    vs pinned" would turn a multi-hour investigation into one look. The driver errors can't
    name the cause; WSL can.
  3. Longer term: a host-floor semantic — some notion of host reserve that vmmem growth
    respects under host memory pressure — would make mixed WSL+host-compute machines safe
    without hand-tuning memory= against the worst day.

Related

#10011 (memory reclaim discussion) covers returning freed memory; this issue is about
memory that is never freed guest-side and the host-side failure mode that results. #12634
(Intel iGPU in WSL2) is adjacent hardware but a different problem. I did not find an
existing issue describing the host-side GPU/NPU allocation failures.

Happy to provide full logs and the before/after memory counters from both sides.


Disclosure: investigation assisted by AI (Claude, Anthropic). I directed and verified;
all numbers were measured on my hardware as described.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions