[BUG] - PixelStreaming 2 - Memory Leak, Please fix it already

[Slipknot765432]

UE Version:
5.8

Frontend Version:
PixelStreamingInfrastructure UE5.8 branch (Wilbur signalling server). Not frontend-dependent —
reproduces with the stock frontend, with a custom frontend, and with the mediasoup SFU.

Problem component
Pixel Streaming C++ plugin (engine-side) — PixelStreaming2 → AVCodecs / NVENC.
Specifically FEncoderNVENC::ApplyConfig in
Engine/Plugins/Experimental/AVCodecs/NVCodecs/Source/NVENC/Private/Video/Encoders/VideoEncoderNVENC.cpp.

Description

Windows commit charge grows continuously while streaming H.264/NVENC on D3D12, and is never
released. The process eventually reaches the system commit limit and is terminated by Windows. UE
usually reports a misleading "Out of video memory", the faulting module is nvEncodeAPI64.dll, and
the Windows System event log records Resource-Exhaustion-Detector event 2004 naming the game
executable with a huge allocation figure.

The defining signature is that commit climbs while the working set stays flat — in one measured
window the process committed +6.9 GB while its working set moved +0.07 GB. Address space is
committed per encoder-reconfigure and never touched. This is why the leak is invisible if you watch
RAM usage or VRAM: physical RAM sat at ~25–41% and VRAM was stable the entire time.

This is the same root cause as #900, which its reporter closed after applying a local source
patch. I am re-filing because:

1. The fix in [BUG] - UE 5.7 PixelStreaming H264 NVENC commit memory grows due to frequent nvEncReconfigureEncoder calls from WebRTC rate control #900 lives in an engine plugin, so anyone on an installed/launcher engine cannot
   apply it without a full source build of UE.
2. The workaround Epic recommends in [BUG] - UE 5.7 PixelStreaming H264 NVENC commit memory grows due to frequent nvEncReconfigureEncoder calls from WebRTC rate control #900 (-AVCodecs.NvEnc.D3D12UsesCUDA=true) does not merely
   fail to help — on this hardware it destroys the encoder outright (details below).
3. The newest available driver still leaks, so "wait for NVIDIA" currently has no end date.
4. I found a launcher-only mitigation that gets ~10× without touching engine source, plus the
   precise reason it cannot reach zero — which points at a small, concrete engine-side change.

Steps to Reproduce:

1. Package a project with PixelStreaming2 enabled. Windows, D3D12, NVIDIA GPU.
2. Launch it pointed at a signalling server, forcing hardware H.264:
   -PixelStreamingConnectionURL=ws://<host>:8888 -PixelStreamingEncoderCodec=H264 -PixelStreamingWebRTCNegotiateCodecs=false -PixelStreamingWebRTCStartBitrate=20000000 -PixelStreamingWebRTCMinBitrate=5000000 -PixelStreamingWebRTCMaxBitrate=25000000
3. Connect one or more browser viewers.
4. Watch the process's private commit — Task Manager's "Commit size" column (not enabled by
   default), or (Get-Process <name>).PrivateMemorySize64. Do not watch RAM %, working set, or
   VRAM; all three stay flat and hide the problem completely.
5. Commit rises monotonically for as long as viewers are connected. Disconnect every viewer and
   growth stops immediately; reconnect and it resumes.

Expected behavior

1. Committed memory should stabilise shortly after streaming begins and stay flat for the life of
   the session.
2. WebRTC rate-control updates (bitrate / framerate changes) should not leak memory. These fire
   continuously and unavoidably during normal streaming.
3. Viewers connecting and disconnecting repeatedly should return the process to a steady baseline.

Screenshots

Measured instead — process-isolated PrivateMemorySize64, sampled every 5s, 1080p60 H.264:

Configuration	Driver	NVENC sessions	Commit growth	Per session
Idle, no viewers connected	610.88	0	~0.04 GB/min (flat)	—
Stock (WebRTC-adaptive bitrate)	610.62	1–2	10–15 GB/min	—
EncoderTargetBitrate pinned, 10 viewers via SFU	610.62	1	1.02 GB/min	1.02
EncoderTargetBitrate pinned, 2 direct viewers	610.88	2	1.90 GB/min	0.95

The leak is ~1 GB/min per NVENC session at 1080p60, linear in session count. Three independent
runs across two driver versions and different viewer counts agree to within ~5%. Viewer count only
matters insofar as it creates encoder sessions — ten viewers behind an SFU (one session) leak less
than two directly-connected viewers (two sessions). Runway is therefore predictable:
commit headroom ÷ (1 GB/min × sessions).

The idle → streaming transition in a single run, one sample per minute:

10.32 GB
10.36   +0.04   <- no viewers
13.30   +2.94   <- viewers connect, encoding starts
14.92   +1.62

Working set across a window where commit grew +6.9 GB: 5.58 GB -> 5.65 GB.

Platform:

• OS: Windows 10 Pro 10.0.19045
• Browser + version: Chrome 150.0.7871.187
• GPU: NVIDIA GeForce RTX 5080 (Blackwell), 16 GB
• Drivers tested: 610.62 (32.0.16.1062) and 610.88 (32.0.16.1088, released 2026-07-29) —
  both leak identically
• Build configs tested: Shipping and DebugGame — both leak
• Encoders tested: H.264/NVENC hardware and VP8 software — both leak

Additional context

1. -AVCodecs.NvEnc.D3D12UsesCUDA=true is not a usable workaround on RTX 50-series.
The CUDA context creates fine, but the instant a viewer connects the encoder dies and never
recovers — 21,514 errors in 3 min 12 s, looping:

LogAVCodecs: Error: Error: Error encoding picture [NVENC 20]
LogAVCodecs: Error: Error Unmapping: Failed to destroy array [CUDA 700]
LogAVCodecs: Error: Error Mapping: Failed to import external memory [CUDA 700]
LogAVCodecs: Error: Error Invalid State: Raw resource is invalid [CUDA]

CUDA 700 is an illegal memory access, which poisons the CUDA context permanently. Commit does go
flat under this flag — but only because nothing is being encoded. It trades the leak for a dead
stream. Worth noting in the docs, since #900's reporter also found it ineffective.

2. The newest driver does not fix it. 610.88 WHQL leaks at the same rate as 610.62. Its release
notes contain no NVENC, encoder, or memory-leak entries at all; "NVENC" appears only under
Limitations regarding old presets being dropped in R590+.

3. Launcher-only mitigation, ~10× improvement, no source patch required:

-PixelStreamingEncoderTargetBitrate=20000000

(PixelStreaming2.Encoder.TargetBitrate, default −1.) At
EpicRtcVideoEncoder.cpp UpdateConfig():

VideoConfig->TargetBitrate = TargetBitrateCVar > -1 ? TargetBitrateCVar : EpicRtcProposedTargetBitrate;

this overrides WebRTC's bitrate entirely, so averageBitRate stops changing, so the
if (AppliedConfig != PendingConfig) guard in FEncoderNVENC::ApplyConfig short-circuits and
nvEncReconfigureEncoder is never reached. Cost: no adaptive bitrate.

4. Why that cannot reach zero — and the concrete ask.
With the bitrate pinned, exactly one field in the NVENC config still moves: TargetFramerate.
It is a uint32 truncated from EpicRtc's double _framerateFps, it has no CVar override, and
under CBR it additionally drives vbvBufferSize / vbvInitialDelay
(VideoEncoderConfigNVENC.cpp). Any render-rate wobble across an integer boundary flips it, and
every flip is another reconfigure and another leak. There is no command-line way to freeze it.

Two options that would resolve this for installed-engine users:

• Preferred: land [BUG] - UE 5.7 PixelStreaming H264 NVENC commit memory grows due to frequent nvEncReconfigureEncoder calls from WebRTC rate control #900's guard upstream — skip nvEncReconfigureEncoder when only
  averageBitRate / maxBitRate / frameRateNum / frameRateDen / vbvBufferSize /
  vbvInitialDelay changed, still reconfiguring on resolution changes. This fixes it for everyone
  without requiring a source build.
• Minimum: expose a CVar to pin TargetFramerate, mirroring
  PixelStreaming2.Encoder.TargetBitrate, so the whole encoder config can be frozen from the
  command line as a supported workaround until NVIDIA ships a driver fix.

5. The leak scales with peer count, since each peer drives its own rate controller. Ten viewers
behind a mediasoup SFU (one encoder session) leaked less than two directly-connected viewers (two
sessions). Routing through an SFU is therefore both a scaling and a leak-severity mitigation.