Mitigate + instrument utility-VM boot-connect timeout (62543002) - #3
Mitigate + instrument utility-VM boot-connect timeout (62543002)#3yeelam-gordon wants to merge 1 commit into
Conversation
Code-review loop — outcomeVerdict: APPROVE. No must-fix items. Verified: the new block scopes only the Tests: |
There was a problem hiding this comment.
Pull request overview
This PR adds missing phase-level telemetry during utility VM initialization by instrumenting the notify-channel accept with SlowOperationWatcher, allowing slow/timeout boot attribution between the mini_init accept and the notify-channel accept phases.
Changes:
- Wrap the notify-channel
AcceptConnection(...)call in a dedicatedSlowOperationWatcher{"WaitForNotifyChannelConnect"}scope. - Add explanatory in-code comments describing why this accept is separately instrumented.
b656210 to
4269ad1
Compare
A healthy-but-slow guest boot (loaded host, cold first boot) trips the KernelBootTimeout watchdog and surfaces as CreateVm/0x800705b4. Raise the default KernelBootTimeout from 30s to 90s for headroom, and wrap the notify-channel accept in its own SlowOperationWatcher so a slow or hung boot is attributed to the correct phase. Diagnostic-only; no control-flow change. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: e76db014-5ee2-46f5-b078-42ad74b5197a
63d1014 to
a0cd1cb
Compare
|
Superseded by upstream microsoft#41165 (same squashed change). |
What
Utility-VM
Initializewaits for the Linux guest to connect back over hvsocket withinKernelBootTimeout— firstmini_init, then the notify channel. When a boot is healthy but slow (loaded host, cold first boot), this watchdog trips and surfaces as.../CreateVm/0x800705b4(ERROR_TIMEOUT). This is the reported failure in bug 62543002 on loaded SP11 ARM64 CI.KernelBootTimeoutis a guest-responsiveness watchdog (a give-up ceiling), not a boot budget — a healthy boot connects in ~1–3s. The 30s default leaves too little headroom for a contended host, and the notify-channel accept was uninstrumented, so the lost time could not be attributed to a phase.Fix
Two small, targeted changes:
KernelBootTimeoutfrom 30s to 90s (WslCoreConfig.h). Still bounds a genuinely stuck VM; just gives a slow-but-alive boot room to finish. User-overridable viawsl2.kernelBootTimeout.SlowOperationWatcher{"WaitForNotifyChannelConnect"}, mirroring the adjacentWaitForMiniInitConnectblock, so slow/hung boots are attributed to the correct phase. Diagnostic-only — no control-flow change.Why this isn't already fixed by microsoft#40519
microsoft#40519 (merged, in this PR's base) protects critical WSL processes via a guest-side
wsl-usercgroup. It does not address this failure: its host-side change is only anIsolateDistroCgroupflag (no host CPU reservation/priority on the VM), and its cgroup tree is built inside the running distro aftermini_inithas already connected — a different phase, and intra-guest contention rather than the host starving the whole VM's vCPUs during boot.Root cause (follow-up)
The underlying disease is host CPU-scheduling starvation of the utility VM's vCPUs during cold first boot: the VM is created with all logical processors but no CPU reservation/weight/priority. The real cure is a host-side CPU guarantee for the VM until
mini_initconnects — to be built after confirming the starved resource via the phase telemetry above plus a host resource snapshot.Testing
SlowOperationWatcherunit tests: Total=6 Passed=6, includingSlowCompletionEmitsOneRecord(realelapsedMs) andHangEmitsOnceAtMaxThenStops(timedOut=truehang backstop).wslservicebuild-verified (CMake configure + build, no errors) with both changes.kernelBootTimeout=1yields the exact.../CreateVm/0x800705b4.