Skip to content

Mitigate + instrument utility-VM boot-connect timeout (62543002) - #3

Closed
yeelam-gordon wants to merge 1 commit into
masterfrom
fix/instrument-notify-channel-accept
Closed

Mitigate + instrument utility-VM boot-connect timeout (62543002)#3
yeelam-gordon wants to merge 1 commit into
masterfrom
fix/instrument-notify-channel-accept

Conversation

@yeelam-gordon

@yeelam-gordon yeelam-gordon commented Jul 24, 2026

Copy link
Copy Markdown
Owner

What

Utility-VM Initialize waits for the Linux guest to connect back over hvsocket within KernelBootTimeout — first mini_init, then the notify channel. When a boot is healthy but slow (loaded host, cold first boot), this watchdog trips and surfaces as .../CreateVm/0x800705b4 (ERROR_TIMEOUT). This is the reported failure in bug 62543002 on loaded SP11 ARM64 CI.

KernelBootTimeout is a guest-responsiveness watchdog (a give-up ceiling), not a boot budget — a healthy boot connects in ~1–3s. The 30s default leaves too little headroom for a contended host, and the notify-channel accept was uninstrumented, so the lost time could not be attributed to a phase.

Fix

Two small, targeted changes:

  1. Mitigate — raise the default KernelBootTimeout from 30s to 90s (WslCoreConfig.h). Still bounds a genuinely stuck VM; just gives a slow-but-alive boot room to finish. User-overridable via wsl2.kernelBootTimeout.
  2. Instrument — wrap the notify-channel accept in its own SlowOperationWatcher{"WaitForNotifyChannelConnect"}, mirroring the adjacent WaitForMiniInitConnect block, so slow/hung boots are attributed to the correct phase. Diagnostic-only — no control-flow change.

Why this isn't already fixed by microsoft#40519

microsoft#40519 (merged, in this PR's base) protects critical WSL processes via a guest-side wsl-user cgroup. It does not address this failure: its host-side change is only an IsolateDistroCgroup flag (no host CPU reservation/priority on the VM), and its cgroup tree is built inside the running distro after mini_init has already connected — a different phase, and intra-guest contention rather than the host starving the whole VM's vCPUs during boot.

Root cause (follow-up)

The underlying disease is host CPU-scheduling starvation of the utility VM's vCPUs during cold first boot: the VM is created with all logical processors but no CPU reservation/weight/priority. The real cure is a host-side CPU guarantee for the VM until mini_init connects — to be built after confirming the starved resource via the phase telemetry above plus a host resource snapshot.

Testing

  • SlowOperationWatcher unit tests: Total=6 Passed=6, including SlowCompletionEmitsOneRecord (real elapsedMs) and HangEmitsOnceAtMaxThenStops (timedOut=true hang backstop).
  • wslservice build-verified (CMake configure + build, no errors) with both changes.
  • Deterministic repro confirmed on the same accept path: setting kernelBootTimeout=1 yields the exact .../CreateVm/0x800705b4.

@yeelam-gordon

Copy link
Copy Markdown
Owner Author

Code-review loop — outcome

Verdict: APPROVE. No must-fix items.

Verified: the new block scopes only the SlowOperationWatcher construction + the m_notifyChannel assignment (member lifetime unaffected); SlowOperationWatcher is already used throughout this TU with the string-literal ctor; purely additive, passive instrumentation mirroring the adjacent WaitForMiniInitConnect block — no control-flow, timeout, or assignment-semantics change.

Tests: SlowOperationWatcher unit tests 6/6 PASS (real elapsedMs; timedOut=true hang backstop). Composes with the elapsed-time telemetry work (build-verified).

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds missing phase-level telemetry during utility VM initialization by instrumenting the notify-channel accept with SlowOperationWatcher, allowing slow/timeout boot attribution between the mini_init accept and the notify-channel accept phases.

Changes:

  • Wrap the notify-channel AcceptConnection(...) call in a dedicated SlowOperationWatcher{"WaitForNotifyChannelConnect"} scope.
  • Add explanatory in-code comments describing why this accept is separately instrumented.

Comment thread src/windows/service/exe/WslCoreVm.cpp Outdated

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.

@yeelam-gordon yeelam-gordon changed the title Instrument the notify-channel accept with SlowOperationWatcher Mitigate + instrument utility-VM boot-connect timeout (62543002) Jul 24, 2026
@yeelam-gordon
yeelam-gordon requested a review from Copilot July 24, 2026 14:53

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.

A healthy-but-slow guest boot (loaded host, cold first boot) trips the KernelBootTimeout watchdog and surfaces as CreateVm/0x800705b4. Raise the default KernelBootTimeout from 30s to 90s for headroom, and wrap the notify-channel accept in its own SlowOperationWatcher so a slow or hung boot is attributed to the correct phase. Diagnostic-only; no control-flow change.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: e76db014-5ee2-46f5-b078-42ad74b5197a
@yeelam-gordon
yeelam-gordon force-pushed the fix/instrument-notify-channel-accept branch from 63d1014 to a0cd1cb Compare July 24, 2026 15:01
@yeelam-gordon

Copy link
Copy Markdown
Owner Author

Superseded by upstream microsoft#41165 (same squashed change).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants