Add diagnostics for Process test hangs - #131907
Conversation
Focus and serialize Windows ProcessStartInfo tests, log startup and test ordering, and capture a WER dump before the Helix timeout. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 54c890f9-b409-4b7d-b810-3701e125fd96
|
/azp run runtime-libraries-coreclr outerloop-windows |
|
/azp run runtime-nativeaot-outerloop |
|
Azure Pipelines: Successfully started running 1 pipeline(s). |
|
Azure Pipelines: Successfully started running 3 pipeline(s). 13 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run. |
|
Azure Pipelines: Successfully started running 1 pipeline(s). |
There was a problem hiding this comment.
Pull request overview
Adds additional diagnostics and run-configuration changes to help investigate hangs in System.Diagnostics.Process tests, primarily on Windows. The PR introduces both logging (per-test + targeted messages) and a watchdog/WER dump configuration mechanism.
Changes:
- Windows-only test execution configuration updates (disable parallelization, enable progress output, and currently filters to a single test class via
XUnitOptions). - New diagnostics helper with a module initializer, WER LocalDumps configuration attempt, and a FailFast watchdog.
- Adds assembly-level xUnit
BeforeAfterTestAttributehook and adds extra diagnostic logging in a couple ofProcessStartInfoTests.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| src/libraries/System.Diagnostics.Process/tests/System.Diagnostics.Process.Tests.csproj | Adds Windows-only test runner settings (parallelization/progress and class filtering) and includes the new diagnostics source file. |
| src/libraries/System.Diagnostics.Process/tests/ProcessTestHangDiagnostics.cs | New diagnostics implementation: per-test logging hook, module initializer logging, WER LocalDumps setup, and watchdog FailFast timeout. |
| src/libraries/System.Diagnostics.Process/tests/ProcessStartInfoTests.cs | Adds targeted diagnostic logs around specific test execution points. |
| src/libraries/System.Diagnostics.Process/tests/AssemblyInfo.cs | Registers the diagnostics attribute at assembly level. |
Suppressed comments (1)
src/libraries/System.Diagnostics.Process/tests/ProcessTestHangDiagnostics.cs:83
- Configuring WER LocalDumps under HKLM requires admin and will typically fail in CI/Helix (and if it succeeds, it leaves a machine-wide setting behind). Using HKCU avoids the elevation requirement and limits scope to the current user, making it more likely the dump configuration actually takes effect.
{
using RegistryKey? key = Registry.LocalMachine.CreateSubKey(keyPath);
if (key is null)
{
Log($"Unable to create WER LocalDumps key HKLM\\{keyPath}.");
| <PropertyGroup Condition="'$(TargetPlatformIdentifier)' == 'windows'"> | ||
| <TestDisableParallelization>true</TestDisableParallelization> | ||
| <XUnitOptions>$(XUnitOptions) -class System.Diagnostics.Tests.ProcessStartInfoTests -parallel none</XUnitOptions> | ||
| <XUnitShowProgress>true</XUnitShowProgress> | ||
| </PropertyGroup> |
| ConfigureWindowsErrorReporting(); | ||
|
|
||
| var watchdog = new Thread(Watchdog) | ||
| { | ||
| IsBackground = true, | ||
| Name = "Process tests hang watchdog" | ||
| }; | ||
| watchdog.Start(); |
| [assembly: CollectionBehavior(CollectionBehavior.CollectionPerAssembly)] | ||
| [assembly: System.Diagnostics.Tests.ProcessTestHangDiagnosticsAttribute] | ||
|
|
|
Tagging subscribers to this area: @dotnet/area-system-diagnostics-process |
Write BOM-less UTF-8 console output and place WER dumps in the Helix work-item upload directory. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 54c890f9-b409-4b7d-b810-3701e125fd96
|
/azp run runtime-libraries-coreclr outerloop-windows |
|
Azure Pipelines: Successfully started running 1 pipeline(s). |
|
/azp run runtime-nativeaot-outerloop |
|
Azure Pipelines: Successfully started running 1 pipeline(s). |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.
Suppressed comments (3)
src/libraries/System.Diagnostics.Process/tests/System.Diagnostics.Process.Tests.csproj:17
- The hard-coded
-class System.Diagnostics.Tests.ProcessStartInfoTestsfilter inXUnitOptionsmeans Windows runs of this test project will only execute that single test class, skipping the rest of the Process test suite on Windows (significant test coverage reduction). This kind of focusing should be passed as a CI/test invocation parameter (e.g.,/p:XUnitClassName=...) rather than baked into the project file.
<PropertyGroup Condition="'$(TargetPlatformIdentifier)' == 'windows'">
<TestDisableParallelization>true</TestDisableParallelization>
<XUnitOptions>$(XUnitOptions) -class System.Diagnostics.Tests.ProcessStartInfoTests -parallel none</XUnitOptions>
<XUnitShowProgress>true</XUnitShowProgress>
src/libraries/System.Diagnostics.Process/tests/ProcessTestHangDiagnostics.cs:49
- The watchdog thread is started unconditionally from the module initializer and will
FailFastafter 3 minutes even for non-hung runs (e.g., slow machines or local debugging). Consider gating this to Helix runs (or an explicit opt-in env var) so it doesn’t introduce new crash behavior in normal Windows test execution.
var watchdog = new Thread(Watchdog)
{
IsBackground = true,
Name = "Process tests hang watchdog"
};
watchdog.Start();
src/libraries/System.Diagnostics.Process/tests/ProcessTestHangDiagnostics.cs:87
- Configuring WER LocalDumps via
Registry.LocalMachinewrites machine-wide state and will typically require elevated permissions; if it fails (UnauthorizedAccess), the dump capture won’t be configured at all. Prefer using a per-user key so the diagnostics can work under normal test permissions and avoid persisting HKLM changes.
using RegistryKey? key = Registry.LocalMachine.CreateSubKey(keyPath);
if (key is null)
{
Log($"Unable to create WER LocalDumps key HKLM\\{keyPath}.");
return;
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
|
/azp run runtime-libraries-coreclr outerloop-windows |
|
Azure Pipelines: Successfully started running 1 pipeline(s). |
|
/azp run runtime-nativeaot-outerloop |
|
Azure Pipelines: Successfully started running 1 pipeline(s). |
There was a problem hiding this comment.
Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.
Note
This error may be related to your runner configuration. You can now configure runners for Copilot code review separately from Copilot cloud agent by creating a copilot-code-review.yml file with your setup steps. Read the docs for details.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6a078136-b226-4ce5-94ff-1732ebcdfe79
|
/azp run runtime-nativeaot-outerloop |
|
Azure Pipelines: Successfully started running 1 pipeline(s). |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.
Suppressed comments (6)
src/libraries/System.Diagnostics.Process/tests/System.Diagnostics.Process.Tests.csproj:17
XUnitOptionscurrently includes-class System.Diagnostics.Tests.ProcessStartInfoTests(andXUnitShowProgressenables-verbose). The-classfilter will skip all other test classes for Windows TFMs, reducing coverage and potentially masking regressions; the verbose output is also unusually chatty to enable unconditionally.
<TestDisableParallelization>true</TestDisableParallelization>
<XUnitOptions>$(XUnitOptions) -class System.Diagnostics.Tests.ProcessStartInfoTests -parallel none</XUnitOptions>
<XUnitShowProgress>true</XUnitShowProgress>
src/libraries/System.Diagnostics.Process/tests/ProcessTestHangDiagnostics.cs:47
- The watchdog thread is started unconditionally, but
Watchdog()currently callsFailFastafter 3 minutes. That means any Windows run longer than 3 minutes will be terminated even if it’s making progress.
var watchdog = new Thread(Watchdog)
{
IsBackground = true,
Name = "Process tests hang watchdog"
};
src/libraries/System.Diagnostics.Process/tests/ProcessTestHangDiagnostics.cs:63
- To support an inactivity-based watchdog,
Logshould update the last-activity timestamp in a thread-safe way before writing output.
internal static void Log(string message)
{
s_log.WriteLine($"[Process test hang diagnostics] {message}");
}
src/libraries/System.Diagnostics.Process/tests/ProcessTestHangDiagnostics.cs:104
Watchdog()currently sleeps once and then unconditionallyFailFasts. With an inactivity timestamp, this can instead trigger only when no test progress has been logged for the timeout window.
private static void Watchdog()
{
Thread.Sleep(WatchdogTimeout);
const string message = "System.Diagnostics.Process.Tests exceeded the diagnostic watchdog timeout.";
Log(message);
src/libraries/System.Diagnostics.Process/tests/ProcessTestHangDiagnostics.cs:35
- The watchdog currently enforces an absolute timeout. Tracking last test activity and having the watchdog trigger on inactivity avoids terminating long-but-progressing test runs.
This issue also appears in the following locations of the same file:
- line 60
- line 100
private static readonly TimeSpan WatchdogTimeout = TimeSpan.FromMinutes(3);
private static readonly TextWriter s_log = TextWriter.Synchronized(
new StreamWriter(Console.OpenStandardError(), new UTF8Encoding(encoderShouldEmitUTF8Identifier: false), bufferSize: 1024, leaveOpen: true) { AutoFlush = true });
src/libraries/System.Diagnostics.Process/tests/ProcessTestHangDiagnostics.cs:52
Registry.GetValuecan throw (e.g., access denied / security policy). Since this is diagnostic-only logging, it should not be able to fail the entire test assembly.
Log("Reading Windows InstallationType.");
object? installationType = Registry.GetValue(InstallationTypeKey, "InstallationType", defaultValue: null);
Log($"InstallationType={installationType ?? "<null>"}");
|
/azp run runtime-nativeaot-outerloop |
|
Azure Pipelines: Successfully started running 1 pipeline(s). |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.
Suppressed comments (3)
src/libraries/System.Diagnostics.Process/tests/System.Diagnostics.Process.Tests.csproj:18
-class System.Diagnostics.Tests.ProcessStartInfoTestsinXUnitOptionsfilters the Windows test run down to a single test class, which effectively disables the rest of the System.Diagnostics.Process test coverage on Windows. If the goal is only to serialize and add diagnostics, drop the class filter and keep only the parallelization settings.
<PropertyGroup Condition="'$(TargetPlatformIdentifier)' == 'windows'">
<TestDisableParallelization>true</TestDisableParallelization>
<XUnitOptions>$(XUnitOptions) -class System.Diagnostics.Tests.ProcessStartInfoTests -parallel none</XUnitOptions>
<XUnitShowProgress>true</XUnitShowProgress>
</PropertyGroup>
src/libraries/System.Diagnostics.Process/tests/ProcessTestHangDiagnostics.cs:63
- The watchdog currently measures wall-clock time since module initialization, so any legitimate run that takes longer than the watchdog timeout can be terminated even if tests are making progress. Track the most recent test activity and have the watchdog trigger only after a period of inactivity.
internal static void Log(string message)
{
s_log.WriteLine($"[Process test hang diagnostics] {message}");
}
src/libraries/System.Diagnostics.Process/tests/ProcessTestHangDiagnostics.cs:106
Watchdogunconditionally callsFailFastafter a single sleep, regardless of whether tests are actively progressing. Switching to an inactivity-based loop avoids terminating long-but-healthy runs and still triggers quickly when a test actually hangs.
private static void Watchdog()
{
Thread.Sleep(WatchdogTimeout);
const string message = "System.Diagnostics.Process.Tests exceeded the diagnostic watchdog timeout.";
Log(message);
Environment.FailFast(message);
}
| Log("Reading Windows InstallationType."); | ||
| object? installationType = Registry.GetValue(InstallationTypeKey, "InstallationType", defaultValue: null); | ||
| Log($"InstallationType={installationType ?? "<null>"}"); | ||
|
|
||
| Log("Evaluating PlatformDetection.IsWindowsNanoServer and IsWindowsServerCore."); | ||
| bool isWindowsNanoServer = PlatformDetection.IsWindowsNanoServer; | ||
| bool isWindowsServerCore = PlatformDetection.IsWindowsServerCore; | ||
| Log($"IsWindowsNanoServer={isWindowsNanoServer}; IsWindowsServerCore={isWindowsServerCore}"); |
Focus and serialize Windows ProcessStartInfo tests, log startup and test ordering, and capture a WER dump before the Helix timeout.
Co-authored-by: Copilot App 223556219+Copilot@users.noreply.github.com