Skip to content

Add diagnostics for Process test hangs - #131907

Draft
steveisok wants to merge 5 commits into
dotnet:mainfrom
steveisok:steveisok-debug-process-timeout
Draft

Add diagnostics for Process test hangs#131907
steveisok wants to merge 5 commits into
dotnet:mainfrom
steveisok:steveisok-debug-process-timeout

Conversation

@steveisok

Copy link
Copy Markdown
Member

Focus and serialize Windows ProcessStartInfo tests, log startup and test ordering, and capture a WER dump before the Helix timeout.

Co-authored-by: Copilot App 223556219+Copilot@users.noreply.github.com

Focus and serialize Windows ProcessStartInfo tests, log startup and test ordering, and capture a WER dump before the Helix timeout.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 54c890f9-b409-4b7d-b810-3701e125fd96
Copilot AI review requested due to automatic review settings August 5, 2026 22:06
@steveisok

Copy link
Copy Markdown
Member Author

/azp run runtime-libraries-coreclr outerloop-windows

@steveisok

Copy link
Copy Markdown
Member Author

/azp run runtime-nativeaot-outerloop

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 3 pipeline(s).
13 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds additional diagnostics and run-configuration changes to help investigate hangs in System.Diagnostics.Process tests, primarily on Windows. The PR introduces both logging (per-test + targeted messages) and a watchdog/WER dump configuration mechanism.

Changes:

  • Windows-only test execution configuration updates (disable parallelization, enable progress output, and currently filters to a single test class via XUnitOptions).
  • New diagnostics helper with a module initializer, WER LocalDumps configuration attempt, and a FailFast watchdog.
  • Adds assembly-level xUnit BeforeAfterTestAttribute hook and adds extra diagnostic logging in a couple of ProcessStartInfoTests.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.

File Description
src/libraries/System.Diagnostics.Process/tests/System.Diagnostics.Process.Tests.csproj Adds Windows-only test runner settings (parallelization/progress and class filtering) and includes the new diagnostics source file.
src/libraries/System.Diagnostics.Process/tests/ProcessTestHangDiagnostics.cs New diagnostics implementation: per-test logging hook, module initializer logging, WER LocalDumps setup, and watchdog FailFast timeout.
src/libraries/System.Diagnostics.Process/tests/ProcessStartInfoTests.cs Adds targeted diagnostic logs around specific test execution points.
src/libraries/System.Diagnostics.Process/tests/AssemblyInfo.cs Registers the diagnostics attribute at assembly level.
Suppressed comments (1)

src/libraries/System.Diagnostics.Process/tests/ProcessTestHangDiagnostics.cs:83

  • Configuring WER LocalDumps under HKLM requires admin and will typically fail in CI/Helix (and if it succeeds, it leaves a machine-wide setting behind). Using HKCU avoids the elevation requirement and limits scope to the current user, making it more likely the dump configuration actually takes effect.
            {
                using RegistryKey? key = Registry.LocalMachine.CreateSubKey(keyPath);
                if (key is null)
                {
                    Log($"Unable to create WER LocalDumps key HKLM\\{keyPath}.");

Comment on lines +14 to +18
<PropertyGroup Condition="'$(TargetPlatformIdentifier)' == 'windows'">
<TestDisableParallelization>true</TestDisableParallelization>
<XUnitOptions>$(XUnitOptions) -class System.Diagnostics.Tests.ProcessStartInfoTests -parallel none</XUnitOptions>
<XUnitShowProgress>true</XUnitShowProgress>
</PropertyGroup>
Comment on lines +41 to +48
ConfigureWindowsErrorReporting();

var watchdog = new Thread(Watchdog)
{
IsBackground = true,
Name = "Process tests hang watchdog"
};
watchdog.Start();
Comment on lines 8 to 10
[assembly: CollectionBehavior(CollectionBehavior.CollectionPerAssembly)]
[assembly: System.Diagnostics.Tests.ProcessTestHangDiagnosticsAttribute]

@dotnet-policy-service

Copy link
Copy Markdown
Contributor

Tagging subscribers to this area: @dotnet/area-system-diagnostics-process
See info in area-owners.md if you want to be subscribed.

Write BOM-less UTF-8 console output and place WER dumps in the Helix work-item upload directory.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 54c890f9-b409-4b7d-b810-3701e125fd96
Copilot AI review requested due to automatic review settings August 6, 2026 14:30
@steveisok

Copy link
Copy Markdown
Member Author

/azp run runtime-libraries-coreclr outerloop-windows

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).

@steveisok

Copy link
Copy Markdown
Member Author

/azp run runtime-nativeaot-outerloop

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

Suppressed comments (3)

src/libraries/System.Diagnostics.Process/tests/System.Diagnostics.Process.Tests.csproj:17

  • The hard-coded -class System.Diagnostics.Tests.ProcessStartInfoTests filter in XUnitOptions means Windows runs of this test project will only execute that single test class, skipping the rest of the Process test suite on Windows (significant test coverage reduction). This kind of focusing should be passed as a CI/test invocation parameter (e.g., /p:XUnitClassName=...) rather than baked into the project file.
  <PropertyGroup Condition="'$(TargetPlatformIdentifier)' == 'windows'">
    <TestDisableParallelization>true</TestDisableParallelization>
    <XUnitOptions>$(XUnitOptions) -class System.Diagnostics.Tests.ProcessStartInfoTests -parallel none</XUnitOptions>
    <XUnitShowProgress>true</XUnitShowProgress>

src/libraries/System.Diagnostics.Process/tests/ProcessTestHangDiagnostics.cs:49

  • The watchdog thread is started unconditionally from the module initializer and will FailFast after 3 minutes even for non-hung runs (e.g., slow machines or local debugging). Consider gating this to Helix runs (or an explicit opt-in env var) so it doesn’t introduce new crash behavior in normal Windows test execution.
            var watchdog = new Thread(Watchdog)
            {
                IsBackground = true,
                Name = "Process tests hang watchdog"
            };
            watchdog.Start();

src/libraries/System.Diagnostics.Process/tests/ProcessTestHangDiagnostics.cs:87

  • Configuring WER LocalDumps via Registry.LocalMachine writes machine-wide state and will typically require elevated permissions; if it fails (UnauthorizedAccess), the dump capture won’t be configured at all. Prefer using a per-user key so the diagnostics can work under normal test permissions and avoid persisting HKLM changes.
                using RegistryKey? key = Registry.LocalMachine.CreateSubKey(keyPath);
                if (key is null)
                {
                    Log($"Unable to create WER LocalDumps key HKLM\\{keyPath}.");
                    return;

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings August 6, 2026 20:02
@steveisok

Copy link
Copy Markdown
Member Author

/azp run runtime-libraries-coreclr outerloop-windows

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).

@steveisok

Copy link
Copy Markdown
Member Author

/azp run runtime-nativeaot-outerloop

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

Note

This error may be related to your runner configuration. You can now configure runners for Copilot code review separately from Copilot cloud agent by creating a copilot-code-review.yml file with your setup steps. Read the docs for details.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 6a078136-b226-4ce5-94ff-1732ebcdfe79
Copilot AI review requested due to automatic review settings August 6, 2026 23:57
@steveisok

Copy link
Copy Markdown
Member Author

/azp run runtime-nativeaot-outerloop

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

Suppressed comments (6)

src/libraries/System.Diagnostics.Process/tests/System.Diagnostics.Process.Tests.csproj:17

  • XUnitOptions currently includes -class System.Diagnostics.Tests.ProcessStartInfoTests (and XUnitShowProgress enables -verbose). The -class filter will skip all other test classes for Windows TFMs, reducing coverage and potentially masking regressions; the verbose output is also unusually chatty to enable unconditionally.
    <TestDisableParallelization>true</TestDisableParallelization>
    <XUnitOptions>$(XUnitOptions) -class System.Diagnostics.Tests.ProcessStartInfoTests -parallel none</XUnitOptions>
    <XUnitShowProgress>true</XUnitShowProgress>

src/libraries/System.Diagnostics.Process/tests/ProcessTestHangDiagnostics.cs:47

  • The watchdog thread is started unconditionally, but Watchdog() currently calls FailFast after 3 minutes. That means any Windows run longer than 3 minutes will be terminated even if it’s making progress.
            var watchdog = new Thread(Watchdog)
            {
                IsBackground = true,
                Name = "Process tests hang watchdog"
            };

src/libraries/System.Diagnostics.Process/tests/ProcessTestHangDiagnostics.cs:63

  • To support an inactivity-based watchdog, Log should update the last-activity timestamp in a thread-safe way before writing output.
        internal static void Log(string message)
        {
            s_log.WriteLine($"[Process test hang diagnostics] {message}");
        }

src/libraries/System.Diagnostics.Process/tests/ProcessTestHangDiagnostics.cs:104

  • Watchdog() currently sleeps once and then unconditionally FailFasts. With an inactivity timestamp, this can instead trigger only when no test progress has been logged for the timeout window.
        private static void Watchdog()
        {
            Thread.Sleep(WatchdogTimeout);
            const string message = "System.Diagnostics.Process.Tests exceeded the diagnostic watchdog timeout.";
            Log(message);

src/libraries/System.Diagnostics.Process/tests/ProcessTestHangDiagnostics.cs:35

  • The watchdog currently enforces an absolute timeout. Tracking last test activity and having the watchdog trigger on inactivity avoids terminating long-but-progressing test runs.

This issue also appears in the following locations of the same file:

  • line 60
  • line 100
        private static readonly TimeSpan WatchdogTimeout = TimeSpan.FromMinutes(3);
        private static readonly TextWriter s_log = TextWriter.Synchronized(
            new StreamWriter(Console.OpenStandardError(), new UTF8Encoding(encoderShouldEmitUTF8Identifier: false), bufferSize: 1024, leaveOpen: true) { AutoFlush = true });

src/libraries/System.Diagnostics.Process/tests/ProcessTestHangDiagnostics.cs:52

  • Registry.GetValue can throw (e.g., access denied / security policy). Since this is diagnostic-only logging, it should not be able to fail the entire test assembly.
            Log("Reading Windows InstallationType.");
            object? installationType = Registry.GetValue(InstallationTypeKey, "InstallationType", defaultValue: null);
            Log($"InstallationType={installationType ?? "<null>"}");

Copilot AI review requested due to automatic review settings August 7, 2026 00:21
@steveisok

Copy link
Copy Markdown
Member Author

/azp run runtime-nativeaot-outerloop

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

Suppressed comments (3)

src/libraries/System.Diagnostics.Process/tests/System.Diagnostics.Process.Tests.csproj:18

  • -class System.Diagnostics.Tests.ProcessStartInfoTests in XUnitOptions filters the Windows test run down to a single test class, which effectively disables the rest of the System.Diagnostics.Process test coverage on Windows. If the goal is only to serialize and add diagnostics, drop the class filter and keep only the parallelization settings.
  <PropertyGroup Condition="'$(TargetPlatformIdentifier)' == 'windows'">
    <TestDisableParallelization>true</TestDisableParallelization>
    <XUnitOptions>$(XUnitOptions) -class System.Diagnostics.Tests.ProcessStartInfoTests -parallel none</XUnitOptions>
    <XUnitShowProgress>true</XUnitShowProgress>
  </PropertyGroup>

src/libraries/System.Diagnostics.Process/tests/ProcessTestHangDiagnostics.cs:63

  • The watchdog currently measures wall-clock time since module initialization, so any legitimate run that takes longer than the watchdog timeout can be terminated even if tests are making progress. Track the most recent test activity and have the watchdog trigger only after a period of inactivity.
        internal static void Log(string message)
        {
            s_log.WriteLine($"[Process test hang diagnostics] {message}");
        }

src/libraries/System.Diagnostics.Process/tests/ProcessTestHangDiagnostics.cs:106

  • Watchdog unconditionally calls FailFast after a single sleep, regardless of whether tests are actively progressing. Switching to an inactivity-based loop avoids terminating long-but-healthy runs and still triggers quickly when a test actually hangs.
        private static void Watchdog()
        {
            Thread.Sleep(WatchdogTimeout);
            const string message = "System.Diagnostics.Process.Tests exceeded the diagnostic watchdog timeout.";
            Log(message);
            Environment.FailFast(message);
        }

Comment on lines +50 to +57
Log("Reading Windows InstallationType.");
object? installationType = Registry.GetValue(InstallationTypeKey, "InstallationType", defaultValue: null);
Log($"InstallationType={installationType ?? "<null>"}");

Log("Evaluating PlatformDetection.IsWindowsNanoServer and IsWindowsServerCore.");
bool isWindowsNanoServer = PlatformDetection.IsWindowsNanoServer;
bool isWindowsServerCore = PlatformDetection.IsWindowsServerCore;
Log($"IsWindowsNanoServer={isWindowsNanoServer}; IsWindowsServerCore={isWindowsServerCore}");
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants