Enable artifact processing for truncated test runs

documentation/general/dotnet-test-artifact-post-processing.md

@@ -33,11 +33,12 @@ when any of the following is true:
 - `--list-tests` was requested (test discovery, not execution).
 - `--no-artifact-post-processing` was passed.
 - The run was cancelled with <kbd>Ctrl</kbd>+<kbd>C</kbd>.
-- The run was cut short by `--maximum-failed-tests` or `--timeout`. A truncated run produced
-  the artifacts of a truncated run — modules that never started contributed nothing, and
-  modules killed mid-flight wrote whatever they had — so merging them into one
-  authoritative-looking report would hide the truncation. The per-module artifacts are left
-  as they are.
+
+Runs cut short by `--maximum-failed-tests` or `--timeout` are handled differently. The SDK
+plans only artifact groups whose processor explicitly advertised support for policy-truncated
+runs. Other groups remain untouched, preventing an authoritative merger such as TRX or coverage
+from making an incomplete run look complete while still allowing summary-style processors to
+describe the truncation accurately.
 
 Post-processing can never change the run's exit code; it only affects which artifacts are
 listed. See [Failure behavior](#failure-behavior).
@@ -55,10 +56,13 @@ The grouping and election logic lives in
 During its handshake, each MTP test application reports the post-processors registered inside
 it through two properties: `SupportedPostProcessorKinds` (reverse-DNS artifact *kinds*, e.g.
 `microsoft.testing.trx`) and `SupportedPostProcessorExtensionsLegacy` (lowercase file
-extensions, for producers that do not tag a kind). Both are semicolon-separated. An
-application that advertises neither simply never participates — it is neither a candidate to
-perform a merge nor a source of mergeable groups. The handshake property ids are defined in
-[`CliConstants`](../../src/Cli/dotnet/Commands/Test/CliConstants.cs).
+extensions, for producers that do not tag a kind). Both are semicolon-separated.
+`SupportedTruncatedRunPostProcessorKinds` and
+`SupportedTruncatedRunPostProcessorExtensionsLegacy` advertise the subsets whose processors
+can consume the incomplete set of complete artifacts observed before
+`--maximum-failed-tests` or `--timeout` stopped the run. An application that advertises no
+capability applicable to the current run does not participate. The handshake property ids are
+defined in [`CliConstants`](../../src/Cli/dotnet/Commands/Test/CliConstants.cs).
 
 ### Grouping
 
@@ -68,9 +72,10 @@ Artifacts are de-duplicated by path and then grouped:
 2. **Extension fallback.** Artifacts with no kind are grouped by lowercase file extension.
 
 A group is a merge candidate only if a compatible application advertised the matching kind or
-extension, and only if it has **at least two inputs** — merging a single file is pointless. (A
-kind group and a matching extension group that individually have one input each can still be
-merged together when a shared application supports both and the combined count reaches two.)
+extension for the current run's completeness, and only if it has **at least two inputs** —
+merging a single file is pointless. (A kind group and a matching extension group that
+individually have one input each can still be merged together when a shared application supports
+both and the combined count reaches two.)
 
 For binary code-coverage artifacts (kind `microsoft.codecoverage` / extension `.coverage`),
 an application is a candidate only when its architecture matches the inputs. Coverage blobs
@@ -99,10 +104,16 @@ and manifest option are defined in
 
 The SDK writes a JSON manifest listing the input artifacts (path, kind, producing module,
 target framework, architecture, execution id) and the output directory, then hands it to the
-relaunched tool. The merged artifacts flow **back over the same pipe** as ordinary
-file-artifact messages, so they re-enter the normal reporter path. In the summary, the SDK
-[removes the inputs it consumed and adds the merged output](../../src/Cli/dotnet/Commands/Test/MTP/ArtifactPostProcessingManager.cs)
-in their place.
+relaunched tool. On a policy-truncated run, the manifest also contains `truncationReason` with
+the value `maximumFailedTests` or `timeout`; Microsoft.Testing.Platform exposes that through
+`ArtifactPostProcessingContext`.
+
+The processed artifacts flow **back over the same pipe** as ordinary file-artifact messages,
+along with the exact input paths each output consumed, so they re-enter the normal reporter
+path. In the summary, the SDK
+[removes only those consumed inputs and adds the processed output](../../src/Cli/dotnet/Commands/Test/MTP/ArtifactPostProcessingManager.cs)
+in their place. Older hosts that do not report input provenance retain the previous
+kind/extension-based replacement behavior.
 
 The original per-module artifacts are **never deleted from disk** — only the run summary
 changes. If you need the individual files (for example a per-module TRX), they are still where
@@ -149,8 +160,9 @@ The relevant public types live in `Microsoft.Testing.Platform.Extensions.Artifac
 
 | Type | Role |
 |---|---|
-| `IArtifactPostProcessor` | The contract: `SupportedKinds`, `SupportedFileExtensionsFallback`, and `ProcessAsync(inputs, outputDirectory, cancellationToken)`. |
+| `IArtifactPostProcessor` | The contract: `SupportsTruncatedRuns`, `SupportedKinds`, `SupportedFileExtensionsFallback`, and `ProcessAsync(inputs, outputDirectory, context, cancellationToken)`. |
 | `IArtifactPostProcessingManager` | Registration, via `AddArtifactPostProcessor(Func<IServiceProvider, IArtifactPostProcessor>)`. |
+| `ArtifactPostProcessingContext` | Whether the run was truncated and whether the reason was `MaximumFailedTests` or `Timeout`. |
 | `InputArtifact` | One input: path, kind, producing test module, target framework, architecture, execution id. |
 | `ProcessedArtifact` | The merged result: path, kind, display name, description. |
 
@@ -168,13 +180,17 @@ internal sealed class MyArtifactPostProcessor : IArtifactPostProcessor
     public string DisplayName => "Contoso report merger";
     public string Description => "Merges Contoso reports.";
 
+    public bool SupportsTruncatedRuns => false;
     public IReadOnlyList<string> SupportedKinds { get; } = ["contoso.myreport"];
     public IReadOnlyList<string> SupportedFileExtensionsFallback { get; } = [".myreport"];
 
     public Task<bool> IsEnabledAsync() => Task.FromResult(true);
 
     public async Task<ProcessedArtifact?> ProcessAsync(
-        IReadOnlyList<InputArtifact> inputs, string outputDirectory, CancellationToken cancellationToken)
+        IReadOnlyList<InputArtifact> inputs,
+        string outputDirectory,
+        ArtifactPostProcessingContext context,
+        CancellationToken cancellationToken)
     {
         if (inputs.Count < 2)
         {
@@ -202,7 +218,12 @@ What `dotnet test` expects of a processor:
 - **Treat inputs as read-only** and write under the supplied `outputDirectory`. Never return one of
   the inputs as your output, and never delete a source file.
 - **Set `Kind` on the artifact you return.** That is what the SDK uses to decide which originals the
-  merged artifact replaced in the run summary.
+  processed artifact represents. Microsoft.Testing.Platform separately reports the exact input
+  paths it consumed so the SDK removes only those originals from the run summary.
+- **Set `SupportsTruncatedRuns` only when the output can accurately describe an incomplete run.**
+  When enabled, inspect `context.TruncationReason` and make the truncation visible in the result.
+  The capability covers an incomplete set of complete artifacts, not malformed or partially
+  written files.
 - **Be deterministic**: the same set of inputs should produce the same output path.
 
 Two SDK behaviors are worth knowing about. The SDK relaunches the *fewest* test applications that

src/Cli/dotnet/Commands/Test/CliConstants.cs

@@ -94,6 +94,8 @@ internal static class HandshakeMessagePropertyNames
     // supported by post-processors registered in the test application.
     internal const byte SupportedPostProcessorKinds = 14;
     internal const byte SupportedPostProcessorExtensionsLegacy = 15;
+    internal const byte SupportedTruncatedRunPostProcessorKinds = 16;
+    internal const byte SupportedTruncatedRunPostProcessorExtensionsLegacy = 17;
 }
 
 internal static class HandshakeMessageExecutionModes

src/Cli/dotnet/Commands/Test/MTP/ArtifactPostProcessingManager.cs

@@ -25,8 +25,19 @@ public void RecordCapabilities(
         string[] extensions = ParseCapabilities(handshakeMessage, HandshakeMessagePropertyNames.SupportedPostProcessorExtensionsLegacy)
             .Select(extension => extension.ToLowerInvariant())
             .ToArray();
+        string[] truncatedRunKinds = ParseCapabilities(
+            handshakeMessage,
+            HandshakeMessagePropertyNames.SupportedTruncatedRunPostProcessorKinds);
+        string[] truncatedRunExtensions = ParseCapabilities(
+            handshakeMessage,
+            HandshakeMessagePropertyNames.SupportedTruncatedRunPostProcessorExtensionsLegacy)
+            .Select(extension => extension.ToLowerInvariant())
+            .ToArray();
 
-        if (kinds.Length == 0 && extensions.Length == 0)
+        if (kinds.Length == 0
+            && extensions.Length == 0
+            && truncatedRunKinds.Length == 0
+            && truncatedRunExtensions.Length == 0)
         {
             return;
         }
@@ -41,6 +52,8 @@ public void RecordCapabilities(
 
             application.SupportedKinds.UnionWith(kinds);
             application.SupportedExtensions.UnionWith(extensions);
+            application.SupportedTruncatedRunKinds.UnionWith(truncatedRunKinds);
+            application.SupportedTruncatedRunExtensions.UnionWith(truncatedRunExtensions);
         }
     }
 
@@ -59,18 +72,20 @@ public void RecordArtifact(
                 module.TargetPath,
                 targetFramework,
                 architecture,
-                executionId));
+                executionId,
+                artifact.InputArtifactPaths));
         }
     }
 
     public async Task ExecuteAsync(
         BuildOptions buildOptions,
         TerminalTestReporter output,
-        CtrlCCancellationManager ctrlC)
+        CtrlCCancellationManager ctrlC,
+        TestRunCancellationReason cancellationReason)
     {
         try
         {
-            await ExecuteCoreAsync(buildOptions, output, ctrlC);
+            await ExecuteCoreAsync(buildOptions, output, ctrlC, cancellationReason);
         }
         catch (Exception ex)
         {
@@ -85,11 +100,13 @@ public async Task ExecuteAsync(
     private async Task ExecuteCoreAsync(
         BuildOptions buildOptions,
         TerminalTestReporter output,
-        CtrlCCancellationManager ctrlC)
+        CtrlCCancellationManager ctrlC,
+        TestRunCancellationReason cancellationReason)
     {
         ArtifactPostProcessingPlan plan = ArtifactPostProcessingPlanner.Plan(
             SnapshotApplications(),
-            SnapshotArtifacts());
+            SnapshotArtifacts(),
+            cancellationReason);
 
         if (plan.Jobs.Count == 0)
         {
@@ -124,7 +141,11 @@ private async Task ExecuteCoreAsync(
                 string manifestPath = Path.Combine(tempDirectory, "manifest.json");
                 string outputDirectory = GetOutputDirectory(buildOptions, job);
                 Directory.CreateDirectory(outputDirectory);
-                WriteManifest(manifestPath, outputDirectory, job.Groups.SelectMany(group => group.Artifacts));
+                WriteManifest(
+                    manifestPath,
+                    outputDirectory,
+                    job.Groups.SelectMany(group => group.Artifacts),
+                    cancellationReason);
 
                 var invocation = new ArtifactPostProcessingInvocation(manifestPath);
                 var toolOptions = new TestOptions(
@@ -232,7 +253,9 @@ internal IReadOnlyList<ArtifactPostProcessingApplication> SnapshotApplications()
                     application.TargetFramework,
                     application.Architecture,
                     new HashSet<string>(application.SupportedKinds, StringComparer.Ordinal),
-                    new HashSet<string>(application.SupportedExtensions, StringComparer.Ordinal)))
+                    new HashSet<string>(application.SupportedExtensions, StringComparer.Ordinal),
+                    new HashSet<string>(application.SupportedTruncatedRunKinds, StringComparer.Ordinal),
+                    new HashSet<string>(application.SupportedTruncatedRunExtensions, StringComparer.Ordinal)))
             ];
         }
     }
@@ -281,17 +304,30 @@ .. job.Groups
         return Path.GetDirectoryName(Path.GetFullPath(preferredInput.Path))!;
     }
 
-    private static void WriteManifest(
+    internal static void WriteManifest(
         string manifestPath,
         string outputDirectory,
-        IEnumerable<ArtifactPostProcessingArtifact> artifacts)
+        IEnumerable<ArtifactPostProcessingArtifact> artifacts,
+        TestRunCancellationReason cancellationReason)
     {
         using FileStream stream = File.Create(manifestPath);
         using var writer = new Utf8JsonWriter(stream, new JsonWriterOptions { Indented = true });
 
         writer.WriteStartObject();
         writer.WriteNumber("schemaVersion", 1);
         writer.WriteString("outputDirectory", outputDirectory);
+        if (cancellationReason != TestRunCancellationReason.None)
+        {
+            writer.WriteString(
+                "truncationReason",
+                cancellationReason switch
+                {
+                    TestRunCancellationReason.MaximumFailedTests => "maximumFailedTests",
+                    TestRunCancellationReason.Timeout => "timeout",
+                    _ => throw new ArgumentOutOfRangeException(nameof(cancellationReason)),
+                });
+        }
+
         writer.WriteStartArray("inputs");
         foreach (ArtifactPostProcessingArtifact artifact in artifacts
             .OrderBy(artifact => artifact.Path, FileUtilities.PathComparer))
@@ -327,24 +363,36 @@ internal static void ApplyOutputs(
         ArtifactPostProcessingJob job,
         IReadOnlyList<ArtifactPostProcessingArtifact> processedArtifacts)
     {
+        var plannedInputPaths = new HashSet<string>(
+            job.Groups.SelectMany(group => group.Artifacts).Select(artifact => artifact.Path),
+            FileUtilities.PathComparer);
+
         foreach (ArtifactPostProcessingArtifact processedArtifact in processedArtifacts)
         {
-            string outputExtension = Path.GetExtension(processedArtifact.Path).ToLowerInvariant();
-            // The dispatcher gives one processor both its kind-tagged inputs and matching legacy
-            // extension inputs, so the returned output consumes both groups.
-            ArtifactPostProcessingGroup[] consumedGroups =
-            [
-                .. job.Groups.Where(group =>
-                    group.IsKind
-                        ? string.Equals(group.Key, processedArtifact.Kind, StringComparison.Ordinal)
-                        : string.Equals(group.Key, outputExtension, StringComparison.Ordinal))
-            ];
-
-            if (consumedGroups.Length > 0)
+            HashSet<string> consumedPaths;
+            if (processedArtifact.InputArtifactPaths is { } inputArtifactPaths)
+            {
+                consumedPaths = new HashSet<string>(inputArtifactPaths, FileUtilities.PathComparer);
+                consumedPaths.IntersectWith(plannedInputPaths);
+            }
+            else
             {
-                var consumedPaths = new HashSet<string>(
-                    consumedGroups.SelectMany(group => group.Artifacts).Select(artifact => artifact.Path),
+                string outputExtension = Path.GetExtension(processedArtifact.Path).ToLowerInvariant();
+                // Older post-processing hosts do not report input provenance. Preserve their
+                // replacement behavior by inferring consumed groups from the output kind/extension.
+                consumedPaths = new HashSet<string>(
+                    job.Groups
+                        .Where(group =>
+                            group.IsKind
+                                ? string.Equals(group.Key, processedArtifact.Kind, StringComparison.Ordinal)
+                                : string.Equals(group.Key, outputExtension, StringComparison.Ordinal))
+                        .SelectMany(group => group.Artifacts)
+                        .Select(artifact => artifact.Path),
                     FileUtilities.PathComparer);
+            }
+
+            if (consumedPaths.Count > 0)
+            {
                 output.RemoveArtifacts(consumedPaths);
             }
 
@@ -366,6 +414,8 @@ private sealed class ApplicationState(TestModule module, string? targetFramework
         public string? Architecture { get; } = architecture;
         public HashSet<string> SupportedKinds { get; } = new(StringComparer.Ordinal);
         public HashSet<string> SupportedExtensions { get; } = new(StringComparer.Ordinal);
+        public HashSet<string> SupportedTruncatedRunKinds { get; } = new(StringComparer.Ordinal);
+        public HashSet<string> SupportedTruncatedRunExtensions { get; } = new(StringComparer.Ordinal);
     }
 }
 
@@ -411,7 +461,8 @@ public void RecordOutput(
                 module.TargetPath,
                 targetFramework,
                 architecture,
-                executionId));
+                executionId,
+                artifact.InputArtifactPaths));
         }
     }
 

src/Cli/dotnet/Commands/Test/MTP/ArtifactPostProcessingPlanner.cs

@@ -11,15 +11,18 @@ internal sealed record ArtifactPostProcessingApplication(
     string? TargetFramework,
     string? Architecture,
     IReadOnlySet<string> SupportedKinds,
-    IReadOnlySet<string> SupportedExtensions);
+    IReadOnlySet<string> SupportedExtensions,
+    IReadOnlySet<string> SupportedTruncatedRunKinds,
+    IReadOnlySet<string> SupportedTruncatedRunExtensions);
 
 internal sealed record ArtifactPostProcessingArtifact(
     string Path,
     string? Kind,
     string ProducingTestModule,
     string? TargetFramework,
     string? Architecture,
-    string ExecutionId);
+    string ExecutionId,
+    IReadOnlyList<string>? InputArtifactPaths = null);
 
 internal sealed record ArtifactPostProcessingGroup(
     string Key,
@@ -40,8 +43,10 @@ internal static class ArtifactPostProcessingPlanner
 
     public static ArtifactPostProcessingPlan Plan(
         IReadOnlyList<ArtifactPostProcessingApplication> applications,
-        IReadOnlyList<ArtifactPostProcessingArtifact> artifacts)
+        IReadOnlyList<ArtifactPostProcessingArtifact> artifacts,
+        TestRunCancellationReason cancellationReason = TestRunCancellationReason.None)
     {
+        bool isTruncated = cancellationReason != TestRunCancellationReason.None;
         ArtifactPostProcessingArtifact[] distinctArtifacts =
             [.. artifacts.DistinctBy(artifact => artifact.Path, FileUtilities.PathComparer)];
         List<ArtifactPostProcessingGroup> groups = [];
@@ -109,9 +114,7 @@ void AddGroup(string key, bool isKind, ArtifactPostProcessingArtifact[] inputs)
             ArtifactPostProcessingApplication[] candidates =
             [
                 .. applications.Where(application =>
-                    (isKind
-                        ? application.SupportedKinds.Contains(key)
-                        : application.SupportedExtensions.Contains(key))
+                    GetCapabilities(application, isKind, isTruncated).Contains(key)
                     && IsArchitectureCompatible(key, isKind, application, inputs))
             ];
 
@@ -122,6 +125,18 @@ .. applications.Where(application =>
         }
     }
 
+    private static IReadOnlySet<string> GetCapabilities(
+        ArtifactPostProcessingApplication application,
+        bool isKind,
+        bool isTruncated)
+        => (isTruncated, isKind) switch
+        {
+            (true, true) => application.SupportedTruncatedRunKinds,
+            (true, false) => application.SupportedTruncatedRunExtensions,
+            (false, true) => application.SupportedKinds,
+            (false, false) => application.SupportedExtensions,
+        };
+
     private static bool CanCombineKindAndExtensionGroups(
         ArtifactPostProcessingGroup first,
         ArtifactPostProcessingGroup second)