Skip to content

Migrate Csc to multithreaded MSBuild execution - #84500

Closed
OvesN wants to merge 10 commits into
dotnet:mainfrom
OvesN:dev/veronikao/migrate-csc
Closed

Migrate Csc to multithreaded MSBuild execution#84500
OvesN wants to merge 10 commits into
dotnet:mainfrom
OvesN:dev/veronikao/migrate-csc

Conversation

@OvesN

@OvesN OvesN commented Jul 14, 2026

Copy link
Copy Markdown

Fixes dotnet/msbuild#14230

Summary

Migrates the Csc MSBuild task to MSBuild's multithreaded task execution model while preserving existing compiler arguments, server reuse, cache behavior, and process-wide temporary-directory behavior.

Changes

Opt Csc into multithreaded execution

  • Adds [MSBuildMultiThreadableTask] to the concrete Csc task. The attribute is intentionally not added to ManagedCompiler, ManagedToolTask, or Vbc.
  • Uses the TaskEnvironment inherited from MSBuild 18.7.1's ToolTask.

Resolve task-local state through TaskEnvironment

  • Uses TaskEnvironment.ProjectDirectory for the working directory sent in compiler-server requests instead of the MSBuild process CWD.
  • Reads LIB, runtime-host variables, tiered-compilation settings, ROSLYN_CACHE_PATH, and command-line logger configuration from TaskEnvironment.
  • Resolves reference probes and command-line source normalization through TaskEnvironment.GetAbsolutePath.
  • Adds a GetFullPathNoThrow overload so task absolutization and canonicalization remain inside the existing I/O-related exception filter.
  • Preserves original null/empty reference behavior: invalid references log MSB3104 instead of throwing during absolutization.

Keep compiler process startup task-aware

  • Passes TaskEnvironment.GetProcessStartInfo() when the task needs to start VBCSCompiler.
  • Keeps runtime-host selection, DOTNET_ROOT calculation, pipe arguments, and server process construction owned by BuildServerConnection.
  • Adds task-aware overloads that receive environment/path accessors without introducing an MSBuild dependency into shared compiler-client code.

Make command-line logging task-aware

  • Adds a BCL-delegate constructor to CompilerServerLogger so the MSBuild task can read the logging variable from TaskEnvironment and resolve relative log paths against the task project directory.
  • Keeps the existing constructor and standalone csc, vbc, and VBCSCompiler process behavior unchanged.
Microsoft Reviewers: Open in CodeFlow

JanProvaznik and others added 3 commits July 9, 2026 11:05
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…for 18.7

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 2 pipeline(s).
There may be pipelines that require an authorized user to comment /azp run to run.

@dotnet-policy-service dotnet-policy-service Bot added the Community The pull request was submitted by a contributor who is not a Microsoft employee. label Jul 14, 2026
OvesN and others added 2 commits July 15, 2026 09:27
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 4145242b-ec7c-4ff0-bace-d006bf403feb
@OvesN
OvesN marked this pull request as ready for review July 27, 2026 12:42
@OvesN
OvesN requested a review from a team as a code owner July 27, 2026 12:42
Copilot AI review requested due to automatic review settings July 27, 2026 12:42
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).
1 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates Roslyn’s MSBuild Csc task and its supporting compiler-server plumbing to work with MSBuild’s multithreaded task execution model by routing environment/path/working-directory decisions through MSBuild’s TaskEnvironment instead of process-global state.

Changes:

  • Opts Csc into multithreaded execution and adjusts MSBuild task behavior to prefer TaskEnvironment (working directory, env vars, path absolutization).
  • Adds task-aware overloads in shared compiler-server utilities to allow server startup/environment decisions to be based on a caller-provided environment/path snapshot.
  • Adds/updates unit tests covering injected environment/path handling and task-directory-relative path resolution.

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
src/Compilers/Shared/RuntimeHostInfo.cs Adds overloads to resolve dotnet path / DOTNET_ROOT using injected env/path accessors.
src/Compilers/Shared/CompilerServerLogger.cs Adds constructor overload to inject env lookup + path absolutization for log-file resolution.
src/Compilers/Shared/BuildServerConnection.cs Adds task-aware server request overloads and environment/process-info helpers for server startup.
src/Compilers/Server/VBCSCompilerTests/BuildServerConnectionTests.cs Adds regression tests validating snapshot-based env/path behavior.
src/Compilers/Core/Portable/InternalUtilities/CompilerOptionParseUtilities.cs Allows feature-flag injection to read ROSLYN_CACHE_PATH via injected env accessor.
src/Compilers/Core/MSBuildTaskTests/RuntimeHostInfoTests.cs Adds test ensuring dotnet discovery works with TaskEnvironment-relative PATH entries.
src/Compilers/Core/MSBuildTaskTests/ManagedCompilerGlobalCacheTests.cs Minor cleanup consistent with env-driven feature-flag behavior.
src/Compilers/Core/MSBuildTaskTests/CscTests.cs Adds tests for task-environment dotnet host path and reference resolution against project directory.
src/Compilers/Core/MSBuildTask/Utilities.cs Adds task-environment-aware GetFullPathNoThrow overload.
src/Compilers/Core/MSBuildTask/ManagedToolTask.cs Uses TaskEnvironment for dotnet discovery and runtime-host related environment behavior.
src/Compilers/Core/MSBuildTask/ManagedCompiler.cs Uses TaskEnvironment for server logger, env probing, path normalization, and server startup.
src/Compilers/Core/MSBuildTask/Csc.cs Marks Csc as [MSBuildMultiThreadableTask].

Comment thread src/Compilers/Shared/BuildServerConnection.cs Outdated
Comment thread src/Compilers/Core/MSBuildTask/ManagedCompiler.cs Outdated
@333fred

333fred commented Jul 27, 2026

Copy link
Copy Markdown
Member

Test failures look legitimate. I have not reviewed the code.

@jaredpar jaredpar left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This change is leaving problematic APIs like Environment.GetEnvironmentVariable in place and callable from code. Also the passing around of ProcessStartInfo in our creation APIs seems to break invariants and it's hard to see a justification for why this is happening.

Overall this change seems to be made from manual inspection of our code vs. running the MT analyzer over it. Is that the case? If so why aren't we using the analyzer here?

Comment thread src/Compilers/Shared/RuntimeHostInfo.cs Outdated
try
{
var filePath = Path.Combine(item, fileName);
var filePath = getFullPath(Path.Combine(item, fileName));

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the wrong fix. If a path in this is not absolute we should just skip it. Yes I realize that the code was broken before but honestly didn't think about that. Don't want to perpetuate the bad behavior.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed, If path is not absolute we skip it now.

Comment thread src/Compilers/Shared/RuntimeHostInfo.cs Outdated
Comment on lines +87 to +88
internal static string GetDotNetPathOrDefault() =>
GetDotNetPathOrDefault(Environment.GetEnvironmentVariable, static path => path);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This feels like the wrong fix here. The is hiding us using process wide state behind a pleasant looking API. We should be threading through the Func<string, string?> getEnvironment parameter here to.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now only overload with explicit Func<string, string?> getEnvironment parameter is called.

/// Static class initializer that initializes logging.
/// </summary>
public CompilerServerLogger(string identifier, string? loggingFilePath = null)
: this(identifier, loggingFilePath, Environment.GetEnvironmentVariable, static path => path)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same feedback about using Environment.GetEnvironmentVariable just hiding the problem here. This should be threaded through.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed.

string pipeName,
Func<string, string?> getEnvironmentVariable,
Func<string, string> getFullPath,
ProcessStartInfo? processStartInfo,

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This breaks contracts around the API. It means that on Windows and Linux we can get different behaviors. There needs to be a substantial reason for adding this here vs. fixing up the implementation to create the process correctly (on all OS).

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are right, I will not just pass a snapshot of current environmentVariables from task

TryCreateServerCore(
clientDirectory,
pipeName,
Environment.GetEnvironmentVariable,

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Similar feedback about this needing to be removed.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed, I now pass snapshot from environmentVariables

internal static bool TryCreateServer(
string clientDirectory,
string pipeName,
ProcessStartInfo processStartInfo,

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same feedback about this breaking our guarantees around TryCreateServer.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed

Comment on lines +586 to +594
// Clear DOTNET_ROOT* variables such as DOTNET_ROOT_X64 before setting our own DOTNET_ROOT.
foreach (var key in environmentVariables.Keys.ToArray())
{
if (key.StartsWith(RuntimeHostInfo.DotNetRootEnvironmentName, StringComparison.OrdinalIgnoreCase))
{
environmentVariables[key] = string.Empty;
}
}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the reason for changing where this logic happens?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I moved it while extracting the common helper but it was indeed unnecessary. I reverted it

@OvesN

OvesN commented Jul 29, 2026

Copy link
Copy Markdown
Author

@jaredpar
If I understood it correctly, pipelines are failing because Roslyn compiles the Csc task against MSBuild 18.7.1, but its SDK integration tests run it inside SDK 10.0.301, which contains MSBuild 18.6.4. That host does not have  ToolTask.TaskEnvironment , causing  MissingMethodException.
dotnet/sdk#55230 raised the baseline to 18.7, but Roslyn still pins the older SDK. Should Roslyn update  global.json , or should Csc temporarily remain compatible with MSBuild 18.6?

…host

discovery and compiler server logging.

Skip relative PATH entries instead of trying to absolutize it.
Copilot AI review requested due to automatic review settings July 29, 2026 14:33
@OvesN
OvesN requested a review from a team as a code owner July 29, 2026 14:33

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 19 out of 19 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

src/Compilers/Shared/CompilerServerLogger.cs:121

  • The PR description says the existing CompilerServerLogger(string identifier, string? loggingFilePath = null) constructor is preserved, but this file currently only exposes the injected-delegate constructor. Reintroducing the original convenience overload keeps the API stable for non-MSBuild callers and avoids the repeated boilerplate at call sites that don't need task-specific environment/path behavior.
        /// <summary>
        /// Initializes logging using the supplied environment variable lookup and path absolutization.
        /// </summary>
        /// <param name="getEnvironmentVariable">Reads the named environment variable.</param>
        /// <param name="makeAbsolutePath">
        /// Resolves a (possibly relative) path to an absolute one before it is used for file system
        /// access.
        /// </param>
        public CompilerServerLogger(
            string identifier,
            string? loggingFilePath,
            Func<string, string?> getEnvironmentVariable,
            Func<string, string> makeAbsolutePath)
        {

…d of

caller-owned ProcessStartInfo instances. Keep process construction consistent
across Windows and Unix

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 21 out of 21 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (1)

src/Compilers/Shared/CompilerServerLogger.cs:121

  • The PR description says the existing CompilerServerLogger constructor is kept, but this change replaces it with a delegate-based constructor and forces all call sites to pass Environment.GetEnvironmentVariable / identity path mapping. Consider reintroducing the original convenience overload (forwarding to the new constructor) to match the stated compatibility goal and avoid repeating boilerplate across entry points.
        /// <summary>
        /// Initializes logging using the supplied environment variable lookup and path absolutization.
        /// </summary>
        /// <param name="getEnvironmentVariable">Reads the named environment variable.</param>
        /// <param name="makeAbsolutePath">
        /// Resolves a (possibly relative) path to an absolute one before it is used for file system
        /// access.
        /// </param>
        public CompilerServerLogger(
            string identifier,
            string? loggingFilePath,
            Func<string, string?> getEnvironmentVariable,
            Func<string, string> makeAbsolutePath)
        {

Comment thread src/Compilers/Core/MSBuildTask/ManagedCompiler.cs
Copilot AI review requested due to automatic review settings July 29, 2026 18:59

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 21 out of 21 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (2)

src/Compilers/Shared/CompilerServerLogger.cs:121

  • The PR description indicates the existing CompilerServerLogger constructor is kept, but this file now only exposes the new delegate-based constructor. Consider reintroducing the prior overload so existing call sites (and external/internal consumers) can keep using the simple API while new task-aware callers can opt into the injected behaviors.
        /// <summary>
        /// Initializes logging using the supplied environment variable lookup and path absolutization.
        /// </summary>
        /// <param name="getEnvironmentVariable">Reads the named environment variable.</param>
        /// <param name="makeAbsolutePath">
        /// Resolves a (possibly relative) path to an absolute one before it is used for file system
        /// access.
        /// </param>
        public CompilerServerLogger(
            string identifier,
            string? loggingFilePath,
            Func<string, string?> getEnvironmentVariable,
            Func<string, string> makeAbsolutePath)
        {

src/Compilers/Core/MSBuildTask/ManagedCompiler.cs:1234

  • CheckAllReferencesExistOnDisk now calls TaskEnvironment.GetAbsolutePath(itemSpec) inside File.Exists. If GetAbsolutePath(...).Value throws for an invalid ItemSpec, this would regress the pre-migration behavior where invalid references were treated as missing (logging MSB3104) rather than throwing. Consider guarding the absolutization and treating failures as missing references.
                var itemSpec = reference.ItemSpec;

                if (string.IsNullOrEmpty(itemSpec) || !File.Exists(this.TaskEnvironment.GetAbsolutePath(itemSpec)))
                {

Copilot AI review requested due to automatic review settings July 29, 2026 19:10

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 21 out of 21 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (2)

src/Compilers/Shared/CompilerServerLogger.cs:121

  • The PR description says the existing CompilerServerLogger(string identifier, string? loggingFilePath = null) constructor is kept, but this file only has the injected-delegate constructor now. If the intent is to preserve the existing API and keep standalone tool call sites simpler, reintroduce the old overload and forward it to the injected implementation.
        /// <summary>
        /// Initializes logging using the supplied environment variable lookup and path absolutization.
        /// </summary>
        /// <param name="getEnvironmentVariable">Reads the named environment variable.</param>
        /// <param name="makeAbsolutePath">
        /// Resolves a (possibly relative) path to an absolute one before it is used for file system
        /// access.
        /// </param>
        public CompilerServerLogger(
            string identifier,
            string? loggingFilePath,
            Func<string, string?> getEnvironmentVariable,
            Func<string, string> makeAbsolutePath)
        {

src/Compilers/Shared/BuildServerConnection.cs:538

  • GetServerEnvironmentVariables is declared as returning a nullable dictionary, and the call sites still branch on environmentVariables != null, but the implementation always constructs and returns a dictionary. Consider either (1) making the method non-nullable and simplifying the null checks, or (2) reintroducing a real null return for the “inherit current process environment” case (if that’s still a supported scenario), to avoid dead code and confusing nullability.
        /// <param name="logger">Optional logger for logging environment variable setup</param>
        /// <returns>Dictionary of environment variables to set</returns>
        internal static Dictionary<string, string>? GetServerEnvironmentVariables(
            Func<string, string?> getEnvironmentVariable,
            IReadOnlyList<KeyValuePair<string, string?>> currentEnvironment,
            ICompilerServerLogger? logger = null)

@OvesN

OvesN commented Jul 29, 2026

Copy link
Copy Markdown
Author

This change is leaving problematic APIs like Environment.GetEnvironmentVariable in place and callable from code. Also the passing around of ProcessStartInfo in our creation APIs seems to break invariants and it's hard to see a justification for why this is happening.

Overall this change seems to be made from manual inspection of our code vs. running the MT analyzer over it. Is that the case? If so why aren't we using the analyzer here?

I think we do not call Environment.GetEnvironmentVariable in the Csc task path? This is true that I started migration through manual code inspection first. The only really unsafe API remains is Path.GetTempPath(). As you discussed with Chet our suggestion in analyzer is not really valid for getTempPath so I need to discuss with my team what we should do in this case.

@jaredpar

Copy link
Copy Markdown
Member

Should Roslyn update global.json , or should Csc temporarily remain compatible with MSBuild 18.6?

What .NET SDK version has the appropriate MSBuild DLLs in it?

The only really unsafe API remains is Path.GetTempPath(). As you discussed with Chet our suggestion in analyzer is not really valid for getTempPath so I need to discuss with my team what we should do in this case.

Not exactly sure what to do here. We have a good understanding fo what the algorithm is but rather than everyone implement it I think it would be best to have a helper in the MSBuild APIs that we can use instead.

buildRequest,
pipeName,
GetClientDirectory(),
Environment.GetEnvironmentVariable,

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should add a Func<string, string> parameter here and require callers to pass the appropriate implementation. My mental model is that eventually these uses are going to be flagged by your analyzer. I want us to be in a state where we have the least number of suppressions possible that reduces the risk that we get anything wrong. For cases like this we should either be

  1. Abstracting out to Func<string, string> and forcing the caller to thread through the appropraite method. In the case of the tasks they will thread through TaskEnvironment.GetEnvironmnetVariable for callers like csc.exe they will thread through Environment.GetEnvironmentVariable.
  2. Add an entry in the msbuild task project file like <DefineConstant>$(DefineConstant);ROSLYN_MSBUILD_TASK and then wrap this in an #if !ROSLYN_MSBUILD_TASK

@baronfel

string pipeName,
string clientDirectory,
Func<string, string?> getEnvironmentVariable,
IReadOnlyList<KeyValuePair<string, string?>> environmentVariables,

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems unnecessary. It's the same information provided two ways. Can we just pass a dictionary here that way we get both in one parameter?

Comment on lines +517 to +527
internal static KeyValuePair<string, string?>[] CreateEnvironmentVariableSnapshot(
IEnumerable<KeyValuePair<string, string>> currentEnvironment)
{
var result = new List<KeyValuePair<string, string?>>();
foreach (var entry in currentEnvironment)
{
result.Add(new KeyValuePair<string, string?>(entry.Key, entry.Value));
}

return result.ToArray();
}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems unnecessary, callers could just use .ToArray()

Comment on lines +505 to +515
internal static KeyValuePair<string, string?>[] CreateEnvironmentVariableSnapshot(System.Collections.IDictionary currentEnvironment)
{
var result = new KeyValuePair<string, string?>[currentEnvironment.Count];
var index = 0;
foreach (System.Collections.DictionaryEntry entry in currentEnvironment)
{
result[index++] = new KeyValuePair<string, string?>((string)entry.Key, (string?)entry.Value);
}

return result;
}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems unnecessary, callers can just use .OfType<DictionaryEntry>().ToArray(x => KeyValuePair.Create(x.Key, x.Value)

Comment on lines +591 to +592
Func<string, string?> getEnvironmentVariable,
IReadOnlyList<KeyValuePair<string, string?>> environmentVariablesSnapshot,

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is one of my top areas of concern about these changes. I 100% get the necessity of no longer using Environment directly in our tasks now. At the same time it seems like there is no interface / API that is a suitable replacement for it. Instead we're getting a mix of Func<string, string>, IROList, IDictionary, etc ...

I think it would be simpler if we just treaded through ITaskEnvironmentDriver here. It is the base abstraction the new APIs are using. If that is causing challenges beacuse of the multi-use of this code we could create a local one that has the same surface area.

@baronfel

@OvesN

OvesN commented Jul 30, 2026

Copy link
Copy Markdown
Author

What .NET SDK version has the appropriate MSBuild DLLs in it?

@jaredpar 10.0.400

@OvesN

OvesN commented Aug 7, 2026

Copy link
Copy Markdown
Author

Csc will be migrated in this PR -> #84701

@OvesN OvesN closed this Aug 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Area-Compilers Community The pull request was submitted by a contributor who is not a Microsoft employee. VSCode

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Multithreaded] Migrate Csc in Roslyn

6 participants