How to test diagnostic suppressor that suppresses rule from NuGet package?

[salkriaf34]

I've written a diagnostic suppressor that suppresses SA1615 from the StyleCop.Analyzers NuGet package.

I've also written a unit test for the suppressor, but I can't get the diagnostic to trigger in it. I get Mismatch between number of diagnostics returned, expected "1" actual "0".

If I copy the code to a new project with the NuGet package installed, the diagnostic is reported.

Suppressor code

namespace MyCustomDiagnosticSuppressions
{
	using System.Collections.Immutable;
	using Microsoft.CodeAnalysis;
	using Microsoft.CodeAnalysis.Diagnostics;

	[DiagnosticAnalyzer(LanguageNames.CSharp)]
	public sealed class SA1615Suppressor
		: DiagnosticSuppressor
	{
		public const string DiagnosticId = "SA1615Suppressor";

		internal static readonly SuppressionDescriptor SuppressionDescriptor = new(
			id: DiagnosticId,
			suppressedDiagnosticId: "SA1615",
			justification: "Custom justification goes here.");

		public override ImmutableArray<SuppressionDescriptor> SupportedSuppressions { get; } = [SuppressionDescriptor];

		public override void ReportSuppressions(SuppressionAnalysisContext context)
		{
			foreach (Diagnostic diagnostic in context.ReportedDiagnostics)
			{
				bool shouldSuppress = true; // Custom logic goes here.

				if (shouldSuppress)
					context.ReportSuppression(Suppression.Create(SuppressionDescriptor, diagnostic));
			}
		}
	}
}

Unit test code

namespace MyCustomDiagnosticSuppressions.Test
{
	using System.Threading.Tasks;
	using Microsoft.CodeAnalysis.CSharp.Testing;
	using Microsoft.CodeAnalysis.Testing;
	using Microsoft.VisualStudio.TestTools.UnitTesting;

	[TestClass]
	public sealed class SA1615SuppressorTests
	{
		[TestMethod]
		public async Task Should_Be_Suppressed()
		{
			string testSourceCode =
				$$"""
				namespace TestNamespace
				{
				    using System;
				    using System.Threading.Tasks;

				    /// <summary>
				    /// Some class.
				    /// </summary>
				    public class TestClass
				    {
				        /// <summary>
				        /// Some method.
				        /// </summary>
				        public async Task TestMethod() => throw new NotSupportedException("Test");
				    }
				}
				""";

			await new CSharpAnalyzerTest<SA1615Suppressor, DefaultVerifier>
			{
				TestState =
				{
					ReferenceAssemblies = ReferenceAssemblies.Net.Net90.AddPackages([new PackageIdentity("StyleCop.Analyzers", "1.2.0-beta.556")]),
					Sources = { testSourceCode },
					ExpectedDiagnostics = { DiagnosticResult.CompilerWarning("SA1615").WithIsSuppressed(true) },
				},
			}.RunAsync().ConfigureAwait(false);
		}
	}
}

Unit test output

Test method MyCustomDiagnosticSuppressions.Test.SA1615SuppressorTests.Should_Be_Suppressed threw exception: 
System.InvalidOperationException: Mismatch between number of diagnostics returned, expected "1" actual "0"

Diagnostics:
    NONE.


  Stack Trace: 
DefaultVerifier.Equal[T](T expected, T actual, String message)
AnalyzerTest`1.VerifyDiagnosticResults(IEnumerable`1 actualResults, ImmutableArray`1 analyzers, DiagnosticResult[] expectedResults, IVerifier verifier)
AnalyzerTest`1.VerifyDiagnosticsAsync(EvaluatedProjectState primaryProject, ImmutableArray`1 additionalProjects, DiagnosticResult[] expected, IVerifier verifier, CancellationToken cancellationToken)
AnalyzerTest`1.RunImplAsync(CancellationToken cancellationToken)
AnalyzerTest`1.RunAsync(CancellationToken cancellationToken)
SA1615SuppressorTests.Should_Be_Suppressed() line 34

Screenshot showing SA1615 for the test code

[bjornhellander]

I suspect that your assignment to ReferenceAssemblies is not enough to make the compiler treat it as an analyzer package. Maybe someone else can confirm this, like @sharwell?

I have found this: https://github.com/tom-englert/Lazy.Fody/tree/master/Lazy.Fody.Analyzer.Tests
Unfortunately, the analyzers in the StyleCop.Analyzers (or actually StyleCop.Analyzers.Unstable) are internal, but I assume that you could modify that code to load the analyzer using reflection instead.

[bjornhellander]

I made a little example of how I got it to work: https://github.com/bjornhellander/RoslynSuppressorTest1
The important parts are

• https://github.com/bjornhellander/RoslynSuppressorTest1/blob/master/RoslynSuppressorTest1/RoslynSuppressorTest1.Test/RoslynSuppressorTest1.Test.csproj#L14-L22 (to get access to the NuGet package code)
• https://github.com/bjornhellander/RoslynSuppressorTest1/blob/master/RoslynSuppressorTest1/RoslynSuppressorTest1.Test/Verifiers/CSharpSuppressorVerifier%601%2BTest.cs (to test analyzers which are internal)
• https://github.com/bjornhellander/RoslynSuppressorTest1/blob/master/RoslynSuppressorTest1/RoslynSuppressorTest1.Test/Verifiers/CSharpSuppressorVerifier%602%2BTest.cs (if you would also like to test analyzers which are public, in an easier way)

I see that the analyzers are actually being run as part of the built as well, which might not be desirable. I have not yet figured out a way to just reference them as a library without having them run.