[tests] Random BundleStructureWithRemoteMac failure from timestamp churn in shared Windows SDK

[rolfbjarne]

Summary

Xamarin.Tests.WindowsTest.BundleStructureWithRemoteMac(iOS, "ios-arm64", All, "Debug") is randomly failing in the Windows remote test leg.

The failure shows up as either:

• Rebuild 1: shared-dotnet.plist: timestamp changed ...
• or _CompileAppManifest unexpectedly executing on what should be a no-op rebuild after only touching AppDelegate.cs.

Example recent failure:

• DevDiv build: https://devdiv.visualstudio.com/DevDiv/_build/results?buildId=13839857&view=results

Diagnostic work

I investigated this using:

• the additional logging added in 0b8b50e94f5226156d582162024901dabd6c6b11
• the failing HTML report / vsdrop artifacts from build 13839857
• the relevant binlogs (windows-remote-dotnet-tests.binlog, the BundleStructure iOS build binlogs, and install-workloads.binlog)

The chain from the failing artifacts is:

1. obj/.../unpack/bindings-framework-test/PartialAppManifest/shared-dotnet.plist gets a newer timestamp.
2. That makes _CompileAppManifest rerun, even though the test only touched AppDelegate.cs.
3. The plist is rewritten because _BeforeUnpackLibraryResources deletes the resource stamp files, so _UnpackLibraryResources runs again.
4. The stamp files are deleted because bindings-framework-test.dll is newer than its stamp.
5. bindings-framework-test.dll is rebuilt because _GenerateBindings/BGen reruns and rewrites FrameworkTest.g.cs.
6. BGen reruns because one of its references (MonoTouch.Dialog.dll) rebuilt.
7. MonoTouch.Dialog.dll rebuilt because MSBuild considered tests/dotnet/Windows/bin/dotnet/sdk/.../Microsoft.CodeAnalysis.CSharp.NetAnalyzers.dll newer than the previous output.

Notably, this analyzer-driven invalidation already happens in the initial build, before the test performs the rebuild step, so it doesn't look like Rebuild 1 itself is what changed that analyzer timestamp.

Conclusions

This looks like an incremental-build / timestamp churn issue in the shared repo-local Windows SDK tree (tests/dotnet/Windows/bin/dotnet), not a semantic change in the BundleStructure project itself.

Two things made the flake visible:

• unchanged unpacked resources were being rewritten, which bumped the timestamp of shared-dotnet.plist
• unchanged bgen outputs were being rewritten, which made the binding project look changed and caused the stamp invalidation chain to continue

The remaining open question is why the analyzer DLL in the shared SDK tree has a later mtime than the referenced-project outputs at test start.

A likely contributor is that the job mutates that shared SDK tree as part of workload installation. The install-workloads.binlog shows:

D:\AzDO\_work\17\s\macios\tests\dotnet\Windows\bin\dotnet\dotnet workload install ios tvos maccatalyst macos ...

So all Windows unit tests are building against a mutable, repo-local SDK installation, and that may be enough to leave SDK inputs newer than previously built referenced projects.

Suggested next steps

1. Confirm what step is refreshing or rewriting files under tests/dotnet/Windows/bin/dotnet/sdk/... and whether that can happen between test runs.
2. Consider isolating workload-installing steps/tests from the rest of the Windows unit tests, or stop sharing the same mutable SDK tree.
3. Keep the extra logging in WindowsTest so the next failure shows the same file/target chain if this still happens.

[dalexsoto]

Confirmed two new occurrences of the same BundleStructureWithRemoteMac(iOS, "ios-arm64", All, "Debug") timestamp-churn signature across two monitored PRs (2 distinct PRs / 2 confirmed occurrences):

• [release/10.0.1xx] Update dependencies from dotnet/dotnet #26443 @ 76abb5bcaa11fd4dedb414f859229b352ef08691 — build 14957450, attempt 1, Windows Integration Tests / Dotnet tests: shared-dotnet.plist changed from 2026-08-13T03:05:50.4306422Z to 2026-08-13T03:07:44.3704989Z.
• [main] Update dependencies from dotnet/dotnet #26444 @ a117ce0ef8f74c17bdd0ebdd2989e1bf1a49b713 — build 14957461, attempt 1, Windows Integration Tests / Dotnet tests: shared-dotnet.plist changed from 2026-08-13T03:09:28.7312747Z to 2026-08-13T03:11:54.0860447Z.

Classification remains a genuine flake caused by incremental-build timestamp churn in the shared repo-local Windows SDK tree, matching this issue. Both PRs are mergeable and within retry budgets. Attempts to rerun the complete Windows Integration Tests stages were rejected because the aggregate pipelines are still running; no retry attempts were created or counted. The stage retries are deferred until each build reaches a completed state.

The issue has no Additional info to collect checklist.