Experiment with trusted cache population and MT consumption

[Jan Provazník (JanProvaznik)]

Experiment status

This is a draft experiment, not a merge-ready rollout. It explores the operational split discussed in #10726: trusted main builds populate MSBuildCache with process nodes, while PR builds consume it read-only with MSBuild's experimental multithreaded mode.

Design

Build	Command shape	Remote mode
Trusted main population	eng\common\msbuild.ps1 -msbuildMultiThreaded:$false ... TestFx.slnx /restore /graph /m /reportfileaccesses /t:Build ...	/p:MSBuildCacheRemoteCacheIsReadOnly=false
PR consumption	eng\common\msbuild.ps1 -msbuildMultiThreaded:$true ... TestFx.slnx /restore /graph /m /t:Build ...	/p:MSBuildCacheRemoteCacheIsReadOnly=true

Population remains process-based because Detours attribution assumes at most one active project per node/process. PR consumption deliberately omits /reportfileaccesses; hits materialize outputs, while misses execute under -mt but cannot publish remotely.

Every nonzero experimental cache-build exit runs the authoritative process-based Arcade fallback. MT changes task routing, so even a source-shaped error can be experiment-specific and must not suppress that fallback.

Patched MSBuild

The repository's pinned SDK does not contain the required engine fixes. Each eligible Windows cache job therefore:

1. Fetches immutable dotnet/msbuild@16e184d749654ec43bf92c67ae4a9bcca2225401 (draft PR #14871).
2. Verifies the checked-out SHA.
3. Builds the Release full-framework bootstrap without System.AccessToken.
4. Verifies artifacts\bin\bootstrap\net472\MSBuild\Current\Bin\amd64\MSBuild.exe (18.12.0.42801 in local validation).
5. Injects it only into the cache graph step through _BuildToolPath and MSBUILD_EXE_PATH.

The job-wide PATH is not modified, so fallback and later validation retain the normal Visual Studio toolchain. If the source build or version check fails, the cache step is skipped and Arcade runs normally. Using the same prototype for seed and consumer avoids engine differences obscuring cache measurements.

Security model

• Only batched/individual CI runs for refs/heads/main can populate remotely.
• PR/manual/nightly consumers remain read-only.
• Fork PRs receive neither the custom-engine cache path nor System.AccessToken; they run Arcade fallback.
• The custom-engine source build has no cache token. The verified engine receives the token only when executing the cache graph step.
• PR code cannot publish into the trusted cache universe.
• CacheClient.log and the OAuth-bearing cache environment remain excluded from published artifacts.

Cache identity and expected behavior

The experiment reuses the existing Azure Pipeline Cache backend and plugin knobs:

• Windows-only jobs on the pinned VS image.
• Separate testfx-v1-Debug and testfx-v1-Release universes.
• Project/TFM, evaluated properties, source inputs, toolchain inputs, package versions, and outputs participate in plugin fingerprints.
• NUGET_PACKAGES=$(Build.SourcesDirectory)\.packages\ stays aligned between seed and consumer.
• MSBuildCacheIdenticalDuplicateOutputPatterns=\** retains writable copy materialization.

Expected behavior:

1. After a matching custom-engine main seed, unchanged nodes hit and materialize under -mt.
2. Changed nodes miss and execute under MT, but a PR cannot publish them remotely.
3. Any prototype/plugin/engine/sign-pack failure preserves diagnostics, cleans partial outputs, and runs Arcade fallback.
4. Existing cache entries may miss until main seeds with the same prototype and configuration.

Required MSBuild fixes

• dotnet/msbuild#14824: preserve FileAccessData across TaskHost deserialization.
• dotnet/msbuild#14826: propagate and gate TaskHost reporting when /reportfileaccesses is absent.
• dotnet/msbuild#14825: reject -mt plus /reportfileaccesses actionably. The underlying Detours attribution model remains incompatible with in-process MT, so population stays process-based.

TestFX-specific blockers and constraints

• Azure Pipeline Cache is the only configured persistent remote backend.
• Fork validation cannot access it by design.
• A fresh GitHub SSO authorization succeeded, but the authenticated account still has repository role READ; direct origin push returns HTTP 403. A TestFX maintainer must push current HEAD ece35e7f3 to a microsoft/testfx branch for credentialed non-fork validation.
• The cache plugin requires a clean Windows static-graph build, full-framework x64 MSBuild, and Git on PATH.
• Arcade's outer Build.proj discovers projects too late for cache interception; the graph pass must invoke TestFx.slnx directly, followed by Arcade sign/pack with NoBuild=true.
• Main must reseed after toolchain/shared-input changes. Azure Pipeline Cache entries are immutable.
• Debug/Release remain isolated to prevent immutable-selector races.
• Copy materialization adds I/O and relaxes identical duplicate-output checking, while different-content duplicate outputs remain errors.

Measurements needed

For Debug and Release, collect:

• graph node and hit/miss/error counts;
• local versus remote hits and bytes materialized;
• lookup, materialization, execution, and total wall-clock time;
• fallback frequency and reason;
• compiler/shared-compilation activity on hits versus misses;
• comparison with process-based consumption and Arcade fallback;
• confirmation that the PR never attempts a remote write.

Validation and live iterations

• Local patched MSBuild build: passed in 125 seconds; version 18.12.0.42801.
• Upstream MSBuild PR build 1572304: passed all Windows, Linux, macOS, MT, source-build, and coverage legs in 55m21s.
• TestFX fork stabilization 1572155: passed all legs before custom-engine wiring.
• TestFX custom-engine fork validation 1572405: passed all legs. The custom-engine and cache steps were correctly skipped because System.PullRequest.IsFork=True; authoritative fallback passed on Windows Release/Debug, app-model, Linux, and macOS.
• Earlier runs 1572106, 1572118, 1572121, and 1572370 were canceled by review-driven follow-up pushes.
• Local pipeline checks passed: eng\validate-affected-tests.ps1, eng\classify-build-change.ps1 -SelfTest, and git diff --check.

The remaining E2E step is a maintainer-owned, non-fork draft run. The fork gate must not be weakened or manually bypassed.

Rollback and exit criteria

Proceed only when a trusted run shows stable read-only hits, no TaskHost/file-access failure, no write attempt, correct outputs/tests, and measurable benefit. Roll back by removing the patched-engine step, consumer -msbuildMultiThreaded:$true, and restoring /reportfileaccesses on process-based consumption. Exit if hit rate is too low, overhead erases benefit, fallback is nontrivial, correctness differs, credentials cannot be safely scoped, or the upstream attribution model changes.

[Copilot]

Pull request overview

Draft experiment separating trusted cache population from multithreaded, read-only PR consumption.

Changes:

• Forces process-based MSBuild for trusted cache seeding.
• Enables -mt and disables file-access reporting for PR cache consumption.
• Documents the design and pending MSBuild prerequisites.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File	Description
azure-pipelines.yml	Configures distinct seed and consumer execution modes.
docs/dev-guide.md	Documents cache commands, constraints, and upstream blockers.

---

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.

[Jan Provazník (JanProvaznik)]

Maintainer help needed for the trusted end-to-end cache validation: please create a branch inside microsoft/testfx from current experiment HEAD ece35e7f3 (or push that commit stack) and open/queue a separate non-fork draft PR against main.

The TestFX branch builds immutable dotnet/msbuild@16e184d749654ec43bf92c67ae4a9bcca2225401 (draft upstream PR dotnet/msbuild#14871) without cache credentials, verifies its x64 .NET Framework bootstrap, and injects it only into the trusted seed and read-only MT consumer cache graph steps. The bootstrap build was validated locally as MSBuild 18.12.0.42801; the authoritative fallback remains on the normal Visual Studio toolchain.

A direct push to microsoft/testfx was attempted and rejected with HTTP 403 (Permission to microsoft/testfx.git denied to JanProvaznik). SSO authorization completed, but GitHub still reports this account's repository role as READ; this is no longer a token-scope or SAML problem. This fork PR correctly skips the custom/cache consumer because System.PullRequest.IsFork=True; that gate must not be weakened or bypassed. The internal run is needed to collect credentialed Azure Pipeline Cache lookup, hit/miss, materialization, fallback, and read-only-write-attempt evidence.

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.