Skip to content

Experiment with trusted cache population and MT consumption #10834

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
Jan Provazník (JanProvaznik) wants to merge 6 commits into
base: main
Choose a base branch
from
+105 −86
Draft

Experiment with trusted cache population and MT consumption #10834

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
7bed29d
Experiment with multithreaded cache consumption
JanProvaznik Aug 28, 2026
f498f61
Clarify read-only cache miss behavior
JanProvaznik Aug 28, 2026
205913a
Always fall back from MT cache failures
JanProvaznik Aug 28, 2026
0e69d76
Align cache fallback documentation
JanProvaznik Aug 28, 2026
1ccd726
Use patched MSBuild for cache experiment
JanProvaznik Aug 28, 2026
ece35e7
Keep patched MSBuild scoped to cache steps
JanProvaznik Aug 28, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
113 changes: 31 additions & 82 deletions azure-pipelines.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,9 @@ variables:
value: true
- name: VSO_DEDUP_REDIRECT_TIMEOUT_IN_SEC
value: 5
# Pinned prototype containing dotnet/msbuild#14824, #14825, and #14826.
- name: _PatchedMSBuildCommit
value: 16e184d749654ec43bf92c67ae4a9bcca2225401
# Microsoft.Testing.Platform report/dump arguments for the Windows Debug unit-test command that uses
# --test-modules. That mode bypasses test/Directory.Build.targets, so the command must restore these options
# explicitly. The full Release runs evaluate the test projects and receive the same options from that targets file.
Expand Down Expand Up @@ -146,6 +149,8 @@ stages:

- template: /eng/pipelines/steps/install-windows-prereqs.yml

- template: /eng/pipelines/steps/build-patched-msbuild.yml

# Seed from every trusted main merge. PR merge refs are read-only consumers, so the cache must be updated as
# soon as main changes to avoid forcing the entire 204-node graph to miss.
- pwsh: |
Expand All @@ -168,6 +173,7 @@ stages:
"-NoProfile",
"-File", "./eng/common/msbuild.ps1",
"-prepareMachine",
"-msbuildMultiThreaded:`$false",
"-warnAsError:`$false",
"./TestFx.slnx",
"/restore",
Expand Down Expand Up @@ -208,7 +214,10 @@ stages:
Write-Host "##vso[task.logissue type=warning]Seeding MSBuildCache failed with exit code $exitCode; continuing with the regular Arcade build."
exit 0
displayName: Seed $(_BuildConfig) project cache
condition: eq(variables['PatchedMSBuildAvailable'], 'true')
env:
_BuildToolPath: $(PatchedMSBuildPath)
MSBUILD_EXE_PATH: $(PatchedMSBuildPath)
SYSTEM_ACCESSTOKEN: $(System.AccessToken)
# MSBuildCache reads NUGET_PACKAGES to build its path normalizer, and every restored package file is a
# fingerprinted input. The seeding and the consuming builds must therefore agree on this folder or every
Expand Down Expand Up @@ -451,23 +460,24 @@ stages:

- template: /eng/pipelines/steps/install-windows-prereqs.yml

- template: /eng/pipelines/steps/build-patched-msbuild.yml
parameters:
condition: and(succeeded(), or(and(eq(variables['Build.Reason'], 'PullRequest'), ne(variables['System.PullRequest.IsFork'], 'True')), and(ne(variables['Build.Reason'], 'PullRequest'), eq(variables['Build.SourceBranch'], 'refs/heads/main'))))

# MSBuildCache requires a direct static graph build; Arcade's outer Build.proj discovers its projects too late
# for the project-cache plugin. Keep the regular Arcade build below as the correctness fallback during rollout.
# for the project-cache plugin. This experimental consumer uses MSBuild's in-process multithreaded mode but
# deliberately does not enable Detours file-access reporting: access attribution assumes one active project
# per process and is only safe on the process-based trusted seed above. Keep the regular Arcade build below as
# the correctness fallback while the required MSBuild fixes from dotnet/msbuild#14824 and #14826 are unreleased.
#
# This step reports one of three outcomes, and the fallback below keys off it:
# This step reports one of two outcomes, and the fallback below keys off it:
# succeeded - the cache supplied the outputs, the Arcade 'Build' step is skipped.
# succeeded with warning - the cache build broke for a reason the fallback can plausibly fix, so the
# Arcade 'Build' step runs. The step deliberately stays green: a step marked
# 'SucceededWithIssues' makes the job PartiallySucceeded, which Azure Repos
# build-validation policies treat as a failure and would block the merge on a
# run that ultimately builds fine (same reason as _MacOSNonBlockingTrailer).
# failed - the sources genuinely do not build, see the classification below.
# succeeded with warning - the experimental MT cache build failed, so the authoritative process-based
# Arcade 'Build' step runs. MT changes task routing, so even a source-shaped error
# can be specific to the experiment and must not suppress the fallback.
- pwsh: |
# This step no longer sets continueOnError, so its result is meaningful and an *unexpected*
# terminating error (Tee-Object, log reading, the classification below) must not fail the job:
# that is an infrastructure failure, which belongs on the fallback path exactly like a crash of
# the build itself. `exit` is flow control rather than an error, so the deliberate `exit 1` for a
# classified source break further down still fails the step as intended.
# This step no longer sets continueOnError, so an unexpected terminating error must not fail the job:
# that is an infrastructure failure, which belongs on the fallback path exactly like a nonzero build.
trap {
Write-Host "##vso[task.setvariable variable=MSBuildCacheBuildSucceeded]false"
Write-Host "##vso[task.logissue type=warning]The MSBuildCache step itself failed unexpectedly ($($_.Exception.Message)); continuing with the regular Arcade build."
Expand All @@ -485,12 +495,12 @@ stages:
"-NoProfile",
"-File", "./eng/common/msbuild.ps1",
"-prepareMachine",
"-msbuildMultiThreaded:`$true",
Comment thread
JanProvaznik marked this conversation as resolved.
"-warnAsError:`$false",
"./TestFx.slnx",
"/restore",
"/graph",
"/m",
"/reportfileaccesses",
"/nr:false",
"/t:Build",
"/v:minimal",
Expand All @@ -507,16 +517,10 @@ stages:
"/p:MSBuildCacheLogDirectory=$(Agent.TempDirectory)\MSBuildCache"
)

# Tee MSBuild's console output (errors included) so the failure can be classified below without
# re-running anything. stderr is deliberately left unredirected, exactly as before, so PowerShell's
# native-command error handling in this step is unchanged.
$logPath = Join-Path "$(Agent.TempDirectory)" "MSBuildCacheGraphBuild.log"

$previousPSNativeCommandUseErrorActionPreference = $PSNativeCommandUseErrorActionPreference
try {
# Unlike Start-Process -Wait, direct native invocation does not wait for detached telemetry descendants.
$PSNativeCommandUseErrorActionPreference = $false
& $pwsh @arguments | Tee-Object -FilePath $logPath
& $pwsh @arguments
$exitCode = $LASTEXITCODE
}
finally {
Expand All @@ -530,68 +534,13 @@ stages:
}

Write-Host "##vso[task.setvariable variable=MSBuildCacheBuildSucceeded]false"

# Decide whether the Arcade fallback can still reach a different result than this build did.
#
# A cache hit only materializes outputs that were stored earlier; it never runs the compiler. Every
# "error XXnnnn:" below was therefore emitted by a project that actually executed - exactly as the
# fallback would execute it - so running the whole solution through Arcade again only reproduces the
# same errors several minutes later. Fail here instead, with the errors surfaced on this step.
#
# Three classes of failure stay on the fallback path because the non-cached build can legitimately
# behave differently: anything MSBuildCache itself reports (plugin, cache or file-access problems),
# restore failures (NU*, which are often transient), and the engine/graph failures classified below -
# whether or not they reached MSBuild's error summary.
$log = if (Test-Path $logPath) { @(Get-Content -LiteralPath $logPath) } else { @() }

$reportedErrorCount = -1
# Matches MSBuild's English summary. A localized agent leaves the count at -1, which falls through
# to the fallback below - the safe direction, and the same outcome as a build that never reached a
# summary at all.
$errorSummary = @($log | Select-String -Pattern '(\d+)\s+Error\(s\)') | Select-Object -Last 1
if ($errorSummary) {
$reportedErrorCount = [int]$errorSummary.Matches[0].Groups[1].Value
}

$errorLines = @($log | Where-Object { $_ -match '(?i):\s*error(\s+[a-z]+\d+)?\s*:\s' } | Select-Object -Unique)
$fallbackMayHelp = '(?i)MSBuildCache|ProjectCache|project cache|cache plugin|CacheClient|file access|\bNU\d{4}\b'
$mayHelpLines = @($errorLines | Where-Object { $_ -match $fallbackMayHelp })

# Some failures do reach the error summary but still are not source breaks the fallback would
# reproduce, so they must not fail fast. These are matched against the whole log rather than against
# $errorLines, because their diagnostic detail usually continues on following lines that are not
# themselves formatted as errors:
# - engine crashes and OOM (MSB4166 "exited prematurely", MSB0001/MSB1025 internal errors,
# MSB4017 logger failure). The node died before the projects it owned could report their real
# result, so the Arcade invocation can legitimately succeed where this one did not.
# - MSB4260, a project reference that cannot be resolved with a static graph. That is a
# limitation of the /graph switch this step passes; the Arcade fallback does not build the
# graph statically, so it can resolve the same reference and succeed.
# - MSBuildCache's own duplicate-output diagnostic, worded "Node ... produced output ... which was
# already produced by another node ...". It carries no plugin or cache token of its own, so
# without this marker it reads as an ordinary build error and would defeat the intent that
# everything the plugin reports goes to the fallback.
# - transient file locks (MSB3021/MSB3027, "used by another process"). Copy contention is timing
# dependent, and copying is heavier here because the cache uses copy rather than hardlink
# semantics, so a differently scheduled build can legitimately succeed.
$fallbackCanDiffer = '(?i)\bMSB4166\b|\bMSB0001\b|\bMSB1025\b|\bMSB4017\b|\bMSB4260\b|\bMSB3021\b|\bMSB3027\b|exited prematurely|OutOfMemoryException|already produced by another node|being used by another process'
$fallbackCanDifferLines = @($log | Where-Object { $_ -match $fallbackCanDiffer })

if ($reportedErrorCount -gt 0 -and $errorLines.Count -gt 0 -and $mayHelpLines.Count -eq 0 -and $fallbackCanDifferLines.Count -eq 0) {
foreach ($line in @($errorLines | Select-Object -First 20)) {
Write-Host "##vso[task.logissue type=error]$line"
}

Write-Host "The solution does not build. The Arcade 'Build' step would report the same $reportedErrorCount error(s), so the build is failed here instead of repeating it."
Write-Host "If you believe an error above is an artifact of a cached dependency rather than a source break, the cache diagnostics for this run are published under artifacts/log/$(_BuildConfig)/MSBuildCache."
exit 1
}

Write-Host "##vso[task.logissue type=warning]MSBuildCache build failed with exit code $exitCode without a build error the fallback would reproduce; continuing with the regular Arcade build."
Write-Host "##vso[task.logissue type=warning]The experimental multithreaded MSBuildCache build failed with exit code $exitCode; continuing with the authoritative process-based Arcade build."
Comment thread
JanProvaznik marked this conversation as resolved.
exit 0
displayName: Build solution graph with MSBuildCache
condition: and(succeeded(), or(and(eq(variables['Build.Reason'], 'PullRequest'), ne(variables['System.PullRequest.IsFork'], 'True')), and(ne(variables['Build.Reason'], 'PullRequest'), eq(variables['Build.SourceBranch'], 'refs/heads/main'))))
condition: and(succeeded(), eq(variables['PatchedMSBuildAvailable'], 'true'), or(and(eq(variables['Build.Reason'], 'PullRequest'), ne(variables['System.PullRequest.IsFork'], 'True')), and(ne(variables['Build.Reason'], 'PullRequest'), eq(variables['Build.SourceBranch'], 'refs/heads/main'))))
env:
_BuildToolPath: $(PatchedMSBuildPath)
MSBUILD_EXE_PATH: $(PatchedMSBuildPath)
SYSTEM_ACCESSTOKEN: $(System.AccessToken)
# Must match the seeding stage exactly. MSBuildCache normalizes observed input paths against
# NUGET_PACKAGES, and every restored package file is a fingerprinted input, so a different package
Expand Down Expand Up @@ -735,8 +684,8 @@ stages:
New-ItemProperty -Path "HKLM:\SOFTWARE\Microsoft\Windows\Windows Error Reporting\LocalDumps" -Name "DumpFolder" -Value "$(Build.SourcesDirectory)\artifacts\CrashDumps" -PropertyType ExpandString -Force
New-ItemProperty -Path "HKLM:\SOFTWARE\Microsoft\Windows\Windows Error Reporting\LocalDumps" -Name "DumpCount" -Value 10 -PropertyType DWord -Force

# Fallback build. Skipped when the MSBuildCache steps above already produced, signed and packed the
# outputs; also skipped when they failed with build errors, because this step would only reproduce them.
# Authoritative fallback build. It is skipped only when the cache graph and any required sign/pack work
# succeed; every cache-path failure runs this process-based build.
- script: eng\common\CIBuild.cmd
-configuration $(_BuildConfig)
-prepareMachine
Expand Down
Loading