Skip to content

Fix intermittent build failure from duplicate graph builds of TemplateEngine projects#54317

Merged
MichaelSimons merged 4 commits into
dotnet:mainfrom
MichaelSimons:fix/template-engine-duplicate-graph-builds
May 18, 2026
Merged

Fix intermittent build failure from duplicate graph builds of TemplateEngine projects#54317
MichaelSimons merged 4 commits into
dotnet:mainfrom
MichaelSimons:fix/template-engine-duplicate-graph-builds

Conversation

@MichaelSimons

Copy link
Copy Markdown
Member

Fix intermittent build failure caused by duplicate graph builds of TemplateEngine projects

Fixes #54020

Problem

The TestBuild: linux (x64) and TestBuild: linux (arm64) CI legs intermittently fail with MSB3894/MSB3026 errors:

error MSB3894: The process cannot access the file
'artifacts/obj/Microsoft.TemplateEngine.Orchestrator.RunnableProjects/Release/net10.0/
Microsoft.TemplateEngine.Orchestrator.RunnableProjects.dll' because it is being used by another process.

This has been a recurring issue tracked as a Known Build Error with BuildRetry: true as the only mitigation.

Root Cause Analysis

The SDK build uses /graph mode (set in eng/build.sh). In static graph builds, MSBuild creates a separate graph node for each unique combination of (project path, global properties). Two nodes with different global properties but the same output path will be scheduled concurrently, causing file lock races.

How the duplicate builds arise

Microsoft.TemplateEngine.Authoring.Tasks.csproj is a packable MSBuild task project that publishes itself during pack via:

<Target Name="AddBuildOutputToPackageNet" DependsOnTargets="Publish" ...>

When the Publish target runs, MSBuild sets PublishDir as a global property. This property then leaks through to the project's three ProjectReference items because they did not specify GlobalPropertiesToRemove="PublishDir":

<!-- Before (PublishDir leaks to all three) -->
<ProjectReference Include="...TemplateLocalizer.Core.csproj" PrivateAssets="all" />
<ProjectReference Include="...Microsoft.TemplateEngine.Edge.csproj" PrivateAssets="all" />
<ProjectReference Include="...Microsoft.TemplateEngine.Orchestrator.RunnableProjects.csproj" PrivateAssets="all" />

This creates two distinct graph nodes for each referenced project:

Graph path PublishDir set? Output path
Solution → RunnableProjects ❌ No artifacts/bin/.../net10.0/
Solution → Authoring.Tasks → (Publish) → RunnableProjects ✅ Yes artifacts/bin/.../net10.0/ (same!)

Because PublishDir doesn't affect the referenced project's output directory, both nodes write to the same path concurrently. When one MSBuild node's compiler holds a lock on the obj DLL while the other node's CopyFilesToOutputDirectory task tries to read it, the build fails.

Evidence from the build log

The build log for build 1421188 confirms the duplicate builds — the same project produces output multiple times for the same TFM:

19:42:42  RunnableProjects -> .../net11.0/...dll
19:43:54  RunnableProjects -> .../net472/...dll
19:43:54  RunnableProjects -> .../net11.0/...dll    ← duplicate!
19:44:12  ERROR MSB3894 on net10.0 copy
19:44:12  RunnableProjects -> .../net10.0/...dll    ← retry succeeded
19:44:13  RunnableProjects -> .../net10.0/...dll    ← duplicate!
19:48:15  ERROR MSB3894 on net10.0 copy             ← build fails

The same pattern affects Microsoft.TemplateEngine.Edge and other transitive dependencies of Authoring.Tasks.

Why MSBUILDALWAYSRETRY=true doesn't help

The retry mechanism fires (the log shows "Beginning retry 1 in 1000ms") and sometimes succeeds. But because the contention comes from a concurrent build of the same project — not a transient file system hiccup — the lock can persist through all retry attempts.

Fix

Added GlobalPropertiesToRemove="PublishDir" to the three ProjectReference items in Microsoft.TemplateEngine.Authoring.Tasks.csproj. This prevents PublishDir from propagating to referenced projects during Publish, ensuring each project gets a single graph node per TFM rather than duplicate nodes that race against each other.

This matches the pattern already established by src/Cli/dotnet/dotnet.csproj and other projects in the src/Cli/ directory, which consistently use GlobalPropertiesToRemove="PublishDir" on all their project references for exactly this reason.

…eEngine projects

Authoring.Tasks.csproj publishes itself during pack (DependsOnTargets=Publish),
which sets PublishDir as a global property. This leaked through to its three
ProjectReference items, creating duplicate graph nodes in /graph mode that built
the same projects to the same output path concurrently, causing MSB3894 file
lock errors.

Add GlobalPropertiesToRemove=PublishDir to the three references, matching the
pattern used by dotnet.csproj and other src/Cli/ projects.

Fixes dotnet#54020

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings May 14, 2026 21:00

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR addresses intermittent CI build failures in static graph builds caused by the PublishDir global property flowing into ProjectReference builds, creating duplicate graph nodes that concurrently write to the same output paths (leading to MSB3894/MSB3026 file-lock errors).

Changes:

  • Prevent PublishDir from propagating to TemplateEngine project references during Publish by adding GlobalPropertiesToRemove="PublishDir" to the relevant ProjectReference items.

@MichaelSimons MichaelSimons requested a review from baronfel May 14, 2026 21:02
@MichaelSimons MichaelSimons requested review from a team and NikolaMilosavljevic and removed request for a team May 18, 2026 16:15
@MichaelSimons

Copy link
Copy Markdown
Member Author

/ba-g - failures appear unrelated to changes.

@MichaelSimons MichaelSimons merged commit c309f47 into dotnet:main May 18, 2026
22 of 24 checks passed
@MichaelSimons MichaelSimons deleted the fix/template-engine-duplicate-graph-builds branch May 18, 2026 19:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Known Build Error] The process cannot access the file Microsoft.TemplateEngine.Orchestrator.RunnableProjects.dll

3 participants