Stage 3: Forward BuildProjectFile* callbacks from OOP TaskHost to worker node

[JanProvaznik]

Summary

Stage 3 of IBuildEngine callback support for out-of-process TaskHost. Implements forwarding of BuildProjectFile* from OOP TaskHost to the owning worker node, enabling tasks ejected to TaskHost (via -mt or explicit TaskHostFactory) to build other projects.

Real-world impact: WPF XAML compilation uses GenerateTemporaryTargetAssembly which calls BuildProjectFile from TaskHost. Without this change, building any WPF project with -mt fails with MSB5022.

Design

Packet pair

• TaskHostBuildRequest (0x20) / TaskHostBuildResponse (0x21): All 4 BuildProjectFile* overloads normalize to the 6-param BuildProjectFilesInParallel canonical form.

Callback flow (no Yield/Reacquire)

Unlike the original design, we do not use Yield/Reacquire for BuildProjectFile callbacks. Instead:

1. Task thread calls BuildProjectFile → gates on CallbacksSupported
2. BlockForCallback() transitions the task from active to blocked (_activeTaskCount--, _blockedTaskCount++)
3. SendCallbackRequestAndWaitForResponse<T> serializes the request and blocks on TaskCompletionSource
4. Worker's TaskHostTask.HandleBuildRequest calls IBuildEngine3.BuildProjectFilesInParallel — exceptions propagate to TaskBuilder naturally (matching in-proc behavior), finally sends the response
5. Response arrives → TCS completes → ResumeAfterCallback() restores counters and environment

A blocked task does not prevent the node from accepting new work — HandleTaskHostConfiguration checks _activeTaskCount == 0 (not blocked count), so nested task dispatch works.

Yield/Reacquire remain no-ops in the OOP TaskHost.

Per-task isolation

TaskExecutionContext stores per-task state via ConcurrentDictionary<int, TaskExecutionContext> + AsyncLocal: configuration, task wrapper, saved environment, debug flags, warning settings, pending callback requests. GetCurrentConfiguration() and EffectiveWarningsAs* read from per-task context on task threads, falling back to _currentConfiguration on the main communication thread.

Handler routing (process reuse)

NodeProviderOutOfProcTaskHost routes packets by TaskHostNodeKey, supporting nested task dispatch when an outer task is blocked on a callback and a new task arrives on the same TaskHost process.

Version gating

Gated behind CallbacksSupported (_parentPacketVersion >= 4 OR MSBUILDENABLETASKHOSTCALLBACKS=1). Each BuildProjectFile overload explicitly checks. Follow-up PR bumps PacketVersion to 4 and removes the env var.

Tests

• Serialization tests — packet round-trip for TaskHostBuildRequest/Response with null/empty/populated properties
• Integration tests — BuildProjectFile via TaskHostFactory, global property forwarding, target output retrieval, callbacks-not-supported error path (MSB5022)
• E2E tests — cross-runtime BuildProjectFile callback, TaskHost process reuse (same PID verification)
• MT mode test — BuildProjectFile during -mt build

Context

• Spec: TaskHost IBuildEngine Callback Support spec #12868
• Stage 1: [IBuildEngine callbacks] Stage 1: Packet infrastructure + IsRunningMultipleNodes #13149 (merged) — IsRunningMultipleNodes
• Stage 2: [IBuildEngine callbacks] Stage 2: RequestCores/ReleaseCores #13306 (merged) — RequestCores/ReleaseCores
• WPF repro: TaskHost Does Not Support Critical IBuildEngine Callbacks #12863

[JanProvaznik]

Post-review notes (automated multi-agent review of new commits)

5 new commits pushed addressing review meeting feedback. Multi-agent review (5 agents: expert-reviewer, 2× Opus 4.6, 2× GPT-5.4) ran against the delta. Findings addressed inline; remaining items for human attention:

Items addressed in these commits

• ✅ ETW refactor: diagnostic strings → ETW events (6 resource strings + 13 xlf translations removed)
• ✅ Shutdown ordering: FailAllPendingCallbackRequests before _taskRunnerThread.Join() (deadlock fix)
• ✅ ObjectDisposedException guard on _taskCompleteEvent.Set()
• ✅ ETW Stop event moved to finally block (all exit paths covered)
• ✅ Non-atomic field reads in ETW params fixed (capture Interlocked return values)
• ✅ TaskHostNestedTaskStart renamed to TaskHostNestedTaskDispatched (point-in-time, not unpaired Start)
• ✅ BuildInParallel=""true"" added to orchestrator tests (was defaulting to sequential execution)
• ✅ Coexistence test doc comment corrected (no longer overclaims handler stack isolation proof)

Items for human reviewer attention

1. _taskRunnerThread?.Join() has no timeout (pre-existing, not introduced by this PR). If a user task hangs (not blocked on TCS), shutdown hangs indefinitely. Nested task threads get Join(TimeSpan.FromSeconds(5)) but the main runner thread does not. Consider adding a timeout in a follow-up.

2. Coexistence test is a smoke test, not a proof of handler-stack isolation. With MaxNodeCount=1, even with BuildInParallel=true, the scheduler may serialize execution. A stronger test would assert PID equality across MT-ejected and explicit TaskHostFactory tasks to prove they shared a process. Left as-is since it still provides value as a "both paths work in one build" test.

3. No cancellation-during-callback test yet. Meeting agreed on OS mutex-based coordination for this (Rainer's suggestion). Deferred to follow-up.

[Copilot]

Pull request overview

Stage 3 of out-of-process TaskHost IBuildEngine callback support: forwards BuildProjectFile* and Yield/Reacquire from the OOP TaskHost back to the owning worker node so ejected tasks (notably WPF XAML compilation) can build projects and cooperate with the scheduler.

Changes:

• Add new callback packet types for BuildProjectFile* and Yield/Reacquire, including serialization round-trip tests.
• Implement callback forwarding in OutOfProcTaskHostNode, worker-side handling in TaskHostTask, and nested-task process reuse routing via a handler stack in NodeProviderOutOfProcTaskHost.
• Add integration/E2E tests and update TaskHost threading documentation plus resource strings.

Reviewed changes

Copilot reviewed 40 out of 40 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
src/Shared/TaskHostBuildRequest.cs	New packet carrying canonical BuildProjectFilesInParallel request data.
src/Shared/TaskHostBuildResponse.cs	New response packet with success + optional target outputs (TaskParameter-serialized).
src/Shared/TaskHostYieldRequest.cs	New packet for Yield/Reacquire operations.
src/Shared/TaskHostYieldResponse.cs	New response packet to unblock TaskHost on Reacquire.
src/Shared/Resources/Strings.shared.resx	Adds new resource strings for callback/yield diagnostics.
src/Shared/Resources/xlf/Strings.shared.*.xlf	Adds corresponding localization entries for the new resource strings.
src/MSBuild/TaskExecutionContext.cs	Introduces per-task execution context for concurrent/nested TaskHost execution.
src/MSBuild/OutOfProcTaskHostNode.cs	Implements forwarding for BuildProjectFile* and Yield/Reacquire; adds per-task context + counters + completion queue.
src/MSBuild/MSBuild.csproj	Includes new shared packets + TaskExecutionContext in MSBuild build.
src/Build/Microsoft.Build.csproj	Includes new shared packets in Microsoft.Build build.
src/Build/Instance/TaskFactories/TaskHostTask.cs	Worker-side handlers for build/yield requests and response emission.
src/Build/BackEnd/Components/Communications/NodeProviderOutOfProcTaskHost.cs	Adds handler stack to support nested TaskHostTask routing and reuse.
src/Build.UnitTests/BackEnd/TaskHostCallbackPacket_Tests.cs	Adds serialization tests for the new request/response packet types.
src/Build.UnitTests/BackEnd/TaskHostCallback_Tests.cs	Adds integration tests validating BuildProjectFile forwarding and fallback behavior.
src/Build.UnitTests/NetTaskHost_E2E_Tests.cs	Adds E2E tests for cross-runtime BuildProjectFile and same-PID reuse for nested builds.
src/Build.UnitTests/BackEnd/BuildProjectFileTask.cs	New test task that triggers BuildProjectFile callback and captures outputs.
src/Build.UnitTests/BackEnd/BuildProjectFileAndReportPidTask.cs	New test task for PID reporting + nested BuildProjectFile call (reuse verification).
src/Build.UnitTests/TestAssets/ExampleNetTask/TestNetTaskBuildCallback/*	New E2E asset project for callback validation.
src/Build.UnitTests/TestAssets/ExampleNetTask/TestNetTaskHostReuse/*	New E2E asset project for nested-build process reuse validation.
src/Build.UnitTests/TestAssets/ExampleNetTask/ExampleTask/ExampleTask.csproj	Links new test tasks into ExampleTask assembly.
documentation/specs/multithreading/taskhost-threading.md	Documents Stage 3 callback flow, yield-for-callback pattern, and handler stack.

[JanProvaznik]

I investigated supporting explicit IBuildEngine3.Yield() / Reacquire() in the OOP TaskHost and decided not to implement it in this change.

The short version is that this is not a small missing feature in the current design—it conflicts with several core assumptions of the OOP task-host architecture due to having one pipe for communication:

• Packet routing breaks down. The parent side routes incoming packets to the top handler on a per-node stack. Once a task explicitly yields and another task is scheduled on the same OOP node, packets from the yielded task can be delivered to the wrong handler.
• Logs can be silently misattributed. If a yielded task’s log packet is routed to the wrong TaskHostTask, the event gets stamped with the wrong BuildEventContext, which would corrupt console/binlog diagnostics.
• We would need per-packet handler identity

By contrast, the existing internal callback-based blocking path works because it blocks the task thread while the callback is in flight. That preserves the single-active-handler assumption and avoids the routing/logging/state-machine issues above.

So the conclusion here is: supporting explicit yield in OOP TaskHost would require broader architectural and wire-protocol work than is appropriate for this PR, and a partial implementation would be brittle and potentially incorrect.

Not implementing Yield does not regress correctness, however negative perf impact in -mt mode is possible for custom tasks that use this feature and are not migrated, and we may revisit this if there is evidence of many customers using Yield while being blocked from migration to -mt aware tasks.