.NET: Add Foundry.Hosting.IntegrationTests#5598
Merged
rogerbarreto merged 13 commits intomicrosoft:mainfrom May 6, 2026
Merged
.NET: Add Foundry.Hosting.IntegrationTests#5598rogerbarreto merged 13 commits intomicrosoft:mainfrom
rogerbarreto merged 13 commits intomicrosoft:mainfrom
Conversation
…tests Add a new integration test project for Foundry hosted agents alongside the existing Foundry.IntegrationTests project. The project provisions a real Foundry hosted agent per scenario via AgentAdministrationClient.CreateAgentVersionAsync, points it at a single test container image (built and pushed out of band by scripts/it-build-image.ps1 in a follow up commit), and exercises the agent through AIProjectClient.AsAIAgent. Six scenario fixtures are introduced, each pointing at the same image but selecting behavior via the IT_SCENARIO environment variable on the HostedAgentDefinition: - HappyPathHostedAgentFixture (round trip, multi turn, stored=false flag) - ToolCallingHostedAgentFixture (server side AIFunctions) - ToolCallingApprovalHostedAgentFixture (approval flow) - ToolboxHostedAgentFixture (Foundry toolbox) - McpToolboxHostedAgentFixture (MCP backed toolbox) - CustomStorageHostedAgentFixture (custom storage provider) 24 tests across 6 test classes are scaffolded. All are tagged Skip pending the test container build and the end to end smoke iteration in follow up commits. Once the container is in place the Skip annotations can be removed scenario by scenario. Adds an IT_HOSTED_AGENT_IMAGE constant to the shared TestSettings so every IT project agrees on the env var name the build script emits.
…nx, README Adds the rest of the integration test infrastructure on top of the previous scaffolding commit: * Foundry.Hosting.IntegrationTests.TestContainer csproj and Program.cs implementing the multi scenario container (one image, IT_SCENARIO env var dispatches between happy-path, tool-calling, tool-calling-approval, toolbox, mcp-toolbox, and custom-storage). The toolbox, mcp-toolbox, and custom-storage branches are placeholders pending API surface stabilization. * Dockerfile and dockerignore in the test container project, using the contributor pattern matching the investigation work (host side dotnet publish, container only does COPY out/). * scripts/it-build-image.ps1 with mandatory Registry parameter (no hardcoded ACR), content hashed tags so unchanged source results in a no op push, and emits IT_HOSTED_AGENT_IMAGE for shells and CI to consume. * slnx entry for both new projects. * README in the IT project covering env vars, image build, scenario table, and current placeholder status. Steps still pending: end to end smoke (step 5) and CI workflow integration (step 6) require a live Foundry deployment and ACR push, so they land in follow up commits.
Contributor
There was a problem hiding this comment.
Pull request overview
Note
Copilot was unable to run its full agentic suite in this review.
Adds a new .NET integration test suite that provisions real Azure AI Foundry hosted agents (via Microsoft.Agents.AI.Foundry.Hosting) plus a shared scenario-driven test container image and supporting build/push script.
Changes:
- Introduces
Foundry.Hosting.IntegrationTestswith scenario fixtures and (currently skipped) end-to-end tests. - Adds
Foundry.Hosting.IntegrationTests.TestContainerASP.NET host that switches behavior viaIT_SCENARIO. - Adds
scripts/it-build-image.ps1to publish/build/push a content-hashed container image and emitIT_HOSTED_AGENT_IMAGE.
Reviewed changes
Copilot reviewed 22 out of 22 changed files in this pull request and generated 10 comments.
Show a summary per file
| File | Description |
|---|---|
| scripts/it-build-image.ps1 | Builds and pushes the shared test container image and outputs IT_HOSTED_AGENT_IMAGE. |
| dotnet/tests/Foundry.Hosting.IntegrationTests/Foundry.Hosting.IntegrationTests.csproj | New multi-targeted integration test project referencing Foundry + Hosting libs. |
| dotnet/tests/Foundry.Hosting.IntegrationTests/README.md | Documents setup, required env vars, and scenario layout. |
| dotnet/tests/Foundry.Hosting.IntegrationTests/Fixtures/HostedAgentFixture.cs | Base fixture that provisions hosted agents, waits for Active, and cleans up. |
| dotnet/tests/Foundry.Hosting.IntegrationTests/Fixtures/HappyPathHostedAgentFixture.cs | Scenario fixture for IT_SCENARIO=happy-path. |
| dotnet/tests/Foundry.Hosting.IntegrationTests/Fixtures/ToolCallingHostedAgentFixture.cs | Scenario fixture for IT_SCENARIO=tool-calling. |
| dotnet/tests/Foundry.Hosting.IntegrationTests/Fixtures/ToolCallingApprovalHostedAgentFixture.cs | Scenario fixture for IT_SCENARIO=tool-calling-approval. |
| dotnet/tests/Foundry.Hosting.IntegrationTests/Fixtures/ToolboxHostedAgentFixture.cs | Scenario fixture for IT_SCENARIO=toolbox. |
| dotnet/tests/Foundry.Hosting.IntegrationTests/Fixtures/McpToolboxHostedAgentFixture.cs | Scenario fixture for IT_SCENARIO=mcp-toolbox. |
| dotnet/tests/Foundry.Hosting.IntegrationTests/Fixtures/CustomStorageHostedAgentFixture.cs | Scenario fixture for IT_SCENARIO=custom-storage. |
| dotnet/tests/Foundry.Hosting.IntegrationTests/HappyPathHostedAgentTests.cs | Happy-path tests for streaming, multi-turn, stored=false behaviors. |
| dotnet/tests/Foundry.Hosting.IntegrationTests/ToolCallingHostedAgentTests.cs | Tool-calling tests validating function call behavior and multi-turn. |
| dotnet/tests/Foundry.Hosting.IntegrationTests/ToolCallingApprovalHostedAgentTests.cs | Approval-flow tests for tool execution requiring approval. |
| dotnet/tests/Foundry.Hosting.IntegrationTests/ToolboxHostedAgentTests.cs | Toolbox tests (currently placeholders where APIs are pending). |
| dotnet/tests/Foundry.Hosting.IntegrationTests/McpToolboxHostedAgentTests.cs | MCP toolbox tests against Learn MCP endpoint (placeholder). |
| dotnet/tests/Foundry.Hosting.IntegrationTests/CustomStorageHostedAgentTests.cs | Custom storage tests (placeholder for container wiring). |
| dotnet/tests/Foundry.Hosting.IntegrationTests.TestContainer/Foundry.Hosting.IntegrationTests.TestContainer.csproj | New ASP.NET container project for scenario implementations. |
| dotnet/tests/Foundry.Hosting.IntegrationTests.TestContainer/Program.cs | Scenario switch that wires the correct agent and maps Foundry Responses endpoints. |
| dotnet/tests/Foundry.Hosting.IntegrationTests.TestContainer/Dockerfile | Runtime image definition for the published container output. |
| dotnet/tests/Foundry.Hosting.IntegrationTests.TestContainer/.dockerignore | Excludes build outputs and repo metadata from Docker context. |
| dotnet/src/Shared/IntegrationTests/TestSettings.cs | Adds IT_HOSTED_AGENT_IMAGE env var constant. |
| dotnet/agent-framework-dotnet.slnx | Registers the new test project and test container project in the solution. |
rogerbarreto
had a problem deploying
to
integration
GitHub Actions
Error
rogerbarreto
had a problem deploying
to
integration
GitHub Actions
Error
Fix issues raised by Copilot review: * it-build-image.ps1: hash file contents, not the path list, so any source edit produces a fresh tag. Normalize Registry input by stripping scheme and trailing slash before deriving the ACR short name. Validate the short name is non empty. * HostedAgentFixture: route GetAgentAsync through _adminClient (which has the FoundryFeaturesPolicy attached) instead of through _projectClient.AgentAdministrationClient (which does not). * HostedAgentFixture FoundryFeaturesPolicy: replace Headers.Add with Remove plus Add so retries cannot accumulate duplicate headers. * HappyPath, ToolCalling, ToolCallingApproval, CustomStorage tests: create the AgentSession before turn 1 and reuse it for both turns. The previous pattern created the session after turn 1 so turn 2 had no link to turn 1, defeating the multi turn assertion.
github-actions
Bot
changed the title
rogerbarreto
temporarily deployed
to
integration
GitHub Actions
Inactive
rogerbarreto
temporarily deployed
to
integration
GitHub Actions
Inactive
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
… format autofix - Set <TargetFrameworks>net10.0</TargetFrameworks>: the project references both Microsoft.Agents.AI.Foundry.Hosting (net8/9/10 only) and AgentConformance.IntegrationTests (net10.0;net472 — inherits the tests-default TFM list). The intersection is net10.0; the previous $(TargetFrameworksCore) triple caused NU1702 + System.Text.Json version conflicts on the net8.0/net9.0 builds because AgentConformance had no matching asset. - Apply `dotnet format` autofix on the test files (IDE0005, IDE0009, IDE0032, IMPORTS).
…UTF-8 BOM CI's check-format requires charset=utf-8-bom per .editorconfig.
…ted agents
Make the integration tests usable end-to-end against a live Foundry deployment, including
a per-run rebuild of the test container so framework code changes are exercised.
Fixture (HostedAgentFixture.cs)
* Switch from per-run unique agent names to stable scenario-keyed names (it-happy-path,
it-tool-calling, ...). The agent's managed identity carries the Azure AI User role on
the project scope, which is required for inbound inference; deleting the agent recycles
the MI and breaks that role assignment, so we keep the agent across runs and only churn
versions.
* Add IT_RUN_ID env var to defeat Foundry's content-addressed version dedup; otherwise a
rerun just receives the existing version and Dispose deletes it.
* PATCH the per-agent endpoint with AgentEndpointConfig (Responses protocol, version
selector at 100% to the new version). Without this, /agents/{name}/endpoint/protocols/
openai/responses returns HTTP 400.
* Build a per-agent ProjectOpenAIClient (not the cached projectClient.ProjectOpenAIClient,
which is bound to the project-level URL); set AgentName in options so the URL routes
through the agent endpoint, and add the Foundry-Features header to the inference
pipeline.
* Use Versions (which serializes to container_protocol_versions) instead of the
deprecated ProtocolVersions; the server now rejects the legacy field.
* On Dispose, delete only the version this fixture created. Never delete the agent.
Tests
* Tag every HostedAgentTests class with [Trait("Category", "FoundryHostedAgents")] so the
CI workflow can route them to a separate Foundry project than the rest of the
integration suite.
CI workflow (.github/workflows/dotnet-build-and-test.yml)
* Add a foundryHosting paths-filter covering Microsoft.Agents.AI.Foundry.Hosting and its
in-repo dependency chain (Foundry, Agents.AI, Agents.AI.Abstractions), the test
container, the test fixture, Directory.Packages.props, the build script, and this
workflow file. Skip the costly hosted-agent steps when none of those changed.
* Add "Build and push Foundry Hosted Agents test container" step that invokes
scripts/it-build-image.ps1 against vars.IT_HOSTED_AGENT_REGISTRY and pipes the resulting
IT_HOSTED_AGENT_IMAGE=<tag> into GITHUB_ENV.
* Add "Run Foundry Hosted Agents Integration Tests" step that filters in only the new
trait, with AZURE_AI_PROJECT_ENDPOINT/AZURE_AI_MODEL_DEPLOYMENT_NAME pointed at
IT_HOSTED_AGENT_PROJECT_ENDPOINT/IT_HOSTED_AGENT_MODEL_DEPLOYMENT_NAME (Tao project,
East US 2; the SK IT project's region does not yet support hosted agents preview).
* Exclude the new trait from the existing "Run Integration Tests" step.
* TEMP: drop the != 'pull_request' guard on the new steps and on Azure CLI Login when the
paths-filter triggers, so PR microsoft#5598 can validate the wiring before promoting to merge
queue only. Restore the original guard after one green PR run.
Build script (scripts/it-build-image.ps1)
* Hash now spans TestContainer source AND its referenced framework projects so any
framework code change forces a fresh tag and a real docker push; the previous
TestContainer-only hash silently reused stale images on framework edits.
Bootstrap script (dotnet/tests/Foundry.Hosting.IntegrationTests/scripts/it-bootstrap-agents.ps1)
* New idempotent script that creates the six stable scenario agents and grants Azure AI
User on the project scope to each agent's MI. Run once per Foundry project. Includes
AAD-graph propagation retries because newly created MIs take time to appear there.
README (dotnet/tests/Foundry.Hosting.IntegrationTests/README.md)
* Document the bootstrap prerequisite, the regional caveat (East US 2 is the only region
we have validated; East US returned "Unsupported region" at the time of writing), the
per-run image rebuild, and the CI wiring including the SP RBAC requirements.
SDK pin (TEMP)
* Bump Microsoft.Agents.AI.Foundry.Hosting's Azure.AI.Projects VersionOverride to
2.1.0-alpha.20260505.1 from the azure-sdk public daily feed (added to nuget.config).
This release is the first that builds the per-agent inference URL as
/agents/{name}/endpoint/protocols/openai (the 2.1.0-beta.1 release builds
.../openai/openai/v1, which the server rejects). Revert both the feed and the override
once the URL fix lands in a stable Azure.AI.Projects release.
… PATCH to bootstrap The alpha SDK pin (Azure.AI.Projects 2.1.0-alpha.20260505.1 from the azure-sdk public daily feed) was needed only for the URL routing fix and the strongly-typed AgentEndpointConfig/PatchAgentOptions wrapper. We do not need either right now: the fixture stays compatible with the public 2.1.0-beta.1 by moving the one-time endpoint PATCH to the bootstrap script (it sets version_selector to FixedRatio @latest, so each new fixture run becomes the served version automatically without a per-run PATCH from the test code). The hosted-agent invocation path will start working end-to-end once the URL routing fix lands in a stable Azure.AI.Projects release; until then the tests stay [Fact(Skip = ...)] as documented. * Revert dotnet/nuget.config: drop the azure-sdk-for-net public feed. * Revert Microsoft.Agents.AI.Foundry.Hosting.csproj VersionOverride to 2.1.0-beta.1. * Revert Microsoft.Agents.AI.Foundry.UnitTests and Microsoft.Agents.AI.Foundry.Hosting.UnitTests Azure.AI.Projects pin (they had been bumped to align Azure.Core 1.54 transitive). * Drop the AgentEndpointConfig PATCH block from HostedAgentFixture.cs (the type is alpha-only). Replace with a comment pointing at the bootstrap script. * Bootstrap script (it-bootstrap-agents.ps1) now also PATCHes each agent's endpoint with version_selector=@latest if not already set. Idempotent.
…e Foundry endpoint
The fixture already constructs ProjectOpenAIClient via the per-agent path that beta.1
supports (new ProjectOpenAIClient(uri, cred, opts { AgentName })), so no SDK pin bump
is required to run the smoke tests end-to-end. Un-skip the 5 tests that pass against
the live test container.
Tests un-skipped (verified passing locally against tao-foundry-prj):
* RunAsync_ReturnsNonEmptyTextAsync
* RunStreamingAsync_YieldsAtLeastOneUpdateAsync
* MultiTurn_WithPreviousResponseId_PreservesContextAsync
* StoredFalse_Baseline_DoesNotPersistResponseAsync
* Instructions_FromContainerDefinition_AreObeyedAsync
Tests still skipped with a more specific reason (4 of 9 in HappyPath plus all
ToolCalling*, McpToolbox, Toolbox, CustomStorage) because the test container does not
yet emit usable response_id / conversation_id chains, and the placeholder scenarios are
not implemented in the test container's Program.cs. These are test container limitations,
not infra bugs, and can be un-skipped as the container surfaces stabilize.
rogerbarreto
changed the title
lokitoth
approved these changes
May 5, 2026
westey-m
reviewed
May 6, 2026
westey-m
reviewed
May 6, 2026
westey-m
reviewed
May 6, 2026
westey-m
approved these changes
May 6, 2026
…, add Workflows dep Address Wesley's review feedback on PR microsoft#5598: 1. Pull Foundry hosted-agent IT into its own dotnet-foundry-hosted-it job that runs in parallel to dotnet-build and dotnet-test. Same path-filter gate keeps it skipped on unrelated edits. Builds only the filtered solution containing Foundry.Hosting.IntegrationTests and src deps. dotnet-build-and-test-check now waits on it too. 2. Add Microsoft.Agents.AI.Workflows to the foundryHosting paths-filter and to hashedDirs in it-build-image.ps1 since Foundry.Hosting transitively depends on it. TFM constraint on the IT csproj stays at net10.0 because AgentConformance.IntegrationTests targets net10/net472 and is consumed by ~12 other IT projects on net472.
westey-m
approved these changes
May 6, 2026
This was referenced May 8, 2026
Open
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a new integration test project,
Foundry.Hosting.IntegrationTests, that provisions real Foundry hosted agents and exercises them throughMicrosoft.Agents.AI.Foundry.Hosting. Sits next toFoundry.IntegrationTests, mirroring the source split betweenMicrosoft.Agents.AI.FoundryandMicrosoft.Agents.AI.Foundry.Hosting. CI wiring builds and pushes the test container image on every IT run, then runs the tests against a Foundry project that has the hosted-agents preview enabled.How it works
Foundry.Hosting.IntegrationTests.TestContainer) hosts all scenario specific behavior. TheIT_SCENARIOenvironment variable on eachHostedAgentDefinitionselects the branch in the container'sProgram.cs. So one image build per source change covers every fixture.IClassFixture<T>derived fromHostedAgentFixture. The base provisions a fresh agent version, polls untilActive, exposes the wrappedAIAgentviaAgent, and onDisposeAsyncremoves only the version it created.it-happy-path). The agent itself is provisioned out of band byscripts/it-bootstrap-agents.ps1and its managed identity holdsAzure AI Useron the project scope. The fixture only adds and deletes versions on the agent. Deleting the agent recycles the MI and would invalidate the role assignment, so the agent is intentionally never deleted.ProjectOpenAIClientconstructed withAgentNameset in the options; the cachedaiProjectClient.ProjectOpenAIClientcannot serve a hosted agent because it is bound to the project-level URL.scripts/it-build-image.ps1 -Registry <login-server>publishes the test container, builds the Docker image with a content hashed tag (covering the test container source AND its referenced framework projects), pushes to ACR, and emitsIT_HOSTED_AGENT_IMAGE=...for shells and CI to consume. Identical content is adocker pushno-op.agent.RunAsync(...)overMicrosoft.Agents.AIabstractions, no raw HTTP.Scenarios
IT_SCENARIOHappyPathHostedAgentFixturehappy-pathit-happy-pathToolCallingHostedAgentFixturetool-callingit-tool-callingToolCallingApprovalHostedAgentFixturetool-calling-approvalit-tool-calling-approvalToolboxHostedAgentFixturetoolboxit-toolboxMcpToolboxHostedAgentFixturemcp-toolboxit-mcp-toolboxCustomStorageHostedAgentFixturecustom-storageit-custom-storage24 tests total. 5 are now executing live in CI, 19 are skipped with explicit reasons (4 with a container-gap message, 15 placeholders). Each placeholder is unblocked by wiring its branch in the test container
Program.csand removing the Skip annotation, in a follow up PR.What's in this PR
Test project and fixtures
dotnet/tests/Foundry.Hosting.IntegrationTests/test project (net10.0),HostedAgentFixturebase, 6 derived fixtures, 24 tests, README.dotnet/tests/Foundry.Hosting.IntegrationTests.TestContainer/ASP.NET test container, Dockerfile,.dockerignore. Happy path and tool calling branches wired; the rest are placeholders.[Trait("Category", "FoundryHostedAgents")]on every test class so the workflow can route them to a separate Foundry project than the rest of the integration suite.Scripts
dotnet/tests/Foundry.Hosting.IntegrationTests/scripts/it-build-image.ps1content-hashed tag (test container source + framework dep chain), idempotent push.dotnet/tests/Foundry.Hosting.IntegrationTests/scripts/it-bootstrap-agents.ps1one-time idempotent bootstrap that creates the six scenario agents and grantsAzure AI Userto each agent's MI on the project scope. AAD-graph propagation retry built in. Also patches the agent endpoint withversion_selector=@latestso each new fixture-created version is automatically served.CI workflow (
.github/workflows/dotnet-build-and-test.yml)foundryHostingpaths-filter scoped toMicrosoft.Agents.AI.Foundry.Hostingand its in-repo dependency chain (Microsoft.Agents.AI.Foundry,Microsoft.Agents.AI,Microsoft.Agents.AI.Abstractions), the test container, the IT project,Directory.Packages.props, the build script, and this workflow file. Skip the costly hosted-agent steps when none of those changed.it-build-image.ps1againstvars.IT_HOSTED_AGENT_REGISTRYand pipesIT_HOSTED_AGENT_IMAGE=<tag>into$GITHUB_ENV.Category=FoundryHostedAgentsagainst the env vars below.Configuration
The
integrationGitHub environment now needs three additional variables:IT_HOSTED_AGENT_PROJECT_ENDPOINTAZURE_AI_PROJECT_ENDPOINTIT_HOSTED_AGENT_MODEL_DEPLOYMENT_NAMEAZURE_AI_MODEL_DEPLOYMENT_NAMEIT_HOSTED_AGENT_REGISTRYit-build-image.ps1)The CI service principal (
AgentFrameworkIntegrationTests) needsAzure AI UserandCognitive Services Data Contributor (Preview)on the hosted-agents Foundry project scope, andAcrPushon the registry. Already granted on the active validation environment.Verification
CI run on
microsoft/agent-framework: https://github.com/microsoft/agent-framework/actions/runs/25395109487Foundry.Hosting.IntegrationTests: succeeded 5, skipped 19, failed 0.acrhostedagenttao.azurecr.io/foundry-hosting-it:838aa1af46cd), consumed by step 14.Microsoft.Agents.AI.Foundry.Hostingimproves from 0% to 75.5% via the live 5 tests.Out of scope (follow up PRs)
ToolCallingApproval,Toolbox,McpToolbox,CustomStorage) in the test containerProgram.cs, then un-skip those tests.response_id/conversation_idchains so the 4 skipped HappyPath tests can run.Azure.AI.Projectsships either a fluentaiProjectClient.GetProjectResponsesClientForHostedAgent(name)helper or a workingaiProjectClient.GetProjectOpenAIClient(opts { AgentName })overload, the per-agentProjectOpenAIClientconstruction inHostedAgentFixturecan be simplified. Beta.1 already supports the per-agent URL via the explicitnew ProjectOpenAIClient(uri, cred, opts { AgentName })ctor that the fixture uses today.x-chat-key/x-user-key). Deferred untilMicrosoft.Agents.AI.Foundry.Hostingexposes per call header injection.