Foundry Hosted Agent Validation

Validating a Foundry Hosted Agent Sample

A hosted agent sample is "validated" when it passes three independent

checks, plus cleanup:

• Local, native runtime — run the sample's own entry point

(python main.py) and invoke it over HTTP.

• Local, via azd ai agent run — the azd local dev loop.
• Deployed — azd deploy to Foundry, then invoke the hosted agent.

Each check must succeed for single-turn and multi-turn (session /

previous_response_id) conversation. Always end with cleanup (delete the

deployed agent, remove the temp azd project, restore the sample dir).

> Read the sample's own README.md and the parent

> .../foundry-hosted-agents/README.md first — they define the run/deploy

> commands and any sample-specific payload. This skill captures the process and

> the non-obvious gotchas the READMEs don't.

Inputs you need before starting

Gather these:

• Foundry project endpoint, e.g.

https://<account>.services.ai.azure.com/api/projects/<project>.

• Foundry project resource id (for non-interactive azd ai agent init):

/subscriptions/<sub>/resourceGroups/<rg>/providers/Microsoft.CognitiveServices/accounts/<account>/projects/<project>.

Find it with az cognitiveservices account list + the project name.

• A real, deployed model name in that project (e.g. gpt-4.1-mini). This is

often different from the model id in agent.manifest.yaml — the actual

deployment name wins.

• An existing ACR to reuse for deployment (login server, e.g.

myacr.azurecr.io). Reusing one avoids azd provision creating resources.

• Whether a like-named agent already exists in the project (remove it first

for a clean validation — see below).

Tooling / auth

• az (logged in: az login) and azd (logged in: azd auth login).
• azd agents extension: azd extension list should show

azure.ai.agents; install with azd extension install azure.ai.agents.

• uv for the native-Python local run. python need not be on PATH — uv

and azd ai agent run provision their own interpreter.

• Docker is not required when you reuse an ACR (remoteBuild: true builds

in ACR Tasks).

Phase 0 — Understand the sample

A responses/invocations sample folder typically contains:

main.py (entry point + ResponsesHostServer/InvocationsHostServer),

agent.manifest.yaml (used by azd ai agent init), agent.yaml (the deployed

agent definition), requirements.txt, Dockerfile, .env.example.

**The sample is the whole directory whose entry point is main.py — not every

.py file in it.** Other Python files in (or alongside) a sample folder are

helper/companion scripts, not standalone samples. Do not treat a helper

script as an individual sample — validate the sample via its main.py host, and

run a helper only when the sample's README.md calls for it as a setup.

Note the protocol (responses or invocations) from agent.yaml /

manifest — it changes the invoke command (--protocol invocations) and the HTTP

path (/responses vs the invocations route).

Phase 1 — Local validation, native runtime (Python)

Run from the sample directory.

uv venv .venv --python 3.12          # 3.12 matches the sample Dockerfile
uv pip install --python .venv/... -r requirements.txt

Create .env from .env.example with the real values:

FOUNDRY_PROJECT_ENDPOINT="https://<account>.services.ai.azure.com/api/projects/<project>"
AZURE_AI_MODEL_DEPLOYMENT_NAME="<real-deployed-model>"

Start the server (python main.py) — it listens on http://localhost:8088.

main.py uses DefaultAzureCredential, so az login must be current.

Invoke (single turn), capture the returned response_id, then reuse it for a

follow-up turn to confirm memory:

curl -X POST http://localhost:8088/responses -H "Content-Type: application/json" \
  -d '{"input": "My name is Tao. Remember it."}'
# take response_id from the JSON, then:
curl -X POST http://localhost:8088/responses -H "Content-Type: application/json" \
  -d '{"input": "What is my name?", "previous_response_id": "<response_id>"}'

PowerShell: use Invoke-WebRequest -Uri http://localhost:8088/responses -Method POST -ContentType application/json -Body '...'.

Pass: HTTP 200, non-empty output[].content[].text, and the second turn

recalls the name. Stop the server afterward.

Recording the playbook (cached replay)

The sample-validation harness caches a playbook so future runs replay your

result without an agent. For a hosted-agent sample the playbook is an

agent-authored Python script that reproduces **only this Phase 1

native-local smoke test** — never the azd/deploy phases (those are

credential- and deployment-bound, non-deterministic, and must not enter the

replay cache).

Emit a self-contained script that the harness runs with the active interpreter

from the python/ directory (exit code 0 = success). It may import only the

sample's own installed deps, the Python stdlib, and httpx. It must:

• Start the sample server in the background with

subprocess.Popen([sys.executable, "<sample>/main.py"]) (path relative to

python/; the shared python/.env and the process env supply credentials).

• Poll http://localhost:8088 until the port accepts a connection (bounded

readiness loop with a timeout), failing if it never comes up.

• POST turn 1 to the protocol's route (/responses for responses samples;

the invocations route for invocations samples), capture the response_id,

then POST turn 2 with previous_response_id to confirm recall.

• Assert HTTP 200, non-empty output[].content[].text, and that turn 2

recalls the fact from turn 1.

• Always terminate the server in a finally block so port 8088 is freed

(the harness force-kills the process group as a backstop, but the script must

still clean up). On any failure, print the captured server output before

exiting non-zero.

Keep the deep three-phase validation (below) for a full manual/azd pass; it is

out of scope for the cached playbook.

Phase 2 — Local validation via azd ai agent run

Init the azd project (once)

Run in an empty temp directory outside the repo (short path avoids Windows

path-length issues, e.g. C:\afval\<sample>). Point -m at the local

manifest so it validates the working-tree sample:

azd ai agent init -m <path>/agent.manifest.yaml \
  --project-id "<project-resource-id>" \
  --model-deployment "<real-deployed-model>" \
  --agent-name "<agent-name-from-manifest>" \
  --no-prompt --force

init downloads the template into a subfolder named after the agent, so the

azd project root is <tempdir>/<agent-name>/. cd there for all later azd

commands.

> Before init, remove any .venv you created in the sample dir — init

> copies the entire manifest directory into src/. (.venv is excluded from

> deploy packaging by .agentignore/.dockerignore, so it is harmless but

> bloats/slows the copy.)

Fix the model deployment name (critical — see Gotcha 1)

azd env set AZURE_AI_MODEL_DEPLOYMENT_NAME "<real-deployed-model>"

Run and invoke locally

azd ai agent run --no-inspector        # auto-creates a uv venv + installs deps; listens on :8088
azd ai agent invoke --local --new-session "My name is Tao. Remember it."
azd ai agent invoke --local "What is my name?"   # same session is reused automatically

Pass: both invokes return text; the second recalls the name (same

Session: id). Stop the run process afterward.

Removing a pre-existing agent (do this before deploying)

init prints a warning if the agent name already exists in the project. To

delete it, note that **azd ai agent delete/show resolve the deployed agent

name from an azd env var, not from the positional argument.** The var is

AGENT_{SERVICEKEY}_NAME, where SERVICEKEY = the azure.yaml service name

uppercased with -/spaces → _.

Example for service agent-framework-agent-basic-responses:

azd env set AGENT_AGENT_FRAMEWORK_AGENT_BASIC_RESPONSES_NAME agent-framework-agent-basic-responses
azd ai agent delete <service-name> --force --no-prompt --output json
# -> {"object":"agent.deleted","name":"...","deleted":true}

(After a successful azd deploy, this var is set automatically, so later

show/delete/invoke work without setting it.)

Phase 3 — Deploy and validate

Reuse an existing ACR (avoid provisioning)

For an existing project + model, do not run azd provision/azd up — the

generated azure.yaml has a deployments block for the manifest's model

(often an auto-selected GlobalProvisionedManaged PTU SKU) that provision would

try to create (costly / quota failures). Instead reuse an ACR:

azd env set AZURE_CONTAINER_REGISTRY_ENDPOINT <acr-login-server>   # e.g. myacr.azurecr.io
azd deploy

azd deploy fails with _"could not determine container registry endpoint"_ if

this is unset and no ACR is provisioned.

Verify the deployed model env var, then invoke

azd ai agent show <agent-name> --output json   # check definition.environment_variables.AZURE_AI_MODEL_DEPLOYMENT_NAME
azd ai agent invoke <agent-name> --new-session "My name is Tao. Remember it."
azd ai agent invoke <agent-name> "What is my name?"

Use --output raw on invoke to see raw SSE events and any failure, e.g.:

event: response.failed
... "code": "DeploymentNotFound" ... 404 ...

DeploymentNotFound means the deployed AZURE_AI_MODEL_DEPLOYMENT_NAME points

at a model that isn't deployed → fix per Gotcha 1 and redeploy (creates a new

version).

Pass: agent reaches status: active, invoke returns text (not empty, no

response.failed), and multi-turn recalls the name.

Cleanup (always)

• Delete the deployed agent: azd ai agent delete <agent-name> --force --no-prompt.
• Delete the temp azd project directory.
• Remove .env/.venv you created in the sample dir; confirm the sample dir is

pristine (git status --porcelain <dir> is empty — .env/.venv are

gitignored).

• Stop any leftover local server still holding port 8088:

Get-NetTCPConnection -LocalPort 8088 -State Listen → Stop-Process -Id <pid>

(Linux/macOS: lsof -ti:8088 | xargs kill). Stopping the shell may leave the

child interpreter running.

Gotchas (the parts that waste the most time)

• The deployed model name comes from agent.yaml, not the azd env.

azd ai agent init ignores --model-deployment in --no-prompt mode and

writes the manifest's model id (e.g. gpt-4.1-mini) as a literal into

both the azd env AZURE_AI_MODEL_DEPLOYMENT_NAME and the generated

src/<agent>/agent.yaml env var. Local runs read the azd env (so

azd env set fixes them), but deployment injects agent.yaml's value.

Fix by setting the generated agent.yaml env var to the template

value: ${AZURE_AI_MODEL_DEPLOYMENT_NAME} (what the repo sample already uses;

init flattens it) and azd env set AZURE_AI_MODEL_DEPLOYMENT_NAME <real>,

then redeploy. A literal value: <real-model> also works.

• azd ai agent delete/show need the AGENT_{SERVICEKEY}_NAME env var — a

bare positional agent name is treated as the _service_ name and the deployed

agent name is looked up from that env var (see "Removing a pre-existing

agent").

• **azd provision/azd up will try to create the manifest's model

deployment** (from azure.yaml's deployments block). Prefer azd deploy

with a reused ACR when the project + model already exist.

• python on PATH is not required. uv venv and azd ai agent run

provision their own interpreter and install requirements.txt.

• init copies the whole manifest directory into src/. Remove a local

.venv from the sample dir first to keep the copy clean/fast.

• Port 8088 can stay bound after stopping the shell — kill the interpreter

by PID (see Cleanup).

Success checklist

• [ ] Native local run: 200 + non-empty text + multi-turn recall.
• [ ] azd ai agent run local: text returned + session reused across invokes.
• [ ] Pre-existing agent removed (if any).
• [ ] azd deploy succeeds; agent status: active.
• [ ] azd ai agent show confirms AZURE_AI_MODEL_DEPLOYMENT_NAME = the real

deployed model.

• [ ] Deployed invoke: text returned (no response.failed) + multi-turn recall.
• [ ] Cleanup done: agent deleted, temp project removed, sample dir pristine,

port 8088 free.