Rerecord Tests

Re-record ai-projects tests and publish recordings

Records the live test suite for @azure/ai-projects, verifies the new

recordings play back green, pushes only the passing recordings to the

Azure/azure-sdk-assets repo, and opens a PR that contains just the updated

assets.json tag.

This package uses the asset-sync workflow: recordings live in

Azure/azure-sdk-assets, not in this repo. The only file that lands in a PR is

assets.json, whose Tag points at the pushed

recordings. See sdk/test-utils/recorder/ASSET_SYNC_WORKFLOW.md.

When to Use

• Recordings are stale and the recorded test suite fails in playback mode.
• New recorded tests were added (e.g. after the author-tests skill) and need

real recordings captured against live Azure resources.

• A deliberate re-record is requested to refresh assets.json.

Prerequisites

• Live Azure access. Record mode hits real Foundry resources. You must be

authenticated (az login, or a working DefaultAzureCredential) and have the

test environment variables populated. Key ones (see

test/public/utils/createClient.ts):

FOUNDRY_PROJECT_ENDPOINT, FOUNDRY_MODEL_NAME, FOUNDRY_AGENT_NAME,

AZURE_AI_PUBLISHED_ENDPOINT, MODEL_ENDPOINT, plus the connection-id vars

for the tool tests. Missing vars only fail the tests that need them — those

recordings are simply not pushed (see "Only passing recordings", below).

• Write access to Azure/azure-sdk-assets. Required by test-proxy push.

Request access here).

• pwsh and the gh CLI installed and authenticated against

Azure/azure-sdk-for-js.

• A clean working tree apart from changes under sdk/ai/ai-projects/.

Procedure

Run everything from sdk/ai/ai-projects/. The flow mirrors the asset-sync

diagram: record → verify in playback → push → commit assets.json → PR.

Step 1: Restore current recordings (baseline)

npx dev-tool test-proxy restore

Pulls the recordings referenced by the current assets.json so re-recording

diffs against a known baseline.

Step 2: Record, verify playback, and (optionally) push

./.github/skills/rerecord-tests/scripts/rerecord.ps1

The script (scripts/rerecord.ps1):

• Runs the suite in record mode (TEST_MODE=record npm run test:node),

writing fresh recordings into the repo-root .assets/ working tree.

• Runs the suite again in playback mode (TEST_MODE=playback) with no

live env vars, to prove the recordings are self-contained.

• If any spec failed playback, it refuses to push and prints the exact

manual steps to drop the failing recordings (so only passing recordings are

ever published — see "Only passing recordings").

• Stops there by default. Pass -Push to also run `npx dev-tool test-proxy

push, which uploads the kept recordings and rewrites the Tag` in

assets.json. -Push is ignored when playback is red.

Useful flags:

• -TestFilter <pattern> — scope to a subset (passed through to the vitest

runner) instead of the whole suite.

• -Push — push recordings and update assets.json after a green playback.
• -SkipRecord — skip record mode and only re-run playback verification

(useful after manually fixing a recording).

Step 3: Push recordings (if not already pushed)

If you ran Step 2 without -Push, push once playback is green:

npx dev-tool test-proxy push

This updates the Tag field in assets.json. Confirm the diff:

git --no-pager diff -- assets.json

Only the Tag line should change. The recordings are now visible at

https://github.com/Azure/azure-sdk-assets/tree/<new-tag>.

Step 4: Open the assets.json PR

./.github/skills/rerecord-tests/scripts/open-assets-pr.ps1

The script (scripts/open-assets-pr.ps1):

• Aborts if assets.json has no change, or if there are changes outside

sdk/ai/ai-projects/assets.json (the only file this PR should touch).

• Creates branch rerecord/ai-projects/<yyyyMMdd-HHmm> from HEAD.
• Commits only assets.json with message

[ai-projects] Re-record tests: update assets.json.

• Pushes to origin (never force) and opens a draft PR via gh, with a

body that links the new assets tag and lists the verification checklist.

Only passing recordings

The user contract is "push the recordings that pass." A recording is "passing"

only if its spec is green in playback mode (Step 2.2). When playback is red,

the script refuses to push and tells you to drop the failing recordings first,

so the assets tag never publishes a broken recording.

To inspect or hand-tune what will be pushed:

• Run npx dev-tool test-proxy diff — it locates the .assets/<hash>/ clone

(via .assets/.breadcrumb) and prints the changed recording files for this

package.

• cd into that clone dir and git restore <file> (or git clean) any

recording you do not want pushed.

• Return to sdk/ai/ai-projects/ and run npx dev-tool test-proxy push (or

./.github/skills/rerecord-tests/scripts/rerecord.ps1 -SkipRecord -Push).

Out of scope

• Does not provision or configure live Azure resources or env vars. If

record mode can't authenticate, surface that to the human — do not fabricate

recordings.

• Does not edit test source (test/**). If a spec needs a code change to

record cleanly, that belongs to the author-tests skill or a human.

• Does not flip the PR to ready-for-review — humans do that.

Safety rails

• Never runs git push --force / --force-with-lease.
• Never commits anything other than assets.json. Recordings live in the

assets repo, not here.

• Never pushes recordings for specs that failed playback.
• Aborts if the working tree is dirty outside sdk/ai/ai-projects/.

Notes

• Record mode is slow and bills real services; prefer -TestFilter while

iterating on a single failing spec, then do a full run before pushing.

• If playback fails right after a successful record, the recording likely

contains an unsanitized value or a non-deterministic field. Re-run with

dev-tool run test:vitest --test-proxy-debug to see sanitizer activity, fix

the sanitizer in test/public/utils/createClient.ts, and re-record.

• npm run test:node runs both recorded and unit-style specs. Pure unit

specs (e.g. test/public/tracing/*.spec.ts) don't produce recordings and are

unaffected by record/playback mode.