Flaky Smoke Tests

Diagnosing Flaky Smoke Tests

Use this skill for failures from the Azure DevOps Flaky Smoke Tests pipeline:

• Organization: https://dev.azure.com/monacotools
• Project: Monaco
• Definition ID: 700
• Pipeline YAML: build/azure-pipelines/product-smoke-flaky.yml
• Scheduled runs: twice daily on main

The pipeline builds VS Code from source and runs the complete Electron smoke

suite once per entry in the iterations parameter. Each platform runs its

iterations sequentially in one job.

Important Pipeline Behavior

• Each iteration is a separate timeline task named

Smoke test iteration <N>/<total> (Electron).

• Iteration tasks use continueOnError: true, so a failed iteration normally

has result succeededWithIssues and later iterations still run.

• Do not infer smoke-test health from the overall build or job result. Inspect

every iteration task.

• Logs are published once per platform job from the shared .build/logs

directory. The artifact is job-scoped, not iteration-scoped:

smoke-test-runner.log can contain multiple iterations, and suite

directories are not separated by iteration.

• Do not cancel a platform job while investigating if you still need its logs

artifact. Cancellation can skip the publish steps entirely, even when one or

more failed iteration task logs are already available. If a validation run

has become predictably red, either let the current job finish or queue a

single diagnostic iteration and let that job publish before iterating.

• Later iterations can overwrite or truncate files in the shared logs

directory. The final artifact may therefore contain the last iteration's

suite directory or runner log but not the failed iteration's detailed

diagnostics. Preserve the exact task log, and use a one-iteration diagnostic

run when extension-host, renderer, mock-server, screenshot, or trace evidence

from the failed attempt is required.

• The exact iteration task log is authoritative for its failure summary and

time range. Correlate that range with the job-level artifact.

• Iterations can see persisted smoke-test state from earlier iterations, such

as historical session rows. Never assume the newest-looking row or a matching

prompt belongs to the active conversation without verifying the active view.

Platform definitions and artifacts:

| Platform | Job | Logs artifact | Crash artifact |

|---|---|---|---|

| macOS arm64 | macOSSmokeFlaky | logs-macos-arm64-smoke-<attempt> | crash-dump-macos-arm64-smoke-<attempt> |

| Windows x64 | WindowsSmokeFlaky | logs-windows-x64-smoke-<attempt> | crash-dump-windows-x64-smoke-<attempt> |

| Linux x64 | LinuxSmokeFlaky | logs-linux-x64-smoke-<attempt> | crash-dump-linux-x64-smoke-<attempt> |

Prerequisites

Use Azure CLI rather than the browser:

az --version
az extension show --name azure-devops
az devops configure --defaults \
  organization=https://dev.azure.com/monacotools \
  project=Monaco

If authentication fails, run az login.

1. Confirm the Build and Source Revision

az pipelines build show \
  --id <BUILD_ID> \
  --org https://dev.azure.com/monacotools \
  --project Monaco \
  --query "{id:id,status:status,result:result,sourceBranch:sourceBranch,sourceVersion:sourceVersion,templateParameters:templateParameters}" \
  --output json

Record sourceVersion. Investigate the exact code built by the pipeline, not

the current working tree.

2. Find Failed Iterations

Download the build timeline:

az devops invoke \
  --org https://dev.azure.com/monacotools \
  --area build \
  --resource timeline \
  --route-parameters project=Monaco buildId=<BUILD_ID> \
  --output json

Filter task records whose names contain Smoke test iteration. Treat

succeededWithIssues, failed, and tasks with error issues as failed

iterations. Record:

• task id
• parent job id
• task name
• startTime and finishTime
• log.id
• issues

If the user supplied an Azure log URL, its j= value is the job ID and its

t= value is the task ID. Query those records directly:

az devops invoke \
  --org https://dev.azure.com/monacotools \
  --area build \
  --resource timeline \
  --route-parameters project=Monaco buildId=<BUILD_ID> \
  --query "records[?id=='<JOB_OR_TASK_ID>'].{id:id,parentId:parentId,name:name,type:type,state:state,result:result,logId:log.id,startTime:startTime,finishTime:finishTime,issues:issues}" \
  --output json

3. Download the Exact Iteration Task Log

Use timeline record log.id. The correct REST resource is logs, not

buildLog:

az devops invoke \
  --org https://dev.azure.com/monacotools \
  --area build \
  --resource logs \
  --route-parameters project=Monaco buildId=<BUILD_ID> logId=<LOG_ID> \
  --out-file task-log.json

The downloaded response is JSON with a value array containing one string per

log line. Convert it to plain text.

PowerShell:

(Get-Content -Raw task-log.json | ConvertFrom-Json).value |
  Set-Content task-log.txt

Bash:

jq -r '.value[]' task-log.json > task-log.txt

The task log provides the concise Mocha failure, stack, and the iteration's

exact time range even before platform artifacts are published. Diagnostics

written through the smoke runner's Logger (including

dumpFailureDiagnostics) may exist only in smoke-test-runner.log, so do not

assume they will be present in the Azure task log.

4. Download the Platform Logs Artifact

List artifacts first; names include the job attempt:

az pipelines runs artifact list \
  --run-id <BUILD_ID> \
  --org https://dev.azure.com/monacotools \
  --project Monaco \
  --output table

Download the relevant platform:

az pipelines runs artifact download \
  --run-id <BUILD_ID> \
  --artifact-name <LOGS_ARTIFACT> \
  --path <DESTINATION> \
  --org https://dev.azure.com/monacotools \
  --project Monaco

Artifacts are available after the platform job publishes its outputs.

An absent platform artifact after a canceled run is expected; it is not

evidence that the artifact name was wrong.

5. Correlate the Failure

Start with <DESTINATION>/smoke-tests-electron/smoke-test-runner.log.

Use the failed task's timestamps and test title to isolate the matching

Test start / Test end interval. Do not use the first occurrence of a test

title because the runner log can contain several iterations.

Within that interval, establish this chain:

• Gesture: Was the expected editor clicked and prompt typed?
• Dispatch: Did the send action fire and did the new-session view close?
• Request: Did the mock server receive the expected scenario tag?
• Tool loop: For shell tests, did the second model request contain the

expected tool result?

• Rendering: Did an assistant response render, and in which session view?
• Routing: Did the active session auto-swap to a new composer while the

completed response remained in another session?

For mock-LLM suites, search for:

• the scenario ID
• request body:
• the expected response marker
• model turn 1/2 and model turn 2/2

The request log distinguishes:

• no dispatch
• wrong provider or scenario
• tool call never executed
• tool result returned but UI rendering/routing lost it

Then inspect the suite directory:

smoke-tests-electron/<N>_suite_<Suite_Name>/

Useful files include:

• window*/exthost/<extension>/<extension>.log
• main.log, renderer.log, and agenthost.log
• copilot-runtime-logs/process-*.log — the Copilot runtime (@github/copilot

CLI) process logs, captured by dumpFailureDiagnostics when a Copilot-runtime

session fails. Check these first for a **hang or "Timed out waiting for

response"**: they are the SDK/CLI's own record (startup, auth, model request,

turn lifecycle, panics, out-of-order or protocol errors) and explain a timeout

the test error alone does not. A tail is also mirrored into

smoke-test-runner.log. Agent Host sessions (Agents Window / local

AgentHost) write a full log run at trace; Chat Sessions editor (Copilot CLI /

Claude) and Local sessions write only a minimal startup log here (whether the

runtime came up), with their detailed diagnostics in GitHub Copilot Chat.log.

(Claude / Codex sessions use a different runtime and are not captured here.)

• playwright-screenshot-*.png
• Playwright trace archives

For native exits or renderer crashes, also download the platform crash artifact.

6. Interpret UI Diagnostics Carefully

Session-list text and active-view text answer different questions:

• A list row can contain a prompt from the current test but a command or title

inherited from a prior request.

• The active view can already be a fresh untitled composer while the response

belongs to a completed, inactive session.

• Broad response selectors can match stale DOM from an earlier session or

iteration.

• Reusing a fixed warm-up scenario marker can let a later warm-up wait match an

earlier response and abandon the actually-running warm-up.

• Standard smoke suites created through installAllHandlers use distinct

randomized user-data directories. Do not attribute a setting from the

preceding suite to the failing suite without comparing the actual

vscode-userdata: or --user-data-dir paths in their logs. Iterations share

broader job state, but sibling suites normally do not share the same profile.

• A setup change can alter rather than fix the symptom. Re-check the exact

stack and the gesture/dispatch/request/rendering chain on every validation

run. For example, moving extension enablement before startup can eliminate an

editor-open timeout while activating the extension before test settings are

written, producing a later request or response timeout instead.

Prefer assertions that establish identity:

• unique scenario IDs or response markers per attempt
• active chat/session resource attributes
• expected response in the active session before follow-up input
• mock-server request count or captured request content after the gesture

Do not "fix" these races by only increasing a timeout. Wait for the actual

state transition or remove duplicate/in-flight work.

7. Find the Introducing Commit

Use the build's sourceVersion and identify the source file from the stack:

git show <SOURCE_VERSION>:test/smoke/src/areas/<area>/<test>.test.ts
git log --oneline <KNOWN_GOOD>..<SOURCE_VERSION> -- <relevant paths>
git blame -L <start>,<end> <file>
git show <SUSPECT_COMMIT> -- <relevant paths>

Trace the whole causal sequence, not only the failing assertion. For example,

inspect setup hooks, warm-ups, session selection, response waits, and teardown.

Distinguish:

• the commit that introduced the racy behavior
• a later unrelated commit after which timing happened to expose it
• a commit that only added diagnostics or made the flake visible

State the introducing commit only when the diff contains the causal behavior

and the pipeline evidence matches it.

8. Queue a Focused Validation Run

The branch and commit must already be pushed to microsoft/vscode. Check for

and cancel obsolete definition-700 runs on the same branch before queueing.

Use az pipelines run directly because iterations is an object parameter:

az pipelines run \
  --id 700 \
  --branch <BRANCH> \
  --commit-id <COMMIT_SHA> \
  --parameters "iterations=[1,2,3,4,5,6]" \
    VSCODE_BUILD_MACOS=true \
    VSCODE_BUILD_LINUX=false \
    VSCODE_BUILD_WIN32=true \
  --org https://dev.azure.com/monacotools \
  --project Monaco \
  --output json

Enable only platforms relevant to the failure. Confirm the queued run's

sourceVersion and templateParameters with az pipelines build show.

Six iterations are a useful quick validation sample. Use the default 20 when

the failure is rare or when validating before declaring a recurring flake

resolved.

9. Validate the Local Change

For smoke-test TypeScript changes:

npm run compile --prefix test/smoke
node --experimental-strip-types build/hygiene.ts <changed-file>

If local dependencies are missing or stale, report that explicitly and rely on

the focused definition-700 run for full compiled validation; do not silently

skip validation or install unrelated tooling.

Expected Report

Summarize:

• build ID and source revision
• failing platform and iteration count
• exact failing test and symptom
• dispatch/request/rendering evidence
• root cause
• introducing commit, with why it is causal
• fix commit
• focused validation run ID, platforms, iterations, and current result