Pytorch Runners Routing

PyTorch CI Runner Routing & Resolution

Scope

Everything about how .github/workflows/*.yml jobs in pytorch/pytorch land on a

specific runner — label definition, which infrastructure serves it, fleet-prefix

resolution, the experiment system, and the EC2-to-ARC translation pipeline. Read this

BEFORE touching any runs-on: value, any test-matrix: entry, any runner_prefix:

interpolation, any check_experiments: / opt_out_experiments: input, arc.yaml, or

runner_determinator.py / map_ec2_to_arc.py.

For the legacy EC2 world in depth (the Lambda autoscaler, scale-config, dot prefixes,

who owns what), see references/old-ali-infra.md.

The Two Runner Infrastructures

Two completely different infrastructures serve PyTorch CI. Which one a job lands on is

encoded in its runner label's prefix.

• Old ALI — Lambda-autoscaled EC2 (legacy). A workflow_job webhook drives a

scale-up Lambda that launches ONE ephemeral EC2 instance per job; the runner

self-registers with an exact-match label set from scale-config.yml, runs the job,

then is torn down. Owned by test-infra/terraform-aws-github-runner (the module),

test-infra:.github/scale-config.yml + lf-scale-config.yml (labels), and

pytorch-gha-infra/runners (Meta deploy). Being decommissioned; still serves

passthrough/identity labels. Depth in references/old-ali-infra.md.

• OSDC — ARC on EKS (current default). Actions Runner Controller schedules the

runner as a POD on a pre-existing EKS nodepool — no per-job VM boot. Owned by the

osdc repo: clusters.yaml sets each cluster's runner_name_prefix, runner defs live

in modules/arc-runners*/defs/. Essentially every routed job lands here now.

The prefix tells you the infra. Dot/bare prefixes = old ALI EC2; dash prefixes =

OSDC ARC. The dot->dash shift *is* the ALI->OSDC migration (completed in pytorch/pytorch

PR #189219, which also removed the arc experiment and the old use-arc output).

Master prefix -> infrastructure map

| Prefix in runs-on | Style | Infrastructure | Operator / funding | Defined / managed in |

|---|---|---|---|---|

| mt- | dash | OSDC — ARC on EKS | Meta | osdc clusters.yaml meta-prod-aws-uw1/ue1/ue2; defs modules/arc-runners*/defs/. Determinator default (META_LABEL_PREFIX); also the error fallback. |

| c-mt- | dash | OSDC — ARC on EKS | Meta (staging/canary) | osdc meta-staging-aws-*; META_CANARY_LABEL_PREFIX, emitted only on the pytorch/pytorch-canary repo. |

| lf- | dash | OSDC — ARC on EKS | Linux Foundation | osdc clusters.yaml lf-prod-aws-ue1/ue2; LF_LABEL_PREFIX, emitted when the lf experiment is on. Same github.com/pytorch org — distinguished only by the scale-set-name prefix. |

| amd-do- | dash | Partner ROCm (AMD dedicated) | AMD | separate amd-do-label-type output; identity-mapped in arc.yaml. |

| linux. and other bare | bare | Old ALI — EC2 | Meta (gh-ci, AWS_PROFILE=fbossci) | test-infra:.github/scale-config.yml, deployed by pytorch-gha-infra. Legacy — determinator no longer emits it. map_ec2_to_arc.py translates ordinary bare linux.* -> mt-… (OSDC ARC); only arc.yaml identity/passthrough entries stay bare and land on literal old-infra/partner runners. |

| lf. | dot | Old ALI — EC2 | Linux Foundation (parallel deploy) | test-infra:.github/lf-scale-config.yml. Dead in the determinator. |

| c. / lf.c. | dot | Old ALI — EC2 canary | Meta / LF | test-infra generated canary-scale-config.yml / lf-canary-scale-config.yml (prefix-substituted from scale-config.yml by validate_scale_config.py --generate). |

| identity passthrough (linux.rocm.gpu.*, linux.idc.xpu, linux.client.xpu, linux.google.tpuv7x.1, linux.dgx.b200.8) | bare | Literal self-hosted (old-infra / partner HW) | arc.yaml identity entries — no prefix; land on a runner registering that exact string. |

Mental Model: Bare Label + Fleet Prefix + Unconditional Translation

For the Linux build/test path, the runs-on: value is assembled at runtime from:

• Bare EC2-style label in the workflow YAML (linux.c7i.2xlarge, linux.aws.h100)

— what a human writes in the test-matrix.

• Fleet prefix from the determinator's label-type output: mt- (default, Meta

OSDC), lf- (LF OSDC, when the lf experiment is on), or c-mt- (canary repo).

amd-do- is exposed *separately* via amd-do-label-type and wired per-job.

• arc.yaml translation via map_ec2_to_arc.py — runs unconditionally on every

test-matrix, rewriting each EC2 label to its ARC equivalent (or leaving identity-

passthrough labels bare), and forcing mt- for meta_only_runners. There is no

use-arc toggle anymore; ARC is the destination for everything the determinator routes.

Examples: linux.c7i.2xlarge -> mt-l-x86iavx512-8-64 (default) or

lf-l-x86iavx512-8-64 (lf on). linux.aws.h100 -> mt-l-x86iamx-22-225-h100

regardless of prefix (H100/B200 pinned to Meta via meta_only_runners, see

Meta-Only Override for H100/B200).

linux.dgx.b200.8 stays linux.dgx.b200.8 (identity passthrough — old runner until

OSDC has capacity).

The 5 Files That Define Runner Routing

| File | Purpose |

|------|---------|

| .github/workflows/_runner-determinator.yml | Reusable workflow. Caller passes check_experiments / opt_out_experiments. Emits label-type (the fleet prefix), amd-do-label-type, runner-config, runner-type, runner-label, ci-docker-hash. |

| .github/scripts/runner_determinator.py | The script. Fetches the rollout config from pytorch/test-infra#5132 (first comment), evaluates per-user opt-in/out + per-workflow allowlist + rollout %, and emits the fleet prefix. |

| .github/scripts/test_runner_determinator.py | Tests for the determinator. Run on PR changes to the script. |

| .github/arc.yaml | EC2 -> ARC label mapping (runner_mapping) AND meta_only_runners override list for H100/B200. |

| .github/scripts/map_ec2_to_arc.py + test_map_ec2_to_arc.py | Script that rewrites a test-matrix's runner: field from EC2 labels to ARC labels using arc.yaml. Invoked unconditionally from _linux-build.yml. |

The Two Reusable Workflows That Consume the Determinator Output

| File | Role |

|------|------|

| .github/workflows/_linux-build.yml | Build job. runs-on: ${{ inputs.runner_prefix }}${{ startsWith(inputs.runner, 'l-') && inputs.runner || contains(inputs.runner, 'arm64') && 'l-arm64g4-16-62' || 'l-x86iavx512-8-64' }} — an inline ternary: the build lands on a fixed small ARC CPU (x86 AVX-512 8/64, or arm64 g4 16/62), unless inputs.runner is already an l--prefixed ARC label (then it passes through). Then it runs map_ec2_to_arc.py --prefix "${RUNNER_PREFIX}" over the test-matrix (via uv run so pyyaml is present even in build images that lack it) and re-emits it as the build job's test-matrix output. That translated matrix is what the test job consumes. |

| .github/workflows/_linux-test.yml | Test job. A single job — runs-on: ${{ matrix.runner }} from the already-translated test-matrix, always in a container with options: "--gpus all", always assuming ARC IAM role arn:aws:iam::308535385114:role/arc. There is no longer a test vs test-osdc branch and no use-arc gate; the ARC path is unconditional. (setup-linux is called with a hardcoded use-arc: true input — that is an action input meaning "we are on ARC", unrelated to the deleted determinator output.) |

Sibling workflows exist for other platforms (_mac-build.yml, _mac-test.yml,

_win-build.yml, _win-test.yml, _rocm-test.yml, _xpu-test.yml, _vllm-build.yml,

_vllm-benchmark.yml, _linux-test-stable-fa3.yml). Not all participate in the

determinator/ARC flow — Linux build + test are the primary path.

Caller Pattern (How a Workflow Plugs Into the Determinator)

Every workflow that wants experiment-driven runner placement has a get-label-type

job at the top calling _runner-determinator.yml, then passes its outputs into the

build/test jobs:

jobs:
  get-label-type:
    name: get-label-type
    uses: pytorch/pytorch/.github/workflows/_runner-determinator.yml@main
    with:
      triggering_actor: ${{ github.triggering_actor }}
      issue_owner: ${{ github.event.pull_request.user.login || github.event.issue.user.login }}
      curr_branch: ${{ github.head_ref || github.ref_name }}
      curr_ref_type: ${{ github.ref_type }}
      check_experiments: lf       # non-default experiments to consider for THIS workflow (optional)
      # opt_out_experiments: lf   # experiments to explicitly skip for THIS workflow (optional)

  some-build:
    uses: ./.github/workflows/_linux-build.yml
    needs: get-label-type
    with:
      runner_prefix: "${{ needs.get-label-type.outputs.label-type }}"   # fleet prefix
      ci-docker-hash: ${{ needs.get-label-type.outputs.ci-docker-hash }} # image tag suffix
      runner: linux.c7i.2xlarge                                          # base build label
      test-matrix: |
        { include: [
          { config: "default", shard: 1, num_shards: 1, runner: "linux.aws.h100" },
        ]}

The default fleet prefix is mt- (Meta OSDC) — the determinator never emits an empty

prefix. The build translates the test-matrix on every run and re-emits it; the test job

consumes needs.<build>.outputs.test-matrix, so matrix.runner is already the

translated ARC label. There is no use-arc: input to forward.

Experiment System

Config Source

The rollout config lives in the first comment of GitHub issue

pytorch/test-infra#5132. The script fetches it at runtime via the GitHub API.

The issue body has two ----separated sections:

• Settings YAML — defines available experiments and their rollout %.
• User opt-in list — @username,experiment[:percent],-experiment_to_opt_out.

The issue number is overridable via the issue_number input on

_runner-determinator.yml (default "5132").

The live experiments are lf (switches the fleet to Linux Foundation OSDC) and

amd-do (routes ROCm mi350 tests to AMD's dedicated runners, exposed via the

separate amd-do-label-type output). Any other experiment name is still parsed, but if

it is neither lf nor amd-do the determinator logs "enabled but no longer affects the

runner label prefix; ignoring" and it changes nothing — there is no arc experiment

anymore.

Experiment Settings

experiments:
  lf:
    rollout_perc: 25
    all_branches: false
    default: true

Field-name caveat: the working field is rollout_perc (the Experiment NamedTuple

field). The docstring example inside runner_determinator.py writes rollout_percent —

a typo. parse_settings_from_text logs any unknown key as an "Unexpected setting" and

never applies it, so a config copied from that docstring gets rollout_perc = 0 (no

percentage rollout).

Per-experiment fields (Experiment NamedTuple in runner_determinator.py):

• rollout_perc: float — % of workflows that get this experiment when no user opted in.
• all_branches: bool — if False, exception branches (main, nightly, release,

landchecks) skip the experiment.

• default: bool — if False, the experiment only runs when the caller passes it in

check_experiments.

• workflows: str — comma-separated allowlist of github.workflow names. "ALL" or

empty = every workflow. "-Name" prefix excludes that workflow even when "ALL" is

present. Exclusions win over inclusions.

User Opt-in / Opt-out

After the --- separator: each line @user,experiment1,experiment2:N,-experiment3:

• Plain entry -> opt-in 100%.
• experiment:N -> per-user N% rollout (0-100).
• -experiment -> explicit opt-out.
• #@user,... -> user opts out of ALL experiments.
• Triggering actor and PR author are both checked (the minimum per-user % among

opted-in requestors wins, so a conservative author % is respected).

Per-Workflow Opt-In / Opt-Out (in the workflow YAML)

| Input | Effect |

|-------|--------|

| check_experiments: lf | Only consider these experiments (overrides default: true for non-listed ones — they will NOT run unless listed). |

| check_experiments unset | Use each experiment's default: flag from the config (so lf is eligible by default). |

| opt_out_experiments: lf | Explicitly skip the named experiment regardless of default: / check_experiments. Higher priority than check_experiments. |

PR-Level Kill Switch

Apply the no-runner-experiments label on a PR -> the determinator adds lf to the

opt-outs, so the run stays on the default Meta fleet (mt-) rather than LF. See

OPT_OUT_LABEL in runner_determinator.py.

Fleet Prefixes the Determinator Emits

| Condition | label-type output | Fleet / infra | Example test label (linux.c7i.2xlarge) |

|---|---|---|---|

| Default (no lf) | mt- (META_LABEL_PREFIX) | Meta OSDC ARC | mt-l-x86iavx512-8-64 |

| lf experiment on | lf- (LF_LABEL_PREFIX) | LF OSDC ARC | lf-l-x86iavx512-8-64 |

| Canary repo (pytorch/pytorch-canary), no lf | c-mt- (META_CANARY_LABEL_PREFIX) | Meta OSDC ARC (staging) | c-mt-l-x86iavx512-8-64 |

| Determinator error / timeout | mt- (fallback) | Meta OSDC ARC | — |

| PR has no-runner-experiments | mt- (opts out of lf) | Meta OSDC ARC | — |

Notes:

• The determinator can NEVER emit "" (bare/EC2) or lf. (dotted). Those are

old-ALI prefixes — see references/old-ali-infra.md.

• lf takes precedence over canary: on the canary repo with lf on, you get lf-,

not c-mt-.

• amd-do is exposed via its own amd-do-label-type output ("amd-do-" when enabled,

else ""), NOT folded into label-type. Consumers wire it per-job.

EC2 -> ARC Label Translation

The translation happens in the build job, unconditionally, so the test job receives

ARC fleet labels instead of EC2 labels:

• _linux-build.yml runs map_ec2_to_arc.py --prefix "${RUNNER_PREFIX}" "${FILTERED_TEST_MATRIX}".
• The script:
• Loads runner_mapping and meta_only_runners from .github/arc.yaml.
• For each include[].runner: strips the prefix, looks the bare label up in

runner_mapping, then re-prepends the prefix (with the passthrough and meta-only

rules below).

• Drops entries whose config is in excluded_configs (currently {"onnx"} — see

Common Failure Modes).

• The translated test-matrix becomes the build job's test-matrix output.
• The test job consumes needs.<build>.outputs.test-matrix, so matrix.runner is the

already-translated ARC label.

arc.yaml — The Mapping Table

runner_mapping:
  linux.c7i.2xlarge: l-x86iavx512-8-64
  linux.12xlarge.memory: l-x86iavx512-48-384
  linux.aws.h100: l-x86iamx-22-225-h100
  linux.aws.h100.4: l-x86iamx-88-900-h100-4
  linux.aws.h100.8: l-bx86iamx-176-1800-h100-8
  linux.dgx.b200: l-x86iamx-22-225-b200
  # ... CPU, A100, H100, B200, A10G, T4, L4, ARM64
  linux.dgx.b200.8: linux.dgx.b200.8                   # passthrough — identity mapping
  linux.rocm.gpu.2: linux.rocm.gpu.2                   # passthrough
  linux.idc.xpu: linux.idc.xpu                         # passthrough

Passthrough rule (identity mapping in the table): mapped == clean means the runner

is not OSDC-managed (ROCm, XPU, TPU, and linux.dgx.b200.8 until OSDC has 8-GPU B200

capacity). The script keeps the original label WITHOUT prefixing — so it lands on the

existing self-hosted / old-infra runner.

ARC label naming convention is documented as a comment at the top of arc.yaml:

{os}-[b]{arch}{vendor}{features}-{vcpu}-{memory}[-{gpu_type}[-{gpu_count}]]

Examples: l-x86iavx512-8-64 (Linux, x86, Intel AVX-512, 8 vCPU, 64 GiB),

l-bx86iamx-176-1800-h100-8 (Linux, bare-metal, x86, Intel AMX, 176 vCPU, 1800 GiB,

8x H100).

Meta-Only Override for H100/B200

H100 and B200 hardware exists only on the Meta OSDC fleet — LF and AWS EC2 don't carry

those machines. Routing an H100/B200 job to lf-l-...-h100 would queue forever.

The fix: the meta_only_runners list in .github/arc.yaml:

meta_only_runners:
  - linux.aws.h100
  - linux.aws.h100.4
  - linux.aws.h100.8
  - linux.dgx.b200

In map_ec2_to_arc.py, the per-entry loop has an early branch for any label in this

set: entry["runner"] = "mt-" + mapped (forces Meta OSDC, overriding whatever

--prefix was passed). This decouples the H100/B200 test runner from the build runner's

fleet — the build CPU is free to land wherever the experiment routes it, but the GPU

test job is always pinned to Meta OSDC.

Important: linux.dgx.b200.8 (8-GPU B200, used by b200-distributed.yml and

b200-symm-mem.yml) is an identity passthrough in runner_mapping and is NOT in

meta_only_runners — OSDC has 8-GPU B200 runners but not enough capacity yet, so tests

stay on the existing linux.dgx.b200.8 runner while the OSDC build runs. When OSDC gains

capacity, replace the identity mapping with a real ARC label AND add the EC2 label to

meta_only_runners.

Build vs Test Runner Decoupling — Why It Matters

A single _runner-determinator.yml call typically feeds the build job's runner_prefix

(and its ci-docker-hash); the test runners come from the translated test-matrix. There

are four patterns in the wild:

• Fully coupled (default) — runner_prefix: "${{ ...label-type }}". Build CPU and

the (translatable) test runners travel together through the experiment.

• Hardcoded mt- (runner_prefix: "mt-") — pins that build job to Meta OSDC,

bypassing the determinator's fleet choice. Widely used for B200 builds, all ROCm

(rocm-*, periodic-rocm-*), XPU (xpu.yml), and several inductor perf builds.

• Per-workflow opt-out (opt_out_experiments: lf) — removes lf from

consideration for that workflow. Used across the inductor family

(inductor.yml, inductor-unittest.yml, inductor-periodic.yml,

inductor-nightly.yml, inductor-perf-test-*), dynamo-unittest.yml, etc.

• meta_only_runners — targets a specific runner *label* (H100/B200) rather than a

whole workflow. The cleanest tool when "the build can land anywhere but this specific

test runner must always be Meta OSDC".

GPU Workflows: Inventory and Mechanics

Workflows touching high-end NVIDIA GPUs (H100, B200, A100). check_experiments /

opt_out_experiments are re-derived from live source. (none) = the get-label-type

job passes no experiment inputs, so defaults apply (lf eligible unless opted out).

| File | GPU test runner(s) | check_experiments | opt_out_experiments | Notes |

|------|--------------------|---------------------|-----------------------|-------|

| test-h100.yml | linux.aws.h100 | lf | — | H100 forced mt- via meta_only_runners |

| h100-cutlass-backend.yml | linux.aws.h100 | lf | — | |

| h100-distributed.yml | linux.aws.h100.8 | lf | — | |

| h100-symm-mem.yml | linux.aws.h100.4 | lf | — | |

| test-b200.yml | linux.dgx.b200 | lf | — | B200 forced mt- via meta_only_runners |

| b200-distributed.yml | linux.dgx.b200.8 | (none) | — | build pinned runner_prefix: "mt-"; get-label-type used only for ci-docker-hash; b200.8 test = identity passthrough -> old bare runner |

| b200-symm-mem.yml | linux.dgx.b200.8 | (none) | — | same as b200-distributed.yml |

| inductor-perf-test-nightly-h100.yml | linux.aws.h100 | (none) | lf | |

| inductor-perf-test-b200.yml | linux.dgx.b200 | (none) | lf | build pinned runner_prefix: "mt-" |

| inductor-pallas.yml | linux.aws.h100 | (none) | — | |

| inductor-periodic.yml | linux.aws.h100 + many g5/a100 | amd-do | lf | |

| attention_op_microbenchmark.yml | linux.aws.a100, linux.aws.h100, linux.dgx.b200 | lf | — | B200 build path pins runner_prefix: "mt-" |

| operator_microbenchmark.yml | linux.aws.h100, linux.aws.a100, linux.dgx.b200 | lf | — | B200 build path pins runner_prefix: "mt-" |

| operator_microbenchmark_compare.yml | conditional H100/A100/B200 via ${{ inputs.gpu }} | (none) | — | B200 path pins runner_prefix: "mt-" |

| vllm-benchmark.yml | external matrix gen | NO get-label-type job | — | does not participate in the determinator |

Common Runner Label Families

CPU build runners (translated to the l-x86i* / l-arm64g* ARC families)

linux.c7i.2xlarge, linux.4xlarge, linux.12xlarge, linux.12xlarge.memory,

linux.r7i.4xlarge, linux.24xlarge.memory, linux.24xl.spr-metal, *.amx, *.avx2,

*.amd. The build job itself always lands on a fixed small ARC CPU (see

_linux-build.yml); these labels matter for the *test* matrix.

NVIDIA GPU test runners

| Label | GPU |

|-------|-----|

| linux.g4dn.4xlarge.nvidia.gpu, linux.g4dn.12xlarge.nvidia.gpu, linux.g4dn.metal.nvidia.gpu | T4 |

| linux.g5.4xlarge.nvidia.gpu, linux.g5.12xlarge.nvidia.gpu, linux.g5.48xlarge.nvidia.gpu | A10G |

| linux.g6.4xlarge.experimental.nvidia.gpu, linux.g6.12xlarge.nvidia.gpu | L4 |

| linux.aws.a100 | A100 (p4de) |

| linux.aws.h100, linux.aws.h100.4, linux.aws.h100.8 | H100 (p5) — Meta-only |

| linux.dgx.b200 | B200 (p6) — Meta-only |

| linux.dgx.b200.8 | B200 8-GPU — identity passthrough (old runner) |

ARM64

linux.arm64.2xlarge, linux.arm64.m7g.4xlarge, linux.arm64.m8g.4xlarge,

linux.arm64.r7g.12xlarge.memory, linux.arm64.m7g.metal, linux.arm64.m8g.metal-24xl.

Partner hardware (identity passthrough — never prefixed/translated)

linux.rocm.gpu.2, linux.rocm.gpu.mi210.1/.2, linux.rocm.gpu.gfx942.1/.4,

linux.rocm.gpu.gfx950.1/.2, linux.rocm.gpu.gfx1100, linux.idc.xpu,

linux.client.xpu, linux.google.tpuv7x.1. The amd-do experiment routes mi350 tests

through the amd-do--prefixed identity entries (amd-do-linux.rocm.gpu.gfx950.1/.2).

_runner-determinator.yml Outputs (Full Reference)

| Output | What it is | Typical consumer |

|--------|-----------|-----------------|

| label-type | The fleet prefix ("mt-", "c-mt-", "lf-") | runner_prefix: input on _linux-build.yml |

| amd-do-label-type | "amd-do-" if the amd-do experiment is enabled, else "" | Per-job AMD pinning |

| ci-docker-hash | git rev-parse HEAD:.ci/docker (the tree hash, from PR head) | ci-docker-hash: input on _linux-build.yml for the image tag |

| runner-config | Normalized runner config (m7g / m8g) | ARM64 metal runner selection |

| runner-type | Runner suffix (metal / metal-24xl) | ARM64 metal runner selection |

| runner-label | Fully qualified ARM label linux.arm64.<config>.<type> | Direct runs-on: on some ARM workflows |

There is no use-arc output (removed with the arc experiment in PR #189219).

Decision Trees / Recipes

"I want this workflow to never use LF runners"

Add opt_out_experiments: lf to the _runner-determinator.yml caller. The prefix will

never be lf- — it stays mt-. Examples: the whole inductor family

(inductor.yml, inductor-unittest.yml, inductor-periodic.yml, inductor-nightly.yml,

inductor-perf-test-*), dynamo-unittest.yml.

"I want this specific GPU test runner to always go to Meta OSDC, but let the build follow the experiment"

Add the EC2 label to meta_only_runners in .github/arc.yaml. No workflow changes

needed. Already done for linux.aws.h100, linux.aws.h100.4, linux.aws.h100.8,

linux.dgx.b200.

"I want this build job to always go to Meta OSDC (skip the determinator's fleet choice)"

Hardcode runner_prefix: "mt-" on the build job. Skips the fleet experiment for that

job. Widely used for B200 builds, all ROCm, XPU, and some inductor perf builds.

"I want to know what infra + label my job will actually run on"

Trace through:

• Read the workflow YAML — find the get-label-type job and the build/test jobs.
• Note check_experiments / opt_out_experiments.
• Pull the current test-infra #5132 config (first comment) to see what's rolling.
• Determine the fleet prefix: mt- unless lf is enabled (lf-) or it's the canary

repo (c-mt-).

• Apply map_ec2_to_arc.py (it runs on EVERY matrix): EC2 label -> ARC label with the

prefix; identity-passthrough labels stay bare (old infra); meta_only_runners force

mt-.

• Dash prefix -> OSDC ARC; bare/dotted -> old ALI EC2 (see references/old-ali-infra.md).

"I added a new EC2 runner type — what do I need to update?"

Add the EC2 -> ARC mapping to runner_mapping in .github/arc.yaml. If it's a

passthrough (not OSDC-managed), add an identity mapping (linux.foo: linux.foo).

Otherwise map_ec2_to_arc.py errors with error: no ARC runner found for '<label>' —

the lookup is strict, there is no implicit passthrough. If it's a new H100/B200 variant,

also add the EC2 label to meta_only_runners.

"How do I add a new experiment?"

Define it in the test-infra #5132 issue body. No code change to runner_determinator.py

is needed UNLESS the experiment must change the label prefix — the script only special-

cases lf (sets the fleet) and amd-do (exposed via amd-do-label-type). Any other

experiment is parsed but "no longer affects the runner label prefix" and does nothing to

routing. Workflows opt in via check_experiments: <name>.

Common Failure Modes

| Symptom | Likely cause |

|---------|-------------|

| Job stuck queued on lf-l-...-h100 or lf-l-...-b200 | H100/B200 not in meta_only_runners. Add it. |

| map_ec2_to_arc.py error error: no ARC runner found for '<label>' | Missing entry in runner_mapping. Add a mapping (or an identity passthrough if not OSDC-managed). |

| GPU test lands on a bare/old label instead of mt-l-... | The bare label is an identity passthrough in runner_mapping (e.g. linux.dgx.b200.8) — intentional until OSDC capacity lands. To move it to OSDC, replace the identity mapping and add a meta_only_runners entry. |

| Job lands on the default Meta fleet despite expecting lf- | User opted out via #@user in #5132, PR has no-runner-experiments (opts out of lf), lf is below rollout %, or the workflow sets opt_out_experiments: lf. Set the user to 100% opt-in to debug. |

| New runner added to a test-matrix breaks the build's map step | Missing from arc.yaml. The per-entry lookup is strict — no implicit passthrough for unknown labels. |

| Onnx tests silently dropped on the ARC path | Intentional. map_ec2_to_arc.py excludes config: onnx because onnxruntime's hardware_concurrency() sees all host CPUs on ARC k8s instead of the container cpuset. See the comment in the script. |

| Determinator script fails or times out | The wrapper falls back to Meta runners (mt-) + no experiments. Look in the get-label-type step logs for the actual error. |

| check_experiments typo / unknown experiment name | Silently ignored. Check spelling against the keys in the #5132 settings YAML (lf, amd-do). |

| A stale determinator checked out from a PR emits lf. | _runner-determinator.yml checks the script out from the merge commit (base), not PR head, precisely so a stale prefix like lf. can't reach main's build job (which has no EC2 fallback). See the checkout comment citing #189113/#189171. |

Things to NEVER Do

• NEVER invent a new ARC label format — use the existing

{os}-[b]{arch}{vendor}{features}-{vcpu}-{memory}[-{gpu_type}[-{gpu_count}]] scheme in

arc.yaml. The convention is reviewed across CI infra.

• NEVER add an EC2 label to meta_only_runners without first confirming it has a

runner_mapping entry. The script will error.

• NEVER hardcode an ARC label (e.g. l-x86iamx-22-225-h100) in a workflow's

test-matrix runner: field. Use the EC2 label and let map_ec2_to_arc.py translate

it — the script only recognizes EC2/bare labels as input.

• NEVER add the lf- / mt- prefix directly to runner_mapping entry values. The

prefix is added separately by the script.

• NEVER make the determinator emit a bare ("") or dotted (lf.) prefix — main's

build/test path has no EC2 fallback, so an old-style prefix breaks routing (see

189113/#189171).

• NEVER edit .ci/docker/ to change runner behavior — that's a Docker image rebuild

trigger, unrelated to runner routing.

Key Cross-References

• pytorch/test-infra#5132 — first comment is the live rollout config. Edit there to

roll experiments, not in code.

• references/old-ali-infra.md — the legacy Lambda-autoscaled EC2 world: scale-config,

dot prefixes, ownership, and how to recognize an old-ALI job today.

• PR #189219 — removed the arc experiment and the use-arc output; completed the

dot->dash (ALI->OSDC) migration. Read when confused why old docs mention use-arc.

• docs/runner_naming_convention.md (in ~/meta/ci-infra/osdc/) — source of truth for

the ARC label naming scheme. The osdc-runners-nodepools skill covers the runner

definition side (what runners exist, what hardware they map to).

• ~/meta/ci-infra/osdc/modules/arc-runners*/defs/*.yaml — the actual ARC runner scale

set definitions. Each runner.name there is what appears as the mapping *target* in

arc.yaml; runnerScaleSetName = {prefix}{runner.name} is the literal runs-on.

Verification Commands

# See the current test-infra #5132 config
gh issue view 5132 --repo pytorch/test-infra --json comments --jq '.comments[0].body'

# Run the determinator locally (read-only, hits the GitHub API)
cd ~/meta/pytorch
python .github/scripts/runner_determinator.py --help

# Run the ARC mapping for a test-matrix (matches the _linux-build.yml uv invocation)
cd ~/meta/pytorch/.github/scripts
uv run --no-project --with pyyaml==6.0.2 \
  python map_ec2_to_arc.py --prefix "lf-" '{ include: [ { config: "x", runner: "linux.aws.h100" } ] }'
# Expected: runner becomes "mt-l-x86iamx-22-225-h100" (meta_only override forces mt-)

# Run the test suites
python .github/scripts/test_runner_determinator.py
python .github/scripts/test_map_ec2_to_arc.py