Sandbox Bench

Sandbox bench: paired A/B perf runs for React and Next.js changes

Measures what a change is actually worth, end to end: two revisions

("arms") built into otherwise-identical Next.js apps, exercised by the

bench/render-pipeline harness on Vercel Sandbox VMs, compared with

paired statistics that treat the VM boot as the unit of replication.

All heavy work happens on sandbox VMs; the laptop only orchestrates.

Scripts live in scripts/ next to this file and run from anywhere.

Arms are git refs, resolved in cached clones of react and next.js;

the Next side defaults to canary. Everything is cached

content-addressed: first use of a new pair builds caches (~45-60 min

extra, once); later runs boot straight into measurement.

One-time setup

• node scripts/config.mjs show — if it reports NOT CONFIGURED, ask

the user which Vercel team and project the sandbox VMs

should run under (these are billed resources; never guess, never

default), then node scripts/config.mjs set team=<slug> project=<name>.

Config lives in ~/.config/sandbox-bench/config.json — never commit

team/project names into the repo.

• The Vercel CLI session must have access to that team. On a 403,

stop launching (don't retry through it) and check whether access is

already back: vercel whoami --scope <team-slug> plus one scoped

read call (e.g. vercel sandbox ls) — grants drop and recover on

their own, and a transient 403 needs no login at all. If

verification still fails, run vercel login <team-slug> yourself

as a background task (the token lives with the CLI session, not

with the user). It opens a browser/device confirmation — relay the

URL if one is printed — but keep re-running the verification pair

every minute or two while it waits: access often returns before

the login flow reports success, and once verification passes, kill

the pending login and resume. After a 403 outage, expect in-flight runs to have died:

run node scripts/bench-status.mjs and follow its recovery

actions (measurement VMs will have hit their ~5h timeout if the

outage was long — those cells need relaunching, not collecting).

• react and next.js clones land in the cache on first use (or point

reactRepo/nextRepo in the config at existing checkouts).

Before the first real run with a new configuration, sanity-check the

plan with --dry-run (prints what would happen, touches nothing).

Workflow

1. Resolve what's being compared

• React PR: --pr <url|number> — base is computed automatically

(merge-base of the PR head with react main).

• React refs: --arms base=<ref>,cand=<ref> — base FIRST. For a

multi-commit branch, base is the merge-base with main, not cand^.

• Next.js PR: --next-pr <url|number>. The React side defaults to

whatever each Next ref vendors (that's what would ship); pass

--react-ref only to pin both arms to one specific React build.

• Next refs: --next-arms base=<ref>,cand=<ref>.

Exactly one side varies; the other is identical in both arms. That

isolation is what makes the numbers attributable — never vary both.

2. Gate correctness before spending bench compute

A bench number from an arm that fails its own tests is meaningless.

For any arm that is not already CI-green upstream (hand-assembled

branches, cherry-picks with resolved conflicts, local commits):

node scripts/sandbox-gate.mjs --arms cand=<ref>

The bench itself enforces the primary gate: every react arm's commit

must have green CI on the react repo, checked automatically before any

build or VM is spent. PRs and main-history commits normally satisfy

this with no extra work. For local or unpushed refs (no CI exists),

gate on a VM with sandbox-gate.mjs and then pass --allow-ungated to

the bench. The VM gate runs the full test suite in prod mode (the

channel that gets benched). PASS requires seeing the actual test

counts in the output. If a gate fails, report the failures and stop —

do not bench a broken arm. Each arm is gated in its own

lockfile's environment. Bench the exact sha the gate prints (a branch

ref can move between gate and bench).

3. Launch the bench (background, non-blocking)

bash -c 'node scripts/sandbox-e2e.mjs --pr <url> --label <slug> \
  2>&1 | grep --line-buffered -v "^live "; exit ${PIPESTATUS[0]}'

For React PRs, launch BOTH suites (separate background tasks; they

share arm builds and caches):

bash -c 'node scripts/sandbox-ssr.mjs --pr <url> --label <slug>-ssr \
  2>&1 | grep --line-buffered -v "^live "; exit ${PIPESTATUS[0]}'

The e2e suite measures the Node path through a real Next.js app; the

ssr suite measures the react repo's flight-ssr-bench fixture — 8

variants (Fizz and Flight+Fizz, Node and Edge web streams, sync and

async), each sequentially with Flight script injection and behind an

HTTP server at c=1/c=10. Edge cells are the ssr suite's headline (the

e2e suite cannot see that path); its Node and Fizz-only cells

attribute an effect to the Flight layer, the Fizz layer, or the

stream plumbing. The fixture (the workload) is pinned to one ref for

both arms — react main by default — so only the React builds differ;

if the PR itself edits the fixture, the launcher says so and the run

does not measure those edits. Next PRs run the e2e suite only.

• Run it as a background task and proceed on its completion

notification. Never hold a foreground wait; never poll in a loop —

the rule is about control flow, not status relay: reading the

output tail to answer "how's it going" is always fine.

• The harness handles the invariants internally: both arms in the same

VM, interleaved ABBA, paired per (vm, run); detached remote

execution (transport drops don't kill runs); build fingerprints

recorded in every result row.

• live ... lines are streaming estimates for progress display only.

Never stop a run early because a live p-value looks good, and never

report a live number — sequential peeking manufactures false

positives. Only the final analysis counts.

• Defaults (16 VMs × 2 paired runs) implement the methodology; don't

reduce VM count to save time — boots are the unit of inference, and

fewer boots means wider intervals, not faster answers.

• Sandbox compute is internal capacity, not a budget: launch, relaunch,

and confirm runs without asking about cost or shrinking them to

save it.

• The Next side's default, canary, is the latest published canary

release (the launcher prints its version and sha), so repeat benches

reuse the built snapshot until a new canary ships.

• Useful flags: --bench-env KEY=VALUE (runtime-only env for the

bench process — it does NOT affect the snapshot's app build), --isolate-routes

(tail investigations), --no-profile (skip the CPU capture that

runs by default after the timed runs), --prepare (build caches

only — use when two cells will share an arm, to avoid duplicate

builds racing).

• CPU profiles are captured by default: one profile pass per arm runs

strictly AFTER the timed runs (it cannot touch the numbers), costs

~45-60 min extra VM wall-clock, and lands in <runDir>/prof-vm<N>/

as standard V8 .cpuprofile files. Cross-VM profile diffs are

highly stable (observed 16/16 sign agreement on real movers), so one

profiled cell suffices to rank hot paths. Analysis caveats:

aggregate by (functionName, line, column) — bare minified names

collide across the bundle — and never diff arms by minified name

(the minifier renames between builds); match positions or code

snippets instead.

• The bench exercises Next's node-streams path

(__NEXT_USE_NODE_STREAMS is inlined as true for the node runtime

at build time). React changes that only touch the EDGE stream

configs are not exercised end-to-end and will (correctly) bench as

no detected difference.

4. Read the result like a skeptical data scientist

The goal is the truth about the change, not making its author feel

good. The final analysis prints, per route/phase/metric, the

boot-level mean, ±95% CI, and p across boots. Apply the policy in

references/methodology.md:

• Claim only boot-level p < 0.01, with the CI, on an A/A-validated

team/config (see methodology).

• The PR is a hypothesis, not an explanation. Claims come from the

analysis output alone. When the numbers agree with the PR's story,

check whether the captured data actually discriminates that

mechanism from alternatives — a latency win attributed to smaller

payloads should come with a document-bytes delta; if the bytes

didn't move, the story doesn't hold and the report says so.

• Use every captured metric, and voice anything that does not add up:

one metric family moving against the others, effects with no

byte-level or RSS trace, throughput moving without latency,

sign flips across boots. An inconsistency you cannot explain

belongs in the report, not in the drawer.

• The within-run p shown in brackets is a diagnostic, never a claim.
• Check the fingerprint header first: two distinct fingerprints = valid

A/B; "inconsistent fingerprints" = invalid, report no numbers. The

fingerprint hashes both bundlers' compiled server files — arms

touching only client files can still legitimately show identical

fingerprints with different version strings.

• Per-boot values are printed; if boots disagree in sign, say so.
• Any claim that will drive a decision gets one independent

confirmation run before it's stated as fact.

Re-analyze any past run without re-running it:

node scripts/bench-analyze.mjs <runDir>.

5. Report

Name what was measured with links: the PR title (printed in the

analysis header, stored in meta.json) linking to the PR; for ref

arms, the commit title. Lead with a table of the significant cells,

each row carrying the effect with its unit, the CI, and p:

## [<PR title>](<PR url>) — e2e, Vercel Sandbox (x86 Xeon), <n> boots

Significant (boot-level p < 0.01, A/A-validated):
| cell | effect | 95% CI | p |
|---|---|---|---|
| /dashboard under load | +14.4% throughput (req/s) | ±3.2% | <0.0001 |
| /dashboard serial | −10.7% median latency (ms) | ±0.6% | <0.0001 |

No detected difference: <every cell not in the table, by name>.
Flags: <cells at 0.01 ≤ p < 0.05, sign disagreements across boots,
fingerprint caveats, anything that does not add up>

One row per cell: rps and median restate each other, so report the

throughput number (add a p95 row only when the tail moves differently

from the median). Document metrics (raw/gzip/Flight KB) get their own

rows when they differ — they are the mechanism evidence. When the

Next side predates the document-metrics harness (vercel/next.js#95828)

those cells are absent; say so instead of silently reporting less. State the platform next to the numbers. Magnitudes

are platform-dependent (GC share differs by CPU); direction and

mechanism transfer, percentages do not. Never present a

noise-compatible delta as a small win or loss — it is "no detected

difference".

Results database

Every collected run lands in one SQLite file,

~/.cache/sandbox-bench/results.db — raw measurements and artifacts

(CPU profiles, logs) only, written exclusively by the importer, never

by hand. The launcher imports and verifies automatically at

collection; bench-analyze reads the db and nothing else, so every

statistic is a pure function of it. Numbers in reports come from the

analysis output verbatim — never retype, recompute, or aggregate them

yourself.

• node scripts/bench-db.mjs ls — all runs with sample/artifact counts.
• node scripts/bench-db.mjs verify [runId] — integrity checks:

sqlite-level, referential, one fingerprint per arm, paired sample

counts, artifact sha256. Run it before drawing on old data.

• node scripts/bench-db.mjs export out.db <runId...> — cut a

self-contained db of specific runs (with their profiles) to send to

someone. It opens in any SQLite tool.

• node scripts/bench-analyze.mjs <runId> — re-analyze anything in

the db; a run-dir argument imports it first.

Keeping the user informed

The launcher narrates itself on stdout: launch facts first (run dir,

arms, CI verdicts), then a progress line every ~2 minutes with rows

collected and interim per-route effects with confidence. Relay to the

user: the run dir and expected duration right after launching,

notable interim shifts if they ask how it's going, and the full

verdict from the final analysis when the completion notification

arrives. The analysis names metrics that were `not captured on this

run` — repeat that in the verdict when it limits what the data can

say (document metrics absent means the payload mechanism is

unverified, not verified-identical).

While a run is active, open any reply with a one-line status per run:

read the tail of the launcher's output and quote its latest progress

line. If the session supports timed wakeups or reminders, schedule a

check at each expected transition (arm builds -> experiment snapshot

-> measuring, then every ~15 minutes of measurement) and post the

progress line; if not, say when the next update will arrive so

silence is never ambiguous. Interim effects in progress lines are

streaming estimates — share them as progress, never as claims.

If a launcher process dies (session teardown, crash), the remote VMs

keep executing their measurement loops — the data is not lost. `node

scripts/bench-collect.mjs <runDir>` reconnects, waits for the loops,

downloads the results, cleans up, and analyzes. Run it before the VMs

hit their ~5h timeout.

Failure recovery

• First move, always: node scripts/bench-status.mjs. Session

restarts silently kill background launchers while their detached VMs

keep measuring, and a dead launcher's log still ends with a

healthy-looking progress line — never infer liveness from log tails

or task output files. bench-status checks each run's recorded

launcher pid and prints the per-run recovery action (running /

collect now / relaunch). Run it at the start of any session that

expects work in flight, after any crash, and before telling the user

what is or isn't running. Launcher crashes are also recorded in the

run's status.json (phase: "failed" plus the error).

• Interrupted local process: remote VMs keep running detached.

vercel sandbox list (with the configured team/project) to find

them; poll each VM's /vercel/sandbox/loop.done, cp its

results.jsonl down when done, then remove the VM and analyze with

bench-analyze.mjs.

• Leaked VMs after any crash: node scripts/sandbox-sweep.mjs

lists this skill's VMs (matched by sbench-\* name AND the

purpose=sandbox-bench tag, and only when older than --min-age-hours,

default 3, so healthy in-flight runs are never touched); --yes

removes them by exact listed name.

• Flaky uploads/transports: the harnesses size-check artifacts and

abort on truncation. A failed cell is safe to relaunch; caches make

the retry cheap. Don't relaunch two cells that need the same uncached

arm at the same moment — they'll race to build it; use --prepare

first instead.

Cost expectations (set these with the user before big runs)

Per cell at defaults: ~18 VMs (8 measurement + build/snapshot VMs),

~1-2h wall-clock cold, ~30-60 min warm. A/A calibration and

confirmation runs are extra cells. VMs are billed to the configured

team — for anything beyond a single PR check, confirm scope first.