Triaging Visual Review Runs

Triaging visual review runs

Visual Review is PostHog's screenshot-regression product: CI captures storybook + playwright screenshots,

diffs them against committed baseline hashes, and gates the PR until a human approves the visible changes.

A PR with visual changes carries a visual-review GitHub status check that stays red until each diffed

snapshot is approved or tolerated in the VR UI.

This skill teaches an agent how to answer the questions a human reviewer would actually ask, by chaining

the VR MCP tools — instead of reaching for gh pr view and tab-hopping to the VR web UI. The read tools

cover status / scope / history / triage. Two are reversible DB-only triage marks (approve-create,

tolerate-create); one ships the change (finalize-create) — it commits the baseline and greens the gate,

and only that one needs explicit per-run human confirmation.

When this skill applies

Trigger this skill on any of:

• A PR number, branch name, or commit SHA paired with words like _visual review_, _VR_, _snapshot_, _screenshot_,

_storybook diff_, _playwright snapshot_, _baseline_, _approve_, _tolerated_, _quarantine_.

• Questions about why a PR is blocked, what visually changed, or whether a diff is real.
• "Is my run done?" / "What's left to review?" / "Has this story flaked recently?"
• A failing visual-review GitHub check or a PR comment from the posthog-bot mentioning visual review.

When the user asks for the rendered diff image itself, the VR web UI

is faster — direct them there. This skill is for everything around the diff: status, scope, history, triage.

Tools

Read tools (safe to call freely):

| Tool | Purpose |

| -------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |

| posthog:visual-review-runs-list | List runs, filter by pr_number / commit_sha / branch / review_state. Start here. |

| posthog:visual-review-runs-retrieve | Full detail for a single run (status, summary counts, supersession). |

| posthog:visual-review-runs-snapshots-list | Per-snapshot results inside a run: identifier, result, diff %, classification, baseline + current artifact URLs. Quarantined snapshots are excluded by default (see quarantined_count); pass include_quarantined=true to see them. |

| posthog:visual-review-runs-snapshot-history-list | A single story's last N runs across master/PRs — the flake check. |

| posthog:visual-review-runs-counts-retrieve | Aggregate counts for queue triage (how many runs in needs_review, etc.). |

| posthog:visual-review-runs-tolerated-hashes-list | Hashes the team has explicitly accepted as "known flake / acceptable variation". |

| posthog:visual-review-repos-list | Repos (one per GitHub repo) — usually only one matters; useful for filtering. |

| posthog:visual-review-repos-retrieve | Repo metadata: baseline file paths, PR-comment configuration. |

Triage tools (reversible, DB-only — they record a review decision but do NOT change the baseline or the gate):

| Tool | Purpose |

| -------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------- |

| posthog:visual-review-runs-approve-create | Mark changed / new snapshots reviewed (approved) in the DB. Does NOT commit or green the gate — ship via finalize. |

| posthog:visual-review-runs-tolerate-create | Mark a single changed snapshot as a known tolerated alternate. Does NOT change the baseline — use for benign variants. |

Ship tool (irreversible, outward-facing — requires explicit per-run human confirmation; see the gate):

| Tool | Purpose |

| -------------------------------------------- | ---------------------------------------------------------------------------------------------------------------- |

| posthog:visual-review-runs-finalize-create | Commit the approved baseline to the PR branch and green the GitHub visual-review check. This ships the change. |

Mark-reviewed call shape (approve-create):

• id (required) — the run UUID. It's the route parameter, so the call fails without it.
• snapshots: [{identifier, new_hash}] — new_hash is the content_hash of each snapshot's current_artifact. This only records the review in the DB; nothing is committed and the gate stays red until you finalize.

Toleration call shape — both fields are required:

• id (required) — the run UUID. It's the route parameter, so the call fails without it.
• snapshot_id (required) — the UUID of the individual snapshot to tolerate (from visual-review-runs-snapshots-list). This identifies _which_ snapshot inside the run; it does not replace the run id.

Finalize call shape (finalize-create) — the all-or-nothing ship action:

• id (required) — the run UUID.
• approve_all: true — approve every still-pending changed/new snapshot before finalizing (tolerated ones are left alone). Use when you've verified every remaining diff is intended.
• Omit approve_all (default false) to finalize a run you've already reviewed snapshot-by-snapshot. Finalize is all-or-nothing: it fails with 409 not_fully_resolved (and lists what's left) unless every changed/new snapshot is approved, tolerated, or quarantined.
• It commits exactly the snapshots approved in the DB — tolerated snapshots keep their baseline and are never overwritten. When a baseline commit is pushed, its SHA comes back on the run's metadata.baseline_commit_sha. It's absent when nothing needed committing (everything resolved by toleration/quarantine — the gate still greens) or when the commit was skipped: no PR, or a 409 sha_mismatch because the PR has newer commits (that one leaves the gate red — re-run CI on the latest commit and finalize again).

If finalize fails with 409 stale_run, the run has been superseded — visual-review-runs-list { pr_number } and finalize the newest one. A successful finalize often kicks off a fresh CI run, which is normal.

The finalize gate

Finalize is the one irreversible, outward-facing action in this skill: it rewrites the baseline committed to the

PR and greens the merge gate. Treat it like pushing to someone's branch — never automatic.

Before _any_ finalize-create call, all of these must hold:

• You verified the diffs. You pulled the current (and, for changed, baseline) PNGs and looked at them, ran

the flake check on anything suspect, and reached a per-snapshot verdict. Metadata alone is never enough.

• You presented the verdict and waited. Show the user, per snapshot, what changed and your recommendation, then stop.
• The user explicitly approved _this_ run. A broad "get the gate green" / "fix the PR" is permission to

investigate and recommend — NOT to finalize. When the task implies finalizing but the human hasn't said it for

this specific run, ask.

approve-create and tolerate-create are reversible triage and don't need this gate — but they don't ship anything

either. The moment you're about to finalize-create and can't point to a specific human "yes" for this run, stop and ask.

Vocabulary cheat sheet

These appear in tool output and matter for interpretation:

• Run review_state: needs_review (open, awaiting human), clean (zero diffs), processing (CI still uploading),

stale (a newer run on the same PR has superseded this one — check superseded_by_id).

• Run run_type: storybook (component snapshots) or playwright (full-page e2e snapshots).
• Snapshot result: unchanged, changed (real diff), new (no baseline yet), removed.
• Snapshot classification_reason: tolerated_hash (matches a known-tolerated hash, no action needed),

below_threshold (under the noise floor), exact (byte-identical), "" (real diff requiring review).

• Snapshot review_state: pending or approved.
• Run summary: total / changed / new / removed / unchanged / unresolved / tolerated_matched —

unresolved is what's actually blocking review.

Workflows

"What's the VR status of this PR?"

The single most common job. Map a PR number to its run state in two calls.

• posthog:visual-review-runs-list { pr_number: <n>, limit: 5 } — sort by created_at desc, take the latest non-stale one.
• If the run has summary.changed > 0 or summary.unresolved > 0, drill in:

posthog:visual-review-runs-snapshots-list { id: <run_id> } and report the changed snapshots.

Report back: PR number, run UUID, review_state, summary counts, and the _posthogUrl deep link so the

user can click straight to the diff viewer.

"Is the diff real or unrelated?"

The most useful judgment a code-aware agent can add. Combine three signals: scope match, flake history,

and the actual rendered images. The agent should look at the screenshots — not just describe metadata.

• Scope check — git diff master...HEAD --stat (or against the PR's base branch) → list of touched paths.

Cross-reference with posthog:visual-review-runs-snapshots-list { id } filtered to result: changed → story identifiers.

Stories are namespaced like <area>-<scene>--<story>--<theme>; e.g. scenes-app-settings-user--settings-user-profile--dark

maps to frontend/src/scenes/settings/user/.... Use this to translate story id → likely source path.

• Visual inspection — for each changed snapshot, the tool result contains current_artifact.download_url

and baseline_artifact.download_url. These are pre-signed S3 URLs to PNG files; pull them and look:

   curl -s -o /tmp/vr-baseline.png "<baseline_artifact.download_url>"
   curl -s -o /tmp/vr-current.png "<current_artifact.download_url>"

Then Read both files (the Read tool renders images visually) and compare. Things to call out:

• The actual visible delta (text changed, button moved, layout shift, color drift, missing element).
• Whether the change is consistent with the diff_pixel_count and diff_percentage in the metadata

(e.g. 54% diff but the images look near-identical → screenshot framing changed, not the UI).

• Whether the baseline and current have different dimensions (width / height fields). Mismatched

dimensions usually mean the story rendered to a different viewport or didn't fully render before

screenshot — a flake signal, not a regression.

• Flake history — run the flake check below for any story that looks suspect.

• Verdict — combine all three:
• Scope plausible + visible regression matches the code change → real diff, recommend approval.
• Scope mismatch + dimensions mismatch + frequent prior changes → flake, recommend tolerating the hash.
• Scope plausible + visible regression looks unintended → push a fix; do not approve.

Always include a one-line description of what you saw in the images — the user uses this to decide whether to

trust your verdict without opening the VR UI themselves.

Flake check: "Has this story been changing?"

Once you have a suspect snapshot identifier:

posthog:visual-review-runs-snapshot-history-list { id: <snapshot_id> } → returns prior outcomes for the same story.

Verdicts:

• Mostly unchanged and this run's diff is the outlier → likely a real regression caused by this PR.
• Frequent changed across unrelated branches/master → flaky story; recommend tolerating the hash via the UI.
• Recent removed or large-jump dimension change → baseline likely stale; recommend re-baselining on master.

Triaging the queue

When the user is doing housekeeping rather than asking about a specific PR:

• posthog:visual-review-runs-counts-retrieve → total queue size.
• posthog:visual-review-runs-list { review_state: needs_review, limit: 50 } (paginate if needed).
• Group by branch author or run_type to surface clusters (e.g., "12 PRs blocked on the same shared

component change" usually means a single underlying root cause to address).

• Prefer surfacing runs whose summary.changed > 0 over runs that are only new — new means no baseline

yet, which is usually trivial to approve; changed is the real review work.

Output expectations

For PR-status questions, lead with the verdict in one line, then 2-4 bullets of supporting context. Always

include the _posthogUrl deep link to the run — humans need to see the rendered images to make the call,

the agent can only describe the metadata.

For triage / aggregate questions, a short table beats prose. Group by what the user is going to act on.

What NOT to do

• Do not approve or tolerate without explicit user confirmation. The verdict is yours to recommend; the

decision to ship belongs to the user. Once they say "approve those" / "tolerate that", call the tool.

• Do not assume the failing GitHub check on a PR is unrelated to VR — if a visual-review check is red on

a PR you're working on, that's the trigger to run this skill.

• Do not declare a verdict from metadata alone when result: changed. Pull the baseline and current PNGs

and look at them; metadata can only say "something changed", not whether the change is intended.