Experiment Audit

Experiment Audit

Verify that a customer's experiment is actually collecting variant data, that downstream attribution survives the funnel, and that the metrics measure what the customer thinks they measure. Output is a Slack-ready writeup grouped by the four questions customers almost always ask.

Step 0 — confirm scope before running

The skill assumes the active PostHog MCP is scoped to the customer's project, not yours. Always start with:

> "What project am I in? List the most recent 5 experiments."

If the project name looks like your own internal project (e.g. "PostHog App + Website", id 2), STOP — the impersonation isn't routing correctly. Re-run the wizard or check /mcp auth before continuing.

Step 1 — pull the experiment config

Use experiment-list with search to find the experiment by name. Then pull the full record. Capture:

• Status (running / draft / stopped / paused) and start date.
• Linked feature flag key and ID.
• Variants and traffic split. Variant *names* must match what the customer's code reads — bucketing bugs are usually case/typo mismatches.
• Holdout, bucketing key (device_id vs user_id), and ensure_experience_continuity.
• Exposure event — default $feature_flag_called or a custom event.
• Primary and secondary metrics. Note action IDs, event names, breakdowns, conversion windows, attribution modes.
• Filter test accounts setting.

Step 2 — pull the feature flag config

For the linked flag:

• Release conditions — read each one carefully.
• Watch for two specific footguns:
• URL targeting via person property ($current_url = ...) — uses the *latest URL the person has been seen on*, not the current page. Stale by definition. Always flag as a problem.
• Exact-match on a path fragment — $current_url is captured as the full URL (https://host/path). Exact-matching /path will never hit.
• Audience filters (desktop-only, geo, cohort). Verify they use person properties or group properties — not URL.
• Rollout percentage and any super-conditions.
• ensure_experience_continuity setting on the flag (this overrides the experiment-level setting).

Step 3 — verify exposure is happening

Pull $feature_flag_called events for the flag key since the experiment start date.

• Total count. If suspiciously low for the time elapsed, dig.
• Break down by $feature_flag_response. Should split close to 50/50 between the variant names (e.g. control / test). Flag a sample ratio mismatch if imbalance exceeds ~5% with non-trivial volume.
• If $feature_flag_response returns false for most events, the user isn't being bucketed into the experiment at all — the release condition is rejecting them. This is the most common cause of "experiment shows 0 exposures."
• Spot-check 5 raw event rows. Note the $current_url, $device_type, $feature_flag, $feature_flag_response, and distinct_id.

Step 4 — verify downstream attribution

For each metric event (CTA click, signup page visit, signup completion, conversion):

• Pull sample rows and confirm they carry the $feature/<flag-key> property with a real variant value (control or test), not false or missing.
• Action-based metrics: check the action filters. If the variant renders different DOM IDs, the action must match all of them or one variant will artificially show 0 events. Action URL filters should match the production page, not a dev preview.
• Metric scoping: if the metric is too broad (e.g. "any $pageview containing /signup"), it will credit both variants for global traffic regardless of source. Suggest scoping by $feature/<flag-key> property or session entry pathname.

Step 5 — verify identity continuity

Cross-domain handoff (Webflow → app, marketing → product, etc.) is where attribution usually dies.

• Pick 5–10 users who reached the final funnel step (e.g. signup completion). Pull their event timeline.
• Confirm they have a prior $feature_flag_called event with a real variant value.
• Confirm $identify fires on the handoff. If users never have an $identify event, the anonymous device profile never stitches to the authenticated user — variant attribution is dead even with a perfectly fired flag.
• Confirm bucketing key + ensure_experience_continuity settings together don't cause re-bucketing. device_id bucketing without continuity = same user on a new device looks fresh.

Step 6 — downstream conversion metric (trial activation / purchase / etc.)

Customers often have a primary conversion event that lives downstream (in their app or warehouse).

• Search the event schema for the expected event name. Try several variants (plus_trial_activated, trial_started, subscription_created).
• If not present, look for warehouse sources via external-data-sources-list. Common pattern: Snowflake/Postgres table like accounts.trial_started_at.
• Recommend the cleaner path: emit a server-side event from the app on activation. Easier than warehouse joins, faster signal, no schema fragility.
• Alternative: use the warehouse table as an experiment metric directly (supported for funnel + trend metrics).

Step 7 — common pitfalls to call out (regardless of what you found)

The customer's actual setup almost always has one of these:

• Person-property URL targeting (always wrong for this use case)
• Exact-match operators on full-URL person properties (never hit)
• Action metrics tied to dev URLs/selectors that won't fire on prod
• Global metrics ("any signup completion") that credit both variants equally
• Missing $identify on the marketing → product domain handoff
• device_id bucketing without ensure_experience_continuity → re-bucketing across sessions
• Default 14-day conversion window too short for downstream conversion events
• Internal/test user filter not configured → QA traffic skews early days
• Sanity check exposure within 24h of launch — a 50/50 that shows <10 events in a week is a wiring bug, not a power problem

Output format

Group the report by the four standard customer questions. Lead with the worst finding:

:warning: [Experiment name] — audit findings

[One-paragraph TL;DR of the headline finding. Be direct.]

---
(a) Does the config look correct?
[Verdict + specific issues with evidence — event counts, sample values, etc.]

---
(b) How to verify attribution (once issues are fixed)
[Concrete steps the customer can run themselves.]

---
(c) What to change about attribution
[Numbered action list, priority order. Each item should be specific
enough that the customer's engineer can act on it directly.]

---
(d) Common pitfalls to watch for
[Subset of step 7's checklist relevant to this customer's setup.
Frame as general guidance, not as accusations.]

---
Bottom line: [one or two sentences. What's the single most important
fix that unblocks the experiment?]

Rules

• Read-only. Do not create insights, dashboards, actions, experiments, or modify any config. You are impersonating the customer's user — any writes land in their actual project.
• No fabrication. If you can't find the experiment or the data is empty, say so explicitly. Do not invent findings to fill the template.
• Cite real numbers. Every claim about exposure counts, sample ratios, or event volumes must come from a query you actually ran in this session.
• Surface ambiguity. If a setting could be intentional (e.g. low conversion window because conversion happens fast), note both interpretations and ask the customer to confirm.
• Match the customer's writing register. Customers using PostHog are usually technical — don't oversimplify. But avoid jargon shorthand they may not know yet.

When the audit is done

Remind the user to:

• Exit Claude Code
• Log out of Django Admin impersonation in their browser
• Optionally disable the posthog plugin: claude plugin disable posthog

The impersonate-audit.sh wrapper handles step 3 prompts automatically on exit.