Exploring The Wizard

Exploring the wizard as an agent

Drive a real wizard run yourself: boot it on an app, read each screen, decide,

act, snapshot.

The MCP server

Everything goes through the wizard-ci MCP server, registered in this

repo's .mcp.json (npx tsx scripts/wizard-ci-mcp.no-jest.ts) — it runs the

wizard from this checkout's source, so whatever branch you're on is what

you're testing. If the tools (open_app, read_state, …) aren't available,

the server isn't approved yet — ask the user to approve wizard-ci, then

retry. For how the harness works underneath, read

e2e-harness/ARCHITECTURE.md.

What it can do

• Boot the real TUI on any app directory and walk every pre-auth screen:

framework detection, gatherContext, feature discovery, warehouse scan,

intro, setup questions, health check, auth.

• Run the real integration (run_agent): OAuth-equivalent bootstrap, the

full agent run, outro, MCP install, skills — creating **real PostHog

resources** (dashboard + insights) in the target project.

• Show you every screen (render_screen) exactly as a user sees it.
• What it can't do: drive two apps at once (open_app replaces the active

wizard), or advance auth/run without credentials.

Credentials

Two modes — pick by how far the run must go:

• Detection-only (no credentials). open_app needs just

{ appDir, projectId }. Everything up to and including the auth screen

runs credential-free — enough to regression-test detection, setup

questions, and screen flow. End the run at auth.

• Full run (credentials required) — anything past auth, i.e.

run_agent. Prompt the user for three things before starting:

• Key — ask "What's the path to your phx key file?" and pass it as

keyFile (preferred: keeps the key out of logs). If they hold the key

in an environment variable instead, have them write it to a file first

(printenv THEIR_VAR > /tmp/phx-key — you never echo it) or pass

apiKey inline as a last resort. Never print or commit the key.

• Project id — the PostHog project the key is scoped to (projectId).
• Region — us (default) or eu (region).

Always copy the target app to a throwaway /tmp copy (never a real

fixture) — the run edits files.

How to drive it

• open_app({ appDir, projectId, keyFile?, region? }) — boots a live

wizard on the app and returns the first screen.

• read_state — current screen, run phase, secret-free session, tasks,

and the actions legal right now. Call after every move.

• perform_action({ action, params? }) — commit a decision:

confirm_setup, dismiss_outage, choose (a setup question, e.g.

{ key, value }), set_mcp_outcome, dismiss_slack, keep_skills.

• render_screen — render the current TUI to ANSI so you can _see_ it.
• run_agent — kicks off the real integration in the background and

returns immediately; it bootstraps credentials, so it's what advances

auth and run. Then poll read_state — runPhase goes

running → completed and the screen advances to outro.

A typical full walk:

open_app → intro → perform_action confirm_setup
read_state → health-check → perform_action dismiss_outage
read_state → auth → run_agent           (returns at once; integration runs in background)
read_state (poll) → runPhase running → completed, screen → outro
outro → perform_action dismiss_outro → … → keep_skills

A detection-only walk (no key):

open_app → read_state (poll until detectionComplete) →
check integration / detectedFrameworkLabel → confirm_setup → auth → done, next app

Snapshot with render_screen at each key moment and save each frame to a

numbered file — /tmp/wz-explore-snaps/NN-<screen>.txt, incrementing NN in

visit order — so the run leaves a readable, ordered record you and the user

can review afterward (the same shape the CI route's .txt frames take).

Capture the run screen as it progresses, not just on screen changes.

Sweeping the workbench

The fixture library lives at

wizard-workbench/apps/basic-integration/<framework>/<app> (sibling repo).

To regression-test detection across every framework, loop the detection-only

walk over each app. Learned the hard way:

• Copy with rsync -a --exclude node_modules --exclude .git (plus

vendor/venv/Pods/build/dist). A plain cp -R of the workbench fills the

disk, and a copy that dies mid-write leaves a truncated app that detects as

null — a false regression. If detection returns null unexpectedly, check

the fixture (package.json present?) before blaming the code.

• open_app returns the first paint, sometimes before detection lands

(detectionComplete: false, integration: null). Poll read_state —

slower detectors (laravel, rails) need a beat.

• Logs: every run appends to /tmp/posthog-wizard.log. Record

wc -c < /tmp/posthog-wizard.log before the sweep and slice with

tail -c +OFFSET after — that's the run's own log, greppable for

detection lines and [bounded-fs] cap warnings.

• **Router modes and other gatherContext results don't appear in

read_state or the log.** An empty setupQuestions implies the mode

resolved (ambiguity would raise a question), but for positive proof call

the util directly with npx tsx against the same fixture (e.g.

getNextJsRouter, getTanStackRouterMode).

• Delete fixtures as you go — a full sweep is multiple GB.

Key facts

• State → screen. You never navigate; you commit a decision (an action) and the

router re-derives the active screen. Name actions, not keys.

• auth and run advance only via run_agent. They expose no action and

don't self-advance. run_agent returns immediately and runs the integration in

the background — poll read_state for runPhase (running → completed).

Everything else is an instant commit.

• run_agent creates real PostHog resources (a dashboard + insights) in the

project; each run duplicates them.

• A green run ≠ a valid integration. runPhase=completed means the flow

finished, not that the wizard understood the framework (e.g. it'll treat a Wasp

app as react-router). Read what it actually changed.

• None of this ships. The harness lives in e2e-harness/, out of src/.