Creating Online Evaluations

Creating online evaluations

An online evaluation automatically scores either each matching $ai_generation or the whole trace

containing it, until disabled. A good eval comes from a real failure mode you've found in production traffic,

not from a guess or a generic metric like "hallucination" or "helpfulness". This skill starts once that

failure mode is identified and turns it into a scoped, continuously-running eval.

First, know what you're evaluating. Finding and ranking the failure modes worth catching is a

separate job. If the user doesn't specify what they want to evaluate, ask them. If they are still vague

about it and don't refer to a specific failure mode, run exploring-ai-failures to scope a use case,

find failing traces, and produce a ranked list of failure modes.

For the mechanics of _writing and iterating_ an evaluator (Hog source vs LLM-judge prompt, dry-running,

debugging a live eval), defer to exploring-llm-evaluations.

Tools

| Tool | Purpose |

| -------------------------------------- | ------------------------------------------------------------- |

| posthog:llma-evaluation-config-get | Check the active provider key used by unpinned judges |

| posthog:llma-provider-key-list | Find a usable (ok state) provider key to pin |

| posthog:llma-evaluation-judge-models | List valid provider+model combos |

| posthog:llma-evaluation-test-hog | Dry-run Hog source against recent generations before creating |

| posthog:llma-evaluation-create | Create the evaluation (always enabled: false first) |

| posthog:llma-evaluation-run | Spot-run a draft eval against one generation |

| posthog:llma-evaluation-update | Iterate config, then flip enabled: true |

| posthog:execute-sql | Verify a condition matches the events and volume you expect |

| posthog:generate-app-url | Build a region- and project-qualified deep link to the eval |

The full create payload (every field, the config schemas, the exact conditions shape) is in

references/evaluation-payload.md.

Phase 1 — Pick the failure mode to evaluate

Start from a real, observed failure, not a metric you picked in advance. If you don't already have one,

run exploring-ai-failures to scope a use case, find failing traces, and produce a ranked list of failure

modes — then come back. With that list in hand, talk with the user to choose what to turn into an eval:

• Most frequent, most painful first. A handful of modes usually cover the majority of failures.
• Pair obvious fixes with the eval, don't skip it. If a prompt tweak would likely fix the failure, set

up the eval anyway and suggest the fix alongside it — a rising pass rate is how you confirm the fix landed.

• One mode per eval. Three failure modes is three evals, not one prompt trying to catch everything.

You should end with a single, crisp, checkable criterion — "the reply must stay on the user's topic", "the

tool call must include an order_id". Then move to Phase 2.

Phase 2 — Build the online eval

2.1 — Choose the eval type

| Use… | When the criterion is… |

| ----------- | ------------------------------------------------------------------------------------------------------------------------------------- |

| hog | Structural / rule-based (JSON parses, length, regex, tool-call shape). Cheap, deterministic, no provider key needed. |

| llm_judge | Subjective / fuzzy (tone, factuality, on-topic). Costs an LLM call per run; needs a provider, model, and usable provider key. |

| sentiment | You want sentiment labels on user messages, not a pass/fail (unless very specifically asked for, usually not relevant to this skill). |

Reach for hog first, escalate to llm_judge if there is no deterministic way to check for what we want to check.

2.2 — Choose the target

| Target | Behavior |

| ------------ | ------------------------------------------------------------------------------------------------------------------ |

| generation | Runs once for each matching $ai_generation, immediately after ingestion. This is the default. |

| trace | Runs once for the whole trace after the first matching generation and a configurable wait for the trace to finish. |

For a trace target, send "target": "trace" plus a settle config that controls when the trace is

evaluated, discriminated on strategy:

• { "strategy": "fixed_window", "window_seconds": 1800 } — evaluate a fixed wait after the first

matching generation. Between 10 seconds and 2 hours, defaults to 30 minutes. A target_config

without a strategy key means this.

• { "strategy": "inactivity", "quiet_period_seconds": 300, "max_age_seconds": 7200 } — evaluate once

the trace has had no new activity for the quiet period (10 seconds to 30 minutes,

defaults to 5 minutes). max_age_seconds caps the total wait from the first matching generation

(1 minute to 2 hours, defaults to 2 hours, must be at least the quiet period).

Conditions still match the generation that triggers the run; the evaluator itself receives

the complete trace. Sentiment evaluations support only the generation target.

New Hog source should use the globals shared by both targets:

| Global | Meaning |

| -------------------------------------- | ------------------------------------------------------------------------------------------------- |

| evaluation_events | One generation event for a generation target, or every captured event for a trace target. |

| target | The target's type, id, total_cost_usd, and total_latency_seconds. |

| item.input_text / item.output_text | Best-effort readable projections; use these for length, keyword, and regex checks. |

| item.input / item.output | Original serialized values; use these when the evaluator needs to parse the captured JSON itself. |

Generation evaluations still expose top-level input, output, properties, and event. Trace evaluations

still expose their original events and trace globals. Those globals are kept for compatibility with saved

evaluators. Do not use target-specific globals in new source that needs to work for both targets. The text

projections recognize common provider payloads but are not authoritative; use item.input / item.output when

exact structure matters.

2.3 — Configure the LLM judge

An llm_judge evaluation requires a valid provider and model. It also needs a usable provider key

when it runs. provider_key_id controls whether the evaluation pins one specific key:

• Set provider_key_id to the UUID of an ok-state key for the same provider to pin it.
• Set provider_key_id to null to use the team's active provider key. The active key must be in the

ok state and use the same provider as model_configuration.provider.

Hog and sentiment evaluations skip this step.

posthog:llma-evaluation-config-get        // check active_provider_key for an unpinned judge
posthog:llma-provider-key-list            // find an ok-state key to pin
posthog:llma-evaluation-judge-models      // { "provider": "openai" } → valid models

Confirm the provider and model with llma-evaluation-judge-models. Prefer pinning the chosen key so a later

team-wide active-key change does not change how the evaluation runs. Leave provider_key_id as null only

after llma-evaluation-config-get confirms the active key is usable and its provider matches.

If there is no usable key, you may still create a disabled draft for the user to review. Do not spot-run or

enable it. Ask the user to add or validate a key in the UI before continuing.

2.4 — Create it disabled

Create with enabled: false so nothing fires until the scope is verified. Minimal hog example:

posthog:llma-evaluation-create
{
  "name": "Output is not empty",
  "description": "Fails when a generation has no readable output",
  "evaluation_type": "hog",
  "evaluation_config": { "source": "let count := 0\nfor (let i, item in evaluation_events) {\n    if (item.event == '$ai_generation') {\n        count := count + 1\n        if (length(trim(item.output_text)) == 0) { return false }\n    }\n}\nreturn count > 0" },
  "output_type": "boolean",
  "output_config": { "allows_na": false },
  "target": "generation",
  "target_config": {},
  "conditions": [
    { "id": "default", "rollout_percentage": 100, "properties": [{ "key": "$ai_model", "type": "event", "operator": "icontains", "value": "gpt" }] }
  ],
  "enabled": false
}

For llm_judge, swap evaluation_config to { "prompt": "…" } and add

"model_configuration": { "provider": "openai", "model": "gpt-5-mini", "provider_key_id": "<uuid of an ok-state key from llma-provider-key-list>" }.

Use null only when the active team key is ok and uses the same provider. Full field reference:

references/evaluation-payload.md.

2.5 — Verify the scope before enabling

conditions is where online evals go wrong: too broad and you evaluate (and bill) a firehose; too narrow

and it never fires. Confirm the filter matches the events you expect, and roughly how many per day:

posthog:execute-sql
SELECT count() AS matched, count() / 7 AS per_day
FROM events
WHERE event = '$ai_generation'
    AND properties.$ai_model ILIKE '%gpt%'      -- mirror each condition property
    AND timestamp >= now() - INTERVAL 7 DAY

For generation targets, count() is the run volume. For trace targets, count distinct non-empty

$ai_trace_id values because matching generations from the same trace schedule only one run.

If volume is high, set rollout_percentage below 100 to sample. Spot-check the evaluator with

llma-evaluation-test-hog (hog) or llma-evaluation-run against one generation (llm_judge).

Both tools currently use generation samples; for a trace target they can check shared source or prompt behavior,

but they do not reproduce the complete settled trace. Review the first live trace results before increasing rollout.

> Watch out: some orgs reuse a single $ai_trace_id across 100k+ events. Scoping by trace-ID prefix

> can match far more than expected — verify volume with the SQL above before enabling.

2.6 — Enable, then close the loop

posthog:llma-evaluation-update
{ "evaluationId": "<uuid>", "enabled": true }

It now runs on every new matching generation, or once per matching trace for a trace target. This isn't

one-and-done: the user should be aware that they need to keep an eye on results and iterate if the outcome

is not the expected one. To wire results into a Slack feed, see feature-usage-feed.

Scoping with conditions

conditions is a list of condition sets — OR between sets, AND within a set's properties. Each

set is { id, rollout_percentage, properties[] }. There is no time window inside conditions; sampling is

only rollout_percentage (0–100). Property filters use the standard PostHog shape

(key, type, operator, value). For trace targets, these filters still select the generation that

triggers the eventual whole-trace evaluation.

"conditions": [
  { "id": "openai",    "rollout_percentage": 100, "properties": [{"key": "$ai_provider", "type": "event", "operator": "exact", "value": "openai"}] },
  { "id": "anthropic", "rollout_percentage": 25,  "properties": [{"key": "$ai_provider", "type": "event", "operator": "exact", "value": "anthropic"}] }
]

Constructing UI links

Build links with posthog:generate-app-url — never hand-write the host or the /project/<id>/ prefix.

The url must be a canonical catalog template; pass concrete ids via params, never inline them into the path.

• Evaluations list: generate-app-url {url: "/ai-evals/evaluations"}
• Single evaluation: generate-app-url {url: "/ai-evals/evaluations/{id}", params: {id: "<evaluation_id>"}}

These resolve to the correct region host and project prefix (e.g.

https://us.posthog.com/project/<id>/ai-evals/evaluations/<evaluation_id>). Surface the link after

creating so the user can review and toggle it in the UI.

Tips

• Evals come from real failures, not generic metrics. Start from a failure found in this product's

traffic (via exploring-ai-failures), not from "let's measure hallucination". A metric nobody traced

back to a real bad output is noise.

• One eval, one failure mode. Different failure modes need different evals; don't make one eval try to

catch everything.

• Suggest changes along with the eval if possible. If it's clear a prompt change would fix the issue, for

instance, set up the eval but also suggest to the user they change the prompt: they should soon see the eval

go from low pass rate to a higher pass rate.

• hog first. No provider key, no AI approval, deterministic. Reach for llm_judge only when the

criterion genuinely can't be coded.

• Always create disabled, verify scope, then enable. An eval firing on the wrong events is worse than

none — noise, and (for llm_judge) cost.

• Configure llm_judge credentials before running. A judge needs a valid provider and model plus a usable

provider key. provider_key_id may be null only when the matching active team key can be used.

• bytecode is server-written for hog evals — never pass it; send only evaluation_config.source.
• For cluster-scoped evals, identify the cluster with exploring-llm-clusters, then translate its event

filter into conditions.