Authoring Signals Scouts

Authoring Signals scouts

A scout is a scheduled agent that wakes on its own interval, looks at one PostHog

project, decides what's genuinely worth surfacing, and emits it as a finding into

the Signals inbox — or closes out empty, which is a real outcome. PostHog ships a fleet

of canonical scouts (a cross-product generalist plus per-surface specialists). This

skill helps you and your agent adapt those canonical scouts to a specific project, or

author new scouts from scratch for a use case the fleet doesn't cover.

A scout is just an LLMSkill whose name starts with signals-scout-. The harness

discovers scouts by globbing signals-scout-* over the project's skills, loads the body

verbatim as the agent's system prompt, and progressively reads any bundled reference

files on demand. **The signals-scout- name prefix is load-bearing: a skill named

anything else will never run as a scout.**

The job before the writing

Don't write a scout in the abstract. Ground it in the target project first — a scout is

only as good as its fit to the data it watches.

• Read the project. posthog:signals-scout-project-profile-get returns the

deterministic snapshot the scout itself cold-starts from: products in use, top events

with reach/burst metrics, integrations, existing inbox counts. If the scout watches a

specific event, confirm it exists and check its shape with posthog:read-data-schema.

A scout for an event the project doesn't capture is dead on arrival.

• See what already runs. posthog:signals-scout-config-list lists every existing

scout on the project with its schedule, enabled, and emit posture, plus each scout's

description (pulled from the skill's frontmatter) so you can tell what a scout watches

without loading its body. Don't duplicate a surface a canonical scout already covers —

adapt that one instead.

• Read the closest canonical scout. It's your template and your reference shape. Pull

it with posthog:llma-skill-get {"skill_name": "signals-scout-<x>"} (per-team rows) or

read it from the repo at products/signals/skills/signals-scout-*/. The generalist

(signals-scout-general) is the broad template; if your scope is domain-tight, pick

the specialist closest to your surface — list the live roster with

posthog:llma-skill-list {"search": "signals-scout"} (specialists exist for most

product surfaces: error tracking, logs, AI observability, experiments, feature flags,

session replay, web analytics, surveys, and more).

• Skim the inbox. posthog:inbox-reports-list shows what findings are actually

landing — calibrate so your scout adds signal, not noise.

Choose the path

There are two independent decisions: what you're building, and where it lives.

What

| Situation | Approach |

| ---------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------- |

| A canonical scout is close but too broad / too noisy / missing a disqualifier for this project | Adapt it — narrow the scope, add disqualifiers, retune thresholds. |

| You want a surface no canonical scout covers (a custom event, a product-specific funnel) | New scout from scratch — copy the closest canonical scout as scaffolding, replace the domain discriminator + explore patterns. |

| You only want to change _when_ / _whether_ a scout runs | No authoring — just tune the config (see Run posture). |

Where

| Path | Mechanism | Use when |

| ------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------- |

| Per-team (the common user path) | Create/edit a signals-scout-* LLMSkill row in the project's skills store via posthog:llma-skill-create / -update / -file-create, then register its config immediately via posthog:signals-scout-config-create. | Customizing for one project. The harness globs the row in on the next tick; canonical sync leaves your edited ("diverged") row alone. |

| Canonical (PostHog contributors) | Edit disk under products/signals/skills/signals-scout-*/, lint/build, open a PR. | Improving a scout for _every_ enrolled project. lazy_seed mirrors it onto all enrolled teams on the next tick. |

Adapting-in-place tradeoff: editing a canonical scout's row for your team marks it

diverged — you stop receiving upstream improvements to that scout. If you only need an

_additional_ behavior, prefer authoring a new, differently-named scout

(signals-scout-<your-scope>) and leaving the canonical one intact.

See references/lifecycle-and-testing.md for the

exact skills-store calls, the build/lint commands, and how seeding works.

Write the scout

First pick the shape. references/scout-patterns.md is a

cookbook of the reference architectures scouts fall into — anomaly watcher, watchlist

explore/exploit, cross-product correlation, recommendation/gap, warehouse-backed source,

custom single-event, open-text theme, external-tool/code — each mapped to a canonical scout

you can copy as scaffolding. It also makes the key point that **a scout can watch any source

PostHog ingests into the data warehouse, not just analytics events** (a Slack channel sync, a

billing system, a CRM, a support inbox), plus external systems reachable from the sandbox.

Find the closest pattern, then write the body.

Follow references/scout-anatomy.md — it has the frontmatter

schema, the canonical body structure (quick close-out → orient → domain discriminator →

explore patterns → save-memory → decide → disqualifiers → close-out), the lean-body rule,

and copy-ready skeleton templates for both a specialist and the generalist.

Two craft references the whole fleet reasons in terms of — a good scout's Decide and

memory sections are built on them, so read them before writing those sections:

• references/emit-contract.md — what emit-signal takes,

the confidence rubric, severity, dedupe keys, finding_id, the description

prose contract, and a worked example. This is how your scout decides _what clears the

bar_ and _how to write the finding_.

• references/dedupe-and-memory.md — the four-states

classifier (net-new / material-update / already-covered / addressed-or-noise), the

scratchpad key-prefix vocabulary, and the cross-project noise patterns. This is how your

scout avoids re-emitting and learns across runs.

The single most important design decision in any scout is its **signal-vs-noise

discriminator** — the cheap profile-shape read that separates "worth investigating" from

"baseline". For error tracking it's the count vs distinct_users ratio; for CSP it's

reach over raw count. Your new scout needs its own. Name it explicitly near the top of the

body so every run anchors on it.

Run posture (config)

A scout's schedule and emit behavior live on its SignalScoutConfig, separate from the

skill body. For a brand-new scout, register the config immediately after creating the

skill with posthog:signals-scout-config-create {"skill_name": "signals-scout-<scope>", ...},

setting any of the fields below in the same call — including creating it disabled or in

dry-run before it ever runs. (It's an upsert: if the coordinator already auto-registered

the row, your fields are applied to it.) Otherwise the coordinator auto-registers an enabled

hourly default on its next tick (up to ~30 min). For an existing scout, tune with

posthog:signals-scout-config-update (find the id via -config-list):

• run_interval_minutes — 10 to 43200. Default 60 (hourly). Slow a chatty or expensive

scout by raising this.

• enabled — false pauses the scout entirely (coordinator skips it).
• emit — defaults to true: the scout writes its findings straight to the inbox. The

standard flow is to make a scout and let it emit — seeing what actually lands is the

fastest way to calibrate it. Set emit=false (dry-run) only when you want to be extra

careful: the scout still runs and logs its reasoning but writes nothing to the inbox.

Reach for dry-run on a scout you expect to be chatty, expensive, or high-stakes; for most

scouts, just emitting and watching the inbox is the better loop.

Test loop

You can't force a synchronous run as a user — scouts fire on their schedule. The standard

loop is emit + inspect: ship the scout live, let it emit, and calibrate against what

actually lands.

• Ship the scout (the default emit=true) with a short run_interval_minutes so it fires

soon — set it at creation via

posthog:signals-scout-config-create {"skill_name": ..., "run_interval_minutes": 10}

right after llma-skill-create, rather than waiting for the coordinator to

auto-register an hourly default.

• After a tick, read what it did: posthog:inbox-reports-list (the findings it actually

emitted), posthog:signals-scout-runs-list (run summaries), -runs-retrieve (full

reasoning for one run), and -scratchpad-search (the durable memory it wrote).

• Refine the body — tighten the discriminator, add disqualifiers for whatever it

false-positived on, fix the emit calibration.

• Once it's landing the right findings, restore the interval to something sustainable

(hourly+).

Want to be extra careful? Set emit=false to dry-run first — create the config with

emit=false via -config-create so the scout never has a live first run; it runs and logs

what it _would_ have emitted (visible via -runs-list / -runs-retrieve) without writing to

the inbox. Inspect, refine, then flip emit=true. Worth it for a scout you expect to be

chatty, expensive, or high-stakes; otherwise just emitting and watching the inbox is the

faster path to a calibrated scout.

Repo contributors get a faster loop — hogli sync:skill and the harness's local run path;

see references/lifecycle-and-testing.md.

To read what your scouts are doing rather than change them — surveying the fleet, inspecting

individual runs, the scratchpad memory, and assessing performance — use the read-only companion

skill exploring-signals-scouts. Keep the two in sync when the scout config / run / scratchpad

surfaces change.

Quality bar for a v1 scout

• A named, cheap signal-vs-noise discriminator anchored near the top.
• A quick close-out so a quiet run is cheap (don't pay for deep exploration when the

watched surface is at baseline or absent).

• 2–4 concrete explore patterns with the actual queries/tools to run — starting

points, not a rigid checklist.

• Disqualifiers listing this project's known noise (single-user quirks, dev-env

bursts, allowlisted entities).

• A Decide section calibrated against the emit contract (confidence ≥ 0.65 to emit;

below that, write memory).

• Save-memory guidance using the scratchpad prefixes so the scout gets smarter each run.
• A lean body (push depth into references/) — every line is a recurring token cost on

every run.