Investigating Logs

Investigating logs

Investigation is a narrowing problem: summarize before you read.

One posthog:logs-patterns call compresses millions of lines into at most 200 templates,

and one posthog:logs-patterns-diff call answers "what is different about now vs. before" directly.

Raw rows (posthog:query-logs) are the last step of an investigation, never the first.

When to use this skill

• "Check the logs" / "is service X healthy?" / "did my deploy (or model bump, config change, migration) break anything?"
• "Why are errors up?" / "explain this spike" / incident triage — "what changed?"
• "What is this service logging?" — orienting in an unfamiliar or noisy stream.
• Finding the log evidence for a failure reported elsewhere (an alert, an error-tracking issue, a user complaint).

When _not_ to use this skill

• Creating or tuning log alerts — that's authoring-log-alerts.
• Analytics over product events, persons, or insights — that's querying-posthog-data.
• HogQL exposes a logs table via posthog:execute-sql, but do not investigate through it:

hand-written SQL over logs routinely hits read-byte caps and re-derives what the tools below do in one cheap call.

Reserve SQL for the rare case of joining log-derived facts with non-log data.

Tools

| Tool | Job |

| --------------------------------------------------------------------- | --------------------------------------------------------------------------------------------- |

| posthog:logs-services-create | Top-25 services with log_count, error_count, error_rate, sparkline. Orientation. |

| posthog:logs-patterns | Mine one window's message templates, ordered by frequency. "What is this stream saying?" |

| posthog:logs-patterns-diff | Diff templates between two windows: new / rate-shifted / gone. "What changed?" |

| posthog:logs-count / posthog:logs-count-ranges | Scalar and time-bucketed counts for a filter. Localize volume before pulling rows. |

| posthog:logs-sparkline-query | Volume over time broken down by severity or service (the one bucketed view with a breakdown). |

| posthog:logs-facet-values-create | Distribution of severity/service (or a resource attribute) under a filter. |

| posthog:logs-attributes-list / posthog:logs-attribute-values-list | Discover attribute keys and values before building filters. |

| posthog:query-logs | Raw rows. Endpoint of every drill-down, entry point of none. |

Each tool's own description documents its parameters and response shape — read it before calling.

Pick the workflow by question shape

"Is it healthy?" — post-deploy / post-change verification

The user changed something (deploy, model bump, config, migration) and wants to know the logs still look right.

• Pin down the change time and the affected service(s). Ask if the user hasn't said; the diff is meaningless without a boundary.
• Orient with posthog:logs-services-create: is the service still logging at all, and what is its error_rate now?

A service that went silent fails verification just as hard as one that started erroring.

• posthog:logs-patterns-diff with query.dateRange from the change time to now and baselineDateRange

set to a comparable window just before the change, scoped to serviceNames.

New error/fatal templates right after a change are the classic regression signature;

large rate_ratio shifts on existing error templates are the second thing to check.

• Check volume continuity with posthog:logs-count-ranges spanning before and after the boundary:

a rate discontinuity (crash loop, restart storm, silence) shows up here even when message content looks unchanged.

• Drill only the suspects: pivot each suspicious pattern to raw lines via its match_regex with posthog:query-logs.

A pass verdict needs all three: no new error templates, no large error rate_ratio shifts, and continuous volume.

Say which windows you compared — "healthy" is only as strong as the baseline.

"Explain this spike"

• Localize it: posthog:logs-count-ranges over the user's window, then recurse into the dense bucket(s) — each bucket's

date_from/date_to feeds the next call. Stop after 3–4 levels.

• Explain it: posthog:logs-patterns-diff with the spike as query.dateRange and the window just before as

baselineDateRange. The top new and rate_shift entries are the explanation. Do not mine both windows

separately and diff by hand — the diff is one call.

Incident triage — "what broke?"

posthog:logs-patterns-diff first: incident window vs. a known-good window just before (or omit the baseline for

same-window-last-week). Suspects are new entries and the biggest rate_ratio shifts; pivot each to raw lines.

If the failing service is unknown, find it first with posthog:logs-facet-values-create faceting service_name

under severityLevels: ["error", "fatal"].

"What is this stream saying?" — unfamiliar service

posthog:logs-patterns over the last hour, scoped to the service. Scan templates by estimated_count and

non-zero error share in severity_counts. Widen the window or add searchTerm only if the answer isn't there.

Known needle — a specific message, attribute, or person

When the target is already precise (an error string, a request id, a distinct_id), skip pattern mining:

discover the right keys with posthog:logs-attributes-list / posthog:logs-attribute-values-list,

size the result with posthog:logs-count, then pull rows with posthog:query-logs.

Rules that keep investigations honest and cheap

• Scope serviceNames (or a resource-attribute filter) on every call once the target service is known.

Unscoped calls scan the whole team's stream and starve the pattern sample budget.

• posthog:query-logs requires an explicit query.dateRange — omitting it is a 400, not a default window.
• Pattern counts are sampled estimates (sampled: true); templates rarer than ~1 in 10,000 rows can be invisible.

Absence of a rare template is not evidence it stopped.

• Before trusting a wall of new entries in a diff, check baseline.total_count —

a tiny or empty baseline (logging only just started) makes everything look new.

• severityLevels matches the six canonical lowercase buckets against severity_text exactly.

Zero rows on a severity filter → check the stored values with posthog:logs-attribute-values-list { key: "severity_text" }.

• Budget: one services call, at most one patterns-diff per window pair, 3–4 count-ranges levels,

and query-logs only for confirmed suspects with limit ≤ 100.

Output

Lead with the verdict, then the evidence:

• Verdict: healthy / regressed / inconclusive, with the windows compared.
• Suspects (if any): template, classification (new / rate_shift), estimated counts or rate_ratio, services, and 1–2 sample raw lines.
• What was checked and what wasn't: services covered, windows, and any sampling or baseline caveats that limit confidence.

The user should be able to act on the verdict without re-running the investigation.