posthog/monitoring-capture-service
> Guide for using the Grafana MCP to monitor and diagnose the capture service (rust/capture) in production. Use when investigating latency, event loss, Kafka backpressure, Redis issues, rate limiting, Envoy proxy issues, or any capture health question. Covers prod-us and prod-eu environments.
npx skills add https://github.com/PostHog/posthog --skill monitoring-capture-serviceThe capture service (rust/capture/) is PostHog's Rust HTTP ingestion endpoint.
It receives events from SDKs, applies quota/rate limits, and produces to Kafka.
Five deployments run the same binary with different CAPTURE_MODE configs, each in its own K8s namespace.
This skill teaches how to discover live metrics using the Grafana MCP tools
rather than memorizing metric names that change as the code evolves.
The Grafana MCP is connected to a single Grafana instance scoped to one environment.
If the user hasn't specified, ask which environment they want to investigate:
Most capture app metrics (e.g. capture_*, http_requests_*, envoy_cluster_*) are
environment-specific by virtue of which Grafana you're connected to — they don't carry
an environment label. MSK and CloudWatch metrics do carry environment labels but
are still scoped to the connected Grafana's AWS account.
Cross-environment comparison requires switching Grafana instances (not possible in one session).
Capture spans seven telemetry domains.
Each has a Grafana datasource and a discovery entry point.
| Domain | Datasource UID | Discovery tool | Scope filter |
| ------------------------------ | ------------------------------- | ------------------------------ | ---------------------------------------------------- |
| App metrics (VictoriaMetrics) | victoriametrics | list_prometheus_metric_names | regex: "capture_.*" |
| App metrics (realtime) | victoriametrics-realtime | same | same (lower retention, higher resolution) |
| Logs | P44D702D3E93867EC (Loki-logs) | list_loki_label_names | app=~"capture.*" |
| Profiling | pyroscope | list_pyroscope_profile_types | service_name="capture-analytics/capture-analytics" |
| Dashboards | n/a | search_dashboards | query "capture" or "ingestion" |
| CloudWatch (ElastiCache, MSK) | P034F075C744B399F | query_prometheus | environment="prod-us" |
| CloudWatch Root (prod-us only) | PAAE47F430CFD1449 | same | root account AWS metrics (does NOT exist in prod-eu) |
These facts change infrequently and are hard to discover dynamically.
Each capture variant runs as a separate K8s deployment in its own namespace.
The primary scope labels are namespace and container (not role — that label
contains pod names and is not useful for filtering).
| Deployment | Namespace | capture_mode | Pipeline | Notes |
| ------------------- | ------------------- | -------------- | ------------------ | --------------------------------------------- |
| capture-analytics | capture-analytics | events | Main events | Highest volume; routes /e, /i/v0/e, etc. |
| capture-ai | capture-ai | events | AI/LLM events | Routes /i/v0/ai; OTel on port 4318 |
| capture-replay | capture-replay | recordings | Session recordings | Routes /s/; CAPTURE_MODE=recordings |
| capture-mirrored | capture-mirrored | events | Mirror/canary | Not always running; same metrics as analytics |
| capture-logs | capture-logs | — | Log ingestion | OTel logs on port 4318 |
All variants share the same Rust binary (ghcr.io/posthog/posthog/capture).
Scope capture metrics with namespace=~"capture-.*" or container=~"capture-.*".
For a single pipeline, scope by namespace (e.g., namespace="capture-analytics").
Envoy metrics use envoy_cluster_name to identify the upstream backend.
Pattern: posthog_{deployment}_{port}.
Capture-related clusters:
posthog_capture-analytics_3000, posthog_capture-ai_3000, posthog_capture-replay_3000,
posthog_capture-mirrored_3000, posthog_capture-logs_4318, posthog_capture-logs-canary_4318.
capture-replay also has a proxy-as-a-service cluster used in KEDA autoscaling:
proxy-as-a-service_capture-replay_3000.
Scope with: envoy_cluster_name=~"posthog_capture-.*".
Capture depends on up to three logical Redis instances,
plus one external instance at the Envoy layer (not in the capture binary).
None emit capture_redis_* metrics — Redis health is inferred from capture-side
metrics and CloudWatch ElastiCache metrics.
1. Primary Redis (REDIS_URL env var)
posthog-solo (prod-us) or posthog-prod-redis-encripted (prod-eu; sic — typo in actual cluster name)REDIS_READER_URL)CaptureQuotaLimiter), session replay overflow limitercapture_billing_limits_loaded_tokens (by cache_key),capture_quota_limit_exceeded (by resource)
events, exceptions, llm_events, recordings, survey_responses@posthog/quota-limits/{resource}, @posthog/capture-overflow/replay2. Global Rate Limiter Redis (GLOBAL_RATE_LIMIT_REDIS_URL, optional)
capture-globalratelimit-{env}-redis (prod-us, prod-eu; not dev)GLOBAL_RATE_LIMIT_REDIS_READER_URLGLOBAL_RATE_LIMIT_ENABLED (may be off in some envs during rollout)global_rate_limiter_* (direct), capture_events_rerouted_overflow{reason="rate_limited"} (proxy signal)capture-globalratelimit-prod-redis3. Event Restrictions Redis (EVENT_RESTRICTIONS_REDIS_URL)
ingestion-prod-redis (separate writable cluster in both envs)capture_event_restrictions_redis_fetch (labels: restriction_type,result in success/not_found/error/parse_error),
capture_event_restrictions_stale, capture_event_restrictions_loaded_count
4. Contour Rate Limit Redis (ratelimit-{env}-redis) — NOT in capture binary
ratelimit_service_* (label: domain="posthog")Every prefix here can be discovered live with list_prometheus_metric_names
using datasourceUid: "victoriametrics" and regex: "<prefix>.*".
| Prefix | Domain | Scope label |
| ------------------------------ | --------------------------------------- | ---------------------------------------------- |
| capture_* | App metrics (~80 metrics) | namespace, container |
| http_requests_* | HTTP layer (shared) | namespace=~"capture-.*" |
| capture_kafka_* | Kafka producer (17 metrics) | namespace, container |
| capture_billing_* | Billing/quota tokens loaded | namespace, cache_key |
| capture_event_restrictions_* | Event restrictions (6 metrics) | namespace, restriction_type |
| capture_ai_otel_* | AI/OTel capture (12 metrics) | namespace="capture-ai" |
| envoy_cluster_* | L7 proxy | envoy_cluster_name=~"posthog_capture-.*" |
| aws_msk_* | MSK broker-side (JMX) | environment="prod-us" or "prod-eu" |
| ratelimit_service_* | Contour rate limit | domain="posthog" |
| overflow_redirect_* | Node.js ingestion overflow (downstream) | ingestion_pipeline |
| kube_* / container_* | K8s resources | namespace=~"capture-.*", pod=~"capture-.*" |
Topics capture produces to (discover live via topic label on capture_kafka_produce_avg_batch_size_bytes).
Partition counts are encoded in topic names and differ by env (EU generally has fewer partitions).
Capture writes to two different Kafka backing systems depending on the pipeline:
warpstream-replay-v2 VC), logs (warpstream-logs VC), traces (warpstream-traces VC)| Topic (prod-us / prod-eu) | Backing | Pipeline |
| ---------------------------------------------------------- | ---------- | -------------------------------------------- |
| ingestion-analytics-main-1024 / -512 | MSK | Main events |
| ingestion-analytics-overflow-128 | MSK | Overflow (rate-limited / high-volume tokens) |
| ingestion-analytics-historical-128 | MSK | Historical backfill events |
| ingestion-analytics-turbo-1024 | MSK | General turbo (prod-us only) |
| ingestion-heatmaps-main-128 | MSK | Heatmaps |
| ingestion-errortracking-main-128 | MSK | Error tracking |
| ingestion-errortracking-overflow-32 | MSK | Error tracking overflow |
| ingestion-clientwarnings-main-16 / -32 | MSK | Client warnings |
| ingestion-sessionreplay-main-512 / -256 | WarpStream | Session replay |
| ingestion-sessionreplay-overflow-64 / -32 | WarpStream | Session replay overflow |
| ingestion-logs | WarpStream | Log ingestion |
| ingestion-traces | WarpStream | Traces ingestion |
| ingestion-analytics-main-dlq (+ per-pipeline DLQ topics) | MSK | Dead letter queues |
| Service name | Deployment |
| ------------------------------------- | ---------------------------- |
| capture-analytics/capture-analytics | Main capture |
| capture-ai/capture-ai | AI capture |
| capture-replay/capture-replay | Replay capture |
| capture-mirrored/capture-mirrored | Mirror/canary (when running) |
| capture-logs/capture-logs | Logs capture |
Profile types: process_cpu:cpu:nanoseconds:cpu:nanoseconds,
wall:wall:nanoseconds:wall:nanoseconds,
memory:inuse_space:bytes:inuse_space:bytes,
memory:inuse_objects:count:inuse_space:bytes.
| UID | Title | Use for |
| ------------------------- | -------------------------- | -------------------------------------------------------------------------- |
| capture | Capture | Overview across capture deployments — start here |
| capture-v1-details | V1 Details | Per-panel detail for the v1 pipeline (/i/v1/analytics/events) |
| ddfkdj56ds11xce | Legacy Details | Per-panel detail for the legacy v0 pipeline (/e/, /batch/, /i/v0/e/) |
| ingestion-health | Ingestion - Health | Cross-service ingestion health |
| ingestion-analytics | Ingestion - Analytics | Downstream analytics pipeline, per lane |
| ingestion-pipelines | Ingestion - Pipelines | End-to-end pipeline throughput and lag |
| ingestion-reliability | Ingestion - Reliability | Ingested-event trends, processing-lag SLOs, pod restarts |
| ingestion-sessionreplay | Ingestion - Session Replay | Session replay ingestion |
| contour | Contour Ingress | Envoy L7 proxy (set envoy_cluster_name=posthog_capture-analytics_3000) |
| AWSRedis | AWS ElastiCache Redis | Per-cluster CloudWatch Redis metrics (filter cacheclusterId) |
The two details boards live under Capture/Ingestion/ and cross-link each other
plus the overview. Sections for machinery shared by both pipelines (Global Rate
Limiter, Event restrictions, Hyper server, Contour, billing/quota) appear on
both details boards, so use whichever one you are already on.
One board you will find via search_dashboards but should not rely on:
capture-3000-envoy-codes is a personal, US-only Envoy investigation board (not
synced to EU).
list_prometheus_metric_names — datasourceUid: "victoriametrics", regex: "capture_.*" to enumerate app metricslist_prometheus_label_names scoped to it — see available dimensionslist_prometheus_label_values — discover actual values for a label(e.g. labelName: "cause" on capture_events_dropped_total)
query_prometheus with PromQL — always scope by namespace (or container) and set a time rangelist_loki_label_names — datasourceUid: "P44D702D3E93867EC" (Loki-logs; do NOT use primary Loki P8E80F9AEF21F6940 which 502s intermittently)list_loki_label_values for app or namespace — find capture containersquery_loki_logs — e.g. {app=~"capture.*"} |= "error"list_pyroscope_profile_types — data_source_uid: "pyroscope"fetch_pyroscope_profile — matchers: '{service_name="capture-analytics/capture-analytics"}',profile_type: "process_cpu:cpu:nanoseconds:cpu:nanoseconds"
search_dashboards — query "capture" or "ingestion"get_dashboard_by_uid — use a known UID (e.g. "capture") to get panel detailsget_dashboard_panel_queries — extract PromQL from existing panelscapture_billing_*, capture_event_restrictions_*,capture_quota_* metrics in VictoriaMetrics
P034F075C744B399F for ElastiCache(CPU, memory, connections, latency).
Cluster IDs: capture-globalratelimit-prod-redis, posthog-solo (prod-us primary)
Categories of what to look for. Discover specific metrics live using the prefixes above.
HTTP layer — request rate, latency distribution (p50/p99), active connections,
error rates by status code. Metrics: http_requests_* scoped by namespace, capture_active_connections.
Event lifecycle — the funnel from received to ingested to dropped/rerouted.
capture_events_received_total -> capture_events_ingested_total -> capture_events_dropped_total.
The cause label on drops has 20+ values (discover live). Key additions since the golden-chart
migration: event_restriction_drop, event_too_big, otel_quota_drop, oversize_event,
ai_opt_in, gathering, invalid_session, no_session_id, no_snapshot.
Rerouting: capture_events_rerouted_overflow with reason label
(rate_limited, force_limited, event_restriction).
Also: capture_events_rerouted_custom_topic for topic-redirect restrictions.
Kafka producer — broker connectivity (capture_kafka_any_brokers_down,
capture_kafka_broker_connected), queue saturation (_queue_depth / _queue_depth_limit),
produce RTT (capture_kafka_produce_rtt_latency_us by quantile and broker),
delivery errors (capture_kafka_produce_errors_total).
Billing and quota — capture_billing_limits_loaded_tokens by cache_key,
capture_quota_limit_exceeded by resource (events, exceptions, llm_events, recordings, survey_responses).
Event restrictions — capture_event_restrictions_* for Redis fetch health,
staleness, loaded count, applied restrictions by restriction_type
(drop_event, force_overflow, redirect_to_topic, skip_person_processing).
Envoy proxy — upstream latency, response codes (2xx/4xx/5xx), connection health,
circuit breakers (_open gauges), backend membership (healthy vs total),
timeouts, retries. Always filter: envoy_cluster_name=~"posthog_capture-.*".
For capture-replay, also check proxy-as-a-service_capture-replay_3000.
Contour rate limit — ratelimit_service_* for per-IP DoS protection.
ratelimit_service_rate_limit_over_limit = actively rate-limited IPs.
MSK broker-side — aws_msk_* JMX metrics for capture-analytics, capture-ai, and other MSK-backed topics.
Key signals: throttle time, network processor idle %, memory pool depletion, request queue size.
Both envs have a dedicated ingestion MSK cluster separate from the events cluster
(prod-us: c21; prod-eu: posthog-prod-eu-ingestion-2026-05-04).
WarpStream — warpstream_agent_* metrics for capture-replay, capture-logs, and traces.
These pipelines produce to in-cluster WarpStream agents, not MSK.
Key signals: warpstream_agent_control_plane_operation_counter (by operation),
warpstream_agent_file_cache_client_fetch_local_or_remote_counter (cache hit/miss ratio).
Dashboards: warpstream (Agent Overview), dbfj5c31spa1ogf (MSK vs WarpStream — Active Produce Topics).
US-only personal dashboards (not synced to EU): ws-coarse-lag-explore (Coarse Lag), 8e93b023-… (CH Consumer Lag).
Per-VC KMinion instances: kminion-warpstream-replay, kminion-warpstream-logs, kminion-warpstream-traces.
K8s resources — container_* and kube_* for CPU, memory, restarts, HPA state.
Scope: namespace=~"capture-.*", pod=~"capture-.*".
See references/investigation-playbooks.md
for step-by-step workflows for common questions:
health checks, event loss, latency, Kafka backpressure, rate limiting, Redis, and cross-env comparison.
Take posthog/monitoring-capture-service from the repository into ~/.claude/skills for personal
use, or into .claude/skills inside a project.
The agent identifies a skill by the name field in its header. Two skills with the
same name cannot sit side by side — one of them will be ignored.