Observability Agent Skill

Observability

You are wiring the *inside* view of a service: when something breaks at 3am, an engineer must be able to answer "what happened, where, and why" from telemetry alone — without adding a console.log and redeploying into the fire. This skill emits a concrete artifact: SDK init code, instrumentation (spans/metrics/structured logs), a Collector config, and alert rules that the instrumentation makes possible. The outside-in half — is it up, who gets paged — is ../monitoring/SKILL.md.

The one rule

Every signal carries the same correlation identity: trace_id, service.name, deployment.environment. A log line you cannot pivot to its trace, or a spiking metric you cannot pivot to an exemplar span, doubles your mean-time-to-resolution — you are back to grepping. Three signals that don't share keys are three disconnected tools; three that do are one queryable system. Set the resource once at SDK init, inject trace_id/span_id into every log, and never emit a metric you can't tie back to a service and environment.

Start from the operator's questions

Before choosing signals, write two to four questions on-call must answer during the likely incident. For example: “Are payment retries recovering?”, “Which dependency and failure class drives exhaustion?”, “Can one payment be charged twice?”, “Which customer-visible operations need intervention now?” Then assign the cheapest signal: metrics say that/how much, traces show where/causal path, logs explain why for this event. If a proposed event or label answers none of the questions, do not emit it.

The three pillars — when each earns its place

Don't emit all three of everything. Each signal answers a different question at a different cost.

| Signal | Answers | Cost | Alert on it? | Main gotcha |

|---|---|---|---|---|

| Logs | "what exactly happened in this one event" | high per-event, cheap to skip | rarely (noisy) | high-cardinality fields belong in the *body*, not in stream labels |

| Metrics | "what's the rate/aggregate over time" | cheap, pre-aggregated | yes — this is your alert source | cardinality explosion if a label is unbounded |

| Traces | "what was the causal path across hops, and where did time go" | medium; sample it | indirectly (via derived RED metrics) | one giant span = no causality; sample or you pay for noise |

The "fourth pillar," continuous profiling (CPU/heap flame graphs over time), is now a first-class OTel signal — reach for it only when traces say "the time is inside *this* function" and you need to know which line.

Architecture

Instrument once, route anywhere. The app talks OTLP to a Collector; the Collector fans the firehose out to backends.

  ┌─────────────┐   OTLP/gRPC :4317        ┌───────────────────────────┐
  │  app + SDK  │   OTLP/HTTP :4318  ───▶   │       OTel Collector       │
  │ (resource:  │   /v1/traces             │  receivers → processors    │
  │ service.name│   /v1/metrics            │  → exporters (per signal)  │
  │  +env+ver)  │   /v1/logs               │  wired in service.pipelines│
  └─────────────┘                          └───────────────────────────┘
                                                 │        │        │
                                            logs │ traces │ metrics│
                                                 ▼        ▼        ▼
                                              Loki     Tempo   Mimir/Prom   (+ Grafana to view)
                                                  └── or a single vendor: Datadog / Honeycomb ──┘

OTLP is the wire format: gRPC on 4317 (TLS + gzip by default), HTTP on 4318 with per-signal paths /v1/traces, /v1/metrics, /v1/logs. Always export to a Collector, never straight to the vendor. Why: the Collector gives you one place to batch (fewer round-trips), retry (survive a backend blip), redact PII, and swap or add a backend without redeploying the app. App SDKs should be dumb pipes; policy lives in the Collector.

Instrument in the right order: auto first, manual second

Never hand-roll a span for something auto-instrumentation already covers (HTTP servers, DB clients, queues). You will miss edges and waste effort. Turn on zero-code instrumentation, confirm traces flow, *then* add manual spans only where your business logic lives.

# Node — zero-code, no app changes. SDK is 2.0+; the register hook is compatible.
npm i @opentelemetry/api @opentelemetry/auto-instrumentations-node
OTEL_SERVICE_NAME=checkout-api \
OTEL_RESOURCE_ATTRIBUTES=deployment.environment=prod,service.version=1.4.2 \
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 \
node --require '@opentelemetry/auto-instrumentations-node/register' app.js

# Python — zero-code via the launcher; it patches known libraries on import.
pip install opentelemetry-distro opentelemetry-exporter-otlp
opentelemetry-bootstrap -a install
OTEL_SERVICE_NAME=checkout-api \
OTEL_RESOURCE_ATTRIBUTES=deployment.environment=prod,service.version=1.4.2 \
opentelemetry-instrument python app.py

Then add a manual span only around a meaningful business operation — and give it attributes and a status, or it tells you nothing:

// Bad — a span with no attributes and no status. You learn that "something ran."
const span = tracer.startSpan('work');
await chargeCard(order);
span.end();

// Good — named for the business op, carries the inputs you'd filter by, records outcome.
const { trace, SpanStatusCode } = require('@opentelemetry/api');
const tracer = trace.getTracer('checkout');
await tracer.startActiveSpan('charge_card', async (span) => {
  span.setAttribute('order.id', order.id);          // searchable dimension
  span.setAttribute('payment.provider', 'stripe');
  try {
    await chargeCard(order);
    span.setStatus({ code: SpanStatusCode.OK });
  } catch (err) {
    span.recordException(err);                        // attaches the stack as an event
    span.setStatus({ code: SpanStatusCode.ERROR, message: err.message });
    throw err;
  } finally {
    span.end();                                       // a span you never end leaks forever
  }
});

Per-stack init (Node SDK 2.0 manual setup, Go SDK, context propagation across HTTP/queue hops, the GenAI span template) lives in references/instrumentation-recipes.md.

Resource and semantic conventions

Use the standard attribute names; never invent bespoke keys. The whole correlation story and every prebuilt backend dashboard assume http.*, db.*, gen_ai.*, service.*. A homegrown mycompany.endpoint attribute is invisible to every tool that expects http.route.

• Set on the resource, once: service.name (required — unset means telemetry lands as unknown_service), service.version, deployment.environment.
• Kubernetes attributes (k8s.*) reached release candidate (2026-03); DB conventions are on their 2nd RC. Prefer the standard names even while an area is still stabilizing.
• For LLM calls, GenAI conventions exited experimental for client spans (early 2026) — use gen_ai.* so the same span feeds your spend view:

// LLM call as a span: model + token attrs. These attributes feed ../cost-tracking/SKILL.md.
await tracer.startActiveSpan('chat gpt-4o', async (span) => {
  span.setAttribute('gen_ai.system', 'openai');
  span.setAttribute('gen_ai.request.model', 'gpt-4o');
  const res = await openai.chat.completions.create({ /* ... */ });
  span.setAttribute('gen_ai.usage.input_tokens', res.usage.prompt_tokens);
  span.setAttribute('gen_ai.usage.output_tokens', res.usage.completion_tokens);
  span.end();
});

This skill emits the *signal* (token attributes on a span). Turning those tokens into a dollar figure and a budget is ../cost-tracking/SKILL.md.

Structured, correlated logs

Logs are JSON, carry the trace context, and never carry PII. Free-text print lines cost you twice: you can't query them, and you can't jump from the log to its trace.

# Bad — unstructured, unsearchable, unlinkable to a trace.
print("charged user " + email + " amount " + str(amount))

# Good — structured, leveled, correlated, no PII (id not email).
import logging, json
from opentelemetry import trace

def log_charge(order_id, amount):
    ctx = trace.get_current_span().get_span_context()
    logging.info(json.dumps({
        "event": "charge.succeeded",
        "level": "info",
        "order_id": order_id,            # an opaque id, not the customer's email
        "amount_cents": amount,
        "trace_id": format(ctx.trace_id, "032x"),  # ← the pivot back to the trace
        "span_id": format(ctx.span_id, "016x"),
    }))

Level discipline: error = a human should look, warn = degraded but handled, info = business milestones, debug = off in prod. If everything is error, nothing is.

Metrics that earn an alert

RED for request-driven services, USE for finite resources. Alerts come from metrics — logs and traces are for *investigating* the alert, not firing it.

• RED (per service/endpoint): Rate (requests/s), Errors (failed/s), Duration (latency as a *histogram*, so you can read p50/p95/p99 — never a single average, which hides the tail).
• USE (per resource — CPU, pool, disk): Utilization, Saturation (queue depth / waiting), Errors.

The cardinality rule — this is the #1 way an observability stack falls over. A metric's total series count is the product of its label cardinalities. Put an unbounded value on a label and you create a near-infinite series count; the time-series database (Prometheus/Mimir) is most often restarted because of exactly this. Loki indexes *labels only, not log contents* — so the same rule binds its stream labels.

# Bad — user_id is unbounded; 5M users = 5M series per metric. OOMs the TSDB.
http_requests_total{route="/checkout", user_id="u_8f3a...", status="200"}

# Good — only bounded, low-cardinality dimensions on the metric.
http_requests_total{route="/checkout", method="POST", status="200"}
# Need to slice by user? That's a trace attribute or a log field, never a metric label.

The Collector config

A minimal valid pipeline. An exporter is inert until it appears in a pipeline — defining one under exporters: does nothing on its own.

# otel-collector.yaml — Collector v0.153.0 shape
receivers:
  otlp:
    protocols:
      grpc: { endpoint: 0.0.0.0:4317 }
      http: { endpoint: 0.0.0.0:4318 }

processors:
  memory_limiter:                 # first line of defense: shed load before OOM
    check_interval: 1s
    limit_percentage: 80
  batch: {}                       # batch before export — fewer, bigger round-trips
  redaction:                      # strip PII before it leaves your network
    allow_all_keys: true
    blocked_values: ["[0-9]{13,16}", "\\b[\\w.]+@[\\w.]+\\b"]  # PANs, emails

exporters:
  otlphttp/traces:  { endpoint: http://tempo:4318 }
  otlphttp/logs:    { endpoint: http://loki:3100/otlp }
  otlphttp/metrics: { endpoint: http://mimir:9009/otlp }

service:
  pipelines:
    traces:  { receivers: [otlp], processors: [memory_limiter, redaction, batch], exporters: [otlphttp/traces] }
    logs:    { receivers: [otlp], processors: [memory_limiter, redaction, batch], exporters: [otlphttp/logs] }
    metrics: { receivers: [otlp], processors: [memory_limiter, batch],            exporters: [otlphttp/metrics] }

Tail sampling, gateway-vs-agent topology, multi-backend fan-out (LGTM and a vendor in parallel), and resourcedetection live in references/collector-config.md. Shipping the Collector as a container or in CI is ../docker/SKILL.md.

Alerts the telemetry now enables

The instrumentation above makes these *possible* — defining them is your job; routing the page to a human is ../monitoring/SKILL.md.

• SLO burn-rate (multi-window) — alert when you're spending the error budget too fast, using a fast window (e.g. 5m) AND a slow window (e.g. 1h) so a brief blip doesn't page but a sustained burn does. This fires on *impact*, not on traffic.
• Latency SLO — p99 of the duration histogram over budget for a sustained window.
• Saturation — the USE "S": queue depth / pool waiters climbing, the leading indicator before errors appear.

Anti-patterns

| Anti-pattern | Why it bites | Do instead |

|---|---|---|

| Unbounded label (user_id, request_id, email) on a metric or Loki stream | Cardinality explosion → TSDB OOM/restart, cost blowup | Keep labels low-cardinality; put the high-cardinality field on a span attribute or log body |

| Logging PII / secrets (email, card, token) | Compliance breach + the leak is now in every log backend | Log opaque ids; redact in the Collector before export |

| 100% trace sampling in prod, no policy | Pay to store noise; backend throttles and drops the traces you needed | Head/tail sampling — keep all errors + slow traces, sample the rest |

| App exports straight to the vendor, no Collector | Can't batch, retry, redact, or swap backends without a redeploy | Always route through a Collector |

| Instrument everything before deciding the question | Noise with no signal; nobody opens the dashboard | Start from "what would I ask during an incident," instrument that path |

| Alert on a raw error *count* | Fires on traffic spikes, silent during a low-traffic outage | Alert on error *rate* / SLO burn |

| One giant span per request (or a thousand contentless ones) | No causality, or context with no detail — both useless | Span per meaningful operation, each with attributes + status |

Verify

Run scripts/verify.sh against the directory holding your Collector config + SDK init. It checks the config is valid, that every defined exporter is actually wired into a pipeline (the classic "defined but unused" footgun), that service.name is set, and warns on high-cardinality metric labels. It is read-only and exits 0 when there's nothing to check.

Then prove the wire with one safe induced failure in a test/staging path. Confirm the expected error-rate metric changes, the trace records the failing operation and error status, and the structured log carries the same trace_id without PII. Query the backend/Collector output; “the instrumentation code ran” is not evidence that usable telemetry arrived. Live paging and escalation proof remains ../monitoring/SKILL.md.