Observability Agent Skill

Observability

Decide *what to measure* so a system can be seen, alerted on, and debugged in

production. Getting this wrong is failure mode #6 — ignoring failure: a design

that works on the whiteboard but goes dark under load, where the first signal of

an outage is a user complaint instead of a page.

When to reach for this

Any production design needs an answer to "how would we know this broke, and how

fast?" Reach for this when defining what the system measures, what pages a human,

what an acceptable level of service is (SLO), or how a request is traced across

services. It is the design move that makes every *other* block's stress section

real — you cannot mitigate a thundering herd or a hot shard you can't see.

When NOT to

Do not build a full metrics-logs-traces stack for a prototype or an internal tool

with no users to disappoint (YAGNI) — a health check and error logging are

enough. Do not invent SLOs nobody will defend, or wire alerts before knowing the

symptom that matters; an alert with no owner and no runbook is noise that trains

the team to ignore pages. This skill owns *what* to measure and alert on; the

high-volume log pipeline (collect → buffer → ship → index → retain) lives in

distributed-logging — summarize and link, don't rebuild it here.

Clarify first

• What is "healthy" from a user's view? The symptom that defines a bad

experience (slow checkout, failed upload) — alerts target this, not CPU.

• SLO target and window? e.g. 99.9% of requests < 300 ms over 30 days. This

sets the error budget and the alert thresholds. (→ back-of-the-envelope for the nines.)

• Request volume and cardinality? QPS drives metric/trace sample rates;

high-cardinality labels (user ID, URL) blow up a metrics store.

• Is this request-driven or resource-driven? Picks RED (services) vs USE

(CPU/disk/queues) as the measurement frame.

• Multi-service request path? If a request crosses services, tracing earns

its keep; a single service may not need it yet.

The options

The three pillars (complementary, not either/or):

• Metrics — cheap numeric time series (counters, gauges, histograms). Use for

dashboards, trend analysis, and *alerting* — the always-on signal.

• Logs — discrete events with context. Use for debugging the specific failure

after an alert fires. (The pipeline that moves them is distributed-logging.)

• Traces — one request's journey across services with timing per hop. Use to

find *which* service or dependency is the latency/error source.

What to measure (pick a frame per component):

• RED (request-driven services): Rate, Errors, Duration. Use for

APIs, web tiers, anything serving requests.

• USE (resources): Utilization, Saturation, Errors. Use for CPU,

memory, disk, connection pools, queues.

• Four golden signals (latency, traffic, errors, saturation) — the superset;

use as the default service dashboard.

Health checks (the signal, consumed by load-balancing/orchestrator):

• Liveness — is the process alive / not deadlocked? Failure ⇒ *restart*. Keep

it cheap; don't check dependencies. Use to recover stuck processes.

• Readiness — can this instance serve traffic *now* (deps reachable, warmup

done)? Failure ⇒ *pull from rotation, don't restart*. Use to gate cold/struggling instances.

Alerting:

• Symptom-based (SLO burn) — page when the user-facing SLO is at risk. Use as

the default; it is actionable and low-noise.

• Cause-based (resource thresholds) — ticket/warn on CPU, disk, saturation.

Use for capacity planning, not paging.

Trade-offs

| Option | What it solves | What it worsens | Change it when |

|---|---|---|---|

| Metrics | Cheap, always-on alerting + trends | No per-request detail; high-cardinality labels explode storage/cost | You need to debug a *specific* request → add traces/logs |

| Logs | Rich context for debugging an incident | Volume + cost; needs the distributed-logging pipeline to scale | Volume is unmanageable → sample, or shift detail to metrics |

| Traces | Pinpoints the slow/failing hop across services | Instrumentation effort; sampling needed at high QPS | Single-service or low volume → defer; metrics suffice |

| RED | Right frame for request services | Misses resource exhaustion that hasn't yet hurt requests | Component is a resource (queue, disk) → use USE |

| USE | Catches saturation before it hurts users | Resource-centric, not user-centric; can be noisy | Alerting on it → switch to symptom/SLO-based |

| Liveness check | Recovers deadlocked processes | Too aggressive ⇒ restart loops, mask real bugs | Restarts hide a crash-loop → fix readiness/root cause |

| Readiness check | Keeps cold/broken instances out of rotation | Flapping if it checks flaky deps ⇒ capacity yo-yo | All instances fail readiness together → it's a dep outage |

| SLO/error budget | Ties alerting + release pace to user pain | Effort to define + defend; wrong SLO misleads | Budget never burns (too loose) or always burns (too tight) |

| Symptom alerting | Low-noise, actionable pages | Slightly slower to localize root cause | Need faster localization → add cause-based *tickets* (not pages) |

Behavior under stress

Observability is most needed exactly when it's most likely to break or mislead.

• The monitoring amplifies the outage. Synchronous logging on the request path

turns a slow log backend into request latency. Per-request trace export with no

sampling melts the collector under a spike. Keep telemetry async and sampled so

it degrades the *signal*, never the service.

• Health-check stampede. Aggressive liveness probes restart instances that are

merely slow, and readiness flaps yank capacity mid-spike — making the spike

worse. The gating/recovery behavior is owned by load-balancing and

resilience-failure; this block owns defining a *stable* signal (sane

thresholds, consecutive-failure counts, deps in readiness not liveness).

• Cardinality blow-up. A label like user_id or raw URL multiplies time series

into the millions and OOMs the metrics store under load — drop or bucket

high-cardinality dimensions.

• Alert storm. One root cause (a DB outage) fires fifty correlated alerts; the

on-call drowns. Alert on the user-facing symptom; group/inhibit downstream alerts.

• Blind spot on recovery. After an incident, dashboards must show whether the

fix worked and whether recovery traffic is overwhelming a cold tier.

Monitor: the four golden signals per service (rate, errors, latency p50/p95/p99,

saturation), SLO error-budget burn rate, healthy-host count and probe flap rate,

and the telemetry pipeline's own lag/drop rate (watch the watcher).

How to apply

• Clarify the inputs — pin the user-facing symptom, the SLO target + window,

request volume/cardinality, and whether the path is multi-service (see *Clarify

first*). No SLO yet? Define the symptom first; the number follows.

• Pick the pillars and frame from the trade-off table — metrics for alerting

always; RED for services and USE for resources; add traces only when a request

crosses services or volume justifies it.

• Set the key knobs — define liveness vs readiness endpoints (deps in

readiness, not liveness), consecutive-failure thresholds, trace sample rate, and

the SLO + error-budget policy. Drop high-cardinality labels up front.

• Stress-test the choice — walk *Behavior under stress*: confirm telemetry is

async/sampled, probes won't stampede, cardinality is bounded, and alerts group

by root cause so one outage isn't fifty pages.

• Size it with numbers — sanity-check metric series count and retention,

trace/log sample volume against the QPS, and that the SLO's nines match the

architecture's redundancy (→ *Numbers that matter*).

• Pick a provider — default to the generic recipe; open a provider file only

if the user named a cloud (see *Choosing a provider*).

• Show the trade-off in the answer — give a `Decision | Solves | Worsens |

Change it when…` table with one row per choice you made here (e.g. the

option, its store, its failure policy); the last column is a concrete trigger

(a number or observable signal). Then name what breaks first.

Dos and don'ts

Do

• Alert on user-facing symptoms (SLO burn), and make every page actionable with a runbook.
• Put dependency checks in *readiness* and keep *liveness* cheap and dependency-free.
• Emit telemetry asynchronously and sampled so it never adds latency or melts the collector.
• Define an SLO + error budget you will actually defend, and tie release pace to it.
• Standardize a per-service dashboard on the four golden signals.

Don't

• Don't page on causes (CPU > 80%) — ticket those; pages are for user pain.
• Don't use high-cardinality labels (user ID, raw URL) as metric dimensions.
• Don't check downstream dependencies in liveness — you'll restart-loop the fleet.
• Don't build a full stack for a prototype, or wire alerts no one owns (YAGNI / alert fatigue).
• Don't reimplement the log pipeline here — that's distributed-logging.

Numbers that matter

The SLO sets everything else: 99.9% over 30 days allows ~43 minutes of budget; if

the architecture's redundancy can't deliver those nines, the SLO is fiction (→

back-of-the-envelope for the availability-nines table). Alert on p95/p99, not

averages — averages hide tail pain. Sample traces (often 1–10% at high QPS) so cost

scales sub-linearly with traffic. Watch metric cardinality (series ≈ product of

label values): one unbounded label can mean millions of series. Don't restate the

latency/QPS/nines tables — they live in back-of-the-envelope.

Interface sketch

Telemetry has contracts worth pinning down. A metric: name + label set +

type (http_requests_total{service,method,status} counter) — labels are

low-cardinality. A health endpoint: GET /livez → 200/503 (process only),

GET /readyz → 200/503 (deps + warmup), with version/build info in the body for

debugging. A trace context: a trace_id + span_id propagated on every

inbound/outbound call (e.g. W3C traceparent header) so traces and logs correlate.

Choosing a provider

Default to the generic recipe above (Prometheus + Grafana + Loki + Jaeger, glued

by OpenTelemetry). If the user names a cloud, read

references/providers/<provider>.md for the managed-service mapping, quotas/limits,

and provider-specific trade-offs. If no file exists for that provider, the generic

recipe is the answer.

Diagram

To visualize the telemetry flow (app → OpenTelemetry SDK → collector → metrics /

logs / traces backends → dashboard + alertmanager) or the alert-to-page path, use

the in-plugin architecture-diagram skill. Telemetry export uses dashed arrows to

show it is off the request path.

Related building blocks

• distributed-logging — *owned-concept lives in* there: the high-volume log

pipeline (collect → buffer → ship → index → retain). This skill decides *what* to

log and alert on; that skill decides *how* to move and store it at scale.

• resilience-failure — *pairs with* this: alerts and SLO burn trigger graceful

degradation and circuit breaking, and the health-check signal defined here is the

input its failover and load-balancing health-gating consume.

• scaling-evolution — *feeds into* it: saturation metrics and the golden signals

reveal the next bottleneck that justifies the next scaling step.

• back-of-the-envelope — *feeds into* this skill: it supplies the availability

nines and QPS that turn an SLO and sample rates into real numbers.

• system-design — *owned-concept lives in* the orchestrator: the reasoning loop,

the trade-off method, and the ten failure modes (this block defends #6).

References

• references/deep-dive.md — pillar mechanics (histograms vs gauges, push vs

pull, sampling), SLI/SLO/error-budget math and burn-rate alerts, RED/USE in

detail, health-check tuning, and alert design. Read when designing the

observability layer in depth.

• references/providers/{generic,aws,azure,gcp}.md — service mappings, limits,

and pitfalls per environment.