ancoleman/implementing-observability
Monitoring, logging, and tracing implementation using OpenTelemetry as the unified standard. Use when building production systems requiring visibility into performance, errors, and behavior. Covers OpenTelemetry (metrics, logs, traces), Prometheus, Grafana, Loki, Jaeger, Tempo, structured logging (structlog, tracing, slog, pino), and alerting.
npx skills add https://github.com/ancoleman/ai-design-components --skill implementing-observabilityImplement production-grade observability using OpenTelemetry as the 2025 industry standard. Covers the three pillars (metrics, logs, traces), LGTM stack deployment, and critical log-trace correlation patterns.
Use when:
Skip if:
OpenTelemetry is the CNCF graduated project unifying observability:
┌────────────────────────────────────────────────────────┐
│ OpenTelemetry: The Unified Standard │
├────────────────────────────────────────────────────────┤
│ │
│ ONE SDK for ALL signals: │
│ ├── Metrics (Prometheus-compatible) │
│ ├── Logs (structured, correlated) │
│ ├── Traces (distributed, standardized) │
│ └── Context (propagates across services) │
│ │
│ Language SDKs: │
│ ├── Python: opentelemetry-api, opentelemetry-sdk │
│ ├── Rust: opentelemetry, tracing-opentelemetry │
│ ├── Go: go.opentelemetry.io/otel │
│ └── TypeScript: @opentelemetry/api │
│ │
│ Export to ANY backend: │
│ ├── LGTM Stack (Loki, Grafana, Tempo, Mimir) │
│ ├── Prometheus + Jaeger │
│ ├── Datadog, New Relic, Honeycomb (SaaS) │
│ └── Custom backends via OTLP protocol │
│ │
└────────────────────────────────────────────────────────┘
Context7 Reference: /websites/opentelemetry_io (Trust: High, Snippets: 5,888, Score: 85.9)
Track system health and performance over time.
Metric Types: Counters (always increase), Gauges (up/down), Histograms (distributions), Summaries (percentiles).
Brief Example (Python):
from opentelemetry import metrics
meter = metrics.get_meter(__name__)
http_requests = meter.create_counter("http.server.requests")
http_requests.add(1, {"method": "GET", "status": 200})
Record discrete events with context.
CRITICAL: Always inject trace_id/span_id for log-trace correlation.
Brief Example (Python + structlog):
import structlog
from opentelemetry import trace
logger = structlog.get_logger()
span = trace.get_current_span()
ctx = span.get_span_context()
logger.info(
"processing_request",
trace_id=format(ctx.trace_id, '032x'),
span_id=format(ctx.span_id, '016x'),
user_id=user_id
)
See: references/structured-logging.md for complete configuration.
Track request flow across distributed services.
Key Concepts: Trace (end-to-end journey), Span (individual operation), Parent-Child (nested operations).
Brief Example (Python + FastAPI):
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor
app = FastAPI()
FastAPIInstrumentor.instrument_app(app) # Auto-traces all HTTP requests
See: references/opentelemetry-setup.md for SDK installation by language.
LGTM = Loki (Logs) + Grafana (Visualization) + Tempo (Traces) + Mimir (Metrics)
┌────────────────────────────────────────────────────────┐
│ LGTM Architecture │
├────────────────────────────────────────────────────────┤
│ │
│ ┌──────────────────────────────────────────────┐ │
│ │ Grafana Dashboard (Port 3000) │ │
│ │ Unified UI for Logs, Metrics, Traces │ │
│ └──────┬──────────────┬─────────────┬─────────┘ │
│ │ │ │ │
│ ▼ ▼ ▼ │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
│ │ Loki │ │ Tempo │ │ Mimir │ │
│ │ (Logs) │ │ (Traces) │ │(Metrics) │ │
│ │Port 3100 │ │Port 3200 │ │Port 9009 │ │
│ └────▲─────┘ └────▲─────┘ └────▲─────┘ │
│ │ │ │ │
│ └──────────────┴─────────────┘ │
│ │ │
│ ┌───────▼────────┐ │
│ │ Grafana Alloy │ │
│ │ (Collector) │ │
│ │ Port 4317/8 │ ← OTLP gRPC/HTTP │
│ └───────▲────────┘ │
│ │ │
│ OpenTelemetry Instrumented Apps │
│ │
└────────────────────────────────────────────────────────┘
Quick Start: Run examples/lgtm-docker-compose/docker-compose.yml for a complete LGTM stack.
See: references/lgtm-stack.md for production deployment guide.
The Problem: Logs and traces live in separate systems. You see an error log but can't find the related trace.
The Solution: Inject trace_id and span_id into every log record.
import structlog
from opentelemetry import trace
logger = structlog.get_logger()
span = trace.get_current_span()
ctx = span.get_span_context()
logger.info(
"request_processed",
trace_id=format(ctx.trace_id, '032x'), # 32-char hex
span_id=format(ctx.span_id, '016x'), # 16-char hex
user_id=user_id
)
use tracing::{info, instrument};
#[instrument(fields(user_id = %user_id))]
async fn process_request(user_id: u64) -> Result<Response> {
// trace_id/span_id automatically included
info!(user_id = user_id, "processing request");
Ok(result)
}
See: references/trace-context.md for Go and TypeScript patterns.
{job="api-service"} |= "trace_id=4bf92f3577b34da6a3ce929d0e0e4736"
Decision Tree:
Bootstrap Script:
python scripts/setup_otel.py --language python --framework fastapi
Manual (Python):
pip install opentelemetry-api opentelemetry-sdk \
opentelemetry-instrumentation-fastapi \
opentelemetry-exporter-otlp
See: references/opentelemetry-setup.md for Rust, Go, TypeScript installation.
Docker Compose (development):
cd examples/lgtm-docker-compose
docker-compose up -d
# Grafana: http://localhost:3000 (admin/admin)
# OTLP: localhost:4317 (gRPC), localhost:4318 (HTTP)
See: references/lgtm-stack.md for production Kubernetes deployment.
See: references/structured-logging.md for complete setup (Python, Rust, Go, TypeScript).
See: references/alerting-rules.md for Prometheus and Loki alert patterns.
OpenTelemetry auto-instruments popular frameworks:
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor
app = FastAPI()
FastAPIInstrumentor.instrument_app(app) # Auto-trace all HTTP requests
Supported: FastAPI, Flask, Django, Express, Gin, Echo, Nest.js
See: references/opentelemetry-setup.md for framework-specific setup.
from opentelemetry import trace
tracer = trace.get_tracer(__name__)
with tracer.start_as_current_span("fetch_user_details") as span:
span.set_attribute("user_id", user_id)
user = await db.fetch_user(user_id)
span.set_attribute("user_found", user is not None)
from opentelemetry.trace import Status, StatusCode
with tracer.start_as_current_span("process_payment") as span:
try:
result = process_payment(amount, card_token)
span.set_status(Status(StatusCode.OK))
except PaymentError as e:
span.set_status(Status(StatusCode.ERROR, str(e)))
span.record_exception(e)
raise
See: references/trace-context.md for background job tracing and context propagation.
# Test log-trace correlation
# 1. Make request to your app
# 2. Copy trace_id from logs
# 3. Query in Grafana: {job="myapp"} |= "trace_id=<TRACE_ID>"
# Validate metrics
python scripts/validate_metrics.py
See: examples/fastapi-otel/ for complete integration.
Setup Guides:
references/opentelemetry-setup.md - SDK installation (Python, Rust, Go, TypeScript)references/structured-logging.md - structlog, tracing, slog, pino configurationreferences/lgtm-stack.md - LGTM deployment (Docker, Kubernetes)references/trace-context.md - Log-trace correlation patternsreferences/alerting-rules.md - Prometheus and Loki alert templatesExamples:
examples/fastapi-otel/ - FastAPI + OpenTelemetry + LGTMexamples/axum-tracing/ - Rust Axum + tracing + LGTMexamples/lgtm-docker-compose/ - Production-ready LGTM stackScripts:
scripts/setup_otel.py - Bootstrap OpenTelemetry SDKscripts/generate_dashboards.py - Generate Grafana dashboardsscripts/validate_metrics.py - Validate metric namingDon't:
Do:
Take ancoleman/implementing-observability from the repository into ~/.claude/skills for personal
use, or into .claude/skills inside a project.
The agent identifies a skill by the name field in its header. Two skills with the
same name cannot sit side by side — one of them will be ignored.
The instructions reference pip.
Without those the skill loads but fails at the first command.