Agentsop Observability Setup

Observability Setup — Which Backend + One-Line Autolog

> *"Instrument before you need it. The cheapest debugging session is a trace you already have."*

This is an ENHANCE overlay. The local skills [[langsmith]], [[phoenix]] and [[mlflow]] each

teach one backend deeply. This skill sits one level up: it answers the question those skills

cannot — *which one, and how do I turn it on in a single line right now* — then hands off to the

chosen backend's own skill for depth.

1. 何时激活 (When to activate)

Activate at one of three moments — earlier is always cheaper:

| Trigger | Signal |

|---|---|

| Starting any LM project | First dspy.LM / ChatOpenAI / LlamaIndex / agent graph in the repo, and no tracing wired yet |

| Before first deploy | About to ship an LM feature to real users with no trace UI — the AP-15 trap (§6) |

| "Why did it do that?" with no traces | A bug surfaced, you reach for history, and there is nothing recorded — you are debugging blind |

| Multi-component pipeline | Retriever + reranker + LLM + tools; per-call printing ([[agentsop-prompt-history-inspect]]) is no longer enough |

| Cost / latency regression | Need aggregate token & latency dashboards across runs, not a single printout |

Do NOT activate when:

• You have already chosen a backend and just need its API — defer to [[langsmith]] / [[phoenix]] / [[mlflow]] directly.
• The task is classic ML experiment tracking with no LLM calls — that is plain MLflow, no decision needed.
• You only need to read one rendered prompt right now — that is [[agentsop-prompt-history-inspect]] (the cheaper first move).

The relationship to [[agentsop-prompt-history-inspect]]: that skill is the *reactive* first move (dump one

prompt, no setup). This skill is the *proactive* layer — wire persistent tracing so the next "why"

is answered by a trace that already exists, not a frantic re-run.

2. 核心心智模型 (Core mental model)

   Instrument BEFORE you need it
   ───────────────────────────────────────────────
   pick backend      one-line          verify         add eval
   by constraints →  autolog turns  →  a trace    →   hooks (judge,
   (stack/scale/      it on            actually        datasets,
    budget)           (~1 line)        landed          alerts)
   ───────────────────────────────────────────────
   cost of skipping each stage compounds: a missing trace at deploy
   becomes a multi-hour blind-debug later (AP-15, §6).

Three load-bearing ideas:

• Choosing is the hard part, not wiring. Every modern backend offers ~one-line autolog

(langsmith env vars, px.launch_app(), mlflow.<framework>.autolog()). The real cost is

picking the one that won't lock you in or under-serve you at scale (§4 decision table, §5 cases).

• The cheapest debugging is a trace you already have. A trace recorded proactively costs

near-zero; reconstructing one after a production bug costs hours and may be impossible (no repro).

• One backend, owned end-to-end. Don't wire two trace backends "to be safe" — pick one,

standardize on it, and let OpenTelemetry GenAI semantics keep you portable if you outgrow it (§7).

3. SOP (Standard operating procedure)

A five-step path. Each step gates the next.

Step 1 — Pick the backend by constraints

Run the decision table in §4 (OP-1). Inputs: stack (is it LangChain/LangGraph? OSS-only? already on MLflow?), scale (dev-only vs high-volume production), budget/hosting (managed-OK vs must-self-host). Output: exactly one backend.

Step 2 — One-line autolog

Wire the single call/env-var for the chosen backend (OP-2). Resist building a custom tracing layer first — autolog gets you a trace today; you can refine later.

Step 3 — Verify traces actually land

Run one real LM call, then open the trace UI and confirm the call appears with inputs, outputs, latency, and token counts (OP-3). A backend that is "configured" but shows no traces is the #1 silent failure (wrong project name, env var not exported, sampling at 0).

Step 4 — Add eval hooks

Once raw traces flow, attach what makes them actionable: an LLM-as-judge or rule evaluator, a dataset built from real traces for regression testing, and basic cost/latency alerts (OP-4). This is what turns "we have logs" into "we catch regressions before users do."

Step 5 — Hand off to the backend skill

Hand off to the chosen backend's own skill ([[langsmith]] / [[phoenix]] / [[mlflow]]) for the deep API once tracing and evals are flowing (OP-6).

Step gating, explicitly. Do not start Step *n+1* until Step *n* is observably true. The

most common failure is jumping from Step 2 (wired) straight to Step 4 (evals) without Step 3

(verify) — you build evaluators on top of a trace stream that was silently empty the whole time.

Each step has a one-line proof: Step 1 → a backend name written down; Step 2 → an autolog call

in the code; Step 3 → one trace visible in the UI with token counts; Step 4 → one evaluator

firing on that trace. If any proof is missing, you are not at that step yet.

4. 操作模型 (Operational model — Trigger / Action / Output / Evidence)

OP-1 · Backend decision table *(the core of this skill)*

• Trigger: Stage 1 — you have an LM project and no backend chosen yet.
• Action: Match your dominant constraint to a row. First matching row wins; ties broken by the "tie-breaker" column.

| If your situation is… | Choose | Why | Tie-breaker / caveat |

|---|---|---|---|

| Stack is LangChain / LangGraph (or LlamaIndex) and managed SaaS is acceptable | LangSmith → [[langsmith]] | First-class, zero-glue integration; "same team builds both" so tracing is native ([R1] swarnendu.de / langgraph AP-15) | Vendor lock-in; pricing scales with trace volume — see Case B (§5) |

| OSS-only / local-first, want OpenTelemetry, no vendor lock-in | Phoenix (Arize) → [[phoenix]] | OTel-based, self-hosted with SQLite/Postgres, framework-agnostic ([R1] phoenix SKILL frontmatter) | Self-host = you run the infra; managed Arize Cloud exists if you outgrow it |

| You are an ML-shop already running MLflow (model registry, experiments) | MLflow → [[mlflow]] | One pane of glass: LLM traces alongside existing runs/registry; mlflow.<fw>.autolog() ([R1] mlflow SKILL; dspy-sop deploy line) | LLM-trace UI is younger than purpose-built LLM tools; fine if MLflow is already your hub |

| Must self-host a purpose-built LLM-observability product (data-residency, privacy) | Langfuse | Self-hostable LLM-native tracing + evals; the common DSPy/CrewAI bolt-on ([R1] dspy-sop "LangFuse-style traces"; crewai obs) | Not a local skill here — install per Langfuse docs; OTel-compatible |

| Mixed stack, want maximum portability, framework churn expected | OpenTelemetry GenAI semantics under any of the above | Standard span schema → swap backends without re-instrumenting | More wiring; pick a concrete backend (Phoenix is OTel-native) to actually view spans |

• Output: Exactly one backend name + its skill to hand off to.
• Evidence: backend frontmatter in [[langsmith]]/[[phoenix]]/[[mlflow]]; langgraph-sop AP-15; llamaindex-sop "LangSmith / Phoenix for observability"; dspy-sop MLflow + LangFuse lines. See references/R1-source-evidence.md.

OP-2 · One-line autolog per backend

• Trigger: Stage 2 — backend chosen, turn it on.
• Action: Use the *minimal* turn-on for the chosen backend:

  # LangSmith — env vars only, no code change (auto-traces LangChain/LangGraph):
  #   export LANGSMITH_TRACING=true
  #   export LANGSMITH_API_KEY=...   export LANGSMITH_PROJECT=my-app
  # (raw SDK / non-LangChain: wrap calls with langsmith.wrappers.wrap_openai or @traceable)

  # Phoenix — launch local app + auto-instrument:
  import phoenix as px; px.launch_app()
  from phoenix.otel import register
  register(auto_instrument=True)        # picks up installed framework instrumentors

  # MLflow — one autolog call per framework:
  import mlflow
  mlflow.set_tracking_uri("http://localhost:5000")
  mlflow.dspy.autolog()                 # or mlflow.langchain / mlflow.openai / mlflow.crewai

• Output: Traces start flowing on the next LM call — no per-call instrumentation written by hand.
• Evidence: [[langsmith]]/[[phoenix]]/[[mlflow]] quick-start sections; dspy-sop mlflow.dspy.log_model deploy line; prompt-history-inspect op-001 (mlflow.dspy.autolog).

OP-3 · Verify a trace actually landed

• Trigger: Stage 3 — autolog wired, before trusting it.
• Action: Run one real call, open the UI (LangSmith project / px localhost / MLflow :5000), confirm the run shows inputs + outputs + latency + token count. If empty: check project name, that the env var is exported in *this* process, and that sampling ≠ 0.
• Output: One verified trace = the backend is real, not just configured.
• Evidence: AP "configured-but-empty" is the most common silent failure; verify-before-trust is standard SOP (references/R2).

OP-4 · Add eval hooks on top of raw traces

• Trigger: Stage 4 — traces flow; make them actionable.
• Action: Attach (a) an evaluator — LLM-as-judge (Phoenix) or dataset evaluators (LangSmith) or logged metrics (MLflow); (b) a dataset built from real traces for regression testing; (c) cost/latency alerts. Defer the deep API to the chosen backend's skill.
• Output: Traces become a regression-catching system, not just a log.
• Evidence: [[langsmith]]/[[phoenix]] evaluation sections; llamaindex-sop "eval loop is what surfaces failures"; dspy-sop metric pattern.

OP-5 · What to trace (minimum payload)

• Trigger: Deciding what each span should capture.
• Action: At minimum capture, per LM call: rendered prompt, response, model id, latency, token in/out, and the upstream context (for RAG: query + retrieved_nodes + scores + index_id; for agents: tool name + args + observation).
• Output: A span that can answer "why did it do that" without a re-run.
• Evidence: llamaindex-sop line 159 ("tracing captures query + retrieved_nodes + scores + index_id + LLM prompt for every failure"); prompt-history-inspect §what-to-dump.

Quick decision: how much to capture. Default to the minimum payload above on *every* span,

plus full request/response bodies on errors and a sampled fraction of successes. Capturing

full bodies on 100% of high-volume traffic is the over-instrumenting trap (§6) and the cost

driver in Case B. Capturing *less* than the minimum recreates AP-15 in slow motion — you have

traces, but they cannot answer the "why."

OP-6 · Handoff to backend skill

• Trigger: Backend wired and verified; need depth (custom evaluators, datasets, dashboards, alerting).
• Action: Stop here and invoke the chosen backend's own skill — [[langsmith]], [[phoenix]], or [[mlflow]]. This overlay deliberately does not duplicate their APIs.
• Output: Clean separation — decision/wiring here, depth there.
• Evidence: overlay design; this skill's boundary (§6).

5. 困境决策案例 (Dilemma cases / worked examples)

Case A — Vendor lock-in: LangSmith vs OSS Phoenix

Situation: A team building on LangGraph wants tracing before launch. LangSmith is one env-var

away and natively integrated; Phoenix is OSS/OTel but needs self-hosting.

Tension: Speed-to-trace + native integration (LangSmith) vs no vendor lock-in + data control

(Phoenix). The langgraph-sop is explicit that LangSmith integration is first-class — "the same team

builds both" ([R1] AP-15).

Resolution: If you are LangChain/LangGraph-first and SaaS is acceptable, take LangSmith

now — the integration tax of Phoenix is real and AP-15 warns against shipping with *no* tracing far

more loudly than against the lock-in. If data must stay in-house or you have a mixed/OSS stack,

take Phoenix and accept the self-host cost. The wrong move is to dither and ship with neither

(AP-15). Either choice beats no choice; OTel semantics (§7) keep migration possible.

Case B — Trace volume at scale (the Replit problem)

Situation: A product graduates from dev to high traffic. Per-trace pricing on a managed backend

turns "nice dashboards" into a line item that scales linearly with users; full-fidelity tracing of

every call becomes both expensive and noisy.

Tension: Full observability vs cost & signal-to-noise at production volume. High-volume LLM

products (the Replit-class case — agent products generating millions of LM calls) cannot afford to

trace 100% on a metered SaaS plan, but turning tracing *off* recreates AP-15.

Resolution: Don't binary-choose between "trace all" and "trace none." (1) Sample —

full-fidelity on errors and a small % of successes; (2) move to a self-hosted backend (Phoenix /

Langfuse) where marginal cost is infra, not per-trace fees, once volume crosses the break-even; (3)

keep eval datasets curated from sampled traces so regression coverage survives the sampling. Decide

the sampling/hosting policy *before* the bill or the noise forces a panicked migration — instrument

before you need it, but scale the instrumentation deliberately.

6. 反模式与边界 (Anti-patterns & boundaries)

Anti-patterns

• AP-15 · No observability in production *(the headline trap)*. Shipping an LM feature with no

trace UI. "Replaying a checkpoint locally only goes so far; production needs the trace UI"

([R1] langgraph-sop AP-15). Symptom: a user hits a bug, you have nothing to inspect, and you

cannot reproduce it. Fix: this skill's Stage 1–3 *before* deploy.

• Over-instrumenting. Wiring two trace backends "to be safe," tracing every internal function

as a span, or building a bespoke tracing layer before trying one-line autolog. Cost: noise,

double bills, maintenance burden. Fix: one backend, autolog first, add detail only where a real

question demands it.

• Configured-but-empty. Declaring victory after wiring autolog without verifying a trace landed

(OP-3). The env var wasn't exported in the running process, the project name is wrong, or sampling

is 0 — and you discover it only when you need a trace and there is none.

• Tracing the prompt but not the response / context. Truncation, refusal, and bad-retrieval

bugs live in the response and the upstream nodes, not the prompt (OP-5).

• Leaving full-fidelity tracing on at scale without a cost plan (Case B).

Boundaries (when this skill is the wrong tool)

• Backend already chosen → use [[langsmith]] / [[phoenix]] / [[mlflow]] directly; this overlay adds nothing.
• Need to read one prompt right now, no setup → [[agentsop-prompt-history-inspect]] is the cheaper move.
• Pure ML experiment tracking, no LLM calls → plain MLflow; there is no backend *decision* to make.
• Pre-call infra failure (auth/network/import) → fix infra; there is nothing to trace yet.

7. 跨框架对照 (Cross-backend comparison)

| Backend | Hosting | Stack fit | One-line turn-on | Lock-in | Local skill |

|---|---|---|---|---|---|

| LangSmith | Managed SaaS (self-host enterprise) | LangChain / LangGraph / LlamaIndex first-class | export LANGSMITH_TRACING=true (+ key + project) | High (proprietary) | [[langsmith]] |

| Phoenix (Arize) | Self-host (SQLite/Postgres) or Arize Cloud | OSS / any framework via OTel | px.launch_app() + register(auto_instrument=True) | Low (OSS, OTel) | [[phoenix]] |

| MLflow | Self-host / managed (Databricks) | ML-shops; framework-agnostic, registry + runs | mlflow.<framework>.autolog() | Low (Apache-2.0) | [[mlflow]] |

| Langfuse | Self-host (LLM-native) or cloud | DSPy / CrewAI bolt-on; privacy/data-residency | from langfuse import Langfuse + framework callback | Low (OSS) | *(no local skill — per docs)* |

| OpenTelemetry GenAI | Backend-agnostic span schema | Any — portability layer under the above | Instrument with OTel GenAI conventions, export to chosen backend | None | *(standard, not a product)* |

Reading the table: pick by the *dominant* constraint, not a feature checklist. Stack-coupling

(LangSmith), OSS/lock-in aversion (Phoenix), existing-MLflow-investment (MLflow), and

hosting/privacy (Langfuse) are the four forces; OpenTelemetry is the escape hatch that keeps any

choice reversible. Then hand off to the matching local skill for depth.

*Overlay note: this skill intentionally stops at decision + one-line wiring + verification + eval

hooks. For backend-specific APIs (custom evaluators, dataset management, dashboards, alerting),

defer to [[langsmith]], [[phoenix]], [[mlflow]]. For the reactive single-prompt dump, see

[[agentsop-prompt-history-inspect]].*