Databricks Mlflow Evaluation

MLflow 3 GenAI Evaluation

Scope vs upstream mlflow/skills

The OSS mlflow/skills repo ships agent-evaluation and related skills (instrumenting-with-mlflow-tracing, analyze-mlflow-trace, retrieving-mlflow-traces, querying-mlflow-metrics) that cover the generic MLflow GenAI evaluation workflow — mlflow.genai.evaluate(), scorers/judges, datasets, tracing setup, and the 5-step evaluation loop.

This skill layers Databricks-specific patterns on top of that workflow rather than restating it. Use this skill when you need any of:

• Unity Catalog trace ingestion — production traces written into UC tables, log-based monitoring (patterns-trace-ingestion.md).
• MemAlign judge alignment via UC SME labeling sessions — aligning custom judges against domain-expert feedback collected in Databricks (patterns-judge-alignment.md).
• optimize_prompts() GEPA loop — Databricks' automated prompt-optimization driver running on a UC dataset (patterns-prompt-optimization.md).
• Databricks-flavored scorer/dataset patterns — UC-table-backed datasets, tagging traces in the Databricks UI for inclusion (patterns-datasets.md, patterns-scorers.md).

For everything else — generic mlflow.genai.evaluate() calls, scorer authoring patterns, dataset creation outside Databricks, MLflow tracing setup that isn't UC-table-bound — the upstream mlflow/skills/agent-evaluation skill is the canonical source and is kept current by the MLflow team.

Before Writing Any Code

• Read GOTCHAS.md - 15+ common mistakes that cause failures
• Read CRITICAL-interfaces.md - Exact API signatures and data schemas

End-to-End Workflows

Follow these workflows based on your goal. Each step indicates which reference files to read.

Workflow 1: First-Time Evaluation Setup

For users new to MLflow GenAI evaluation or setting up evaluation for a new agent.

| Step | Action | Reference Files |

|------|--------|-----------------|

| 1 | Understand what to evaluate | user-journeys.md (Journey 0: Strategy) |

| 2 | Learn API patterns | GOTCHAS.md + CRITICAL-interfaces.md |

| 3 | Build initial dataset | patterns-datasets.md (Patterns 1-4) |

| 4 | Choose/create scorers | patterns-scorers.md + CRITICAL-interfaces.md (built-in list) |

| 5 | Run evaluation | patterns-evaluation.md (Patterns 1-3) |

Workflow 2: Production Trace -> Evaluation Dataset

For building evaluation datasets from production traces.

| Step | Action | Reference Files |

|------|--------|-----------------|

| 1 | Search and filter traces | patterns-trace-analysis.md (MCP tools section) |

| 2 | Analyze trace quality | patterns-trace-analysis.md (Patterns 1-7) |

| 3 | Tag traces for inclusion | patterns-datasets.md (Patterns 16-17) |

| 4 | Build dataset from traces | patterns-datasets.md (Patterns 6-7) |

| 5 | Add expectations/ground truth | patterns-datasets.md (Pattern 2) |

Workflow 3: Performance Optimization

For debugging slow or expensive agent execution.

| Step | Action | Reference Files |

|------|--------|-----------------|

| 1 | Profile latency by span | patterns-trace-analysis.md (Patterns 4-6) |

| 2 | Analyze token usage | patterns-trace-analysis.md (Pattern 9) |

| 3 | Detect context issues | patterns-context-optimization.md (Section 5) |

| 4 | Apply optimizations | patterns-context-optimization.md (Sections 1-4, 6) |

| 5 | Re-evaluate to measure impact | patterns-evaluation.md (Pattern 6-7) |

Workflow 4: Regression Detection

For comparing agent versions and finding regressions.

| Step | Action | Reference Files |

|------|--------|-----------------|

| 1 | Establish baseline | patterns-evaluation.md (Pattern 4: named runs) |

| 2 | Run current version | patterns-evaluation.md (Pattern 1) |

| 3 | Compare metrics | patterns-evaluation.md (Patterns 6-7) |

| 4 | Analyze failing traces | patterns-trace-analysis.md (Pattern 7) |

| 5 | Debug specific failures | patterns-trace-analysis.md (Patterns 8-9) |

Workflow 5: Custom Scorer Development

For creating project-specific evaluation metrics.

| Step | Action | Reference Files |

|------|--------|-----------------|

| 1 | Understand scorer interface | CRITICAL-interfaces.md (Scorer section) |

| 2 | Choose scorer pattern | patterns-scorers.md (Patterns 4-11) |

| 3 | For multi-agent scorers | patterns-scorers.md (Patterns 13-16) |

| 4 | Test with evaluation | patterns-evaluation.md (Pattern 1) |

Workflow 6: Unity Catalog Trace Ingestion & Production Monitoring

For storing traces in Unity Catalog, instrumenting applications, and enabling continuous production monitoring.

| Step | Action | Reference Files |

|------|--------|-----------------|

| 1 | Link UC schema to experiment | patterns-trace-ingestion.md (Patterns 1-2) |

| 2 | Set trace destination | patterns-trace-ingestion.md (Patterns 3-4) |

| 3 | Instrument your application | patterns-trace-ingestion.md (Patterns 5-8) |

| 4 | Configure trace sources (Apps/Serving/OTEL) | patterns-trace-ingestion.md (Patterns 9-11) |

| 5 | Enable production monitoring | patterns-trace-ingestion.md (Patterns 12-13) |

| 6 | Query and analyze UC traces | patterns-trace-ingestion.md (Pattern 14) |

Workflow 7: Judge Alignment with MemAlign

For aligning an LLM judge to match domain expert preferences. A well-aligned judge improves every downstream use: evaluation accuracy, production monitoring signal, and prompt optimization quality. This workflow is valuable on its own, independent of prompt optimization.

| Step | Action | Reference Files |

|------|--------|-----------------|

| 1 | Design base judge with make_judge (any feedback type) | patterns-judge-alignment.md (Pattern 1) |

| 2 | Run evaluate(), tag successful traces | patterns-judge-alignment.md (Pattern 2) |

| 3 | Build UC dataset + create SME labeling session | patterns-judge-alignment.md (Pattern 3) |

| 4 | Align judge with MemAlign after labeling completes | patterns-judge-alignment.md (Pattern 4) |

| 5 | Register aligned judge to experiment | patterns-judge-alignment.md (Pattern 5) |

| 6 | Re-evaluate with aligned judge (baseline) | patterns-judge-alignment.md (Pattern 6) |

Workflow 8: Automated Prompt Optimization with GEPA

For automatically improving a registered system prompt using optimize_prompts(). Works with any scorer, but paired with an aligned judge (Workflow 7) gives the most domain-accurate signal. For the full end-to-end loop combining alignment and optimization, see user-journeys.md Journey 10.

| Step | Action | Reference Files |

|------|--------|-----------------|

| 1 | Build optimization dataset (inputs + expectations) | patterns-prompt-optimization.md (Pattern 1) |

| 2 | Run optimize_prompts() with GEPA + scorer | patterns-prompt-optimization.md (Pattern 2) |

| 3 | Register new version, promote conditionally | patterns-prompt-optimization.md (Pattern 3) |

Reference Files Quick Lookup

| Reference | Purpose | When to Read |

|-----------|---------|--------------|

| GOTCHAS.md | Common mistakes | Always read first before writing code |

| CRITICAL-interfaces.md | API signatures, schemas | When writing any evaluation code |

| patterns-evaluation.md | Running evals, comparing | When executing evaluations |

| patterns-scorers.md | Custom scorer creation | When built-in scorers aren't enough |

| patterns-datasets.md | Dataset building | When preparing evaluation data |

| patterns-trace-analysis.md | Trace debugging | When analyzing agent behavior |

| patterns-context-optimization.md | Token/latency fixes | When agent is slow or expensive |

| patterns-trace-ingestion.md | UC trace setup, monitoring | When setting up trace storage or production monitoring |

| patterns-judge-alignment.md | MemAlign judge alignment, labeling sessions, SME feedback | When aligning judges to domain expert preferences |

| patterns-prompt-optimization.md | GEPA optimization: build dataset, optimize_prompts(), promote | When running automated prompt improvement |

| user-journeys.md | High-level workflows, full domain-expert optimization loop | When starting a new evaluation project or running the full align + optimize cycle |

Critical API Facts

• Use: mlflow.genai.evaluate() (NOT mlflow.evaluate())
• Data format: {"inputs": {"query": "..."}} (nested structure required)
• predict_fn: Receives **unpacked kwargs (not a dict)
• MemAlign: Scorer-agnostic (works with any feedback_value_type -- float, bool, categorical); token-heavy on the embedding model so set embedding_model explicitly
• Label schema name matching: The label schema name in the labeling session MUST match the judge name used in evaluate() for align() to pair scores
• Aligned judge scores: May be lower than unaligned judge scores -- this is expected and means the judge is now more accurate, not that the agent regressed
• GEPA optimization dataset: Must have both inputs AND expectations per record (different from eval dataset)
• Episodic memory: Lazily loaded -- get_scorer() results won't show episodic memory on print until the judge is first used
• optimize_prompts: Requires MLflow >= 3.5.0

See GOTCHAS.md for complete list.

Related Skills

• databricks-docs - General Databricks documentation reference
• databricks-model-serving - Deploying models and agents to serving endpoints
• databricks-agent-bricks - Building agents that can be evaluated with this skill
• databricks-python-sdk - SDK patterns used alongside MLflow APIs
• databricks-unity-catalog - Unity Catalog tables for managed evaluation datasets