SDK AI Bot Eval Dataset Agent Skill

QA Bot Evaluation Dataset

Create a new per-scenario evaluation dataset or add cases to an existing one for the

QA bot evaluation package at tools/sdk-ai-bots/azure-sdk-qa-bot-evaluation. Datasets

are per-scenario JSONL files under evaluation_datasets/<target>/<scenario>.jsonl

(target = basic or perf) holding inputs + expectations only.

Run all commands from tools/sdk-ai-bots/azure-sdk-qa-bot-evaluation with the package

.venv active and az login done. See schema and workflows for the canonical row format and step-by-step recipes.

Triggers

USE FOR: create a new evaluation dataset (new scenario file); add cases to an existing per-scenario dataset; curate cases from storage markdown; promote reviewed staging cases; upload a dataset as a Foundry asset

WHEN: "add eval dataset item", "add a test case", "new evaluation dataset", "create dataset", "add question to dataset", "curate eval data", "promote staging cases", "upload dataset asset", "new scenario dataset"

DO NOT USE FOR: running evaluations, pipeline troubleshooting, knowledge-graph indexing

Rules

• A dataset is one file: evaluation_datasets/<target>/<scenario>.jsonl. Creating a new dataset = creating a new <scenario>.jsonl in basic/ or perf/.
• The canonical dedup key is the normalized query (applied at curation). testcase titles may legitimately repeat (e.g. Untitled) — never dedup or fail on testcase.
• Only reviewed: "pass" rows are curated/committed; see the review status lifecycle for the three states and how leftovers are finalized.
• evaluation_datasets/_staging/ is committed (shared review state) so concurrent contributors don't re-curate the same cases; basic/, perf/ and registry.json are committed too.
• Always validate before upload, and after editing any curated file.

Environment

Before running any command that touches Azure, ensure the required variables are set

and remind the user to configure them. Dataset prep loads a local .env (copy and fill

in tools/sdk-ai-bots/azure-sdk-qa-bot-evaluation/env-variables) and authenticates with

az login.

| Command | Requires |

| ------------------------------------ | ---------------------------------------------------------------------------------------- |

| dataset.curate | az login, STORAGE_BLOB_ACCOUNT, AI_ONLINE_PERFORMANCE_EVALUATION_STORAGE_CONTAINER |

| dataset.upload | az login, AZURE_AI_PROJECT_ENDPOINT |

| dataset.validate, dataset.review | none (local file operations) |

If a required variable is missing the command fails (KeyError / auth error) — set it in

.env or the shell and re-run. A purely manual add (edit JSONL + validate) needs no

env vars; only dataset.upload then requires AZURE_AI_PROJECT_ENDPOINT + az login.

Choose a workflow

| Goal | Workflow |

| ------------------------------------------------- | ------------------------------------------------------------------------------- |

| Add a few specific cases you already have | Manual add |

| Harvest new cases from collected storage markdown | Curate from blob |

| Create a brand-new scenario dataset | New dataset |

Core commands

# Validate a file or folder (--require-reviewed gates official datasets on reviewed=="pass")
python -m dataset.validate evaluation_datasets/<target>/<scenario>.jsonl --require-reviewed

# Promote reviewed (pass) staging rows; leftover items are finalized to abandoned
python -m dataset.review --target <basic|perf> [--scenario <scenario>]

# Upload one versioned Foundry asset per scenario; writes registry.json
python -m dataset.upload --target <basic|perf> [--scenario <scenario>]

After adding or creating a dataset: validate → upload → commit the per-scenario

file, registry.json, and updated _staging/ files.

Steps

• Pick a workflow from the table above (manual add, curate from blob, or new dataset).
• Add or stage canonical rows in evaluation_datasets/<target>/<scenario>.jsonl.
• For staged cases, promote the reviewed ones with python -m dataset.review.
• Validate the file with python -m dataset.validate ... --require-reviewed.
• Publish with python -m dataset.upload, then commit the per-scenario file,

registry.json, and updated _staging/.