Skill Grader

Skill Grader

Structured evaluation rubric for Claude Agent Skills. Produces letter grades (A+ through F) on 10 axes plus an overall grade, with specific improvement recommendations for each axis.

Designed for sub-agents and non-expert reviewers who need a mechanical, repeatable process for assessing skill quality without deep domain expertise.

When to Use

✅ Use for:

• Auditing a single skill's quality
• Comparing skills against each other
• Prioritizing which skills to improve first
• Quality control sweeps across a skill library
• Generating improvement roadmaps

❌ NOT for:

• Creating new skills (use skill-architect)
• Grading code quality or non-skill documents
• Evaluating agent performance (different from skill quality)

Grading Process

flowchart TD
  A[Read SKILL.md + all files] --> B[Score each of 10 axes]
  B --> C[Assign letter grade per axis]
  C --> D[Compute overall grade]
  D --> E[Write improvement recommendations]
  E --> F[Produce grading report]

Step-by-Step

• Read the entire skill folder — SKILL.md, all references, scripts, CHANGELOG, README
• Score each axis — Use the rubric below (0-100 per axis)
• Convert to letter grade — See grade scale
• Compute overall grade — Weighted average (Description and Scope are 2x weight)
• Write 1-3 specific improvements per axis scoring below B+
• Produce the grading report in the output format below

The 10 Evaluation Axes

Axis 1: Description Quality (Weight: 2x)

Does the description follow [What] [When] [Keywords]. NOT for [Exclusions]?

| Grade | Criteria |

|-------|----------|

| A | Specific verb+noun, domain keywords users would type, 2-5 explicit NOT exclusions, 25-50 words |

| B | Has keywords and NOT clause, but slightly vague or missing synonym coverage |

| C | Too generic, missing NOT clause, or >100 words of process detail |

| D | Single vague sentence ("helps with X") or name/description mismatch |

| F | Missing or empty description |

Axis 2: Scope Discipline (Weight: 2x)

Is the skill narrowly focused on one expertise type, or a catch-all?

| Grade | Criteria |

|-------|----------|

| A | One clear expertise domain, "When to Use" and "NOT for" sections both present and specific |

| B | Mostly focused, minor boundary ambiguity |

| C | Covers 2-3 related but distinct domains, should probably be split |

| D | Catch-all skill ("helps with anything related to X") |

| F | No scope boundaries defined at all |

Axis 3: Progressive Disclosure

Does the skill follow the three-layer architecture (metadata → SKILL.md → references)?

| Grade | Criteria |

|-------|----------|

| A | SKILL.md <300 lines, heavy content in references, reference index in SKILL.md with 1-line descriptions |

| B | SKILL.md <500 lines, some references used, index present |

| C | SKILL.md >500 lines, or all content inlined with no references |

| D | SKILL.md >800 lines, or references exist but aren't indexed in SKILL.md |

| F | Single massive file with no structure |

Axis 4: Anti-Pattern Coverage

Does the skill encode expert knowledge that prevents common mistakes?

| Grade | Criteria |

|-------|----------|

| A | 3+ anti-patterns with Novice/Expert/Timeline template, LLM-mistake notes |

| B | 1-2 anti-patterns with clear explanation |

| C | Anti-patterns mentioned but no structured template |

| D | No anti-patterns, just positive instructions |

| F | Contains advice that IS an anti-pattern (outdated, harmful) |

Axis 5: Self-Contained Tools

Does the skill include working tools (scripts, MCPs, subagents)?

| Grade | Criteria |

|-------|----------|

| A | Working scripts with CLI interface, error handling, dependency docs; OR valid "no tools needed" justification |

| B | Scripts exist and work but lack error handling or docs |

| C | Scripts referenced but are templates/pseudocode |

| D | Phantom tools (SKILL.md references files that don't exist) |

| F | References non-existent tools AND no acknowledgment |

Note: Not every skill needs tools. A pure decision-tree skill can score A if tools aren't applicable.

Axis 6: Activation Precision

Would the skill activate correctly on relevant queries and stay silent on irrelevant ones?

| Grade | Criteria |

|-------|----------|

| A | Description has specific keywords matching user language, clear NOT clause, no obvious false-positive vectors |

| B | Good keywords, minor false-positive risk |

| C | Generic keywords that overlap with other skills |

| D | No specific keywords, or NOT clause contradicts intended use |

| F | Description would cause constant false activation |

Axis 7: Visual Artifacts

Does the skill use Mermaid diagrams, code examples, and tables effectively?

| Grade | Criteria |

|-------|----------|

| A | Decision trees as Mermaid flowcharts, tables for comparisons, code examples for concrete patterns |

| B | Some diagrams or tables, but key decision trees still in prose |

| C | Tables used but no Mermaid diagrams for processes |

| D | Prose-only, no visual structure |

| F | Wall of text with no formatting aids |

Axis 8: Output Contracts

Does the skill define what it produces in a format consumable by other agents?

| Grade | Criteria |

|-------|----------|

| A | Explicit output format (JSON schema, markdown template, or structured sections), subagent-consumable |

| B | Output format implied but not explicitly documented |

| C | No output format, but content is structured enough to infer |

| D | Unstructured prose output expected |

| F | N/A (pure reference skill) — exempt from this axis |

Axis 9: Temporal Awareness

Does the skill track when knowledge was current and what has changed?

| Grade | Criteria |

|-------|----------|

| A | Timelines in anti-patterns, "as of [date]" markers, CHANGELOG with dates |

| B | Some temporal context, CHANGELOG exists |

| C | No dates on knowledge, but CHANGELOG exists |

| D | No temporal context anywhere, knowledge could be stale |

| F | Contains demonstrably outdated advice without warning |

Axis 10: Documentation Quality

README, CHANGELOG, and reference organization.

| Grade | Criteria |

|-------|----------|

| A | README with quick start, CHANGELOG with dated versions, references well-organized with clear filenames |

| B | README and CHANGELOG exist, references present |

| C | SKILL.md is the only file, but it's well-structured |

| D | No README, no CHANGELOG, disorganized references |

| F | SKILL.md is the only file and it's poorly structured |

Grade Scale

| Letter | Score Range | Meaning |

|--------|------------|---------|

| A+ | 97-100 | Exemplary — sets the standard |

| A | 93-96 | Excellent — minor improvements possible |

| A- | 90-92 | Very good — a few small gaps |

| B+ | 87-89 | Good — notable room for improvement |

| B | 83-86 | Solid — several areas need work |

| B- | 80-82 | Above average — meaningful gaps |

| C+ | 77-79 | Average — significant improvements needed |

| C | 73-76 | Below average — major gaps |

| C- | 70-72 | Barely adequate |

| D+ | 67-69 | Poor — fundamental issues |

| D | 63-66 | Very poor — needs major rework |

| D- | 60-62 | Near-failing quality |

| F | <60 | Failing — start over |

Overall Grade Computation

Axes 1 (Description) and 2 (Scope) carry 2x weight. All others carry 1x weight. If Axis 8 (Output Contracts) is marked exempt, remove it from the calculation.

Overall = (2×Axis1 + 2×Axis2 + Axis3 + Axis4 + Axis5 + Axis6 + Axis7 + Axis8 + Axis9 + Axis10) / 12

Convert the numeric average to a letter grade using the scale above.

Output Format

Produce this exact structure:

# Skill Grading Report: [skill-name]

**Graded**: [date]
**Overall Grade**: [letter] ([score]/100)

## Axis Grades

| # | Axis | Grade | Score | Key Finding |
|---|------|-------|-------|-------------|
| 1 | Description Quality | [grade] | [score] | [1-line finding] |
| 2 | Scope Discipline | [grade] | [score] | [1-line finding] |
| 3 | Progressive Disclosure | [grade] | [score] | [1-line finding] |
| 4 | Anti-Pattern Coverage | [grade] | [score] | [1-line finding] |
| 5 | Self-Contained Tools | [grade] | [score] | [1-line finding] |
| 6 | Activation Precision | [grade] | [score] | [1-line finding] |
| 7 | Visual Artifacts | [grade] | [score] | [1-line finding] |
| 8 | Output Contracts | [grade] | [score] | [1-line finding] |
| 9 | Temporal Awareness | [grade] | [score] | [1-line finding] |
| 10 | Documentation Quality | [grade] | [score] | [1-line finding] |

## Top 3 Improvements (Highest Impact)

1. **[Axis]: [Specific action]** — [Why this matters, expected grade improvement]
2. **[Axis]: [Specific action]** — [Why this matters, expected grade improvement]
3. **[Axis]: [Specific action]** — [Why this matters, expected grade improvement]

## Detailed Notes

### [Axis name] ([grade])
[2-3 sentences of specific feedback with examples from the skill]

[Repeat for each axis scoring below B+]

Quick Grading (Abbreviated)

For rapid triage across many skills, produce only:

| Skill | Overall | Desc | Scope | Disc | Anti | Tools | Activ | Visual | Output | Temp | Docs |
|-------|---------|------|-------|------|------|-------|-------|--------|--------|------|------|
| [name] | [grade] | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |

Anti-Patterns in Grading

Grade Inflation

Wrong: Giving B+ because "it's pretty good" without checking criteria.

Right: Match observations to the rubric table literally. If the description lacks a NOT clause, it cannot score above C on Axis 1.

Missing Context

Wrong: Grading a pure decision-tree skill poorly on Axis 5 (tools) because it has no scripts.

Right: Mark Axis 5 as "A — tools not applicable for this skill type."

Ignoring Phantoms

Wrong: Scoring Axis 5 as B because scripts are "referenced."

Right: Actually check if every referenced file exists. If scripts/validate.py is mentioned but doesn't exist, that's D.