Skillhone

SkillHone

SkillHone is a measurement harness plus an optimization toolkit for agent

skills. The important abstraction is not a fixed workflow; it is the evidence

trail created when a skill is run against private eval tasks.

The harness has these layers:

• Skill repo: public behavior to improve (SKILL.md, scripts, references).
• Eval repo: private measurement contract (datasets, verifier, synthesis

contract, task-local validators).

• Solver workdirs: isolated per-item sandboxes containing artifacts and

trajectory.jsonl.

• Observation records: redacted probe results, trajectory diagnosis,

compiler/validator diagnosis, issues, PRs, and wiki pages.

When improving a skill, identify which layer explains the failure before

changing code. A score drop may point to skill instructions, but it can also be

a harness, verifier, compiler, artifact-path, or infrastructure problem.

Standalone scripts live under scripts/. Pick one based on the task in front of you.

Before starting a diagnosis, development, PR review, merge, or optimization cycle on a Forgejo-backed repo, run scripts/status.py so you know the current issue/PR state and do not duplicate work or merge the wrong PR.

Eval Synthesis Contract

When creating or synthesizing eval data, the task spec / README is the source of

truth for both prompts and verification. Do not generate verifiers that only

check the gold answer if the task spec also requires observable output

properties.

If the README says the answer must have a format, file, syntax, parser/compile

success, render success, required section, count range, banned token, fixed

style token, palette, local-only dependency, or other deterministic acceptance

criterion, include a corresponding scores key in the verifier. This applies

whether you use skillhone-synthesis or write a small generation script by

hand.

Subjective requirements may be approximated by deterministic proxies. Truly

uncheckable preferences should be noted as unverified, not silently ignored.

Scripts

| Script | Use when the user asks to... | Details |

|--------|-------------------------------|---------|

| scripts/status.py | inspect the current Forgejo repo's issue/PR state before acting | references/cli.md |

| scripts/eval.py | run a probe / test evaluation on a skill, get a score | references/evaluation.md |

| scripts/optim.py | iterate on / optimize / auto-improve a skill | references/optim.md |

| scripts/new.py | create a new skill+eval repo pair (a new experiment) | references/cli.md |

| scripts/seed.py | scaffold SKILL.md / scripts/ for an empty skill repo | references/seed.md |

| scripts/synth.py | synthesise a probe split into the eval repo (calls into skillhone-synthesis) | references/cli.md |

| scripts/serve.py | listen for Forgejo webhooks (PR/issue/push) | references/cli.md |

| scripts/quality/static_check.py | validate skill structure and script discoverability | references/quality_scoring_rubric.md |

Load the relevant reference file only when you're about to run that specific script. Do not load them up front.

Upstream binding & sync

The skillhone skill is bound to <https://github.com/Tencent/SkillHone>;

the repository is the actual implementation. When the skill is delivered

through a hub (SkillHub, an internal mirror, any agentskills.io-protocol

runtime), the SKILL.md alone is not enough — the runtime has to pull the

full upstream repository before the harness scripts can run.

Follow references/upstream.md whenever you need to:

• Install for the first time — ~/.skillhone/skills/ is empty.
• Refresh / update / pull latest — refresh existing skill folders.

Both run the same idempotent procedure (shallow clone + cp -R into

~/.skillhone/skills/<skill>/). Settings, run history, and the cache stay

untouched.

Quick examples

# Status — read-only dashboard of Issues + PRs for the current Forgejo repo
python3 scripts/status.py

# Evaluate — runs probe split, writes result.json
python3 scripts/eval.py --skill-dir ./my-skill --eval-dir ./my-skill-eval \
                        --split probe --output result.json

# Optimize — agent-driven loop (5 iters, stop after 2 with no gain)
python3 scripts/optim.py --repo http://forgejo/skillhone/my-skill.git \
                         --iters 5 --patience 2

# New experiment — creates skill-repo + eval-repo on Forgejo
python3 scripts/new.py deep-research \
                       --instruction README.md --data-dir ./data --no-run

# Seed an empty skill — generate SKILL.md from a brief
python3 scripts/seed.py --repo http://forgejo/skillhone/my-skill.git

# Webhook listener
python3 scripts/serve.py --port 8790

Gotchas

• ~/.skillhone/settings.json is required before any script runs. It holds the Forgejo URL/token and three model profiles: improver (drives optim.py), executor (runs the skill under eval), and optional synthesis (used by synth.py). See references/configuration.md.
• Start with scripts/status.py on Forgejo repos. It is read-only and shows the open/closed Issue and PR state for the current repo; use it before creating issues, developing fixes, reviewing PRs, or merging.
• scripts/eval.py never writes to the skill repo — it reads SKILL.md and writes a JSON result. Safe to run read-only.
• scripts/optim.py spawns subagents (issue-reporter → developer → reviewer) via the Agent tool. Do *not* add Agent / Task to disallowed_tools in settings.json.improver or the loop will no-op.
• Eval repo must stay private. The optimizing agent must never see it. optim.py only passes the skill repo path + a redacted probe result into the loop.
• Probe ≠ test. A probe improvement does not guarantee a test improvement. See references/evaluation.md → "Probe vs Test".
• All state lives under ~/.skillhone/ (logs, run artifacts, workspaces). Override with $SKILLHONE_HOME.

How this skill composes

skillhone-prd   →  skillhone-synthesis  →  skillhone  →  skillhone-evaluation  →  skillhone-optimization
(spec the PRD)     (generate eval data)    (eval/optim)    (score + diagnose)       (optimize via PR)

skillhone is the orchestrator entry point; evaluation and optimization skills

are loaded on demand inside optim.py's agent loop. VCS operations are provided

by a separate backend skill such as forgejo.

Orchestration: "synthesise and optimise a skill from <PRD>"

When the user asks to "synthesise and optimise a skill from <path/to/PRD.md>"

(typical phrasing for the worked examples under examples/), run the four

scripts below in order. The contract is one persistent skill repo + one

private eval repo on Forgejo, with a regression-aware synth step gating the

expensive optim phase.

• scripts/new.py <skill-name> --instruction <PRD.md>

Creates the public <skill-name> and private <skill-name>-eval repos

on Forgejo. Auto-redacts the PRD's `## ...Evaluation/Verifier/Scoring/

Rubric...` section so the public README never exposes the grading rubric

to the improver. The unredacted PRD lands in the eval repo.

• scripts/seed.py --repo <skill-url>

Reads the redacted public README, generates a real (but unoptimised)

SKILL.md plus minimal scaffolding, and commits as the seed point. This

is the baseline the synth-stage regression scores against — without a

real seed, the regression is meaningless. Skip this step ONLY when the

PRD has no ## 3.5 Synth-stage acceptance gate and you are intentionally

running an old-style single-shot synth.

• scripts/synth.py --repo <skill-url> --target 10 --splits probe ...

Synthesises probe.jsonl from the eval-side PRD. When the PRD declares

a synth-stage acceptance gate (§3.5 in the worked examples), pass

--target-pass-rate-max <X> --max-resynth <N> (typical: 0.30 and 3)

so synth runs eval.py --mode seed --split probe after each draft and

redrafts when the seed solves more than X of the probes. Each iteration's

observations are written to the eval repo's synthesis_observations/

directory and pushed alongside the final probe.jsonl. Without these

flags, synth is single-shot (the historical behaviour).

> Synth is optional. If you already have a curated eval set (golden

> items from a benchmark, hand-written probes, an exported test bank,

> etc.), skip synth.py entirely and push your own probe.jsonl

> (and optionally test.jsonl) directly into the eval repo. The

> format the rest of the harness expects is documented in

> references/evaluation.md. As long as

> the verifier contract is satisfied, optim.py does not care whether

> the data came from synth.py or git push.

• scripts/optim.py --repo <skill-url> --iters 3 --patience 2

The agent-driven PR loop: diagnose probe failures → file Issue → land

focused PR → re-evaluate → write Iteration-N-Observation wiki page.

Each merged PR is one atomic skill change.

Skip steps 2 + 3's --target-pass-rate-max only for prototype runs where you

explicitly want to see what synth produces without a regression gate. For any

example whose PRD includes §3.5, skipping the gate defeats the point.

References (load on demand)

• references/evaluation.md — eval.py CLI, output JSON schema, solver architecture. Read before running eval.py or interpreting result.json.
• references/optim.md — optim.py loop, subagent roles, stop conditions. Read before running optim.py.
• references/seed.md — original SkillHone seed scaffold and validation rules. Read before running seed.py.
• references/quality_scoring_rubric.md — rubric used by quality reviewers.
• references/configuration.md — ~/.skillhone/settings.json schema, directory layout, env vars. Read on first setup or when the user asks "where do I configure X?".
• references/cli.md — flag-by-flag reference for every script. Read when a user asks about a flag you're not sure about.