Night Market Diagnostics Toolkit

Night Market Diagnostics Toolkit

Measure instead of eyeballing. Every claim about the health of this

repo ("the skill graph is clean", "descriptions fit the budget",

"no compromised dependencies") has a script that produces a number

or a PASS/FAIL. This skill catalogs those scripts, their exact

invocations, and how to read their output.

Conventions for everything below:

• Run all commands from the repo root

(/home/alext/claude-night-market).

• Plain python3 is enough for every script in the table except

mutation testing, which needs uv.

• All checks here are read-only. None of them mutate the repo.

The ratchet-baseline concept

Several checks are ratchets. A ratchet check counts a category of

existing debt (for example, SKILL.md files missing an Exit Criteria

section), compares the live count against a frozen number in a

baseline JSON file, and:

• passes while the live count is at or below the baseline

(pre-existing debt is tolerated),

• fails the moment a NEW violation pushes the count above the

baseline (debt may not grow),

• nudges you to lower the baseline number when the live count drops

(locking in the win so the debt cannot silently return).

The goal is a baseline that only shrinks, eventually to zero. The

two ratchet baselines live at scripts/skill_graph_baseline.json

(keys max_dangling_bugs, max_uncalled_libraries) and

scripts/skill_exit_criteria_baseline.json

(key max_missing_exit_criteria). When a ratchet prints a

"dropped to N (baseline M). Lower ... to lock the win." line, edit

the baseline JSON down to N in the same PR.

Tool table

Jargon used in the table: "dangling Skill() ref" means a

Skill(plugin:name) reference in a skill/command/agent file whose

target skill does not exist. "Uncalled library" means a

library-role skill no other skill invokes (the risk targeted by

.claude/rules/shared-utility-consumer-rule.md).

| Tool | Invocation | What it measures | How to interpret | When to run |

|------|------------|------------------|------------------|-------------|

| Plugin structure validator | python3 plugins/abstract/scripts/validate_plugin.py plugins/<name> | plugin.json validity, kebab-case naming, directory layout | Exit 0 with "Plugin validation passed". Any FAIL line names the broken file | Before committing plugin changes. make validate-all loops it over every plugin |

| Skill-graph drift ratchet | python3 scripts/check_skill_graph_drift.py | Dangling Skill() refs and uncalled libraries vs skill_graph_baseline.json | Exit 0 at/below baseline. Nonzero exit lists the new dangling refs. "Lower ... to lock the win" means shrink the baseline | After adding, renaming, or deleting skills or Skill() refs |

| Exit-criteria drift ratchet | python3 scripts/check_skill_exit_criteria_drift.py | SKILL.md files under plugins/ missing an ## Exit Criteria heading vs skill_exit_criteria_baseline.json | Same ratchet semantics as above | After adding or editing any SKILL.md |

| Pinned-version checker | python3 scripts/check_pinned_versions.py | GitHub-sourced tool pins vs the latest upstream release (needs network) | "N pin(s) current" is healthy. A "holding X at ..." line documents an intentional hold with its reason (for example bandit 1.8.6 for the Python 3.9 hook floor) | In pre-commit, and whenever a CI tool-setup step breaks |

| Capabilities sync | bash scripts/capabilities-sync-check.sh | plugin.json registrations vs the generated capabilities reference in book/src/ | "PASSED: All capabilities are in sync" plus counts. On drift, run /sanctum:sync-capabilities --fix | After changing any skill, command, or agent registration |

| Supply-chain scan | python3 scripts/supply_chain_scan.py | Lockfiles vs the known-compromised-versions blocklist, plus known malicious artifacts | Two [OK] lines is clean. Any hit must be resolved before release | When adding dependencies, during incidents, before releases |

| Token stats | python3 plugins/abstract/scripts/context_optimizer.py stats plugins/ --format json | Bytes and estimated tokens per SKILL.md, bucketed small/medium/large | Skills over ~5,000 estimated tokens breach the quality-gates file limit and are split candidates | When a skill feels bloated, before modularizing |

| Description budget | python3 plugins/abstract/scripts/validate_budget.py | Sum of all skill/command description characters vs the 90,000-char ecosystem ceiling (ADR-0004, 160 chars per description) | Prints used/ceiling and headroom. Exit 0 means within budget | After editing any frontmatter description |

| skrills validate | skrills validate --skill-dir plugins --target claude | Skill frontmatter validity per target framework (Claude, Codex, Copilot) | "Validated N skills: ..." then an Errors (n): list with file:line. Fix every listed error | Skill audits. make validate-skills falls back to uv run python scripts/check_plugin_hooks.py when skrills is absent |

| skrills analyze | skrills analyze --skill-dir plugins | Skill token usage and dependency structure | Large-token outliers are split candidates | Budget planning. make analyze-skills falls back to scripts/generate_dependency_map.py |

| Mutation testing | cd plugins/<name> && uv run mutmut run --paths-to-mutate=scripts/,src/ --tests-dir=tests/ | Whether the test suite kills injected code mutations (surviving mutants = untested behavior) | Exit 0 = no survivors. Exit 2 = survivors found (CI treats this as pass-with-report). Any other exit = crash, investigate | Weekly CI (Sunday) or manually before hardening a test suite |

| Markdown link checker | python3 scripts/check-markdown-links.py [file.md ...] | Broken relative links and anchors. No args = scan the whole repo | Exit 1 prints each broken link as file: link | After moving or renaming docs or skills |

| Lint-suppression guard | python3 scripts/check_noqa.py <files...> | Inline lint/type suppressions that lack a stated reason | "BLOCKED" plus a hit list. Fix the issue, or append a reason after the suppression marker | Pre-commit runs it on changed files |

| Docstring quality | python3 scripts/check_docstring_quality.py <files.py...> | Docstrings that merely restate the function name | Each hit says "delete it or add information". Do exactly that | Pre-commit runs it on changed Python files |

| JSON-utils drift | bash scripts/shared/check-json-utils-drift.sh | Vendored JSON helper copies in plugin hooks vs the canonical scripts/shared/json_utils.sh | "OK: all vendored JSON utilities match canonical." Anything else names the drifted copy | After editing json_utils.sh or any inlined copy |

| Export stats | python3 scripts/clawhub_export.py --stats | Skill counts per plugin and top-20 membership for cross-framework export | Totals line plus a per-plugin table | Before cross-framework publishing |

| Framework detect | python3 scripts/framework_detect.py [--json] | Which agent-framework capabilities the current directory exposes (skills, agents, hooks, commands, mcp, a2a) | [+]/[-] capability checklist | When debugging cross-framework export targets |

Interpretation notes

• **context_optimizer stats prints nothing in the default text

format.** This is a verified quirk: stats on a directory with

no --format flag produces empty output and exit 0. Always pass

--format json (clean JSON) rather than --format table, which

dumps a raw Python dict.

• Ratchet failures name the new violations. A ratchet that

fails after your change means your change added debt. Fix the

new ref or add the missing section. Never raise a baseline number

to make a check pass, with one documented exception: a brand-new

library skill legitimately starts uncalled, and

scripts/check_skill_graph_drift.py itself (plus the _comment

in scripts/skill_graph_baseline.json) instructs you to raise

max_uncalled_libraries to record the 30-day consumer grace

period that .claude/rules/shared-utility-consumer-rule.md

grants (the rule grants the grace period; the script defines the

baseline mechanics). night-market-architecture-contract records

the same exception in its ratchet invariants.

• Mutation exit codes are the contract. CI

(.github/workflows/mutation-testing.yml) treats exit 0 and

exit 2 as success and posts survivors to the step summary. Only

exits other than 0 and 2 fail the workflow. So a green mutation

run does NOT mean zero survivors. Read the report.

• check_pinned_versions needs network access. It queries GitHub

for latest releases. Offline, expect failures that are

environmental, not real findings.

Health snapshot in one command

scripts/health-snapshot.sh (in this skill directory) runs the six

cheap checks and prints a PASS/FAIL table. It is read-only,

needs only bash and python3, requires no network, and completes in

about 6 seconds (measured 2026-07-02).

bash .claude/skills/night-market-diagnostics-toolkit/scripts/health-snapshot.sh

It runs: plugin structure validation (all plugins), capabilities

sync, supply-chain scan, both drift ratchets, and the description

budget. Exit 0 means all six passed. Exit 1 means at least one

failed, and the failing check's full output is printed to stderr

above the table. Run it at session start, before a PR, and after

any bulk skill edit.

Blocking vs advisory

Which failures stop a merge and which are signals:

| Check | Gate | Effect on failure |

|-------|------|-------------------|

| validate_plugin.py | pre-commit (19 hook entries) | Blocks commit |

| Skill-graph drift ratchet | pre-commit | Blocks commit |

| Exit-criteria drift ratchet | pre-commit | Blocks commit |

| check_noqa.py | pre-commit | Blocks commit |

| check_docstring_quality.py | pre-commit | Blocks commit |

| check-markdown-links.py | pre-commit | Blocks commit |

| validate_budget.py | pre-commit | Blocks commit |

| check-json-utils-drift.sh | pre-commit | Blocks commit |

| check_pinned_versions.py | pre-commit | Blocks commit |

| capabilities-sync-check.sh | pre-commit and CI (capabilities-sync.yml) | Blocks commit and PR |

| Mutation testing | CI (weekly and manual dispatch) | Advisory: survivors reported, workflow stays green |

| supply_chain_scan.py | manual / make supply-chain-scan (not wired to CI) | Advisory day-to-day, treat as blocking before any release |

| context_optimizer, skrills analyze, clawhub stats, framework_detect | manual | Advisory: measurement only |

The dimension-level policy lives in .claude/quality_gates.json:

enforce_blocking is true with max_critical_issues: 3, and only

the security dimension sets block_on_violation: true. The

performance (20KB / 5,000-token file limits, 60-line functions,

complexity under 12), maintainability (debt ratio under 0.3,

nesting under 5), and compliance dimensions are advisory

(block_on_violation: false). So: security findings block,

size/complexity findings warn.

When NOT to use

• Running the test suite, lint, typecheck, or a release:

use night-market-operations (command anatomy and artifacts).

• You already have a failure and need symptom-to-cause triage:

use night-market-debugging-playbook.

• You need the meaning and default of a config knob rather than a

measurement: use night-market-config-catalog.

• You are deciding whether evidence is sufficient to claim work

complete: use night-market-validation-and-qa.

Exit Criteria

• [ ] bash .claude/skills/night-market-diagnostics-toolkit/scripts/health-snapshot.sh

runs from the repo root, prints a six-row PASS/FAIL table,

and exits 0 when all checks pass and 1 when any fails.

• [ ] Every invocation in the tool table is copy-pasteable from the

repo root and executes without a usage error.

• [ ] A reader who has only this file can state the ratchet

contract: baseline freezes existing debt, new violations

fail, shrinking the baseline locks in wins.

• [ ] The blocking/advisory table matches .pre-commit-config.yaml

entries and the block_on_violation flags in

.claude/quality_gates.json.

• [ ] Mutation exit-code semantics (0 clean, 2 survivors tolerated,

anything else is a crash) match

.github/workflows/mutation-testing.yml.

Provenance and maintenance

Compiled 2026-07-02 against repo v1.9.15 on branch

discussions-fix-1.9.14. Volatile numbers observed that day:

197 skills, 134 commands, 56 agents registered; description budget

73,135 of 90,000 chars used (81.3%); ratchet live counts 5 dangling

refs (baseline 31), 7 uncalled libraries (baseline 8), 1 missing

Exit Criteria (baseline 127); 13 version pins current with bandit

intentionally held at 1.8.6.

Re-verification one-liners:

# Baselines and live ratchet counts
python3 scripts/check_skill_graph_drift.py; python3 scripts/check_skill_exit_criteria_drift.py

# Registration counts and budget usage
bash scripts/capabilities-sync-check.sh | tail -3; python3 plugins/abstract/scripts/validate_budget.py | tail -3

# Pre-commit wiring of the blocking table
rg -n "check_skill_graph_drift|check_noqa|validate_budget|check_pinned_versions" .pre-commit-config.yaml

# Mutation exit-code contract
rg -n "exit_code" .github/workflows/mutation-testing.yml

# Gate dimension block flags
python3 -c "import json;print(json.load(open('.claude/quality_gates.json')))"