Run Health Review Agent Skill

run-health-review

Unattended work fails quietly. A host restarts and children sit idle, a quota runs out at 2am, a prompt drifts, the pool empties, CI goes red and every worker politely no-ops. Without a watchdog, the human computes "time since last merge" by eye every morning. This skill makes that check a job: measure health from GitHub state, name the stall, change one thing (or nothing), and pick up work that was left half-done.

When to use

• "Why did delivery stall overnight?", "nothing merged for 8 hours", "this was great the first few days, now it's slow".
• An automation failed (automation_failed) or its useful-output rate dropped.
• 403/429 rate-limit errors, or CI minutes or another quota ran out.
• On a schedule: every 30–60 minutes during an active unattended run; weekly for automations at steady state.

When not to use

• Nothing is running unattended. Use progress-contract for attended work.
• A single PR is red. That is pr-shepherd's job.

Steps

• Set the window and baseline. Default: last 24 h vs the prior 7-day daily average.
• Pull health signals, read-only, in one pass:
• Time since last merge to the default branch (the headline watchdog metric) and since the last release.
• Merged PRs, open PRs by state: green-unmerged, red, conflicted, waiting on review, draft; age of each.
• Ready-pool size (see keep-the-pool-full).
• Per automation or worker: runs, ACTED / NOOP / BLOCKED mix, failures by reason (reported, crashed, timed_out), latest revision and date.
• Rate-limit and quota hits: 403/429 counts, CI minutes, API calls per merged PR.
• Idle or interrupted sessions and branches with no commits since the stall began.
• Stay silent if healthy. If time since last merge is under STALL_THRESHOLD and no signal crossed its threshold, end NOOP(healthy) and post nothing.
• Classify each stall with the taxonomy below. Cite the evidence for each (the run, PR, log line or count). One stall can have one primary cause; list secondary causes, don't act on them.
• Take over orphaned work if TAKEOVER allows (protocol below).
• Propose one change, or none. Pick the single change with the largest expected effect on the headline metric. State the prediction ("time since last merge back under 45 min within 24 h") and how you will check it. "No change" is a valid outcome when the stall was transient or the cause is outside your control; say what you'll watch instead.
• Route human-only blockers once. Permissions, budget increases, sign-ins and policy calls go in a single list with a recommended answer, surfaced in the next scheduled report. Never ask the same question twice; never block a stream on it.
• Report in the status format: time since last merge: x → y, merged count, stalls by class, the one change (or "no change") with its prediction, takeovers, human-only list. Next run, check last run's prediction first.

Stall taxonomy

| Class | Signal | Typical one change |

|---|---|---|

| Supply | Pool empty or low; workers NOOP(no eligible work) | Run keep-the-pool-full; raise its target |

| CI | Main red, or PRs red on the same job | Route to main-red triage; quarantine a proven flake |

| Rate limit (secondary) | 403 or 429 with a rate-limit body or Retry-After; bursts of content-creating writes from many lanes on one identity | Classify as retryable, not permission. Record "not attempted, retry after <absolute time>". Pace all lanes through one budget; serialize writes |

| Quota (primary) | Hourly API, CI-minute or credit cap exhausted | Lower concurrency or trigger breadth; adopt the shared API budget below |

| Permission | BLOCKED on a missing tool, scope or ruleset | One human-only request; or remove the step that needs it |

| Prompt drift | Rising BLOCKED or invalid outputs after a revision; humans overriding outputs | Revert to the last good revision, or propose one contract-rule change |

| Orphaned | Branch or PR with an owner that stopped (host restart, interrupted session) | Take over (below) |

| Trigger | Eligible items with no run (filter too narrow); or many runs per item (too broad) | Adjust one trigger filter; add a sweep |

| Review / merge | Green PRs waiting on threads or approvals | Thread-resolution sweep; check the autonomy level is what the human intended |

| Busy-but-idle | Session reports busy, but there has been no new turn or tool call for longer than IDLE_AFTER | Rekick with what changed and what's expected. Judge liveness by the last tool timestamp, not a status flag |

| Host restart | Coordinator and children went idle at the same time, right after a deploy of the platform they run on | Resume the root first, then its children. Make "resume self" the last step of any self-deploy |

| Self-echo / no-op storm | No-op wakes above 50%, or any wakes triggered by the agent's own writes | Tighten the wake filter (see automate-it step 3) |

| Tool-profile mismatch | A tool fails on every call; a write tool sits in a read-only profile | Remove it, or fix the profile. Every-call failures are a config bug, not flakiness |

Takeover protocol

• Find candidates: open PRs or branches with no commits for longer than ORPHAN_AFTER, whose owner session or automation is no longer running.
• Verify state: head SHA, CI on that SHA, unresolved threads, mergeability, and the linked issue's acceptance check.
• Claim visibly: comment "taking over from <owner> at <sha>: <next step>" and assign yourself (or the worker that will continue).
• Continue from the existing branch. Don't rewrite or force-push someone else's commits; add commits on top.
• If the work is obsolete (superseded, already fixed on main), close it with a one-line reason and a link, only if TAKEOVER allows closing.
• Record the takeover. If the original owner reappears, hand back with the current state.

Shared API budget

Quota is the failure mode that grows with success: more agents, more polling, more calls per merge.

• Inventory callers: every automation, worker and local loop that reads GitHub, with calls per run.
• Set a per-unit budget, for example at most 75 REST calls per merged PR across all callers, and a per-hour ceiling below the token's limit.
• Pace through one budget, not N independent ones. Priority order: merge-path work first, reports last.
• Cut calls before adding retries: read the head SHA only; one batched GraphQL read per wake; bounded pages; cached job lists; wait on events instead of polling.
• Alert on overspend: calls per merged PR above budget, or any 403/429, is a Rate limit or Quota stall.

Dials

| Dial | Default | Options |

|---|---|---|

| STALL_THRESHOLD | 2 h without a merge during active hours, report only | Minutes; per repo |

| Schedule | Hourly during active runs; weekly otherwise | interval; automation_failed |

| TAKEOVER | propose (list candidates, don't act) | continue (claim and push), continue_and_close |

| ORPHAN_AFTER | 2 h without commits and no live owner | Duration |

| Change authority | Propose only | Apply prompt changes via a human; never self-update automations |

| Merge stalled green PRs | Off | On, at the level set in docs/autonomy.md |

| API budget | 75 REST calls per merged PR | Integer; per-hour ceiling |

| Quiet hours | None | No human pings between X and Y; batch into the morning report |

Outputs

• A health report (silent when healthy) with stalls, evidence, one change or "no change", and a prediction.
• Takeover comments and records.
• At most one proposed change per run, for example a propose_automation_update or a PR to a prompt file.

To run the weekly automation half unattended, install automations/automation-health.

This skill's own feedback loop

• Headline metric: stall minutes per day (time since last merge above threshold), and human rekicks or "why did it stall?" asks per week (target zero).
• Prediction score: for each proposed change, did the predicted metric move within the window? Keep a running hit rate. If "no change" runs are followed by the same stall twice, escalate the class.

Failure modes it prevents

• The human as the watchdog, computing time since last merge by eye.
• Changing three things at once and learning nothing.
• Half-done PRs left by interrupted sessions.
• Agents exhausting a shared API quota and taking every other agent down with them.
• Asking the same human-only question every hour.

Why this exists

• Morning messages in one long unattended run: "no merged commits in 19 hours", "no commit all night", "lots of draft PRs, nothing merged for 8 hours". The human computed the metric himself for about two weeks before it became a 30-minute job; once encoded, overnight stalls became visible and short.
• Rekicking interrupted child sessions was the top manual chore (at least 7 explicit requests) until it became a rule that the human should never have to ask.
• Scheduled single-shot runs spend 7–13% of runs checking state and exiting. Event-woken, long-lived sessions are different: in one sample, about 62% of wakes were no-ops and about 16% were self-echo. A watchdog must know which kind of lane it is judging.
• CI babysitters once made about 45k API calls in 3 hours and hit the rate limit. The fix was a shared budget and fewer calls per merge, not retries.
• In one fleet of hosted agents, the secondary rate limit was mentioned more than 15 times as often as flaky tests. The damage came from misreading a rate-limited 403 as access-denied and reporting merges as failed. The fix was classification plus one paced budget, not retries.
• Revisions that changed one thing (a model swap, a prompt cut 15% with the same rules) could be judged; "no change" was often the right answer.