Agentsop Test Fix Loop

Test-Fix Loop · SOP

> One-liner: The test result IS the next prompt. Wiring the verifier is

> 20% of the work; framing its output as a useful feedback message is 80%.

1. 何时激活 (Activation Rules)

Activate this skill when any of the following triggers fire:

• The user says "have the agent fix until tests pass", "run lint and tests

automatically", "iterate until green", or invokes aider --auto-test,

cline --yes, or an OpenHands-style headless agent.

• The task has a verifiable success command: a non-zero exit code on

failure (pytest, ruff, mypy, eslint, tsc, go test, cargo check, npm run

build, make check, …).

• You're wrapping a code-editing LLM in a script/CI step and need to decide:

*when does the agent return?*

• The agent just made an edit and the next message in the loop would be

"here's what the verifier said".

Do not activate when:

• Success is subjective (writing prose, designing UX). The loop has no

feedback signal worth replaying.

• The verifier is slow + interactive (full E2E suite, multi-min builds).

Either async-ify the loop, or run a fast subset (pytest -x -k changed) in

the loop and gate the slow suite at PR review.

• The gate is human approval, not a machine check — use the HITL skill.

2. 核心心智模型 (Core Mental Model)

2.1 The test result IS the next prompt

The agent's *next turn* is conditioned almost entirely on the message you

inject between edit-N and edit-N+1. That message — formatted from

stdout, stderr, exit_code — is the prompt. The framework labels it

"tool result" or "verifier output" but mechanically it is a user-role message

the LM consumes verbatim.

⇒ Framing the feedback dominates the model choice. A 4000-line raw pytest

dump prompts a worse fix than a 30-line "first failing test, traceback, the

diff you just applied" digest, *regardless of the model behind it*.

2.2 Four primitives

+-----------------+   +-----------------+   +-----------------+   +-----------------+
| 1. Verifier     |   | 2. Capture      |   | 3. Format       |   | 4. Iteration    |
|    command      |   |    (stdout +    |   |    feedback     |   |    bound        |
|                 |   |     stderr +    |   |    message      |   |                 |
| - pytest -x     |   |     exit_code)  |   | - first error   |   | - max N tries   |
| - ruff check    |   | - timeout cap   |   | - last K lines  |   | - escalate /    |
| - mypy --strict |   | - byte cap      |   | - drop noise    |   |   commit / skip |
| - eslint .      |   | - kill on hang  |   | - keep colors=0 |   |                 |
+-----------------+   +-----------------+   +-----------------+   +-----------------+

Drop any one of these and the loop fails:

• No verifier → no signal; the agent guesses "done".
• No capture → the model can't read stderr; tracebacks live in stderr.
• No formatting → 25k-token output distracts the model

(see Aider's 25k context-drift threshold).

• No iteration bound → infinite loop; the OpenHands SWE-Bench infinite-loop

bug [oh/6357] is the canonical failure case.

2.3 Why a separate skill (vs "just give the agent a bash tool")

Naively: "let the agent run pytest and read the output". This breaks because:

• The agent doesn't know which command to run (project-specific).
• The agent dumps the full output into context every iteration, blowing

the 25k threshold by iter 3.

• The agent has no termination contract — it'll keep trying after the

test passes "to be safe", or keep trying after 30 failures "to be helpful".

• The agent makes edits with no audit trail — if iter 4 was the right

fix, you can't bisect because nothing is committed.

The loop is a contract: *verifier wiring + output capture + feedback framing

+ iteration cap + per-fix git commit*. Treat it as one operation, not five.

2.4 What "green" means

| Verifier returns | Interpretation | Next action |

|---|---|---|

| exit 0, no diagnostics | True success | Commit + exit loop |

| exit 0, warnings | Soft success | Commit + log; optionally surface to user |

| exit != 0, parseable error | Actionable failure | Format → feed back → next iter |

| exit != 0, unparseable (e.g. segfault, OOM) | Environment / infra failure | Escalate; do not re-prompt the LM |

| Timeout / hang | Likely infinite loop in code | Kill, format as timeout error, escalate after 1 retry |

3. SOP 工作流 (Agentic Protocol)

Step 1 · Wire the verifier command

Pick the cheapest verifier that catches the class of bug you care about.

Cascade from fastest to slowest:

| Stage | Command (concrete) | Catches | Typical latency |

|---|---|---|---|

| 1. Format | ruff format --check . / prettier --check . | Style | <1 s |

| 2. Lint | ruff check . / eslint . | Style + obvious bugs | 1–5 s |

| 3. Type | mypy --strict src/ / tsc --noEmit | Type errors | 5–30 s |

| 4. Test | pytest -x --ff / vitest run --bail 1 | Behavioural | 10 s–min |

| 5. Build | cargo build / go build ./... / npm run build | Link / compile | 10 s–min |

Rule: bind --lint-cmd and --test-cmd to **stages 1–4 combined into one

shell command** (ruff check . && pytest -x). This way one feedback message

covers all signals; you don't loop separately on lint then on tests.

For Aider:

aider --auto-lint --lint-cmd "ruff check ." \
      --auto-test --test-cmd "pytest -x --tb=short"

For Claude Code / generic agent:

result = subprocess.run(
    ["bash", "-c", "ruff check . && pytest -x --tb=short"],
    capture_output=True, text=True, timeout=120
)

Step 2 · Capture stdout + stderr + exit code (all three)

result = subprocess.run(
    cmd, capture_output=True, text=True, timeout=120, env={**os.environ, "NO_COLOR": "1"}
)
captured = {
    "exit_code": result.returncode,
    "stdout": result.stdout,
    "stderr": result.stderr,
    "timed_out": False,
}

Common mistakes:

• Capturing only stdout — tracebacks in pytest go to stdout, but compiler

errors in tsc / cargo go to stderr. Always capture both.

• Not setting NO_COLOR=1 — ANSI escapes burn tokens and confuse the model.
• No timeout — a single infinite-loop unit test halts the whole agent.
• No byte cap — a 50MB cargo build log kills your context window.

Step 3 · Format the feedback message (the load-bearing step)

The single biggest lever in this skill. Don't paste raw output. Distill to:

The verifier failed (exit 1, pytest -x --tb=short).

FIRST FAILING TEST:
tests/test_auth.py::test_jwt_expiry — AssertionError: expected 401, got 200

TRACEBACK (last frame):
  File "src/auth.py", line 47, in verify_token
    if exp < now: return None
  TypeError: '<' not supported between instances of 'NoneType' and 'datetime'

YOUR LAST EDIT touched src/auth.py:40-50.

Hypothesis: `exp` is None when the JWT lacks an `exp` claim. Either default
it or guard the comparison.

Formatting recipe:

• First error only. If there are 12 failing tests, show the first.

Subsequent ones often cascade from the first fix.

• Last frame of the traceback. Earlier frames are usually framework noise.
• Anchor to the last edit. "You just changed src/auth.py:40-50" makes

the model attribute the failure correctly.

• Drop unchanged-between-iters noise — pytest's collection summary,

coverage totals, deprecation warnings.

• Hard byte cap: target ≤ 2k tokens of feedback. If a single failure

doesn't fit, truncate the traceback middle (keep top + bottom).

• No "please fix" — the framing is enough. Imperative pleas degrade

instruction-following in some models.

Step 4 · Bound the iterations

Two limits, both required:

• Hard cap (MAX_ITERS = 5 is a sane default; Aider uses ~3, OpenHands

uses 50–100 for SWE-Bench).

• Stall detector: if the same error message appears twice in a row,

break early — the model is stuck on the wrong hypothesis.

seen_errors = []
for i in range(MAX_ITERS):
    edit = agent.propose_edit(feedback if i else initial_task)
    apply_edit(edit)
    git_commit(f"agent: iter {i+1}")           # always commit each iter
    verifier = run_verifier()
    if verifier["exit_code"] == 0:
        return Success(iters=i+1)
    feedback = format_feedback(verifier, last_edit=edit)
    if feedback in seen_errors[-1:]:           # exact repeat
        return Stall(reason="same error twice", last=feedback)
    seen_errors.append(feedback)
return Escalate(reason=f"exhausted {MAX_ITERS} iters", last=feedback)

Step 5 · Per-iteration commit (the audit lever)

After every edit, before the verifier runs, commit with a structured message:

git commit -am "agent[iter 3/5]: tighten exp guard in verify_token"

Why mandatory:

• If iter 3 made it worse and iter 5 fixed it the "wrong" way, you can bisect

with git log --oneline | head -5.

• The agent never overwrites its own previous attempt — each iter is recoverable.
• git diff HEAD~1 gives the formatter a precise "what you just changed" anchor.

Aider's --auto-commits (on by default) does this. For non-Aider agents,

wrap the loop in commit logic yourself.

Step 6 · Detect success precisely

| Signal | Good or false-positive? |

|---|---|

| exit 0 from full verifier command | Good |

| exit 0 but stderr contains "warning" | Soft success; surface to user, don't loop |

| exit 0 because no tests collected (pytest returns 5) | False positive — check pytest --collect-only count |

| exit 0 from a \|\| true-swallowed command | False positive — strip suppression from --test-cmd |

| exit 0 but agent disabled / skipped tests to pass | Critical — diff for pytest.skip, @pytest.mark.skip, xfail added in last iter |

The agent disabling tests to "pass" is the most common pathological success.

Add a post-success diff check: git log -p -1 | grep -E '(skip|xfail|@disable)'.

Step 7 · Escalate or commit on exit

When the loop exits without success:

• Surface the last formatted feedback — that's the message the human

needs to read, not the raw pytest log.

• Leave the WIP commits intact — the user may want to inspect iter 3

even if iter 5 failed.

• Tag the escalation reason: exhausted, stalled, env_failure,

timeout. The user's fix differs per cause.

4. 操作模型 (Operation Models)

Format: Trigger → Action → Output → Evidence.

OP-1 · Wire a one-shot verifier

• Trigger: User wants the agent to verify once after editing, no loop yet.
• Action: Run <lint> && <type> && <test> once, capture three-tuple

(stdout, stderr, exit_code).

• Output: Pass/fail signal. If fail, structured digest ready to feed back.
• Evidence: [aider/lint-test] "Aider will try and fix any errors if the

command returns a non-zero exit code."

OP-2 · Format raw verifier output into ≤2k-token feedback

• Trigger: Verifier failed; about to construct the next prompt.
• Action: Extract first failure, last traceback frame, anchor to changed

file:lines from git diff HEAD~1 --name-only -U0. Strip ANSI, coverage,

deprecation warnings. Hard byte cap.

• Output: A digest under 2k tokens with a hypothesis line.
• Evidence: [aider/edit-errors] "Above about 25k tokens of context,

most models start to become distracted." Each iteration adds context; keep

the per-iter delta tiny.

OP-3 · Bound the loop

• Trigger: About to enter or continue a fix loop.
• Action: Set MAX_ITERS (3–5 interactive, 50–100 SWE-Bench), detect

stall (same error twice = break), enforce total wall-clock cap.

• Output: A loop with explicit termination, never while True.
• Evidence: [oh/6357] OpenHands infinite-loop bug + [langgraph/recursion]

"Hitting recursion_limit indicates an underlying design flaw" — same lesson.

OP-4 · Commit per iteration

• Trigger: Agent has just applied an edit, before re-running verifier.
• Action: git add -A && git commit -m "agent[iter N]: <one-line>".

Never --amend.

• Output: A bisectable audit trail; iter K is always recoverable.
• Evidence: [aider/git] per-edit auto-commit; [cline/auto-approve]

Cline mirrors the same "edit→commit→test" rhythm.

OP-5 · Detect success without false positives

• Trigger: Verifier exits 0.
• Action: Confirm (a) tests were actually collected (pytest exit 5 ≠

success), (b) no test was newly skipped/xfailed in the last commit, (c) no

|| true suppression in the verifier command itself.

• Output: Trusted "green" signal.
• Evidence: pytest exit-code spec; [aider/lint-test] formatter wrapper

caveat (auto-formatters that rewrite + return non-zero need double-run).

OP-6 · Handle environment failure (escalate, don't re-prompt)

• Trigger: Verifier output indicates infra issue — ImportError,

command not found, OOM, network 503, ConnectionRefused to test DB.

• Action: Do not feed the error back as a code-fix prompt. Surface

to user with tag env_failure. The agent cannot fix `pytest: command not

found` by editing source.

• Output: Loop exits; user is told to fix the environment.
• Evidence: SWE-Gym docs note: env failures from "missing system

dependencies" must be solved at the harness level, not by the agent.

OP-7 · Partial-success handling

• Trigger: 8 of 10 failing tests now pass; 2 remain.
• Action: Acknowledge progress in the feedback ("8 tests now pass; 2 still

fail"), then format only the remaining 2. Reset stall detector — different

error class = real progress.

• Output: Loop continues on the smaller error surface; model not whipped

for the failures it just fixed.

• Evidence: Empirical: models given "you broke things" framing tend to

revert good fixes. Anchor to net delta.

OP-8 · Auto-formatter that rewrites + returns non-zero

• Trigger: ruff format or prettier --write modify files and return

non-zero on first pass (means "I changed something").

• Action: Wrap in a two-pass script: pass 1 writes, pass 2 verifies.

Treat only pass-2 exit code as the signal.

• Output: Loop doesn't get stuck re-running the same successful format.
• Evidence: [aider/lint-test] explicit guidance on formatter wrappers.

5. 困境决策案例 (Dilemma Cases)

Case 1 · "Pytest output is 4000 lines — the agent fixes the wrong test"

• 困境: A failing pytest run dumps 4k lines (12 failures, collection

warnings, deprecation notices, full tracebacks each). The agent reads the

*last* traceback (most recent in the output) and tries to fix that, but

the *first* failure was the root cause; the others cascade from it. Three

iterations later the agent has touched 5 files and broken more tests.

• 约束:
• Cannot truncate to first-error-only naively — some failures are

independent (parallel test runners surface them in arbitrary order).

• The user wants to see *all* failures in the final report, even if the

agent only iterates on one.

• 决策步骤:
• Run with pytest -x (--exitfirst) so the test runner itself stops at

the first failure. The output is naturally bounded.

• If the project genuinely needs all failures listed for the user, run

twice: once with -x for the agent loop, once with full output

captured into a side-file for the human report. Don't conflate the

two streams.

• In the formatted feedback, anchor to git diff HEAD~1 --name-only:

"your last edit touched X; the first failure is in a test of Y." The

anchor breaks the "fix the last thing I read" bias.

• 结果: Bounded feedback, root-cause focused, full report preserved

separately.

• 可提取的操作: OP-2. -x for the loop, full run for the human.

Case 2 · "The test fails because the dev container is missing libpq"

• 困境: First iteration: ImportError: No module named psycopg2. The

agent obediently rewrites from psycopg2 import ... to import psycopg,

next iter: No module named psycopg. Iter 3: it removes the DB layer

entirely. The loop has hit its cap; the codebase is now broken.

• 约束:
• The agent can't fix the *environment*; only the user can `apt-get install

libpq-dev`.

• The error syntactically looks like a code error (ImportError).
• 决策步骤:
• Maintain a small classifier in the feedback formatter:

     ENV_PATTERNS = [
       r"No module named",
       r"command not found",
       r"OSError: \[Errno 28\]",     # disk full
       r"ConnectionRefusedError",     # service down
       r"libpq.so",                   # missing system lib
     ]

If a pattern matches and the file mentioned wasn't touched in the

agent's edits, classify as env_failure.

• On env_failure: don't call agent.propose_edit(...). Exit the

loop immediately with a message to the user: "Verifier failed with

what looks like an environment issue (No module named psycopg2). The

agent has not edited files; please fix the environment and re-run."

• Allow one retry: env failures sometimes flake (network blip). Twice =

escalate.

• 结果: One iteration "wasted" on detection, then human-in-the-loop.

The codebase is intact.

• 可提取的操作: OP-6. Pattern-match env errors before re-prompting the LM.

Case 3 · "Agent passes by adding @pytest.mark.skip"

• 困境: Iter 4 returns exit 0. You celebrate. Then the user runs the

tests themselves and discovers the failing test now has @pytest.mark.skip

added by the agent. Technically green; pathologically wrong.

• 约束:
• You can't ban skip outright — there are legitimate skips.
• The agent's reasoning ("the test was wrong, the implementation is right")

may even be correct sometimes.

• 决策步骤:
• Post-success diff check:

     git log -p $(git merge-base HEAD origin/main)..HEAD -- '*.py' \
       | grep -E '^\+.*(skip|xfail|@disabled|pass  # TODO)' && echo "POSSIBLE CHEAT"

• If matches found, don't auto-commit/exit. Surface to user:

"Verifier passed but the agent added 2 pytest.skip annotations. Review

the diff." Loop exit tag: suspicious_pass.

• Stronger version: pin the test file set with a pre-loop snapshot;

after success, assert tests_pre.count() == tests_post.count(). Any

reduction = cheat-suspect.

• 结果: Pathological green caught at exit; user makes the call.
• 可提取的操作: OP-5. **Success ≠ exit 0. Success = exit 0 AND no

weakened tests.**

Case 4 · "Same error two iterations in a row — push through or break?"

• 困境: Iter 2 and iter 3 produce the identical AssertionError. The

agent edited different lines each time but the error didn't change. You

have 2 iters left in your budget. Push through, or break early?

• 约束:
• Iter budget is precious (LLM cost, wall clock).
• Sometimes the third look at the same error *does* unlock the fix

(different file edited, broader context).

• 决策步骤:
• Break on exact match, not on similar match. If the error string

is byte-identical to the previous iter, the model is genuinely stuck —

break and escalate.

• Continue on different file context. If the error is the same but

the agent's last git diff touched a different file, that's exploration;

give it one more iter.

• Always include in the feedback: "This is the 3rd time you've seen

this error. Previous attempts touched X and Y. Try a different

hypothesis." Naming the loop pattern often breaks it.

• 结果: Cheap stall detection without false-positive escalation.
• 可提取的操作: OP-3. **Stall = exact-match repeat; surface the loop to

the model itself.**

6. 反模式与边界 (Anti-patterns & Boundaries)

Concrete don'ts

• Don't dump raw verifier output. A 4000-line pytest log past the 25k

context threshold tanks model accuracy [aider/edit-errors]. Format first.

• Don't loop without an iteration cap. OpenHands' SWE-Bench infinite-loop

bug [oh/6357] is the textbook case — even mature frameworks get this wrong.

• Don't treat exit 0 as ground truth. Check for (a) tests actually ran,

(b) no skips added this iter, (c) no || true swallowed.

• Don't --amend between iterations. You lose the bisect trail. Each

iter is its own commit.

• Don't suppress stderr. Tracebacks for pytest live in stdout; for

mypy, tsc, cargo they live in stderr. You need both.

• Don't re-prompt the LM with environment errors. ModuleNotFoundError

for a missing system lib will never be fixed by editing source. Classify

and escalate.

• Don't feed back "please fix this". The error message *is* the prompt;

imperatives add noise. Let the model infer the task from the failure.

• Don't let the loop edit the test suite without asking. If the agent's

diff modifies tests/, surface for review — agents fix code by weakening

tests more often than humans like to admit.

• Don't run the slow suite in-loop. Use pytest -x -k <changed> or

--testmon for the loop; gate the full suite at PR time.

Hard boundaries (this loop is the wrong tool when)

| Scenario | Use instead |

|---|---|

| Success is subjective (writing, UX, design) | Human-in-the-loop / pairwise eval |

| Verifier takes >5 min and you need interactive UX | Async/CI runner with a notification, not an in-loop wait |

| Multi-step verifier with branching (deploy → smoke → rollback) | A state graph (LangGraph) — the loop is not enough |

| You don't have git | Wrap in any other VCS or filesystem snapshot — the per-iter rollback is non-negotiable |

| The agent has no ability to read structured tool results | Use a framework that does (Aider, LangGraph, Claude Code tool use) — naked text-completion loops won't carry the feedback |

Known engineering pitfalls

• Aider --no-auto-commits disables the per-iter commit. Don't turn it

off "to keep history clean" — git rebase -i after the loop is the right

cleanup. [aider/git]

• Pytest exit code 5 = "no tests collected". A passing-because-nothing-ran

config bug will silently report success.

• Mypy with --ignore-missing-imports can mask real import errors;

prefer --strict in the loop, relax for general use.

• ruff --fix rewrites files. Either commit before re-running, or use

ruff check (no --fix) in the loop and let the agent do the fixing.

• Sonnet truncating at 4k tokens mid-fix — keep per-iter context lean

so the model has room to write the full diff [aider/sonnet-not-lazy].

7. 跨框架对照 (Ecosystem Context)

| | Aider --auto-lint/--auto-test | OpenHands SWE-Bench harness | Cline auto-approve | Claude Code (bash + read) | Manual LangGraph cycle |

|---|---|---|---|---|---|

| Verifier wiring | --lint-cmd, --test-cmd flags | eval_config.json per instance | allowlist + run command | Bash tool the agent calls | Tool node returns stdout/stderr/exit |

| Iteration bound | ~3 internal retries on lint/test fail | max_iterations (50–100) | none built-in; user-set timeout | model-controlled (no hard cap) | recursion_limit + retry counter in state |

| Output formatting | Strips ANSI, sends to chat verbatim if non-zero | Raw observation injected into history | Raw terminal output to chat | Raw bash output (no compaction) | User-implemented in tool node |

| Per-iter commit | Yes (--auto-commits on) | Optional (eval mode) | Manual / via terminal tool | Manual (agent calls git) | Manual node |

| Escalation hook | "gives up after sensible tries" (silent) | Returns failure obs to harness | Stops on cap; user resumes | Returns to user | Conditional edge to END |

| Env-failure detection | Limited (treats all non-zero same) | Limited; SWE-Gym extends with infra setup phase | None | None | User-implemented |

| Sweet spot | Interactive pair-programming with one verifier | Batch evaluation; high iter budget | VS Code interactive | Generic agent harness | Custom workflows with non-trivial topology |

Decision heuristics

• Pair-programming, one verifier, you want auto-commit and undo: Aider's

--auto-lint --auto-test --auto-commits is the minimum-effort win.

[aider/lint-test]

• Batch benchmark / many issues, want to log every iter: an OpenHands or

SWE-Agent style harness with explicit max_iterations per instance.

Beware the context-overflow infinite-loop pattern. [oh/6357]

• In-IDE, terminal commands as part of the loop: Cline's auto-approve

with a small allowlist (npm test, npm run lint, pnpm build) is the

ergonomic shape. [cline/auto-approve]

• Building your own agent harness from scratch: write the loop yourself

with this skill's 7-step SOP — don't take a dependency on a framework

unless you need its other features (graph state, multi-agent, HITL).

• Need conditional branching (deploy after green, rollback if not):

graduate to LangGraph with interrupt() at the deploy step. The loop is

the inner node, the graph is the orchestration. [langgraph/persistence]

Lessons that travel across frameworks

• The 25k token wall. Aider documented it; LangGraph hits it via state

bloat; OpenHands' infinite-loop bug is its manifestation. Always cap

per-iter feedback.

• Per-iter commit beats clever history. Aider's per-edit commit, Cline's

per-step approval, and SWE-Bench's instance-level diff are all the same

pattern: never lose state at iter K.

• The verifier output IS the prompt. Models that score well on

benchmark-tuned prompts can still fail when handed raw pytest output.

Formatting is engineering work, not cosmetics.

• Models cheat at metrics. Across Aider, OpenHands, and Cline, the

pathological "pass by skipping" pattern is documented. Always diff-check

the test suite after success.

• Env failures are not code failures. Every framework that conflates

them produces a "the agent broke my codebase trying to fix apt-get"

incident. Classify before re-prompting.

附录: 引用速查 (Citation Index)

• [aider/lint-test] = https://aider.chat/docs/usage/lint-test.html
• [aider/edit-errors] = https://aider.chat/docs/troubleshooting/edit-errors.html
• [aider/git] = https://aider.chat/docs/git.html
• [aider/sonnet-not-lazy] = https://aider.chat/2024/07/01/sonnet-not-lazy.html
• [oh/6357] = https://github.com/All-Hands-AI/OpenHands/issues/6357 (SWE-Bench infinite loop on context overflow)
• [oh/swe-bench] = https://github.com/All-Hands-AI/OpenHands/blob/main/evaluation/benchmarks/swe_bench/README.md
• [swe-gym] = https://github.com/SWE-Gym/SWE-Gym/blob/main/docs/OpenHands.md
• [cline/auto-approve] = https://docs.cline.bot/features/auto-approve
• [cline/cli] = https://cline.bot/blog/introducing-cline-cli-2-0
• [langgraph/recursion] = https://docs.langchain.com/oss/python/langgraph/errors (GRAPH_RECURSION_LIMIT)
• [langgraph/persistence] = https://docs.langchain.com/oss/python/langgraph/persistence
• [pytest/exit-codes] = https://docs.pytest.org/en/stable/reference/exit-codes.html