Debug Test Failures

Debug Test Failures

Overview

A systematic process for diagnosing and fixing test failures. Use this skill when tests fail, CI is red, or someone reports a broken build. The key principle: find the root cause, not just the symptom.

Key Concepts

Failure Categories

• Deterministic failure: Same result every time. Usually a code bug or outdated test.
• Flaky failure: Intermittent. Usually caused by timing, ordering, shared state, or external dependencies.
• Environment failure: Works locally, fails in CI. Usually paths, permissions, or missing dependencies.

Patterns & Techniques

Systematic Debugging Process

Step 1: Reproduce

Run the failing test in isolation:

# Python tests
cd hastelib && hatch run test:pytest

# UI lint
cd ui && npm run lint

Run it 2-3 times. If intermittent, it's likely flaky.

Step 2: Analyze

• Read the error message and stack trace carefully
• Identify the assertion that failed: expected vs actual values
• Check if the failure is in the test or the code under test

Step 3: Investigate

• Read the test code to understand what it's testing
• Read the production code being tested
• Check recent changes to both
• Common root causes:
• Changed function signatures or return types
• Modified data structures
• New dependencies not mocked
• Environment differences (paths, timezones, locale)
• Race conditions in async code

Step 4: Fix

• Fix the root cause, not just the symptom
• If the test is wrong (testing outdated behavior), update the test
• If the code is wrong, fix the code
• Run the full test suite: cd hastelib && hatch run test:pytest

Step 5: Report

Summarize: root cause, fix applied, verification that all tests pass.

Decision Framework

| Symptom | Likely Cause | Action |

|---------|-------------|--------|

| Test always fails | Code bug or outdated test | Read both, fix the wrong one |

| Test passes locally, fails in CI | Environment difference | Check paths, deps, permissions |

| Test fails intermittently | Timing, shared state, or ordering | Add isolation, remove shared state |

| Many tests fail at once | Breaking change in shared code | Find the common dependency |

Common Pitfalls

• Fixing the test to match broken code — Verify the code is correct before changing the test
• Adding sleep() to fix flaky tests — Fix the root cause (usually missing await or shared state)
• Only running the fixed test — Always run the full suite to check for regressions