Mutation Testing

Mutation Testing

This skill makes an AI agent set up and interpret mutation testing: StrykerJS for JS/TS projects (Jest or Vitest runners) and mutmut for Python. Mutation testing injects small bugs (mutants) into source code and re-runs the tests — if the tests still pass, the mutant "survived" and your suite has a blind spot that line coverage never showed. Trigger this when a team claims high coverage but still ships regressions, when reviewing test suite quality, or when the user mentions Stryker, mutmut, or mutation score.

Core Principles

• Line coverage measures execution, mutation score measures verification. A test that calls a function and asserts nothing gives 100% line coverage and a 0% mutation score. Mutation score is the honest metric.
• Never mutate everything on every run. A full mutation run on a real codebase takes hours. Use incremental mode locally and scope CI runs to changed files. Full runs belong in nightly jobs.
• A surviving mutant is either a missing assertion or dead code. Both are findings. If you cannot write a test that kills a mutant, the mutated code path is unreachable — delete it instead of suppressing the mutant.
• Set break thresholds, not aspirational targets. Start at your current score minus 2, fail the build below it, and ratchet up monthly. A threshold nobody enforces is documentation fiction.
• Equivalent mutants exist; budget for them. Some mutations (e.g., i < len to i <= len on an array that is never exactly full) produce identical behavior. Mark them ignored explicitly rather than chasing 100%.
• Kill mutants with stronger assertions, not more tests. The fix for a survived > → >= mutant is usually one boundary-value assertion in an existing test, not a new test file.

Setup: StrykerJS (TypeScript + Vitest)

npm install --save-dev @stryker-mutator/core @stryker-mutator/vitest-runner
npx stryker init

// stryker.config.json
{
  "$schema": "./node_modules/@stryker-mutator/core/schema/stryker-schema.json",
  "testRunner": "vitest",
  "mutate": ["src/**/*.ts", "!src/**/*.test.ts", "!src/**/*.d.ts", "!src/generated/**"],
  "coverageAnalysis": "perTest",
  "reporters": ["html", "clear-text", "progress", "json"],
  "thresholds": { "high": 85, "low": 70, "break": 65 },
  "incremental": true,
  "incrementalFile": ".stryker-tmp/incremental.json",
  "timeoutMS": 10000,
  "concurrency": 4
}

For Jest projects swap the runner:

npm install --save-dev @stryker-mutator/core @stryker-mutator/jest-runner

// stryker.config.json (Jest variant)
{
  "testRunner": "jest",
  "jest": { "projectType": "custom", "configFile": "jest.config.js" },
  "coverageAnalysis": "perTest",
  "mutate": ["src/**/*.ts", "!src/**/*.test.ts"],
  "thresholds": { "high": 85, "low": 70, "break": 65 }
}

Run it:

npx stryker run                          # full run, writes reports/mutation/mutation.html
npx stryker run --mutate src/pricing.ts  # scope to one file while fixing survivors

Reading a Survived Mutant and Killing It

Stryker output:

[Survived] ArithmeticOperator
src/pricing.ts:14:31
-   return subtotal * (1 - discountRate);
+   return subtotal / (1 - discountRate);
Ran 3 tests, none failed.

The suite never asserts a discounted price with a nonzero rate. Kill it with a boundary assertion:

// src/pricing.test.ts
import { describe, expect, it } from 'vitest';
import { applyDiscount } from './pricing';

describe('applyDiscount', () => {
  it('multiplies subtotal by the inverse discount rate', () => {
    // 200 * (1 - 0.25) = 150; the division mutant would return 266.67
    expect(applyDiscount(200, 0.25)).toBe(150);
  });

  it('returns the subtotal unchanged at rate 0', () => {
    expect(applyDiscount(99.5, 0)).toBe(99.5);
  });
});

Ignore a genuinely equivalent mutant inline instead of lowering the threshold:

// Stryker disable next-line EqualityOperator: loop bound is equivalent for empty input
for (let i = 0; i < items.length; i++) {

Setup: mutmut (Python + pytest)

pip install mutmut pytest

# pyproject.toml
[tool.mutmut]
paths_to_mutate = ["src/"]
tests_dir = ["tests/"]
also_copy = ["conftest.py"]

mutmut run                 # mutates src/, runs pytest per mutant, caches results
mutmut results             # summary: killed / survived / timeout / suspicious
mutmut show src.pricing.apply_discount__mutmut_3   # diff of one survivor
mutmut run --max-children 4                        # parallel workers

Killing a Python survivor follows the same pattern — the mutant tells you the missing assertion:

# tests/test_pricing.py
import pytest
from src.pricing import apply_discount


def test_apply_discount_uses_multiplication():
    # mutant changed * to /; 200 * 0.75 == 150, 200 / 0.75 == 266.67
    assert apply_discount(200, 0.25) == 150


def test_apply_discount_rejects_rate_above_one():
    # kills the mutant that removed the validation guard
    with pytest.raises(ValueError):
        apply_discount(100, 1.5)

CI Integration

Incremental Stryker on pull requests, full run nightly:

# .github/workflows/mutation.yml
name: mutation
on:
  pull_request:
    paths: ['src/**', 'stryker.config.json']
  schedule:
    - cron: '0 2 * * *'

jobs:
  stryker:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0 # incremental mode diffs against git history
      - uses: actions/setup-node@v4
        with:
          node-version: 20
          cache: npm
      - run: npm ci
      - name: Restore incremental cache
        uses: actions/cache@v4
        with:
          path: .stryker-tmp/incremental.json
          key: stryker-incremental-${{ github.base_ref || 'main' }}
      - name: Mutation test (incremental on PRs, full nightly)
        run: |
          if [ "${{ github.event_name }}" = "pull_request" ]; then
            npx stryker run --incremental
          else
            npx stryker run --force
          fi
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: mutation-report
          path: reports/mutation/mutation.html

The thresholds.break: 65 in config makes stryker run exit non-zero below 65% — no extra scripting needed.

Best Practices

• Start mutation testing on your most critical module (pricing, auth, parsers), not the whole repo. One high-signal report beats a week-long full run.
• Use coverageAnalysis: "perTest" — Stryker then only runs the tests that cover each mutant, often a 10x speedup.
• Commit .stryker-tmp/incremental.json to CI cache, never to git.
• Track mutation score per module in the JSON report; aggregate scores hide a 30% module behind a 90% repo average.
• Set timeoutMS explicitly. Mutants that create infinite loops are killed by timeout; the default factor-based timeout misbehaves on very fast suites.
• In Python, run mutmut run against a single module first: paths_to_mutate = ["src/billing.py"].

Anti-Patterns

• Chasing 100% mutation score. Past ~90% you are writing tests for equivalent mutants. Mark them ignored with a reason and move on.
• Running full mutation on every PR. It will take 40+ minutes and the team will delete the workflow within a month. Incremental on PRs, full nightly.
• Treating "survived" as a Stryker bug. Over years of use, genuine false positives are rare; assume your suite is the problem first.
• Mutating test files or generated code. Always exclude *.test.ts, *.spec.ts, migrations, and codegen output in mutate globs.
• Lowering break to make a red build green. The threshold ratchets up, never down; fix the survivors or explicitly ignore equivalents inline.
• Writing assertion-free "kill tests" that snapshot huge objects just to kill mutants. A surviving mutant deserves a precise boundary assertion, not a snapshot blanket.

When to Trigger This Skill

• The user mentions Stryker, mutmut, mutation score, or "are my tests actually good".
• A project has high line coverage but recurring production regressions in covered code.
• Reviewing or hardening a test suite for a critical module (payments, auth, data migration).
• Setting up CI quality gates beyond line coverage thresholds.
• A stryker.config.json, stryker.conf.mjs, or [tool.mutmut] section already exists in the repo.