vibeeval/mutation-testing
Mutation testing ile test suite kalitesini olc. Stryker, mutmut, go-mutesting destegi.
npx skills add https://github.com/vibeeval/vibecosystem --skill mutation-testingMutation testing, test suite'inin kalitesini olcen bir tekniktir. Kaynak kodda kucuk degisiklikler (mutasyonlar) yapilir ve testlerin bu degisiklikleri yakalayip yakalamadigina bakilir.
Code coverage "kodun ne kadari calistiriliyor?" sorusunu yanitlar.
Mutation testing "testler gercekten bir seyi kontrol ediyor mu?" sorusunu yanitlar.
%100 code coverage'a sahip ama assertion'i olmayan testler mutation testing'de FAIL alir.
# Install
npm install --save-dev @stryker-mutator/core
npx stryker init
# Jest runner
npm install --save-dev @stryker-mutator/jest-runner
# Vitest runner
npm install --save-dev @stryker-mutator/vitest-runner
# TypeScript support
npm install --save-dev @stryker-mutator/typescript-checker
Config (stryker.config.mjs):
/** @type {import('@stryker-mutator/api/core').PartialStrykerOptions} */
export default {
mutate: [
'src/**/*.ts',
'!src/**/*.test.ts',
'!src/**/*.spec.ts',
'!src/**/*.d.ts',
'!src/**/index.ts'
],
testRunner: 'jest',
checkers: ['typescript'],
reporters: ['html', 'clear-text', 'progress', 'json'],
coverageAnalysis: 'perTest',
thresholds: {
high: 80,
low: 60,
break: null // Set to 60 to fail CI on low kill ratio
},
timeoutMS: 60000,
concurrency: 4
};
Run:
npx stryker run
# Report: reports/mutation/mutation.html
pip install mutmut
Config (pyproject.toml):
[tool.mutmut]
paths_to_mutate = "src/"
tests_dir = "tests/"
runner = "python -m pytest -x --tb=short -q"
dict_synonyms = "Struct,NamedStruct"
Run:
# Full run
mutmut run
# Results
mutmut results
# Show specific mutant
mutmut show 42
# HTML report
mutmut html
go install github.com/zimmski/go-mutesting/cmd/go-mutesting@latest
Run:
# Full run
go-mutesting ./...
# Specific package
go-mutesting ./pkg/calculator/...
# With score threshold
go-mutesting --score 0.8 ./...
a + b -> a - b, a * b, a / b
a * b -> a / b, a + b
a++ -> a--
Neyi test eder: Matematiksel hesaplamalarin dogrulugu
a > b -> a >= b
a < b -> a <= b
a >= b -> a > b
a <= b -> a < b
Neyi test eder: Boundary condition'lar, off-by-one hatalari
true -> false
a && b -> a || b
a || b -> a && b
!a -> a
Neyi test eder: Boolean logic, branch coverage
if (condition) -> if (!condition)
while (x > 0) -> while (x <= 0)
Neyi test eder: Kontrol akisinin dogrulugu
return x -> return 0
return true -> return false
return "hello" -> return ""
return obj -> return null
Neyi test eder: Return value assertion'lari
"hello" -> ""
"hello" -> "Stryker was here!"
Neyi test eder: String handling, empty string kontrolu
doSomething(); -> (removed)
x = calculate() -> (removed)
Neyi test eder: Side effect'lerin test edilip edilmedigi
| Seviye | Kill Ratio | Anlami |
|--------|-----------|--------|
| Mukemmel | 90%+ | Test suite cok guclu |
| Iyi | 80-89% | Kabul edilebilir, kucuk iyilestirmeler |
| Orta | 60-79% | Ciddi iyilestirme gerekli |
| Zayif | < 60% | Test suite guvenilemez |
Hedef: Her projede minimum %80 kill ratio
Bir mutant survive ettiyse su adimlari takip et:
Dosya: src/calculator.ts:15
Original: if (balance > 0) { ... }
Mutant: if (balance >= 0) { ... }
Durum: SURVIVED
balance === 0 durumunu test etmiyorit('should handle zero balance', () => {
const result = processBalance(0);
expect(result).toBe('no_funds'); // Bu test mutant'i oldurur
});
npx stryker run --mutate "src/calculator.ts"
Survived mutant > -> >= ise:
// Her boundary icin 3 test yaz: altinda, ustunde, tam sinirda
it('rejects when below minimum', () => expect(validate(-1)).toBe(false));
it('rejects at exact minimum', () => expect(validate(0)).toBe(false));
it('accepts above minimum', () => expect(validate(1)).toBe(true));
Survived mutant return x -> return 0 ise:
// Testlerde return value'yu MUTLAKA assert et
const result = calculate(5, 3);
expect(result).toBe(8); // Spesifik deger kontrolu
Survived mutant && -> || ise:
// Her boolean kombinasyonu test et
it('fails when only A is true', () => expect(check(true, false)).toBe(false));
it('fails when only B is true', () => expect(check(false, true)).toBe(false));
it('passes when both are true', () => expect(check(true, true)).toBe(true));
it('fails when both are false', () => expect(check(false, false)).toBe(false));
Survived mutant statement removal ise:
// Side effect'leri de test et
calculate(5);
expect(mockLogger.info).toHaveBeenCalledWith('Calculated: 5');
expect(mockMetrics.increment).toHaveBeenCalledWith('calculations');
Survived mutant !x -> x ise:
// Her iki yolu da test et
it('handles truthy input', () => expect(process(true)).toBe('A'));
it('handles falsy input', () => expect(process(false)).toBe('B'));
name: Mutation Testing
on:
pull_request:
branches: [main]
schedule:
- cron: '0 2 * * 0' # Haftalik tam tarama
jobs:
mutation-test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: 20
cache: npm
- run: npm ci
- run: npx stryker run
- uses: actions/upload-artifact@v4
with:
name: mutation-report
path: reports/mutation/
- name: Check kill ratio
run: |
SCORE=$(cat reports/mutation/mutation.json | jq '.schemaVersion' -r)
# Custom threshold check script
mutation-test:
stage: test
script:
- npm ci
- npx stryker run
artifacts:
paths:
- reports/mutation/
expire_in: 7 days
only:
- merge_requests
allow_failure: true # Ilk baslarken, sonra kaldir
PR'larda sadece degisen dosyalari mutate et:
- name: Get changed files
id: changed
run: |
FILES=$(git diff --name-only origin/main...HEAD -- '*.ts' | grep -v test | tr '\n' ',')
echo "files=$FILES" >> $GITHUB_OUTPUT
- name: Run incremental mutation
if: steps.changed.outputs.files != ''
run: npx stryker run --mutate "${{ steps.changed.outputs.files }}"
Sadece degisen dosyalari mutate et:
# Stryker
npx stryker run --mutate "src/changed-file.ts"
# mutmut
mutmut run --paths-to-mutate src/changed_module/
Stryker'da coverageAnalysis: 'perTest' kullan. Her mutant sadece ilgili testlerle calistirilir.
Sonsuz donguye giren mutant'lar icin makul timeout:
timeoutMS: 60000, // 60 saniye max
timeoutFactor: 1.5 // Normal surenin 1.5 kati
CPU sayisina gore paralel calistir:
concurrency: 4 // veya os.cpus().length - 1
Onceki sonuclari cache'le:
incremental: true,
incrementalFile: 'reports/stryker-incremental.json'
Bazi mutasyonlar kodun davranisini degistirmez:
// Original
const i = 0;
// Mutant (equivalent - davranis ayni)
const i = -0;
Cozum: Equivalent mutant'lari rapordan cikar, survived olarak sayma.
while (true) veya for(;;) gibi durumlar:
Cozum: Timeout ayarini dogru yap, timeout mutant'larini "killed" say.
Buyuk codebase'lerde saatlerce surebilir:
Cozum: Incremental mode, per-test coverage, parallelism kullan.
Mutant, baska testleri de etkiler:
Cozum: Testlerin bagimsiz oldugunu dogrula, shared state kullanma.
Flaky testler mutant'lari yanlis killed gosterebilir:
Cozum: Once flaky testleri duzelt, sonra mutation test calistir.
Config dosyalarini mutate etmenin anlami yok:
Cozum: mutate pattern'indan config, constants, types dosyalarini haric tut.
Bu skill su durumlarda aktive olur:
Take vibeeval/mutation-testing from the repository into ~/.claude/skills for personal
use, or into .claude/skills inside a project.
The agent identifies a skill by the name field in its header. Two skills with the
same name cannot sit side by side — one of them will be ignored.
The instructions reference pip, npm, npx, go.
Without those the skill loads but fails at the first command.