Bisect Agent Skill

ClickHouse Bisect Skill

Bisect a ClickHouse regression using utils/auto-bisect/bisect.sh, which downloads pre-built CI binaries and manages the bisect loop automatically.

Arguments

• $0 (required): Path to a SQL repro file (e.g., /tmp/repro.sql)
• $1 (optional): Good ref — a git tag, branch, or SHA known to be good (default: latest v*.1.1-* tag)
• $2 (optional): Bad ref — a git tag, branch, or SHA known to be bad (default: HEAD)

How the auto-bisect script works

utils/auto-bisect/bisect.sh orchestrates the full bisect:

• For each commit, helpers/download.sh fetches the release binary from CI S3
• env/<option>.sh installs configs and starts a ClickHouse server (configs go to utils/auto-bisect/data/etc/, never to /etc/)
• Your test script is called with $COMMIT_SHA, $CH_PATH (binary), and $GIT_WORK_TREE in the environment
• Exit code from your test determines good/bad/skip

The working tree is never checked out — BISECT_HEAD is written for each step so git history is untouched.

Test script contract

The test script receives these environment variables:

• $COMMIT_SHA — full SHA of the commit being tested
• $CH_PATH — path to the downloaded binary (e.g., utils/auto-bisect/data/clickhouse)
• $GIT_WORK_TREE — path to the ClickHouse repository

Exit codes:

• 0 — good (bug not present)
• 1 — bad (bug present)
• 125 — skip (handled upstream by download.sh, not typically needed in the test itself)

Process

1. Validate inputs

• Read and understand the repro SQL
• Resolve good/bad refs to full SHAs: git rev-parse <ref>
• If no good ref given, find the previous release branch point:

  git tag -l 'v*' --sort=-creatordate | grep '\.1\.1-' | head -5

Pick the nearest v<MAJOR>.<MINOR>.1.1-* tag before the bad ref.

2. Write the test script

Create /tmp/bisect_test.sh. Use $CH_PATH for the binary — do not download it yourself, the framework handles that.

For clickhouse local repros (no server needed — use --env nothing):

#!/bin/bash
set -e

OUTPUT=$("$CH_PATH" local --multiquery 2>&1 <<'SQL'
-- paste repro SQL here
SQL
)

if echo "$OUTPUT" | grep -q "FAILURE_PATTERN"; then
    exit 1  # bad
else
    exit 0  # good
fi

For server-based repros (use --env single, the default):

#!/bin/bash
# Server is already running when this script is called.
# Use clickhouse-client on the default port (9000).
set -e

OUTPUT=$("$CH_PATH" client --multiquery 2>&1 <<'SQL'
-- paste repro SQL here
SQL
)

if echo "$OUTPUT" | grep -q "FAILURE_PATTERN"; then
    exit 1
else
    exit 0
fi

Failure pattern examples:

• Exception: grep -q "LOGICAL_ERROR\|Code: [0-9]"
• Wrong result: compare OUTPUT to expected string
• Segfault: [ "${PIPESTATUS[0]}" = "139" ]

3. Confirm the bug range

Before running the full bisect, verify the repro:

GOOD_SHA=$(git rev-parse <good-ref>)
BAD_SHA=$(git rev-parse <bad-ref>)

Optionally do a quick walker sanity check over just the two endpoints:

CH_PATH=/usr/local/bin/clickhouse /tmp/bisect_test.sh  # should exit 0 on good

4. Run bisect

cd utils/auto-bisect
./bisect.sh \
  --good <good-sha> \
  --bad  <bad-sha> \
  --path /path/to/ClickHouse \
  --test /tmp/bisect_test.sh \
  --env  nothing          # or: single (default), replicateddb, sharedcatalog

Use --env nothing when the test uses clickhouse local (no server).

Use --env single (default) when the test needs a running server.

Walker mode — walk every commit linearly instead of bisecting (useful when you want to see the progression or the range is small):

./bisect.sh --good <sha> --bad <sha> --path ... --test ... --walker
# Or provide explicit commits:
./bisect.sh --path ... --test ... --walker --walker-commits "sha1 sha2 sha3"
# Or limit to N evenly-spaced steps:
./bisect.sh --good <sha> --bad <sha> --path ... --test ... --walker --walker-steps 10

Set a generous timeout (up to 10 min) — each step downloads ~200 MB.

5. Report findings

After bisect completes, the output contains <sha> is the first bad commit.

git show --stat <bad-sha>

Report:

• First bad commit SHA and message
• PR number from message (e.g., "Merge pull request #NNNNN") → link to https://github.com/ClickHouse/ClickHouse/pull/<N>
• Brief description from git show --stat
• Any skipped commits (no CI binary) that fall in the range

Notes

• Binaries are downloaded to utils/auto-bisect/data/clickhouse and overwritten each step (not cached between steps — use walker if you want to re-test)
• Only first-parent merge commits on master have CI binaries; bisect.sh uses --first-parent automatically
• Configs are installed to utils/auto-bisect/data/etc/clickhouse-server/ — the host's /etc/clickhouse-* is never touched
• The working tree is never modified during bisect or walker runs; BISECT_HEAD is used instead of git checkout
• For private CI builds, set CH_CI_USER / CH_CI_PASSWORD and pass --private