Debug Reference Tests

Debugging ClickHouse reference test failures

The reference suites in tests/ parse every .sql file in

tests/clickhouse-reference/ and compare this library's output against the

committed expected files. Each suite checks one dimension:

| Test suite | Compares against | Expected file suffix | Diff command |

| ------------------------------------------- | ----------------------------- | --------------------------- | --------------- |

| clickhouse-reference-ast.test.ts | parse() (meta stripped) | .expected.ast.json | diff:ast |

| clickhouse-reference-format.test.ts | format(parse(...)) | .expected.formatted.sql | diff:format |

| clickhouse-reference-explain.test.ts | formatExplain(parse(...)) | .expected.explain.txt | diff:explain |

| clickhouse-reference-round-trip.test.ts | parse(format(parse(...))) | (none — internal stability) | see note below |

The diff:* scripts reproduce *exactly* what each suite compares, then print the

actual output, the expected output, and a colored unified diff (expected → actual).

Workflow

1. Identify the failing dimension and case(s)

Run the suite (or read the failure output). Vitest names each failing case by its

.sql filename under a describe block — e.g. `clickhouse reference - format >

00123_foo.sql. The describe` label tells you the dimension; the test name is the

case file.

To re-run a single dimension and filter to one case:

npx vitest run tests/clickhouse-reference-format.test.ts -t 00123_foo.sql

2. Diff that case

Pick the diff:* command for the failing dimension and pass the case (the .sql

suffix is optional). Start with --diff-only for a compact view:

npm run diff:format -- 00123_foo --diff-only

Drop --diff-only to also see the full actual and expected blocks (useful when the

diff hunks lack enough context). Other flags: --only-diffs (skip matching cases),

--actual-only / --expected-only, --no-color. -h prints usage.

The <reference> argument also accepts a comma-separated list or a glob, so you can

diff a batch of failures at once and let --only-diffs filter to the broken ones:

npm run diff:ast -- '0012*,00200_bar' --only-diffs --diff-only

The script exits non-zero if any case differs, so it doubles as a quick check.

3. Fix the cause — in the code, never the fixtures

tests/clickhouse-reference/ and the test files are fixtures: do not edit them

to make a test pass. The expected outputs are the source of truth. Fix the

production code so its output matches:

• diff:ast failure → wrong/missing parse result. Fix src/grammar.pegjs

(PEG rules, not JS), then npm run generate:parser. The AST must hold structured

data, not raw syntax strings (see CLAUDE.md).

• diff:format failure → src/format.ts. The formatter must only render AST

fields — no re-parsing. If it lacks data, the fix may belong in the grammar/AST.

• diff:explain failure → src/explain.ts.

After a grammar change, always npm run generate:parser before re-diffing — the

diff:* scripts import src/ directly via tsx and use the generated

src/parser.js.

4. Re-verify

Re-run the diff command (expect exit 0 / "no differences"), then the suite:

npm run diff:format -- 00123_foo --diff-only
npx vitest run tests/clickhouse-reference-format.test.ts -t 00123_foo.sql

Confirm you didn't regress others by running the full dimension suite, since a

grammar/formatter change can affect many cases.

Notes

• Explain placeholders: expected explain files may contain <Query Parameters>

or <Explain Error> entries (statements ClickHouse couldn't explain). The suite

and diff:explain skip these, so they never show as diffs.

• Round-trip failures have no expected file — they assert format() output

re-parses to the same AST. Diff its two parse() results manually: get the

formatted SQL with npm run format -- <case>, then re-parse it with

npm run parse -- --sql "<formatted SQL>" and compare against npm run parse -- <case>.

(parse/format/explain take a reference case as a positional arg, or raw SQL

via --sql.)

• A <Error> ... line in the actual block means parse()/format()/

formatExplain() threw for that case — a parser/formatter crash, not a mismatch.