I Round-Tripped 2,249 Test Fixtures Through sqlfluff's Auto-Fixer. Eight Came Back Unparsable.

The Dangers of Auto-Fix in Linters

Auto-fix is the most dangerous feature a linter has. Every other part of the tool only reads your code. --fix writes to it, usually across a whole repository, usually in CI, usually without anyone reading the diff. So it is worth asking a blunt question: does the fixer ever produce output the tool itself cannot read back?

A Cheap Way to Find Out

There is a cheap way to find out, and the input you need is already sitting in the repository. The invariant is simple: If a fixture parses cleanly, its fix output must parse cleanly too. That is the whole idea. One line. No fuzzer, no grammar model, no corpus to build.

Using Test Fixtures as Adversarial Input

It works because a linter's test fixtures are the best adversarial input anyone will ever hand you for free. They were written by the maintainers, deliberately, to sit on the edges of the grammar: every operator, every dialect quirk, every syntax someone once filed a bug about. Nobody could commission a better corpus, and it ships with the source.

Running the Fixer on a Large Corpus

The run sqlfluff, at a54076a. 2,249 fixtures, 28 dialects, roughly an hour single-process. For each fixture: parse it. If it parses, fix it, then parse the output. Classify the result:

• CORRUPTION - parsed before, does not parse after.
• UNSTABLE - fixing twice does not converge.
• FIX_CRASH - the fixer threw.

Raw result: 8 CORRUPTION, 31 UNSTABLE. Thirty-nine hits. Thirty-seven of them were not bugs.

Triage and the Importance of Discipline

Triage is the entire job. Finding 39 things is easy. Knowing which 2 are real is the skill, and getting it wrong in public is how you teach a maintainer to ignore you. UNSTABLE is almost always noise. All 31, in this run. sqlfluff's CLI loops the fixer until output stabilises, so a single pass that has not yet converged is expected behaviour, not a defect.

Identifying Real Issues

The eight collapse into two root causes. Not eight bugs. Two:

Bug One: Whitespace Fix Deletes Code

Bug one: a whitespace fix that deletes your code. sqlfluff removes whitespace it considers redundant. Sometimes the characters on either side of that whitespace are tokens that fuse into a different token when they touch.

Bug Two: Fixer Strips Load-Bearing Quotes

Bug two: the fixer strips quotes that were load-bearing. RF06 removes unnecessary quotes from identifiers. In MySQL and MariaDB, 'jeffrey'@'localhost' is not two quoted identifiers. It is one account specification, and the quotes are part of it.

The Importance of Validating Output

A clean re-parse does not mean the output is safe. 1 * --5 parses perfectly. It just means something else now. So I added a second check, for output that parses but has lost meaning.

The Check that Found Nothing

The check that found nothing is a check for output that introduces a comment. Nothing sqlfluff legitimately does creates a comment, so a new one after fixing means source text was swallowed.

Lessons Learned

Two things that cost me time:

• Extracted statements lie. Pull a statement out of a fixture to build a minimal repro and it may not parse standalone even though the whole file does.
• Run the real fixture, then minimise, then re-verify against the real fixture again. Redirect, do not pipe.

Generalising the Methodology

Nothing above is specific to SQL. The requirements are only:

• The tool has an auto-fix mode.
• The tool can validate its own output.
• The project ships a fixture corpus.

That describes ruff, prettier, rustfmt, black, ESLint, gofmt, and most of the formatting layer our industry now runs unattended in CI.