A new audit says BIRD’s gold labels are noisy enough to blur 21 leaderboard places

A text-to-SQL leaderboard can rank systems precisely while measuring them against an imprecise answer key. SALUS, a new SIGMOD 2027 benchmark-auditing system, reports that problem at a scale large enough to change how BIRD development scores should be interpreted.

What was measured

The authors built BIRD-Clean-xs from 298 manually verified BIRD development tasks: 166 labeled erroneous and 132 correct. Two PhD students reviewed each natural-language/SQL pair independently, reaching 86.24% initial agreement and a Cohen’s kappa of about 0.73; two database professors adjudicated disagreements. The set combines a random 200-task sample with 109 errors identified through version differences, with 11 tasks overlapping.

On that evaluation set, SALUS reaches 0.9194 F1 for annotation-error detection. A four-agent majority vote reaches 0.8895, the paper’s SAPAR verifier comparison reaches 0.8389, and SQLDriller reaches 0.6456. SALUS combines 44 weak-labeling functions across execution, SQL structure and question intent, then learns a query-conditioned decision plane from high-confidence pseudo-labels.

The manually reviewed random sample is the clearest directly measured prevalence result: 68 of 200 BIRD tasks were erroneous, or 34%. The larger benchmark-wide figures are calibrated estimates rather than exhaustive manual audits. SALUS estimates a 37% error rate for the November 2025 BIRD development set, with a reported 95% interval of 30%–43%, and 27% for Spider development, with a 21%–33% interval. The released repository includes recorded model outputs, the 298-task labeled set, predictions, pinned dependencies and commands that reproduce the pipeline without API keys or database downloads.

Why the leaderboard gap matters

The paper’s uncertainty analysis says that, under the estimated 37% BIRD annotation-error rate, the first-ranked development result cannot be reliably distinguished from the next 20 systems. It places the first statistically trusted separation at rank 22 and says score gaps of 5.09 percentage points or less are not distinguishable at 95% confidence.

That warning applies to development-set interpretation, not directly to BIRD’s hidden test ranking. The live BIRD leaderboard currently separates its top three hidden-test entries by only 0.67 points: GrainSQL at 82.95 execution accuracy, DataGallery at 82.39 and SiriusAI-SQL at 82.28. The audit does not prove those hidden-test labels have the same error rate, and the public site explicitly distinguishes development and test scores.

The practical change is methodological: leaderboard users should stop treating sub-point development gains as decisive unless the underlying items have been adjudicated. SALUS is also an auditor, not a repair system; it flags likely errors but does not produce corrected benchmark labels. Its strongest contribution is therefore the released evidence trail: a human-checked mini-benchmark, per-task predictions and a runnable audit pipeline that benchmark maintainers can inspect before accepting another narrow state-of-the-art claim.