ReToolSQL’s hidden BIRD test score is stronger than its development result
BIRD has now added the JPMorganChase system at 78.14% test execution accuracy—second in the single-trained-model track and 3.37 points above its dev score.
The test-set result is now official
BIRD merged ReToolSQL into its Single Trained Model Track on September 30, 2026. The 31B JPMorganChase system posts 78.14% execution accuracy on the hidden test split, placing it second in that track behind Gemini-SQL2 at 80.04%. The same row reports 74.77% on development, so ReToolSQL’s test score is 3.37 percentage points higher than its dev result.
That split matters because the ReToolSQL paper originally centered its headline on BIRD development. It reported 74.32% dev execution accuracy for a single pass and 74.77% with self-consistency, describing the latter as first on the single-model development leaderboard at the time. The merged BIRD row now adds the result that was missing from that account: performance on the benchmark’s withheld test set.
Difficulty still bites
The pull request includes the evaluator output for all 1,789 test questions. ReToolSQL reaches 84.30% EX on 949 simple questions, 76.04% on 555 moderate questions, and 61.75% on 285 challenging questions. That is a 22.55-point fall from simple to challenging, even though the aggregate test result is stronger than the published dev result.
The same official evaluation reports 79.06% Soft F1 and 72.54% R-VES overall. Those are separate metrics and should not be substituted for execution accuracy: EX checks whether the generated query returns the correct result, while R-VES adds an efficiency-sensitive view. On challenging questions, ReToolSQL’s R-VES falls to 56.87%, below its 61.75% EX.
What changed—and what did not
This is a genuine leaderboard movement, but not a new overall BIRD record. GrainSQL remains the main test leaderboard leader at 82.95% EX, while the single-trained-model track isolates systems trained as one model rather than broader pipelines. Within that narrower track, the current page orders ReToolSQL behind Gemini-SQL2 and ahead of Gemini-SQL at 77.14% test EX.
The useful signal is not simply that another row appeared. ReToolSQL’s paper argued that supervised fine-tuning expands the set of solvable questions while reinforcement fine-tuning teaches a single dense model to use read-only database tools for proposing, checking and repairing SQL. BIRD’s hidden-test evaluation does not independently prove that causal story, but it does show that the reported development gain survived the move to the withheld split—and improved in aggregate rather than collapsing.
sources
comments · 0