A new GRPO text-to-SQL adapter improves correctness while making more queries fail to execute

A newly released Qwen2.5-3B adapter offers a useful, compact example of why text-to-SQL teams should read the whole evaluation table rather than its best number.

The Hugging Face model card reports 62.67% execution accuracy on all 1,034 examples in the Spider development set. That is 1.84 percentage points above the untuned Qwen2.5-3B-Instruct base at 60.83%, and 0.97 points above the supervised fine-tuned checkpoint at 61.70%. The adapter is MIT-licensed and the training and evaluation code are public.

But another column moves in the opposite direction. The base model’s SQL executed successfully 88.78% of the time, compared with 82.40% for the GRPO adapter. The base also retained the higher mean reward, 0.770 versus 0.761. Those figures are not contradictions: “execution success” asks whether a query runs, while execution accuracy asks whether its result matches the reference. A system can become less willing to emit harmless-but-wrong SQL while still producing more statements that fail outright.

What was actually trained

The published results file says the recipe starts with one supervised epoch over 8,659 Spider training rows, then runs group-relative policy optimization over 3,000 prompts with four generations each. Its reward assigns 20% to validity, 30% to successful execution and 50% to correctness.

The evaluation script uses deterministic decoding, sets seed 42 and scores generated SQL against each example’s SQLite database and gold query. That makes the comparison inspectable, but it is still a single project-authored run on Spider—not an independent reproduction, a production-schema test or evidence about access controls and unsafe statements. The model card explicitly labels the adapter research/demo software and tells users to validate SQL before running it against real data.

The deployment lesson

For a pilot, keep both published metrics and add a third bucket: requests where the system should refuse or ask for clarification. Then record at least four outcomes separately—correct result, executable but wrong result, execution failure, and safe refusal. A single blended reward can hide movement between those failure modes.

The release is modest rather than state of the art, and that is what makes it instructive. Its headline metric improves, yet its operability metric remains below the base. Teams evaluating post-training for enterprise NL2SQL should require gains that survive their own schemas, SQL dialects, permissions and ambiguous questions—not just a higher aggregate score on a legacy benchmark.