COAL-SQL gets 64.9% on BIRD dev by training on failures, not just more examples

A smaller training set, chosen deliberately

COAL-SQL reports 64.9% execution accuracy on the BIRD development split after post-training Qwen2.5-Coder-7B-Instruct on 12,644 distinct examples. The paper says the same backbone scores 50.9% before this post-training, a 14.0-point gain. That is an author-reported development result, not the BIRD hidden-test score, and it should not be compared directly with test results on the live leaderboard. The paper states the split, backbone and data count explicitly.

The interesting result is less the absolute rank than the training allocation. COAL-SQL first converts queries into schema-independent skeletons, preserving operators, clause structure, joins, nesting and set operations while masking database-specific names and values. It then uses metric K-center selection to add examples whose structures complement the seed set rather than sampling more SQL indiscriminately. The released repository includes the augmentation, skeleton-retrieval and training modules.

Train on the places reinforcement learning stalls

The second half of the method targets what the authors call Solve-None examples: prompts for which none of eight sampled rollouts executes correctly. COAL-SQL keeps GRPO as its main objective, but adds supervised reasoning traces for newly observed failures and retrieves structurally related practice from failures accumulated across an epoch. In the paper’s initial policy, 15.1% of training examples were Solve-None, concentrated around nested queries, multi-table joins and aggregation combinations. Those definitions and measurements come from the evaluation and method sections.

The ablations support a modest but consistent contribution from that targeting. GRPO on the augmented set reaches 63.2% BIRD dev execution accuracy; adding step-level correction raises it to 63.8%, and adding epoch-level expansion reaches 64.9%. Replacing failure-aware epoch retrieval with random practice lowers BIRD dev to 64.3%. Replacing coverage-guided K-center selection with random augmentation lowers it to 64.4%. The paper reports all of these on the BIRD development split.

What the number does—and does not—show

Within the paper’s comparison, COAL-SQL exceeds OmniSQL’s 63.9% while using about 12.6K rather than 2.5 million training examples. It also reports 36.2% execution accuracy on EHRSQL and 53.9% on Science Benchmark, versus 24.3% and 45.2% for the untrained backbone. The authors audit all 12,644 training questions against 1,534 BIRD dev questions and report no exact duplicate, no shared 13-gram and no SQL pair with token containment of 0.5 or more. These are the paper’s own results and contamination checks.

That still leaves independent reproduction as the next test. The repository publishes runnable code and configuration scaffolding, but does not bundle the benchmark datasets; users must supply BIRD and the external SQL collection themselves. The repository documents those prerequisites. For now, 64.9% is evidence that carefully chosen structures and failure-driven supervision can improve data efficiency on BIRD dev—not a new BIRD test leader.