EliteReplay keeps more correct SQL strategies alive—but its 74.8% is a 128-sample result

A new post-training study argues that text-to-SQL accuracy can improve while the model quietly loses alternative ways to solve the same question. EliteReplay addresses that failure mode by combining Group Relative Policy Optimization with replay from an archive of verified, structurally distinct SQL solutions.

The headline result needs careful labeling. On BIRD dev, EliteReplay reaches 74.8% execution pass@128: the chance that at least one of 128 sampled queries executes to the gold result. Its single-sample pass@1 is 52.5%. Those are not interchangeable with the ordinary one-query execution-accuracy figures shown on the public BIRD leaderboard. The same paper reports 93.2% pass@128 and 78.6% pass@1 on Spider 1.0 dev, used as an out-of-distribution test after training on 5,000 BIRD questions.

The result is about reach, not just accuracy

Standard RL post-training improved BIRD pass@1 from the Qwen3.5-4B-Base model’s 25.3% into a 52–56% range, but several methods fell below the base model at larger sampling budgets. GRPO, SetPO and DPH-F reached only 68–69% pass@128, versus 73.6% for the untrained base. EliteReplay reached 74.8%, narrowly ahead of MaxRL’s 73.7%.

The larger difference appeared in strategy diversity. At 128 samples per question, EliteReplay produced 9.1 distinct correct SQL strategies on BIRD, compared with 6.2 for MaxRL. On Spider, the corresponding figures were 9.4 and 5.3. EliteReplay retained 62% of correct base-model strategies on BIRD and 60% on Spider; the compared baselines retained 34–39% and 30–37%, respectively.

That distinction matters when a request has more than one valid reading. On AmbiQT, which gives each question two valid interpretations, EliteReplay’s 128 samples covered both readings for 27.0% of questions. The strongest alternative post-training baseline reached 14.2%, while the base model reached 37.0%.

Strong reproducibility trail, narrow scope

The authors released code, model weights, a BIRD archive and evaluation tooling. The repository says the archive contains 2,708 occupied cells across 780 questions, with 1,500 replay examples, and documents BIRD and Spider generation plus execution checking.

The caveat is scope. The study uses one 4B base model, trains on BIRD, evaluates SQL diversity through AST-derived structures, and draws 128 samples per question. That makes it evidence that replay can preserve solution reach during RL—not evidence that EliteReplay is a new one-shot BIRD leaderboard leader. The paper’s most useful contribution may be the measurement lesson: pass@k can look healthy even when a model has collapsed onto fewer correct strategies.