MRTPO improves six-test SQL accuracy by assigning each reward to the tokens that earned it

A text-to-SQL training run can earn several rewards at once: one for formatting, one for producing executable SQL and one for matching the reference result. The new MRTPO study argues that the usual mistake is not how those rewards are combined, but where their credit lands. Instead of spreading every reward across every output token, MRTPO replays the response with and without each verifier’s information and assigns more of that reward to the tokens whose next-token distribution changes most.

The six-test result

The authors trained Qwen2.5-Coder-3B-Instruct and 7B-Instruct for 100 steps on 6,915 Spider training prompts, using 32 prompts and eight rollouts per step. They evaluated greedy execution accuracy on Spider dev, test, Syn, Realistic and DK, plus BIRD dev. Three independent seeds were reported for the SQL experiments.

With the 7B model, MRTPO averaged 73.15% across those six evaluations, compared with 71.55% for GDPO—a 1.60-point gain. Its individual scores were 81.33% on Spider dev, 81.65% on Spider test, 73.53% on Spider-Syn, 75.92% on Spider-Realistic, 70.04% on Spider-DK and 56.42% on BIRD dev. Adding SAW’s response-level reward reweighting lifted the six-test average to 73.41% and BIRD dev to 57.53%.

The smaller 3B model showed the same direction: MRTPO averaged 68.60%, versus 67.32% for GDPO. The paper therefore reports gains of 1.3 and 1.6 points for the two model sizes, without conflating BIRD dev with any held-out BIRD test score.

Why the result matters

The ablation is stronger than the headline average. On the 7B model, shuffling MRTPO’s learned token weights within each response reduced the six-test average to 71.03%, below GDPO’s 71.55%. A shared “joint replay” profile reached 72.56%, still below reward-specific MRTPO at 73.15%. That supports the authors’ narrower claim: the location of credit, and the separation of one reward from another, contributed to the improvement.

There is also an efficiency result, but not a free one. MRTPO passed GDPO’s final six-test accuracy after 60 of 100 training steps, and the 3B model’s responses became 40% shorter. However, verifier-conditioned replay made each training step 10–20% slower. The released repository specifies eight H20 GPUs per SQL run, which limits what “faster convergence” means in wall-clock or compute terms.

The scope is equally important. The study covers 3B and 7B Qwen coder models, programmatic rewards and a fixed allocation strength. It does not establish the result for frontier models, LLM-judged rewards or private enterprise schemas. What it does provide is a reproducible test of a precise training choice: when SQL correctness depends on a few table, column or operator tokens, rewarding the whole response uniformly leaves measurable accuracy on the table.