FROST’s Spider gain is a data-budget result—not a new leaderboard score

A smaller synthetic pool beats the full one

A new Meta-led preprint reports that FROST, an online synthetic-data filter, reached 65.7% execution accuracy on the Spider 1.0 development split while discarding 20.4% of its synthetic training examples. The relevant comparison is not the current BIRD or Spider 2.0 leaderboard: this is a controlled fine-tuning experiment on the older Spider 1.0 dev set, using Qwen2.5-Coder-0.5B-Instruct as the base model. The paper reports one epoch of full fine-tuning on 8,659 human-annotated Spider pairs plus a 34,636-example subset of SynSQL.

The full, unfiltered mixture scored 64.8% EX, so FROST’s result is a 0.9 percentage-point gain with about one fifth less synthetic supervision. The real-only run scored 57.1%, showing that most of the measured lift still came from adding synthetic data at all; FROST improved how that pool was used rather than replacing it. At the same 20.4% drop ratio, random selection scored 64.1%, GradNorm 64.2%, and GREATS 65.3%. Those matched-budget results make the filtering comparison more informative than an unmatched “less data, better score” headline.

The aggregate gain hides a trade-off

The difficulty breakdown is not uniformly better. Against all synthetic data, FROST moved Easy from 83.5% to 83.5%, Medium from 67.0% to 68.6%, Hard from 56.9% down to 54.6%, and Extra Hard from 39.2% up to 42.8%. In other words, the overall gain depends partly on a 3.6-point improvement in Extra Hard while Hard loses 2.3 points. The same table therefore supports a narrower claim: online filtering improved aggregate dev EX under this training recipe, not every difficulty bucket.

FROST scores synthetic examples by aligning their gradients with an exponential-moving-average reference built from real training batches. It keeps moderate-utility batches intact and applies sample-level interquartile-range filtering only to batches outside that band. An ablation reinforces that both stages matter: batch-only filtering dropped 80.0% of synthetic data and scored 60.7% EX, while sample-only filtering dropped 32.2% and scored 63.1%; full FROST retained the 65.7% result with a smaller 20.4% drop. The ablation is reported on the same Spider 1.0 dev split.

What the result does not establish

This is a preprint result on one development split, one 0.5B-parameter model, and one synthetic pool. The paper does not report a Spider test-set score or place FROST on the active BIRD or Spider 2.0 boards; its table also does not provide confidence intervals for the 0.9-point text-to-SQL gain. The useful finding is consequently about training-data efficiency: under a fixed recipe, gradient-guided filtering beat both the complete synthetic pool and three matched-budget selectors. The paper’s own setup and limits should travel with the number.