FROST’s Spider result says synthetic training data should earn its place

Less synthetic data, slightly better execution accuracy

A new Meta-led paper reports that FROST, an online synthetic-data filtering method, reached 65.7% execution accuracy on the Spider 1.0 development split while dropping 20.4% of the available synthetic training examples. The same Qwen2.5-Coder-0.5B-Instruct model trained with the entire synthetic pool reached 64.8%, so the filtered run gained 0.9 percentage points despite seeing less synthetic data.

The experimental setup matters. The authors fully fine-tuned a 0.5B-parameter model for one epoch using 8,659 human-annotated Spider 1.0 training pairs plus a sampled pool of 34,636 SynSQL examples—four synthetic examples for every real one. Every selection method retained all real data. FROST, random downsampling, GradNorm and GREATS were compared at the same 20.4% synthetic-data drop ratio.

The matched-budget comparison favors FROST, but only narrowly. GREATS reached 65.3% overall execution accuracy, versus 64.2% for GradNorm and 64.1% for random selection. Training on real data alone reached 57.1%. That makes the central result more specific than “synthetic data helps”: the full pool already delivered a 7.7-point gain over real-only training, while utility-based filtering recovered another 0.9 points.

The difficult questions tell a less tidy story

FROST’s overall lead did not extend uniformly across Spider’s difficulty buckets. It scored 83.5% on Easy, 68.6% on Medium, 54.6% on Hard and 42.8% on Extra Hard. GREATS was lower overall, but higher on Hard at 57.5%; FROST’s advantage came chiefly from Medium and Extra Hard, where GREATS scored 68.4% and 40.4%, respectively.

That breakdown is a useful warning against turning a 0.9-point aggregate gain into a broad claim about SQL reasoning. FROST estimates each synthetic example’s utility by aligning its gradient with a smoothed gradient reference from real training batches. The method therefore selects examples predicted to improve the current training objective; it does not directly optimize every benchmark difficulty stratum.

What the result does—and does not—establish

This is a Spider 1.0 dev-split fine-tuning experiment, not an official hidden-test leaderboard submission and not evidence of state-of-the-art text-to-SQL performance. Its contribution is about training-data efficiency: a learner-aware filter beat using every synthetic example and three matched-budget alternatives in one controlled setup.

The next test should be transfer. The paper evaluates one small model, one synthetic source and one public SQL benchmark. Repeating the comparison across model sizes, synthetic generators and harder contamination-resistant evaluations would show whether the 0.9-point gain reflects a durable selection principle or a useful but local fit to Spider’s development distribution.