HarnessSQL’s 32.6-point gain comes from training the agent inside its SQL tools

The result

HarnessSQL reports that Qwen3-14B rises from 22.2% to 54.8% execution accuracy on the 135-task Spider 2.0-SQLite split after full-trajectory supervised fine-tuning and execution-reward reinforcement learning. The 8B model moves from 15.5% to 45.2% under the same two-stage recipe. Those are gains of 32.6 and 29.7 percentage points, respectively.

The split matters. Spider 2.0-SQLite is the paper’s in-domain evaluation, assembled from Spider 2.0-Lite tasks and run through a dedicated SQLite harness. It is not Spider 2.0-Snow, and the numbers should not be compared with Snow leaderboard scores.

What the ablations actually support

The paper’s strongest evidence is not simply that reinforcement learning helped. Supervised fine-tuning alone raises the 14B model from 22.2% to 37.8%; reinforcement learning then adds 17.0 points to reach 54.8%. For the 8B model, direct reinforcement learning from the base checkpoint reaches only 20.0%, versus 45.2% when reinforcement learning follows trajectory supervision.

The harness itself also changes the result. On the paper’s 8B comparison, adding the interactive DSH-SQL environment raises the untrained base model from 2.2% without the harness to 15.5% with it. With inference fixed to DSH-SQL, harness-native supervised training reaches 30.4%, while non-harness training reaches 9.6%. That controlled comparison supports the paper’s central claim: tool use is part of the behavior being trained, not a neutral wrapper added at inference time.

HarnessSQL limits the agent to four operations—listing tables, inspecting schemas, executing bounded read-only queries and submitting SQL—and trains on 2,512 verified multi-turn trajectories drawn from 79 databases. Its context ablation is also consequential: the 8B system rises from 27.4% at 8K context to 45.2% at 32K, suggesting that truncating prior tool feedback damages long-horizon SQL work.

The boundary

Transfer results are positive but far below the in-domain score. The 14B model reaches 11.3% on BIRD-Interact Mini and 28.5% on LiveSQLBench Base-Lite SQLite; the 8B model reaches 9.0% and 24.8%. The authors also identify dialect coverage as the main limitation: the current study focuses primarily on SQLite rather than PostgreSQL, Snowflake or BigQuery.

The defensible conclusion is therefore narrower than “compact models solved Spider 2.0.” HarnessSQL shows that matching training trajectories to the exact execution interface can produce a large, controlled gain on a specific SQLite split. Whether that advantage survives a different database engine, tool contract or production observation format remains the important reproduction test.