SIRIUS-SQL’s 75.88% BIRD score comes from 32 candidates—not one answer
The paper’s ablations show candidate diversity contributes more than execution repair or reranking, while leaving latency and dollar cost unreported.
SIRIUS-SQL reports 75.88% execution accuracy on the 1,534-question BIRD development split, but the headline number describes a search-and-selection system, not a single model response. For each question, the strongest configuration generates as many as 32 SQL candidates—16 from an RL-trained SIRIUS-32B specialist and 16 from Gemini 3.1 Pro Preview—then executes, repairs and reranks them. The same configuration reports 91.20% on the Spider 1.0 test split; those are different benchmarks and should not be compared as one leaderboard series. The paper reports both results in Table 1.
The ablations make the source of the gain unusually visible. Under a matched 32-candidate budget on BIRD dev, a Qwen3-Coder-only pool reaches 72.08%, a SIRIUS-32B-only pool reaches 73.27%, and the hybrid SIRIUS-32B plus Qwen3-Coder pool reaches 74.51%. That assigns 2.43 percentage points over the generalist-only pipeline—and 1.24 points over the specialist-only pipeline—to complementary candidate generation, with the lifecycle and selector held constant. Table 2 reports the matched-budget comparison.
Execution repair helps, but less. With exact-result majority voting fixed as the selector, removing the lifecycle lowers BIRD-dev accuracy from 75.03% to 74.32%, a 0.71-point change. Of 49,088 candidates in the SIRIUS-32B plus Gemini pool, 97.4% were already clean; repair recovered 46% of empty-result candidates, 89% of timeouts and all 162 validation failures. Table 3 and Figure 5 contain the lifecycle ablation and repair counts.
Selection contributes another 0.85 points over majority voting: the paper’s execution-consensus and pairwise-judgment merge reaches 75.68%, and a deterministic structural gate lifts that to 75.88%. The gate itself is worth 0.20 points. Table 4 isolates the selector changes.
The practical caveat is cost visibility. The paper specifies 32 candidates per question, a 120-second execution timeout per lifecycle query, a 30-second timeout during selection, hosted Gemini calls and substantial specialist-training hardware, but reports no end-to-end latency, token count or dollar cost. Its implementation details and limitations describe those budgets. The result is a credible accuracy gain, but not yet a deployability comparison: buyers cannot tell whether the extra 0.98 points over the paper’s 74.90% prior BIRD baseline justify the additional generation and execution work.
comments · 0