Text-to-SQL now has three jobs: answer, clarify and abstain

One score cannot describe three decisions

Execution accuracy asks whether generated SQL returns the expected result. Three October evaluations show why that is no longer enough: a production system must first decide whether to answer, ask for clarification or abstain.

PremiseSQL isolates the abstention problem. Its 1,800 flawed Spider/BIRD-derived requests are paired with 1,800 unmodified originals, and 16 models process both under base, implicit and explicit prompting. Under ordinary SQL-generation prompts, aggregate flaw detection was only 0.319. Explicitly telling models that a flaw exists raised detection to 0.931—but valid-query SQL generation fell to 0.523. The intervention improved apparent safety partly by teaching models to reject requests, including valid ones.

That makes “refusal rate” a poor reliability metric by itself. PremiseSQL’s strongest warning is the paired control: every flaw has an original that should still receive SQL. Qwen3.5-397B reached 0.968 detection under explicit prompting but generated SQL for only 0.004 of valid originals. Gemini 3.1 Pro Preview retained 0.941 generation on those originals. Those systems can look similarly cautious on flawed inputs while behaving very differently when the input is sound.

Clarification requires more than uncertainty

CircleClar addresses the middle option. Its premise is that ambiguity is model-dependent: an input can admit several real interpretations, but a particular model may know how to answer only some of them. On the executable AMBROSIA text-to-SQL evaluation, CircleClar produced the best answer-set F1 for two of four tested models—not three, as our initial wire item incorrectly stated. For GPT-5.5, its gain over one-shot enumeration was only 0.61 points; other methods led for Qwen3-4B and GPT-5.5.

That result argues against adding a universal “ask a question when uncertain” wrapper. A clarification policy should be evaluated with the model it controls, because the set of interpretations the model can actually resolve changes with the model.

Diversity is useful only if the system can route it

EliteReplay tests a third component: preserving multiple correct SQL strategies during reinforcement-learning post-training. Its headline BIRD-dev result is 74.8% pass@128, not single-shot accuracy, while retaining 62% of the base model’s correct strategies. On ambiguous questions, the paper reports covering both interpretations nearly twice as often as its strongest baseline.

That diversity creates options, but not a complete product policy. A system still needs evidence to decide whether alternatives are equivalent, represent genuine ambiguity or reveal a broken premise.

Together, the three evaluations suggest a better reliability report. Publish SQL correctness when the system answers; valid-query coverage when it abstains; interpretation coverage and clarification quality when ambiguity exists. Keep the denominators separate. A model that generates less SQL can appear safer, and a model that samples 128 candidates can appear more capable, without either improvement translating into a better one-shot user decision.

The benchmark frontier is therefore shifting from “Can the model write the query?” to “Does it choose the right action before writing one?” That is a harder target—and a more faithful one for deployed analytics agents.