ARCS shows SQL agents lose 22.8 points when they must work out what the user means

A text-to-SQL system can write valid SQL and still answer the wrong question. ARCS, a new benchmark for naturally ambiguous requests, measures that failure separately from query generation—and the separation is large.

The 22.8-point intent gap

On ARCS, gpt-6-sol with medium reasoning scored 74.60% execution accuracy when the intended interpretation was supplied, but only 51.77% end to end when it had to identify that intent itself. The paper reports the difference as 22.83 percentage points. Its best open model, gpt-oss-120b, reached 52.09% with the ambiguity resolved and 26.05% end to end, a 26.04-point drop.

Those are full-benchmark results; the paper does not define a development/test split. They should not be compared with BIRD or Spider leaderboard scores, even though ARCS draws five databases from BIRD and one from Spider 2.0. The six databases were converted to SQLite and span 8.3 GB across banking, retail, student activities, NBA, Stack Exchange and GitHub domains.

The benchmark makes ambiguity concrete. One published sample asks for “total revenue for each nation in 1995,” but exposes four separate choices: gross or net revenue, whether to include returned items, customer or supplier nation, and which of four date fields defines 1995. Producing syntactically correct SQL before resolving those choices is not success; it is an undocumented guess.

A form beat a conversation on effort

ARCS compares free-form clarification with a structured interface that presents constrained interpretations. Using o4-mini at medium reasoning, unconstrained conversation reached 43.73% execution accuracy with user effort of 34. Structured disambiguation reached 42.44%—1.29 points lower—but cut the paper’s user-effort measure to 18, a 47% reduction. That measure weights simulator input tokens at 0.1 and output tokens at 1, so it is a benchmark proxy rather than a human usability study.

The benchmark is inspectable rather than paper-only: the project page links code, a dataset, an interactive demo and a leaderboard through the TabulaFlow repository. That matters because the construction still needs scrutiny. The paper’s contribution summary says ARCS contains 311 tasks, while its evaluation section describes 331 task instances sampled from 101 unique questions and 413 SQL queries. Main experiments use an LLM-based user simulator, and costs exclude that simulator.

ARCS’s useful result is therefore not a new universal text-to-SQL ranking. It is a diagnosis: on this released ambiguity suite, supplying intent can add 22 to 26 execution points. Benchmarking only the final query hides that gap—and production systems cannot assume users will state every business definition before asking.