DSABench shows why finding the right databases is only half the text-to-SQL problem

DSABench changes a common text-to-SQL assumption: the system is not told which database contains the answer. Instead, an agent must discover every required source, infer how those sources depend on one another, generate source-local work and complete the combined workflow. On that end-to-end test, the strongest reported clarified configuration succeeds on just 16.99% of the benchmark’s full frozen set.

A catalog problem before a SQL problem

The benchmark contains 815 human-reviewed tasks drawn from Spider, BIRD and LiveSQLBench: 537, 254 and 24 tasks, respectively. The authors report that 618 tasks require multiple databases to contribute, while 105 tasks admit two possible interpretations of the needed sources. Agents can inspect public catalog evidence, but the evaluator withholds the gold source set, dependency graph and source-local SQL.

That boundary matters because plausible retrieval is not the same as complete retrieval. The paper reports 93.83% average recall of annotated target-list entries in the candidate top 50, but the same candidate stage contains the entire target list for only 86.50% of tasks. After reranking to ten candidates, complete coverage falls to 57.18%.

Coordination cuts the score nearly in half

Across seven model backends, the best routing-only exact-source-set result is 60.55%, achieved by GPT-5.6 Sol with the paper’s Hermes router. The best score falls to 34.66% when the metric also requires the correct workflow topology and applicable dependency edges; that result comes from Claude Opus 5 with Hermes.

The distinction is essential. A system can name the right databases but still misunderstand whether work should happen in parallel, sequentially or through exploration. In the paper’s fixed non-oracle setting, only 55.1%–58.4% of source-set-correct task weight also passes this graph-aware test. Of the graph-correct weight, only 32.3%–37.0% completes execution correctly.

Asking once helps—but does not close the workflow gap

The clearest intervention is clarification. On the 105 ambiguous tasks, allowing one question raises exact branch-source recovery by 39.8 to 43.2 percentage points. Interactive end-to-end success rises from 0% under “Never Ask” to 35.91%–38.02% under “Ask Once,” although the authors caution that the zero baseline partly reflects a strict protocol requiring the interactive topology label.

Across all 815 tasks and non-privileged inputs, median Route+Exec-Graph success is only 6.69%–7.98% without clarification and 11.44%–13.25% with one clarification. The best clarified configuration reaches 16.99%.

The result should not be read as a clean model ranking. The authors say the routing policies are not compute-matched, the best-of-grid figure is selected on evaluation data, and execution comparators differ across the three source corpora. DSABench is also a single frozen evaluation set rather than a conventional train/dev/test leaderboard.

Its useful finding is narrower and stronger: retrieving likely databases, writing locally correct SQL and completing the right multi-database workflow are separate failure surfaces. A clarification step can repair much of the ambiguity at the first surface, while most of the end-to-end problem remains.