BEAVER gives SQL agents the answers to five subtasks—and they still miss three quarters of queries

BEAVER’s most revealing result is not that enterprise text-to-SQL is hard. It is that the benchmark can hand systems the correct intermediate facts and still leave most queries unsolved. In the paper’s evaluation, ReFoRCE averages 9.5% execution accuracy across seven models in the end-to-end setting. Giving it oracle annotations for table retrieval, join keys and column mappings raises that average to 18.9%; supplying annotations for all five subtasks raises it to 25.9%. The May 13, 2026 v3 paper therefore separates failures in finding context from failures in assembling correct analytical SQL. (BEAVER paper)

The strongest run is still 11.4%

On the paper’s model-by-model table, ReFoRCE with Claude 4.5 Sonnet scores 11.4% execution accuracy, the best reported end-to-end combination. ReFoRCE with GPT-5.2 scores 10.8%. These are results on BEAVER’s evaluation reported without a train/dev/test label in the paper’s result table; they should not be compared as though they were BIRD-dev or a Spider 2.0 leaderboard score. The project site separately says BEAVER contains 9,128 queries, of which 7,978 are public and the remaining 1,150 are held out as a private test set. (BEAVER paper; BEAVER project)

That caveat matters because BEAVER is structurally unlike the public benchmarks it is meant to challenge. The paper reports three private enterprise warehouses spanning 812 tables and 19 domains, with an average of 101.5 tables and 869.4 columns per database. It contrasts that with BIRD’s reported averages of 6.8 tables and 72.5 columns. The dataset combines 254 queries collected from one month of real logs and reports with 8,874 expert-verified synthetic queries generated from structural templates extracted from those workloads. (BEAVER paper)

Oracle context does not solve analytical SQL

The all-hints setting improves ReFoRCE’s table-retrieval F1 to 91.9, join-key F1 to 75.7, and column-mapping F1 to 83.4, yet execution accuracy remains 25.9%. The paper’s audit of 275 incorrect queries after hints attributes 46.8% of residual errors to analytical and structural mistakes, including 22.8% for grouping and ordering and 14.2% for advanced functions. Schema-linking errors still account for 35.2%, while predicate errors account for 18.0%. (BEAVER paper)

The practical lesson is narrower than “enterprise data is messy.” Better retrieval is necessary, but it is not sufficient. BEAVER’s controlled oracle settings show that even when systems are handed much of the correct schema and decomposition context, they still fail on SQL semantics such as aggregation grain, window definitions, join types and predicates. That is the layer a production evaluation must test after retrieval looks healthy. (BEAVER paper)