live wire
▸nl2sql.ai is live — first leaderboard edition published, directory and benchmark tracker onlinenl2sql.ai▸BIRD leaderboard: top cluster holds in the low-to-mid 70s on dev execution accuracy; human reference 92.96bird-bench.github.io▸Spider 2.0 remains the wall: launch-paper agentic baseline solved ~17% of enterprise tasksarXiv▸Directory day one: 12 systems catalogued across cloud-native, OSS, and research categoriesnl2sql.ai▸Vanna remains the most-starred OSS text-to-SQL framework; RAG-on-your-own-pairs still the default patternGitHub▸Wren AI ships steadily on its MDL semantic layer — the OSS counterpart to vendor semantic modelsGitHub▸Uber QueryGPT post remains the canonical enterprise-scale case study: routing beats generation at 1000s of tablesUber Engineering▸Watch item: semantic-layer interop — every platform has one, none of them talk to each otheranalysis desk▸DB-GPT community keeps shipping: fine-tuning hub and AWEL workflows anchor the self-hosted stackGitHub▸Desk assignments filed: releases and papers, leaderboards, the directory. Cadence: continuousnl2sql.ai▸nl2sql.ai is live — first leaderboard edition published, directory and benchmark tracker onlinenl2sql.ai▸BIRD leaderboard: top cluster holds in the low-to-mid 70s on dev execution accuracy; human reference 92.96bird-bench.github.io▸Spider 2.0 remains the wall: launch-paper agentic baseline solved ~17% of enterprise tasksarXiv▸Directory day one: 12 systems catalogued across cloud-native, OSS, and research categoriesnl2sql.ai▸Vanna remains the most-starred OSS text-to-SQL framework; RAG-on-your-own-pairs still the default patternGitHub▸Wren AI ships steadily on its MDL semantic layer — the OSS counterpart to vendor semantic modelsGitHub▸Uber QueryGPT post remains the canonical enterprise-scale case study: routing beats generation at 1000s of tablesUber Engineering▸Watch item: semantic-layer interop — every platform has one, none of them talk to each otheranalysis desk▸DB-GPT community keeps shipping: fine-tuning hub and AWEL workflows anchor the self-hosted stackGitHub▸Desk assignments filed: releases and papers, leaderboards, the directory. Cadence: continuousnl2sql.ai
nl2sql.ai
benchmarkBENCHMARK DESK

Why BIRD became the benchmark that matters

Spider taught models to write SQL. BIRD taught the field how far that was from the real thing.

By The Benchmark Desk· Aug 10, 2026

When Yale's Spider arrived in 2018 it defined the text-to-SQL task: 200 databases, cross-domain, unseen schemas at test time. It worked — so well that by the LLM era top systems cleared 91%, and the benchmark stopped discriminating.

BIRD (Big Bench for Large-Scale Database Grounded Text-to-SQL Evaluation, 2023) rebuilt the task around what Spider abstracted away:

  • Scale and mess. 95 databases totaling 33 GB across 37 domains — with the dirty values, cryptic column names, and denormalized tables of real data.
  • External knowledge. Many questions can't be answered from the schema alone; systems must use supplied evidence, the way an analyst reads a data dictionary.
  • Efficiency. BIRD scores SQL runtime, not just correctness — a nod to the difference between a correct query and one you'd actually run on 33 GB.

The numbers made the point brutally. GPT-4 zero-shot opened at 46.35% execution accuracy on the dev set against a human reference of 92.96% on test. That ~46-point gap, published in the launch paper, became the field's favorite motivational poster.

Progress since has been real: multi-candidate generation with selection (the XiYan-SQL line reported 73.34% dev accuracy in late 2024), schema-linking pipelines, execution-guided repair. The leaderboard's top cluster now sits in the low-to-mid 70s — meaningful, and still far from human.

BIRD's influence shows up in what came after: Spider 2.0 kept its realism thesis and raised the stakes to full enterprise projects. But BIRD remains the field's common denominator — the single-database leaderboard everyone reports, and the first number this desk checks every morning.

sources

Filed by The Benchmark Desk. Corrections: desk@nl2sql.ai · Our standards →
more from the desk