GrainSQL takes BIRD’s overall lead—but its method is still mostly a name

A new overall leader

GrainSQL now sits first on BIRD’s overall leaderboard with 82.95 test execution accuracy (EX), according to a submission merged by the benchmark maintainers on September 30. The same submission reports 77.77 dev EX, 84.10 test R-VES, and 83.66 test Soft F1; it says the BIRD team returned the hidden-test results by email on September 26.

The row moves the test leader without producing a matching development lead. DataGallery-Text2SQL remains higher on the public development split at 78.10 dev EX, versus GrainSQL’s 77.77, but lower on the hidden test split at 82.39 test EX. GrainSQL’s test advantage is therefore 0.56 points, while its development score is 0.33 points lower. Those are separate splits, not interchangeable measurements.

What the submission reveals

The pull request identifies GrainSQL’s base model as Claude Opus 5 accessed by API, with no fine-tuning, and places the result in BIRD’s overall track. That matters because this is not a trained-model-only comparison: an overall-track system can combine a frontier model with prompting, retrieval, orchestration, voting, correction or other inference-time machinery.

What the public record does not yet provide is the machinery. GitHub repository search returns no public GrainSQL repository, and the merged submission links no paper, code, method card, prompt, cost report or latency measurement. The leaderboard score is benchmark-verified; the route to that score is not independently inspectable from the submitted materials.

That leaves the result useful for ranking but weak for replication. Readers can say that GrainSQL achieved 82.95 test EX on BIRD’s overall split. They cannot yet say which part of the system produced the gain, what it cost, whether its improvement survives reruns, or how much of the result depends on Claude Opus 5 rather than GrainSQL’s surrounding pipeline.

The practical reading

A 0.56-point hidden-test lead is enough to change first place, but not enough to justify a broad claim that one architecture has surpassed another—especially when one entrant publishes no method. The more defensible conclusion is narrower: BIRD’s highest verified overall test EX is now 82.95, while the leading row remains non-reproducible from public artifacts.

The next meaningful update is not another decimal point. It is a method card: exact model version, schema-linking and retrieval procedure, number of sampled candidates, selection or repair policy, prompts, compute budget, and per-run variance. Until then, GrainSQL is the leaderboard leader, not yet a reproducible recipe.