On Qwen2.5-Coder-7B, reward-specific token credit reached 73.15% average execution accuracy—1.60 points above GDPO—at a 10–20% training-step cost.

The new RL recipe improves BIRD and Spider reach while resisting strategy collapse. Its strongest numbers measure repeated sampling, not leaderboard-style single-query accuracy.

Across 16 models, explicit flaw warnings lifted aggregate detection to 93.1% but cut SQL generation on paired valid questions to 52.3%; the released code is inspectable, though the paper’s data and outputs are separate.

The official leaderboard verifies the score; Genloop’s site explains its context product, not enough of the evaluated system to reproduce the result.

An open authentication issue shows how infrastructure access can become part of a benchmark’s effective difficulty—and exclude automated reruns.

One clarification question sharply improves source recovery on ambiguous requests, yet the benchmark’s best full-workflow result reaches only 16.99%.

Two late-September Base-Full v1 entries improve the leaderboard’s top half, while their cost and verification labels show why success rate alone is not enough.

ReFoRCE’s average execution accuracy rises from 9.5% to 25.9% with oracle annotations, isolating a failure layer beyond schema retrieval and decomposition.

The new runnable benchmark separates SQL generation from intent resolution—and finds that a correct query is often hiding behind the wrong interpretation.

The paper’s ablations show candidate diversity contributes more than execution repair or reranking, while leaving latency and dollar cost unreported.

The result is strong evidence for train-deploy alignment, but it is a 135-task SQLite evaluation—not the official Spider 2.0-Snow leaderboard.

The vendor-run CostBench result is unusually inspectable, but it excludes Databricks’ beta real-time warehouse and combines modeled cost with accumulated runtime.

SALUS reaches 0.9194 F1 on a manually verified audit set, then estimates 37% annotation error on BIRD dev and 27% on Spider dev.

A new semantics-aware synthesis pipeline reports sharply lower human-rated errors and better execution accuracy, while its most important reproducibility artifact remains scheduled for November.

Sentinel lifts GPT-4o’s low-stakes EHRSQL Reliability Score from +0.08 to +0.24, but its high-stakes operating point converges on abstention.

A controlled biomedical test lifted accuracy from 50% to 78.2%, but its most useful lesson is architectural: precomputed concept profiles beat a seven-tool graph traversal agent.

The result is a useful argument for testing data design before swapping models, with an equally useful warning about benchmark disclosure.

FLINT reached 68.8% and 65.2% on two proprietary production-query sets, while seven baselines stayed below 50% despite executing successfully at least 99% of the time.

AptMQL-Bench reports that foreign-key-driven migration failed outright on six BIRD databases and silently lost rows on eight more—before any model generated a query.

A UCI pipeline generated ten workload-specific protocols and improved nine through evolutionary search; its correctness test is empirical, not a proof over every interleaving.

A fresh reproduction found one deterministic false positive in an otherwise unusually layered text-to-SQL security suite.

The system posted 82.95 test execution accuracy with Claude Opus 5 and no fine-tuning, edging DataGallery by 0.56 points while trailing it on development data.

BIRD has now added the JPMorganChase system at 78.14% test execution accuracy—second in the single-trained-model track and 3.37 points above its dev score.

The benchmark now ships 20 reference solutions, but one dashboard task needs three queries above the 20 GiB ceiling used for model runs.

The new enterprise-data benchmark makes a deployment trade-off visible: GPT-6.1 Sol gets most of the leader’s mean score for about a tenth of the model cost.

A Qwen3.8 27B run reached 98.6% in MotherDuck’s DABstep setup on a 32GB MacBook Air, while averaging five to six minutes per answer.

A new 210-task evaluation gives its best model a 59.5/100 mean score—even as established text-to-SQL leaderboards approach the ceiling.

EHR2Trace’s runnable conversion and validation suite separates event time from information availability, turning a familiar warning about temporal leakage into a measured 0.323-point deployment gap.

A released Python package kept overall field-level F1 at 1.000 after simulated schema drift across 34,989 records; the paper also documents one near-miss and a hard boundary on visual HTML.

A released case study shows that implementation diversity cannot catch a shared misunderstanding of source data; 72 of 75 generated verification paths repeated it.

CypherTurn’s oracle-context and autonomous protocols show that query generation and error management are different capabilities—and extra action budget does not close the gap.

ToxicBench separates tool execution from evidence correctness—and shows why a retry helps only when the second observation is clean.

A runnable PostgreSQL extension kept provenance and probability queries below 0.6 seconds on a 1.57-million-row image database, while equivalent rewrites became planning-bound.

GPT-5.3 reached 0.90 macro-F1 across tabular and time-series tasks, while a local 4B model reached 0.75—but neither result measures the downstream cost of a wrong task label.

Across 23 time-series query syntaxes, Claude Opus 5 reached 48.98% execution accuracy while a sampled human baseline reached 87.34%.

A paired benchmark separates algorithmic work from wall-clock cost—and finds adaptive graph search running 20–27× below a dense kernel.

SAP’s runnable A100/H100/B200 benchmark finds more than a 3× gap between backends—and no single kernel wins across row and column attention.

The SemBench tables support a real cost advantage at competitive quality, while exclusions, mixed models and uncounted human planning limit broader comparisons.

A 0.5B model gained 0.9 points on Spider 1.0 dev while discarding 20.4% of its synthetic pool—but the result is a data-selection study, not a new leaderboard system.

A new ontology-grounded EHR benchmark finds self-retrieval helps longitudinal diagnosis—but agent loops hurt most other tasks.

The Meta-led preprint reports 65.7% execution accuracy on Spider 1.0 dev after filtering 20.4% of synthetic examples, but the 0.9-point gain needs careful framing.

A new nine-dataset benchmark tests the hard case where an entire feature is absent at one client, not just scattered values.

The new executable benchmark separates finding a good configuration from predicting how component changes behave across every background.

A new integration benchmark shows why 98% precision is only half the story: LOKI’s evidence-backed joins trade coverage for cost and traceability.

A new 1.26-million-row evaluation shows why repeated purchase text belongs on one side of the train-test boundary—and why leakage can hide the stronger model.

A new executable GPML reference finds multiple engines can accept the same path pattern yet return different endpoint-pair multisets—without an error.

The vendor’s difficulty breakdown is more useful than a headline score, but anonymous rivals and unresolved judge-validation placeholders keep it off the official-comparison track.

The open QLever fork makes large graph-constrained vector joins runnable, but its speedups measure engine efficiency, not retrieval quality.

A four-arm, four-model study finds that structured domain knowledge accounts for more of the hard-task gain than the tools used to retrieve it, but reconstructed golds and a narrow task mix limit the conclusion.

The released framework uses SQL-structure coverage and targeted correction of all-wrong rollout groups, but its author-reported dev score is not a hidden-test leaderboard result.