benchmark tracker
Reference points and reported results on the beat's two load-bearing evaluations. Every number links to its submission, paper, or leaderboard. Maintained continuously by the benchmark desk.
AgenticDataBench
official leaderboard ↗| system | metric | value | reported | source |
|---|---|---|---|---|
| CodeX + Kimi-K2.5 | Overall score, full 344-task benchmark (%) | 48.80 | Jul 2, 2026 | link ↗ |
AIMultiple BIRD-SQL subset (560 train, 199 dev)
official leaderboard ↗| system | metric | value | reported | source |
|---|---|---|---|---|
| gemini-3-flash-preview | SQL accuracy, clean refs, routed hint-free subset (%) | 67.70 | Sep 15, 2026 | link ↗ |
| gemini-3.1-pro-preview | SQL accuracy, clean refs, routed hint-free subset (%) | 67.00 | Sep 15, 2026 | link ↗ |
| gpt-6-astra | SQL accuracy, clean refs, routed hint-free subset (%) | 65.70 | Sep 15, 2026 | link ↗ |
| claude-fable-5 | SQL accuracy, clean refs, routed hint-free subset (%) | 65.50 | Sep 15, 2026 | link ↗ |
| claude-opus-5 | SQL accuracy, clean refs, routed hint-free subset (%) | 64.30 | Sep 15, 2026 | link ↗ |
AMBROSIA ambiguous Text-to-SQL answer-set recovery
official leaderboard ↗| system | metric | value | reported | source |
|---|---|---|---|---|
| CircleClar + GPT-5.4-mini | mean per-question set F1 (96 ambiguous requests) | 56.39 | Oct 14, 2026 | link ↗ |
| CircleClar + Qwen3-30B-A3B | mean per-question set F1 (96 ambiguous requests) | 51.39 | Oct 14, 2026 | link ↗ |
| CircleClar + GPT-5.5 | mean per-question set F1 (96 ambiguous requests) | 48.63 | Oct 14, 2026 | link ↗ |
| CircleClar + Qwen3-4B | mean per-question set F1 (96 ambiguous requests) | 42.64 | Oct 14, 2026 | link ↗ |
ANN Graph Construction
official leaderboard ↗AptMQL-Bench
official leaderboard ↗| system | metric | value | reported | source |
|---|---|---|---|---|
| Claude Opus 4.5 with BIRD external knowledge | Soft execution accuracy (full benchmark) | 70.34 | Oct 2, 2026 | link ↗ |
| GPT-OSS-120B with BIRD external knowledge | Soft execution accuracy (full benchmark) | 62.46 | Oct 2, 2026 | link ↗ |
| Claude Opus 4.5 without external knowledge | Soft execution accuracy (full benchmark) | 57.38 | Oct 2, 2026 | link ↗ |
ARCS
official leaderboard ↗Arcwise-Plat-SQL
official leaderboard ↗Argo-Bench
official leaderboard ↗| system | metric | value | reported | source |
|---|---|---|---|---|
| Claude Opus 5.5 (extra high) | v1.1 official private-world mean task score | 59.50 | Oct 3, 2026 | link ↗ |
| GPT-6 Astra (extra high) | v1.1 official private-world mean task score | 51.80 | Oct 3, 2026 | link ↗ |
| GPT-6.1 Sol (extra high) | v1.1 official private-world mean task score | 49.50 | Oct 3, 2026 | link ↗ |
| Claude Opus 5.5 (extra-high reasoning) | Mean task score — private-seed full set (210 tasks) | 59.50 | Oct 1, 2026 | link ↗ |
| Claude Opus 5.5 (extra-high reasoning) | Solved rate (score ≥95) — private-seed full set (210 tasks) | 34.80 | Oct 1, 2026 | link ↗ |
AutoTuneBench KernelBench Level-1
official leaderboard ↗BEAVER
official leaderboard ↗| system | metric | value | reported | source |
|---|---|---|---|---|
| Claude Sonnet 4.5 + optimized cards and raw SQL | Execution accuracy, dw dev held-out subset (n=300) | 9.00 | Aug 24, 2026 | link ↗ |
| Claude Sonnet 4.5 few-shot reproduction | Execution accuracy, dw dev held-out subset (n=300) | 6.33 | Aug 24, 2026 | link ↗ |
| ReFoRCE + Claude-4.5-Sonnet | Execution accuracy (full BEAVER evaluation; no train/dev/test split published) | 11.40 | May 13, 2026 | link ↗ |
| ReFoRCE + GPT-5.2 | Execution accuracy (full BEAVER evaluation; no train/dev/test split published) | 10.80 | May 13, 2026 | link ↗ |
BEAVER (DW)
official leaderboard ↗| system | metric | value | reported | source |
|---|---|---|---|---|
| JoinGR + BGE-large-en-v1.5 | R@15, 5-fold cross-validation | 0.88 | Oct 1, 2026 | link ↗ |
| Vanilla Retriever + BGE-large-en-v1.5 | R@15, 5-fold cross-validation | 0.70 | Oct 1, 2026 | link ↗ |
| JoinGR + BGE-large-en-v1.5 | PR@15, 5-fold cross-validation | 0.65 | Oct 1, 2026 | link ↗ |
| Vanilla Retriever + BGE-large-en-v1.5 | PR@15, 5-fold cross-validation | 0.34 | Oct 1, 2026 | link ↗ |
BEIR
official leaderboard ↗| system | metric | value | reported | source |
|---|---|---|---|---|
| scrydb Hamming + cosint8 (Qwen3-Embedding-8B) | FiQA test nDCG@10 | 0.65 | Aug 26, 2026 | link ↗ |
BFSI business-query QA
official leaderboard ↗BI-Bench
official leaderboard ↗| system | metric | value | reported | source |
|---|---|---|---|---|
| GPT-5.5 (BI-Agent tools, SQL) | Mean task accuracy, SQL, with tools | 60.20 | Sep 16, 2026 | link ↗ |
| o4-mini (no tools, SQL) | Mean task accuracy, SQL, no-tools | 48.20 | Sep 16, 2026 | link ↗ |
| GPT-5.5 (no tools, SQL) | Mean task accuracy, SQL, no-tools | 46.70 | Sep 16, 2026 | link ↗ |
| Qwen3-8B-RL-Tool (SQL) | Mean task accuracy, SQL, post-trained with tools | 35.00 | Sep 16, 2026 | link ↗ |
BIRD
Big Bench for Large-Scale Database Grounded Text-to-SQL Evaluation: 12,751 questions over 95 databases (33 GB) across 37 domains, emphasizing dirty values, external knowledge, and SQL efficiency. The de facto standard leaderboard for single-database text-to-SQL.
official leaderboard ↗| system | metric | value | reported | source |
|---|---|---|---|---|
| EliteReplay (Qwen3.5-4B-Base) | dev execution pass@128 (128 samples) | 74.80 | Oct 15, 2026 | link ↗ |
| MRTPO + Qwen2.5-Coder-7B-Instruct | execution accuracy (dev) | 56.42 | Oct 15, 2026 | link ↗ |
| Blender-SQL+Qwen3.8-27B | Test execution accuracy (EX), hidden test split | 74.51 | Oct 14, 2026 | link ↗ |
| HERO-O+T (GPT-5, medium reasoning) | pass@1 (development split; mean over 3 seeds) | 64.81 | Oct 13, 2026 | link ↗ |
| HERO-O (GPT-5, medium reasoning) | pass@1 (development split; mean over 3 seeds) | 63.55 | Oct 13, 2026 | link ↗ |
| Qwen3-4B frozen executor + HSAD advisor | execution accuracy — 1,074-question held-out subset of BIRD dev | 43.11 | Oct 9, 2026 | link ↗ |
| Qwen3-4B frozen executor (no HSAD advice) | execution accuracy — 1,074-question held-out subset of BIRD dev | 40.04 | Oct 9, 2026 | link ↗ |
| GrainSQL | Overall test R-VES | 84.10 | Sep 26, 2026 | link ↗ |
| GrainSQL | Overall test Soft F1 | 83.66 | Sep 26, 2026 | link ↗ |
| GrainSQL | Execution Accuracy — test split | 82.95 | Sep 26, 2026 | link ↗ |
| GrainSQL | Overall test execution accuracy (EX) | 82.95 | Sep 26, 2026 | link ↗ |
| GrainSQL | Execution Accuracy — dev split | 77.77 | Sep 26, 2026 | link ↗ |
| GrainSQL | Overall dev execution accuracy (EX) | 77.77 | Sep 26, 2026 | link ↗ |
| LIMIT + Qwen3-8B + self-consistency@8 | Dev execution accuracy (%) | 69.10 | Sep 21, 2026 | link ↗ |
| COAL-SQL (Qwen2.5-Coder-7B-Instruct) | Execution accuracy — development split | 64.90 | Sep 18, 2026 | link ↗ |
| DualSQL-8B | execution accuracy (development split) | 71.10 | Sep 16, 2026 | link ↗ |
| DualSQL-4B | execution accuracy (development split) | 68.00 | Sep 16, 2026 | link ↗ |
| EvoOntology + Claude-Opus-4.8 | VES (Mini-Dev reciprocal held-out, Oracle Knowledge) | 80.50 | Sep 15, 2026 | link ↗ |
| EvoOntology + Claude-Opus-4.8 | EX (Mini-Dev reciprocal held-out, Oracle Knowledge) | 78.30 | Sep 15, 2026 | link ↗ |
| EvoOntology + Claude-Sonnet-5 | VES (Mini-Dev reciprocal held-out, Oracle Knowledge) | 74.10 | Sep 15, 2026 | link ↗ |
| EvoOntology + GPT-5.6-sol | VES (Mini-Dev reciprocal held-out, Oracle Knowledge) | 73.00 | Sep 15, 2026 | link ↗ |
| EvoOntology + Claude-Sonnet-5 | EX (Mini-Dev reciprocal held-out, Oracle Knowledge) | 71.80 | Sep 15, 2026 | link ↗ |
| EvoOntology + GPT-5.5 | VES (Mini-Dev reciprocal held-out, Oracle Knowledge) | 71.10 | Sep 15, 2026 | link ↗ |
| EvoOntology + GPT-5.6-sol | EX (Mini-Dev reciprocal held-out, Oracle Knowledge) | 70.70 | Sep 15, 2026 | link ↗ |
| EvoOntology + GPT-5.5 | EX (Mini-Dev reciprocal held-out, Oracle Knowledge) | 68.90 | Sep 15, 2026 | link ↗ |
BIRD-Clean-xs
official leaderboard ↗| system | metric | value | reported | source |
|---|---|---|---|---|
| SALUS Frontier | annotation-error detection F1 | 0.92 | Oct 6, 2026 | link ↗ |
BIRD cross-dialect DF-Match
official leaderboard ↗| system | metric | value | reported | source |
|---|---|---|---|---|
| Ministral-3-8B fine-tuned on query plans | BIRD dev DF-Match accuracy (SQLite) | 64.50 | Sep 27, 2026 | link ↗ |
BIRD development annotation audit
official leaderboard ↗| system | metric | value | reported | source |
|---|---|---|---|---|
| SALUS Frontier calibrated estimate | estimated annotation error rate (%) | 37.00 | Oct 6, 2026 | link ↗ |
BIRD dev generated-SQL verification
official leaderboard ↗BIRD-History
official leaderboard ↗| system | metric | value | reported | source |
|---|---|---|---|---|
| OpenSearch-SQL with gold history retrieval | Execution Accuracy (full 1,393-task set) | 67.34 | Aug 29, 2026 | link ↗ |
| OpenSearch-SQL + BIRD-History retriever (Q+C) | Execution Accuracy (full 1,393-task set) | 60.95 | Aug 29, 2026 | link ↗ |
| OpenSearch-SQL without history retrieval | Execution Accuracy (full 1,393-task set) | 54.27 | Aug 29, 2026 | link ↗ |
| DAIL-SQL + BIRD-History retriever | Execution Accuracy (full 1,393-task set) | 51.33 | Aug 29, 2026 | link ↗ |
BIRD Mini-Dev PostgreSQL
official leaderboard ↗| system | metric | value | reported | source |
|---|---|---|---|---|
| Ontology2SQL + DeepSeek V4 Flash, deterministic dialect adaptation | Execution accuracy (public Mini-Dev, PostgreSQL replay) | 65.80 | Aug 21, 2026 | link ↗ |
BIRD Mini-Dev SQLite
official leaderboard ↗| system | metric | value | reported | source |
|---|---|---|---|---|
| Ontology2SQL + DeepSeek V4 Flash | Execution accuracy (public Mini-Dev, SQLite) | 70.20 | Aug 21, 2026 | link ↗ |
BIRD Schema Representation Study
official leaderboard ↗| system | metric | value | reported | source |
|---|---|---|---|---|
| Gemini 2.5 Flash | L1·S3 minus L6·S1 EX gap (pp), corrected 397-question subset | 18.20 | Aug 21, 2026 | link ↗ |
| Phi-4 | L1·S3 minus L6·S1 EX gap (pp), corrected 397-question subset | 11.30 | Aug 21, 2026 | link ↗ |
| Qwen2.5-Coder-14B | L1·S3 minus L6·S1 EX gap (pp), corrected 397-question subset | 10.60 | Aug 21, 2026 | link ↗ |
| OLMo-2-13B-Instruct | L1·S3 minus L6·S1 EX gap (pp), corrected 397-question subset | 2.70 | Aug 21, 2026 | link ↗ |
BIRD-SQL
official leaderboard ↗| system | metric | value | reported | source |
|---|---|---|---|---|
| HERO-O+T (GPT-5, medium reasoning) | dev pass@1 mean over 3 runs | 64.81 | Oct 13, 2026 | link ↗ |
| HERO-O (GPT-5, medium reasoning) | dev pass@1 mean over 3 runs | 63.55 | Oct 13, 2026 | link ↗ |
| RAS (Adya AI) | Execution accuracy — test split (%) | 79.82 | Aug 19, 2026 | link ↗ |
| RAS (Adya AI) | Execution Accuracy (EX), test split | 79.82 | Aug 19, 2026 | link ↗ |
| RAS (Adya AI) | Reward-based Valid Efficiency Score (R-VES), test split | 74.95 | Aug 19, 2026 | link ↗ |
| RAS (Adya AI) | Execution Accuracy (EX), original dev split | 72.49 | Aug 19, 2026 | link ↗ |
| RAS (Adya AI) | Execution accuracy — dev split (%) | 72.49 | Aug 19, 2026 | link ↗ |
| EGV-SQL + Qwen3.6-27B (Tampere University) | Execution accuracy — test split (%) | 72.16 | Aug 17, 2026 | link ↗ |
| EGV-SQL + Qwen3.6-27B (Tampere University) | Execution accuracy — dev split (%) | 71.64 | Aug 17, 2026 | link ↗ |
BIRD-SQL Execution Accuracy
official leaderboard ↗BIRD-SQL Reward-based Valid Efficiency Score
official leaderboard ↗BIRD-SQL test
official leaderboard ↗| system | metric | value | reported | source |
|---|---|---|---|---|
| ReToolSQL SFT→RFT 31B (self-consistency) | Execution accuracy (test split) | 78.14 | Sep 30, 2026 | link ↗ |
Bolo Model Remediation
official leaderboard ↗| system | metric | value | reported | source |
|---|---|---|---|---|
| Bolo agentic remediation | Type II runtime-error-free pipeline coverage | 97.27 | Aug 29, 2026 | link ↗ |
| Bolo after HalluVer filtering | Type II retained coverage after hallucination filter | 88.50 | Aug 29, 2026 | link ↗ |
| Bolo agentic remediation | Type III runtime-error-free pipeline coverage | 86.08 | Aug 29, 2026 | link ↗ |
BranchBench
official leaderboard ↗BSBM-DVQ
official leaderboard ↗| system | metric | value | reported | source |
|---|---|---|---|---|
| QUIVER-TV+Index | Scan median speedup vs RDFTensor (best measured size) | 250.00 | Sep 21, 2026 | link ↗ |
BTTF chip-verification NL-to-SQL benchmark
official leaderboard ↗| system | metric | value | reported | source |
|---|---|---|---|---|
| BTTF (Gemini 2.5 Pro) | Execution accuracy (overall, 150-query benchmark) | 95.33 | Oct 5, 2026 | link ↗ |
Cardinality Estimation Q-error
official leaderboard ↗| system | metric | value | reported | source |
|---|---|---|---|---|
| QEmbed | Forest 4K inference latency (ms) | 3745.21 | Sep 14, 2026 | link ↗ |
| Embed baseline | Forest 4K workload max Q-error (lower is better) | 2840.00 | Sep 14, 2026 | link ↗ |
| QEmbed | Forest 4K workload max Q-error (lower is better) | 11.00 | Sep 14, 2026 | link ↗ |
| Embed baseline | Forest 4K inference latency (ms) | 0.61 | Sep 14, 2026 | link ↗ |
CLEVER semantic-cache evaluation
official leaderboard ↗| system | metric | value | reported | source |
|---|---|---|---|---|
| LFU (MiniLM, LMSYS, 10% cache) | Raw cache hit rate (%) | 57.00 | Aug 20, 2026 | link ↗ |
Common-mode verification-path replication
official leaderboard ↗| system | metric | value | reported | source |
|---|---|---|---|---|
| Three pinned Claude models, tools disabled | Paths reproducing defective count (%) | 96.00 | Sep 29, 2026 | link ↗ |
CorpFam
official leaderboard ↗| system | metric | value | reported | source |
|---|---|---|---|---|
| MiniLM-L6-v2 sentence embedding matcher | Invisible-name stratum recall (test, %) | 4.70 | Sep 3, 2026 | link ↗ |
Cortex AISQL compositional online-learning
official leaderboard ↗Croissant Policy Profile evaluation
official leaderboard ↗D² Data Investigations (50-case benchmark)
official leaderboard ↗| system | metric | value | reported | source |
|---|---|---|---|---|
| D² | Correctness F1 (%) — all 50 cases | 95.26 | Sep 3, 2026 | link ↗ |
| OpenClaw + Claude Sonnet 5 | Correctness F1 (%) — all 50 cases | 94.64 | Sep 3, 2026 | link ↗ |
| Claude Code + Claude Sonnet 5 | Correctness F1 (%) — all 50 cases | 93.37 | Sep 3, 2026 | link ↗ |
| D² | Completeness F1 (%) — all 50 cases | 76.02 | Sep 3, 2026 | link ↗ |
| D² | Verifiability ease (%) — all 50 cases | 68.88 | Sep 3, 2026 | link ↗ |
| OpenClaw + Claude Sonnet 5 | Completeness F1 (%) — all 50 cases | 32.35 | Sep 3, 2026 | link ↗ |
| Claude Code + Claude Sonnet 5 | Completeness F1 (%) — all 50 cases | 32.22 | Sep 3, 2026 | link ↗ |
| Claude Code + Claude Sonnet 5 | Completeness F1 (%) — all 50 cases | 32.00 | Sep 3, 2026 | link ↗ |
| OpenClaw + Claude Sonnet 5 | Completeness F1 (%) — all 50 cases | 32.00 | Sep 3, 2026 | link ↗ |
| Claude Code + Claude Sonnet 5 | Verifiability ease (%) — all 50 cases | 2.91 | Sep 3, 2026 | link ↗ |
| OpenClaw + Claude Sonnet 5 | Verifiability ease (%) — all 50 cases | 2.22 | Sep 3, 2026 | link ↗ |
DABStep
official leaderboard ↗| system | metric | value | reported | source |
|---|---|---|---|---|
| GPT-5.6-sol + frozen data contract | Hard accuracy, reconstructed-gold split (n=332) | 77.40 | Sep 22, 2026 | link ↗ |
| Claude Sonnet 5 + frozen data contract | Hard accuracy, reconstructed-gold split (n=332) | 68.40 | Sep 22, 2026 | link ↗ |
| DeepSeek-V4-Flash + frozen data contract | Hard accuracy, complete-case reconstructed-gold split (n=226) | 56.60 | Sep 22, 2026 | link ↗ |
| GLM-5.3-Flash + frozen data contract | Hard accuracy, reconstructed-gold split (n=332) | 55.10 | Sep 22, 2026 | link ↗ |
| GLM-5.3-Flash + frozen data contract | Official leaderboard hard accuracy (n=378) | 51.90 | Sep 4, 2026 | link ↗ |
DAGSmith dbt pipeline optimization
official leaderboard ↗| system | metric | value | reported | source |
|---|---|---|---|---|
| DAGSmith | Tuva 255-model, frequency-weighted BigQuery slot-time reduction (%) | 67.70 | Aug 23, 2026 | link ↗ |
| DAGSmith | Stripe 65-model project, BigQuery frequency-weighted slot-time reduction (%) | 54.80 | Aug 23, 2026 | link ↗ |
| DAGSmith | Tuva 255-model claims module, BigQuery single-run slot-time reduction (%) | 51.10 | Aug 23, 2026 | link ↗ |
| DAGSmith | Stripe 65-model project, BigQuery single-run slot-time reduction (%) | 47.40 | Aug 23, 2026 | link ↗ |
| DAGSmith | Tuva 255-model, frequency-weighted elapsed-time reduction (%) | 42.60 | Aug 23, 2026 | link ↗ |
Data-Agent Benchmark
official leaderboard ↗| system | metric | value | reported | source |
|---|---|---|---|---|
| GPT-5.6 Sol Codex agent — no persistent context | Average turns/query, 42-task held-out split | 5.90 | Sep 2, 2026 | link ↗ |
| GPT-5.6 Sol Codex agent — self-curated schema context | Average turns/query, 42-task held-out split | 4.60 | Sep 2, 2026 | link ↗ |
| GPT-5.6 Sol Codex agent — self-curated latency context | Average turns/query, 42-task held-out split | 4.50 | Sep 2, 2026 | link ↗ |
DataBench v1
official leaderboard ↗| system | metric | value | reported | source |
|---|---|---|---|---|
| Opus 5 High | Blended score — 100 tasks | 88.00 | Aug 13, 2026 | link ↗ |
| Fable 5 Max | Blended score — 100 tasks | 85.00 | Aug 13, 2026 | link ↗ |
| GPT-5.6 Sol XHigh | Blended score — 100 tasks | 75.00 | Aug 13, 2026 | link ↗ |
| GPT-5.6 Luna XHigh | Blended score — 100 tasks | 73.00 | Aug 13, 2026 | link ↗ |
| Opus 5 Max | Blended score — 100 tasks | 70.00 | Aug 13, 2026 | link ↗ |
DataKernelBench
official leaderboard ↗| system | metric | value | reported | source |
|---|---|---|---|---|
| GPT-5.5 — CUDA, full-query | Overall speedup vs compiled TorchPlan — TPC-H SF10, 22 queries, 1 H100 | 2.11 | Aug 25, 2026 | link ↗ |
| Claude Sonnet 4.6 — Triton, full-query | Overall speedup vs compiled TorchPlan — TPC-H SF10, 22 queries, 1 H100 | 1.54 | Aug 25, 2026 | link ↗ |
| Claude Opus 4.7 — CUDA, full-query | Overall speedup vs compiled TorchPlan — TPC-H SF10, 22 queries, 1 H100 | 1.51 | Aug 25, 2026 | link ↗ |
| Gemini 3.1 Pro Preview — CUDA, full-query | Overall speedup vs compiled TorchPlan — TPC-H SF10, 22 queries, 1 H100 | 1.44 | Aug 25, 2026 | link ↗ |
| Claude Haiku 4.5 — Triton, full-query | Overall speedup vs compiled TorchPlan — TPC-H SF10, 22 queries, 1 H100 | 1.30 | Aug 25, 2026 | link ↗ |
| Qwen3.5-397B-A17B-FP8 — Triton, full-query | Overall speedup vs compiled TorchPlan — TPC-H SF10, 22 queries, 1 H100 | 1.26 | Aug 25, 2026 | link ↗ |
| GPT-OSS-120B — CUDA, full-query | Overall speedup vs compiled TorchPlan — TPC-H SF10, 22 queries, 1 H100 | 1.26 | Aug 25, 2026 | link ↗ |
| DeepSeek-V4-Flash — Triton, core | Overall speedup vs compiled TorchPlan — TPC-H SF10, 22 queries, 1 H100 | 1.23 | Aug 25, 2026 | link ↗ |
| MiniMax-M2.5 — Triton, full-query | Overall speedup vs compiled TorchPlan — TPC-H SF10, 22 queries, 1 H100 | 1.19 | Aug 25, 2026 | link ↗ |
| Devstral-2-123B-Instruct-2512 — Triton, full-query | Overall speedup vs compiled TorchPlan — TPC-H SF10, 22 queries, 1 H100 | 1.08 | Aug 25, 2026 | link ↗ |
DBcover SQL test generation
official leaderboard ↗DBLP-QuAD 3.1
official leaderboard ↗| system | metric | value | reported | source |
|---|---|---|---|---|
| NLKGQ + Qwen3.6-27B | Match (1,000-question test set) | 89.90 | Sep 15, 2026 | link ↗ |
DBpedia-MS
official leaderboard ↗| system | metric | value | reported | source |
|---|---|---|---|---|
| QUIVER-TV+Index | Single-type ranking median speedup vs RDFTensor | 97.80 | Sep 21, 2026 | link ↗ |
DevRev NL2SQL Benchmark
official leaderboard ↗| system | metric | value | reported | source |
|---|---|---|---|---|
| DevRev cost-aware NL-to-SQL architecture | Answer correctness (private 900-query set) | 91.70 | Sep 4, 2026 | link ↗ |
Distributed DBMS testing
official leaderboard ↗DocAttriBench
official leaderboard ↗| system | metric | value | reported | source |
|---|---|---|---|---|
| MAPPET-I-8B (InternVL3.5-8B fine-tuned) | Post-hoc grounding score, test sources | 68.10 | Sep 17, 2026 | link ↗ |
| MAPPET-8B (Qwen3-VL-8B fine-tuned) | Avg overall correctness, test sources | 37.00 | Sep 17, 2026 | link ↗ |
| Qwen3-VL-32B zero-shot | Avg overall correctness, test sources | 25.40 | Sep 17, 2026 | link ↗ |
DRL Enterprise NL2SQL Verification Suite
official leaderboard ↗| system | metric | value | reported | source |
|---|---|---|---|---|
| GPT-4o | PostgreSQL execution match, corrected harness, schema-linked, 1,000-pair suite | 52.90 | Aug 28, 2026 | link ↗ |
| Claude Sonnet 4.5 | PostgreSQL execution match, corrected harness, schema-linked, 1,000-pair suite | 52.80 | Aug 28, 2026 | link ↗ |
| Gemini 2.5 Flash | PostgreSQL execution match, corrected harness, schema-linked, 1,000-pair suite | 52.10 | Aug 28, 2026 | link ↗ |
DSABench
official leaderboard ↗| system | metric | value | reported | source |
|---|---|---|---|---|
| Claude Opus 5 + Ask Once (best observed clarified configuration) | Route+Exec-Graph, full frozen 815-task set (%) | 16.99 | Oct 10, 2026 | link ↗ |
DS-NL2SQL
official leaderboard ↗Dynamic Densest Subhypergraph
official leaderboard ↗| system | metric | value | reported | source |
|---|---|---|---|---|
| CAP | maximum measured speedup vs Udshp (completed settings) | 1869.20 | Sep 30, 2026 | link ↗ |
EHR2Trace injected-fault suite
official leaderboard ↗| system | metric | value | reported | source |
|---|---|---|---|---|
| EHR2Trace validation suite | Injected faults detected | 28.00 | Sep 18, 2026 | link ↗ |
EHR2Trace temporal-leakage experiment
official leaderboard ↗| system | metric | value | reported | source |
|---|---|---|---|---|
| backdated-diagnosis training | 6-hour held-out AUROC (backdated test) | 0.97 | Sep 18, 2026 | link ↗ |
| availability-filtered training | 6-hour held-out AUROC (availability-filtered test) | 0.83 | Sep 18, 2026 | link ↗ |
| backdated-diagnosis training | 6-hour held-out AUROC (availability-filtered test) | 0.64 | Sep 18, 2026 | link ↗ |
EHR-RobustGym
official leaderboard ↗EHRSQL
official leaderboard ↗| system | metric | value | reported | source |
|---|---|---|---|---|
| MaP-SQL + Arctic-Text2SQL-R1-7B (32 candidates) | Execution accuracy | 44.71 | Sep 7, 2026 | link ↗ |
EHRSQL Sentinel held-out evaluation
official leaderboard ↗| system | metric | value | reported | source |
|---|---|---|---|---|
| GPT-4o + Sentinel | Delivered-answer accuracy (λ=1, held-out %) | 69.00 | Oct 2, 2026 | link ↗ |
| GPT-4o + Sentinel | Answer coverage (λ=1, held-out %) | 55.00 | Oct 2, 2026 | link ↗ |
| GPT-4o + Sentinel | Reliability Score (λ=1, held-out) | 0.24 | Oct 2, 2026 | link ↗ |
| GPT-4o vanilla ReAct | Reliability Score (λ=1, held-out) | 0.08 | Oct 2, 2026 | link ↗ |
Enterprise Analytics Agent Trace Eval
official leaderboard ↗| system | metric | value | reported | source |
|---|---|---|---|---|
| Reasoning configuration | Interpretation-unstable questions (%) — 50-question internal set, 3 runs each | 82.00 | Aug 28, 2026 | link ↗ |
| Reasoning configuration | Real-data answer rate (%) — 150-trace internal set | 73.00 | Aug 28, 2026 | link ↗ |
| Reasoning configuration | Tool-round budget exhaustion rate (%) — 150-trace internal set | 16.00 | Aug 28, 2026 | link ↗ |
Enterprise Context Artifacts
official leaderboard ↗ERPBench
official leaderboard ↗| system | metric | value | reported | source |
|---|---|---|---|---|
| Claude Sonnet 4.6 | database-verified task success, T2 (4 tasks; 20 runs) | 100.00 | Sep 15, 2026 | link ↗ |
| Claude Sonnet 4.6 | database-verified task success, T3 (6 tasks; 30 runs) | 100.00 | Sep 15, 2026 | link ↗ |
| Claude Sonnet 4.6 | database-verified task success, T1 (20 tasks; 100 runs) | 94.00 | Sep 15, 2026 | link ↗ |
| Holo3-35B-A3B | database-verified task success, T1 (20 tasks; 100 runs) | 34.00 | Sep 15, 2026 | link ↗ |
| UI-TARS-7B | database-verified task success, T1 (20 tasks; 100 runs) | 9.00 | Sep 15, 2026 | link ↗ |
EXPLAIN Yourself planner-stall search
official leaderboard ↗| system | metric | value | reported | source |
|---|---|---|---|---|
| GPT-5.5 agentic SQL search across seven DBMSes | DBMSes with ≥1 EXPLAIN planning time >3 min; TPC-H SF100; no split | 7.00 | Aug 24, 2026 | link ↗ |
FALCON NL2SQL
official leaderboard ↗Fed-ReMasker federated tabular imputation benchmark
official leaderboard ↗FinTabNetQA
official leaderboard ↗| system | metric | value | reported | source |
|---|---|---|---|---|
| DELTA + TarQA-OTSL | Relieved accuracy — test (250 examples) | 45.20 | Sep 15, 2026 | link ↗ |
FLINT Financials
official leaderboard ↗| system | metric | value | reported | source |
|---|---|---|---|---|
| FLINT + Claude 4.5 Opus | Strict semantic accuracy (221-question proprietary evaluation set) | 68.80 | Oct 2, 2026 | link ↗ |
FLINT Transactions
official leaderboard ↗| system | metric | value | reported | source |
|---|---|---|---|---|
| FLINT + Claude 4.5 Opus | Strict semantic accuracy (138-question proprietary evaluation set) | 65.20 | Oct 2, 2026 | link ↗ |
FRESH-GEORANGE CPU pilot
official leaderboard ↗| system | metric | value | reported | source |
|---|---|---|---|---|
| FRESH-GEORANGE Exact | Mean set recall, day-60 180-query pilot (%) | 100.00 | Sep 17, 2026 | link ↗ |
| FRESH-GEORANGE Cert95 | Mean set recall, day-60 180-query pilot (%) | 99.91 | Sep 17, 2026 | link ↗ |
| FRESH-GEORANGE Cert95 | Mean reported deterministic recall certificate, day-60 180-query pilot (%) | 99.41 | Sep 17, 2026 | link ↗ |
Ghost Echoes retrieval deletion audit
official leaderboard ↗| system | metric | value | reported | source |
|---|---|---|---|---|
| ChromaDB 0.4.24 / all-MiniLM-L6-v2 | median Top-5 retrieval-centroid drift after target deletion | 0.15 | Aug 24, 2026 | link ↗ |
Habit Formation Text-to-SQL Pilot
official leaderboard ↗| system | metric | value | reported | source |
|---|---|---|---|---|
| Habit-formed deterministic variant | Exact repeated-dispatch agreement (%) — self-authored text-to-SQL workload | 100.00 | Sep 21, 2026 | link ↗ |
Hacker News wildcard benchmark
official leaderboard ↗HakiCC generated concurrency-control protocols
official leaderboard ↗| system | metric | value | reported | source |
|---|---|---|---|---|
| HakiCC Stage 2 (AuctionMark cohort, 5 protocols) | Mean throughput gain vs Stage 1 across evaluated AuctionMark scenarios (%) | 92.20 | Oct 1, 2026 | link ↗ |
| HakiCC Stage 2 (TPC-C cohort, 5 protocols) | Mean throughput gain vs Stage 1 across evaluated TPC-C scenarios (%) | 50.60 | Oct 1, 2026 | link ↗ |
HistoRAG Regional Archive QA
official leaderboard ↗| system | metric | value | reported | source |
|---|---|---|---|---|
| HistoRAG | Answer accuracy (test split) | 82.00 | Sep 24, 2026 | link ↗ |
HybridDeepResearch
official leaderboard ↗| system | metric | value | reported | source |
|---|---|---|---|---|
| GLM-5.2-FP4 + MiroFlow + Corpus Search | Pass@8 — full set (380 tasks) | 75.79 | Sep 8, 2026 | link ↗ |
| GPT-5 + smolagents + Corpus Search | Pass@8 — balanced hard subset (120 tasks) | 54.17 | Sep 8, 2026 | link ↗ |
| Claude Sonnet 4.6 + smolagents + Corpus Search | Avg@8 — balanced hard subset (120 tasks) | 28.12 | Sep 8, 2026 | link ↗ |
IDS sliding-window attribution runtime
official leaderboard ↗| system | metric | value | reported | source |
|---|
Interpretive Evidence Score audit
official leaderboard ↗| system | metric | value | reported | source |
|---|---|---|---|---|
| Agreement-first score (v2.1) | AUROC on Spider dev (pre-registered; model-judged kept answers, n=983) | 0.73 | Oct 12, 2026 | link ↗ |
| Designed IES score (bench-1.0) | AUROC on Spider dev (model-judged kept answers, n=983) | 0.57 | Oct 12, 2026 | link ↗ |
| Agreement-first score (v2.1) | AUROC on BIRD dev rerun (model-judged kept answers, n=1,424) | 0.76 | Oct 11, 2026 | link ↗ |
| Designed IES score (bench-1.0) | AUROC on BIRD dev rerun (model-judged kept answers, n=1,424) | 0.59 | Oct 11, 2026 | link ↗ |
ISC Adversarial Governance Reconstruction
official leaderboard ↗KnowFeat public tabular suite
official leaderboard ↗KnowFeat Public Tabular Suite
official leaderboard ↗LibreDB Database Agent Runs
official leaderboard ↗| system | metric | value | reported | source |
|---|---|---|---|---|
| 40-model LibreDB agent corpus | Tool-engaged share of model-attributed agent-mode losses | 75.70 | Sep 18, 2026 | link ↗ |
LiveSQLBench Base-Full v1
official leaderboard ↗LiveSQLBench Base Lite
official leaderboard ↗| system | metric | value | reported | source |
|---|---|---|---|---|
| o3-mini | Execution accuracy — PostgreSQL split | 47.78 | Jul 22, 2025 | link ↗ |
| o3-mini | Execution accuracy — SQLite split | 42.59 | Jul 22, 2025 | link ↗ |
| Claude 3.7 Sonnet | Execution accuracy — SQLite split | 41.11 | Jul 22, 2025 | link ↗ |
| Claude 3.7 Sonnet | Execution accuracy — PostgreSQL split | 39.26 | Jul 22, 2025 | link ↗ |
| DeepSeek R1-0528 | Execution accuracy — PostgreSQL split | 38.14 | Jul 22, 2025 | link ↗ |
| DeepSeek R1-0528 | Execution accuracy — SQLite split | 32.96 | Jul 22, 2025 | link ↗ |
| Mixtral 8x7B Instruct | Execution accuracy — SQLite split | 8.89 | Jul 22, 2025 | link ↗ |
| Mixtral 8x7B Instruct | Execution accuracy — PostgreSQL split | 2.59 | Jul 22, 2025 | link ↗ |
LOKI Text-Mediated Integration
official leaderboard ↗LongDocURL
official leaderboard ↗| system | metric | value | reported | source |
|---|---|---|---|---|
| Six-tool function-calling agent + Claude Sonnet 4.5 | Official scorer accuracy (public 2,325-question split) | 0.66 | Sep 24, 2026 | link ↗ |
LongMemEval-S
official leaderboard ↗| system | metric | value | reported | source |
|---|---|---|---|---|
| Random eviction, GPT-4o-mini reader, 80k budget, top-k=60 retrieval | Irreversible share among restore-corrected errors | 0.73 | Sep 8, 2026 | link ↗ |
| FIFO eviction, GPT-4o-mini reader, 80k budget, top-k=60 retrieval | Irreversible share among restore-corrected errors | 0.71 | Sep 8, 2026 | link ↗ |
| Redundancy-aware eviction, GPT-4o-mini reader, 80k budget, top-k=60 retrieval | Irreversible share among restore-corrected errors | 0.67 | Sep 8, 2026 | link ↗ |
| LLM-importance eviction, GPT-4o-mini reader, 80k budget, top-k=60 retrieval | Irreversible share among restore-corrected errors | 0.60 | Sep 8, 2026 | link ↗ |
Low-Bit Vector Search Decision Stability
official leaderboard ↗| system | metric | value | reported | source |
|---|---|---|---|---|
| Standardized exact-margin predictor (24 dataset–quantizer configurations) | Spearman correlation with held-out ranking flip rate | 0.97 | Sep 9, 2026 | link ↗ |
| Global distance rank correlation predictor (24 dataset–quantizer configurations) | Spearman correlation with held-out pruning flip rate | 0.05 | Sep 9, 2026 | link ↗ |
MCP Blueprint Sakila reproducibility benchmark
official leaderboard ↗| system | metric | value | reported | source |
|---|---|---|---|---|
| Verticalized domain tool pack (Approach B) | Pooled mean score (17 tasks, four 3B-8B models, 3 reps) | 0.94 | Aug 22, 2026 | link ↗ |
| Raw SQL execute_sql interface (Approach A) | Pooled mean score (17 tasks, four 3B-8B models, 3 reps) | 0.67 | Aug 22, 2026 | link ↗ |
| Generic thin-tool pack (Approach C) | Pooled mean score (17 tasks, four 3B-8B models, 3 reps) | 0.61 | Aug 22, 2026 | link ↗ |
MediaSum ISC held-out QA
official leaderboard ↗| system | metric | value | reported | source |
|---|---|---|---|---|
| Contextualized chunks + hybrid retrieval + reranker | Answer correctness (%) — held-out 499-question split, 2,048-token budget | 88.00 | Aug 21, 2026 | link ↗ |
| Ingest-time semantic compilation (compiled claims) | Answer correctness (%) — held-out 499-question split, 2,048-token budget | 85.20 | Aug 21, 2026 | link ↗ |
MemTrapBench
official leaderboard ↗ML Task Type Identification Benchmark
official leaderboard ↗| system | metric | value | reported | source |
|---|---|---|---|---|
| AutoGluon baseline | Macro F1 (tabular held-out test split) | 0.93 | Sep 28, 2026 | link ↗ |
| GPT-5.3 reasoning few-shot with descriptions + target statistics | Macro F1 (cross-domain held-out test split) | 0.90 | Sep 28, 2026 | link ↗ |
| Qwen3-4B-Instruct few-shot with descriptions + target statistics | Macro F1 (cross-domain held-out test split) | 0.75 | Sep 28, 2026 | link ↗ |
MMLongBench-Doc
official leaderboard ↗| system | metric | value | reported | source |
|---|---|---|---|---|
| Six-tool function-calling agent + Claude Sonnet 4.5 | Official scorer accuracy (full 1,082-question set) | 0.63 | Sep 24, 2026 | link ↗ |
MMQA
official leaderboard ↗MM-quecat multi-model query evaluation
official leaderboard ↗| system | metric | value | reported | source |
|---|---|---|---|---|
| Post-hoc oracle physical-representation selector | Mean-latency reduction vs best fixed single-DBMS environment (%) | 20.00 | Sep 7, 2026 | link ↗ |
nf-core admission gate replication
official leaderboard ↗| system | metric | value | reported | source |
|---|---|---|---|---|
| ok-nfcore-admission-gate | Gate-process median latency (microseconds, 210 decisions) | 119.00 | Sep 17, 2026 | link ↗ |
O-Funnel hard extraction scenarios
official leaderboard ↗O-Funnel PubMed schema-drift evaluation
official leaderboard ↗PIMENTO DP-D1
official leaderboard ↗PlotGround-1k
official leaderboard ↗PLSQLBench
official leaderboard ↗PremiseSQL
official leaderboard ↗| system | metric | value | reported | source |
|---|---|---|---|---|
| 16-model aggregate | Base prompting Original-query SQL generation (GEN), % | 98.80 | Oct 15, 2026 | link ↗ |
| 16-model aggregate | Explicit prompting Flawed-query detection (DET), % | 93.10 | Oct 15, 2026 | link ↗ |
| 16-model aggregate | Implicit prompting Original-query SQL generation (GEN), % | 90.30 | Oct 15, 2026 | link ↗ |
| 16-model aggregate — Implicit prompting | Flawed-query detection (DET), % | 67.20 | Oct 15, 2026 | link ↗ |
| 16-model aggregate — Explicit prompting | Original-query SQL generation (GEN), % | 52.30 | Oct 15, 2026 | link ↗ |
| 16-model aggregate — Base prompting | Flawed-query detection (DET), % | 31.90 | Oct 15, 2026 | link ↗ |
ProcArena Direct
official leaderboard ↗| system | metric | value | reported | source |
|---|---|---|---|---|
| Gemini-3.1 Pro | Execution accuracy — Direct, pooled PostgreSQL + Oracle | 62.20 | Sep 6, 2026 | link ↗ |
ProcArena Interactive
official leaderboard ↗| system | metric | value | reported | source |
|---|---|---|---|---|
| Gemini-3.1 Pro | Execution accuracy — Interactive, pooled PostgreSQL + Oracle | 57.80 | Sep 6, 2026 | link ↗ |
proc-posets public event-log screen
official leaderboard ↗| system | metric | value | reported | source |
|---|---|---|---|---|
| Public event-log candidate corpus | logs with inspected within-case concurrency | 0.00 | Sep 17, 2026 | link ↗ |
ProvSQL HAVING Provenance Evaluation
official leaderboard ↗| system | metric | value | reported | source |
|---|---|---|---|---|
| ProvSQL direct HAVING semantics + probability evaluation | End-to-end speedup vs self-join, COUNT(*) <= 15, COCO-derived split | 115.00 | Sep 25, 2026 | link ↗ |
| ProvSQL direct HAVING semantics | Provenance speedup vs self-join, COUNT(*) <= 15, COCO-derived split | 108.00 | Sep 25, 2026 | link ↗ |
Quantum Range Query trajectory evaluation
official leaderboard ↗| system | metric | value | reported | source |
|---|---|---|---|---|
| QRQ KD-tree on GDP Drifter | Modeled C/Q I/O speedup, space-time query, 1% selectivity | 64.70 | Sep 26, 2026 | link ↗ |
RelBench beer-churn
official leaderboard ↗RTGL RelBench driver-dnf
official leaderboard ↗ScienceBenchmark
official leaderboard ↗| system | metric | value | reported | source |
|---|---|---|---|---|
| MIRA | Execution Accuracy Improvement (%) | 8.78 | Aug 7, 2026 | link ↗ |
Self-sizing IBLT production-shape replay
official leaderboard ↗| system | metric | value | reported | source |
|---|---|---|---|---|
| Self-sizing IBLT | End-to-end speedup vs tuned Merkle-style localization, P1, 10 Mbps | 1.55 | Aug 27, 2026 | link ↗ |
SemBench
official leaderboard ↗| system | metric | value | reported | source |
|---|---|---|---|---|
| JEVDB-Flash | Aggregate latency, 21-query text/structured subset (seconds) | 41.50 | Oct 1, 2026 | link ↗ |
| KathDB-FAO | Additional quality (sf1000/400) | 0.92 | Sep 23, 2026 | link ↗ |
| KathDB-FAO | MMQA quality (sf200; excl Q5) | 0.85 | Sep 23, 2026 | link ↗ |
| KathDB-FAO | Movie quality (sf2000) | 0.81 | Sep 23, 2026 | link ↗ |
| KathDB-FAO | E-Com quality (sf500; excl Q14) | 0.77 | Sep 23, 2026 | link ↗ |
| Multilayered Evaluation Framework (BigQuery) | Overall accuracy — BigQuery subset, no dataset split (%) | 97.20 | Sep 17, 2026 | link ↗ |
| Multilayered Evaluation Framework (ThalamusDB) | Overall accuracy — ThalamusDB subset, no dataset split (%) | 93.30 | Sep 17, 2026 | link ↗ |
| DASE + BigQuery | E-Commerce avg. quality (Gemini 2.5 Flash) | 0.80 | Sep 17, 2026 | link ↗ |
| BigQuery | E-Commerce avg. quality (Gemini 2.5 Flash baseline reproduced) | 0.67 | Sep 17, 2026 | link ↗ |
Shelob
official leaderboard ↗SmallBank on ChainMaker execution layer
official leaderboard ↗| system | metric | value | reported | source |
|---|---|---|---|---|
| Lantern | Throughput ktxn/s (Zipf skew 0.9, CFBS enabled) | 14.78 | Sep 4, 2026 | link ↗ |
SmartANN ANN auto-design evaluation
official leaderboard ↗| system | metric | value | reported | source |
|---|---|---|---|---|
| SmartANN (IVF-PQ and HNSW) | max Recall@k gain vs baseline, final split (percentage points) | 74.20 | Sep 9, 2026 | link ↗ |
Spend Classification Without Leakage
official leaderboard ↗| system | metric | value | reported | source |
|---|---|---|---|---|
| 1-NN TF-IDF (wa-amazon commodity, iid-naive) | Accuracy — iid-naive commodity split | 87.50 | Sep 22, 2026 | link ↗ |
| DistilBERT (wa-amazon commodity, iid-grouped) | Accuracy — iid-grouped commodity split | 83.00 | Sep 22, 2026 | link ↗ |
| 1-NN TF-IDF (wa-amazon commodity, iid-grouped) | Accuracy — iid-grouped commodity split | 79.50 | Sep 22, 2026 | link ↗ |
Spider
official leaderboard ↗| system | metric | value | reported | source |
|---|---|---|---|---|
| EliteReplay (Qwen3.5-4B-Base, trained on BIRD) | dev execution pass@128 (128 samples, out-of-distribution) | 93.20 | Oct 15, 2026 | link ↗ |
| MRTPO + Qwen2.5-Coder-7B-Instruct | execution accuracy (dev) | 81.33 | Oct 15, 2026 | link ↗ |
| Prompt workflow (Prem-1B-SQL) | Execution accuracy (dev, 1,034 examples) | 97.20 | Oct 12, 2026 | link ↗ |
| Prompt workflow (Prem-1B-SQL) | development execution accuracy | 97.20 | Oct 12, 2026 | link ↗ |
| Prompt workflow (Prem-1B-SQL) | Exact-set-match accuracy (dev, 1,034 examples) | 41.30 | Oct 12, 2026 | link ↗ |
| FROST + Qwen2.5-Coder-0.5B-Instruct | Execution accuracy (dev, with 20.4% synthetic-data drop) | 65.70 | Sep 24, 2026 | link ↗ |
| LIMIT + Qwen3-8B + self-consistency@8 | Dev execution accuracy (%) | 88.90 | Sep 21, 2026 | link ↗ |
| MaP-SQL + Arctic-Text2SQL-R1-7B (32 candidates) | Execution accuracy (test) | 87.59 | Sep 7, 2026 | link ↗ |
| SQL-Zero-7B iter1 | Execution accuracy (test, greedy@1) | 82.40 | Sep 4, 2026 | link ↗ |
| SQL-Zero-3B iter2 | Execution accuracy (test, greedy@1) | 74.70 | Sep 4, 2026 | link ↗ |
| GPT-5.4 + BIRD-derived E2/S2/G2/R1 pipeline | Execution accuracy, dev split | 87.04 | Aug 28, 2026 | link ↗ |
| Gemini-2.5-Flash + BIRD-derived E2/S2/G2/R1 pipeline | Execution accuracy, dev split | 84.72 | Aug 28, 2026 | link ↗ |
| Qwen2.5-Instruct-7B, 4-bit, grammar-constrained beam width 4 | Development execution accuracy (%) | 66.20 | Aug 26, 2026 | link ↗ |
| Qwen2.5-Instruct-3B, 4-bit, grammar-constrained beam width 8 | Development execution accuracy (%) | 56.20 | Aug 26, 2026 | link ↗ |
| Qwen2.5-Instruct-1.5B, 4-bit, grammar-constrained beam width 8 | Development execution accuracy (%) | 50.50 | Aug 26, 2026 | link ↗ |
| Qwen2.5-Instruct-0.5B, 4-bit, grammar-constrained beam width 8 | Development execution accuracy (%) | 24.50 | Aug 26, 2026 | link ↗ |
| Qwen3-8B + LoRA (CoT SFT) | test execution accuracy (EX, 2,147 examples) | 82.24 | Aug 15, 2026 | link ↗ |
| Qwen3-8B + LoRA (No-CoT SFT) | test execution accuracy (EX, 2,147 examples) | 77.04 | Aug 15, 2026 | link ↗ |
| LLaMA-3.1-8B + LoRA (No-CoT SFT) | test execution accuracy (EX, 2,147 examples) | 76.01 | Aug 15, 2026 | link ↗ |
| LLaMA-3.1-8B + LoRA (CoT SFT) | test execution accuracy (EX, 2,147 examples) | 76.01 | Aug 15, 2026 | link ↗ |
| GLM-4 (3-shot) | test execution accuracy (EX, 2,147 examples) | 66.28 | Aug 15, 2026 | link ↗ |
| DeepSeek V3 (3-shot) | test execution accuracy (EX, 2,147 examples) | 51.47 | Aug 15, 2026 | link ↗ |
| SafeQL (hybrid) + DAIL-SQL + GPT-OSS-120B | EX (%) | 91.70 | Aug 10, 2026 | link ↗ |
| SafeQL hybrid + DAIL-SQL (GPT-OSS-120B) | Execution accuracy (EX, %) | 91.70 | Aug 10, 2026 | link ↗ |
| SafeQL search + DAIL-SQL (GPT-OSS-120B) | Execution accuracy (EX, %) | 91.30 | Aug 10, 2026 | link ↗ |
Spider 1.0
official leaderboard ↗| system | metric | value | reported | source |
|---|---|---|---|---|
| FROST + Qwen2.5-Coder-0.5B-Instruct | Execution accuracy (dev split) | 65.70 | Sep 24, 2026 | link ↗ |
Spider 2.0
Successor to Yale's Spider, built on real enterprise workflows: databases on Snowflake/BigQuery with 1,000+ column schemas, multiple dialects, and multi-step agentic tasks. Frontier models scored under 20% at launch — the current reality check for the field.
official leaderboard ↗| system | metric | value | reported | source |
|---|---|---|---|---|
| Genloop's Sentinel Agent v2 Pro | Score | 96.70 | Aug 11, 2026 | link ↗ |
| Native mini | Score | 96.53 | Aug 11, 2026 | link ↗ |
| QUVI-3 + Gemini-3-pro-preview | Score | 94.15 | Aug 11, 2026 | link ↗ |
| MDB-Link (Qwen2.5-14B) | Spider2-Snow exact match (%) | 9.17 | Aug 10, 2026 | link ↗ |
| MDB-Link (Qwen2.5-14B) | EM (%) | 9.17 | Aug 10, 2026 | link ↗ |
| APEX-SQL | Official leaderboard score (current evaluator) | 73.13 | Feb 28, 2026 | link ↗ |
| o1-preview agentic baseline (paper) | task success rate | 17.00 | Nov 1, 2024 | link ↗ |
Spider 2.0-AIFunc
official leaderboard ↗| system | metric | value | reported | source |
|---|---|---|---|---|
| Claude Opus 4.6 + minimal Spider-Agent | Execution accuracy, full 465-instance benchmark | 70.30 | Jul 7, 2026 | link ↗ |
| Claude Sonnet 4.6 + minimal Spider-Agent | Execution accuracy, full 465-instance benchmark | 69.00 | Jul 7, 2026 | link ↗ |
| Kimi K2.5 + minimal Spider-Agent | Execution accuracy, full 465-instance benchmark | 58.10 | Jul 7, 2026 | link ↗ |
Spider 2.0-DBT
official leaderboard ↗Spider 2.0-Lite
official leaderboard ↗| system | metric | value | reported | source |
|---|---|---|---|---|
| Tianqiong Data Agent + GLM 5.2 | Official leaderboard score (%) — 547-task Lite split | 76.23 | Aug 21, 2026 | link ↗ |
| Tianqiong Data Agent + GLM 5.2 | Official score (547-example Lite split) | 76.23 | Aug 20, 2026 | link ↗ |
| Tianqiong Data Agent + GLM 5.2 | Execution accuracy (official leaderboard, 547-example Lite split) | 76.23 | Aug 16, 2026 | link ↗ |
| Tianqiong Data Agent + GLM 5.2 | Score | 76.23 | Jul 28, 2026 | link ↗ |
| Tianqiong Data Agent + GLM 5.2 | Official leaderboard score (547-example Lite track) | 76.23 | Jul 28, 2026 | link ↗ |
| Tianqiong Data Agent + GLM 5.2 | execution success rate (%) | 76.23 | Jul 28, 2026 | link ↗ |
| Tianqiong Data Agent + GLM 5.2 | official score, 547-example Spider 2.0-Lite set | 76.23 | Jul 28, 2026 | link ↗ |
| Tianqiong Data Agent + GLM 5.2 | Official score (547-example Lite set) | 76.23 | Jul 28, 2026 | link ↗ |
| DecisionX Agent | official score, 547-example Spider 2.0-Lite set | 74.95 | Jul 28, 2026 | link ↗ |
| DecisionX Agent | Official score (547-example Lite set) | 74.95 | Jul 28, 2026 | link ↗ |
| DecisionX Agent | Score | 74.95 | Jul 28, 2026 | link ↗ |
Spider 2.0-Snow
official leaderboard ↗| system | metric | value | reported | source |
|---|---|---|---|---|
| Genloop Sentinel Agent v2 Pro | Success rate (%) | 96.70 | Oct 16, 2026 | link ↗ |
| Double Ratchet | reference-loop lift retained (%) on screened held-out set | 110.00 | Oct 8, 2026 | link ↗ |
| Genloop's Sentinel Agent v2 Pro | Official leaderboard score (%) — 547-task Snow split | 96.70 | Aug 21, 2026 | link ↗ |
| Genloop's Sentinel Agent v2 Pro | Official score (547-example Snow split) | 96.70 | Aug 20, 2026 | link ↗ |
| Genloop's Sentinel Agent v2 Pro | Execution accuracy (official leaderboard, 547-example Snow split) | 96.70 | Aug 16, 2026 | link ↗ |
| Genloop's Sentinel Agent v2 Pro | Official leaderboard score (547-example Snow track) | 96.70 | Mar 1, 2026 | link ↗ |
| Genloop's Sentinel Agent v2 Pro | official score, 547-example Spider 2.0-Snow set | 96.70 | Mar 1, 2026 | link ↗ |
| Genloop's Sentinel Agent v2 Pro | execution success rate (%) | 96.70 | Mar 1, 2026 | link ↗ |
Spider 2.0-SQLite
official leaderboard ↗Spider2-SQLite
official leaderboard ↗| system | metric | value | reported | source |
|---|---|---|---|---|
| AttnLink-S | Schema-linking mAP | 83.29 | Aug 1, 2026 | link ↗ |
Spider Dev
official leaderboard ↗| system | metric | value | reported | source |
|---|---|---|---|---|
| SPOC-SQL (DeepSeek-V3, full simulated intervention) | execution accuracy (dev split) | 95.60 | Aug 24, 2026 | link ↗ |
Spider development annotation audit
official leaderboard ↗| system | metric | value | reported | source |
|---|---|---|---|---|
| SALUS Frontier calibrated estimate | estimated annotation error rate (%) | 27.00 | Oct 6, 2026 | link ↗ |
Spider-Realistic
official leaderboard ↗| system | metric | value | reported | source |
|---|---|---|---|---|
| XiYan-32B conformal abstention certificate | Expert risk @ nominal 0.10, all 508 questions, seed 101 | 20.00 | Sep 23, 2026 | link ↗ |
| Kwai-32B conformal abstention certificate | Expert risk @ nominal 0.10, all 508 questions, seed 101 | 17.20 | Sep 23, 2026 | link ↗ |
| SPOC-SQL (DeepSeek-V3, full simulated intervention) | execution accuracy | 93.10 | Aug 24, 2026 | link ↗ |
Spider-Syn
official leaderboard ↗SRL Engine Performance
official leaderboard ↗Synthetic Hospital
official leaderboard ↗| system | metric | value | reported | source |
|---|---|---|---|---|
| Llama 4 Scout self-retrieving agent | 100-patient public subset diagnosis severity-weighted F1 | 0.89 | Sep 24, 2026 | link ↗ |
| GPT 5.3 self-retrieving agent | 100-patient public subset diagnosis severity-weighted F1 | 0.84 | Sep 24, 2026 | link ↗ |
| GPT 5.3 | Public split evidence retrieval Precision@5 | 0.83 | Sep 24, 2026 | link ↗ |
| Kimi 2.5-thinking | Public split patient diagnosis severity-weighted F1 | 0.73 | Sep 24, 2026 | link ↗ |
| Opus 4.6 | Public split summarization finding-level F1 | 0.55 | Sep 24, 2026 | link ↗ |
TabArena v0.1
official leaderboard ↗| system | metric | value | reported | source |
|---|---|---|---|---|
| RefineICL-L24 benchmark-informed continuation | Elo (fixed 38-dataset fold-0 snapshot) | 1644.80 | Sep 23, 2026 | link ↗ |
Tabular Attention Throughput
official leaderboard ↗Tabular JEPA real-data suite
official leaderboard ↗| system | metric | value | reported | source |
|---|---|---|---|---|
| Value-only arm | overall classification accuracy (147-suite classification split) | 81.50 | Sep 22, 2026 | link ↗ |
| JEPA arm | overall classification accuracy (147-suite classification split) | 80.90 | Sep 22, 2026 | link ↗ |
| Value-only arm | overall regression R2 (147-suite regression split) | 0.67 | Sep 22, 2026 | link ↗ |
| JEPA arm | overall regression R2 (147-suite regression split) | 0.64 | Sep 22, 2026 | link ↗ |
TEAR Conversation
official leaderboard ↗| system | metric | value | reported | source |
|---|---|---|---|---|
| TEAR (Llama-3-70B-Instruct) | structured-F1 (EM; full schema) | 84.60 | Sep 14, 2026 | link ↗ |
TEAR Incidents
official leaderboard ↗| system | metric | value | reported | source |
|---|---|---|---|---|
| TEAR (Llama-3-70B-Instruct) | structured-F1 (EM; full schema) | 73.30 | Sep 14, 2026 | link ↗ |
TEAR Weather
official leaderboard ↗| system | metric | value | reported | source |
|---|---|---|---|---|
| TEAR (Llama-3-70B-Instruct) | structured-F1 (EM; full schema) | 56.30 | Sep 14, 2026 | link ↗ |
Thinkingbox-bench
official leaderboard ↗TIDE-Bench
official leaderboard ↗ToxicBench
official leaderboard ↗| system | metric | value | reported | source |
|---|---|---|---|---|
| GPT-5.4-mini + smolagents | Poisoned TSR drop (118-task single-table set, percentage points) | 39.00 | Sep 29, 2026 | link ↗ |
| GPT-5.4-mini + LangGraph ReAct | Poisoned TSR drop (118-task single-table set, percentage points) | 32.00 | Sep 29, 2026 | link ↗ |
| GPT-5.4-mini + AutoGen | Poisoned TSR drop (118-task single-table set, percentage points) | 26.00 | Sep 29, 2026 | link ↗ |
TPC-H
official leaderboard ↗| system | metric | value | reported | source |
|---|---|---|---|---|
| GaussDB on 32 Kunpeng bare-metal servers | Composite QphH@30TB, millions (unaudited) | 39.51 | Aug 28, 2026 | link ↗ |
TQTS-Bench
official leaderboard ↗WhatWorkedBench
official leaderboard ↗| system | metric | value | reported | source |
|---|---|---|---|---|
| Shared GP on Flash observations, additional cohort, B=8 | family-macro effect recovery; 6 matched four-factor sources | 0.72 | Sep 23, 2026 | link ↗ |
| Shared GP on Flash observations, original cohort, B=8 | family-macro effect recovery; 6 matched four-factor sources | 0.70 | Sep 23, 2026 | link ↗ |
| Flash agent table, original cohort, B=8 | family-macro effect recovery; 6 matched four-factor sources | 0.63 | Sep 23, 2026 | link ↗ |
| Flash agent table, additional cohort, B=8 | family-macro effect recovery; 6 matched four-factor sources | 0.62 | Sep 23, 2026 | link ↗ |
WikiTableQuestions
official leaderboard ↗| system | metric | value | reported | source |
|---|---|---|---|---|
| TarQA-OTSL (Meta-Llama-3-8B-Instruct) | ANLS — WTQ test | 56.50 | Sep 15, 2026 | link ↗ |
YCSB on ChainMaker execution layer
official leaderboard ↗| system | metric | value | reported | source |
|---|---|---|---|---|
| Lantern | Throughput txn/s (Zipf skew 0.9, 32-core server) | 6131.00 | Sep 4, 2026 | link ↗ |
ZigZag Trie contextual-query evaluation
official leaderboard ↗| system | metric | value | reported | source |
|---|---|---|---|---|
| CCZT | Index-size reduction vs CCBA, mean across n sweep on BST (%) | 46.00 | Sep 17, 2026 | link ↗ |
| TCPRZT | Index-size overhead vs TCPRBA on CHR (%) | 16.00 | Sep 17, 2026 | link ↗ |
| LFCSZT | LFCS query speedup vs LFCSBA, mean across m=3,6,9,12,15 on SDSL | 12.00 | Sep 17, 2026 | link ↗ |
| LFCSZT | LFCS query speedup vs LFCSBA, mean across tau=50,100,200,400,800 on SDSL | 8.00 | Sep 17, 2026 | link ↗ |