Argo-Bench exposes the gap between writing SQL and making the right business decision

The headline result from Argo-Bench is not that Claude Opus 5.5 leads. It is how far the leader remains from reliable enterprise work: at extra-high reasoning effort, Opus averages 59.5/100 and reaches the benchmark’s “solved” threshold of 95 points on 34.8% of 210 tasks. The next three systems—GPT-6 Astra, Claude Sonnet 5.5 and GPT-6.1 Sol—trail by 7.7 to 10.0 mean-score points, while their own intervals overlap.

That result lands against a very different picture on established SQL leaderboards. The live Spider 2.0 board puts Genloop Sentinel Agent v2 Pro at 96.70, while BIRD lists GrainSQL at 82.95% test execution accuracy. Those numbers are not comparable to Argo-Bench’s score—the tasks, graders and scales differ—but the contrast explains why the new evaluation matters. A system can look close to saturation when the endpoint is a result table and still fail when it must choose an economically correct action.

Argo-Bench changes both the environment and the target. Its simulated New York food-delivery business is exported as an Oracle E-Business Suite-style warehouse with 235 tables, 7.49 billion rows and 81 million orders. Agents receive read-only SQL access plus a Python environment, then file outcomes such as account bans, forecasts, journal entries, budget allocations and dashboard data sources. The grader uses latent simulator state to score the consequences rather than comparing one generated query with a gold SQL string.

The failure analysis is more useful than the ranking. In one budget-cutting family, GPT-6 Astra correctly estimates where courier bonuses buy the fewest additional hours, yet its plan loses $86,281 because it optimizes the wrong objective. Opus instead models how bonuses substitute for surge pay and saves $3.09 million out of an attainable $3.12 million, scoring 99. Across 4,553 forecast series, nominal 80% intervals cover the realized value only 44.8% of the time. And 66.9% of dashboard data sources that satisfy their structural contract still score zero on values.

There is an important reproducibility boundary. The repository releases the 210 questions, harness, model configurations, 20 reference solutions and a 71 GiB sibling warehouse build. But the paper’s model scores use a separate private-seed world with held-out answer keys; local runs can confirm that agents file correctly, not reproduce official scores without maintainer grading. The paper also reports one run per task and one shared simulated world, so its confidence intervals capture task choice rather than run-to-run model variance.

The practical lesson is precise: SQL execution accuracy measures whether a system can retrieve an answer under a benchmark’s contract. Argo-Bench asks whether the system identified the right record, quantity and objective before acting. The leading score says that second problem is nowhere near solved.