Argo-Bench’s reference suite cannot fully replay under its published BigQuery scan cap

Argo-Bench has added something most agent benchmarks lack: executable reference solutions that pass through the same SQL and Python tools as a model run. The new suite covers 20 of the benchmark’s 210 questions, plus a no-data smoke test. But the release also documents a useful constraint failure: one reference task cannot be replayed on BigQuery under the benchmark’s published per-query scan limit.

The exception is explicit

The reference-suite documentation says every reference statement remains below the 20 GiB BigQuery scan cap except three statements in ds-22-margin-monthly-per-order. Those queries read a full year of books, so the maintainers direct users to replay that task on DuckDB instead.

That matters because Argo-Bench’s main harness specification says model runs receive a 20 GiB cap on each BigQuery query. The released reference is therefore not a complete proof that all 20 tasks are solvable through the exact BigQuery resource envelope imposed on evaluated agents. It is a proof for 19 reference-covered tasks under that envelope, plus a DuckDB replay path for the exception.

The scores expose a second useful boundary

The same reference table reports perfect 1.00 scores on both the paper warehouse and the released warehouse for 17 of the 20 tasks. Three are lower on at least one build: col-10-refund-partnerships scores 0.98 on the paper warehouse and 0.94 on the released one; fc-12-true-ups-may scores 0.94 and 0.87; ds-22-margin-monthly-per-order scores 1.00 on both.

Those figures do not invalidate the benchmark. They make its limits inspectable. The repository says the released 71 GiB warehouse is a sibling build generated with a different random seed, while entity-specific prompts were redrawn using each question’s selection rule. A reference procedure can therefore be structurally correct without receiving a perfect grade on both data builds.

What submissions should disclose

A reproduction should now identify both its engine and whether it used the benchmark’s 20 GiB BigQuery ceiling. A DuckDB reference replay and a capped BigQuery model run are not identical execution conditions, even when they target the same question and grading key.

The larger lesson is methodological: an executable reference suite is more valuable when it reveals where the benchmark’s own resource contract bends. Argo-Bench has surfaced that exception rather than hiding it. The next useful step would be a capped BigQuery reference strategy for ds-22, or an explicit task-level exemption in reported results.