WhatWorkedBench asks whether research agents understand why experiments work

A benchmark for what an experiment teaches

A research agent can find the best configuration without learning why it works. WhatWorkedBench, a new executable benchmark released with code and recorded evidence, is designed to measure that difference.

The benchmark gives an agent a workflow with four or six binary component choices, two free anchor measurements, and a fixed budget for additional experiments. The agent must then predict the utility of every legal configuration — 16 cells for four choices or 64 for six. Exhaustive CPU execution supplies the reference table, letting the evaluator score not only the chosen optimum but every conditional component effect.

That distinction matters because an option can help in one background and hurt in another. The paper reports at least one sign reversal in 35 of 36 task conditions; 24 still contain a reversal when effects in both directions must exceed 5% of the task’s response range. A single average effect or winning configuration hides those interactions.

Optimization is not reconstruction

The public release spans 36 task conditions drawn from 30 sources and eight workflow families, including classification, regression, clustering, forecasting, image restoration, retrieval, beat detection and graph link prediction. Its 1,248 native outcomes yield 3,392 conditional effects and 342 mean pair interactions. The reported study adds 4,206 numerical-control records and 108 core agent episodes.

On 22 four-factor sources with a budget of eight new measurements, the paper’s pair-effect ridge method selected an exact optimum in 15 cases but met the strict reconstruction criterion in only three. Its family-macro effect-recovery score was 0.612. An effect-variance Gaussian process reached 0.701 recovery and 16 exact choices, yet strictly reconstructed only one source. The result is not a clean win for either method: optimization, average recovery and worst-case effect error rank systems differently.

Agent runs show another gap between collecting evidence and using it. Applying a shared Gaussian-process estimator to the same observations raised recovery from 0.632 to 0.698 in the original Flash cohort and from 0.621 to 0.720 in an additional cohort. Because the measurements are held fixed, those gains come from reconstruction rather than extra experiments.

Code knowledge is useful only when enforced

The strongest diagnostic concerns program structure. Across six paired six-option workflows at a 20-measurement budget, encoding exact code equivalences raised Gaussian-process recovery from 0.248 to 0.462. In eight diagnostic episodes, agents identified all six registered rules but still submitted tables that violated them. Projecting the submitted tables onto the valid equivalence classes after the fact raised mean recovery from 0.338 to 0.507.

WhatWorkedBench therefore tests more than whether an agent can run experiments or report a high score. It asks whether the final artifact captures the intervention landscape those experiments were meant to reveal. The Apache-2.0 repository includes public tasks, a standalone evaluator, numerical baselines, native replay code and recorded result artifacts; its maintainers say the prepared-input pack can replay all 1,248 native records outside the repository.