A successful tool call can still cut a data agent’s accuracy by 39 points

A tool call that returns without an error is not necessarily trustworthy evidence. ToxicBench tests that distinction by keeping source tables fixed while a proxy changes otherwise successful tool responses. Across a retained 118-task single-table set, poisoning reduced task success by 32 percentage points for GPT-5.4-mini with LangGraph ReAct, 39 points with smolagents and 26 points with AutoGen.

The adapter changes the failure rate

Those three runs used the same model but produced materially different outcomes. LangGraph fell from 0.99 clean task success to 0.67 when poisoned; smolagents fell from 0.98 to 0.59; AutoGen fell from 0.92 to 0.66. The benchmark’s blind-compliance rate was also adapter-dependent: 0.38, 0.53 and 0.27 respectively. The paper’s Table 1 therefore measures more than model quality: orchestration determines whether plausible bad evidence becomes the final answer.

ToxicBench’s primary expanded comparison contains 58 numerical and 60 semantic/schema tasks after two ambiguous numerical prompts were removed. Its poison operators include scaling aggregates, flipping signs, swapping rank labels, changing schema semantics and returning biased retrieval results. The released repository includes task definitions, fixtures, selected trajectories, scoring code and reference and delivery audits.

Retrying works—until the retry is poisoned

On the same 118 tasks, a separate Claude Haiku 4.5 experiment found that an ordinary second pass raised poisoned task success from 0.89 for the base route to 1.00. The gain was 0.110, with a template-cluster bootstrap 95% confidence interval of 0.056 to 0.171. An explicit verification-only route scored 0.90, while a generic guard scored 0.97; the authors did not find a clear advantage for explicit verification over an ordinary retry in that one-shot condition. Each two-route method had a 10-step limit per route and a 3,072-output-token cap per model request, but realized compute was not matched.

The control experiment explains the result. Across 24 tasks on three public tables, second-stage retries were correct in 48 of 48 runs when given fresh clean evidence and zero of 48 when all target aliases remained corrupted. Reviews showed the same boundary: 47 of 48 with clean evidence and zero of 48 under full-target corruption. In other words, asking again is useful when it creates an independent observation—not merely a second opinion over the same poisoned channel.

Repeated poisoning makes that distinction operational. Across 1,572 retained trajectories, the scorer found 101 cases where the agent adopted the poisoned answer despite a qualifying validation event. Seventy-eight of those follow-up checks were themselves corrupted; 23 included an unmodified check. The result supports a narrower conclusion than “agents ignore verification”: checking cannot recover truth when the evidence path stays compromised.

The benchmark is still bounded. Its core uses small synthetic tables, and the authors say generalization to production workflows and naturally occurring errors remains untested. But its scoring received 96% task-success agreement on a post-freeze human audit of 200 trajectories, giving the central result a useful audit trail: tool reliability has to be measured at the evidence layer, not inferred from a successful API response.