DABStep ablation separates semantic content from retrieval scaffolding

A new four-arm ablation on DABStep asks a question that most context-layer demos leave unresolved: does an agent improve because it receives better semantic content, or because retrieval tools and instructions make it work more carefully? Across four model families, the paper’s answer is that both can help, but the semantic content contributes more on the hard split.

Four arms, one scorer

The experiment compares a bare schema, the vendor manual pasted into the prompt, a governed tool layer backed by a deliberately hollow data contract, and the same tools backed by the full frozen contract. The authors report all arms with DABStep’s scorer and the same tasks within each model run. Their released repository includes per-row results plus 6,416 compressed transcripts, split into four 1,604-file archives with published SHA-256 digests.

On hard tasks, the full-contract arm scored 55.1% with GLM-5.3-Flash, 56.6% with DeepSeek-V4-Flash, 68.4% with Claude Sonnet 5 and 77.4% with GPT-5.6-sol. The hollow-contract control scored 19.3%, 22.6%, 38.0% and 51.2%, respectively. That leaves a content contribution of 35.8, 34.0, 30.4 and 26.2 percentage points—larger than the corresponding scaffolding step on every model.

The scaffolding result is not zero, however. Relative to a bare schema, the hollow contract added 14.2 points on GPT-5.6-sol and 15.1 points on Sonnet 5, while adding 5.4 points on GLM and none on DeepSeek. This is a useful correction to the simpler claim that tools are irrelevant: the released analysis says tooling was significant on the two frontier-model runs, but still did not explain most of the full contract’s advantage.

The important limits

These are not four runs on the official hidden test golds. The main tables use reconstructed answers for 401 of DABStep’s 450 tasks; the hard splits contain 332 tasks for GLM, Sonnet and GPT, while provider rate limits reduced DeepSeek to a 226-hard-task complete-case set. The paper says its reconstructed hard scoring is optimistic, and reports an official leaderboard check for the GLM contract arm at 51.9% on 378 hard tasks, versus 55.1% on the reconstructed 332-task hard set.

The domain mix is another constraint. The authors report that 294 of those 332 reconstructed hard tasks mention fees, so the experiment strongly tests whether structured fee semantics beat prose delivery. It does not establish the same effect across unrelated enterprise schemas.

Still, this is unusually auditable evidence for a context-layer claim. The frozen and hollow contracts, scorer revision, result rows and transcript archives are public. The practical takeaway is narrower than “semantic layers solve text-to-SQL,” but more defensible: on this DABStep setup, agents benefited most when domain rules were available as structured, retrievable content, and stronger models did not make that content unnecessary.