MotherDuck says a schema change lifted hard-query accuracy sixfold—but the landing page is not enough to reproduce it

MotherDuck has published an AI-analytics evaluation guide built around a striking controlled experiment: it says the same question set and agent were rerun after one schema change, with the improvement holding across four model families without model-specific retuning. On the hard-question slice, the company reports Gemini 3 Flash rising from 7.2% accuracy to 44.4%, while Claude Opus 4.7 rose from 31.2% to 59.9%. MotherDuck

Those numbers make a practical point that procurement scorecards often miss. A team can spend weeks comparing models while leaving the schema—the interface the model must reason over—unchanged. MotherDuck’s stated intervention was column naming, and its page says the Flash result improved at lower cost. That supports testing a data-design change before concluding that the model is the bottleneck. MotherDuck

The experiment is promising, not yet independently auditable

The public landing page calls the workload a “public benchmark,” but it does not identify the benchmark in the visible methodology, enumerate the hard-question sample, publish prompts or link to run artifacts. It also says the Flash and Opus outcomes were respectively 3.1× and 4.1× the paper’s best published hard result without naming that paper on the page. Those omissions prevent a reader from checking whether the baseline, schema intervention and comparison result used identical execution conditions. MotherDuck

That does not make the finding useless. It makes the right takeaway narrower: schema naming is a high-leverage variable worth isolating in an internal evaluation, not a guaranteed sixfold improvement for another warehouse.

Copy the test design, not the headline number

MotherDuck’s seven-step playbook is more transferable than its model ranking. It recommends starting with 20–30 business questions, separating training and held-out cases, stabilizing the data, logging accuracy, speed and cost for every question, changing one variable at a time, iterating on failures only against the training set, and scoring the held-out set once. It then adds production telemetry so failures can become new evaluation cases. MotherDuck

The page also identifies three confounders worth putting directly into an experiment plan: batching several changes together, allowing context to become stale relative to the schema, and comparing runs across model-provider snapshot changes. Its remedy is simple: pin the data and execution setup, move one lever, and preserve the held-out set alongside the harness. MotherDuck

For practitioners, the actionable test is not “Does our preferred model reach 44.4%?” It is “When the model, questions and execution budget stay fixed, does a specific schema change improve held-out accuracy without making latency or cost worse?” MotherDuck’s result gives teams a reason to run that experiment. Reproducible artifacts would be needed before its exact uplift should be treated as a benchmark.