MotherDuck’s local SQL agent is cheap and accurate—but each answer takes minutes

The result

MotherDuck says a locally run, 4-bit Qwen3.8 27B model reached 98.6% accuracy in its agentic-SQL evaluation while running on an M5 MacBook Air with 32GB of memory. A more compressed 3-bit run on a 16GB M1 Pro reached 96.4%. The vendor’s write-up says the full test covers more than 400 questions after a 26-question training-and-improvement loop. MotherDuck’s report

The cost claim is attention-grabbing: MotherDuck estimates the local run at less than $0.50 of electricity, compared with more than $8 for GPT-5.6 Luna Max in its comparison. That is a vendor-run estimate, not an independent benchmark, and it excludes the laptop’s purchase price. MotherDuck separately estimates that allocating laptop depreciation would lift the cost to roughly $6 per 1,000 questions. MotherDuck’s report

The latency boundary

The tradeoff is time. MotherDuck reports that local Qwen took five to six minutes per question on average, versus about 25 seconds for Gemini-3-Flash and 40 seconds for GPT-5.6 Luna. The local setup also handled only one prompt at a time, while the hosted-model run used concurrency of 15. Those constraints make the result more compelling for overnight evaluation, background analysis and cost-capped experimentation than for an interactive analytics assistant. MotherDuck’s report

Hardware matters too. The reported 4-bit result used 32GB of unified memory; the 16GB machine needed a more aggressive 3-bit quantization and lost accuracy. MotherDuck measured about five to seven generated tokens per second on its test laptops. MotherDuck’s report

What teams can reproduce

The evaluation is more inspectable than a chart alone. MotherDuck links a public harness that uses an agent skill plus a progressively loaded semantic layer, then queries data through the same MCP server. The repository documents the 26 template questions, a 419-question cleaned held-out set, strict scoring, configurable models and commands for running the evaluation. It also states that five disputed gold answers are excluded from the cleaned test set. MotherDuck Labs repository

One important caveat: the repository’s current README highlights a 417/419 Gemini-3-Flash result; the Qwen figures are presented in the MotherDuck article rather than as the README’s headline reproducible run. Practitioners should therefore rerun the harness on their own hardware before treating 98.6% or the electricity estimate as portable. MotherDuck Labs repository

The practical conclusion is narrower than “cloud models are obsolete.” A local model can now be good enough for serial, background SQL work when cost containment and data locality matter more than response time. Interactive workloads still favor hosted models—or materially faster local hardware—because minutes per answer changes the user experience completely.