KathDB-FAO cuts multimodal query cost—but its 58.8% headline needs careful reading

KathDB-FAO’s central idea is unusually concrete: turn a natural-language multimodal query into a plan whose operators are synthesized for that query, rather than repeatedly invoking generic model-powered operators. Its paper reports 58.8% lower average execution cost than the next-best system at comparable or better quality across its evaluated SemBench scenarios. The underlying tables support a cost-quality advantage, but not a simple claim that KathDB-FAO is always the cheapest system in every row. (paper)

What the evaluation actually covers

The authors evaluate three original SemBench scenarios—Movie, E-Com and MMQA—plus an “Additional” set of ten queries drawn from E-Com and MMQA at larger scale factors. Results are averaged over three runs per query. MMQA excludes Q5, E-Com excludes Q14, and the BigQuery E-Com average also excludes Q10 and Q11; those boundaries matter when comparing the aggregate numbers. (paper)

On MMQA at scale factor 200, KathDB-FAO scores 0.85 quality with $0.031 execution cost. The next-highest quality in the table is Palimpzest at 0.78, costing $7.161; BigQuery scores 0.32 at $0.142. On Movie at scale factor 2,000, KathDB-FAO and ClaudeCode both score 0.81, while their reported execution costs are $0.004 and $1.402, respectively. (paper)

The picture is less absolute on E-Com at scale factor 500. ClaudeCode+ reaches 0.79 quality, slightly above KathDB-FAO’s 0.77, but costs $0.176 to execute versus $0.079. CAESURA+ is cheaper than KathDB-FAO on Movie—$0.003 versus $0.004—but its quality is 0.44, far below KathDB-FAO’s 0.81. That is why the paper’s claim depends on “comparable or better quality,” rather than simply selecting the lowest-cost cell. (paper)

The comparison is not fully apples-to-apples

KathDB-FAO uses Claude Opus 4.7 for plan generation and gpt-4o-mini for AI operators. BigQuery’s inherited SemBench results use Gemini 2.5 Flash, which the authors say had higher per-token pricing, and BigQuery was not rerun in the same environment. Palimpzest and BigQuery start from expert-written plans, so their tables omit the human planning cost; ClaudeCode combines planning and execution, making its planning cost inseparable. KathDB-FAO itself reports separate planning charges ranging from $0.34 to $0.73 across the four scenarios, which should not be confused with its much smaller per-run execution costs. (paper)

The defensible result is therefore narrower than the headline: on these SemBench workloads, under the paper’s model and pricing setup, synthesized query-specific operators buy substantial execution-cost savings without a broad quality collapse. Whether that advantage survives different model prices, excluded queries and repeated-plan amortization remains an open benchmark question—not a weakness to hide, but the next test the result now makes worth running.