A text-to-MQL benchmark found the conversion pipeline can corrupt the benchmark

A new text-to-MQL benchmark makes an uncomfortable point: the benchmark-construction pipeline can fail before the model does.

AptMQL-Bench, posted October 2, converts BIRD’s relational databases and questions into a MongoDB-native evaluation. Its authors compared three schema strategies: a one-table-to-one-collection mapping, foreign-key-driven embedding, and their access-pattern-driven design. In their reimplementation, foreign-key embedding failed to migrate six of 21 BIRD databases. Three contained self-referential foreign keys; three collapsed into documents that exceeded MongoDB’s 16 MB document limit. Eight more databases migrated only partially, with missing-parent relationships silently dropping rows. The worst case, toxicology, lost 25.9% of source rows.

That is not a model-quality problem. A generated query cannot recover records removed during benchmark creation. It also means that a leaderboard built on a mechanically converted database can reward systems against an altered task while presenting the result as a translation of the original one.

Schema design changes both integrity and cost

The authors’ alternative designs each document schema around expected access patterns, deciding relationship by relationship whether to embed or reference data. Their resulting AptMQL-Bench contains 21 databases and 3,186 natural-language requests with associated MQL queries. They report lossless migration after collection-level, cross-collection and source-anchored validation.

The tradeoff is labor. For 11 development databases, the paper estimates schema design and validation at about 27 reviewer minutes per database, ranging from roughly 13 to 42 minutes. Initial SQL-to-MQL translations were checked automatically for result equivalence, but 1,444 queries—45.32% of the set—received a manual optimization pass to remove SQL-shaped artifacts.

Executable is not the same as correct

The benchmark also separates execution success from semantic correctness. Eight models generated valid MQL almost all the time: the Claude models’ execution success was about 98.5%. Yet Claude Opus 4.5 reached only 57.38% soft execution accuracy without BIRD’s external knowledge and 70.34% with it. GPT-OSS-120B reached 62.46% with that knowledge.

“Soft” execution accuracy still requires the predicted and reference answers to have the same rows and matching values for every reference field; it merely permits additional returned fields. That matters for document databases, where returning a richer document can still answer the question. Opus scored 50.44% under strict execution accuracy versus 70.34% under the soft metric with knowledge. The authors say all 634 Opus disagreements between those metrics contained the correct answer plus extra fields, not extra rows or conflicting values.

The result is author-run, not an independent reproduction. The foreign-key comparison also relies on a reimplementation validated against three released conversions because the earlier TEND-v2 work did not publish enough material to reproduce every database. Those limits should temper the conclusion, but not erase it: benchmark builders need migration-integrity tests before they compare models. A high-quality evaluator must verify the database transformation as carefully as the generated query.