Do not collapse text-to-SQL accuracy into one number

A text-to-SQL system does not have one kind of accuracy. Querio’s evaluation guide separates five measurements—exact match, execution accuracy, semantic testing, functional correctness and speed—and the useful lesson is not that teams must adopt those labels verbatim. It is that each measurement rejects a different failure.

Syntax is not an answer

Exact match compares generated SQL with a reference query character by character. That makes it easy to automate for regression testing, but the guide notes that equivalent joins, aliases or query structures can fail even when they return the same result. Exact match is therefore a stability check, not sufficient evidence that the answer is right.

Execution accuracy moves the comparison to returned rows. This accepts different SQL formulations when they produce the same result set. But Querio also lists the traps: incomplete test data can omit nulls, duplicates or unusual types; changing database state can make comparisons inconsistent; and formatting or floating-point differences can create false failures. A passing execution test also says nothing about whether the query is unnecessarily expensive.

Test meaning and operating behavior separately

Semantic suites test whether the system preserves intent across paraphrases, ambiguity and increasing query complexity. The guide recommends grouping related questions rather than relying on isolated examples, so evaluators can see where a system begins to fail—for example when a simple customer filter becomes a comparison against an annual average.

Functional checks ask whether the query performed the intended operations even when its structure differs from the reference. Speed metrics then add an operational constraint: a correct query can still be unsuitable if it is too slow on realistic data volumes.

That produces a practical review stack:

1. Use exact match when reproducibility of a known query form matters.
2. Compare result sets for semantic equivalence.
3. Build suites around business intent, paraphrases and edge cases.
4. Inspect required operations, not only output on one fixture.
5. Measure latency on representative data.

The guide does not publish a benchmark, thresholds or measured results for Querio itself. Its value is methodological: it makes clear why a single “accuracy” percentage is incomplete unless the vendor says what was compared, on which data, with which edge cases and under what performance limit. Buyers should ask for that decomposition before treating any headline score as production evidence.