This text-to-SQL prototype gates releases on result sets—not fluent answers

A text-to-SQL system can produce a convincing explanation while querying the wrong rows. Satwiki De, who identifies himself as a senior consultant at Microsoft, has published a personal open-source prototype that makes that mismatch a release-blocking failure rather than a secondary quality signal.

The important choice is the order of evaluation. The project’s primary gate is SQL execution accuracy: run the reference SQL and the agent’s last successful generated statement against the same database, normalize the outputs, and compare the result sets. Answer correctness, schema adherence and contextual relevance are additional metrics, but polished prose does not rescue a query that returns different data.

Compare behavior, not query strings

The accompanying write-up explains why literal SQL matching is too brittle. Different queries can produce the same correct result, while a syntactically valid query can still answer the wrong business question. Its evaluator therefore compares outputs rather than requiring the generated SQL to match one canonical string.

The comparison is more specific than a generic “execution accuracy” label. Column names are lowercased, nulls are handled consistently and numeric values use a common representation with an absolute tolerance of 0.01. Row order is enforced when the reference query contains ORDER BY; otherwise the evaluator compares order-independent collections while preserving duplicates.

That last distinction matters in analytical workloads. Treating unordered rows as ordered creates false failures, while collapsing duplicates can hide a wrong join or aggregation.

The gate is wired into the runner

The repository’s evaluation runner supports a deterministic --skip-llm mode that omits judge-based metrics and runs execution checks without an LLM evaluator. It builds an isolated SQLite fixture from the ground-truth snapshot unless an operator deliberately supplies another database. The process returns a failure exit code when any sample fails SQL Execution Accuracy, making the metric usable in CI rather than leaving it as a dashboard observation.

Each golden sample can include the question, schema context, reference SQL, stored reference result, expected answer and supporting claims. The runner records generated SQL and query history as metadata, which gives a failed release gate an inspectable trail.

What this does—and does not—prove

This is a local e-commerce prototype, not evidence of production accuracy at enterprise scale. The author explicitly says a hosted or on-premises deployment may need a frozen database snapshot so generated and reference queries execute against the same state. The article reports no aggregate score that can be compared with BIRD, Spider or a vendor benchmark.

But the evaluation contract is useful precisely because it is modest. It separates three questions that teams often blur together: did the SQL execute, did it return the same analytical result, and did the final narrative faithfully communicate that result? For a release pipeline, the first two can fail deterministically before a subjective judge ever sees the prose.