A text-to-SQL comparison needs an architecture test—and a vendor-disclosure test

Enterprise buyers often compare text-to-SQL systems as if they were interchangeable chat boxes. A new Promethium guide offers a more useful starting point: classify the architecture first. But the guide also demonstrates why procurement teams need a second control—treat the author’s category rankings as vendor claims, not neutral test results.

Five products can answer the same question in five different ways

Promethium divides the market into five categories: platform-native stacks, BI agents, catalog agents, direct schema-to-query wrappers and what it calls “AI Insights Fabrics.” The categories expose real deployment boundaries. A platform-native agent can inherit the data platform’s controls but is strongest inside that platform. A BI agent starts from already modeled metrics. A catalog agent may help a user find and understand data without executing the final analysis. A wrapper can connect quickly but leaves more context and control work to the buyer.

That taxonomy is more useful than comparing a single benchmark score because it forces the team to ask where query execution happens, which semantic definitions are available and whether a question crosses systems.

Turn the taxonomy into acceptance tests

The guide proposes four scenarios that procurement teams can convert into tests: a multi-source join, an ambiguous business term, role-based data access and consistency of a metric across differently worded questions.

Those tests should run against the buyer’s own identities, schemas and business definitions. For example, “revenue” should resolve to the approved measure for that user and stay consistent when the follow-up asks for quarter-over-quarter growth. A cross-system question should expose whether the product truly queries multiple sources or depends on a prebuilt extract. An access test should verify the returned rows, not merely whether the interface says governance is enabled.

The same discipline applies to cost. Promethium’s guide warns that generated SQL can create warehouse expense and argues that semantic modeling is an ongoing investment rather than a one-time setup. Buyers should therefore record bytes scanned, latency and retry count alongside answer correctness during evaluation.

Read the scorecard with its conflict attached

Promethium places its own product in the “AI Insights Fabric” category and scores that category favorably across federation, governance and cross-platform querying. That is a disclosed product position, not an independent bake-off. The guide does not report a hands-on, same-workload comparison of every named product.

The practical takeaway is not to accept or reject Promethium’s winner. It is to reuse the architecture categories while replacing the vendor’s table with evidence from your environment. Require every supplier to run the same questions, under the same user identities, against the same data, and preserve the generated SQL and execution metrics.

A useful text-to-SQL comparison is therefore two documents: an architecture map that explains where each system can operate, and an acceptance report that proves what it did on your workload. Without the first, products that solve different problems look equivalent. Without the second, vendor positioning looks like measured performance.