Domo’s text-to-SQL checklist turns a flashy demo into a five-part production test

Domo published a new comparison of nine text-to-SQL products on October 15, but the useful part is not the vendor ranking. It is the evaluation procedure buried beneath it: a compact set of tests that an enterprise can run before allowing a conversational interface near production data.

The source is plainly commercial. Domo includes itself among the nine products and presents its own governed, agentic platform first. That conflict should make readers skeptical of comparative judgments about competitors. It does not invalidate the operational tests, which are specific enough to reproduce during a pilot.

Start with two identities, not one demo user

Domo’s strongest recommendation is to test the same question with two accounts that have different row-level policies. The expected result is not merely a correct SQL statement; each account must see only the rows it is authorized to access. The guide tells buyers to determine whether the tool passes the requester’s identity through to the warehouse or runs queries through a generic service account.

That distinction changes the meaning of a successful demo. A service account can produce the right aggregate while quietly bypassing the requester’s row or column restrictions. Identity continuity therefore belongs in the acceptance test, not in a post-pilot security review.

Try to make the assistant write

The guide also recommends explicitly prompting the system to generate a write operation and verifying that it refuses. Teams should pair that adversarial check with known-answer questions and with inefficient-query tests. A syntactically correct answer can still be unsafe if it issues a damaging statement or scans enough data to create an unexpected warehouse bill.

This is a better bar than headline accuracy. It separates three failure classes that require different controls: wrong answers, unauthorized operations and economically harmful queries.

Test the context layer

Domo advises asking a question that depends on a company-specific metric and checking whether the generated query uses the approved definition from a semantic layer or catalog. The guide also recommends messy schemas and ambiguous questions rather than a curated demonstration dataset.

That test exposes whether the product can retrieve shared business meaning or is only matching natural-language terms to table and column names. A pilot should include a metric whose definition changed, a synonym that maps to more than one field and a question that should trigger clarification rather than SQL.

Measure the whole request path

Finally, the guide calls for concurrent-request tests, cache checks and monitoring of inference plus warehouse-compute costs. Those measurements should be recorded together. A cheap model can still drive an expensive query, while a fast cached answer can conceal the latency of a first-time request.

The practical takeaway is a five-part gate: identity isolation, write protection, known-answer accuracy, semantic grounding and end-to-end cost/performance. Domo’s comparison is marketing material, not an independent benchmark. Its checklist is still a defensible starting point because each item can be turned into a pass/fail test against an organization’s own data and policies.