A finance text-to-SQL test found valid SQL was still wrong most often at the filter IDs

The query can run and still answer the wrong question

A new production-oriented text-to-SQL evaluation isolates a failure that execution success does not catch: systems can produce valid SQL while choosing the wrong opaque identifiers for metrics, periods or transaction types.

The FLINT paper evaluates seven established text-to-SQL approaches on two proprietary financial datasets using the same Claude 4.5 Opus model. The Financials set contains 221 questions over 24 tables and averages 6.0 joins per gold query. The Transactions set contains 138 questions over 18 tables and averages 5.2 joins. Both sets were drawn from six months of production queries and paired with expert-written SQL.

Every compared system achieved at least 99% execution success, yet none of the seven baselines exceeded 50% semantic accuracy. The paper attributes 64% to 96% of baseline failures on Financials to incorrect numeric filter IDs. That distinction matters: iterative repair can respond to a database error, but a query that executes with the wrong metric ID supplies no failure signal.

FLINT improves the score, but does not solve the task

FLINT combines a lookup agent for reference tables, an expert-authored query bank, entity resolution, schema linking and one execution-reflection attempt. It scored 68.8% strict semantic accuracy on Financials and 65.2% on Transactions. Those are separate evaluation sets and should not be averaged or compared as if they were one leaderboard split.

The ablation is more informative than the headline win. Replacing the dynamic lookup step with static mappings reduced accuracy by 14 points on Financials and 32 points on Transactions. Removing retrieved examples cost 11 and 25 points respectively. The paper reports that FLINT correctly resolved all required IDs in 94% of Financials queries but only 75% of Transactions queries.

The system still produced 117 incorrect predictions across the two sets. Column selection accounted for 71 errors, while 30 failures occurred after the right tables and columns had been selected because lookup constraints were still wrong. FLINT also requires substantial human configuration: schema documentation, domain rules and a curated bank of 15–30 examples, with the authors estimating one to three person-weeks depending on complexity.

What the result measures

This is not a public hidden-test leaderboard: the questions, production schemas and gold queries are proprietary, and semantic equivalence includes an LLM judge for roughly 40% of unresolved comparisons. The authors report at least 94% judge agreement with humans on a validation sample.

The useful finding is narrower and stronger than a universal state-of-the-art claim. On deeply normalized schemas, execution rate can be nearly perfect while answer correctness remains below 50%. A benchmark or production monitor that records only whether SQL ran will miss the dominant error class exposed here.