A clinical text-to-SQL monitor improves reliability mostly by answering fewer questions

The gain is real—but it is selective prediction, not better SQL

A new clinical text-to-SQL evaluation reports that a confidence layer called Sentinel raises GPT-4o’s Reliability Score on an EHRSQL held-out pool from +0.08 to +0.24 when a wrong answer carries the same one-point cost as a correct answer’s reward. The mechanism does not make the underlying ReAct agent generate more correct SQL. It monitors the question, reasoning trajectory, SQL and database outputs, then refuses, stops or withholds work that looks unreliable.

That distinction changes how the result should be read. GPT-4o’s delivered-answer accuracy rises from 54% to 69%, but answer coverage falls from 84% to 55%. Sentinel is improving the quality of what reaches the user by declining nearly half of the evaluation pool at that operating point, not by turning a 54%-accurate agent into a 69%-accurate agent at equal coverage.

Higher stakes collapse toward “do not answer”

The paper evaluates the EHRSQL Reliability Score at penalty weights λ=1, 4 and 9. A correct answer or correct refusal earns +1; a missed answer earns 0; a delivered mistake loses λ points. On GPT-4o, the unmonitored agent scores −3.02 at λ=9. Sentinel reaches +0.03, approximately the score obtained by refusing every question.

That is a useful negative result. At the paper’s high-stakes threshold, the tested agent is not reliable enough to deliver meaningful coverage under the stated utility function. The monitor recognizes that boundary, but the score should not be summarized as broad clinical readiness. It shows that abstention can prevent a deeply negative outcome when errors are expensive.

Across five tested agents, the same pattern is visible. At λ=1, full Sentinel scores range from +0.04 for Phi-4-mini to +0.24 for GPT-4o. At λ=9, all five systems sit at roughly +0.03 to +0.04, effectively the always-abstain neighborhood.

The split and baseline matter

These are not official EHRSQL test results. The authors construct a 2,101-question held-out pool from the official train and validation material while leaving the official test split untouched. Only 60 held-out questions are unanswerable, which makes always abstaining worth about +0.03.

The strongest low-stakes gains clear that baseline: GPT-4o’s improvement over always abstaining is +0.209, with a template-cluster bootstrap interval of +0.113 to +0.306. At λ=4, however, GPT-4o is −0.010 relative to always abstaining, and at λ=9 the difference is only +0.001.

The monitor also saves computation. At λ=1, early stopping removes 13% of GPT-4o steps; across the five agents, low-stakes savings range from 13% to 28%. But the paper limits the claim to one benchmark and one database, trains a separate confidence head for each agent, reports one training seed and notes that a question-only baseline already reaches 0.955 AUROC for detecting EHRSQL’s unusually templated unanswerable questions.

The practical benchmark lesson is precise: report reliability together with coverage and the error penalty. Without those coordinates, +0.24 looks like an accuracy improvement. With them, it is evidence that a learned monitor can find a safer operating point—and that the safest point at high stakes may still be silence.