live wire
▸Snowflake agent evals can pin judge versions; unpinned metrics may roll forward and shift scoresSnowflake▸ThoughtSpot makes Spotter 3 default for new instances; it requires download rights and always shares query data with an LLMThoughtSpot documentation▸MotherDuck adds image classification and object location to prompt_jev(), returning typed SQL outputs with confidence scoresMotherDuck▸BigQuery conversational analytics adds Montréal and Toronto; mixed-region agents still default to globalGoogle Cloud Documentation▸Fabric enables Azure OpenAI Copilot access by default, but OpenAI-subprocessor and cross-region processing/storage switches remain offMicrosoft Learn▸Snowflake Cortex Agents now continue by default when named tools are inaccessible, emitting warnings instead of failing the runSnowflake documentation▸Databricks Genie One can use group memberships to tailor answers, while Unity Catalog remains the access-control boundaryDatabricks documentation▸Fabric data agents can query Power BI models with Read alone; Build and workspace access are unnecessaryMicrosoft Learn▸Fabric data-agent Git configs expose column flags, but selecting a table still includes every columnMicrosoft Learn▸Google Data Agent Kit offers remote Spark kernels, but notebook exports can still move results into local Pandas or ArrowGoogle Cloud Documentation▸Cube says production analytics agents need explicit partial-result states, caller-filtered model search and durable artifactsCube▸Thinking Machines names Bridgewater behind ReViSQL-K2.6, citing 92.97% SC-16 accuracy at 15% of frontier-model costThinking Machines Lab▸AWS says a mismatched data/tool/prompt release cut its Field Sales agent pass rate from 78% to 38% in pre-productionAWS Artificial Intelligence Blog▸NL2SQL Multi-Agent ships evaluation artifacts across BIRD, CSpider, Archer and KDD DataSpace—but no software licenseGitHub▸Snowflake agent evals can pin judge versions; unpinned metrics may roll forward and shift scoresSnowflake▸ThoughtSpot makes Spotter 3 default for new instances; it requires download rights and always shares query data with an LLMThoughtSpot documentation▸MotherDuck adds image classification and object location to prompt_jev(), returning typed SQL outputs with confidence scoresMotherDuck▸BigQuery conversational analytics adds Montréal and Toronto; mixed-region agents still default to globalGoogle Cloud Documentation▸Fabric enables Azure OpenAI Copilot access by default, but OpenAI-subprocessor and cross-region processing/storage switches remain offMicrosoft Learn▸Snowflake Cortex Agents now continue by default when named tools are inaccessible, emitting warnings instead of failing the runSnowflake documentation▸Databricks Genie One can use group memberships to tailor answers, while Unity Catalog remains the access-control boundaryDatabricks documentation▸Fabric data agents can query Power BI models with Read alone; Build and workspace access are unnecessaryMicrosoft Learn▸Fabric data-agent Git configs expose column flags, but selecting a table still includes every columnMicrosoft Learn▸Google Data Agent Kit offers remote Spark kernels, but notebook exports can still move results into local Pandas or ArrowGoogle Cloud Documentation▸Cube says production analytics agents need explicit partial-result states, caller-filtered model search and durable artifactsCube▸Thinking Machines names Bridgewater behind ReViSQL-K2.6, citing 92.97% SC-16 accuracy at 15% of frontier-model costThinking Machines Lab▸AWS says a mismatched data/tool/prompt release cut its Field Sales agent pass rate from 78% to 38% in pre-productionAWS Artificial Intelligence Blog▸NL2SQL Multi-Agent ships evaluation artifacts across BIRD, CSpider, Archer and KDD DataSpace—but no software licenseGitHub
nl2sql.ai
benchmarkREPRODUCTION

QueryClarify’s security test catches its own UI copy—not a leaked credential

A fresh reproduction found one deterministic false positive in an otherwise unusually layered text-to-SQL security suite.

False positive security scan on UI copy
AI-generated illustration
By The Benchmark Desk· Oct 4, 2026the quick take — two AI hosts go live when you do

The useful failure

QueryClarify is a newly updated text-to-SQL project that lets users upload SQLite, CSV, TSV or Excel data, then asks an agent to retrieve schema context, generate SQL and execute it. Its repository describes a defense-in-depth design: uploaded SQLite files are rebuilt with views, triggers and virtual tables removed; user databases are opened read-only; SQLite’s authorizer permits reads but refuses writes, ATTACH and extension loading; and MySQL access can be restricted to an allow-list and a separately audited SELECT-only account.

That is a much stronger test surface than a README promise. On October 4, I cloned the repository at its current default branch and ran its standard-library unit-test command. The runner discovered 156 tests. It reported 145 passes, nine errors because this clean environment lacked the documented pandas dependency, one skipped frontend-bundle check, and one actual assertion failure.

The failure was reproducible by running that test alone. The repository-wide credential scanner flagged frontend/src/App.tsx line 541 as a hard-coded credential because the UI contains the status assignment Passed = "****". The test expected zero findings, so it failed. This is not evidence of an exposed password; it is evidence that the scanner’s credential-literal rule currently mistakes presentation text for a secret.

Why this still matters

False positives are not harmless in a security gate. A check that fails on normal interface copy either blocks releases or trains maintainers to waive the check. The smallest defensible fix is not to disable repository scanning, but to narrow the credential rule or add a tightly scoped allow-marker for this exact non-secret string, then preserve the planted-secret tests that verify real detections.

The rest of the visible design deserves credit. QueryClarify says uploaded databases are isolated by browser client ID, expire after seven days by default and return the same 404 for another user’s identifier as for a nonexistent one. Its built-in MySQL path fails closed when no allow-list can be loaded, and its startup audit reports privileges beyond SELECT. The repository also states an important limitation: identity is anonymous local-storage state, history is in memory, and there is no built-in rate limiter or job queue.

This reproduction is not an accuracy benchmark, and the partial suite run is not a full deployment audit. The dependency-related errors say nothing about the CSV/Excel implementation. The finding is narrower and directly repeatable: the project ships meaningful engine-layer guardrails, but its own repository-wide secret gate is red on current source because of a false positive in frontend copy.

Filed by The Benchmark Desk. Corrections: desk@nl2sql.ai · Our standards →

comments · 0

    Comments are moderated before they appear. Your email is used once to confirm it is you — never shown, never sold. Corrections and questions get an answer from the desk when we have one.