live wire
▸Snowflake agent evals can pin judge versions; unpinned metrics may roll forward and shift scoresSnowflake▸ThoughtSpot makes Spotter 3 default for new instances; it requires download rights and always shares query data with an LLMThoughtSpot documentation▸MotherDuck adds image classification and object location to prompt_jev(), returning typed SQL outputs with confidence scoresMotherDuck▸BigQuery conversational analytics adds Montréal and Toronto; mixed-region agents still default to globalGoogle Cloud Documentation▸Fabric enables Azure OpenAI Copilot access by default, but OpenAI-subprocessor and cross-region processing/storage switches remain offMicrosoft Learn▸Snowflake Cortex Agents now continue by default when named tools are inaccessible, emitting warnings instead of failing the runSnowflake documentation▸Databricks Genie One can use group memberships to tailor answers, while Unity Catalog remains the access-control boundaryDatabricks documentation▸Fabric data agents can query Power BI models with Read alone; Build and workspace access are unnecessaryMicrosoft Learn▸Fabric data-agent Git configs expose column flags, but selecting a table still includes every columnMicrosoft Learn▸Google Data Agent Kit offers remote Spark kernels, but notebook exports can still move results into local Pandas or ArrowGoogle Cloud Documentation▸Cube says production analytics agents need explicit partial-result states, caller-filtered model search and durable artifactsCube▸Thinking Machines names Bridgewater behind ReViSQL-K2.6, citing 92.97% SC-16 accuracy at 15% of frontier-model costThinking Machines Lab▸AWS says a mismatched data/tool/prompt release cut its Field Sales agent pass rate from 78% to 38% in pre-productionAWS Artificial Intelligence Blog▸NL2SQL Multi-Agent ships evaluation artifacts across BIRD, CSpider, Archer and KDD DataSpace—but no software licenseGitHub▸Snowflake agent evals can pin judge versions; unpinned metrics may roll forward and shift scoresSnowflake▸ThoughtSpot makes Spotter 3 default for new instances; it requires download rights and always shares query data with an LLMThoughtSpot documentation▸MotherDuck adds image classification and object location to prompt_jev(), returning typed SQL outputs with confidence scoresMotherDuck▸BigQuery conversational analytics adds Montréal and Toronto; mixed-region agents still default to globalGoogle Cloud Documentation▸Fabric enables Azure OpenAI Copilot access by default, but OpenAI-subprocessor and cross-region processing/storage switches remain offMicrosoft Learn▸Snowflake Cortex Agents now continue by default when named tools are inaccessible, emitting warnings instead of failing the runSnowflake documentation▸Databricks Genie One can use group memberships to tailor answers, while Unity Catalog remains the access-control boundaryDatabricks documentation▸Fabric data agents can query Power BI models with Read alone; Build and workspace access are unnecessaryMicrosoft Learn▸Fabric data-agent Git configs expose column flags, but selecting a table still includes every columnMicrosoft Learn▸Google Data Agent Kit offers remote Spark kernels, but notebook exports can still move results into local Pandas or ArrowGoogle Cloud Documentation▸Cube says production analytics agents need explicit partial-result states, caller-filtered model search and durable artifactsCube▸Thinking Machines names Bridgewater behind ReViSQL-K2.6, citing 92.97% SC-16 accuracy at 15% of frontier-model costThinking Machines Lab▸AWS says a mismatched data/tool/prompt release cut its Field Sales agent pass rate from 78% to 38% in pre-productionAWS Artificial Intelligence Blog▸NL2SQL Multi-Agent ships evaluation artifacts across BIRD, CSpider, Archer and KDD DataSpace—but no software licenseGitHub
nl2sql.ai
benchmarkBENCHMARK WATCH

Graph-query conformance suite finds silent semantic splits across engines

A new executable GPML reference finds multiple engines can accept the same path pattern yet return different endpoint-pair multisets—without an error.

Graph query benchmark reveals silent semantic mismatches across engines.
Chart: figures from the story
By The Benchmark Desk· Sep 23, 2026the quick take — two AI hosts go live when you do

A new conformance suite for the graph pattern-matching language shared by GQL and SQL/PGQ finds that apparently valid path queries can produce different answers across shipping engines. The authors’ 17-construct test reports that five external engines differ from its executable reference on seven constructs; across all disagreements, 15 of 26 were silent, meaning the query returned a different multiset without raising an error. That puts the reported silence ratio at 0.58. The paper and artifact were released publicly.

This is a correctness benchmark, not a speed race

The suite covers path modes and restrictors, quantified segments, selectors, direction, parallel edges and self-loops. Each case is scored as conforming, diverging, rejected or inexpressible, based on an exact comparison of endpoint-pair multisets. Every cell was repeated three times to check determinism, according to the generated results.

The evaluated releases are Kùzu 0.11.3, DuckPGQ on DuckDB 1.4.1, Neo4j 5.26.30 and 2026.04.0, Memgraph 5.9.0, and Apache AGE 1.8.0. The project also reports Samyama-Graph 1.8.0 but excludes that engine from aggregates because the authors develop it and its release incorporated fixes prompted by the suite.

The reference implementation is not a majority vote among products. It encodes the standard’s GPML path semantics and must reproduce six worked answers from the standards committee’s reference exposition before engines are measured. That gate caught a bug in the authors’ first implementation, according to the repository documentation.

Not every difference is an implementation bug

The artifact’s second scoring pass compares each divergence with the path mode declared in that engine’s own documentation. Of 15 answer divergences in the aggregate map, nine are documented language differences; the remaining six are departures from the standard implemented by the relevant engine. Separately, oracle-free metamorphic checks report 39 self-consistency violations across six graph fixtures, attributed to Kùzu and DuckPGQ in the published results.

The practical signal is narrower—and more useful—than a conventional leaderboard rank: syntax compatibility does not guarantee answer compatibility. A workload moved between graph engines can keep running while changing its result, especially around path repetition and selection semantics.

There are important limits. This is a small semantic-conformance suite, not a broad workload benchmark. Its headline map compares endpoint-pair multisets rather than full path bindings, and it contains no latency or throughput measurements. Results are also tied to the named versions. Still, the repository includes the reference implementation, fixtures, adapters, expected outputs and minimal reproducers, making each reported mismatch inspectable rather than leaving it as a paper-only claim.

sources

Filed by The Benchmark Desk. Corrections: desk@nl2sql.ai · Our standards →

comments · 0

    Comments are moderated before they appear. Your email is used once to confirm it is you — never shown, never sold. Corrections and questions get an answer from the desk when we have one.

    more from the desk