The best model solves 64.7% of Cypher turns—but only 4.6% of full conversations

A turn score can hide a broken conversation

CypherTurn measures something most database-language benchmarks leave out: whether an agent can keep a chain of related queries correct from beginning to end. The new benchmark contains 721 sessions and 5,927 turns across seven synthetic Neo4j graphs. Each retained gold query was executed against its graph, and the paper reports Cohen’s κ of 0.83 for the two-annotator accept/reject review. (paper; repository)

The headline result is a sharp mismatch between turn-level and session-level performance. Under the guided protocol, which supplies gold prior history, Claude Opus 4.7 leads the 15 evaluated models at 64.7% execution accuracy. But it completes only 4.6% of sessions with every turn correct. GPT-5.5 reaches 62.5% turn accuracy and 2.5% session exact match. Those are not competing splits: they are two metrics on the same guided evaluation, and the session metric is deliberately unforgiving. (paper; results table)

Autonomy changes the ranking

CypherTurn also runs an agentic ×3 protocol in which models must inspect the schema, execute Cypher, maintain their own history and manage a shared action budget. Here Gemini 3.1 Flash Lite ranks first at 43.2% execution accuracy, while the guided leader falls to 40.2%. GPT-5.5 scores 40.1%. The overall guided-versus-agentic ordering remains correlated—Spearman ρ is 0.87—but the top of the table changes, which the authors call the “Autonomy Divergence.” (repository; paper)

That distinction matters because the agentic run compounds its own mistakes. Claude Opus 4.7’s chain error rate rises from 43.4% guided to 71.2% agentic ×3; GPT-5.5 rises from 42.4% to 69.8%. In other words, after an incorrect turn, the next turn is much more likely to fail when the model must carry its own history. (paper)

Giving agents more room does not solve the problem. The study tests shared budgets of ×3, ×5 and ×10 actions per turn, yet reports that strong frontier models still use roughly two actions per turn and the larger allowance does not close the autonomy gap. (paper)

What the benchmark does—and does not—prove

CypherTurn is reproducible as a framework: its repository includes the 721 sessions, gold Cypher and answers, seven graph dumps, Docker-pinned Neo4j services, both protocols and deterministic execution-based metrics. It does not ship the reference model trajectories, and the maintainers warn that reproducing reasoning-model rows requires adding thinking-token stripping that is absent from the published API client. Proprietary model access is also left to the reproducer. (repository)

The practical lesson is narrower than “model X is best.” A system chosen on isolated or oracle-context query accuracy may not be the best system for a real analytical conversation. CypherTurn makes that gap measurable: generation skill, history management and recovery from prior errors need separate scores.