ThoughtSpot’s 82.8% BIRD result comes with an unfinished audit trail

ThoughtSpot has published an unusually detailed vendor benchmark for its Spotter AI analyst: 82.8% accuracy on 1,533 questions across 11 databases from a BIRD development split, with one attempt per question. The post says a frontier LLM reached 78.4% and a coding agent 72.5% under the same setup. Those are useful measurements—but they are not yet independently reproducible results. (ThoughtSpot)

The valuable part is the slope

ThoughtSpot breaks the result out by BIRD’s simple, moderate and challenging bands rather than stopping at one aggregate. Spotter reached 70.6% on 231 challenging questions, versus 66.7% for an unnamed warehouse assistant, 58.4% for an unnamed lakehouse agent, 56.7% for an unnamed frontier model and 46.8% for an unnamed coding agent. The company also reports that Spotter fell 16.4 points from simple to challenging, while the coding-agent category fell 33.5 points. (ThoughtSpot)

That framing is genuinely useful for buyers. Averages can conceal systems that perform well on straightforward questions and deteriorate on multi-join or nested queries. A difficulty curve gives teams a better hypothesis to test against their own workload than a single percentage.

ThoughtSpot also describes a controlled model swap: moving its harness from Claude Opus 4.8 to Claude Opus 5 raised overall accuracy from 79.4% to 82.8% and challenging-question accuracy from 65.8% to 70.6%, while tool calls stayed around 2.1 per question. Those are vendor-reported measurements, not an independent reproduction. (ThoughtSpot)

The audit trail is not finished

The appendix says BIRD’s official execution scorer could not run end to end because the compared products did not all emit portable SQLite SQL. ThoughtSpot instead used an exact-match pre-pass followed by a rubric judge that saw the question, expected output and returned data. But the live page still describes that judge as “[model]” and its human validation as “[N] questions ([X]% agreement)”. It also contains a bracketed conditional note for a possible re-score. (ThoughtSpot)

The four competitors remain category labels, and the post says exact products, versions, configurations, per-question outputs and judge transcripts are available only on request. The company further says the 3.9-point lead over the nearest rival on the challenging band has not cleared a paired per-question significance test. (ThoughtSpot)

That makes the result a vendor evaluation built on a public dataset—not an entry readers should compare directly with the systems on the official BIRD leaderboard. The distinction matters because the scoring path, product identities and validation details determine what the percentage means.

The practical takeaway is two-sided: copy ThoughtSpot’s single-attempt protocol and difficulty-band reporting for internal evaluations, but withhold comparative conclusions until the judge identity, audit sample, agreement rate and named configurations are published. The post has the shape of a transparent benchmark; its remaining placeholders show exactly what must be filled in before it becomes one.