Spider 2.0 can give opposite joins the same answer when the data never reaches them

An open Spider 2.0 pull request identifies a sharp failure mode for execution-based evaluation: two SQL programs with opposite final join predicates can receive the same observable result when an earlier join is empty. The proposed repair adds eight synthetic rows across eight IPL tables in the Spider 2.0-Lite assets so the incorrect query finally produces a different answer.

The affected workflow asks for players who scored at least 100 runs in a match while playing for the losing team. Both candidate queries build the same player_runs, losing_teams and players_in_losing_teams common table expressions. They differ only at the last join: the intended query uses p.player_id = plt.player_id; the counter-query uses p.player_id <> plt.player_id.

On the released IPL sample described in the pull request, ball_by_ball and batsman_scored have no overlapping key on (match_id, over_id, ball_id, innings_no). That makes the first CTE empty. Once no row reaches the last join, equality and inequality both return an empty result, so execution alone cannot distinguish the intended program from its semantic opposite.

The patch constructs a minimal discriminator rather than changing the evaluator. It adds one row apiece to ball_by_ball, batsman_scored, extra_runs, match, player, player_match, team and wicket_taken. The new batting row supplies 100 runs on a shared synthetic ball key. A synthetic match marks team 0 as the loser, while the player and membership rows are arranged so the equality query remains empty and the inequality query becomes non-empty. The changed-files diff confirms that the proposal modifies exactly those eight JSON table assets.

This is not an official benchmark correction. Pull request #181, opened March 12, remains unmerged as of August 30, and its contributor—not the Spider 2.0 maintainers—provides the reproduction claim. The live Spider 2.0 site still defines Lite as a 547-example text-to-SQL setting spanning BigQuery, Snowflake and SQLite; no leaderboard score has been recomputed from this patch.

The broader lesson is about test-data geometry, not this single cricket question. Execution accuracy can validate only distinctions that the database instance makes observable. A gold query and a wrong query may disagree in logic yet agree on every returned row when joins are empty, predicates never bind, or values lack a counterexample. Adding adversarial rows is therefore part of benchmark design, not merely data cleanup.

For practitioners, the useful regression test is simple: mutate one decisive operator at a time—= to <>, AND to OR, inner join to left join—and require at least one database row to separate the outputs. If the mutation survives, the evaluation case is under-specified even when its gold SQL is correct.