Spider 2.0’s free Snowflake track has a reproducibility problem before the SQL even runs

Spider 2.0-Snow is designed as a no-cost, 547-example text-to-SQL track hosted on Snowflake. But an open issue in the benchmark’s repository documents a more basic obstacle: some participants cannot get automated evaluation code through Snowflake’s authentication layer at all. Spider 2.0 Issue #143

The issue was opened on September 23, 2025 after a participant reported that delegated username-and-password credentials no longer worked for containerized queries because multi-factor authentication was required. The proposed alternative—a programmatic access token—also failed because the account lacked the required network policy. The issue remains open. Issue #143

This was not a one-user configuration mistake that disappeared after documentation changed. A second participant reported the same problem on October 20. A Spider 2.0 collaborator replied on October 22 that the team was working on a fix and provided a fallback account for Python access. Another participant reported a TOTP authentication failure on November 7. On January 23, 2026, a fourth user said both personal and fallback-account access failed despite following the documentation; that report received three positive reactions. A fifth user confirmed the same failure on January 27 and said the documented token-generation control was unavailable. Issue #143 comments

The current access guide is more explicit. It says accounts should be created within 12 hours, requires authenticator-based MFA for interactive login, and tells Python users to place a programmatic access token in the benchmark credential file. If the token button is missing, it provides an ALTER USER statement intended to create a role-restricted token. Snowflake access guideline

That is a useful remediation path, but the unresolved issue matters for interpreting participation. Spider 2.0-Snow’s leaderboard score is execution-based, yet a would-be reproducer must first obtain an account, enroll MFA, create a token with sufficient privileges and successfully connect an automated client. The public site describes Snow as “NO COST,” which is true of the query quota; it does not mean the evaluation is frictionless or locally reproducible. Spider 2.0

This does not invalidate the leaderboard’s reported results, and the issue does not show that any score is wrong. It does show that benchmark infrastructure is part of the measurement system. Until the maintainers close the issue or demonstrate a working non-interactive path for affected users, independent reruns face an access gate before model quality can be measured. Issue #143