Parity turns text-to-SQL result regressions into a CI failure

A new open-source project is trying to move text-to-SQL evaluation into the pull-request loop. Parity was created on September 3 as a Python 0.1.0 alpha package under the MIT license; at verification time it had no stars, forks or tagged GitHub releases. Its narrow proposition is useful: generated SQL can be syntactically valid and still change a business number, so a regression gate should compare what queries return rather than merely whether they execute.

Three ways to define “correct”

Parity’s runner implements three test modes. A golden test executes agent SQL and an oracle query over the same fixture, normalizes both result sets and compares them with a configurable numeric tolerance. An invariant test checks built-in conditions—currently non-negative values and exact row counts. A differential test executes both a candidate agent and a baseline agent, then compares their results.

That makes the tool more specific than a general prompt-evaluation framework. Its configuration example asks teams to provide schema and seed SQL, business questions, and either oracle SQL or invariants. The agent adapter may return a SQL string or a structured response. A failing test produces exit code 1, giving CI a conventional pass/fail boundary.

The gate has real boundaries

Before execution, Parity’s static guard allows one statement beginning with SELECT or WITH and rejects a built-in list of DDL, DML and external-access tokens. The sandbox runs SQLite in a spawned child process, opens file databases read-only, kills work after a timeout, and applies best-effort memory limits on Unix-like systems. Those controls reduce risk, but they do not make the current release a database-agnostic production executor: the package metadata and example configuration target SQLite, and Windows memory limiting is explicitly skipped.

Parity also writes a run manifest with SHA-256 hashes for available config, schema, seed, model-ID and prompt-version inputs. That helps identify what changed between runs, although a hash records identity rather than proving that a fixture represents production behavior.

The repository’s GitHub Actions workflow installs the package, invokes a stated 114-test unit/integration suite, and then runs Parity against its included good-agent demo. The README says all 114 tests pass; this desk inspected the workflow and implementation but did not independently rerun the suite.

The result is a credible early reference for teams that want business-metric regressions to block a merge. It is not yet evidence of field adoption or cross-engine reliability. The next meaningful signals will be external users, tagged releases, and fixtures against production SQL dialects—not a larger badge count.