Two independent data-pipeline checks agreed perfectly—and were wrong together

A data pipeline can compute the same number twice and still be wrong in exactly the same way. A newly released case study provides an unusually complete example: Python set logic and SPARQL queries over an RDF graph agreed on seven published counts, yet three counts were wrong. The largest, a claimed 932 disposition disagreements between two satellite catalogues, fell first to 261 and then to 220 after the authors read the source vocabulary and object histories more carefully.[^1]

The shared premise defeated the independent implementations

The two verification paths were technically different. One read source files and counted Python sets; the other queried a materialized graph of 2,330,660 triples. Both nevertheless imported the same classification constants and loaders. Those constants treated every unrecognized flight-phase status as meaning an object remained in orbit. That collapsed transition events—including docking and attachment—into a claim about current disposition.[^1]

The failure was therefore above the layer that redundancy tested. The gate verified that two implementations applied one interpretation consistently; it did not verify that the interpretation matched the source catalogue. The published 932 count included 640 transition-coded objects, 500 involving docking or attachment. A separate coverage count fell from 900 to 622 after a fourth catalogue file was found, while an undisclosed-tracking-loss count moved from 1,104 to 1,094.[^1]

Even the first correction was not final. Reading each object’s later phase history removed 42 artifacts from the 261-count correction, producing 220. A second reading from the source’s derived current catalogue produced 219 and shared 208 objects with the 220-object set; the paper assigns causes to all 23 differences and explicitly avoids treating either register as ground truth.[^1]

Generated checks inherited the same meaning

The authors then asked three pinned Claude models to generate independent verification paths, with tools disabled and an empty workspace. Across 75 trials, 72 returned the same defective count of 932. Supplying the source’s status definitions did little: 29 of 30 paths still returned 932. Only one path would have exposed the defect, by stopping on unclassified status codes rather than silently assigning them.[^1]

This is a narrow experiment—one task, three models from one family and five trials per cell—not a general model ranking. Its value is forensic rather than universal. The repository publishes the gate scripts, object ledger, controlled-replication prompts, raw outputs and behavioral labels; frozen source files are omitted, but checksums and retrieval locations are documented.[^2]

The practical lesson is sharper than “add another checker.” Independence must apply to premises, not merely code paths. For analytics pipelines and agents, a useful review asks which definitions, source inventories and semantic assumptions both routes share—and adds at least one check that goes back to the source documentation or an external history rather than recomputing the same derived quantity.

[^1]: Rovai, “Independent Verification Paths Are Not Independent,” arXiv:2609.37603 [^2]: Released gate and replication artifacts