A random split made nearest-neighbor spend classification look eight points better

The same purchase, on both sides

A new benchmark for enterprise spend classification finds that ordinary row-level random splits put repeated purchase descriptions into both training and test data at industrial scale. Across 343,484 usable California purchase-order lines and 914,683 usable Washington State Amazon lines, normalized item text from the test set also appeared in training for 60.7% and 61.2% of rows, respectively, under the paper’s iid-naive split. Even byte-identical overlap reached 54.7% in California and 60.6% in Washington.

That matters because the test stops measuring only generalization to unfamiliar purchases. At the eight-digit commodity-code level on the Washington corpus, TF-IDF 1-nearest-neighbor retrieval scored 87.5% accuracy under the row-random split. Grouping identical normalized text so that every copy stayed on one side of the boundary cut the score to 79.5%—an 8.0-point drop. On California, the same model fell from 43.0% to 34.5%, an 8.6-point drop.

The paper evaluates four protocols: row-random iid-naive, leak-free iid-grouped, a temporal holdout and a supplier holdout. Across learned lexical-model cells, the inflation from the naive split ranged from 0.8 to 8.6 percentage points. The grouped protocol is the cleanest direct test of novel text; temporal and supplier splits deliberately retain whatever repeat purchasing occurs naturally across time or vendors.

Leakage can reverse the useful comparison

The more important result is not merely that every score gets larger. Leakage can blur the difference between systems. On Washington commodity classification, 1-NN reached 87.5% under the naive split while a 67-million-parameter DistilBERT reached 87.1%. On the grouped split, DistilBERT scored 83.0% against 79.5% for 1-NN. The temporal and supplier protocols also put DistilBERT ahead, by 3.6 and 2.3 points. A benchmark that looked like a tie under random rows consistently preferred the transformer under three harder protocols.

Seen-versus-novel reporting exposes the mechanism. On California, a Complement Naive Bayes baseline scored 56.0% on repeated text but 31.9% on novel text, a 24.1-point gap. Embedding retrieval improved the novel-text score to 34.8% and narrowed its gap to 19.1 points, but did not remove it.

Read the denominator before the accuracy

The two corpora also should not be compared as if they were interchangeable. At full commodity depth, only 365,085 of Washington’s 914,683 usable rows were evaluable. Conflicting labels for identical text impose a text-only upper bound of 79.4% on California commodity accuracy, versus 98.4% on Washington. Those are properties of these corpora and annotations, not universal limits on procurement classification.

The paper’s scope is deliberately bounded: two US public-sector datasets, buyer-assigned labels, one DistilBERT run per cell and no LLM baseline. Its deployment check—351 of 1,084 model suggestions accepted unchanged, or 32.4% with a 29.7–35.2% Wilson interval—is author-reported corroboration, not independent validation.

The practical benchmark rule is simpler than the model stack: publish overlap rates, keep duplicate text in one partition, and report performance separately for seen and novel inputs. Otherwise, a system can be rewarded for recognizing a purchase the test set was supposed to keep new.