A 625-dataset benchmark tests whether an LLM can identify the task before AutoML begins

AutoML normally starts after a user has already said whether a dataset represents classification, regression or forecasting. A new open benchmark asks whether a model can infer that decision from the dataset itself, given only the target column.

What was tested

The benchmark contains 625 public datasets: 326 tabular and 299 time-series datasets drawn from OpenML, UCI, Kaggle and the Time Series Classification Archive. Its labels cover binary classification, multiclass classification and regression or forecasting within each domain. The authors publish both the annotated collection on Zenodo and their evaluation code on GitHub.

They split the collection 20%/20%/60% into stratified development, validation and held-out test partitions. Prompt design and few-shot examples were developed on the first partition, configurations were selected on validation, and final results were measured once on the test split. Reported uncertainty comes from 1,000 bootstrap resamples of that fixed test set—not repeated retraining.

On tabular data alone, AutoGluon reached 0.93 macro-F1. LLM configurations that received a dataset description plus target-specific statistics reached 0.97–0.98, with Qwen3-14B at 0.98. Statistics without the description still produced 0.95–0.97, suggesting that target cardinality and distribution carry much of the signal when the domain is already known.

Crossing domains is harder

When the model also had to distinguish tabular data from time series, GPT-5.3 with reasoning, few-shot examples, descriptions and target statistics led the held-out test split at 0.90 macro-F1. Qwen2.5-14B reached 0.84 in the comparable few-shot setting; Qwen3-14B reached 0.76. The paper says the common errors confused the same downstream task across the two domains.

The smallest deployable configuration exposed the cost of removing semantic context. Qwen3-4B-Instruct reached 0.75 macro-F1 and 0.74 balanced accuracy with descriptions and statistics, but fell to 0.50 macro-F1 when the description was removed. Its reasoning-oriented 4B counterpart reached only 0.62 in few-shot and 0.45 in zero-shot settings.

What the score does not establish

The benchmark is runnable and unusually explicit about split use, but its endpoint remains a label. It does not measure whether a wrong label actually changes preprocessing, model selection or final predictive quality. The target column is also supplied by the user, and the public datasets may have appeared in model pretraining; the authors say direct memorization of their new labels is impossible but content contamination remains possible.

So 0.90 macro-F1 is evidence that a large model can infer a plausible task type across heterogeneous public data. It is not yet evidence that inserting that model before an AutoML pipeline improves the pipeline’s final result.