The fastest attention backend for tabular models depends on which axis it reads

A new runnable benchmark from SAP shows that attention-kernel choice for tabular foundation models cannot be reduced to “use the newest FlashAttention.” The winning backend changes with the table axis, sequence length and GPU generation—and the gap between the best and worst backend can exceed 3× for the same tensor shape.

The September 25 study evaluates five training-capable backends—PyTorch SDPA efficient, SDPA cuDNN, FlashAttention 2, 3 and 4—plus the inference-only vLLM and SageAttention implementations. Its main configuration uses 12 heads, a head dimension of 64 and bfloat16 across NVIDIA A100, H100 and B200 GPUs. Column tests hold rows at 1,024 and sweep 16–2,048 columns; row tests hold columns at 64 and sweep 32–131,072 rows.

The table axis changes the answer

Column attention operates over the contiguous dimension of the benchmark’s row-first tensor layout. Here, cuDNN is a strong cross-platform default: it leads H100 column throughput between 128 and 1,000 columns, while FlashAttention 3 edges ahead only at longer sequences. On B200, FlashAttention 4 leads through 256 columns, but cuDNN overtakes it at longer column sequences.

Row attention reverses that pattern because PyTorch’s SDPA path must make contiguous copies of query, key and value. On A100 and H100, FlashAttention 3 leads except below 128 rows, reaching up to 3.5× the SDPA-efficient baseline and about 1.5× cuDNN. On B200, FlashAttention 4 leads most of the tested range, but cuDNN retakes the lead beyond 8,000 rows.

Hardware alone therefore does not determine the kernel. The paper reports that B200’s best backends exceed the best H100 column-attention throughput by more than 2× and A100 by more than 5×, yet even B200 changes winners as sequence length grows.

A benchmark teams can actually rerun

This result clears a bar many systems papers do not: SAP published the benchmark runner, tests, backend-specific dependency groups, plotting scripts and raw-result workflow. The repository includes make targets for A100/H100/B200 runs, stores results by GPU, dtype and backend, and flushes each completed shape to JSON. That makes the crossover claims inspectable rather than merely plotted in a paper.

The practical H100 recommendation for the paper’s main shape is deliberately mixed: use cuDNN for column attention and FlashAttention 3 for row attention, with SDPA efficient as a possible fallback for very short sequences. For inference beyond 16,000 rows, SageAttention can lead row throughput, but it is approximate and quantized; predictive quality must be checked separately.

The limit is equally important. This is an isolated-kernel benchmark on synthetic tensors, not an end-to-end TabPFN or TabICL training run. It ignores surrounding layers, data movement and the train/query cross-attention details of specific model families. The benchmark establishes that backend selection materially changes kernel throughput. It does not yet establish the same multiplier for full-model latency or model quality.