Fed-ReMasker wins most federated imputation tests—but its benchmark is still a controlled world

A federated imputation system faces a harder problem than filling scattered blank cells when one participating institution has never measured a feature at all. A new benchmark for Fed-ReMasker makes that feature-level case explicit, and reports the lowest error in 96.7% of 243 feature-level scenarios and 93.2% of 324 value-level scenarios in its homogeneous grid. The paper was posted with reproduction code and configurations.

What the benchmark measures

The study evaluates six synthetic datasets and three real numeric datasets: Codon, PhysioNet and NHANES. It varies the feature-missing ratio from 0 to 0.3, the value-missing ratio from 0.1 to 0.5, and the number of clients across 3, 5 and 10. Error is normalized root mean squared error, reported separately for ordinary missing values and positions belonging to a feature absent from a client.

Averaged across datasets, Fed-ReMasker records 0.061 value-level NRMSE and 0.065 feature-level NRMSE. The closest baseline in the aggregate table, Cafe, records 0.079 and 0.090, respectively. The benchmark also includes Fed-MIWAE, FedHF-Impute and mean imputation.

The feature-level result is the more informative number. A client with some observations of a column can learn its local distribution; a client with none must depend on relationships transferred through the federated model. Fed-ReMasker adapts a masked autoencoder and aggregates client updates with standard FedAvg.

The exceptions matter

This is not a clean sweep. Fed-ReMasker loses 22 of 324 value-level scenarios and 8 of 243 feature-level scenarios. Of those 30 exceptions, 21 occur with 10 clients. The largest deficits are small—at most 0.0058 NRMSE for value-level and 0.0044 for feature-level imputation—but the concentration at the largest client count is a useful stress signal.

Under four client-heterogeneity conditions on the three real datasets, Fed-ReMasker leads all 36 value-level scenarios and at least 35 of 36 feature-level scenarios against each baseline. More elaborate server strategies do not help consistently: FedAvg is best or tied in 29 of 30 dataset-condition-metric cells compared with FedProx and FedAdam.

What remains untested

The paper’s main grid uses missing-completely-at-random masks and a single partition seed. Three seeds are used only for the smaller heterogeneity study. It does not jointly combine the heterogeneity types, evaluate small clinical cohorts, test categorical features, measure downstream prediction quality, or report communication and computation costs. It also applies neither secure aggregation nor differential privacy, so “federated” here means raw records are not pooled—not a formal privacy guarantee.

The accompanying repository exposes the scenario generator, sweep script, evaluation code and scripts used to produce the reported tables. That makes the result more inspectable than a score alone, but independent reproduction will still need to establish whether the advantage survives less controlled missingness and smaller sites.