A clinical model scored 0.965 AUROC—until the benchmark stopped leaking future diagnoses

Clinical-agent benchmarks can be wrong before the agent issues its first query. A newly released, runnable data-conversion study shows how assigning a diagnosis to the start of an admission can make a downstream model look dramatically better than it is under information that would actually have been available at prediction time.

The EHR2Trace paper reports a six-hour held-out AUROC of 0.965 when a mortality-proxy model is trained and tested on histories with diagnoses backdated to admission. When the test histories are filtered by when information became available, the same backdated-training setup falls to 0.642. Training with the availability filter as well raises performance on that deployment-like test to 0.829. Those are separate evaluation conditions, not interchangeable leaderboard scores: the 0.323-point fall is the penalty exposed by changing the test-time information boundary, while the 0.187-point recovery comes from matching training to that boundary.

The timestamp is part of the label

EHR2Trace stores event time and availability time separately. Its inclusion rule requires both to be no later than the prediction time. In the paper’s example, a laboratory specimen may have been collected before a decision but only stored afterward; treating collection time as availability gives the model information a clinician did not yet possess. The system also keeps medication orders, dispensing and administration as distinct events, rather than treating an intended treatment as one actually delivered.

The authors ran the converter across three clinical datasets, producing 846.4 million canonical events. The validation layer contains 55 checks over saved outputs, including source links, identities, timestamps, terminology, splits and cross-format consistency. In an injected-fault catalogue, it detected 28 of 28 faults. Repeated synthetic builds produced 60 byte-identical artifacts across four execution configurations, according to the paper.

This one can be inspected

Unlike a result available only as a paper table, the Apache-2.0 repository includes the converter, dataset configurations, synthetic fixtures, fault catalogue, experiment scripts and aggregate records. Its README gives a credential-free route through the 100-patient MIMIC-IV demonstration subset and says CI reruns the fault-detection, reproducibility and demo-conversion paths. It does not include patient data, clinical vocabularies or private conversion outputs, so reproducing the complete 846.4-million-event run still requires restricted source datasets and substantial compute.

The broader benchmark lesson is concrete: “time of event” and “time knowable” are different fields. A clinical agent can execute valid SQL against a faithfully converted table and still receive future information. Evaluation reports should therefore publish the availability rule alongside the split, metric and timestamp policy—and test on histories filtered exactly as deployment will filter them.