Synthetic Hospital separates agent retrieval gains from agent-loop overhead

A benchmark with a database behind it

Synthetic Hospital is not a text-to-SQL leaderboard, but it tests the neighboring problem that analytics agents increasingly face: retrieving evidence from a governed longitudinal record before producing a structured answer. The benchmark contains 1,268 synthetic patients, 5,602 encounters and 12,014 task instances, served through FHIR R4 resources, role-based access controls and an Epic-style function-calling interface. Its five tasks cover patient diagnosis, general and specialty summarization, evidence retrieval and imaging indication. The authors released patient-disjoint train and public splits and reserved a held-out split for scorer-only evaluation. The paper reports 7,619 training instances, 1,859 public instances and 2,536 held-out instances.

That structure matters because the benchmark can separate reasoning quality from evidence access. In the single-turn public-split evaluation, Kimi 2.5-thinking led patient diagnosis at 0.732 severity-weighted F1. Opus 4.6 led general summarization at 0.550 finding-level F1, while GPT 5.3 led evidence retrieval at 0.833 Precision@5 and imaging indication at 0.518 concept F1. No single model won every task: GPT 5.3 had the best mean rank, but three models supplied the task-level best scores. Table 3 is therefore more useful as a task profile than as a single-model hierarchy.

Retrieval helped diagnosis, not everything

The sharpest result comes from the paper’s controlled agent experiment on a 100-patient subset of the public split. The authors compared a full-context single-turn model, a full-context agent and a self-retrieving agent using the same 13-tool API and a 40-action budget. For GPT 5.3, diagnosis fell from 0.705 in single-turn form to 0.661 with the full-context agent loop, then rose to 0.841 when the agent retrieved its own evidence. Mistral Large followed the same pattern: 0.669, 0.628, then 0.810. Llama 4 Scout improved from 0.554 to 0.794 and then 0.894.

That gain did not generalize. GPT 5.3 summarization declined from 0.342 finding-level F1 in the full-context condition to 0.196 with an agent loop and 0.142 with self-retrieval. Its retrieval Precision@5 dropped from 0.826 to 0.668 and recovered only to 0.674. Table 4 supports a narrower conclusion than “agents are better”: self-retrieval helped a task whose answer depends on gathering dispersed longitudinal evidence, while extra orchestration often hurt when the relevant context was already available.

A useful result with explicit limits

The benchmark’s ground truth is ontology-backed rather than copied from a clinician’s chart. A preregistered physician check found that the graph recovered 111 of 119 clinically required diagnosis relationships, or 93%, but the authors leave false-positive relationship measurement to future work. They also note that the records do not explicitly simulate missingness, contradictions or documentation errors, and that the agent study covers only three models, one scaffold and 100 patients.

Those limits keep Synthetic Hospital from standing in for production clinical evaluation. They do not erase its central measurement lesson: agent architecture should be scored as an intervention, not assumed to improve the underlying model.