LOKI finds text-mediated joins precisely—but misses more than half

The headline number hides the real trade-off

LOKI, a new system for discovering relationships between otherwise disjoint tables through supporting text, reported 98.0% micro typed-pair precision on an end-to-end test built from 382 MIMIC-IV hospital admissions. But its corresponding micro recall was only 43.8%, meaning the system found highly reliable links while leaving more than half of the labeled relationships unrecovered. The authors’ pooled counts make the imbalance concrete: 2,823 true positives, 59 false positives and 3,618 false negatives from 2,882 predictions for the GPT-OSS configuration. [Rahman et al., arXiv, Sept. 22, 2026]

That is still a meaningful result because LOKI is not merely classifying relationships handed to it. It first scores table-document pairs, extracts row-to-sentence links, composes those links into row-sentence-row join paths, clusters them by relationship and then materializes typed tables with sentence-level provenance. Its end-to-end evaluation used diagnosis and medication tables connected through clinical notes; the benchmark’s fine-grained labels were generated by three frontier-model annotators, consolidated by majority vote and checked by a human annotator. [Paper, §5.1 and §5.4]

Direct prompting buys coverage

The strongest direct baseline, Qwen-3.7 through an API, reached 99.7% micro precision and 63.9% micro recall, with 4,135 true positives and 2,333 false negatives. Its micro F1 was 0.779, versus 0.606 for LOKI with GPT-OSS 20B. Direct local Qwen-3.6 landed between them at 99.5% precision, 47.7% recall and 0.645 F1. These are not perfectly symmetric systems: the direct baselines receive the relevant diagnosis table, medication table and notes, while LOKI performs the preceding discovery step as well. [Paper, Table 7 and evaluation protocol]

The efficiency argument favors LOKI. The paper reports that it cuts downstream language-model tokens by 67%–69%, to roughly one-third of direct prompting. With GPT-OSS handling relationship naming, the full pipeline was about 2× faster than direct local Qwen-3.6 and 3.9× faster than the frontier API baseline; under the authors’ API-equivalent pricing assumptions it was about 11× and 44× cheaper, respectively. [Paper, RQ2]

How to read this benchmark

The result is best read as a selective integration system, not a comprehensive one. High precision and explicit evidence are valuable where a wrong join is costly, but 43.8% recall is too low if the application expects broad relationship discovery. The authors identify coverage as the main limitation and also note that candidate discovery currently scores every table-document pair, so scaling cost grows with the product of the table and document counts. [Paper, conclusion]

For practitioners, the useful question is not whether LOKI “beats” direct prompting. It is whether provenance and lower token cost justify missing more relationships. This first benchmark says the answer may be yes for precision-first workflows—but not yet for exhaustive integration.