O-Funnel survives five PubMed schema renames—but costs 40× more than regex

A new document-extraction benchmark makes a useful trade explicit: structural resilience is measurable, but it is not free. In the O-Funnel paper, the released Python system kept overall field-level F1 at 1.000 after five PubMed XML elements were renamed. A hand-written tag-regex baseline fell from 1.000 to 0.199 on the same drifted corpus.

The comparison is unusually inspectable

The evaluation used 34,989 records from two files in the public PubMed 2026 baseline and requested six fields: PMID, article title, ISSN, language, publication year and DOI. The authors then renamed PMID, ArticleTitle, ISSN, Language and PubDate while leaving the extraction requirements unchanged. Table 3 reports the aggregate result above; O-Funnel’s per-field language F1 was 0.999, while the other five fields rounded to 1.000.

That language miss matters. The paper says 38 records contained an OtherAbstract language attribute that became the only exact key match after the article-language element was renamed. Adding an Article scope removed the error, but the authors reported the unscoped configuration. That is better evidence than a flawless headline: the resolver can still select the wrong structurally plausible field when requirements are underspecified.

This clears the desk’s artifact gate because the MIT-licensed repository ships the library and benchmark scripts, and the package is installable from PyPI. The paper identifies the evaluated version as 0.1.1.

Robustness came with a throughput bill

On 120 constructed hard scenarios, byte regexes scored 0.43 F1, plain O-Funnel scored 0.84, and O-Funnel with its scavenger enabled reached 1.00. On a separate 117-document perturbation suite, O-Funnel improved from 0.91 to 0.94 F1 after learning aliases from residue; fully opaque keys moved from 0.00 to 0.67.

But Table 8 reports about 0.295 ms per small pristine document for O-Funnel, versus 0.008 ms for regex—roughly a 40× slowdown. The system still processed about 3,400 small documents per second on one CPU core. On the larger PubMed records, including capture, completeness checking and six requirements, throughput was about 144 records per second.

The boundary is as important as the win

O-Funnel is not a semantic extractor. On the first 500 rows of an SWDE-derived web set, it recovered only 0.2% of values from the attribute name alone because it does not model visual label adjacency. The paper also limits its claims to well-formed supported formats and small documents; PDFs, scans and very wide schemas remain outside the test.

The practical result is narrower than “regex is obsolete.” When schemas are stable and latency dominates, regex remains much faster. When renamed fields, format changes and decoys are routine failure modes, this released benchmark shows that structural extraction can buy substantial resilience—and quantifies both the benefit and the cost.