The false positive problem: how LLM judges mislead table OCR benchmarks · Reducto Engineering

We created an internal benchmark where the ground truth is mathematically known, then used it to show that frontier LLM judges hallucinate table OCR errors at non-trivial rates. A parsed table needs to be faithful across multiple dimensions, such as the right content in the right cells, structure (rows, columns, spans, headers), and formatting (bold, currency symbols, blank separator rows). Human review doesn’t scale, and rule-based metrics miss too much nuance or contain bias to the people who make the rules.

So most teams reach for LLM judges to show the model the original image and the parsed HTML. We ran this experiment with an end goal to rigorously test how reliable LLM judges actually are on table parsing. The trick was building a dataset where we know the correct answer by design.

We started with FinePDFs, a wide open-source repository of documents. For each table, we extracted the image from the document, parsed it with Reducto to get HTML, then rendered that HTML back into an image. The rendered image is derived from the HTML, so by construction, every image-HTML pair is a perfect match. There’s no labeling ambiguity and a judge that says “wrong” on a clean pair is certainly wrong.

We then created a corrupted version of each pair by injecting a single error, such as a misspelled cell, a dropped column, a punctuation swap, and ran judges on both. A good judge should accept clean pairs and catch corrupted ones.

The full benchmark: 538 clean pairs and 538 corrupted pairs, tested across seven frontier models, evaluated on three criteria per pair:

1. Content accuracy: correct cell values, headers, no missing or extra data across both numeric and text content
2. Structural preservation: correct rows/columns/spans, cell-to-header mapping
3. Formatting fidelity: bold/italic on headers, currency and percent symbols, blank separator rows, capitalization

Our findings

On the clean pairs (reminder: the ones that are provably correct), here’s how often each model said something was wrong anyway:

The precision/recall tradeoff

The picture gets more interesting when you look at error detection, which is how often each model actually catches injected mistakes. We report:

• TPR: the judge marks at least one rubric False on a corrupted pair.
• Injected-rubric detection: the judge marks the specific rubric targeted by the corruption False.

Some observations:

• GPT-5.6 Luna is cheap and relatively conservative, but it has trouble detecting actual injected mistakes.
• Claude Opus 4.8 is the best choice when false alarms are expensive.
• GPT-5.6 Sol is a better default when cost matters, but its clean-table false-positive rate is more than twice Claude 4.8’s. It has the same capabilities as GPT 5.5, at less than half the estimated cost, making it a more efficient model.
• Gemini is attractive when missing an error is worse than reviewing extra flags.

A few patterns stand out from the per-criterion breakdown:

1. Formatting remains the main differentiator inside the GPT-5.6 family:

The corrupted HTML makes every letter-containing body cell bold and italic. Sol and Terra mark formatting_fidelity=False; Luna returns all three rubrics as True and describes the injected emphasis as faithful.

2. With content, models do not stick to comparing to the HTML, but tend to correct typos: The image below really says Pre-Primany / Primany. Claude Opus 4.8, Sol, Terra, GPT-5.5 and GPT-5.4 reject the matching HTML because they claim the image says Pre-Primary / Primary.

How this ties into your eval pipeline

The failure modes above are specific to naive, single-call judgment: showing a model the image and the full HTML and asking it to evaluate everything at once. That approach conflates three distinct questions into one prompt, and the noise compounds. If you’re using this pattern to evaluate table parsing, your benchmarks may be systematically skewed.

The fix isn’t abandoning LLM judges but rather being more deliberate about how you use them. Breaking the judgment into separate calls for content, structure, and formatting independently produces less noise per decision than one combined prompt. Grounding the judge with a reference output rather than asking it to reason from scratch helps further. When we apply this approach internally, the false positive rate drops significantly enough that LLM judges become a practical part of our evaluation pipeline rather than a liability.

Intention while choosing the judge also matters. Report false alarms and corrupted-pair rejection together, since a judge can appear conservative simply because it misses errors. If false alarms erode trust in your regression evals, use a conservative judge, but if missing real errors is the bigger risk, optimize for recall. And wherever possible, lean on comparative evals, since asking “which of these two outputs is better” is harder to get wrong than “is this output correct.”

99% is never good enough for prod — we’re always pushing toward perfection at Reducto. See the micro1 long-extraction benchmark, where Reducto leads on complex document extraction.

We’re releasing the benchmark: 538 image-HTML pairs from FinePDFs, clean and corrupted, along with the judge prompts we used: GitHub, Hugging Face.

Engineering by Jean Ghislain Billa, Yifei Hu, and the Reducto ML team. Written by Palak Agarwal.