From Frontier APIs to Fine-Tuned Models: Benchmarking Document Visual Grounding Across Every Major AI Provider

Your document AI extracts the right answer. But when an auditor asks ‘show me exactly where on the page’, can it? We benchmarked every major frontier model on that combined task, and the results reveal a capability gap most teams haven’t accounted for.

• This is Part 2 of our Document Visual Grounding series. Part 1 covered what DVG is, why spatial localisation matters for enterprise document AI, and how NATIVE and OCR-assisted pipelines work. If you haven’t read it, start there: “From OCR-Bound to OCR-Free” (Quantiphi Blog, Dec 2025). This post picks up with a direct question: how well do today’s frontier models actually perform on that task, and what does the answer mean for your AI stack?

Document Visual Grounding (DVG) is the task of extracting an answer from a document image and simultaneously localising it with a precise bounding box, not just finding the right value, but knowing exactly where on the page it lives. Models can approach this in two ways: NATIVE mode, where the model predicts the bounding box directly from the image in a single call, or OCR-assisted mode, where an OCR engine pre-extracts word-level boxes and the model selects from them. If this is new to you, our earlier post covers DVG and both approaches in depth. Here, we focus on how frontier models actually perform on this task, and what the results mean for production deployments.

We evaluated across three benchmarks: an internal set combining DOCILE (invoices, purchase orders, tax forms), Form-NLU (filled forms), referred to as the RFT eval, and DOGR-Bench, covering charts, posters, and PDF documents (Tencent Research, ICCV 2025). Results are reported on three metrics, F1text (answer text accuracy), F1loc (bounding box localisation accuracy), and F1all, the strictest measure which requires both text and location to be correct simultaneously. Getting one right without the other counts as a miss.

What we found was not a story of incremental progress. It was a story of a capability gap, and, in one case, a generation jump that most teams running these models in production have not yet acted on.

Key Findings:

The Gemini Generation Shift, A Capability Jump Most Teams Have Missed

To understand why the generation shift matters, you need to know what changed. In Gemini 2.5, native grounding, asking the model to predict a bounding box directly from a document image, was unreliable. F1loc (localisation accuracy) sat at 42–50% in NATIVE mode, meaning the model got the location wrong roughly half the time. Teams learned to compensate with an OCR pipeline, where an OCR engine pre-extracts word-level bounding boxes and the model selects from them rather than predicting coordinates itself.

Gemini 3.x changed this. The Figure 1 below shows the shift in NATIVE mode F1all (joint text + location accuracy) across both our business document benchmark and DOGR-Bench, which covers charts, posters, and PDFs:

• Key takeaway: Gemini 3.0 Flash NATIVE scores 59% on business documents and 71.5% on charts/PDFs, 15–26% above Gemini 2.5 NATIVE on the same tasks. This is not a marginal improvement.

The next question is whether this makes the OCR pipeline redundant. The answer depends on the generation:

• Key takeaway: For Gemini 2.5: OCR mode wins by 10–15%, keep the pipeline. For Gemini 3.x: NATIVE mode wins by 7–9%, the pipeline is now a liability, not an asset. If you are running Gemini 2.x + OCR today, upgrading to Gemini 3.x NATIVE is the single highest-impact change you can make.

Where Claude and GPT Fall Short, and Why OCR Alone Cannot Fix It

The comparison gets starker when you look at Claude and GPT. Neither model currently produces bounding box coordinates when reading a document image. This is not a configuration issue or a prompt engineering problem, these models were not trained with an explicit spatial grounding objective. The Figure 3 below shows F1loc (localisation accuracy only) in NATIVE mode on DOGR-Bench across all providers. The contrast is difficult to miss:

• Key takeaway: Claude Sonnet 4.6 NATIVE: 0.9% F1loc. GPT-5.4 NATIVE: 16.0% F1loc. Gemini 3.0 Flash NATIVE: 76.5% F1loc. These are not close results.

To understand why the overall F1all gap is so large as shown in Figure 5, it helps to isolate localisation accuracy (F1loc) specifically. Text extraction is reasonable across all providers, but the ability to point to the right location on the page is essentially absent for Claude and GPT in NATIVE mode, as  can be seen in Figure 3. A ~20% gap in joint accuracy that no OCR configuration will close. For teams standardised on OpenAI or Anthropic, OCR mode is the right approach, but accuracy expectations should be set accordingly.

• This is not a permanent indictment of these models. It reflects that document grounding requires training with an explicit spatial objective, something Gemini has invested in and the others have not yet. Capabilities may evolve.

A Small Fine-Tuned Model as a Viable Alternative

This is the question that motivated our own work. If Gemini is not available, due to data governance restrictions, cloud policy, cost at scale, or data privacy requirements, what are the realistic options? Can a purpose-built small model close the gap?

What we built

We fine-tuned Qwen3-VL-4B, a 4 billion parameter vision-language model, specifically for document visual grounding. The training used two stages: Supervised Fine-Tuning (SFT) on annotated business documents from DOCILE (invoices, purchase orders, tax forms) and Form-NLU (filled forms), followed by Reinforcement Learning (RL) with a reward signal that jointly penalises text extraction errors and localisation failures. We call the resulting model the Visual Grounding Model (VGM).

The RL stage is the differentiating factor. SFT teaches the model what correct answers look like. RL trains it to optimize the hardest part of the task, getting both the text and the bounding box right simultaneously.

• Key takeaway: On business documents, the domain it was trained on, the Visual Grounding Model scores 71.8% F1all versus 58.1% for Gemini 3.0 Flash. A 4B model beats a frontier API by 13.7% on its target task (Figure 4).

On benchmarks it was never trained on (DOGR-Bench, Figure 5), it continues to outperform all Claude and GPT configurations. On DOGR-Bench it sits 3% behind Gemini 3.0 Flash, for a 4B model evaluated out-of-domain, that is a strong result. There is no catastrophic forgetting.

• The Visual Grounding Model is not a Gemini replacement. It is a precision option for teams where APIs are restricted, cost is a constraint, or where domain-specific accuracy on business documents is the priority. On your documents, it can actually beat Gemini.

This work was accepted at the ICML 2026 EMMQA Workshop.

The Full Picture, All Models Ranked

With all models introduced, here is the complete ranking on DOGR-Bench, the benchmark with the broadest provider coverage and the most direct head-to-head comparison.

• Key takeaway: Gemini 3.x NATIVE leads. Our Visual Grounding Model sits 3rd, above every OCR-assisted frontier model, on a benchmark it was never trained on. Claude and GPT in NATIVE mode are at the bottom.

What Should You Do?

The data points to clear, situation-specific actions: