Building the Optimal RAG Pipeline: What 40+ Experiments Taught Us

Introduction

Every document Q&A system you build involves five chained design decisions. You pick a parser, a chunking strategy, an embedding model, a retrieval method, and a generation model. Get any one of these wrong and the entire pipeline degrades and you may not find out until it is already running on real data.

We ran more than 40 controlled experiments across a filtered subset of Salesforce UniDoc-Bench, with 800 questions covering text-answer and table-answer question types, drawn from 545 source PDFs spanning 7,403 pages across eight business domains. UniDoc-Bench in full covers 70,000+ real-world PDF pages across four answer modalities (text-only, table-required, image-only, and multimodal). This study focuses on the two textual modalities where answers can be grounded purely in extracted text. Benchmarking image and multimodal question types remains open for future work.

Every combination was evaluated with corpus-level retrieval across all 545 documents simultaneously. The system had to identify the correct document and page purely from the query as it would in a real deployment. This post covers what we found.

The Five-Decision Pipeline

A RAG pipeline is a sequence of five decisions, each of which feeds the next. Weakness at any stage limits the ceiling of every stage downstream.

Figure 1. The five-stage RAG pipeline and the winning configuration at each stage

We held each upstream decision fixed when testing the next downstream one. The parser × chunker ablation tested 15 combinations with embedding and retrieval fixed. The retrieval strategy ablation then tested BM25, dense, and hybrid approaches with the winning parser+chunker fixed. Finally, the generation ablation tested model and prompt combinations with the winning retrieval configuration fixed. This controlled design means every comparison is apples-to-apples.

Finding 1: The Parser Barely Matters

The first question most teams agonize over is which parser to use. Should you pay for an LLM-based OCR service? Use a layout-aware parser? Does text quality matter?

The answer is: not much, as long as you are using fixed-token chunking. Across every fixed-token configuration, the gap between PyMuPDF (fast, free, no API calls) and LiteParse (layout-aware, PDF.js-based) was less than 0.005 NDCG. That is statistically negligible.

The LLM-native parser deserves special mention. It uses a single Gemini 2.5 Flash call to OCR the entire document, returning tables as HTML, equations as LaTeX, with semantic chunk boundaries. It is architecturally elegant. Yet it ranks 13th out of 15 configurations on NDCG@10. Two factors likely contribute: 1.7% of documents failed to parse entirely, introducing systematic zero-recall for affected questions; and the markup-heavy output (<table>, <tr>, <td>) may create a format mismatch with an embedding model trained on clean prose, though we have not isolated this as the sole cause.

Finding 2: Chunking is the Most Critical Retrieval Decision

Of all five pipeline decisions, chunking strategy has the largest impact on retrieval quality. The gap between best and worst configuration is 0.055 NDCG, larger than any other variable we tested. And the winner is not the sophisticated option.

Figure 2. NDCG@10 across all chunking strategies (PyMuPDF parser, Gemini Embedding-001, dense FAISS, 800 questions).

Fixed-300-token chunks and sentence-boundary chunks both reach NDCG 0.835, tied at the top. Every other strategy scores lower. Here is why:

Why Fixed-300 Wins

A 300-token chunk with a 50-token overlap is tight enough to represent one idea. Our hypothesis is that this granularity helps the embedding model produce a cleaner, more focused vector per chunk which in turn ranks the genuinely relevant chunks higher. The NDCG and MRR (Table 2) numbers support this: fixed-300 leads on both despite having lower raw recall than larger chunks.

Why Fixed-1024 Underperforms Despite Best Coverage

This is the most counter-intuitive result from the chunker ablation. Fixed-1024 achieves the highest recall@10 of 0.965 as shown in Figure 4, meaning 96.5% of answer pages are present in the top-10 results. Yet its NDCG@10 is only 0.829, ranking 5th. A likely explanation is that a 1024-token chunk spans 2–3 pages, so it physically covers more ground regardless of relevance, inflating recall without improving ranking quality. NDCG penalises this by rewarding only chunks that are ranked high and genuinely relevant.

Why Semantic Chunking Fails

Semantic chunking (MiniLM cosine threshold) scores the lowest of any non-page strategy at NDCG 0.793 and recall@10 0.793, meaning it misses ~21% of answer pages entirely. Our hypothesis is that MiniLM, which was designed for clean prose, may struggle with the noisy nature of PDF-extracted text, mid-word line breaks, bullet points, and page numbers, producing chunk boundaries that don’t align well with meaningful content units. We did not run controlled ablations to isolate this, but the performance gap relative to every fixed-token strategy is consistent and notable.

One more data point worth closing the loop on: the LLM-native configuration from Finding 1, where the model itself inserts semantic chunk boundaries via <CHUNK_BREAK> tags, also scored NDCG 0.793, placing it alongside semantic chunking at the bottom of the rankings. Model-generated chunk boundaries, at least in this setup, offered no advantage over fixed-token splitting.

Finding 3: Gemini Embedding-001 Leads, Free Alternatives Are Close

Across both chunking strategies, Gemini Embedding-001 (3,072-dim, $0.15/1M tokens) consistently achieves the best NDCG and hit@1 scores. The hit@1 gap is the most meaningful metric here: 0.764 versus 0.638–0.685 for alternatives. When the top-ranked chunk is the right one, generation quality is maximised.

Two surprises worth noting. First: Gemini Embedding-V2 (preview) performs worse than even thenlper/gte-large despite being 33% more expensive than V1. As a natively multimodal model, text retrieval is not its primary objective. Avoid it until it exits preview. Second: Alibaba-NLP/gte-large-en-v1.5 underperforms thenlper/gte-large despite being the newer version. Version number does not imply better retrieval quality, always validate empirically.

Finding 4: Three Retrieval Tiers for Different Needs

With the parser and chunker fixed, we tested three retrieval strategies: dense vector search (Gemini Embedding-001, FAISS), BM25 lexical search (Whoosh with StemmingAnalyzer), and Hybrid RRF (BM25 + dense, fused with Reciprocal Rank Fusion).

A note on BM25 implementation quality: our initial BM25 used naive whitespace tokenization, scoring NDCG 0.707–0.768. Upgrading to Whoosh with StemmingAnalyzer (stemming + stopword removal) improved BM25 NDCG by up to +0.069. If your BM25 comparison looks weak, check your tokenizer first.

Figure 3. NDCG@10 across retrieval strategies (Whoosh BM25, Gemini dense, Hybrid RRF), grouped by chunking config.

The results split cleanly by chunk size:

Why BM25 still trails on semantic questions: questions use conceptual language (“Federal Reserve’s evaluations concerning inflation predictions”) while answers use domain-specific vocabulary (“assessments from the Beige Book”, “Fed Funds Rate June 2023”). Stemming bridges morphological variants but cannot bridge semantic gaps. Dense retrieval handles these cases; BM25 cannot.

Finding 5: More Coverage Hurts Answer Quality

With retrieval fully characterised, we carried the three best-performing chunker configurations i.e. fixed_300, sentence, and fixed_1024 into generation testing. Before looking at which model and prompt performs best, one result from that evaluation demands attention on its own: the configuration with the highest retrieval recall produces the worst answers.

Figure 4. Recall@10 vs. answer correctness for the two chunkers evaluated end-to-end (800 questions).

Fixed-1024 chunks retrieve the ground-truth page 96.5% of the time. Fixed-300 chunks retrieve it 90.9% of the time. Yet fixed-300 generates correct answers 17.8% more often. A second metric, faithfulness, which measures how well an answer’s claims are actually supported by the retrieved context, rather than invented or drawn from outside it, reinforces this: 0.472 for fixed-300 versus 0.252 for fixed-1024, suggesting that answers generated from larger chunks are less grounded in the retrieved context, possibly because the model has more surrounding noise to contend with.

When the retriever returns 10 fixed-1024 chunks, the generator receives roughly 6,000–7,000 words of context to find a specific answer in. We observe lower correctness and lower faithfulness as a result. Whether this is due to attention dilution, context confusion, or simply harder extraction is an open question, but the pattern is consistent across every model and prompt combination we tested.

Finding 6: Generation Model and Prompt, Two Winners Depending on What You Optimise For

With the top three retrieval configurations established: fixed_300, sentence, and fixed_1024, we carried all three into the generation ablation. All three are close on NDCG@10 at retrieval time (0.835 / 0.835 / 0.829), so we let generation quality decide the winner.

The result was clear, fixed_300 won end-to-end on this benchmark, average correctness 0.638 versus sentence at 0.632 and fixed_1024 at 0.535. It also had the highest faithfulness (0.447 vs 0.438 for sentence), meaning its answers stayed most grounded in the retrieved context. fixed_1024 despite having the best retrieval recall collapsed at generation due to the noise dilution effect as described in Finding 4. Sentence chunking remains a valid choice if indexing cost is a priority (it saves ~17% on embedding tokens), but on this benchmark fixed_300 delivered the best end-to-end quality.

In an initial screening stage, we tested every combination across 3 retrieval configurations, 3 models (Gemini 2.5 Flash, Gemini 2.5 Pro, Claude Sonnet), and 3 prompting strategies, 27 combinations in total, evaluated on a smaller sample of questions. From these, we carried the 7 strongest and most representative combinations forward to the full 800-question evaluation reported here. The three prompting strategies tested were:

• Zero-shot (ZS): a direct instruction to answer the question from the retrieved context, with no additional guidance.
• Chain-of-Thought (CoT): the model is asked to reason step-by-step before producing its final answer, rather than answering immediately.
• doc_qa: a stricter prompt that instructs the model to answer using only the retrieved context and explicitly say “not found in context” if the answer isn’t present, reducing fabrication.

The chart below shows these 7 combinations ranked by correctness, with the LLM judge score alongside. Two distinct winners emerge depending on what you are optimising for:

Figure 5. All evaluated generation combinations ranked by correctness

RAGAS correctness decomposes each answer into claims and checks them against the reference answer. The LLM judge evaluates holistic quality as a human would. Claude produces richer, more narrative answers that judges reward but deviate slightly from the exact phrasing of reference answers. Gemini Flash CoT produces concise, precise extractions that match reference answers closely but read less naturally.

Prompt strategy is model-specific. CoT consistently helps Gemini Flash (up to +0.045 over doc_qa). It does not help Claude Claude reasons internally by default, so the explicit prompt adds verbosity without quality gain. The doc_qa strict-grounding prompt helps Claude correctness on fixed-300 (+0.033 over zero-shot) but hurts Gemini Flash across all configurations. There is no universal “best prompt”, prompt strategy must be matched to the model.

What the Study Points To

Across all combinations tested, one configuration consistently led end-to-end. The numbers below reflect what it achieved on this benchmark. Treat them as a directional reference, not a universal prescription.