Advanced RAG Patterns

Last reviewed: August 2026 | This is a fast-moving area subject to quarterly review.

Note

For RAG basics (vector stores, embeddings, basic retrieval), read the RAG section in Getting Started and Vector Stores and Embeddings first. This document builds on those basics with advanced patterns that raise production quality (chunking, re-ranking, hybrid search, query expansion, evaluation).

Limitations of Basic RAG

Simply “document → embedding → retrieve → pass to LLM” is insufficient for production quality. Common problems cited by Azure and AWS official guides:

• Poor chunking breaks context, degrading retrieval quality.
• Without re-ranking, the LLM references irrelevant context from retrieved results.
• Ambiguous user queries (pronouns, abbreviations) defeat vector search alone.

Sources:

• Azure — Develop a RAG Solution: Chunking Phase
• AWS — Writing best practices to optimize RAG applications

Chunking Strategies

How much and how documents are split determines retrieval quality.

Chunking Methods

Method	Description	Best For
Fixed-size	Split at fixed token count (e.g., 512)	General text, blogs
Sentence-based	Split by sentence boundaries	Natural language documents
Recursive	Hierarchical: paragraph → sentence → word	Structured documents
Semantic	Group semantically similar sentences	Long explanatory text
Document-structure	Split by headings/sections	Manuals, wikis, technical docs

Azure recommends trying Fixed-size → Recursive → Document-structure in order of increasing sophistication (see the Chunking Phase guide).

Chunk Size Guide

• Too small — Insufficient context; retrieved fragments lose meaning.
• Too large — Multiple topics in one chunk degrades precision; increases token consumption.

General starting point (Azure guide):

• Chunk size: 500–1500 tokens
• Overlap: 10–20% between chunks to prevent context loss

Note

Chunk size is never “set and forget.” Measure retrieval quality with representative queries and adjust iteratively.

Vendor Chunking Options

Vendor	Supported Methods	Reference
AWS Bedrock Knowledge Bases	Default, fixed-size, hierarchical, semantic	KB Chunking Options
Azure AI Search (Foundry IQ)	Auto-chunking with integrated vectorization, customizable	Azure AI Search Chunking
Vertex AI RAG Engine	Chunk size/overlap configuration, RagManagedDb auto-management	RAG Engine

Managed RAG Pipelines

Instead of building chunking, embedding, retrieval, and re-ranking yourself, managed services handle the entire pipeline.

Vendor	Service	Strengths
AWS	Bedrock Managed Knowledge Base	GA June 2026. 6 native data connectors (S3, SharePoint, Confluence, Web Crawler, Google Drive, OneDrive), Smart Parsing (automatic multi-format parsing), Agentic Retriever (agent decomposes and searches complex multi-step queries), managed vector store. AgentCore Gateway MCP integration
Azure	Azure AI Search (Foundry IQ)	Integrated vectorization, built-in semantic ranker, custom skill pipeline. Also serves as managed knowledge layer in the Microsoft Foundry portal
Google Cloud	RAG Engine (Gemini Enterprise Agent Platform)	Source → embedding → retrieval unified. Cross Corpus Retrieval (simultaneous retrieval across multiple RAG corpora, Preview). RagManagedDb for automatic infra management

Note

Managed RAG is ideal for rapid prototyping. For fine-grained control over chunking logic or search algorithms, custom builds may be preferable.

Re-ranking

Vector search is fast but doesn’t rank by true relevance. Re-ranking re-scores the top-N results with a separate model.

graph LR
    Q[Query] --> V[Vector Search<br/>Top 50]
    V --> R[Re-ranker<br/>Relevance Re-score]
    R --> T[Top 5]
    T --> L[LLM]

Vendor Re-ranking Services

Vendor	Service	Reference
AWS	Bedrock Knowledge Bases Reranker (Amazon Rerank, Cohere Rerank)	Reranker Guide
Azure	Azure AI Search Semantic Ranker	Semantic Ranker
Google Cloud	Vertex AI Ranking API	Ranking API
OCI	Cohere Rerank (OCI Enterprise AI)	OCI Enterprise AI Models

Hybrid Search

Vector search is weak at exact string matching (product codes like SKU-12345, proper nouns). Hybrid search combines vector search with traditional keyword search (BM25).

Vendor	Hybrid Approach	Reference
AWS	OpenSearch Vector + BM25 (RRF algorithm)	Hybrid Search
Azure	Azure AI Search hybrid query	Hybrid Search
Google Cloud	Vertex AI Search (auto hybrid)	Vertex AI Search
OCI	OCI AI Vector Search with SQL combination	OCI AI Vector Search

Query Expansion and Transformation

When user queries are short or ambiguous, use an LLM to rewrite or expand the query.

• Query Rewriting — Resolve pronouns/abbreviations explicitly (e.g., “that” → “policy X discussed in the last meeting”).
• Multi-Query — Generate multiple query versions and search each.
• HyDE (Hypothetical Document Embeddings) — LLM generates a hypothetical answer, then embeds that answer for retrieval.

Official guides:

• Azure — Enrichment Phase of RAG
• AWS — RAG Optimization Guide

Evaluation

RAG systems require measuring retrieval quality and response quality separately.

Retrieval Quality Metrics

Metric	Meaning
Recall@K	Fraction of relevant docs in top-K results
MRR (Mean Reciprocal Rank)	Average reciprocal rank of correct document
NDCG	Ranking quality with position-weighted scoring

Response Quality Metrics

Metric	Meaning
Faithfulness	Is the generated answer grounded in retrieved documents?
Answer Relevance	Does the answer actually address the question?
Context Precision/Recall	How accurate and sufficient is the retrieved context?

Evaluation Tools

Tool	Description
RAGAS	Open-source RAG evaluation framework
Azure AI Evaluation SDK	Built-in Faithfulness, Relevance metrics
Bedrock Evaluations	Integrated model/RAG evaluation
Vertex AI Evaluation Service	Gen AI evaluation framework

Common Mistakes

• Setting chunk size once and never adjusting — Without measuring retrieval quality on representative queries, chunks may be too fragmented or mixed-topic.
• Passing vector search results directly to LLM without re-ranking — Irrelevant docs in top results cause hallucinations.
• Not considering hybrid search — Product codes and proper nouns requiring exact string matching won’t be found by vector search alone.

Checklist

• Chunk size and overlap tuned by measuring retrieval quality with representative queries
• Re-ranking (Semantic Ranker, Cohere Rerank, etc.) applied to improve result accuracy
• RAG evaluation metrics (Faithfulness, Answer Relevance) measured regularly

References

AWS

• RAG Options and Architectures
• Writing Best Practices to Optimize RAG Applications
• Bedrock Knowledge Bases Chunking
• Bedrock Knowledge Bases Reranker

Azure

• Design and Develop a RAG Solution
• RAG Chunking Phase
• Azure AI Search Hybrid Search
• Azure AI Search Semantic Ranker

Google Cloud

• RAG Engine (Gemini Enterprise Agent Platform)
• Vertex AI Ranking API
• Vertex AI Evaluation Service

OCI

• OCI AI Vector Search
• OCI Enterprise AI Models