What is RAG? Retrieval-Augmented Generation Explained

Retrieval-Augmented Generation (RAG) is an AI architecture that gives a Large Language Model (LLM) access to external knowledge at inference time. Instead of relying solely on what the model learned during training, RAG retrieves relevant documents from a knowledge base and augments the prompt before asking the LLM to generate an answer.

RAG solves two of the biggest problems with standalone LLMs: knowledge cutoff (the model does not know recent events) and hallucination (the model invents facts it does not know). It is now the most widely used pattern for building production AI applications.

RAG in 30 seconds

How RAG works — step by step

User query
    │
    ▼
[1. Embed query]  →  query vector
    │
    ▼
[2. Vector search]  →  top-K relevant chunks
    │
    ▼
[3. Build prompt]  →  "Context: <chunks>\n\nQuestion: <query>"
    │
    ▼
[4. LLM generates]  →  grounded answer
    │
    ▼
Answer (with source citations)

Offline indexing (done once or periodically)

1. Load documents — PDFs, web pages, database rows, Markdown files
2. Chunk — split documents into passages (e.g., 512 tokens with 50-token overlap)
3. Embed — convert each chunk to a dense vector using an embedding model
4. Store — write vectors + metadata to a vector database (Chroma, Pinecone, pgvector, etc.)

Online retrieval (every query)

1. Embed the user question using the same embedding model
2. Similarity search — find the top-K chunks whose vectors are closest to the query vector
3. Stuff into prompt — insert retrieved text as context for the LLM
4. Generate — the LLM reads the context and answers the question

RAG architecture diagram

Documents ──► Chunker ──► Embedder ──► Vector DB
                                           │
User Query ──────► Embedder ──► Retriever ─┘
                                    │
                                    ▼
                            [Context + Query]
                                    │
                                    ▼
                                   LLM
                                    │
                                    ▼
                                 Answer

Simple RAG pipeline in Python

# pip install openai chromadb

import openai
import chromadb
from chromadb.utils import embedding_functions

# 1. Setup
client = openai.OpenAI()   # uses OPENAI_API_KEY env var
embed_fn = embedding_functions.OpenAIEmbeddingFunction(
    model_name="text-embedding-3-small"
)

chroma = chromadb.Client()
collection = chroma.create_collection("docs", embedding_function=embed_fn)

# 2. Index documents
documents = [
    "Python was created by Guido van Rossum and released in 1991.",
    "FastAPI is a modern Python web framework based on type hints.",
    "RAG stands for Retrieval-Augmented Generation.",
    "Vector databases store high-dimensional embeddings for similarity search.",
]

collection.add(
    documents=documents,
    ids=[f"doc_{i}" for i in range(len(documents))],
)

# 3. Query function
def rag_query(question: str, top_k: int = 2) -> str:
    # Retrieve relevant chunks
    results = collection.query(query_texts=[question], n_results=top_k)
    context = "\n".join(results["documents"][0])

    # Build augmented prompt
    prompt = f"""Answer the question using ONLY the context below.
If the context does not contain the answer, say "I don't know."

Context:
{context}

Question: {question}
Answer:"""

    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
        temperature=0,
    )
    return response.choices[0].message.content

# 4. Use it
print(rag_query("When was Python released?"))
# "Python was released in 1991."

print(rag_query("What is FastAPI?"))
# "FastAPI is a modern Python web framework based on type hints."

print(rag_query("Who invented JavaScript?"))
# "I don't know."  <- grounded, no hallucination

RAG with LangChain

# pip install langchain langchain-openai langchain-chroma

from langchain_openai import OpenAIEmbeddings, ChatOpenAI
from langchain_chroma import Chroma
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.runnables import RunnablePassthrough
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain_community.document_loaders import TextLoader

# Load and split
loader = TextLoader("my_docs.txt")
docs = loader.load()
splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=50)
chunks = splitter.split_documents(docs)

# Embed and store
vectorstore = Chroma.from_documents(chunks, OpenAIEmbeddings())
retriever = vectorstore.as_retriever(search_kwargs={"k": 3})

# RAG chain
prompt = ChatPromptTemplate.from_template("""
Answer the question based only on the following context:

{context}

Question: {question}
""")

llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)

chain = (
    {"context": retriever, "question": RunnablePassthrough()}
    | prompt
    | llm
)

answer = chain.invoke("What does this document say about X?")
print(answer.content)

Chunking strategies

How you split documents dramatically affects retrieval quality.

Rule of thumb: 256–512 tokens per chunk with 10–15% overlap. Experiment with your data.

Embedding models

Always use the same embedding model for indexing and querying.

Vector databases

For most projects: Chroma (dev) → pgvector or Pinecone (production).

RAG vs Fine-tuning vs Prompt engineering

Decision rule:

• Need factual accuracy over private docs → RAG
• Need the model to learn a new tone or format → Fine-tuning
• Want the model to use existing knowledge better → Prompt engineering
• Best results → RAG + fine-tuned model

Common RAG failure modes

Advanced RAG patterns

Hybrid search (dense + sparse)

Combine vector similarity with keyword search (BM25) for better recall:

# Using Weaviate hybrid search
results = collection.query.hybrid(
    query="Python web framework",
    alpha=0.5,  # 0 = BM25 only, 1 = vector only
    limit=5
)

Reranking

Use a cross-encoder to re-score retrieved chunks before passing to LLM:

from sentence_transformers import CrossEncoder

reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")

scores = reranker.predict([(query, chunk) for chunk in top_chunks])
sorted_chunks = [c for _, c in sorted(zip(scores, top_chunks), reverse=True)]
top_reranked = sorted_chunks[:3]

HyDE (Hypothetical Document Embeddings)

Ask the LLM to generate a hypothetical answer first, then embed that for retrieval:

# 1. Generate hypothetical answer
hypo_answer = llm.generate(f"Write a passage that answers: {question}")
# 2. Embed the hypothetical answer (not the question)
query_vector = embed(hypo_answer)
# 3. Search with this vector — often retrieves better results

Multi-query retrieval

Generate multiple query variants to improve recall:

queries = llm.generate(f"Generate 3 different phrasings of: {question}")
all_chunks = [retrieve(q) for q in queries]
unique_chunks = deduplicate(all_chunks)

RAG evaluation metrics

# pip install ragas
from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_precision

results = evaluate(dataset, metrics=[faithfulness, answer_relevancy, context_precision])
print(results)

RAG stack options

Common mistakes

RAG vs related terms

Frequently asked questions

Why use RAG instead of just making the context window bigger? Even 1M-token context windows have limits: cost scales linearly with tokens, latency increases, and LLMs suffer "lost in the middle" — they attend worse to content in the middle of a long context. RAG retrieves only the handful of relevant passages, keeping prompts focused and cheap.

Does RAG eliminate hallucination? No — it reduces hallucination when the answer exists in the knowledge base. If the retrieved chunks are irrelevant or the LLM ignores them, it can still hallucinate. Combine RAG with faithfulness evaluation and a strong system prompt such as "answer ONLY from the context".

When should I fine-tune instead of RAG? Fine-tune when you need the model to change its behavior or format — writing in your brand tone, following a specific output schema, or learning domain-specific abbreviations. Use RAG when you need the model to access specific facts or documents.

How do I choose chunk size? Start with 512 tokens and 50-token overlap. Run retrieval experiments: if recall is low (the right chunk is never retrieved), try smaller chunks. If the LLM complains about missing context, try larger chunks or increase top-K.

Can RAG work without a vector database? Yes — for small knowledge bases (fewer than 100 documents), you can embed everything in memory with FAISS or even a simple cosine similarity loop. Vector databases add scalability, persistence, filtering, and hybrid search at the cost of infrastructure.

What is the difference between RAG and an AI agent? A basic RAG pipeline retrieves documents in a fixed sequence every time. An AI agent decides dynamically which tools to call, when to retrieve, how many times to retry, and can chain multiple retrieval steps. Think of RAG as one tool an agent might use, not a mutually exclusive pattern.