The Open-Book Exam
Remember taking exams in school? There were two kinds: closed-book (rely entirely on memory) and open-book (reference materials allowed). Most people prefer open-book: when you can look things up, your answers are more accurate, not pieced together from fuzzy memories. RAG gives AI the same advantage: it can look up relevant information before answering. Here’s how it works.Core Concepts
The Three-Step Dance: Retrieve, Augment, Generate
RAG stands for Retrieval Augmented Generation, and those three words describe exactly what happens: Step 1: Retrieve When you ask a question, the system searches a knowledge base (the collection of documents it can draw on) for information relevant to your question. This isn’t like typing keywords into a search engine. It’s a semantic search, which matches by meaning rather than exact words (more on this in Lesson 3). Step 2: Augment The retrieved information is added to your original question. Think of it as handing the AI a cheat sheet along with the question. “Here’s what the user asked, and here’s some relevant information that might help you answer.” Step 3: Generate The AI generates a response, but now it’s informed by the retrieved context. Instead of relying only on training data, it can reference the specific information that was just retrieved. That’s it: three steps. The quality of the answer depends on how well each step is done.A Concrete Example
Imagine you have a RAG system connected to your company’s HR documentation, and an employee asks:“How many vacation days do new employees get?”Without RAG (traditional AI): The AI would try to answer from its general training data. It might say something like “Vacation policies vary by company, typically ranging from 10-20 days for new employees.” Technically true, but no help for your company. With RAG:
- Retrieve: The system searches the HR documents and finds a chunk (a short passage of a document) that says “New employees receive 15 days of paid vacation in their first year, increasing to 20 days after two years of employment.”
- Augment: This retrieved text gets combined with the original question into a prompt like: “Based on the following company policy: ‘New employees receive 15 days of paid vacation in their first year, increasing to 20 days after two years of employment.’ Please answer this question: How many vacation days do new employees get?”
- Generate: The AI responds: “New employees receive 15 days of paid vacation in their first year. After two years with the company, this increases to 20 days.”
The Library Analogy
Here’s another way to think about it. Imagine you’re a reference librarian answering a question. Bad librarian approach: Try to answer from memory, even if you’re not sure. Good librarian approach:- Listen to the question
- Walk to the relevant section of the library
- Pull out the books that might help
- Read the relevant passages
- Synthesize an answer based on what you found
What Makes Retrieval “Smart”?
How does the system know which documents are relevant? Traditional keyword search looks for exact word matches. If you search for “vacation days,” it finds documents containing those exact words. RAG typically uses semantic search, which understands meaning rather than just matching keywords. So if you ask about “time off for new hires,” it can still find the document about “vacation days for new employees” because it understands these concepts are related. Lesson 3 explains how this works, using embeddings.The Augmented Prompt
The “augment” step is where retrieval meets everything you learned about prompting in AI 102. The retrieved information needs to be presented to the AI in a way that’s clear and useful. A typical augmented prompt might look like this:Why “Augmented” and Not “Replaced”?
RAG adds to what the AI can do. It doesn’t replace it. The AI still uses its general knowledge and reasoning abilities. The retrieved context provides specific information, but the AI synthesizes, summarizes, and presents that information using its language capabilities. This helps because:- The AI can combine retrieved facts with general knowledge
- It can explain concepts in accessible language
- It can draw connections the source documents don’t explicitly make
- It can format responses appropriately for the question
Try It Yourself
Exercise 1: Trace the RAG Steps
Think of a question you might ask a company’s internal knowledge base. For example: “What’s the process for requesting a new laptop?” Now trace through what would happen at each step:- Retrieve: What documents or sections would be relevant? (IT policies? Equipment request forms? Procurement procedures?)
- Augment: What would the combined prompt look like?
- Generate: How would the AI synthesize this into a helpful answer?
Exercise 2: Compare With and Without
Take any factual question about a specific organization or domain. Write out:- What a general AI might say (from training data only)
- What specific information would need to be retrieved to improve the answer
- What the improved answer might look like
Exercise 3: Design the Context
Imagine you’re building a RAG system for a recipe website. A user asks: “What can I make with chicken and broccoli that’s ready in under 30 minutes?” What information would you want retrieved? How would you structure the augmented prompt? Think about what context would be most helpful for generating a good response.Common Pitfalls
Pitfall 1: Retrieving Too Much
If you retrieve every vaguely related document and stuff it all into the prompt, you create problems. The AI might get confused by contradictory information, miss the most relevant details in the noise, or hit token limits (the cap on how much text, counted in tokens, a model can take in at once). The fix: Quality over quantity. Retrieve the most relevant chunks, not everything that might be related.Pitfall 2: Retrieving Too Little
The opposite problem: if retrieval is too narrow, you might miss important context. The AI can only work with what it’s given. The fix: Find the right balance. Test your retrieval to ensure it captures the necessary information without overwhelming the system.Pitfall 3: Ignoring the Augmentation Step
Some people focus entirely on retrieval or generation and treat augmentation as an afterthought. But how you structure the prompt matters a lot. The fix: Pay attention to your prompt templates. Clear instructions about how to use the context lead to better outputs.Pitfall 4: Forgetting That AI Can Still Hallucinate
Even with RAG, the AI can make mistakes. It might misinterpret the retrieved context, combine information incorrectly, or fill gaps with plausible-sounding fiction. The fix: RAG reduces hallucinations but doesn’t eliminate them. Critical information should still be verified. See Search, Generate, Verify.Level Up
Here’s your challenge: Design a RAG workflow on paper. Scenario: You’re creating a RAG-powered assistant for a university library that helps students find and understand research papers. For each of these questions, outline what the three RAG steps would look like:- “What are the main findings of the Smith 2023 paper on climate migration?”
- “Find me papers that compare machine learning approaches for medical diagnosis.”
- “Explain the methodology used in recent behavioral economics studies.”
- What would be retrieved for each?
- How would the augmented prompt be structured?
- What would make the generated response useful?
Key Takeaway
RAG retrieves relevant information, augments the question with it, then generates an informed response. It works best when all three steps work well together: smart retrieval, clear augmentation, and grounded generation.What’s next
You now understand the three-step RAG process, and one technology makes the retrieval step work so well: embeddings. To build the three steps yourself, see the Search guide, the RAG pipeline tutorial and Building Trustworthy Grounded Applications.Lesson 3: Embeddings: How AI Understands Meaning
Explore how AI converts text into numbers that capture meaning, so RAG can find relevant information even when the exact words don’t match.

