Skip to main content

The Honest Conversation

RAG solves many problems well. It also has limits, and ignoring them leads to disappointment and failed projects. This lesson covers when RAG shines, when it struggles, and how to tell whether your RAG system is actually working well, so you can pick good projects and fix problems at the source.

Core Concepts

When RAG Shines

RAG is at its best when: You have good source material RAG retrieves and uses what’s in your knowledge base. If your documents are accurate, well written and complete, RAG can pass that quality on. If they’re not, no retrieval system can fix that. Questions have answers in the documents RAG finds and presents existing information. It doesn’t create new knowledge. If the answer to a question is somewhere in your knowledge base, RAG can find it. If it isn’t, RAG can only acknowledge the gap. Users ask relatively specific questions “What’s the refund policy?” works well. “Tell me everything about your company” is too broad; RAG won’t know what to retrieve. Accuracy and grounding matter When you need answers tied to trusted sources rather than general AI knowledge, RAG provides that grounding. Being able to cite sources is a real advantage. Information is relatively stable If your knowledge base doesn’t change every hour, RAG can provide consistent, reliable answers based on that stable foundation.

When RAG Struggles

RAG struggles when: Knowledge bases are messy or low-quality Garbage in, garbage out. If your documents are outdated, contradictory, poorly organized, or incomplete, RAG will retrieve and use that problematic content. Questions require reasoning beyond the documents RAG finds information. On its own, it doesn’t do new analysis. “Based on these market trends, what should our strategy be?” requires reasoning that goes beyond retrieval. Questions are too vague or too broad “Help me with my project” gives retrieval nothing to work with. Vague questions lead to vague retrieval, which leads to vague answers. The answer requires combining many sources If answering a question requires synthesizing information from 20 different documents in a complex way, retrieval might not surface all the pieces, and generation might struggle with the synthesis. Real-time or rapidly changing information RAG knowledge bases need to be updated. If information changes by the minute (stock prices, live events), RAG may not keep up. Questions are adversarial or tricky Users who try to trick the system, or ask deliberately confusing questions, can sometimes make RAG behave in unexpected ways.

The Needle in a Haystack Problem

One limit worth understanding: as knowledge bases get very large, finding exactly the right information gets harder. Imagine searching for one specific fact in a million documents. Even with good embeddings and semantic search, the right chunk might not surface in the top results. It’s there, but it’s buried among content that’s only somewhat related. Implications:
  • Smaller, focused knowledge bases often outperform massive ones
  • Curation matters; don’t include everything just because you can
  • For very large knowledge bases, extra organization becomes important, such as categories and metadata filtering (narrowing a search to chunks with certain tags, dates or sources)

Hallucinations Aren’t Gone

RAG reduces hallucinations (made-up details that sound right) but doesn’t eliminate them. Why hallucinations can still occur:
  1. The AI misinterprets the context: The retrieved chunk says one thing; the AI understands it differently.
  2. The AI fills gaps: If the context is incomplete, the AI might fill in plausible-sounding but inaccurate details.
  3. The AI over-generalizes: The context mentions one example; the AI treats it as a universal rule.
  4. The retrieval itself fails: If retrieval returns irrelevant chunks, the AI has bad material to work with.
The takeaway: RAG makes AI more reliable, not perfectly reliable. Critical information should still be verified.

Evaluating RAG Quality

To know whether your RAG system is working well, evaluate it systematically. Testing and Observability shows how to build a test set and score it repeatably. Retrieval evaluation:
  • Are the retrieved chunks actually relevant to the query?
  • Is the most relevant information making it into the top results?
  • Are important chunks being missed?
Generation evaluation:
  • Are responses faithful to the retrieved context?
  • Does the AI admit uncertainty when appropriate?
  • Are responses well-formatted and useful?
End-to-end evaluation:
  • Are users getting their questions answered?
  • How often do users find the responses helpful?
  • What percentage of queries lead to good outcomes?
Common metrics:
  • Relevance: Do retrieved chunks relate to the question?
  • Faithfulness: Does the response accurately reflect the retrieved information?
  • Helpfulness: Did the response actually help the user?
  • Groundedness: Is every claim in the response supported by retrieved content?

When to Consider Alternatives

Sometimes RAG isn’t the right tool. Consider alternatives when: You need pure reasoning: Math problems, logic puzzles or analysis that doesn’t depend on specific documents may be better served by the model on its own, or by a reasoning model. You need real-time data: For live information (current prices, weather, sports scores), APIs that fetch real-time data are a better fit than RAG. With tool use, a model can call those APIs for you. The task is creative: Writing fiction, brainstorming ideas, or generating creative content typically doesn’t benefit from retrieval. Simple lookups suffice: If users just need to look up a single fact from a structured database, a simple search or database query might be simpler than RAG. The domain is too specialized: If your domain is so specialized that general embedding models don’t understand the terminology, you might need custom models or different approaches. RAG is one tool among many. Knowing when to use it, and when not to, is a sign of expertise.

Try It Yourself

Exercise 1: Diagnose the Problem

For each scenario, identify whether the problem is with retrieval, generation, or the underlying knowledge base:
  1. The system gives an answer, but it’s from a policy that was updated last month. The old policy is still in the system.
  2. The system retrieves a relevant document about vacation policy, but the response says employees get 20 days when the document says 15.
  3. A user asks about a product feature, but the feature isn’t documented anywhere. The system confidently describes a feature that doesn’t exist.
  4. The system retrieves chunks about three different products when the user asked about one specific product.

Exercise 2: Design an Evaluation

You’re deploying a RAG system for customer support. Design an evaluation approach:
  1. What sample queries would you use for testing?
  2. How would you measure retrieval quality?
  3. How would you measure response quality?
  4. What would “good enough” look like for launch?
  5. How would you continue monitoring after launch?

Exercise 3: Right Tool for the Job

For each task, decide whether RAG is the right approach or suggest an alternative:
  1. Answering questions about your company’s HR policies
  2. Generating creative marketing slogans
  3. Calculating compound interest
  4. Finding relevant research papers on a topic
  5. Getting the current temperature in a city
  6. Explaining how your software product works

Common Pitfalls

Pitfall 1: Blaming RAG for Content Problems

When the system gives outdated or wrong answers, it’s tempting to blame the technology. But often the real problem is that the knowledge base has outdated or wrong content. The fix: When problems occur, check the source content first. Is the right information in the knowledge base?

Pitfall 2: Not Testing with Real Queries

Developers test with questions they think users will ask. Users ask questions developers never imagined. Real-world performance often differs from test performance. The fix: Collect real user queries (even before launch, through research) and evaluate against them. Let real usage guide improvements.

Pitfall 3: Ignoring Failure Cases

It’s tempting to focus on the times RAG works great and ignore the times it fails. But failure cases reveal the most about how to improve. The fix: Systematically review failures. Look for patterns. Address root causes rather than just celebrating successes.

Pitfall 4: Over-trusting Metrics

Metrics can mislead. High retrieval relevance scores don’t guarantee users are happy. Good-looking evaluations don’t mean the system works in practice. The fix: Combine quantitative metrics with qualitative review. Read actual responses. Talk to actual users.

Level Up

Here’s a diagnostic challenge: Scenario: You’ve deployed a RAG system for technical support. Users report that answers are “usually okay but sometimes completely wrong.” Leadership wants to know if RAG is the right approach or if you should abandon it. Your task: Design a systematic investigation.
  1. How would you gather data on where the system succeeds and fails?
  2. What categories of failure would you look for?
  3. How would you determine if failures are fixable (better retrieval, better prompts, better content) or fundamental limitations?
  4. What would you recommend to leadership?
Write out your investigation plan.

Key Takeaway

RAG works best with good source material, specific questions, and a need for grounded, verifiable responses. It struggles with poor-quality knowledge bases, complex reasoning, and very broad queries, and it reduces hallucinations without eliminating them. Evaluate systematically to know whether it’s meeting user needs.
Try it on Gloo: Evaluate Before You Ship shows how to test a grounded app before you launch it.

What’s next

One step remains before you build: making answers people can check. After that, Lesson 11: Getting Started with RAG gives you a clear roadmap for turning your RAG knowledge into RAG practice. For grounded apps in production, see Evaluate Before You Ship and the Grounded Responses guide.

Lesson 10: Citations and Verification

Show users where a grounded answer came from, check its claims against the retrieved sources, and decide what to do when no sources come back.