Skip to main content

Why Your AI Assistant Doesn’t Know About Yesterday

Here’s a frustrating scenario you might have experienced: You ask an AI assistant about a news event from last week, and it confidently tells you it doesn’t have information about events after its training cutoff date. Or worse, it makes something up that sounds plausible but is completely wrong. You’ve just hit one of AI’s basic limits. Understanding it is the first step toward solving it. RAG (Retrieval Augmented Generation) means the AI retrieves relevant information first, augments (adds to) its prompt with what it found, and then generates an answer. By the end of this course, you’ll know how to make AI much more useful by connecting it to accurate, up-to-date information.

Core Concepts

The Knowledge Cutoff Problem

Every AI model was trained on data up to a certain date, called its knowledge cutoff. The model knows nothing about what happened after that date. It’s like a brilliant colleague who has been away for a long time: they know a lot about everything up to the day they left, and nothing about what happened since. This creates real problems:
  • Ask about recent events, and the AI is clueless
  • Ask about your company’s latest product, and it has no idea
  • Ask about a paper published last month, and it can’t help
The AI simply doesn’t have that information. A model doesn’t learn new things after training, and on its own it can’t see your company’s documents or today’s news. Some assistants add a web search tool, but that doesn’t give them your private material.

When AI Confidently Makes Things Up

Here’s something trickier than “I don’t know”: sometimes AI fills gaps with fiction that sounds confident. AI predicts the most likely next words based on patterns (as earlier lessons covered). When it lacks good information, it doesn’t stop and say “I’m uncertain.” It generates text that sounds like a reasonable answer based on similar patterns. This is called hallucination, and it’s one of the biggest challenges with AI assistants. The AI might:
  • Invent statistics that don’t exist
  • Create fake citations to papers that were never written
  • Describe events that never happened
  • Attribute quotes to people who never said them
The dangerous part? These hallucinations often sound just as confident and well-written as accurate responses. There’s no flashing warning sign that says “I made this up.”

The Difference Between Knowing Everything and Knowing the Right Things

Here’s a surprising point: we don’t need AI to know everything. We need it to know the right things for our specific situation. Imagine you’re a customer support agent for a software company. You don’t need an AI that knows every fact in the world. You need an AI that knows:
  • Your product’s features and limitations
  • Your company’s policies and procedures
  • Your documentation and help articles
  • Common customer questions and their answers
A general-purpose AI might know a little about everything but not enough about your domain to be truly helpful. You want an AI that can access your specific knowledge when it needs it.

What If AI Could Look Things Up?

Instead of relying solely on what it learned during training, what if the AI could:
  1. Receive your question
  2. Search through relevant documents to find information that might help
  3. Read that information
  4. Generate a response based on what it just found
This is exactly what RAG does. It’s like giving the AI access to a library and telling it, “Before you answer, go check the relevant books first.” Think of it as the difference between:
  • Closed-book exam: Answer from memory only (traditional AI)
  • Open-book exam: Look up information before answering (RAG)
Which would you trust more for accurate, specific answers? The open-book version. That’s what RAG gives you.

Try It Yourself

Exercise 1: Spot the Limitation

Open your favorite AI assistant and try these prompts:
  1. “What were the top news stories from last week?”
  2. “What’s the current price of [any stock]?”
  3. “What’s the weather like right now in New York?”
Notice how the AI either admits it doesn’t know, gives outdated information, or (in some cases) makes something up. This is the knowledge cutoff problem in action.

Exercise 2: Domain-Specific Questions

Now try asking about something very specific to a niche domain:
  1. “What are the specific return policies for [a local store you know]?”
  2. “What’s the process for submitting expense reports at [your company]?”
  3. “What are the side effects of [a specific medication] according to the latest research?”
The AI might give generic answers, but it won’t have the specific, authoritative information you need. This is where RAG shines.

Exercise 3: Hallucination Hunting

Ask the AI to provide specific citations or statistics on a topic you know well. For example:
“Give me three academic papers about [topic you’re expert in] with authors and publication years.”
Then verify: Do those papers actually exist? You might be surprised how often the AI invents plausible-sounding but completely fictional sources.

Common Pitfalls

Pitfall 1: Assuming AI “Just Knows”

Many people treat AI like an all-knowing oracle and assume its answers are accurate. But AI’s knowledge is limited to its training data, and even within that data, it can make mistakes. The fix: Always consider whether your question requires information the AI might not have (recent events, specific organizations, specialized domains).

Pitfall 2: Not Recognizing Hallucinations

Hallucinated content often sounds perfectly reasonable. It uses proper grammar, follows logical structure, and presents information confidently. The only way to catch hallucinations is to verify important claims. The fix: For any information that matters, verify it against authoritative sources. Don’t assume confident-sounding text is accurate. In an app, Building Trustworthy Grounded Applications shows checks you can run in code.

Pitfall 3: Thinking RAG Is Just “Better Search”

Some people hear about RAG and think it’s just a fancy search engine. But RAG does more than find information. It combines what it finds into a clear answer to your question. The fix: Think of RAG as search plus writing: search finds the information, and the AI turns it into an answer.

Pitfall 4: Expecting RAG to Fix Everything

RAG helps with many questions, but not all of them. If your knowledge base has bad information, RAG will use that bad information. If your question requires reasoning beyond what’s in the documents, RAG alone won’t solve it. The fix: Think of RAG as a tool that makes AI more informed, not omniscient. The quality of outputs depends heavily on the quality of inputs.

Level Up

Here’s a challenge to test your understanding: Scenario: You’re building a customer support chatbot for a software company. The chatbot needs to answer questions about the product, troubleshoot common issues, and explain company policies. Your task: List three specific types of questions where a standard AI would struggle but a RAG-enabled AI would excel. For each, explain:
  1. Why standard AI would fail
  2. What information RAG would need to retrieve
  3. How the answer would be better with RAG

Key Takeaway

AI models don’t know about recent events, lack domain-specific knowledge, and sometimes make things up. RAG addresses these limits by letting AI look up relevant information before it answers: an open-book exam instead of a closed-book one.
Try it on Gloo: Grounded Completions answer from content you upload, so the model can look things up instead of guessing. Step 1 of the RAG chatbot tutorial shows the difference on a real question.

What’s next

You now understand why RAG exists and what problems it solves, but how does the AI actually “look things up” and use what it finds? To see the fix on Gloo, read the Grounded Responses guide, try the RAG chatbot tutorial, and see the failure modes to design against.

Lesson 2: How RAG Actually Works

Break the RAG process into three clear steps (Retrieve, Augment, and Generate), and see how to use RAG effectively and where things can go wrong.