Skip to main content

Finding the Needle in the Haystack

You have a knowledge base with thousands of chunks, each converted to an embedding. A user asks a question. Now what? The retrieval step is where the system searches all that information for the chunks most relevant to the user’s question. Get this right, and the AI has great material to work with. Get it wrong, and even the smartest AI will give unhelpful answers. Here’s how retrieval works.

Core Concepts

From Question to Embedding

The retrieval process starts by converting the user’s question into an embedding using the same model that embedded the knowledge base. The question and the documents need to be in the same “embedding space” (the same map of meaning) to be compared, like a conversation where everyone speaks the same language. If the question is embedded with a different model than the documents, the comparisons mean nothing. So when a user asks “What’s the refund policy for digital products?”, that question becomes a vector of numbers, just like all the chunks in your knowledge base.

Similarity Search: Finding the Closest Matches

With the question and all the chunks represented as embeddings, we can compare them mathematically. The most common approach is cosine similarity, which measures the angle between two vectors:
  • Vectors pointing in the same direction: high similarity (close to 1)
  • Vectors at right angles to each other: little or no similarity (close to 0)
  • Vectors pointing in opposite directions: negative similarity (close to -1)
The system calculates the similarity between the question embedding and every chunk embedding in the knowledge base, then ranks them from most similar to least similar. Chunks at the top of this ranking are semantically closest to the question and most likely to contain relevant information.

Top-K Retrieval: Why We Don’t Just Take the Best Match

Why not just use the single most similar chunk? In practice, systems usually retrieve several: the “top K” chunks, where K is the number to retrieve, typically 3-10. Here’s why: 1. Relevant information might be spread across chunks The answer to “How do I apply for a refund and how long does it take?” might require information from two different sections. 2. The top match isn’t always the best Similarity scores are imperfect. The second or third match might actually be more useful for a particular question. 3. More context is often better Within reason, giving the AI more relevant context leads to more comprehensive answers. 4. Redundancy provides confirmation If multiple chunks say the same thing, the AI can be more confident in that information. K is a setting you tune. Too few chunks, and you might miss relevant information. Too many, and the useful parts get buried, or you hit the context window limit (how much text the model can read at once).

Beyond Simple Similarity: Ranking Strategies

Basic similarity search is a great start, but many RAG systems add more strategies. Some re-rank results: they re-score the first matches so the best ones end up on top. Others change how the search itself is done: Recency weighting: More recent documents might be more relevant, especially for policies or news. You can boost the scores of newer content. Authority weighting: Some sources are more authoritative than others. Official documentation might be weighted higher than blog posts. Diversity sampling: If the top 5 results all say the same thing, you might want to include a 6th result that offers a different perspective, even if its similarity score is lower. Query expansion: Sometimes the user’s question doesn’t perfectly match how information is phrased in documents. You can expand the query with synonyms or related terms to improve retrieval. Hybrid search: Combine semantic similarity with keyword matching. Sometimes exact keyword matches are important (like product names or technical terms that embeddings might not handle perfectly).

The Relevance Threshold

Not every query will have good matches in your knowledge base. If someone asks about a topic that isn’t covered in your documents, even the “best” matches might be irrelevant. Many systems use a relevance threshold: if no chunks score above a certain similarity level, the system might:
  • Return no results
  • Acknowledge that it doesn’t have information on this topic
  • Fall back to the AI’s general knowledge (with appropriate caveats)
This prevents the system from confidently presenting irrelevant information just because it was the “best” match available.

Speed vs. Accuracy Trade-offs

Comparing a question against millions of embeddings one by one would be slow. In practice, RAG systems use specialized tools: Vector databases: Purpose-built databases optimized for similarity search, like Pinecone, Weaviate, or Chroma. Approximate nearest neighbor (ANN) algorithms: Instead of comparing against every embedding, these algorithms use an index to quickly find matches that are very nearly the best. They give up a little accuracy for a large gain in speed. For most applications, these approximate methods are good enough: the chunk that’s technically ranked 5th might show up as 4th, but overall quality stays high.

Try It Yourself

Exercise 1: Think Like a Retrieval System

Imagine a knowledge base about a software product with these chunks:
  1. “Installation guide for Windows users…”
  2. “Pricing plans start at $10/month…”
  3. “To reset your password, click ‘Forgot Password’…”
  4. “System requirements: Windows 10+, 4GB RAM…”
  5. “Our refund policy allows returns within 30 days…”
For each user question, which chunk(s) would you expect to be retrieved?
  • “How do I install the software?”
  • “What are the hardware requirements?”
  • “I forgot my password”
  • “How much does it cost?”
  • “Can I get my money back?”
Now think: Are there questions where the right chunk might NOT be the top similarity match? (Hint: “What do I need to run this software?” might match “System requirements” better than “Installation guide,” even though both are relevant.)

Exercise 2: Design Your Top-K Strategy

For a customer support RAG system, consider:
  1. What value of K would you choose? Why?
  2. Would you use any ranking adjustments beyond similarity? (Recency? Authority?)
  3. What would you do when no results pass the relevance threshold?
There are no perfect answers; this builds your intuition for the trade-offs.

Exercise 3: Edge Cases

Think of questions that would be challenging for retrieval:
  1. Questions that use very different words than the source documents
  2. Questions that require combining information from multiple unrelated sections
  3. Questions about topics not covered in the knowledge base
  4. Questions with ambiguous terms that could match multiple topics
How might you handle each of these?

Common Pitfalls

Pitfall 1: Retrieving Based on Topic, Not Relevance

A chunk might be about the same general topic as the question but not actually answer it. “Our company history began in 1995…” is about the company but doesn’t help someone asking “What products do you sell?” The fix: Evaluate retrieval quality with real questions. Look at what actually gets retrieved and ask: does this help answer the question? Build a test set from your own examples shows how.

Pitfall 2: Ignoring Retrieval Failures

If retrieval returns poor results, the AI will generate poor responses. But it’s easy to blame the AI when the real problem is retrieval. The fix: When RAG responses are bad, check what was retrieved first. The generation step can only work with what retrieval provides.

Pitfall 3: One-Size-Fits-All K

Using the same number of retrieved chunks for every query isn’t optimal. A simple factual question might need just one chunk. A complex analysis might need ten. The fix: Consider dynamic retrieval strategies that adjust K based on query complexity or result quality.

Pitfall 4: Not Testing with Real Queries

Retrieval that works well for your test queries might fail for the queries real users ask. Users phrase things differently than you expect. The fix: Collect real user queries and regularly evaluate retrieval performance against them. Let real usage inform your tuning.

Level Up

Here’s a challenge to deepen your understanding: Scenario: You’re building a RAG system for a legal research application. Lawyers need to find relevant case law and statutes. Consider these complications:
  1. Legal documents use very specific terminology. How would you ensure retrieval handles this?
  2. A query like “cases involving breach of contract in California from 2020-2023” requires matching on concepts (breach of contract), location (California), and time range (2020-2023). How would pure semantic search handle this?
  3. Some legal precedents are more authoritative than others. How might you incorporate this?
Design a retrieval strategy that addresses these challenges. What combinations of techniques would you use?

Key Takeaway

Retrieval is where RAG succeeds or fails. Getting it right requires tuning: how many chunks to retrieve, what ranking strategies to use, and when to acknowledge that the knowledge base doesn’t have the answer. Bad retrieval leads to a bad answer, however good the model is.
Try it on Gloo: run a query against your uploaded content with Search, or try it in the browser with the Search Playground. Building Custom Search with the Search API turns it into a working app.

What’s next

You’ve found the relevant chunks; now they need to be combined with the user’s question in a way that helps the AI generate a great response. To run retrieval yourself on Gloo, see the Search guide, Run retrieval yourself and Part 2: Content Lifecycle of the RAG pipeline tutorial.

Lesson 6: The Augmentation Step

Learn about prompt templates, managing context length, and telling the AI how to use (and not misuse) the information you’ve provided.