Skip to main content

Where Retrieval Meets Prompting

You’ve retrieved relevant chunks from your knowledge base. The next step, augmentation, is easy to overlook but important to get right. You take the user’s question, add the retrieved context, and write a prompt that helps the AI give a useful, grounded answer (one based on the retrieved material). If you’ve taken AI 102: Prompting, this is where those skills pay off in RAG. On Gloo, the grounded endpoints do this step for you: they add the passages they retrieve to the model’s context before it answers. Knowing how augmentation works still helps you write better instructions and understand the answers you get.

Core Concepts

The Anatomy of an Augmented Prompt

A basic augmented prompt has three components: 1. Instructions: Tell the AI how to behave and use the provided context. 2. Retrieved Context: The chunks retrieved from your knowledge base. 3. User Question: What the user actually asked. Here’s what a simple template might look like:
The structure matters. Instructions at the top set how the AI should behave. The context comes next. The question at the end makes clear what needs answering.

Instruction Design: Setting the Rules

The instructions establish how the AI should handle the task. Common elements include: Role setting: “You are a customer support agent for Acme Corp.” Context usage guidance: “Base your answer on the provided context. Do not use information from outside the context.” Handling uncertainty: “If the context doesn’t contain enough information, say ‘I don’t have enough information to answer that fully.’” Tone and format: “Answer in a friendly, professional tone. Use bullet points for lists.” Constraints: “Do not make up information. Do not speculate beyond what the context supports.” These instructions shape everything that follows. Without them, the AI might hallucinate, ignore the context, or respond in ways that don’t fit your use case.

Presenting Retrieved Context

How you present the retrieved chunks affects how well the AI uses them: Simple concatenation: Join all chunks with line breaks. Simple, but it can get messy.
Labeled chunks: Add markers to help the AI track sources.
Structured format: Use clear delimiters and organization.
Labeled approaches help the AI cite sources and distinguish between different types of information.

Managing Context Length

Every AI model has a context window, the maximum amount of text it can handle in one request. Every token (a word or part of a word) in your instructions, context, question and the response counts against that limit. This creates trade-offs: More context = More information for the AI to work with, but:
  • Higher cost (most APIs charge by token)
  • Risk of hitting context limits
  • Possible dilution of focus (too much noise)
Less context = Cheaper and more focused, but:
  • Might miss relevant information
  • Less comprehensive answers
Strategies for managing context length: Compress or summarize: Instead of including full chunks, include summaries. Prioritize: If you retrieved 10 chunks, maybe include only the top 5. Truncate strategically: If a chunk is too long, include the most relevant portion. Use models with larger context windows: When available and cost-effective. The right balance depends on your use case, budget, and quality requirements.

Instructing the AI on Context Usage

Tell the AI how to use (and not misuse) the context. Retrieved text is data, not instructions; see Keep outside content as data. Without clear instructions, you might encounter: Ignoring context: The AI answers from general knowledge instead of the provided information. Overclaiming: The AI states things definitively that the context only implies or doesn’t support. Mixing context with hallucination: The AI starts with real information but adds fabricated details. Effective instructions address these risks:

Handling Multiple Chunks

When you retrieve multiple chunks, they might: Overlap: Say the same thing in different words Complement: Each provides different pieces of the answer Conflict: Provide contradictory information Your prompt template should help the AI navigate these situations:

Question Reformulation

Sometimes the user’s question could be improved before being sent to the AI. Common enhancements: Adding specificity: If the user asks “What’s the policy?”, but you know from context they’re looking at the refund page, you might expand to “What’s the refund policy?” Removing ambiguity: Clarifying pronouns or vague references based on conversation history. Decomposing complex questions: Breaking “What’s the price and how do I sign up?” into sub-questions that can be addressed separately. This is more advanced, but it can improve answers a lot.

Try It Yourself

Exercise 1: Write a Prompt Template

Design a prompt template for a RAG system that answers questions about a company’s product documentation. Include:
  1. A role/persona for the AI
  2. Clear instructions on how to use the context
  3. Guidance on what to do when information is missing
  4. Placeholders for retrieved context and user question
Test your template mentally with a few example questions. Would it produce good results?

Exercise 2: Compare Template Approaches

Here are two different instruction sets. For each, consider what kind of responses they would produce: Template A:
Template B:
Which would you trust more for customer support? Why?

Exercise 3: Handle Conflicting Context

Imagine you retrieve these two chunks: Chunk 1 (from 2022 policy): “Refunds are available within 14 days of purchase.” Chunk 2 (from 2024 policy update): “Refunds are available within 30 days of purchase.” How would you structure your prompt to help the AI handle this conflict? Write out the specific instructions you would include.

Common Pitfalls

Pitfall 1: No Instructions at All

Giving the AI context and a question with no guidance leads to inconsistent, unpredictable responses. The fix: Always include clear instructions. Even a few sentences make a difference.

Pitfall 2: Instructions Too Vague

“Answer based on the context” is a start, but doesn’t address edge cases. What if the context is insufficient? What if it conflicts? The fix: Think through edge cases and address them explicitly in your instructions.

Pitfall 3: Overloading Context

Stuffing every retrieved chunk into the prompt, regardless of relevance or length, leads to bloated prompts that dilute focus and inflate costs. The fix: Be selective. Prioritize quality over quantity in what you include.

Pitfall 4: Ignoring Source Attribution

If the AI responds without indicating where information came from, users can’t verify the answer and trust erodes. The fix: Include instructions for the AI to reference sources. Use labeled chunks so the AI can cite them.

Level Up

Here’s a design challenge: Scenario: You’re building a RAG system for a medical information service. Users ask health questions, and the system retrieves from peer-reviewed medical literature. Constraints:
  • Responses must not be interpreted as personal medical advice
  • Sources must be clearly cited
  • Uncertainty must be acknowledged
  • Information should be accessible to non-medical users
Your task: Design a complete prompt template that addresses all these requirements. Include:
  1. An appropriate role/persona
  2. Detailed instructions on context usage and limitations
  3. Specific guidance on disclaimers and uncertainty
  4. A format for presenting retrieved context
  5. Instructions for generating the response

Key Takeaway

Augmentation is where retrieval meets prompting, and the quality of your augmented prompt directly affects the quality of the answer. A good one gives clear instructions on using the context, presents retrieved information in a well-organized way, and covers edge cases like missing information or conflicting sources.
Try it on Gloo: Step 4 of the Embeddings tutorial builds an augmented prompt from retrieved chunks. With Grounded Responses, Step 3 of the tutorial tells the model how to use the sources Gloo retrieves.

What’s next

You’ve prepared a strong augmented prompt; now the AI generates a response. For instructions you can reuse, see The Prompt Contract, the Grounded Responses guide and Step 3 of the Grounded Responses tutorial.

Lesson 7: The Generation Step

Explore how the AI uses retrieved context to produce grounded responses: staying faithful to source material, encouraging appropriate uncertainty, and making responses verifiable through citations.