Lesson 2: Conversations That Remember

The Model Doesn’t Remember You

The model has no memory between requests. Each request starts fresh. If a user asks “What about the second one?”, the model only knows what “the second one” means if your app sends the earlier messages along with the question. So a conversation is something your app builds. You keep the history, send it with every request, and decide what to drop when it gets too long.

Core Concepts

Your App Holds the History

On Gloo, with every request, your app sends the whole conversation so far:

On the Responses endpoints, the conversation goes in input, as a list of messages with a role (user or assistant) and content.
On the Chat Completions endpoints, it goes in messages.

OpenAI’s server-side conversation features, such as previous_response_id and conversation, aren’t supported on Gloo’s Responses endpoints. If you’re moving code that uses them, send the earlier turns in input instead. Instructions that apply to the whole conversation, such as the assistant’s role, go in instructions on Responses (or a system message on Chat Completions). You send them each time too.

The Context Window

Every model has a context window: the most tokens it can take in one request. (A token is a small piece of text, often part of a word.) Everything you send counts toward it: the instructions, every earlier message and the new question. Context windows vary a lot between models. The model catalog shows each model’s context_window and max_input_tokens. If a request is bigger than the model’s limit, it fails with a 400 error and the code context_length_exceeded (1002). This error isn’t retryable: sending the same request again fails the same way. The error body includes the model’s max_input_tokens and a suggestion naming a model with a larger window.

Long Conversations Cost More

Because you resend the history each time, every turn makes the next request bigger. A 30-turn conversation sends all 30 turns again on turn 31. You pay for those input tokens each time, and big requests take longer. Lesson 8 shows how prompt caching can lower the cost of the part that repeats.

Trimming

Trimming means dropping the oldest turns before you send the conversation, so it stays inside a limit you choose. A simple approach:

Always keep the instructions.
Keep only the most recent messages, up to a maximum count.
If that’s still too long, drop more of the oldest turns until it fits a size limit.

Trim by size as well as by count. A few long answers can be bigger than many short ones.

Get Started

This example keeps a conversation in a list, trims it before each request and adds the model’s answer back in. If a request fails, it drops the unanswered question so the next request doesn’t send it again.

pip install openai

Python

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["GLOO_API_KEY"],
    base_url="https://platform.ai.gloo.com/ai/v2/direct",
)

INSTRUCTIONS = "You are a friendly assistant for a church's events team. Keep answers short."
MAX_MESSAGES = 20
MAX_CHARS = 60_000

history = []  # the conversation so far: user and assistant messages


def trim_history(messages):
    """Keep the most recent messages, by count and then by size."""
    recent = messages[-MAX_MESSAGES:]
    # Always keep at least the newest message.
    while len(recent) > 1 and sum(len(m["content"]) for m in recent) > MAX_CHARS:
        recent.pop(0)
    return recent


def ask(question):
    history.append({"role": "user", "content": question})
    history[:] = trim_history(history)  # store the trimmed form, not just send it
    try:
        response = client.responses.create(
            model="gloo-anthropic-claude-sonnet-5.5",
            instructions=INSTRUCTIONS,
            input=history,
        )
    except Exception:
        history.pop()  # don't keep a question that got no answer
        raise
    answer = response.output_text
    history.append({"role": "assistant", "content": answer})
    return answer


print(ask("Suggest three ideas for a summer family night."))
print(ask("Make the second one work for a rainy evening."))

The second question only makes sense because the first question and answer are in history. Remove them and the model has no idea what “the second one” is. The request body on the wire looks like this:

{
  "model": "gloo-anthropic-claude-sonnet-5.5",
  "instructions": "You are a friendly assistant for a church's events team. Keep answers short.",
  "input": [
    { "role": "user", "content": "Suggest three ideas for a summer family night." },
    { "role": "assistant", "content": "1. Outdoor movie night..." },
    { "role": "user", "content": "Make the second one work for a rainy evening." }
  ]
}

Handle the Error Anyway

Trimming makes the error rare, but a single very long message can still go over. Catch it and tell the user, rather than retrying. On the direct endpoints, every error carries an X-Gloo-Error-Code header to branch on:

import openai

def ask_safely(question):
    try:
        return ask(question)
    except openai.BadRequestError as err:
        if err.response.headers.get("X-Gloo-Error-Code") == "context_length_exceeded":
            return "That message is too long for this model. Try a shorter one."
        raise

With Streaming

If you stream the answer, as in Lesson 1: Streaming Responses, the history works the same way, except when you add the answer: collect the text deltas as they arrive, then append the full answer to history once the stream completes. If the stream fails or stops partway, don’t store the partial text as if it were a complete answer. Drop the unanswered question instead, as ask does on an error:

import openai

def ask_streaming(question):
    history.append({"role": "user", "content": question})
    history[:] = trim_history(history)
    parts, completed = [], False
    try:
        with client.responses.create(
            model="gloo-anthropic-claude-sonnet-5.5",
            instructions=INSTRUCTIONS,
            input=history,
            stream=True,
        ) as stream:  # closes the stream however the loop ends
            for event in stream:
                if event.type == "response.output_text.delta":
                    print(event.delta, end="", flush=True)
                    parts.append(event.delta)
                elif event.type == "response.completed":
                    completed = True
    except openai.APIError:
        pass  # the stream dropped partway: handled below
    finally:
        if not completed:
            history.pop()  # even on Ctrl+C: don't keep the unanswered question
    if not completed:
        print("\n[The answer was interrupted.]")  # the user saw part of an answer
        return None
    print()
    answer = "".join(parts)
    history.append({"role": "assistant", "content": answer})
    return answer

Good to Know

Trim from the oldest end. The latest question and the turns just before it matter most. Keep the instructions at the top no matter what.
Keep facts the conversation depends on. If an early turn holds something the user still needs, such as a name or a choice they made, trimming will drop it. Put lasting facts in instructions instead.
Sending output back is fine. On the Responses endpoints you can send output items back in input as you received them. Reasoning from earlier turns isn’t carried forward: the model reasons afresh each turn from the conversation you send. See Reasoning across turns.
Grounded apps need a little more. If you answer from your own content, keep the sources for each turn so follow-ups like “which source says that?” use the same ones. Building Trustworthy Grounded Applications covers this.

Key Takeaway

The model only knows what you send it. Keep the conversation in your app, send it with every request, and trim the oldest turns, by size as well as count, so it stays inside the model’s context window. Handle context_length_exceeded with a clear message instead of a retry.

Try it on Gloo: Direct Responses takes the conversation in input and your instructions in instructions. The Responses tutorial has a multi-turn example in several languages.

Learn more

Direct Responses: Request format: the input and instructions fields
Building Trustworthy Grounded Applications: Multi-Turn Conversations: trimming and keeping sources across turns
Errors: Error Code Reference: 1002 / CONTEXT_LENGTH_EXCEEDED and the rest
Supported Models: each model’s context window

What’s next

Next, you’ll get the model to return JSON your code can use, and check it before you trust it.

Lesson 3: Structured Output

JSON schemas, and validating the result with Pydantic.

AI 101: Fundamentals

AI 102: Prompting

AI 103: RAG

AI 104: Building with Gloo

Lesson 2: Conversations That Remember

The Model Doesn’t Remember You

Core Concepts

Your App Holds the History

The Context Window

Long Conversations Cost More

Trimming

Get Started

Handle the Error Anyway

With Streaming

Good to Know

Key Takeaway

Learn more

What’s next

Lesson 3: Structured Output

​The Model Doesn’t Remember You

​Core Concepts

​Your App Holds the History

​The Context Window

​Long Conversations Cost More

​Trimming

​Get Started

​Handle the Error Anyway

​With Streaming

​Good to Know

​Key Takeaway

​Learn more

​What’s next

Lesson 3: Structured Output

The Model Doesn’t Remember You

Core Concepts

Your App Holds the History

The Context Window

Long Conversations Cost More

Trimming

Get Started

Handle the Error Anyway

With Streaming

Good to Know

Key Takeaway

Learn more

What’s next