> ## Documentation Index
> Fetch the complete documentation index at: https://docs.gloo.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Conversations and Context Windows

> Best practices for multi-turn conversations on Gloo AI: sending history, staying inside the context window, trimming and summarising, switching models mid-conversation, and keeping each turn affordable.

A chat feels continuous to your user, but each request to the model stands alone: your app sends the conversation every time. That makes long conversations slower and more expensive turn by turn, and eventually too big for the model. This page is for developers building chat and other multi-turn features who want conversations that stay coherent, fit the model and don't grow the bill without limit.

> [!TIP]
>
> New to conversation history? [Conversations That Remember](https://docs.gloo.com/ai-learning-center/gloo-ai-104/conversations-that-remember.md) in the AI Learning Center explains how history and the [context window](https://docs.gloo.com/ai-learning-center/gloo-ai-101/prompting-and-generation.md#context-window) work, with a working example.

This page covers how to:

* hold and send the conversation
* stay inside the model's context window
* trim or summarise long conversations
* switch models partway through a conversation
* keep each turn affordable

## Hold the history in your app

On every Gloo AI generation endpoint, your app holds the conversation and sends it with each request: in `input` on the Responses endpoints, in `messages` on Chat Completions. Instructions that apply to the whole conversation go in `instructions` (Responses) or a `system` message (Chat Completions), and you send them each time too.

* **Don't rely on server-side state.** Gloo doesn't store your conversations: `previous_response_id`, `conversation` and `store` don't keep earlier turns on Gloo's Responses endpoints, and on some models sending them can make the request fail. Leave them out and send the earlier turns in `input`. See [Moving off `encrypted_content` and `previous_response_id`](https://docs.gloo.com/api-guides/direct-responses#moving-off-encrypted_content-and-previous_response_id).
* **Store each conversation durably** with the turns as you sent them and as the model returned them, so a user can come back to it and you can investigate problems later.
* **Drop a question that got no answer.** If a request fails, don't keep the unanswered turn in the history, or the next request sends it twice.
* **Send output items back as you received them** on the Responses endpoints, including tool calls and their results. Reasoning from earlier turns isn't carried forward: the model reasons afresh each turn. See [Reasoning across turns](https://docs.gloo.com/api-guides/reasoning.md#reasoning-across-turns).

## Stay inside the context window

Each model's `context_window` and `max_input_tokens` are in the [model catalog](https://docs.gloo.com/api-guides/supported-models.md). Everything you send counts toward the input limit: instructions, every earlier turn, documents, images and tool definitions. The answer needs room too, up to the output limit you set.

* **Track the size of each conversation.** The `usage` on each response tells you how many input tokens the last request used (`input_tokens` on Responses, `prompt_tokens` on Chat Completions). The next request is roughly that, plus the answer, plus the new message.
* **Set your own budget below the model's limit.** Trimming at, say, half of `max_input_tokens` keeps requests fast and affordable and leaves room for long messages.
* **Handle the error anyway.** A request bigger than the model's window fails with `400` and the code `context_length_exceeded` (`1002 / CONTEXT_LENGTH_EXCEEDED`). It isn't retryable: shorten the conversation, or move to a model with a larger window, before you send it again. The error body includes the model's `max_input_tokens` and a `suggestion` naming a model with a larger window. See the [error code reference](https://docs.gloo.com/api-reference/general/errors.md#error-code-reference).

## Trim or summarise long conversations

Pick a strategy before your first long conversation, not after the first error.

* **Trim from the oldest end.** Keep the instructions and the most recent turns; drop older ones until the conversation fits your budget. Trim by size as well as count, because a few long answers can outweigh many short ones.
* **Keep tool calls with their results.** A `function_call` and its `function_call_output` share a `call_id` and belong together: if you drop one, drop the other.
* **Summarise what you drop.** For long-running conversations, have a small, fast model summarise the dropped turns into a short note, and send that note at the start of the history. Mark it as a summary so the model doesn't treat it as something the user just said.
* **Move lasting facts into the instructions.** A user's name, a choice they made or the document you're discussing shouldn't depend on an old turn surviving the trim.

This trims a Responses-shaped history to a budget, estimating size from characters and keeping tool calls with their results. Calibrate the estimate against `usage.input_tokens` for your own traffic:

```python Python theme={null}
import json

CHARS_PER_TOKEN = 4  # rough for English text; check against usage.input_tokens


def estimate_tokens(item: dict) -> int:
    return len(json.dumps(item)) // CHARS_PER_TOKEN + 1


def trim(history: list[dict], budget_tokens: int) -> list[dict]:
    """Keep the newest items that fit the budget, never splitting a tool call from its result."""
    # A tool call and its output share a call_id and are kept or dropped as one unit.
    unit_tokens: dict[str, int] = {}
    for item in history:
        if "call_id" in item:
            unit_tokens[item["call_id"]] = unit_tokens.get(item["call_id"], 0) + estimate_tokens(item)
    kept: list[dict] = []
    kept_calls: set[str] = set()
    total = 0
    full = False
    for item in reversed(history):
        call_id = item.get("call_id")
        if call_id in kept_calls:
            kept.append(item)  # the other half of a pair already counted
            continue
        if full:
            continue
        size = unit_tokens[call_id] if call_id else estimate_tokens(item)
        if total + size > budget_tokens:
            if not kept:
                raise ValueError("The newest item is over the budget on its own; shorten it before sending.")
            full = True
            continue
        kept.append(item)
        total += size
        if call_id:
            kept_calls.add(call_id)
    kept.reverse()
    return kept
```

```typescript TypeScript theme={null}
const CHARS_PER_TOKEN = 4; // rough for English text; check against usage.input_tokens

function estimateTokens(item: unknown): number {
  return Math.floor(JSON.stringify(item).length / CHARS_PER_TOKEN) + 1;
}

/** Keep the newest items that fit the budget, never splitting a tool call from its result. */
function trim(history: any[], budgetTokens: number): any[] {
  // A tool call and its output share a call_id and are kept or dropped as one unit.
  const unitTokens = new Map<string, number>();
  for (const item of history) {
    if (item.call_id) unitTokens.set(item.call_id, (unitTokens.get(item.call_id) ?? 0) + estimateTokens(item));
  }
  const kept: any[] = [];
  const keptCalls = new Set<string>();
  let total = 0;
  let full = false;
  for (let i = history.length - 1; i >= 0; i--) {
    const item = history[i];
    if (item.call_id && keptCalls.has(item.call_id)) {
      kept.push(item); // the other half of a pair already counted
      continue;
    }
    if (full) continue;
    const size = item.call_id ? unitTokens.get(item.call_id)! : estimateTokens(item);
    if (total + size > budgetTokens) {
      if (kept.length === 0) throw new Error("The newest item is over the budget on its own; shorten it before sending.");
      full = true;
      continue;
    }
    kept.push(item);
    total += size;
    if (item.call_id) keptCalls.add(item.call_id);
  }
  return kept.reverse();
}
```

If the newest item (or newest tool call with its result) is over the budget on its own, `trim` raises an error instead of returning a history that would still be too long: catch it and shorten or summarise that item. Send your instructions separately, in `instructions`, so trimming never removes them. Character counts are only an estimate, and images count very differently from text, so leave a margin.

## Switch models with care

Routing a conversation between models, for cost or capability, is common. Check that the next model can take the whole conversation as it stands:

* **Context window.** A conversation that fits a 1,000,000-token model may not fit a smaller one. Trim to the smaller model's budget before you switch.
* **Images.** A conversation that holds an image can only go to a model with `supports_vision: true`, or you can describe the image once and send the description instead. See [Switching models mid-conversation](https://docs.gloo.com/best-practices/images-in-conversations.md#switching-models-mid-conversation).
* **Tools.** A conversation with tool calls in it needs a model that supports tools if you still send `tools`.
* **Replacements.** When a model you use is deprecated, its requests can be served by a replacement with a smaller context window; `deprecation_note` says so when that happens. See [Choosing, Pinning and Updating Models](https://docs.gloo.com/best-practices/choosing-and-updating-models.md).
* **Format.** Keep history in plain messages and standard output items, not provider-specific blocks, so any model can read it.

## Keep each turn affordable

Because the whole conversation is sent each turn, every turn bills all the turns before it as input again.

* **Append, don't rewrite.** Add new turns at the end and leave earlier ones exactly as they were. The unchanged start of the conversation can then be read from the [prompt cache](https://docs.gloo.com/ai-learning-center/gloo-ai-104/cost-and-speed.md#prompt-caching) at a lower rate. Rewriting or reordering earlier turns breaks the cache.
* **Put stable content first:** instructions, then reference documents, then the conversation, then the new question.
* **Trim earlier rather than later.** A smaller budget costs less on every turn, not just the long ones.
* **Shrink images or describe them once.** An image is billed as input tokens on every turn it stays in the history.

See [Controlling Cost](https://docs.gloo.com/best-practices/cost-control.md) for the rest of the levers.

## Summary

* Your app sends the whole conversation every turn; Gloo doesn't keep it, so don't rely on `previous_response_id`.
* Track size from `usage`, trim to your own budget below `max_input_tokens`, and handle `context_length_exceeded` without retrying.
* Trim from the oldest end, keep tool calls with their results, and summarise what you drop.
* Before switching models, check the context window, vision and tool support.
* Append new turns without rewriting old ones, so the start of the conversation stays cacheable.

## Next steps

> [card] [Conversations That Remember](https://docs.gloo.com/ai-learning-center/gloo-ai-104/conversations-that-remember.md)
>
> The lesson: history, context windows and trimming, with code.

> [card] [Images in Conversations](https://docs.gloo.com/best-practices/images-in-conversations.md)
>
> Keep conversations with images working across models.

> [card] [Direct Responses](https://docs.gloo.com/api-guides/direct-responses.md#multi-turn-behavior)
>
> How earlier turns and reasoning items are handled.

> [card] [Controlling Cost](https://docs.gloo.com/best-practices/cost-control.md)
>
> Caching, trimming and the other levers on your bill.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.