Conversations and Context Windows

• hold and send the conversation
• stay inside the model’s context window
• trim or summarise long conversations
• switch models partway through a conversation
• keep each turn affordable

​

Hold the history in your app

• Don’t rely on server-side state. Gloo doesn’t store your conversations: previous_response_id, conversation and store don’t keep earlier turns on Gloo’s Responses endpoints, and on some models sending them can make the request fail. Leave them out and send the earlier turns in input. See Moving off encrypted_content and previous_response_id.
• Store each conversation durably with the turns as you sent them and as the model returned them, so a user can come back to it and you can investigate problems later.
• Drop a question that got no answer. If a request fails, don’t keep the unanswered turn in the history, or the next request sends it twice.
• Send output items back as you received them on the Responses endpoints, including tool calls and their results. Reasoning from earlier turns isn’t carried forward: the model reasons afresh each turn. See Reasoning across turns.

​

Stay inside the context window

• Track the size of each conversation. The usage on each response tells you how many input tokens the last request used (input_tokens on Responses, prompt_tokens on Chat Completions). The next request is roughly that, plus the answer, plus the new message.
• Set your own budget below the model’s limit. Trimming at, say, half of max_input_tokens keeps requests fast and affordable and leaves room for long messages.
• Handle the error anyway. A request bigger than the model’s window fails with 400 and the code context_length_exceeded (1002 / CONTEXT_LENGTH_EXCEEDED). It isn’t retryable: shorten the conversation, or move to a model with a larger window, before you send it again. The error body includes the model’s max_input_tokens and a suggestion naming a model with a larger window. See the error code reference.

​

Trim or summarise long conversations

• Trim from the oldest end. Keep the instructions and the most recent turns; drop older ones until the conversation fits your budget. Trim by size as well as count, because a few long answers can outweigh many short ones.
• Keep tool calls with their results. A function_call and its function_call_output share a call_id and belong together: if you drop one, drop the other.
• Summarise what you drop. For long-running conversations, have a small, fast model summarise the dropped turns into a short note, and send that note at the start of the history. Mark it as a summary so the model doesn’t treat it as something the user just said.
• Move lasting facts into the instructions. A user’s name, a choice they made or the document you’re discussing shouldn’t depend on an old turn surviving the trim.

import json

CHARS_PER_TOKEN = 4  # rough for English text; check against usage.input_tokens


def estimate_tokens(item: dict) -> int:
    return len(json.dumps(item)) // CHARS_PER_TOKEN + 1


def trim(history: list[dict], budget_tokens: int) -> list[dict]:
    """Keep the newest items that fit the budget, never splitting a tool call from its result."""
    # A tool call and its output share a call_id and are kept or dropped as one unit.
    unit_tokens: dict[str, int] = {}
    for item in history:
        if "call_id" in item:
            unit_tokens[item["call_id"]] = unit_tokens.get(item["call_id"], 0) + estimate_tokens(item)
    kept: list[dict] = []
    kept_calls: set[str] = set()
    total = 0
    full = False
    for item in reversed(history):
        call_id = item.get("call_id")
        if call_id in kept_calls:
            kept.append(item)  # the other half of a pair already counted
            continue
        if full:
            continue
        size = unit_tokens[call_id] if call_id else estimate_tokens(item)
        if total + size > budget_tokens:
            if not kept:
                raise ValueError("The newest item is over the budget on its own; shorten it before sending.")
            full = True
            continue
        kept.append(item)
        total += size
        if call_id:
            kept_calls.add(call_id)
    kept.reverse()
    return kept

const CHARS_PER_TOKEN = 4; // rough for English text; check against usage.input_tokens

function estimateTokens(item: unknown): number {
  return Math.floor(JSON.stringify(item).length / CHARS_PER_TOKEN) + 1;
}

/** Keep the newest items that fit the budget, never splitting a tool call from its result. */
function trim(history: any[], budgetTokens: number): any[] {
  // A tool call and its output share a call_id and are kept or dropped as one unit.
  const unitTokens = new Map<string, number>();
  for (const item of history) {
    if (item.call_id) unitTokens.set(item.call_id, (unitTokens.get(item.call_id) ?? 0) + estimateTokens(item));
  }
  const kept: any[] = [];
  const keptCalls = new Set<string>();
  let total = 0;
  let full = false;
  for (let i = history.length - 1; i >= 0; i--) {
    const item = history[i];
    if (item.call_id && keptCalls.has(item.call_id)) {
      kept.push(item); // the other half of a pair already counted
      continue;
    }
    if (full) continue;
    const size = item.call_id ? unitTokens.get(item.call_id)! : estimateTokens(item);
    if (total + size > budgetTokens) {
      if (kept.length === 0) throw new Error("The newest item is over the budget on its own; shorten it before sending.");
      full = true;
      continue;
    }
    kept.push(item);
    total += size;
    if (item.call_id) keptCalls.add(item.call_id);
  }
  return kept.reverse();
}

​

Switch models with care

• Context window. A conversation that fits a 1,000,000-token model may not fit a smaller one. Trim to the smaller model’s budget before you switch.
• Images. A conversation that holds an image can only go to a model with supports_vision: true, or you can describe the image once and send the description instead. See Switching models mid-conversation.
• Tools. A conversation with tool calls in it needs a model that supports tools if you still send tools.
• Replacements. When a model you use is deprecated, its requests can be served by a replacement with a smaller context window; deprecation_note says so when that happens. See Choosing, Pinning and Updating Models.
• Format. Keep history in plain messages and standard output items, not provider-specific blocks, so any model can read it.

​

Keep each turn affordable

• Append, don’t rewrite. Add new turns at the end and leave earlier ones exactly as they were. The unchanged start of the conversation can then be read from the prompt cache at a lower rate. Rewriting or reordering earlier turns breaks the cache.
• Put stable content first: instructions, then reference documents, then the conversation, then the new question.
• Trim earlier rather than later. A smaller budget costs less on every turn, not just the long ones.
• Shrink images or describe them once. An image is billed as input tokens on every turn it stays in the history.

​

Summary

• Your app sends the whole conversation every turn; Gloo doesn’t keep it, so don’t rely on previous_response_id.
• Track size from usage, trim to your own budget below max_input_tokens, and handle context_length_exceeded without retrying.
• Trim from the oldest end, keep tool calls with their results, and summarise what you drop.
• Before switching models, check the context window, vision and tool support.
• Append new turns without rewriting old ones, so the start of the conversation stays cacheable.

​

Next steps