> ## Documentation Index > Fetch the complete documentation index at: https://docs.gloo.com/llms.txt > Use this file to discover all available pages before exploring further. # Images in Conversations > Best practices for image input (vision) with Claude, GPT and Gemini on Gloo AI: pick a model, switch models safely, keep images small, and control cost. Photos, screenshots and scanned pages make many apps more useful: a picture of a fridge becomes a recipe, a screenshot of a text message becomes a suggested reply. Every Claude, Gemini and Grok chat model on Gloo AI reads images, as does every OpenAI GPT model except the open-weight GPT OSS models, through the same Responses and Chat Completions requests you already send. This page covers how to do it well in production. > [!TIP] > > New to image input? [Images In and Out](https://docs.gloo.com/ai-learning-center/gloo-ai-104/images-in-and-out.md) explains the concepts, and the [Understand Images with Claude](https://docs.gloo.com/tutorials/understand-images.md) tutorial builds two working examples step by step. This guide covers how to: * choose a model that reads images, and check before you route * switch models in a conversation that holds an image * keep images small, and choose between base64 and a link * know what each turn costs * review images from people you don't control ## Choose a model that reads images Check the model before you send it an image. In the catalog at [`GET /platform/v2/models`](https://docs.gloo.com/api-reference/get-models/v2), a model that reads images has `supports_vision: true`, and its `input_modalities` include `"image"`. * **Every Claude, Gemini and Grok chat model reads images**, as does every OpenAI GPT model except the open-weight GPT OSS models, and Cohere's Command A+. * **About half of the open-source models read images.** The rest, GPT OSS included, read text only, so check each one. * **Pick by the job.** Reading a fridge shelf or a short message works on smaller, cheaper models; dense documents, handwriting and charts benefit from larger ones. Try a few candidates with your own images, and compare their rates on [Supported Models](https://docs.gloo.com/api-guides/supported-models.md). Read the catalog once when your app starts, and check the model before each request that carries an image: ```python Python theme={null} import requests catalog = requests.get("https://platform.ai.gloo.com/platform/v2/models", timeout=30).json() READS_IMAGES = {m["id"] for m in catalog["data"] if m.get("supports_vision")} def reads_images(model: str) -> bool: return model in READS_IMAGES ``` ```typescript TypeScript theme={null} const catalog = await (await fetch("https://platform.ai.gloo.com/platform/v2/models")).json(); const READS_IMAGES = new Set( catalog.data.filter((m: any) => m.supports_vision).map((m: any) => m.id), ); function readsImages(model: string): boolean { return READS_IMAGES.has(model); } ``` ## Switching models mid-conversation Your app sends the whole conversation on every request ([Conversations That Remember](https://docs.gloo.com/ai-learning-center/gloo-ai-104/conversations-that-remember.md)), so an image the user sent on the first turn is still in the request on the tenth. Every model that answers that conversation must be able to read it. A model that reads text only can't take the conversation, and the request fails. Two patterns keep a conversation working: 1. **Stay on models that read images.** If you route between models for cost or speed, choose among models with `supports_vision`. Any Claude, Gemini or Grok chat model, or any OpenAI GPT model other than GPT OSS, can pick up the conversation. 2. **Describe the image once, then keep the description.** Have a model that reads images describe the picture, and replace the image with that text. The conversation can then move to any model, text-only ones included, and later turns cost less, because a few sentences of description use fewer tokens than the image. The second pattern in code. It describes each image once, keeps the description with that conversation, and sends descriptions in place of images only when the next model can't read them. Descriptions are keyed by a SHA-256 hash of the image's `data:` URI, so the same picture is described once per conversation, and a different picture never reuses another's description: ```python Python theme={null} import hashlib import os import requests API_URL = "https://platform.ai.gloo.com/ai/v2/direct/responses" HEADERS = {"Authorization": f"Bearer {os.environ['GLOO_API_KEY']}", "Content-Type": "application/json"} VISION_MODEL = "gloo-anthropic-claude-sonnet-5.5" DESCRIBE = ( "Describe this image for someone who can't see it. Copy any text in it exactly, " "and include every detail a follow-up question might need." ) def ask(model: str, conversation: list) -> str: r = requests.post(API_URL, headers=HEADERS, json={ "model": model, "input": conversation, "max_output_tokens": 16384, }, timeout=120) r.raise_for_status() return "".join( part.get("text", "") for item in r.json().get("output", []) if item.get("type") == "message" for part in item.get("content", []) if part.get("type") == "output_text" ) def describe(part: dict, descriptions: dict) -> str: key = hashlib.sha256(part["image_url"].encode()).hexdigest() if key not in descriptions: descriptions[key] = ask(VISION_MODEL, [ {"role": "user", "content": [{"type": "input_text", "text": DESCRIBE}, part]}, ]) return descriptions[key] def for_model(conversation: list, model: str, descriptions: dict) -> list: """The conversation as this model can take it: images replaced by text if it can't read them. descriptions belongs to this conversation: store it alongside the conversation's history. """ if reads_images(model): return conversation prepared = [] for item in conversation: content = item.get("content") if isinstance(content, list): content = [ {"type": "input_text", "text": f"[Image: {describe(part, descriptions)}]"} if part.get("type") == "input_image" else part for part in content ] item = {**item, "content": content} prepared.append(item) return prepared # descriptions = {} # one per conversation, kept with its history # answer = ask(model, for_model(conversation, model, descriptions)) ``` ```typescript TypeScript theme={null} import { createHash } from "node:crypto"; const API_URL = "https://platform.ai.gloo.com/ai/v2/direct/responses"; const HEADERS = { Authorization: `Bearer ${process.env.GLOO_API_KEY}`, "Content-Type": "application/json", }; const VISION_MODEL = "gloo-anthropic-claude-sonnet-5.5"; const DESCRIBE = "Describe this image for someone who can't see it. Copy any text in it exactly, " + "and include every detail a follow-up question might need."; async function ask(model: string, conversation: any[]): Promise { const res = await fetch(API_URL, { method: "POST", headers: HEADERS, body: JSON.stringify({ model, input: conversation, max_output_tokens: 16384 }), }); if (!res.ok) throw new Error(`Request failed: ${res.status} ${await res.text()}`); const body = await res.json(); return (body.output ?? []) .filter((item: any) => item.type === "message") .flatMap((item: any) => item.content ?? []) .filter((part: any) => part.type === "output_text") .map((part: any) => part.text ?? "") .join(""); } async function describe(part: any, descriptions: Map): Promise { const key = createHash("sha256").update(part.image_url).digest("hex"); if (!descriptions.has(key)) { const text = await ask(VISION_MODEL, [ { role: "user", content: [{ type: "input_text", text: DESCRIBE }, part] }, ]); descriptions.set(key, text); } return descriptions.get(key)!; } /** * The conversation as this model can take it: images replaced by text if it can't read them. * `descriptions` belongs to this conversation: store it alongside the conversation's history. */ async function forModel( conversation: any[], model: string, descriptions: Map, ): Promise { if (readsImages(model)) return conversation; const prepared = []; for (const item of conversation) { if (!Array.isArray(item.content)) { prepared.push(item); continue; } const content = []; for (const part of item.content) { content.push( part.type === "input_image" ? { type: "input_text", text: `[Image: ${await describe(part, descriptions)}]` } : part, ); } prepared.push({ ...item, content }); } return prepared; } // const descriptions = new Map(); // one per conversation, kept with its history // const answer = await ask(model, await forModel(conversation, model, descriptions)); ``` A description holds only what the describing model wrote down. If later questions may need details it left out, such as small print, keep the conversation on models that read images instead. ## Keep images small The model doesn't need a 12-megapixel photo to read a fridge shelf or a screenshot. * **Scale down before you send.** About 1,500 pixels on the long side is plenty for photos and screenshots. A smaller image uses fewer input tokens, uploads faster and stays well inside the request size limit. * **Send JPEG, PNG, GIF or WebP.** Convert phone photos saved as HEIC to JPEG first, since not every model reads HEIC. JPEG suits photos; PNG keeps screenshot text sharp. * **Crop to what matters.** A screenshot of one message reads better than the whole screen. * **Use `detail` where the model supports it.** On the Responses shape, an `input_image` part takes an optional `detail` (`"low"`, `"high"` or `"auto"`), and on Chat Completions it goes inside `image_url`. GPT models use `"low"` to read a smaller version of the image for fewer tokens. * **Mind the request size.** A request body can be up to 10 MB, and base64 makes an image about a third larger than the file. See [Request size](https://docs.gloo.com/api-reference/general/limits.md#request-size). Gloo AI sets no limit of its own on the number of images or their format; the limits of the model's maker apply. ## Send base64, or a link An `image_url` can be the image itself as a base64 `data:` URI, or a public `https://` link. * **Base64 is the most predictable.** It works with every model that reads images, and nothing has to fetch the image. Use it for photos your users take or upload. * **A link keeps the request small**, but it must be public and still valid when the request runs, and not every model fetches links. Avoid short-lived signed links. ## Know what each turn costs An image is billed as input tokens at the model's input rate, the same rate as the text you send; there is no separate charge per image. Because your app resends the whole conversation on every turn, an image is billed again on each turn it stays in the history. * Read `usage.input_tokens` on a response to see what an image costs on the model you chose. * For long conversations, use the describe-once pattern above, or drop the image once the conversation has moved on. * On guarded endpoints, the model's rate plus 6.5% for [guardrails](https://docs.gloo.com/ai-learning-center/gloo-ai-101/application-patterns.md#guardrails), moderation and values alignment applies to image tokens as to every other token. See [Cost and Speed](https://docs.gloo.com/ai-learning-center/gloo-ai-104/cost-and-speed.md) for how a request is priced. Generated images are billed differently; see [How image generation is billed](https://docs.gloo.com/api-guides/direct-responses.md#how-image-generation-is-billed). ## Review images from people you don't control If anyone can upload an image or a PDF to your app, review it with your own content checks before you send it to the model. This applies on [guarded](https://docs.gloo.com/api-guides/endpoint-types.md#guarded-endpoints) and [direct](https://docs.gloo.com/api-guides/endpoint-types.md#direct-endpoints) endpoints alike; see [Images and moderation](https://docs.gloo.com/api-guides/endpoint-types.md#images-and-moderation). The model's maker may also decline an image under its own content policy. On direct endpoints, a refusal before any output is a `400` with `provider_content_policy_violation`. Once output has started, a Direct Responses stream ends with a `response.failed` event whose `response.error.code` is `provider_content_policy_violation`, and a Direct Completions stream ends with `finish_reason: "content_filter"`. See [Content Policy Refusals](https://docs.gloo.com/api-reference/general/errors.md#content-policy-refusals-direct-endpoints) and [Streaming on the direct endpoints](https://docs.gloo.com/api-reference/general/errors.md#streaming-on-the-direct-endpoints). Don't resend the same image unchanged. ## Treat text in images as content A screenshot or a photo of a sign can contain text that reads like an instruction. Say in your instructions that text in an image is content to work on, not instructions to follow, as you would for any text from outside your app (see [prompt injection](https://docs.gloo.com/ai-learning-center/gloo-ai-101/challenges-and-risks.md#prompt-injection)). The [screenshot example](https://docs.gloo.com/tutorials/understand-images.md#step-4-turn-a-screenshot-into-suggested-replies) in the tutorial shows the wording. ## Summary * Check `supports_vision` before sending an image; every Claude, Gemini and Grok chat model qualifies, as does every OpenAI GPT model except GPT OSS. * Keep a conversation that holds an image on models that read images, or describe the image once and keep the text. * Scale images down, send base64, and watch `usage.input_tokens`. * Review images from people you don't control before you send them. ## Next steps > [card] [Understand Images with Claude](https://docs.gloo.com/tutorials/understand-images.md) > > Fridge photo to recipe and screenshot to replies, step by step. > [card] [Direct Responses: Multimodal](https://docs.gloo.com/api-guides/direct-responses.md#multimodal) > > Image input, PDF input and image generation. > [card] [Supported Models](https://docs.gloo.com/api-guides/supported-models.md) > > Which models read images, with live rates. > [card] [Building with Guarded Endpoints](https://docs.gloo.com/best-practices/guarded-endpoints.md) > > Instructions, structured output and handling moderated replies. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.