Cut Cost and Latency with Prompt Caching

Overview

Most production prompts repeat themselves. A support assistant sends the same instructions and the same handbook on every request, and only the user’s question changes at the end. Prompt caching lets the model provider reuse the tokens it has already processed for that repeated opening text, the prompt prefix. Cached tokens are billed at a reduced rate and the response starts sooner. In this tutorial you’ll take a prompt that never hits the cache, restructure it so it does, turn on caching for Claude and OpenAI models, and prove it works by reading the cache fields in usage.

What You’ll Build

You’ll build:

A cache-friendly prompt layout with a stable prefix and a changing suffix
A usage reader that pulls cache reads and writes out of any Gloo AI response shape
Explicit caching for Claude, with the X-Cache-TTL header on Completions and cache_control on the Anthropic Messages endpoint
Cache-aware routing for OpenAI models, with prompt_cache_key
A before/after comparison that reports cache hits, latency and input-token savings for your own prompt

Prerequisites

You need:

A Gloo AI Studio account
Your API key from the API Keys page, exported as GLOO_API_KEY
Authentication setup: complete the Authentication Tutorial first
Python 3.10+ with requests, or Node.js 18+ (the TypeScript examples use the built-in fetch)

You also need a long, stable piece of text to cache: a style guide, a policy handbook, product documentation, or a long system prompt. Save it as handbook.md next to your script. Caching only starts above a minimum prompt length (see Step 1), so aim for a few thousand words.

How Caching Works Through Gloo AI

Each provider caches in its own way. Through Gloo AI you either do nothing, or set one header or field:

Provider	Type	What you do
Anthropic (Claude)	Explicit	Send the `X-Cache-TTL` header (`5m` or `1h`). On Anthropic Messages, use `cache_control`.
OpenAI	Implicit, plus an optional key	Nothing for prompts of 1,024+ tokens. Add `prompt_cache_key` to improve hit rates.
DeepSeek, Gemini, Qwen, MiniMax and others	Implicit	Nothing. Repeated prefixes are cached by the provider.

Whatever the provider, a cache hit needs the same thing: the beginning of the prompt must be identical, token for token, to a recent request. Most of this tutorial is about getting that right. The Supported Models catalog shows which models cache and how. For every provider’s rates and rules, see the Prompt Caching guide.

Step 1: Put the Stable Content First

A cache matches from the first token forward and stops at the first difference. Anything that changes per request, placed early, breaks the match for everything after it.

A prompt that never caches

{
  "messages": [
    {
      "role": "system",
      "content": "Today is 2026-10-29 14:02:11. You are helping Maria at Grace Church.\n\nYou are a support assistant for our ministry software...\n\n<the 6,000-token handbook>"
    },
    { "role": "user", "content": "How do I export the giving report?" }
  ]
}

The timestamp and the user’s name sit at the very start, so no two requests share a prefix. Every request pays full price for the handbook.

The same prompt, cache-friendly

{
  "messages": [
    {
      "role": "system",
      "content": "You are a support assistant for our ministry software...\n\n<the 6,000-token handbook>"
    },
    {
      "role": "user",
      "content": "Context: today is 2026-10-29. The user is Maria at Grace Church.\n\nQuestion: How do I export the giving report?"
    }
  ]
}

Now the system message is byte-for-byte identical across users and requests, and everything that varies comes after it.

Rules for a stable prefix

Order content from most stable to least stable: instructions, then reference documents, then conversation history, then the new question.
Move per-request values to the end: timestamps, user names, request IDs, retrieved snippets that differ per question.
Keep the prefix deterministic: build it from the same template every time. Don’t shuffle examples, re-serialize JSON with a different key order, or reorder tool definitions between requests.

Make the prefix long enough to cache. Providers only cache above a minimum length:

Provider	Minimum prompt length
OpenAI	1,024 tokens
Gemini 2.5 Flash, 2.5 Flash Lite	2,048 tokens
Gemini 3 Flash, 3.1 Flash Lite, 3.5 Flash	4,096 tokens
Claude	Varies by model (for example 4,096 tokens for Claude Haiku 4.5); see Anthropic’s prompt caching docs

Version the prefix. When you change the instructions or the handbook, treat it as a new prefix. The first request after a change writes a new cache entry.

Put per-request values in the user message, not the system message, so the cached prefix stays identical on every call and the layout works on every provider.

This prompt builder, reused in every step, loads the handbook once so the system message is identical on every call:

# File: prompt.py
from pathlib import Path

INSTRUCTIONS = (
    "You are a support assistant for our ministry software. "
    "Answer only from the handbook below. If the handbook does not cover "
    "the question, say so and suggest contacting support."
)
HANDBOOK = Path("handbook.md").read_text(encoding="utf-8")

# Built once at startup: the stable prefix.
SYSTEM_PROMPT = f"{INSTRUCTIONS}\n\n<handbook>\n{HANDBOOK}\n</handbook>"


def build_messages(question: str, user_context: str = "") -> list[dict]:
    """Stable system message first; everything that varies goes last."""
    suffix = f"Context: {user_context}\n\nQuestion: {question}" if user_context else question
    return [
        {"role": "system", "content": SYSTEM_PROMPT},
        {"role": "user", "content": suffix},
    ]

// File: prompt.ts
import { readFileSync } from "node:fs";

const INSTRUCTIONS =
  "You are a support assistant for our ministry software. " +
  "Answer only from the handbook below. If the handbook does not cover " +
  "the question, say so and suggest contacting support.";
const HANDBOOK = readFileSync("handbook.md", "utf-8");

// Built once at startup: the stable prefix.
export const SYSTEM_PROMPT = `${INSTRUCTIONS}\n\n<handbook>\n${HANDBOOK}\n</handbook>`;

export function buildMessages(question: string, userContext = "") {
  // Stable system message first; everything that varies goes last.
  const suffix = userContext ? `Context: ${userContext}\n\nQuestion: ${question}` : question;
  return [
    { role: "system", content: SYSTEM_PROMPT },
    { role: "user", content: suffix },
  ];
}

Step 2: Read the Cache Fields in `usage`

Every response reports its token counts in usage. Cache activity appears there too, but the field names depend on the endpoint’s response shape:

Response shape	Endpoints	Prompt tokens	Read from cache	Written to cache
Chat Completions	Direct Completions, Completions V2	`prompt_tokens`	`prompt_tokens_details.cached_tokens`	`prompt_tokens_details.cache_write_tokens`
Responses	Direct Responses, Guarded Responses	`input_tokens`	`input_tokens_details.cached_tokens`	`input_tokens_details.cache_write_tokens`
Anthropic Messages	Anthropic Messages	`input_tokens`	`cache_read_input_tokens`	`cache_creation_input_tokens`

For example, a Direct Completions response reports:

"usage": {
  "prompt_tokens": 45,
  "completion_tokens": 42,
  "total_tokens": 87,
  "prompt_tokens_details": { "cached_tokens": 0, "cache_write_tokens": 0 },
  "cache_creation_input_tokens": 0,
  "cache_read_input_tokens": 0
}

The detail objects are present when the provider reports them, so read them defensively. This helper pulls the cache reads and writes out of all three shapes:

# File: cache_stats.py
def cache_stats(usage: dict | None) -> dict:
    """Normalize cache counts from Chat Completions, Responses or Anthropic usage."""
    usage = usage or {}
    details = usage.get("prompt_tokens_details") or usage.get("input_tokens_details") or {}
    return {
        "cache_read": details.get("cached_tokens") or usage.get("cache_read_input_tokens") or 0,
        "cache_write": details.get("cache_write_tokens") or usage.get("cache_creation_input_tokens") or 0,
    }

// File: cacheStats.ts
export interface CacheStats {
  cacheRead: number;
  cacheWrite: number;
}

export function cacheStats(usage: Record<string, any> | null | undefined): CacheStats {
  // Normalize cache counts from Chat Completions, Responses or Anthropic usage.
  const u = usage ?? {};
  const details = u.prompt_tokens_details ?? u.input_tokens_details ?? {};
  return {
    cacheRead: details.cached_tokens || u.cache_read_input_tokens || 0,
    cacheWrite: details.cache_write_tokens || u.cache_creation_input_tokens || 0,
  };
}

What to expect:

First request with a new prefix: cache_read is 0. On Claude, cache_write is above 0: the prefix was just stored.
The same prefix again, soon after: cache_read is above 0. That’s a cache hit.
OpenAI, Gemini and other implicit providers: watch cache_read. Many of these models charge nothing to write the cache, but some do, including the Gemini models and some newer OpenAI models; the models endpoint shows which.

Step 3: Turn On Caching for Claude with `X-Cache-TTL`

Claude models cache only when you ask. On the guarded Completions V2 endpoint, add the X-Cache-TTL header to opt in; you don’t change your message content.

`X-Cache-TTL`	Cache lifetime	Cache write billed at	Cache read billed at
`5m`	5 minutes	125% of the input rate	10% of the input rate on most models
`1h`	1 hour	200% of the input rate	10% of the input rate on most models

Some newer Claude models read from the cache for less: 5% of the input rate on Claude Opus 5.5 and Sonnet 5.5, and 2.5% on Claude Fable 5.1. GET /platform/v2/models lists the exact cache_read and cache_write rates for each model. Send the same question twice and compare usage:

# File: claude_cache.py
import os
import time

import requests

from cache_stats import cache_stats
from prompt import build_messages

API_URL = "https://platform.ai.gloo.com/ai/v2/guarded/chat/completions"
HEADERS = {
    "Authorization": f"Bearer {os.environ['GLOO_API_KEY']}",
    "Content-Type": "application/json",
    "X-Cache-TTL": "5m",
}


def ask(question: str) -> dict:
    payload = {
        "model": "gloo-anthropic-claude-opus-5.5",
        "messages": build_messages(question),
        "stream": False,
    }
    started = time.perf_counter()
    response = requests.post(API_URL, headers=HEADERS, json=payload, timeout=300)
    response.raise_for_status()
    elapsed = time.perf_counter() - started
    stats = cache_stats(response.json().get("usage"))
    print(f"{elapsed:5.2f}s  {stats}")
    return stats


ask("How do I export the giving report?")    # writes the cache
ask("How do I add a new volunteer role?")    # reads the cache

// File: claudeCache.ts
import { cacheStats } from "./cacheStats";
import { buildMessages } from "./prompt";

const API_URL = "https://platform.ai.gloo.com/ai/v2/guarded/chat/completions";

async function ask(question: string) {
  const started = performance.now();
  const response = await fetch(API_URL, {
    method: "POST",
    headers: {
      Authorization: `Bearer ${process.env.GLOO_API_KEY}`,
      "Content-Type": "application/json",
      "X-Cache-TTL": "5m",
    },
    body: JSON.stringify({
      model: "gloo-anthropic-claude-opus-5.5",
      messages: buildMessages(question),
      stream: false,
    }),
  });
  if (!response.ok) throw new Error(`HTTP ${response.status}: ${await response.text()}`);
  const data = await response.json();
  const elapsed = (performance.now() - started) / 1000;
  const stats = cacheStats(data.usage);
  console.log(`${elapsed.toFixed(2)}s`, stats);
  return stats;
}

async function main() {
  await ask("How do I export the giving report?"); // writes the cache
  await ask("How do I add a new volunteer role?"); // reads the cache
}

main().catch((err) => {
  console.error(err);
  process.exit(1);
});

curl -X POST 'https://platform.ai.gloo.com/ai/v2/guarded/chat/completions' \
  -H "Authorization: Bearer $GLOO_API_KEY" \
  -H 'Content-Type: application/json' \
  -H 'X-Cache-TTL: 5m' \
  -d '{
    "model": "gloo-anthropic-claude-opus-5.5",
    "messages": [
      {"role": "system", "content": "You are a support assistant... <the full handbook>"},
      {"role": "user", "content": "How do I export the giving report?"}
    ],
    "stream": false
  }'

What You’ll See

The two questions differ, but the system message is the same, so the second request reads the prefix from cache. Your counts depend on your handbook and the model; the pattern looks like this:

 9.84s  {'cache_read': 0, 'cache_write': 6150}
 6.10s  {'cache_read': 6150, 'cache_write': 0}

If cache_read stays at 0, jump to Troubleshooting.

Choosing `5m` or `1h`

The write premium is paid once per cache entry; the read discount is earned on every hit. Measured in input-token equivalents for each cached prefix token:

5m: a write costs an extra 0.25, each hit saves 0.9 at a 10% read rate (0.95 on Claude Opus 5.5, the model in this tutorial). One hit within five minutes already pays for the write.
1h: a write costs an extra 1.0, each hit saves 0.9 or more. It pays for itself from the second hit, and keeps the cache warm through quieter periods.

Use 5m for steady traffic, where the same prefix is requested every few minutes. Consider 1h when requests for a prefix are spread out, for example a document a team asks about a few times an hour.

Your own cache breakpoints

With X-Cache-TTL on Completions V2, add at most three cache_control markers of your own on your message content. If your request already has four of its own, the request still succeeds, but your system messages aren’t cached on that call.

Step 4: Use `cache_control` on Anthropic Messages

If your code uses the Anthropic SDK, point it at the Anthropic Messages endpoint. There, cache_control works as it does on Anthropic’s own API: set it at the top level to cache automatically, or on individual content blocks for fine-grained control. Marking the system block caches everything up to and including it:

# File: anthropic_cache.py
import os

from anthropic import Anthropic

from cache_stats import cache_stats
from prompt import SYSTEM_PROMPT

client = Anthropic(
    base_url="https://platform.ai.gloo.com/ai/v2/direct/anthropic",
    auth_token=os.environ["GLOO_API_KEY"],
)


def ask(question: str) -> None:
    message = client.messages.create(
        model="gloo-anthropic-claude-opus-5.5",
        max_tokens=8192,
        system=[
            {
                "type": "text",
                "text": SYSTEM_PROMPT,
                "cache_control": {"type": "ephemeral"},
            }
        ],
        messages=[{"role": "user", "content": question}],
    )
    print(cache_stats(message.usage.model_dump()))


ask("How do I export the giving report?")
ask("How do I add a new volunteer role?")

// File: anthropicCache.ts
import Anthropic from "@anthropic-ai/sdk";
import { cacheStats } from "./cacheStats";
import { SYSTEM_PROMPT } from "./prompt";

const client = new Anthropic({
  baseURL: "https://platform.ai.gloo.com/ai/v2/direct/anthropic",
  authToken: process.env.GLOO_API_KEY,
});

async function ask(question: string) {
  const message = await client.messages.create({
    model: "gloo-anthropic-claude-opus-5.5",
    max_tokens: 8192,
    system: [
      { type: "text", text: SYSTEM_PROMPT, cache_control: { type: "ephemeral" } },
    ],
    messages: [{ role: "user", content: question }],
  });
  console.log(cacheStats(message.usage as any));
}

async function main() {
  await ask("How do I export the giving report?");
  await ask("How do I add a new volunteer role?");
}

main().catch((err) => {
  console.error(err);
  process.exit(1);
});

# Top-level cache_control: automatic caching for the request
curl https://platform.ai.gloo.com/ai/v2/direct/anthropic/v1/messages \
  -H "Authorization: Bearer $GLOO_API_KEY" \
  -H "anthropic-version: 2023-06-01" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gloo-anthropic-claude-opus-5.5",
    "max_tokens": 8192,
    "cache_control": {"type": "ephemeral"},
    "system": "You are a support assistant... <the full handbook>",
    "messages": [
      {"role": "user", "content": "How do I export the giving report?"}
    ]
  }'

Confirm the hit in usage.cache_creation_input_tokens (first call) and usage.cache_read_input_tokens (second call). Pass your Gloo API key as the SDK’s auth token (auth_token in Python, authToken in TypeScript); see Set up the SDK.

Anthropic Messages is a direct endpoint: it bills the model’s published rate, and no guardrails or values alignment run. Pick the endpoint for what your application needs; caching works on both.

Step 5: Improve OpenAI Hit Rates with `prompt_cache_key`

OpenAI caches automatically once a prompt reaches 1,024 tokens, with no write fee on most models. Requests are routed to servers that recently processed the same prefix. Add prompt_cache_key to the request body to make requests that share a prefix more likely to land on the same server:

# File: openai_cache.py
import os

import requests

from cache_stats import cache_stats
from prompt import build_messages

API_URL = "https://platform.ai.gloo.com/ai/v2/guarded/chat/completions"
HEADERS = {
    "Authorization": f"Bearer {os.environ['GLOO_API_KEY']}",
    "Content-Type": "application/json",
}
CACHE_KEY = "support-handbook-v1"  # bump to -v2 when the handbook changes


def ask(question: str) -> None:
    payload = {
        "model": "gloo-openai-gpt-5.2",
        "messages": build_messages(question),
        "prompt_cache_key": CACHE_KEY,
        "stream": False,
    }
    response = requests.post(API_URL, headers=HEADERS, json=payload, timeout=300)
    response.raise_for_status()
    print(cache_stats(response.json().get("usage")))


ask("How do I export the giving report?")
ask("How do I add a new volunteer role?")

// File: openaiCache.ts
import { cacheStats } from "./cacheStats";
import { buildMessages } from "./prompt";

const API_URL = "https://platform.ai.gloo.com/ai/v2/guarded/chat/completions";
const CACHE_KEY = "support-handbook-v1"; // bump to -v2 when the handbook changes

async function ask(question: string) {
  const response = await fetch(API_URL, {
    method: "POST",
    headers: {
      Authorization: `Bearer ${process.env.GLOO_API_KEY}`,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({
      model: "gloo-openai-gpt-5.2",
      messages: buildMessages(question),
      prompt_cache_key: CACHE_KEY,
      stream: false,
    }),
  });
  if (!response.ok) throw new Error(`HTTP ${response.status}: ${await response.text()}`);
  console.log(cacheStats((await response.json()).usage));
}

async function main() {
  await ask("How do I export the giving report?");
  await ask("How do I add a new volunteer role?");
}

main().catch((err) => {
  console.error(err);
  process.exit(1);
});

Good cache keys are stable and describe the prefix, not the request:

Good:  "support-handbook-v1", "sermon-generator", "theology-assistant-v1"
Avoid: "request-12345", "cache-abc", a user ID, a timestamp

When the instructions or the handbook change, change the key too. prompt_cache_key is also accepted on Responses and Direct Responses, and is silently ignored for models from other providers, so it’s safe to send on every request. The cache read rate depends on the model: usually 10% of the input rate on GPT-5 and later models, and 25% on GPT-4.1. Some newer models, such as GPT-5.6 and GPT-6, also charge a cache write. Check each model’s exact rates with GET /platform/v2/models.

Step 6: Gemini, DeepSeek and Other Implicit Providers

Nothing to switch on: no X-Cache-TTL, cache_control or prompt_cache_key. Repeated prefixes are cached by the provider, and hits are billed as a separate cache_read segment at the model’s reduced rate. The work is all in Step 1: a stable prefix above the model’s minimum length. Run the same two-question test from Step 3 without the header, and check cache_read.

Cache read rates vary by model. Gemini cache reads are usually 10% of the input rate, and Gemini models also charge a cache write. Open-weight models such as DeepSeek, Qwen and MiniMax range from under 1% to more than half of the input rate, and some Qwen models charge a cache write too. Not every model in this group caches. The models endpoint is the source of truth for each model’s cache pricing.

Step 7: Compare Before and After

Measure the effect on your own traffic. This script sends the same set of questions twice: once with a prompt that changes at the start (the “before” layout from Step 1), and once with the stable prefix. It totals cache reads, cache writes and latency, then converts the cache counts into input-token equivalents saved, using the rates from the Prompt Caching guide:

saved = cache_read × (1 − read_ratio) − cache_write × (write_ratio − 1)

For Claude with X-Cache-TTL: 5m, write_ratio is 1.25 and read_ratio is 0.10 on most models (0.05 on Claude Opus 5.5, the model used here). For a GPT-5 model with no write fee, such as GPT-5.5, read_ratio is 0.10 and write_ratio is 1.0. For any other model, take both ratios from the models endpoint: divide cache_read and cache_write by input.

# File: compare.py
import os
import time

import requests

from cache_stats import cache_stats
from prompt import SYSTEM_PROMPT, build_messages

API_URL = "https://platform.ai.gloo.com/ai/v2/guarded/chat/completions"
MODEL = "gloo-anthropic-claude-opus-5.5"
READ_RATIO, WRITE_RATIO = 0.05, 1.25  # Claude Opus 5.5, X-Cache-TTL: 5m
HEADERS = {
    "Authorization": f"Bearer {os.environ['GLOO_API_KEY']}",
    "Content-Type": "application/json",
    "X-Cache-TTL": "5m",
}
QUESTIONS = [
    "How do I export the giving report?",
    "How do I add a new volunteer role?",
    "Can two admins edit the same event?",
    "Where do I change the church logo?",
]


def unstable_messages(question: str) -> list[dict]:
    """The 'before' layout: a per-request value at the very start."""
    stamp = time.strftime("%Y-%m-%d %H:%M:%S")
    return [
        {"role": "system", "content": f"Request time: {stamp}.\n\n{SYSTEM_PROMPT}"},
        {"role": "user", "content": question},
    ]


def run(label: str, make_messages) -> None:
    totals = {"cache_read": 0, "cache_write": 0}
    seconds = 0.0
    for question in QUESTIONS:
        started = time.perf_counter()
        response = requests.post(
            API_URL,
            headers=HEADERS,
            json={"model": MODEL, "messages": make_messages(question), "stream": False},
            timeout=300,
        )
        response.raise_for_status()
        seconds += time.perf_counter() - started
        for key, value in cache_stats(response.json().get("usage")).items():
            totals[key] += value
    saved = totals["cache_read"] * (1 - READ_RATIO) - totals["cache_write"] * (WRITE_RATIO - 1)
    print(f"{label:<8} read={totals['cache_read']:>6}  "
          f"write={totals['cache_write']:>6}  saved≈{saved:>8.0f} input-token eq.  "
          f"avg {seconds / len(QUESTIONS):.2f}s")


run("before", unstable_messages)
run("after", build_messages)

// File: compare.ts
import { cacheStats } from "./cacheStats";
import { SYSTEM_PROMPT, buildMessages } from "./prompt";

const API_URL = "https://platform.ai.gloo.com/ai/v2/guarded/chat/completions";
const MODEL = "gloo-anthropic-claude-opus-5.5";
const READ_RATIO = 0.05;
const WRITE_RATIO = 1.25; // Claude Opus 5.5, X-Cache-TTL: 5m
const QUESTIONS = [
  "How do I export the giving report?",
  "How do I add a new volunteer role?",
  "Can two admins edit the same event?",
  "Where do I change the church logo?",
];

// The 'before' layout: a per-request value at the very start.
function unstableMessages(question: string) {
  return [
    { role: "system", content: `Request time: ${new Date().toISOString()}.\n\n${SYSTEM_PROMPT}` },
    { role: "user", content: question },
  ];
}

async function run(label: string, makeMessages: (q: string) => object[]) {
  const totals = { cacheRead: 0, cacheWrite: 0 };
  let seconds = 0;
  for (const question of QUESTIONS) {
    const started = performance.now();
    const response = await fetch(API_URL, {
      method: "POST",
      headers: {
        Authorization: `Bearer ${process.env.GLOO_API_KEY}`,
        "Content-Type": "application/json",
        "X-Cache-TTL": "5m",
      },
      body: JSON.stringify({ model: MODEL, messages: makeMessages(question), stream: false }),
    });
    if (!response.ok) throw new Error(`HTTP ${response.status}: ${await response.text()}`);
    const stats = cacheStats((await response.json()).usage);
    seconds += (performance.now() - started) / 1000;
    totals.cacheRead += stats.cacheRead;
    totals.cacheWrite += stats.cacheWrite;
  }
  const saved = totals.cacheRead * (1 - READ_RATIO) - totals.cacheWrite * (WRITE_RATIO - 1);
  console.log(
    `${label.padEnd(8)} read=${totals.cacheRead} ` +
      `write=${totals.cacheWrite} saved≈${saved.toFixed(0)} input-token eq. ` +
      `avg ${(seconds / QUESTIONS.length).toFixed(2)}s`,
  );
}

async function main() {
  await run("before", unstableMessages);
  await run("after", buildMessages);
}

main().catch((err) => {
  console.error(err);
  process.exit(1);
});

What You’ll See

With the “before” layout, every request starts with a different timestamp, so every request writes a new cache entry and none reads one. With the stable prefix, the first request writes the cache and the next three read it. Illustrative output for a 6,000-token handbook:

before   read=     0  write= 24620  saved≈   -6155 input-token eq.  avg 9.71s
after    read= 18450  write=  6150  saved≈   15990 input-token eq.  avg 6.88s

Note the negative number in the first row: caching switched on with an unstable prefix pays the write premium on every request and never earns a read, so it costs more than no caching at all. Read the result as:

read is the number of prompt tokens billed at the cache read rate instead of the full input rate.
saved is how many full-price input tokens those reads were worth, after paying for the write. Multiply by the model’s input rate from GET /platform/v2/models to turn it into money for your own volume.
Latency typically drops on cache hits; how much depends on the model and prompt length, so measure it for your workload.

The models endpoint returns each segment’s published rate; guarded endpoints are priced at the model’s published rate plus 6.5% (see Pricing). For what you actually paid, check the Billing dashboard and API usage in Gloo AI Studio over a few days of real traffic.

Step 8: Keep the Cache Warm in Production

These habits keep the hit rate high in production, expanding on the caching best practices in the guide:

Pre-warm on startup. Send one request with your standard system prompt when your service starts, so the first real user reads from cache rather than paying for the write.
Expect a dip after failover. Caches are held by each provider. When the provider serving your model fails and Gloo AI fails over to another provider of the same model, that request doesn’t read the cache built on the first provider. Cached tokens recover as the cache warms on the new provider. Don’t alert on a single cold request; watch the hit rate over time.
Log cache stats. Record cache_read and cache_write with each request, next to the model and the prefix version (or prompt_cache_key). A sudden drop to zero usually means something started changing the prefix.
Keep long conversations cache-friendly. In multi-turn chat, append new turns to the end and never rewrite earlier ones, so each request extends the previous prefix.

For handling the errors and retries around these requests, see Build a Resilient Client.

Troubleshooting

cache_read is always 0 on Claude : Check that the X-Cache-TTL header is sent with 5m or 1h on every request (on Anthropic Messages, check cache_control instead). Check the prefix is above the model’s minimum cacheable length. If your request already carries four cache_control markers of its own, your system messages aren’t cached; remove one. cache_read is 0 on the second request, but cache_write is above 0 both times : Something in the prefix changes between requests. Diff the two system messages byte for byte: look for timestamps, IDs, user names, randomized examples, or JSON serialized with a different key order. Hits stop after a few minutes : X-Cache-TTL: 5m sets a five-minute cache lifetime. If your traffic is sparse, try 1h and compare the totals with Step 7. OpenAI requests rarely hit : Make sure the prompt is at least 1,024 tokens, and add a stable prompt_cache_key. A key that changes per request or per user spreads requests across servers and lowers the hit rate. prompt_cache_key has no effect : It only applies to OpenAI models and is ignored for other providers. For Claude, use X-Cache-TTL or cache_control. Cached tokens drop for a while, then recover : The request was most likely served by a different provider of the same model after a failover, which starts with a cold cache. See Provider Failover. The detail objects are missing from usage : They’re present when the provider reports them. The cache_stats helper treats a missing field as 0; if a model never reports cache counts, check its caching support in Supported Models.

Next Steps

Prompt Caching Guide

Every provider’s caching type, billing rates and minimum lengths.

Build a Resilient Client

Retries, backoff and stream recovery around your cached requests.

Provider Failover

What happens to your request, and your cache, when a provider fails.

Get Models API

Current per-model rates and capabilities.

Cost and Speed

How caching fits with model choice and reasoning effort to lower cost.

Context Is King

How to structure the context you put in a prompt.

Best Practices

Production patterns for reliable, trustworthy applications.

Get Started

Responses & Completions

Production

Content

Search & Recommendations

Embeddings

Legacy

Cut Cost and Latency with Prompt Caching

Overview

What You’ll Build

Prerequisites

How Caching Works Through Gloo AI

Step 1: Put the Stable Content First

A prompt that never caches

The same prompt, cache-friendly

Rules for a stable prefix

Step 2: Read the Cache Fields in `usage`

Step 3: Turn On Caching for Claude with `X-Cache-TTL`

What You’ll See

Choosing `5m` or `1h`

Your own cache breakpoints

Step 4: Use `cache_control` on Anthropic Messages

Step 5: Improve OpenAI Hit Rates with `prompt_cache_key`

Step 6: Gemini, DeepSeek and Other Implicit Providers

Step 7: Compare Before and After

What You’ll See

Step 8: Keep the Cache Warm in Production

Troubleshooting

Next Steps

Prompt Caching Guide

Build a Resilient Client

Provider Failover

Get Models API

Cost and Speed

Context Is King

Best Practices

​Overview

​What You’ll Build

​Prerequisites

​How Caching Works Through Gloo AI

​Step 1: Put the Stable Content First

​A prompt that never caches

​The same prompt, cache-friendly

​Rules for a stable prefix

​Step 2: Read the Cache Fields in usage

​Step 3: Turn On Caching for Claude with X-Cache-TTL

​What You’ll See

​Choosing 5m or 1h

​Your own cache breakpoints

​Step 4: Use cache_control on Anthropic Messages

​Step 5: Improve OpenAI Hit Rates with prompt_cache_key

​Step 6: Gemini, DeepSeek and Other Implicit Providers

​Step 7: Compare Before and After

​What You’ll See

​Step 8: Keep the Cache Warm in Production

​Troubleshooting

​Next Steps

Prompt Caching Guide

Build a Resilient Client

Provider Failover

Get Models API

Cost and Speed

Context Is King

Best Practices

Overview

What You’ll Build

Prerequisites

How Caching Works Through Gloo AI

Step 1: Put the Stable Content First

A prompt that never caches

The same prompt, cache-friendly

Rules for a stable prefix

Step 2: Read the Cache Fields in `usage`

Step 3: Turn On Caching for Claude with `X-Cache-TTL`

What You’ll See

Choosing `5m` or `1h`

Your own cache breakpoints

Step 4: Use `cache_control` on Anthropic Messages

Step 5: Improve OpenAI Hit Rates with `prompt_cache_key`

Step 6: Gemini, DeepSeek and Other Implicit Providers

Step 7: Compare Before and After

What You’ll See

Step 8: Keep the Cache Warm in Production

Troubleshooting

Next Steps