> ## Documentation Index
> Fetch the complete documentation index at: https://docs.gloo.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Cut Cost and Latency with Prompt Caching

> Structure prompts around a stable prefix, turn on caching for Claude and OpenAI models, confirm cache hits in the usage fields, and measure the token savings.

## Overview

Most production prompts repeat themselves. A support assistant sends the same instructions and the same handbook on every request, and only the user's question changes at the end. [Prompt caching](https://docs.gloo.com/api-guides/prompt-caching.md) lets the model provider reuse the [tokens](https://docs.gloo.com/ai-learning-center/gloo-ai-101/prompting-and-generation.md#tokens) it has already processed for that repeated opening text, the *prompt prefix*. Cached tokens are billed at a reduced rate and the response starts sooner.

In this tutorial you'll take a prompt that never hits the cache, restructure it so it does, turn on caching for Claude and OpenAI models, and prove it works by reading the cache fields in `usage`.

## What You'll Build

You'll build:

* **A cache-friendly prompt layout** with a stable prefix and a changing suffix
* **A usage reader** that pulls cache reads and writes out of any Gloo AI response shape
* **Explicit caching for Claude**, with the `X-Cache-TTL` header on Completions and `cache_control` on the Anthropic Messages endpoint
* **Cache-aware routing for OpenAI models**, with `prompt_cache_key`
* **A before/after comparison** that reports cache hits, latency and input-token savings for your own prompt

## Prerequisites

You need:

* A Gloo AI Studio account
* Your API key from the [API Keys page](https://docs.gloo.com/studio/manage-api-credentials.md), exported as `GLOO_API_KEY`
* **Authentication setup**: complete the [Authentication Tutorial](https://docs.gloo.com/tutorials/authentication.md) first
* Python 3.10+ with `requests`, or Node.js 18+ (the TypeScript examples use the built-in `fetch`)

You also need a long, stable piece of text to cache: a style guide, a policy handbook, product documentation, or a long [system prompt](https://docs.gloo.com/ai-learning-center/gloo-ai-101/prompting-and-generation.md#system-prompt). Save it as `handbook.md` next to your script. Caching only starts above a minimum prompt length (see [Step 1](#step-1-put-the-stable-content-first)), so aim for a few thousand words.

***

## How Caching Works Through Gloo AI

Each provider caches in its own way. Through Gloo AI you either do nothing, or set one header or field:

| Provider | Type | What you do |
| :- | :- | :- |
| Anthropic (Claude) | Explicit | Send the `X-Cache-TTL` header (`5m` or `1h`). On [Anthropic Messages](https://docs.gloo.com/api-guides/anthropic-messages.md), use `cache_control`. |
| OpenAI | Implicit, plus an optional key | Nothing for prompts of 1,024+ tokens. Add `prompt_cache_key` to improve hit rates. |
| DeepSeek, Gemini, Qwen, MiniMax and others | Implicit | Nothing. Repeated prefixes are cached by the provider. |

Whatever the provider, a cache hit needs the same thing: **the beginning of the prompt must be identical, token for token, to a recent request.** Most of this tutorial is about getting that right.

The [Supported Models](https://docs.gloo.com/api-guides/supported-models.md#prompt-caching) catalog shows which models cache and how. For every provider's rates and rules, see the [Prompt Caching guide](https://docs.gloo.com/api-guides/prompt-caching.md).

***

## Step 1: Put the Stable Content First

A cache matches from the first token forward and stops at the first difference. Anything that changes per request, placed early, breaks the match for everything after it.

### A prompt that never caches

```json theme={null}
{
  "messages": [
    {
      "role": "system",
      "content": "Today is 2026-10-29 14:02:11. You are helping Maria at Grace Church.\n\nYou are a support assistant for our ministry software...\n\n<the 6,000-token handbook>"
    },
    { "role": "user", "content": "How do I export the giving report?" }
  ]
}
```

The timestamp and the user's name sit at the very start, so no two requests share a prefix. Every request pays full price for the handbook.

### The same prompt, cache-friendly

```json theme={null}
{
  "messages": [
    {
      "role": "system",
      "content": "You are a support assistant for our ministry software...\n\n<the 6,000-token handbook>"
    },
    {
      "role": "user",
      "content": "Context: today is 2026-10-29. The user is Maria at Grace Church.\n\nQuestion: How do I export the giving report?"
    }
  ]
}
```

Now the system message is byte-for-byte identical across users and requests, and everything that varies comes after it.

### Rules for a stable prefix

1. **Order content from most stable to least stable**: instructions, then reference documents, then conversation history, then the new question.

2. **Move per-request values to the end**: timestamps, user names, request IDs, retrieved snippets that differ per question.

3. **Keep the prefix deterministic**: build it from the same template every time. Don't shuffle examples, re-serialize JSON with a different key order, or reorder tool definitions between requests.

4. **Make the prefix long enough to cache.** Providers only cache above a minimum length:

   | Provider | Minimum prompt length |
   | :- | :- |
   | OpenAI | 1,024 tokens |
   | Gemini 2.5 Flash, 2.5 Flash Lite | 2,048 tokens |
   | Gemini 3 Flash, 3.1 Flash Lite, 3.5 Flash | 4,096 tokens |
   | Claude | Varies by model (for example 4,096 tokens for Claude Haiku 4.5); see [Anthropic's prompt caching docs](https://docs.anthropic.com/en/docs/build-with-claude/prompt-caching) |

5. **Version the prefix.** When you change the instructions or the handbook, treat it as a new prefix. The first request after a change writes a new cache entry.

> [!TIP]
>
> Put per-request values in the user message, not the system message, so the cached prefix stays identical on every call and the layout works on every provider.

This prompt builder, reused in every step, loads the handbook once so the system message is identical on every call:

```python Python theme={null}
# File: prompt.py
from pathlib import Path

INSTRUCTIONS = (
    "You are a support assistant for our ministry software. "
    "Answer only from the handbook below. If the handbook does not cover "
    "the question, say so and suggest contacting support."
)
HANDBOOK = Path("handbook.md").read_text(encoding="utf-8")

# Built once at startup: the stable prefix.
SYSTEM_PROMPT = f"{INSTRUCTIONS}\n\n<handbook>\n{HANDBOOK}\n</handbook>"


def build_messages(question: str, user_context: str = "") -> list[dict]:
    """Stable system message first; everything that varies goes last."""
    suffix = f"Context: {user_context}\n\nQuestion: {question}" if user_context else question
    return [
        {"role": "system", "content": SYSTEM_PROMPT},
        {"role": "user", "content": suffix},
    ]
```

```typescript TypeScript theme={null}
// File: prompt.ts
import { readFileSync } from "node:fs";

const INSTRUCTIONS =
  "You are a support assistant for our ministry software. " +
  "Answer only from the handbook below. If the handbook does not cover " +
  "the question, say so and suggest contacting support.";
const HANDBOOK = readFileSync("handbook.md", "utf-8");

// Built once at startup: the stable prefix.
export const SYSTEM_PROMPT = `${INSTRUCTIONS}\n\n<handbook>\n${HANDBOOK}\n</handbook>`;

export function buildMessages(question: string, userContext = "") {
  // Stable system message first; everything that varies goes last.
  const suffix = userContext ? `Context: ${userContext}\n\nQuestion: ${question}` : question;
  return [
    { role: "system", content: SYSTEM_PROMPT },
    { role: "user", content: suffix },
  ];
}
```

***

## Step 2: Read the Cache Fields in `usage`

Every response reports its token counts in `usage`. Cache activity appears there too, but the field names depend on the endpoint's response shape:

| Response shape | Endpoints | Prompt tokens | Read from cache | Written to cache |
| :- | :- | :- | :- | :- |
| Chat Completions | [Direct Completions](https://docs.gloo.com/api-reference/completions/direct.md), [Completions V2](https://docs.gloo.com/api-reference/completions/v2.md) | `prompt_tokens` | `prompt_tokens_details.cached_tokens` | `prompt_tokens_details.cache_write_tokens` |
| Responses | [Direct Responses](https://docs.gloo.com/api-reference/responses/v2.md), [Guarded Responses](https://docs.gloo.com/api-reference/responses/governed.md) | `input_tokens` | `input_tokens_details.cached_tokens` | `input_tokens_details.cache_write_tokens` |
| Anthropic Messages | [Anthropic Messages](https://docs.gloo.com/api-reference/messages/anthropic.md) | `input_tokens` | `cache_read_input_tokens` | `cache_creation_input_tokens` |

For example, a Direct Completions response reports:

```json theme={null}
"usage": {
  "prompt_tokens": 45,
  "completion_tokens": 42,
  "total_tokens": 87,
  "prompt_tokens_details": { "cached_tokens": 0, "cache_write_tokens": 0 },
  "cache_creation_input_tokens": 0,
  "cache_read_input_tokens": 0
}
```

The detail objects are present when the provider reports them, so read them defensively. This helper pulls the cache reads and writes out of all three shapes:

```python Python theme={null}
# File: cache_stats.py
def cache_stats(usage: dict | None) -> dict:
    """Normalize cache counts from Chat Completions, Responses or Anthropic usage."""
    usage = usage or {}
    details = usage.get("prompt_tokens_details") or usage.get("input_tokens_details") or {}
    return {
        "cache_read": details.get("cached_tokens") or usage.get("cache_read_input_tokens") or 0,
        "cache_write": details.get("cache_write_tokens") or usage.get("cache_creation_input_tokens") or 0,
    }
```

```typescript TypeScript theme={null}
// File: cacheStats.ts
export interface CacheStats {
  cacheRead: number;
  cacheWrite: number;
}

export function cacheStats(usage: Record<string, any> | null | undefined): CacheStats {
  // Normalize cache counts from Chat Completions, Responses or Anthropic usage.
  const u = usage ?? {};
  const details = u.prompt_tokens_details ?? u.input_tokens_details ?? {};
  return {
    cacheRead: details.cached_tokens || u.cache_read_input_tokens || 0,
    cacheWrite: details.cache_write_tokens || u.cache_creation_input_tokens || 0,
  };
}
```

What to expect:

* **First request with a new prefix**: `cache_read` is `0`. On Claude, `cache_write` is above `0`: the prefix was just stored.
* **The same prefix again, soon after**: `cache_read` is above `0`. That's a cache hit.
* **OpenAI, Gemini and other implicit providers**: watch `cache_read`. Many of these models charge nothing to write the cache, but some do, including the Gemini models and some newer OpenAI models; [the models endpoint](https://docs.gloo.com/api-guides/prompt-caching.md#checking-your-rates) shows which.

***

## Step 3: Turn On Caching for Claude with `X-Cache-TTL`

Claude models cache only when you ask. On the guarded [Completions V2](https://docs.gloo.com/api-guides/completions-v2.md#prompt-caching) endpoint, add the `X-Cache-TTL` header to opt in; you don't change your message content.

| `X-Cache-TTL` | Cache lifetime | Cache write billed at | Cache read billed at |
| :- | :- | :- | :- |
| `5m` | 5 minutes | 125% of the input rate | 10% of the input rate on most models |
| `1h` | 1 hour | 200% of the input rate | 10% of the input rate on most models |

Some newer Claude models read from the cache for less: 5% of the input rate on Claude Opus 5.5 and Sonnet 5.5, and 2.5% on Claude Fable 5.1. `GET /platform/v2/models` lists the exact `cache_read` and `cache_write` rates for each model.

Send the same question twice and compare `usage`:

```python Python theme={null}
# File: claude_cache.py
import os
import time

import requests

from cache_stats import cache_stats
from prompt import build_messages

API_URL = "https://platform.ai.gloo.com/ai/v2/guarded/chat/completions"
HEADERS = {
    "Authorization": f"Bearer {os.environ['GLOO_API_KEY']}",
    "Content-Type": "application/json",
    "X-Cache-TTL": "5m",
}


def ask(question: str) -> dict:
    payload = {
        "model": "gloo-anthropic-claude-opus-5.5",
        "messages": build_messages(question),
        "stream": False,
    }
    started = time.perf_counter()
    response = requests.post(API_URL, headers=HEADERS, json=payload, timeout=300)
    response.raise_for_status()
    elapsed = time.perf_counter() - started
    stats = cache_stats(response.json().get("usage"))
    print(f"{elapsed:5.2f}s  {stats}")
    return stats


ask("How do I export the giving report?")    # writes the cache
ask("How do I add a new volunteer role?")    # reads the cache
```

```typescript TypeScript theme={null}
// File: claudeCache.ts
import { cacheStats } from "./cacheStats";
import { buildMessages } from "./prompt";

const API_URL = "https://platform.ai.gloo.com/ai/v2/guarded/chat/completions";

async function ask(question: string) {
  const started = performance.now();
  const response = await fetch(API_URL, {
    method: "POST",
    headers: {
      Authorization: `Bearer ${process.env.GLOO_API_KEY}`,
      "Content-Type": "application/json",
      "X-Cache-TTL": "5m",
    },
    body: JSON.stringify({
      model: "gloo-anthropic-claude-opus-5.5",
      messages: buildMessages(question),
      stream: false,
    }),
  });
  if (!response.ok) throw new Error(`HTTP ${response.status}: ${await response.text()}`);
  const data = await response.json();
  const elapsed = (performance.now() - started) / 1000;
  const stats = cacheStats(data.usage);
  console.log(`${elapsed.toFixed(2)}s`, stats);
  return stats;
}

async function main() {
  await ask("How do I export the giving report?"); // writes the cache
  await ask("How do I add a new volunteer role?"); // reads the cache
}

main().catch((err) => {
  console.error(err);
  process.exit(1);
});
```

```bash cURL theme={null}
curl -X POST 'https://platform.ai.gloo.com/ai/v2/guarded/chat/completions' \
  -H "Authorization: Bearer $GLOO_API_KEY" \
  -H 'Content-Type: application/json' \
  -H 'X-Cache-TTL: 5m' \
  -d '{
    "model": "gloo-anthropic-claude-opus-5.5",
    "messages": [
      {"role": "system", "content": "You are a support assistant... <the full handbook>"},
      {"role": "user", "content": "How do I export the giving report?"}
    ],
    "stream": false
  }'
```

### What You'll See

The two questions differ, but the system message is the same, so the second request reads the prefix from cache. Your counts depend on your handbook and the model; the pattern looks like this:

```text theme={null}
 9.84s  {'cache_read': 0, 'cache_write': 6150}
 6.10s  {'cache_read': 6150, 'cache_write': 0}
```

If `cache_read` stays at `0`, jump to [Troubleshooting](#troubleshooting).

### Choosing `5m` or `1h`

The write premium is paid once per cache entry; the read discount is earned on every hit. Measured in input-token equivalents for each cached prefix token:

* **`5m`**: a write costs an extra 0.25, each hit saves 0.9 at a 10% read rate (0.95 on Claude Opus 5.5, the model in this tutorial). One hit within five minutes already pays for the write.
* **`1h`**: a write costs an extra 1.0, each hit saves 0.9 or more. It pays for itself from the second hit, and keeps the cache warm through quieter periods.

Use `5m` for steady traffic, where the same prefix is requested every few minutes. Consider `1h` when requests for a prefix are spread out, for example a document a team asks about a few times an hour.

### Your own cache breakpoints

With `X-Cache-TTL` on Completions V2, add at most three `cache_control` markers of your own on your message content. If your request already has four of its own, the request still succeeds, but your system messages aren't cached on that call.

***

## Step 4: Use `cache_control` on Anthropic Messages

If your code uses the Anthropic SDK, point it at the [Anthropic Messages](https://docs.gloo.com/api-guides/anthropic-messages.md) endpoint. There, `cache_control` works as it does on Anthropic's own API: set it at the top level to cache automatically, or on individual content blocks for fine-grained control.

Marking the system block caches everything up to and including it:

```python Python theme={null}
# File: anthropic_cache.py
import os

from anthropic import Anthropic

from cache_stats import cache_stats
from prompt import SYSTEM_PROMPT

client = Anthropic(
    base_url="https://platform.ai.gloo.com/ai/v2/direct/anthropic",
    auth_token=os.environ["GLOO_API_KEY"],
)


def ask(question: str) -> None:
    message = client.messages.create(
        model="gloo-anthropic-claude-opus-5.5",
        max_tokens=8192,
        system=[
            {
                "type": "text",
                "text": SYSTEM_PROMPT,
                "cache_control": {"type": "ephemeral"},
            }
        ],
        messages=[{"role": "user", "content": question}],
    )
    print(cache_stats(message.usage.model_dump()))


ask("How do I export the giving report?")
ask("How do I add a new volunteer role?")
```

```typescript TypeScript theme={null}
// File: anthropicCache.ts
import Anthropic from "@anthropic-ai/sdk";
import { cacheStats } from "./cacheStats";
import { SYSTEM_PROMPT } from "./prompt";

const client = new Anthropic({
  baseURL: "https://platform.ai.gloo.com/ai/v2/direct/anthropic",
  authToken: process.env.GLOO_API_KEY,
});

async function ask(question: string) {
  const message = await client.messages.create({
    model: "gloo-anthropic-claude-opus-5.5",
    max_tokens: 8192,
    system: [
      { type: "text", text: SYSTEM_PROMPT, cache_control: { type: "ephemeral" } },
    ],
    messages: [{ role: "user", content: question }],
  });
  console.log(cacheStats(message.usage as any));
}

async function main() {
  await ask("How do I export the giving report?");
  await ask("How do I add a new volunteer role?");
}

main().catch((err) => {
  console.error(err);
  process.exit(1);
});
```

```bash cURL theme={null}
# Top-level cache_control: automatic caching for the request
curl https://platform.ai.gloo.com/ai/v2/direct/anthropic/v1/messages \
  -H "Authorization: Bearer $GLOO_API_KEY" \
  -H "anthropic-version: 2023-06-01" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gloo-anthropic-claude-opus-5.5",
    "max_tokens": 8192,
    "cache_control": {"type": "ephemeral"},
    "system": "You are a support assistant... <the full handbook>",
    "messages": [
      {"role": "user", "content": "How do I export the giving report?"}
    ]
  }'
```

Confirm the hit in `usage.cache_creation_input_tokens` (first call) and `usage.cache_read_input_tokens` (second call). Pass your Gloo API key as the SDK's auth token (`auth_token` in Python, `authToken` in TypeScript); see [Set up the SDK](https://docs.gloo.com/api-guides/anthropic-messages.md#set-up-the-sdk).

> [!NOTE]
>
> Anthropic Messages is a [direct endpoint](https://docs.gloo.com/api-guides/endpoint-types.md): it bills the model's [published rate](https://docs.gloo.com/getting-started/pricing.md), and no [guardrails](https://docs.gloo.com/ai-learning-center/gloo-ai-101/application-patterns.md#guardrails) or values alignment run. Pick the endpoint for what your application needs; caching works on both.

***

## Step 5: Improve OpenAI Hit Rates with `prompt_cache_key`

OpenAI caches automatically once a prompt reaches 1,024 tokens, with no write fee on most models. Requests are routed to servers that recently processed the same prefix. Add `prompt_cache_key` to the request body to make requests that share a prefix more likely to land on the same server:

```python Python theme={null}
# File: openai_cache.py
import os

import requests

from cache_stats import cache_stats
from prompt import build_messages

API_URL = "https://platform.ai.gloo.com/ai/v2/guarded/chat/completions"
HEADERS = {
    "Authorization": f"Bearer {os.environ['GLOO_API_KEY']}",
    "Content-Type": "application/json",
}
CACHE_KEY = "support-handbook-v1"  # bump to -v2 when the handbook changes


def ask(question: str) -> None:
    payload = {
        "model": "gloo-openai-gpt-5.2",
        "messages": build_messages(question),
        "prompt_cache_key": CACHE_KEY,
        "stream": False,
    }
    response = requests.post(API_URL, headers=HEADERS, json=payload, timeout=300)
    response.raise_for_status()
    print(cache_stats(response.json().get("usage")))


ask("How do I export the giving report?")
ask("How do I add a new volunteer role?")
```

```typescript TypeScript theme={null}
// File: openaiCache.ts
import { cacheStats } from "./cacheStats";
import { buildMessages } from "./prompt";

const API_URL = "https://platform.ai.gloo.com/ai/v2/guarded/chat/completions";
const CACHE_KEY = "support-handbook-v1"; // bump to -v2 when the handbook changes

async function ask(question: string) {
  const response = await fetch(API_URL, {
    method: "POST",
    headers: {
      Authorization: `Bearer ${process.env.GLOO_API_KEY}`,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({
      model: "gloo-openai-gpt-5.2",
      messages: buildMessages(question),
      prompt_cache_key: CACHE_KEY,
      stream: false,
    }),
  });
  if (!response.ok) throw new Error(`HTTP ${response.status}: ${await response.text()}`);
  console.log(cacheStats((await response.json()).usage));
}

async function main() {
  await ask("How do I export the giving report?");
  await ask("How do I add a new volunteer role?");
}

main().catch((err) => {
  console.error(err);
  process.exit(1);
});
```

Good cache keys are **stable and describe the prefix**, not the request:

```text theme={null}
Good:  "support-handbook-v1", "sermon-generator", "theology-assistant-v1"
Avoid: "request-12345", "cache-abc", a user ID, a timestamp
```

When the instructions or the handbook change, change the key too.

`prompt_cache_key` is also accepted on [Responses](https://docs.gloo.com/api-guides/responses.md) and [Direct Responses](https://docs.gloo.com/api-guides/direct-responses.md), and is silently ignored for models from other providers, so it's safe to send on every request.

The cache read rate depends on the model: usually 10% of the input rate on GPT-5 and later models, and 25% on GPT-4.1. Some newer models, such as GPT-5.6 and GPT-6, also charge a cache write. Check each model's exact rates with [`GET /platform/v2/models`](https://docs.gloo.com/api-guides/prompt-caching#checking-your-rates).

***

## Step 6: Gemini, DeepSeek and Other Implicit Providers

Nothing to switch on: no `X-Cache-TTL`, `cache_control` or `prompt_cache_key`. Repeated prefixes are cached by the provider, and hits are billed as a separate `cache_read` segment at the model's reduced rate.

The work is all in [Step 1](#step-1-put-the-stable-content-first): a stable prefix above the model's minimum length. Run the same two-question test from Step 3 without the header, and check `cache_read`.

> [!NOTE]
>
> Cache read rates vary by model. Gemini cache reads are usually 10% of the input rate, and Gemini models also charge a cache write. Open-weight models such as DeepSeek, Qwen and MiniMax range from under 1% to more than half of the input rate, and some Qwen models charge a cache write too. Not every model in this group caches. The [models endpoint](https://docs.gloo.com/api-guides/prompt-caching.md#checking-your-rates) is the source of truth for each model's cache pricing.

***

## Step 7: Compare Before and After

Measure the effect on your own traffic. This script sends the same set of questions twice: once with a prompt that changes at the start (the "before" layout from Step 1), and once with the stable prefix. It totals cache reads, cache writes and latency, then converts the cache counts into **input-token equivalents saved**, using the rates from the [Prompt Caching guide](https://docs.gloo.com/api-guides/prompt-caching.md#providers-at-a-glance):

```text theme={null}
saved = cache_read × (1 − read_ratio) − cache_write × (write_ratio − 1)
```

For Claude with `X-Cache-TTL: 5m`, `write_ratio` is `1.25` and `read_ratio` is `0.10` on most models (`0.05` on Claude Opus 5.5, the model used here). For a GPT-5 model with no write fee, such as GPT-5.5, `read_ratio` is `0.10` and `write_ratio` is `1.0`. For any other model, take both ratios from [the models endpoint](https://docs.gloo.com/api-guides/prompt-caching.md#checking-your-rates): divide `cache_read` and `cache_write` by `input`.

```python Python theme={null}
# File: compare.py
import os
import time

import requests

from cache_stats import cache_stats
from prompt import SYSTEM_PROMPT, build_messages

API_URL = "https://platform.ai.gloo.com/ai/v2/guarded/chat/completions"
MODEL = "gloo-anthropic-claude-opus-5.5"
READ_RATIO, WRITE_RATIO = 0.05, 1.25  # Claude Opus 5.5, X-Cache-TTL: 5m
HEADERS = {
    "Authorization": f"Bearer {os.environ['GLOO_API_KEY']}",
    "Content-Type": "application/json",
    "X-Cache-TTL": "5m",
}
QUESTIONS = [
    "How do I export the giving report?",
    "How do I add a new volunteer role?",
    "Can two admins edit the same event?",
    "Where do I change the church logo?",
]


def unstable_messages(question: str) -> list[dict]:
    """The 'before' layout: a per-request value at the very start."""
    stamp = time.strftime("%Y-%m-%d %H:%M:%S")
    return [
        {"role": "system", "content": f"Request time: {stamp}.\n\n{SYSTEM_PROMPT}"},
        {"role": "user", "content": question},
    ]


def run(label: str, make_messages) -> None:
    totals = {"cache_read": 0, "cache_write": 0}
    seconds = 0.0
    for question in QUESTIONS:
        started = time.perf_counter()
        response = requests.post(
            API_URL,
            headers=HEADERS,
            json={"model": MODEL, "messages": make_messages(question), "stream": False},
            timeout=300,
        )
        response.raise_for_status()
        seconds += time.perf_counter() - started
        for key, value in cache_stats(response.json().get("usage")).items():
            totals[key] += value
    saved = totals["cache_read"] * (1 - READ_RATIO) - totals["cache_write"] * (WRITE_RATIO - 1)
    print(f"{label:<8} read={totals['cache_read']:>6}  "
          f"write={totals['cache_write']:>6}  saved≈{saved:>8.0f} input-token eq.  "
          f"avg {seconds / len(QUESTIONS):.2f}s")


run("before", unstable_messages)
run("after", build_messages)
```

```typescript TypeScript theme={null}
// File: compare.ts
import { cacheStats } from "./cacheStats";
import { SYSTEM_PROMPT, buildMessages } from "./prompt";

const API_URL = "https://platform.ai.gloo.com/ai/v2/guarded/chat/completions";
const MODEL = "gloo-anthropic-claude-opus-5.5";
const READ_RATIO = 0.05;
const WRITE_RATIO = 1.25; // Claude Opus 5.5, X-Cache-TTL: 5m
const QUESTIONS = [
  "How do I export the giving report?",
  "How do I add a new volunteer role?",
  "Can two admins edit the same event?",
  "Where do I change the church logo?",
];

// The 'before' layout: a per-request value at the very start.
function unstableMessages(question: string) {
  return [
    { role: "system", content: `Request time: ${new Date().toISOString()}.\n\n${SYSTEM_PROMPT}` },
    { role: "user", content: question },
  ];
}

async function run(label: string, makeMessages: (q: string) => object[]) {
  const totals = { cacheRead: 0, cacheWrite: 0 };
  let seconds = 0;
  for (const question of QUESTIONS) {
    const started = performance.now();
    const response = await fetch(API_URL, {
      method: "POST",
      headers: {
        Authorization: `Bearer ${process.env.GLOO_API_KEY}`,
        "Content-Type": "application/json",
        "X-Cache-TTL": "5m",
      },
      body: JSON.stringify({ model: MODEL, messages: makeMessages(question), stream: false }),
    });
    if (!response.ok) throw new Error(`HTTP ${response.status}: ${await response.text()}`);
    const stats = cacheStats((await response.json()).usage);
    seconds += (performance.now() - started) / 1000;
    totals.cacheRead += stats.cacheRead;
    totals.cacheWrite += stats.cacheWrite;
  }
  const saved = totals.cacheRead * (1 - READ_RATIO) - totals.cacheWrite * (WRITE_RATIO - 1);
  console.log(
    `${label.padEnd(8)} read=${totals.cacheRead} ` +
      `write=${totals.cacheWrite} saved≈${saved.toFixed(0)} input-token eq. ` +
      `avg ${(seconds / QUESTIONS.length).toFixed(2)}s`,
  );
}

async function main() {
  await run("before", unstableMessages);
  await run("after", buildMessages);
}

main().catch((err) => {
  console.error(err);
  process.exit(1);
});
```

### What You'll See

With the "before" layout, every request starts with a different timestamp, so every request writes a new cache entry and none reads one. With the stable prefix, the first request writes the cache and the next three read it. Illustrative output for a 6,000-token handbook:

```text theme={null}
before   read=     0  write= 24620  saved≈   -6155 input-token eq.  avg 9.71s
after    read= 18450  write=  6150  saved≈   15990 input-token eq.  avg 6.88s
```

Note the negative number in the first row: caching switched on with an unstable prefix pays the write premium on every request and never earns a read, so it costs more than no caching at all. Read the result as:

* **`read`** is the number of prompt tokens billed at the cache read rate instead of the full input rate.
* **`saved`** is how many full-price input tokens those reads were worth, after paying for the write. Multiply by the model's input rate from `GET /platform/v2/models` to turn it into money for your own volume.
* **Latency** typically drops on cache hits; how much depends on the model and prompt length, so measure it for your workload.

The models endpoint returns each segment's published rate; guarded endpoints are priced at the model's published rate plus 6.5% (see [Pricing](https://docs.gloo.com/getting-started/pricing.md)). For what you actually paid, check the [Billing dashboard](https://studio.ai.gloo.com/billing) and [API usage](https://docs.gloo.com/studio/api-usage.md) in Gloo AI Studio over a few days of real traffic.

***

## Step 8: Keep the Cache Warm in Production

These habits keep the hit rate high in production, expanding on the [caching best practices](https://docs.gloo.com/api-guides/prompt-caching.md#best-practices) in the guide:

* **Pre-warm on startup.** Send one request with your standard system prompt when your service starts, so the first real user reads from cache rather than paying for the write.
* **Expect a dip after failover.** Caches are held by each provider. When the provider serving your model fails and Gloo AI [fails over](https://docs.gloo.com/api-guides/provider-failover.md) to another provider of the same model, that request doesn't read the cache built on the first provider. Cached tokens recover as the cache warms on the new provider. Don't alert on a single cold request; watch the hit rate over time.
* **Log cache stats.** Record `cache_read` and `cache_write` with each request, next to the model and the prefix version (or `prompt_cache_key`). A sudden drop to zero usually means something started changing the prefix.
* **Keep long conversations cache-friendly.** In multi-turn chat, append new turns to the end and never rewrite earlier ones, so each request extends the previous prefix.

For handling the errors and retries around these requests, see [Build a Resilient Client](https://docs.gloo.com/tutorials/resilient-client.md).

***

## Troubleshooting

**`cache_read` is always `0` on Claude**
: Check that the `X-Cache-TTL` header is sent with `5m` or `1h` on every request (on Anthropic Messages, check `cache_control` instead). Check the prefix is above the model's minimum cacheable length. If your request already carries four `cache_control` markers of its own, your system messages aren't cached; remove one.

**`cache_read` is `0` on the second request, but `cache_write` is above `0` both times**
: Something in the prefix changes between requests. Diff the two system messages byte for byte: look for timestamps, IDs, user names, randomized examples, or JSON serialized with a different key order.

**Hits stop after a few minutes**
: `X-Cache-TTL: 5m` sets a five-minute cache lifetime. If your traffic is sparse, try `1h` and compare the totals with Step 7.

**OpenAI requests rarely hit**
: Make sure the prompt is at least 1,024 tokens, and add a stable `prompt_cache_key`. A key that changes per request or per user spreads requests across servers and lowers the hit rate.

**`prompt_cache_key` has no effect**
: It only applies to OpenAI models and is ignored for other providers. For Claude, use `X-Cache-TTL` or `cache_control`.

**Cached tokens drop for a while, then recover**
: The request was most likely served by a different provider of the same model after a failover, which starts with a cold cache. See [Provider Failover](https://docs.gloo.com/api-guides/provider-failover.md#what-you-observe).

**The detail objects are missing from `usage`**
: They're present when the provider reports them. The `cache_stats` helper treats a missing field as `0`; if a model never reports cache counts, check its caching support in [Supported Models](https://docs.gloo.com/api-guides/supported-models.md#prompt-caching).

***

## Next Steps

> **[Prompt Caching Guide](https://docs.gloo.com/api-guides/prompt-caching.md)**
>
> Every provider's caching type, billing rates and minimum lengths.

> **[Build a Resilient Client](https://docs.gloo.com/tutorials/resilient-client.md)**
>
> Retries, backoff and stream recovery around your cached requests.

> **[Provider Failover](https://docs.gloo.com/api-guides/provider-failover.md)**
>
> What happens to your request, and your cache, when a provider fails.

> **[Get Models API](https://docs.gloo.com/api-reference/get-models/v2.md)**
>
> Current per-model rates and capabilities.

> **[Cost and Speed](https://docs.gloo.com/ai-learning-center/gloo-ai-104/cost-and-speed.md#prompt-caching)**
>
> How caching fits with model choice and reasoning effort to lower cost.

> **[Context Is King](https://docs.gloo.com/ai-learning-center/gloo-ai-102/context-is-king.md)**
>
> How to structure the context you put in a prompt.

> **[Best Practices](https://docs.gloo.com/best-practices/overview.md)**
>
> Production patterns for reliable, trustworthy applications.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.