> ## Documentation Index
> Fetch the complete documentation index at: https://docs.gloo.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Lesson 8: Cost and Speed

> Understand what a Gloo request costs, lower it with prompt caching, and choose a model and reasoning effort that fit the job.

## Every Token Counts

Your app works. Now it has users, and two questions come up quickly: *how much is this costing?* and *why does that answer take so long?*

Both come down to a few choices: which model you call, how much text you send and get back, whether repeated text is cached, and how hard you ask the model to think. This lesson shows how each one works on Gloo and how to measure it.

***

## Core Concepts

### How a Request Is Priced

You pay per **token**, at the rate of the model you call. Input tokens (what you send) and output tokens (what the model writes) have separate rates, listed per million tokens.

The rates come from the public model catalog, [`GET /platform/v2/models`](https://docs.gloo.com/api-reference/get-models/v2). Each model's `pricing` object has `input` and `output`, each with `rate_per_1m_tokens`. These are **list prices**. What you pay depends on the endpoint:

| Endpoint family | What you pay |
| :- | :- |
| **Direct** | The model's published rate, nothing added. |
| **Guarded** | The published rate plus 6.5% for guardrails, output moderation and values alignment (answers shaped to a faith tradition you choose). |
| **Grounded** | Grounded Responses: the published rate, with no separate charge for retrieval. Grounded Completions builds on guarded Completions and has the guarded platform price. |

Every response tells you what it used, in `usage`. On the Responses endpoints that's `input_tokens` and `output_tokens`; on Chat Completions, `prompt_tokens` and `completion_tokens`.

### Prompt Caching

Many requests start with the same text: a long system prompt, a shared document, the early turns of a conversation. **Prompt caching** lets the provider reuse its work on that repeated beginning, called the **prompt prefix**. Cached tokens are billed at a discounted rate, and responses arrive faster.

How you turn it on depends on the provider:

| Provider | Type | What you do |
| :- | :- | :- |
| OpenAI | Implicit, plus an optional key | Nothing for prompts of 1,024 tokens or more. Add `prompt_cache_key` to improve cache hits. |
| Anthropic | Explicit | Add the `X-Cache-TTL` header (`5m` or `1h`) on the guarded and grounded Completions endpoints. |
| Gemini, DeepSeek and others | Implicit | Nothing. |

For the discount to apply, the repeated text has to come first. Put your instructions and any shared documents at the start, and the parts that change, such as the user's question, at the end.

Rates differ by provider. For example, Anthropic cache reads cost 10% of the input rate on most models and less on some newer ones (5% on Claude Opus 5.5 and Sonnet 5.5, 2.5% on Claude Fable 5.1), while cache writes cost 125% (for `5m`) or 200% (for `1h`). Most OpenAI models charge nothing to write the cache, though some newer ones do. The [Prompt Caching](https://docs.gloo.com/api-guides/prompt-caching.md) guide explains each provider, and the models endpoint lists [every model's exact rates](https://docs.gloo.com/api-guides/prompt-caching.md#checking-your-rates).

### Choosing a Model

Pick a model for each job:

* **Start from what the task needs.** Check the catalog for the capabilities you rely on (`supports_tools`, `supports_vision`, `supports_reasoning`) and a `context_window` big enough for your inputs.
* **Compare rates.** Models that can do the job may differ a lot in price.
* **Test with your own prompts.** Run a handful of real examples through two or three candidates and compare quality, speed and cost. The [Chat Playground](https://docs.gloo.com/studio/playground-chat.md) is a quick way to try them, and the [Model Explorer](https://studio.ai.gloo.com/models) compares them side by side, with speed ratings.
* **Pin the model in production.** A fixed model ID means routing never swaps in a different model. When a pinned model is retired, its requests go to a replacement that can be priced differently, so check your models for deprecations (Lesson 9's [Models Come and Go](https://docs.gloo.com/ai-learning-center/gloo-ai-104/production-ready.md#models-come-and-go)).

### Reasoning Effort

Reasoning models think before they answer, and that thinking is billed as **output tokens**, whether or not you see it. More effort usually means a better answer on hard problems, and also more tokens and a longer wait. (AI 102's [Models That Think Before They Answer](https://docs.gloo.com/ai-learning-center/gloo-ai-102/models-that-think.md) explains reasoning from the start.)

On the Responses and Chat Completions endpoints, you set it with `reasoning: {"effort": ...}` (or `reasoning_effort` on Chat Completions). The levels run from `none` through `minimal`, `low`, `medium`, `high` and `xhigh` to `max`. Four things to know:

* **Send a level the model lists.** Each model's `reasoning.effort_levels` in the catalog shows the levels it runs as sent. A level it doesn't list isn't guaranteed: the request may run and list it in `ignored_params`, or return `400`.
* **Many models reason by default.** With no setting, a model does what its provider does by default, and you pay for that reasoning too. The catalog's `thinks_by_default` tells you which ones.
* **`none` doesn't always turn reasoning off.** It does on models whose `effort_levels` include `none`. On others, such as Claude 5 models, it isn't reliable, and the reasoning they do is still billed.
* **Reasoning uses your output limit.** It counts toward `max_output_tokens` (or `max_tokens`). A high effort with a small limit can use it all before the answer starts.

***

## Get Started

This script looks up a model's rates and reasoning levels in the catalog, sends a request at a level the model lists, and estimates what it cost at list price.

```bash theme={null}
pip install openai requests
```

```python Python theme={null}
import os
import requests
from openai import OpenAI

MODEL = "gloo-anthropic-claude-sonnet-5.5"
ORDER = ["none", "minimal", "low", "medium", "high", "xhigh", "max"]

# 1. Read the model's entry from the public catalog (no key needed).
catalog = requests.get("https://platform.ai.gloo.com/platform/v2/models").json()["data"]
model = next(m for m in catalog if m["id"] == MODEL)

input_rate = float(model["pricing"]["input"]["rate_per_1m_tokens"])
output_rate = float(model["pricing"]["output"]["rate_per_1m_tokens"])
print(f"{MODEL}: ${input_rate} in / ${output_rate} out per 1M tokens")


# 2. Pick the nearest effort the model lists (from the Reasoning guide).
def pick_effort(model, wanted):
    reasoning = model.get("reasoning")
    if reasoning is None:
        return None  # the model doesn't reason
    levels = reasoning["effort_levels"]
    if wanted == "none":
        return "none" if "none" in levels else None
    graded = [level for level in levels if level != "none"]
    if not graded:
        return None  # no graded levels: an effort would change nothing
    return min(graded, key=lambda level: abs(ORDER.index(level) - ORDER.index(wanted)))


effort = pick_effort(model, "low")
print("Effort to send:", effort)

# 3. Send the request.
client = OpenAI(
    api_key=os.environ["GLOO_API_KEY"],
    base_url="https://platform.ai.gloo.com/ai/v2/direct",
)
response = client.responses.create(
    model=MODEL,
    input="Suggest a title for a talk on gratitude.",
    max_output_tokens=4000,
    **({"reasoning": {"effort": effort}} if effort else {}),
)
print(response.output_text)

# 4. Estimate the cost from usage, at list price, before any cache discount.
usage = response.usage
cost = usage.input_tokens * input_rate / 1e6 + usage.output_tokens * output_rate / 1e6
print(f"{usage.input_tokens} in, {usage.output_tokens} out, about ${cost:.6f}")
```

Direct Responses bills the published rate, so with no cached tokens this estimate is what you pay. With [prompt caching](https://docs.gloo.com/api-guides/prompt-caching.md), cached tokens bill at their cache read and write rates instead. On a guarded endpoint, the price is the model's rate plus 6.5%.

### Turn On Caching

For an Anthropic model on the guarded Completions endpoint, add one header:

```bash cURL theme={null}
curl -X POST 'https://platform.ai.gloo.com/ai/v2/guarded/chat/completions' \
  -H "Authorization: Bearer $GLOO_API_KEY" \
  -H 'Content-Type: application/json' \
  -H 'X-Cache-TTL: 5m' \
  -d '{
    "model": "gloo-anthropic-claude-opus-5.5",
    "messages": [
      { "role": "system", "content": "You are a helpful assistant for a church office. <long, unchanging guidance here>" },
      { "role": "user", "content": "What does the Bible say about forgiveness?" }
    ]
  }'
```

For an OpenAI model, add a stable `prompt_cache_key` to the body, and change it when you change the system prompt:

```json theme={null}
{
  "model": "gloo-openai-gpt-5.2",
  "messages": [
    { "role": "system", "content": "You are a theological assistant..." },
    { "role": "user", "content": "Explain the Trinity" }
  ],
  "prompt_cache_key": "theology-assistant-v1"
}
```

Caching only starts once the repeated beginning is long enough: 1,024 tokens on OpenAI, and a model-specific minimum on Anthropic. Replace the placeholder text above with your full system prompt before you test it.

To see whether caching is working, look at `cached_tokens` in the response's usage (`input_tokens_details.cached_tokens` on Responses, `prompt_tokens_details.cached_tokens` on Chat Completions), and compare your input costs over time in the [billing dashboard](https://studio.ai.gloo.com/billing).

**Go further:** the [Prompt Caching tutorial](https://docs.gloo.com/tutorials/prompt-caching.md) restructures a prompt so it hits the cache, turns caching on for Claude and OpenAI models and measures the savings. The [Reasoning tutorial](https://docs.gloo.com/tutorials/reasoning.md) picks an effort level the model supports, sizes the output limit and catches cut-off answers.

***

## Good to Know

* **Measure before you optimize.** Log `usage` for every request so you know where tokens go. Conversation history (Lesson 2) grows with every turn, so watch long chats.
* **Stable text first, changing text last.** This one habit helps caching on every provider.
* **Failover means fewer cache hits for a while.** Caches are held by each provider, so a request served by a different provider doesn't read the cache built on the original one.
* **Check `ignored_params`.** When an effort isn't applied as sent, the response lists it there. Match on the `params` list, not the `reason` text.
* **Stream for speed you can feel.** Streaming (Lesson 1) doesn't make the answer cheaper, but the first words appear much sooner.

***

## Key Takeaway

Cost is tokens times the model's rate, plus 6.5% on guarded endpoints and Grounded Completions. Lower it by choosing a model that fits the task, putting repeated text first so it can be cached, and setting a reasoning effort the model lists, no higher than the task needs. Measure with `usage` on every response and check real spend in Studio.

> [!TIP]
>
> **Try it on Gloo:** [Supported Models](https://docs.gloo.com/api-guides/supported-models.md) shows live rates, caching support and effort levels for every model, and the [Prompt Caching](https://docs.gloo.com/api-guides/prompt-caching.md) guide covers each provider's setup and rates.

***

## Learn more

* [Pricing at a Glance](https://docs.gloo.com/getting-started/pricing.md) and [Billing & Plans](https://docs.gloo.com/studio/billing.md): how credits and pricing work
* [Prompt Caching](https://docs.gloo.com/api-guides/prompt-caching.md): every provider's caching type, rates and best practices
* [Reasoning](https://docs.gloo.com/api-guides/reasoning.md): effort levels, defaults by model family and the output limit
* [Prompt Caching tutorial](https://docs.gloo.com/tutorials/prompt-caching.md) and [Reasoning tutorial](https://docs.gloo.com/tutorials/reasoning.md): step-by-step projects in Python and TypeScript
* [Endpoint Types: Choosing a model](https://docs.gloo.com/api-guides/endpoint-types.md#choosing-a-model): pinning, capabilities and deprecation

***

## What's next

In the last lesson, you'll handle rate limits and errors, see how failover keeps your app running, and keep up with model changes.

> [card] [Lesson 9: Production-Ready](https://docs.gloo.com/ai-learning-center/gloo-ai-104/production-ready.md)
>
> Rate limits and retries, failover and routing, and model lifecycle.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.