Lesson 8: Cost and Speed

​

Every Token Counts

​

Core Concepts

​

How a Request Is Priced

​

Prompt Caching

​

Choosing a Model

• Start from what the task needs. Check the catalog for the capabilities you rely on (supports_tools, supports_vision, supports_reasoning) and a context_window big enough for your inputs.
• Compare rates. Models that can do the job may differ a lot in price.
• Test with your own prompts. Run a handful of real examples through two or three candidates and compare quality, speed and cost. The Chat Playground is a quick way to try them, and the Model Explorer compares them side by side, with speed ratings.
• Pin the model in production. A fixed model ID means routing never swaps in a different model. When a pinned model is retired, its requests go to a replacement that can be priced differently, so check your models for deprecations (Lesson 9’s Models Come and Go).

​

Reasoning Effort

• Send a level the model lists. Each model’s reasoning.effort_levels in the catalog shows the levels it runs as sent. A level it doesn’t list isn’t guaranteed: the request may run and list it in ignored_params, or return 400.
• Many models reason by default. With no setting, a model does what its provider does by default, and you pay for that reasoning too. The catalog’s thinks_by_default tells you which ones.
• none doesn’t always turn reasoning off. It does on models whose effort_levels include none. On others, such as Claude 5 models, it isn’t reliable, and the reasoning they do is still billed.
• Reasoning uses your output limit. It counts toward max_output_tokens (or max_tokens). A high effort with a small limit can use it all before the answer starts.

​

Get Started

pip install openai requests

import os
import requests
from openai import OpenAI

MODEL = "gloo-anthropic-claude-sonnet-5.5"
ORDER = ["none", "minimal", "low", "medium", "high", "xhigh", "max"]

# 1. Read the model's entry from the public catalog (no key needed).
catalog = requests.get("https://platform.ai.gloo.com/platform/v2/models").json()["data"]
model = next(m for m in catalog if m["id"] == MODEL)

input_rate = float(model["pricing"]["input"]["rate_per_1m_tokens"])
output_rate = float(model["pricing"]["output"]["rate_per_1m_tokens"])
print(f"{MODEL}: ${input_rate} in / ${output_rate} out per 1M tokens")


# 2. Pick the nearest effort the model lists (from the Reasoning guide).
def pick_effort(model, wanted):
    reasoning = model.get("reasoning")
    if reasoning is None:
        return None  # the model doesn't reason
    levels = reasoning["effort_levels"]
    if wanted == "none":
        return "none" if "none" in levels else None
    graded = [level for level in levels if level != "none"]
    if not graded:
        return None  # no graded levels: an effort would change nothing
    return min(graded, key=lambda level: abs(ORDER.index(level) - ORDER.index(wanted)))


effort = pick_effort(model, "low")
print("Effort to send:", effort)

# 3. Send the request.
client = OpenAI(
    api_key=os.environ["GLOO_API_KEY"],
    base_url="https://platform.ai.gloo.com/ai/v2/direct",
)
response = client.responses.create(
    model=MODEL,
    input="Suggest a title for a talk on gratitude.",
    max_output_tokens=4000,
    **({"reasoning": {"effort": effort}} if effort else {}),
)
print(response.output_text)

# 4. Estimate the cost from usage, at list price, before any cache discount.
usage = response.usage
cost = usage.input_tokens * input_rate / 1e6 + usage.output_tokens * output_rate / 1e6
print(f"{usage.input_tokens} in, {usage.output_tokens} out, about ${cost:.6f}")

​

Turn On Caching

curl -X POST 'https://platform.ai.gloo.com/ai/v2/guarded/chat/completions' \
  -H "Authorization: Bearer $GLOO_API_KEY" \
  -H 'Content-Type: application/json' \
  -H 'X-Cache-TTL: 5m' \
  -d '{
    "model": "gloo-anthropic-claude-opus-5.5",
    "messages": [
      { "role": "system", "content": "You are a helpful assistant for a church office. <long, unchanging guidance here>" },
      { "role": "user", "content": "What does the Bible say about forgiveness?" }
    ]
  }'

{
  "model": "gloo-openai-gpt-5.2",
  "messages": [
    { "role": "system", "content": "You are a theological assistant..." },
    { "role": "user", "content": "Explain the Trinity" }
  ],
  "prompt_cache_key": "theology-assistant-v1"
}

​

Good to Know

• Measure before you optimize. Log usage for every request so you know where tokens go. Conversation history (Lesson 2) grows with every turn, so watch long chats.
• Stable text first, changing text last. This one habit helps caching on every provider.
• Failover means fewer cache hits for a while. Caches are held by each provider, so a request served by a different provider doesn’t read the cache built on the original one.
• Check ignored_params. When an effort isn’t applied as sent, the response lists it there. Match on the params list, not the reason text.
• Stream for speed you can feel. Streaming (Lesson 1) doesn’t make the answer cheaper, but the first words appear much sooner.

​

Key Takeaway

​

Learn more

• Pricing at a Glance and Billing & Plans: how credits and pricing work
• Prompt Caching: every provider’s caching type, rates and best practices
• Reasoning: effort levels, defaults by model family and the output limit
• Prompt Caching tutorial and Reasoning tutorial: step-by-step projects in Python and TypeScript
• Endpoint Types: Choosing a model: pinning, capabilities and deprecation

​

What’s next