Controlling Cost

• know what a request costs, from the catalog and from usage
• match the model to each step of your app
• control reasoning and output tokens
• get more cache hits
• keep conversations and images from growing the bill
• track spend and set a ceiling

​

Know what a request costs

• Input tokens are everything you send: instructions, conversation history, documents and images. They bill at pricing.input.
• Output tokens are everything the model writes, including any reasoning, whether or not you see it. They bill at pricing.output, which is usually several times the input rate.
• Cached tokens bill at pricing.cache_read, and on some models writing the cache bills at pricing.cache_write. See Prompt Caching.

import requests

catalog = requests.get("https://platform.ai.gloo.com/platform/v2/models", timeout=30).json()["data"]
RATES = {m["id"]: m["pricing"] for m in catalog}


def per_token(pricing: dict, segment: str) -> float:
    return float(pricing[segment]["rate_per_1m_tokens"]) / 1_000_000


def estimate_cost(model: str, usage: dict, guarded: bool) -> float:
    """List-price estimate from a Responses or Chat Completions `usage` object."""
    pricing = RATES[model]
    input_tokens = usage.get("input_tokens", usage.get("prompt_tokens", 0))
    output_tokens = usage.get("output_tokens", usage.get("completion_tokens", 0))
    details = usage.get("input_tokens_details") or usage.get("prompt_tokens_details") or {}
    cached = details.get("cached_tokens") or 0

    cost = (input_tokens - cached) * per_token(pricing, "input")
    cost += output_tokens * per_token(pricing, "output")
    if cached and "cache_read" in pricing:
        cost += cached * per_token(pricing, "cache_read")
    return cost * (1.065 if guarded else 1.0)

const res = await fetch("https://platform.ai.gloo.com/platform/v2/models");
const RATES = new Map<string, any>((await res.json()).data.map((m: any) => [m.id, m.pricing]));

function perToken(pricing: any, segment: string): number {
  return Number(pricing[segment].rate_per_1m_tokens) / 1_000_000;
}

/** List-price estimate from a Responses or Chat Completions `usage` object. */
function estimateCost(model: string, usage: any, guarded: boolean): number {
  const pricing = RATES.get(model);
  const inputTokens = usage.input_tokens ?? usage.prompt_tokens ?? 0;
  const outputTokens = usage.output_tokens ?? usage.completion_tokens ?? 0;
  const details = usage.input_tokens_details ?? usage.prompt_tokens_details ?? {};
  const cached = details.cached_tokens ?? 0;

  let cost = (inputTokens - cached) * perToken(pricing, "input");
  cost += outputTokens * perToken(pricing, "output");
  if (cached && pricing.cache_read) cost += cached * perToken(pricing, "cache_read");
  return cost * (guarded ? 1.065 : 1);
}

​

Match the model to each step

• Split your app into steps (classify, extract, draft, check) and choose a model per step. Short classifications, routing decisions and extractions often run well on small, fast models; long reasoning and careful writing may need a larger one.
• Test the cheaper model first on your own examples, and move up only where quality falls short. See Testing, Evaluation and Observability.
• Pin the model you chose. A pinned model is billed at that model’s catalog rate on every call, rather than a rate that changes per request. On guarded Completions, model_family and auto-routing can pick a different model, at a different price, per request.
• Watch for replacements. Pinning isn’t a price guarantee: when a model you use is deprecated, its requests can be redirected to a replacement priced differently, and deprecation_note states any increase. Check catalog rates on a schedule, log the X-Gloo-Model-Redirected-From header, and follow Model Updates. See Choosing, Pinning and Updating Models.

​

Control reasoning and output tokens

• Know which models think unprompted. A model with reasoning.thinks_by_default: true reasons, and bills that reasoning, even when you send no reasoning setting.
• Lower the effort, not the limit. The reasoning effort is the main control over how long a model thinks. Use low, or none on a model whose effort_levels include it, for short tasks. Lowering max_output_tokens (or max_tokens) doesn’t make the model think less; it cuts the answer off when thinking uses the limit.
• The limit is a ceiling, not a charge. You pay for the output tokens the model uses, not for the limit you set. Size the limit so answers aren’t cut off, then let effort set the cost. See Size the limit.
• Read where output went. Where the model reports it, usage.output_tokens_details.reasoning_tokens (Responses) or usage.completion_tokens_details.reasoning_tokens (Chat Completions) is the part of the output that was reasoning. Claude models report it as 0 or leave it out on some endpoints, and their reasoning is still in the output count.
• Ask for shorter answers. “Answer in two sentences” or a tight output schema cuts output tokens on every model.

​

Get more cache hits

• Put stable content first. Order each request from most stable to least: instructions, reference documents, tool definitions, conversation history, then the new question. Keep timestamps, user names and per-request snippets at the end.
• Keep the prefix byte-identical. Build it from the same template every time; don’t reorder examples, tools or JSON keys between requests.
• Turn on caching where it’s opt-in. Claude models cache when you send the X-Cache-TTL header (5m or 1h); on Anthropic Messages use cache_control. OpenAI, DeepSeek, Gemini and most other models cache automatically above a minimum prompt length.
• Use a stable prompt_cache_key on OpenAI models, named after the prompt it caches, such as "support-assistant-v3", and change it when the instructions change. Per-request keys defeat the cache.
• Check that it works. Log the cached-token count from usage with every request. A hit rate that drops to zero usually means something started changing the prefix. Expect fewer hits for a while after a provider failover, since each provider holds its own cache.

​

Keep conversations and images in check

• Trim or summarise long histories. Keep the instructions and the recent turns; drop or summarise the oldest. See Conversations and Context Windows.
• Append, don’t rewrite. Adding new turns to the end keeps the earlier conversation as a cacheable prefix.
• Mind images. An image is billed as input tokens at the model’s input rate, and again on every turn it stays in the history. Scale images down before you send them, and describe an image once and keep the text when the conversation moves on. See Images in Conversations.
• Send only the context the question needs. Retrieving the three most relevant passages costs less than pasting a whole document, and often answers better.

​

Track spend and set a ceiling

• Watch the Usage page. API Usage in Gloo AI Studio shows spend, requests and tokens (input, output and cache) over time. Filter by API key to see which application is spending, so give each application and environment its own API key.
• Log usage per feature. Studio shows totals; your own logs tell you which feature or customer drove them. Record the model, the token counts and your cost estimate with every request.
• Know your ceiling. Organizations on prepaid credits have a monthly API spend limit, $4,000 per calendar month (UTC) unless Gloo has set a different one. When it’s reached, requests return 429 with MONTHLY_SPEND_LIMIT_REACHED until the month resets. Contact support to raise or lower it, including lowering it as a safety net for a new app.
• Cap your own loops. Retries, tool loops and agents can multiply requests. Bound them with a retry budget and a step limit. See Error Handling and Retries and Tools, Structured Output and Agents.

​

Summary

• Output costs more than input, and reasoning is output: control it with effort, not by shrinking the limit.
• Pick the smallest model that passes your tests for each step, and pin it.
• Put stable content first, turn on caching where it’s opt-in, and check the cached-token count.
• Trim long conversations and shrink images, since every turn resends them.
• Log usage per feature, filter Studio usage by API key, and know your monthly spend limit.

​

Next steps