Lesson 1: Streaming Responses

Don’t Make People Wait

A model writes its answer a few words at a time. Without streaming, your app waits until the whole answer is finished and then shows it in one go. For a long answer, that can mean several seconds of a blank screen. Streaming sends each piece of the answer to your app as soon as the model writes it. The first words appear almost at once, and the rest fill in as the model goes. It’s the reason chat apps feel quick, and it’s the first thing most real AI apps need.

Core Concepts

How a Stream Arrives

When you set "stream": true, the API keeps the connection open and sends a series of small messages called Server-Sent Events (SSE). Each event carries one piece of the answer, called a delta. Your app adds each delta to what it already has. On the Responses endpoints, each event has a type. The ones you’ll use most:

Event	What it means
`response.output_text.delta`	A new piece of answer text
`response.completed`	The response is finished, with `usage`
`response.incomplete`	The answer stopped at `max_output_tokens`
`response.failed`	Something went wrong after the stream started (Direct Responses)
`error` (or another `error.` event)	Something went wrong after the stream started (guarded Responses)

On the Chat Completions endpoints, each chunk carries its text in choices[0].delta.content, and the stream ends with a finish_reason such as "stop".

When to Stream

Stream when a person is waiting to read the answer: chat windows, writing assistants, anything interactive. Don’t bother when your code needs the whole answer before it can do anything, such as a background job or a step that parses JSON (Lesson 3). A normal request is simpler there.

Errors Can Arrive Mid-Stream

A stream can fail at two points:

Before any text arrives. On the direct endpoints you get a normal HTTP error, such as a 503, and the OpenAI SDK raises it as an exception.
After text has started. The HTTP status is already 200, so the error arrives in the stream: a response.failed event on Direct Responses, or an error chunk on Chat Completions. The OpenAI SDK raises an error chunk as an exception from the loop, so catch it.

Gloo retries some of these for you. If a stream fails before the answer starts, Gloo can retry it on another provider, so the stream just starts a little later. For models that support it, a stream cut off mid-answer is continued by another provider in the same response. Lesson 9 covers what your app should still handle.

Get Started

This example streams a short answer from Direct Responses and prints it as it arrives.

pip install openai

import os
import openai
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["GLOO_API_KEY"],
    base_url="https://platform.ai.gloo.com/ai/v2/direct",
)

stream = client.responses.create(
    model="gloo-anthropic-claude-sonnet-5.5",
    input="Write a short, encouraging note for a volunteer's first day.",
    stream=True,
)

try:
    for event in stream:
        if event.type == "response.output_text.delta":
            print(event.delta, end="", flush=True)
        elif event.type == "response.incomplete":
            print("\n[Stopped at the output limit]")
        elif event.type == "response.failed":
            print("\n[The answer was interrupted]")
except openai.APIError:
    # The connection dropped after text had started. Keep what was printed.
    print("\n[The answer was interrupted]")
print()

curl -N -X POST 'https://platform.ai.gloo.com/ai/v2/direct/responses' \
  -H "Authorization: Bearer $GLOO_API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "gloo-anthropic-claude-sonnet-5.5",
    "input": "Write a short, encouraging note for a volunteer'\''s first day.",
    "stream": true
  }'

With cURL, -N turns off buffering so you see each event as it arrives. The raw stream looks like this (trimmed):

event: response.output_text.delta
data: {"type":"response.output_text.delta","output_index":0,"delta":"Welcome "}

event: response.output_text.delta
data: {"type":"response.output_text.delta","output_index":0,"delta":"to the team! "}

event: response.completed
data: {"type":"response.completed",...}

The Same Thing on Chat Completions

If your app uses the Chat Completions shape, read delta.content from each chunk. Add stream_options to get token usage in a final chunk:

import openai

stream = client.chat.completions.create(
    model="gloo-anthropic-claude-sonnet-5.5",
    messages=[{"role": "user", "content": "Write a short, encouraging note for a volunteer's first day."}],
    stream=True,
    stream_options={"include_usage": True},
)

try:
    for chunk in stream:
        if not chunk.choices:
            print("\nUsage:", chunk.usage)  # the final usage chunk
            continue
        if chunk.choices[0].delta.content:
            print(chunk.choices[0].delta.content, end="", flush=True)
except openai.APIError:
    # An error chunk after text has started. Keep what was printed.
    print("\n[The answer was interrupted]")

Good to Know

Keep your API key on the server. Don’t call the API from browser code, where anyone can read the key. Stream from your server and pass the text on to the browser. The streaming tutorial builds a small proxy that does this. See API Key Security.
Check the status before reading. A request that fails before the stream starts returns a JSON error, not events, so check for it first so a 401 shows a clear message instead of a parsing error.
Some chunks have no text. The first and last chunks often carry no content. Skip them quietly.
Keep what the user has seen. If a stream fails partway, leave the partial text on screen and mark it as interrupted, rather than letting it vanish. Handling Streaming Failures covers this in production.
Reasoning can stream too. Reasoning models can send their reasoning before the answer, in separate events. Reasoning shows how to read them.

Key Takeaway

Streaming shows the answer as the model writes it, so your app feels fast. Set "stream": true, add each text delta as it arrives, and be ready for a stream to end early: at the output limit, or with an error after some text has already appeared.

Try it on Gloo: the Streaming tutorial builds a full streaming client and server-side proxy in several languages. The Direct Responses streaming reference lists every event type.

Learn more

Direct Responses: Streaming: every event type on the Responses stream
Streaming AI Responses in Real Time: a step-by-step tutorial with a server-side proxy
Handling Streaming Failures: retry budgets, continuing an interrupted answer and partial-output UX
Streaming on the direct endpoints: how errors arrive before and after output starts

What’s next

Next, you’ll keep a conversation going across many turns, and learn what to do when it gets too long.

Lesson 2: Conversations That Remember

Keep history, stay inside the context window and trim old turns.

AI 101: Fundamentals

AI 102: Prompting

AI 103: RAG

AI 104: Building with Gloo

Lesson 1: Streaming Responses

Don’t Make People Wait

Core Concepts

How a Stream Arrives

When to Stream

Errors Can Arrive Mid-Stream

Get Started

The Same Thing on Chat Completions

Good to Know

Key Takeaway

Learn more

What’s next

Lesson 2: Conversations That Remember

​Don’t Make People Wait

​Core Concepts

​How a Stream Arrives

​When to Stream

​Errors Can Arrive Mid-Stream

​Get Started

​The Same Thing on Chat Completions

​Good to Know

​Key Takeaway

​Learn more

​What’s next

Lesson 2: Conversations That Remember

Don’t Make People Wait

Core Concepts

How a Stream Arrives

When to Stream

Errors Can Arrive Mid-Stream

Get Started

The Same Thing on Chat Completions

Good to Know

Key Takeaway

Learn more

What’s next