Skip to main content
With streaming, your application receives tokens one at a time as the model generates them instead of waiting for the full answer. It feels faster and is the standard pattern for chat and content-generation products. In this tutorial you’ll build a streaming client from scratch, plus a server-side proxy that shields your API credentials from the browser. New to streaming? Streaming Responses in the AI Learning Center explains the ideas first. For production failure handling, see Handling Streaming Failures.

What you’ll build

You’ll build:
  • SSE stream parser that reads tokens as they arrive from the API
  • Token accumulator that assembles the full response with timing and token count
  • Streaming-aware error handler that catches auth and rate-limit errors before reading the stream
  • Terminal renderer that displays tokens in real time with a typing effect
  • Server-side proxy that relays the stream to browser clients without exposing your credentials
You’ll build it in seven steps: † No new implementation — run the demo, test the proxy via API, and explore the browser client.
Server-Sent Events. When you set "stream": true in a completions request, the API switches from a single JSON response to an SSE stream. Each token arrives as a line formatted data: <json>, with blank lines separating events:
The stream ends when a chunk arrives with a non-null finish_reason (typically "stop").Two approaches: direct vs. proxyComponent architecture

Prerequisites

You need:
  • A Gloo AI Studio account with an API key
  • Your API key from the API Keys page. Keep it on your server and out of source control; see API Key Security. Authentication uses your GLOO_API_KEY environment variable — set it in your .env file and it’s used directly as a Bearer token. No token exchange needed.
  • Authentication setup — complete the Authentication Tutorial first

Set up the starter project

You’ll build the streaming client step by step. The starter code provides the scaffolding, with TODO markers guiding each step.
1

Download the starter code

Download the starter project for your language:

Python

Python 3.9+ · requests · Flask

JavaScript

Node.js 18+ · native fetch · Express

TypeScript

TypeScript 5+ · typed SSE chunks

PHP

PHP 8.1+ · cURL write callback

Go

Go 1.20+ · bufio.Scanner · http.Flusher

Java

Java 17+ · HttpClient · Maven
2

Install dependencies and add your key

3

Test your setup

Run the entry point — it should load your credentials and confirm the stubs are in place:
You should see credentials load, followed by NotImplementedError (or equivalent) from the first stub. That means setup is complete.

Step 1: Environment Setup & Auth Verification

Before writing any streaming logic, confirm your GLOO_API_KEY works with the streaming endpoint.

What You’ll Verify

  1. Credentials load correctly from .env
  2. Your API key is set and valid
  3. A request to the completions endpoint returns 200 OK with Content-Type: text/event-stream

Testing Your Setup

Run the Step 1 checkpoint:
Checkpoint: Auth Verification. Confirm this step works before you move on.
Your output should look similar to the following:
If tests fail, check:
  • .env file exists in the language directory (not just .env.example)
  • GLOO_API_KEY is set correctly.
  • You’ve completed the Authentication Tutorial prerequisites

Step 2: Streaming-Aware Error Handling

Implement the stream error handler, which maps HTTP status codes to descriptive exceptions before any stream data is read.

Key Concepts

Two-Phase Error Handling

Streaming errors arrive in two phases. Pre-stream errors are HTTP status codes you can check before reading any bytes, which is what this step’s handler does. Mid-stream errors happen while you read (a network drop, server restart or timeout), so Step 4 catches them around the read loop and keeps any partial text.
This tutorial handles the basics. For production, including which stream errors to retry, retry ceilings, continuing an interrupted answer, and content-filter stops, see Handling Streaming Failures.

Implementation Guide

In your streaming client file, find the error handler method (one case per status code), review the TODO comments, then implement it:
The handler throws an authentication error on 401 (key missing, invalid or revoked), an authorization error on 403 (valid key without permission), a rate limit error on 429 (rejected before the API spent any compute), and a generic error with the response body for any other non-200 status. On 200 it returns so the caller can read the stream.
Checkpoint: Error Handling. Confirm this step works before you move on.
Run the error handling test:
Your output should look similar to the following:
If tests fail, check:
  • Status 200 must not raise an exception
  • The error message for non-200 includes the status code
  • The response body is truncated (first 200 chars) to avoid enormous error messages

Step 3: Streaming Requests & SSE Parsing

Next, open a persistent HTTP connection to the completions API and write the parser that turns raw SSE lines into usable data.

What You’ll Implement

  1. A function to initiate a streaming request
  2. A function to parse individual SSE lines

Making the Streaming Request

Why stream: true Changes Everything

Without stream: true, the API buffers the entire response and returns it as a single JSON object. With stream: true, it switches to SSE mode: the connection stays open and bytes arrive incrementally as the model generates them. That’s why you return the raw response object rather than parsed JSON: the body isn’t fully available yet. The caller reads it line by line in the next steps.

Fail Fast Before Reading

Check the HTTP status code before reading the stream. A 401 response never produces SSE data; it returns a JSON error body. Parsing it as SSE gives confusing parse errors instead of a clear “authentication failed” message.

Implementation Guide

In the same file, find the streaming request method, review the TODO comments, then implement it:
The code sets Authorization and Content-Type headers, and the payload sets stream: true to enable SSE mode and model: "gloo-anthropic-claude-opus-5.5" to pin the model.
PHP note: cURL’s streaming architecture doesn’t allow inspecting the HTTP status before the write callback fires. The status check happens on the first data chunk instead. This is the idiomatic PHP pattern for streaming with cURL.

Parsing SSE Lines

The SSE Wire Format

SSE is a simple text protocol. Each event is one line starting with data: , terminated by a blank line. In practice, the Gloo AI stream looks like:
Skip blank lines (they’re separators, not errors) and any line that doesn’t start with data: , such as event: or : comment lines.

Defensive Parsing

Wrap the JSON parse in a try/catch. Mid-stream network hiccups can produce partial lines, and one malformed chunk shouldn’t crash the stream. Return null for unparseable lines and let the accumulation loop move on.

Implementation Guide

In the same file, find the SSE line parser method, review the TODO comments, then implement it:
The parser returns null for blank and non-data: lines, strips the data: prefix, and detects the [DONE] sentinel before parsing JSON, returning it as a string to signal the end of the stream.
Checkpoint: Streaming Request & SSE Parsing. Confirm this step works before you move on.
Run the validation test for this step:
Your output should look similar to the following:
If tests fail, check:
  • The streaming request function sets stream to true in the payload
  • The SSE line parser strips exactly 6 characters ("data: " has a space after the colon)
  • The [DONE] check happens before the JSON parse

Step 4: Token Extraction & Accumulation

Next, pull the token out of each parsed SSE chunk and write the loop that assembles the complete result.

What You’ll Implement

  1. A function to extract token content from a parsed SSE chunk
  2. A function to collect the full stream into a result object

Extracting Token Content

Why Content Can Be Absent

Not every SSE chunk carries text. The first chunk establishes the role (delta: {"role": "assistant"}), while the final chunk carries the finish reason with an empty or absent delta. Only chunks in the middle carry actual content. An absent content field is normal, so return an empty string rather than throwing. The accumulation loop skips empty strings when counting tokens.

Null-Safe Navigation

Different languages handle missing keys differently. In Python, .get() returns None without raising; in JavaScript/TypeScript, optional chaining (?.) does the same. In Go and Java the struct is fully typed, so missing content simply maps to the zero value. The goal in all languages is the same: never throw when a field is absent.

Implementation Guide

In the same file, find the token content extractor, review the TODO comments, then implement it:
The extractor reads delta.content from the first choice and returns an empty string if choices is empty, the field is absent or null, or the chunk has an unexpected shape.

Accumulating the Full Response

Two Ways to Consume a Stream

You can either accumulate all tokens into a string (what the function in this step does) or print each token immediately as it arrives (what the function in Step 5 does). The choice depends on whether you need the full text before taking action:
  • Accumulate: useful when you need to parse the full response, log it, or return it from an API
  • Print immediately: useful for CLI tools and browser UIs where you want the typing effect

The Line Buffer (JS/TS/PHP)

In Python and Go, the HTTP libraries provide line-at-a-time iteration. In JavaScript, TypeScript, and PHP, you read raw bytes and split on \n yourself. This requires a line buffer: keep any incomplete final chunk in a variable and prepend it to the next read’s output. Without it, tokens near chunk boundaries get split across two parse calls.

Implementation Guide

In the same file, find the accumulation loop method. It brings together everything from the previous steps, with TODO comments marking each stage. Trace through the structure before implementing it.
The loop records the start time before opening the stream, so elapsed time includes connection overhead. It skips null results, stops on a non-null finish_reason or a [DONE] sentinel and returns the text, token count, elapsed milliseconds and finish reason.
Checkpoint: Token Extraction & Accumulation. Confirm this step works before you move on.
Run the validation test for this step:
Your output should look similar to the following:
If tests fail, check:
  • The token content extractor returns "" (not None/null) when content is absent
  • The accumulation loop reads finish_reason from choices[0], not from the top-level chunk
  • The line buffer (buffer = lines.pop()) is in place for JS/TS/PHP

Step 5: Typing-Effect Terminal Renderer

Implement the terminal renderer, which prints each token to stdout as it arrives, without a newline, for a live typing effect. It reuses the streaming request, SSE parsing and token extraction functions but skips the accumulation loop.

Key Concepts

Unbuffered Output

Most languages buffer stdout, holding output until the buffer fills or the program exits. For a typing effect, force each token out immediately:

Direct Stream Consumption vs. Accumulation

Step 4’s function returns once the stream is complete; the renderer prints as it goes, so the user sees output before the model finishes. Pick based on whether the output must be complete before it’s useful.

Implementation Guide

Open the renderer file referenced in the code block. It has a single method to implement. Review the TODO comments, then implement it:
The renderer prints the user’s message as a header, writes and flushes each token as it arrives, and ends with a summary line showing the token count and finish reason.
Checkpoint: Terminal Renderer. Confirm this step works before you move on.
Run the validation test:
Your output should look similar to the following:
With a short prompt like this, tokens arrive so quickly that the response may appear all at once. That’s expected. Longer responses make the typing effect clear, and it’s the pattern your chat UI will use.
If tests fail, check:
  • Each token is written with no trailing newline
  • flush() or equivalent is called after each write
  • The summary line format is [N tokens, finish_reason=X]

Step 6: Server-Side Proxy

Implement the proxy server’s stream handler: the route that receives browser requests, forwards them to Gloo AI with the server-side API key, and pipes the SSE response back.

Key Concepts

Why a Proxy?

Browser JavaScript can’t safely hold an API key: anyone can see client code in DevTools. With a proxy, the browser POSTs to your server, your server adds the API key and calls Gloo AI, and the SSE stream flows back through your server. The proxy can also add rate limiting, logging and multi-tenant auth without touching client code.

SSE Headers That Matter

Three headers tell the browser (and any reverse proxies like nginx) that this is a live stream, not a buffered response:

Language-Specific Flushing

Each language needs an explicit flush to push bytes to the client immediately:

Implementation Guide

Open the proxy server file referenced in the code block; server setup and routing are already in place. Find the stream handler (or route handler) method, review the TODO comments, and implement the relay logic:
The handler sets the SSE headers before writing any data, answers OPTIONS preflight requests so browsers can POST cross-origin, injects stream: true into the forwarded body, and relays and flushes each non-blank SSE line. If the upstream request fails, it writes a structured error SSE frame instead of closing silently.
PHP, Go, and Java use a generic HTTP handler that receives all request methods, so they include an explicit 405 check before the streaming logic. Python, JavaScript, and TypeScript register the route for POST only, so the framework rejects other methods automatically.
Checkpoint: Proxy Server. Confirm this step works before you move on.
Run the proxy server validation test:
Your output should look similar to the following:
If tests fail, check:
  • CORS headers are set before sending the response headers (Java)
  • X-Accel-Buffering: no is present (required to disable nginx buffering)
  • Go: the flusher interface assertion must succeed — this panics if the ResponseWriter doesn’t support flushing
  • PHP: clear any existing output buffers before setting SSE headers

Step 7: Testing Your Complete Implementation

With all six steps implemented, run the demo, test the proxy via API, and try the browser demo.

Run the Demo Script

The entry point runs both examples back-to-back: first it accumulates a full response and prints it, then it streams a second response to the terminal with a typing effect.
Your output should look similar to:

Test the Proxy Server via API

Start the proxy server in one terminal:
Then send a request from another terminal using curl:
The SSE stream arrives line by line:
Each line is a JSON-encoded delta. The final chunk signals the end of the stream with a non-null finish_reason.

Browser Demo

The browser demo is a standalone HTML file separate from the language starter projects — no install step required.

frontend-example/

Download or clone this directory alongside your language starter
The file connects to the proxy over HTTP, so it works with any language’s proxy server. With the proxy running on port 3001, serve the browser client from the frontend-example/ directory with any of these:
Do not open index.html directly via File > Open. When loaded as a file:// URL, the browser reports Origin: null, which the proxy’s CORS policy rejects. You must serve the file over HTTP so the origin is http://localhost:3000.
Then open http://localhost:3000 in your browser, type a question, and click Send — tokens appear one by one as they arrive from the proxy.
Gloo AI Streaming Demo browser page showing a streamed response to "What is my purpose in life"

How the Browser Connects to the Stream

The browser’s built-in EventSource API only supports GET, and the completions API needs a POST body, so the demo page uses fetch() with a ReadableStream, which supports any HTTP method:
The same line buffer, SSE parser and token extractor pattern from the terminal renderer applies.

Markdown Rendering

AI responses often contain markdown. Inserting raw tokens into the DOM breaks mid-stream (**bo appears before ld** closes the bold span). Instead, accumulate tokens and re-parse the full buffer on each token:
marked.parse() runs on every token — slightly redundant but always produces valid HTML. DOMPurify.sanitize() prevents XSS from any HTML in the AI response.
For production, serve the browser client from the same origin as the proxy, or set PROXY_CORS_ORIGIN in your .env to match your frontend domain. For React applications, the Vercel AI SDK useChat hook handles streaming, markdown rendering, and state management out of the box — it’s a higher-level alternative to building this pattern manually.

Complete code

A working reference is in the tutorial repository:

Completed Project

Browse the complete implementation in all six languages — Python, JavaScript, TypeScript, PHP, Go, and Java.

Troubleshooting

These fixes cover the tutorial code. For stream failures in general, see the decision tree in Handling Streaming Failures.
Verify "stream": true is in the request payload. Without it, the API returns a single buffered JSON response, so the connection may appear to hang.
The line buffer is missing or incorrect. In JS/TS/PHP, raw bytes must be accumulated and split on \n before parsing. Make sure buffer = lines.pop() saves the incomplete last fragment.
Your .env file is missing GLOO_API_KEY, or the values are incorrect. Run the Step 1 checkpoint to verify credentials load correctly.
Browsers enforce same-origin policy. Direct calls from browser JavaScript to platform.ai.gloo.com will be blocked. Use the proxy server (Step 6) so API calls happen server-side.
The proxy allows requests only from http://localhost:3000 by default. If your file server uses a different port (e.g. VS Code / Cursor Live Server on port 5500, or python -m http.server 8080), the browser’s Origin header won’t match and the proxy blocks the request. Fix: set PROXY_CORS_ORIGIN in your .env to the exact origin shown in your browser’s address bar, then restart the proxy.
Note that http://localhost:5500 and http://127.0.0.1:5500 are treated as different origins by the browser even though they resolve to the same address. Copy the origin directly from the address bar to avoid a mismatch.
PHP’s output buffering is active. Call ob_end_flush() (or while (ob_get_level() > 0) ob_end_flush()) before the SSE loop to disable buffering.
Your http.ResponseWriter doesn’t implement http.Flusher. This shouldn’t happen with the standard net/http server, but will happen with some test wrappers. Make sure you’re using http.ResponseWriter directly.
Wrap the read loop in try/catch (or check errors in Go). If fullText already has content when the error occurs, return it rather than re-raising — partial responses are usually more useful than nothing.
Follow the decision tree in Handling Streaming Failures. It covers retryable and non-retryable stream errors, content-filter stops, and when to continue rather than retry.
Do not insert raw tokens into innerHTML. Accumulate the full buffer and call marked.parse(buffer) on every token — this ensures the markdown is always valid HTML at each step.

Next steps

New to function calling? See Tools and MCP in the AI Learning Center.

Grounded Completions

Add retrieved context from your content library to improve response accuracy.

Tool Use

Combine streaming with function calling for real-time tool-augmented responses.

Completions API reference

Explore all available parameters including tradition, model_family, and model.