Part 1: Set Up the Pipeline

This is Part 1 of the Build an End-to-End RAG Pipeline series: the ingestion side of RAG. The publisher and content you set up here are the same ones you’ll manage in Part 2 and harden in Part 3. The retrieval side (semantic search and grounded answers) is covered in the Building Custom Search tutorial and the Grounded Completions guide. In this part you’ll create a publisher, upload content with descriptive metadata, and verify that it’s fully indexed and ready for retrieval.

The Full Journey

A production RAG application connects several Gloo AI features into one path. The diagram below shows it end to end: you set up your content and keep adding to it, Gloo answers questions against it, and you keep it current over time. Each stage maps to a step in this series or a focused tutorial.

The end-to-end Gloo journey in three phases. Set up your content: 1 create a publisher in Studio, 2 ingest content into the Data Engine, 3 attach metadata. Answer questions (at request time): 4 semantic search finds matching snippets, 5 grounded completions answer from your content, 6 your app displays the sources. Keep it current (ongoing): 7 update or delete items, 8 monitor status and debug errors. Purple stages are handled by you and your app; green stages are handled by Gloo.

This series covers stages 1-3 and 7-8 in depth. Stages 4-6, the retrieval side, are covered in the tutorial and guide linked above.

Pipeline at a Glance

Publisher setup (Studio)

Create the publisher that owns your content: covered below.

Ingest content with metadata

Upload files and enrich them: covered below, with a deep dive in Upload Files to Data Engine.

Verify indexing

Poll item status until your content is searchable: covered below.

Semantic search

Query your content. Deep dive: Building Custom Search.

Grounded completions with sources

Answer questions from your content with citations. Deep dive: Grounded Completions, which sets rag_publisher to your publisher name.

Content lifecycle

Update, bulk-edit, and delete content: Part 2.

Verification, errors & resilience

Error handling and retry patterns: Part 3.

Prerequisites

Before starting, you need:

A Gloo AI Studio account
API key - Set the GLOO_API_KEY environment variable to your key from the API Keys page. Keep it on your server and out of source control; see API Key Security

All API calls in this series authenticate with an API key. The snippets below read GLOO_API_KEY and set API_KEY so the same variable works in Part 2 and Part 3.

This series requires Data Engine access. Data Engine is currently at capacity for new organizations: existing Pro and Enterprise organizations already have access. New organizations can request access and the team will follow up, or review Billing & Plans. Without access you cannot complete the steps below.

Step 1: Create Your Publisher

Content in the Data Engine belongs to a publisher. Create one in Gloo AI Studio:

In Gloo AI Studio, click your user account in the bottom-left corner and select Manage Organizations
Select the organization you want to add the publisher to, then click View Publishers
Click Create Publisher and give the new publisher a name
Note two values for this publisher. Each parameter in this series takes a different one:
- Publisher ID (a UUID): the ingestion endpoints take it as the publisher_id field
- Publisher name: the grounded endpoints take it as the rag_publisher parameter

See Manage Publishers for the full Studio walkthrough.

Keep both values handy. The Grounded Completions guide documents rag_publisher as your publisher name, which you can read from the same Publishers page.

Use one dedicated publisher for this series. Parts 2 and 3 operate on the content you upload here; a dedicated publisher keeps them separate from your production content.

Step 2: Upload Content with a Producer ID

Upload a file to POST /ingestion/v2/files. The producer_id query parameter attaches your own stable identifier to the item. This is what makes the pipeline manageable later: re-running the upload detects a duplicate instead of creating a copy, and in Part 2 you’ll update, bulk-edit, and delete these same items. This series uses a short Markdown article as its sample content: grab building-stronger-communities.md from the cookbook repository and save it next to your script. (Any Markdown, text, PDF, or Word file of your own works too. Just adjust the filename.) Then upload it:

Content not on local disk? See Preparing Content for Ingestion for the source-agnostic pattern: pulling from a CMS, API, database, file library, or feed, and how to structure content before it reaches this upload step.

import os

import requests

API_KEY = os.getenv("GLOO_API_KEY")
token = API_KEY  # Used by parts 2 and 3
PUBLISHER_ID = "your_publisher_id"
PRODUCER_ID = "rag-pipeline-part1-building-stronger-communities"

# Upload the file with a stable producer ID
with open("building-stronger-communities.md", "rb") as f:
    response = requests.post(
        "https://platform.ai.gloo.com/ingestion/v2/files",
        headers={"Authorization": f"Bearer {token}"},
        params={"producer_id": PRODUCER_ID},
        files={"files": ("building-stronger-communities.md", f)},
        data={"publisher_id": PUBLISHER_ID},
    )
result = response.json()

# A fresh upload returns the new item ID in "ingesting";
# re-uploading the same content returns it in "duplicates".
item_id = (result["ingesting"] or result["duplicates"])[0]
print(f"Item ID: {item_id}")

import { readFile } from 'node:fs/promises';

const API_KEY = process.env.GLOO_API_KEY;
const token = API_KEY; // Used by parts 2 and 3
const PUBLISHER_ID = 'your_publisher_id';
const PRODUCER_ID = 'rag-pipeline-part1-building-stronger-communities';

// Upload the file with a stable producer ID
const form = new FormData();
form.append('publisher_id', PUBLISHER_ID);
form.append(
  'files',
  new Blob([await readFile('building-stronger-communities.md')]),
  'building-stronger-communities.md',
);

const response = await fetch(
  `https://platform.ai.gloo.com/ingestion/v2/files?producer_id=${encodeURIComponent(PRODUCER_ID)}`,
  { method: 'POST', headers: { Authorization: `Bearer ${token}` }, body: form },
);
const result = await response.json();

// A fresh upload returns the new item ID in "ingesting";
// re-uploading the same content returns it in "duplicates".
const itemId = (
  result.ingesting.length ? result.ingesting : result.duplicates
)[0];
console.log(`Item ID: ${itemId}`);

import { readFileSync } from 'node:fs';

const API_KEY = process.env.GLOO_API_KEY;
const token = API_KEY; // Used by parts 2 and 3
const PUBLISHER_ID = 'your_publisher_id';
const PRODUCER_ID = 'rag-pipeline-part1-building-stronger-communities';

interface UploadResponse {
  ingesting: string[];
  duplicates: string[];
}

(async () => {
  // Upload the file with a stable producer ID
  const form = new FormData();
  form.append('publisher_id', PUBLISHER_ID);
  form.append(
    'files',
    new Blob([readFileSync('building-stronger-communities.md')]),
    'building-stronger-communities.md',
  );

  const response = await fetch(
    `https://platform.ai.gloo.com/ingestion/v2/files?producer_id=${encodeURIComponent(PRODUCER_ID)}`,
    {
      method: 'POST',
      headers: { Authorization: `Bearer ${token}` },
      body: form,
    },
  );
  const result = (await response.json()) as UploadResponse;

  // A fresh upload returns the new item ID in "ingesting";
  // re-uploading the same content returns it in "duplicates".
  const itemId = (
    result.ingesting.length ? result.ingesting : result.duplicates
  )[0];
  console.log(`Item ID: ${itemId}`);
})();

<?php

$API_KEY = getenv('GLOO_API_KEY');
$token = $API_KEY; // Used by parts 2 and 3
$publisherId = 'your_publisher_id';
$producerId = 'rag-pipeline-part1-building-stronger-communities';

// Upload the file with a stable producer ID
$url = 'https://platform.ai.gloo.com/ingestion/v2/files?producer_id=' . urlencode($producerId);
$ch = curl_init($url);
curl_setopt_array($ch, [
    CURLOPT_RETURNTRANSFER => true,
    CURLOPT_POST => true,
    CURLOPT_HTTPHEADER => ["Authorization: Bearer $token"],
    CURLOPT_POSTFIELDS => [
        'publisher_id' => $publisherId,
        'files' => new CURLFile('building-stronger-communities.md', 'text/markdown'),
    ],
]);
$result = json_decode(curl_exec($ch), true);
curl_close($ch);

// A fresh upload returns the new item ID in "ingesting";
// re-uploading the same content returns it in "duplicates".
$itemId = ($result['ingesting'] ?: $result['duplicates'])[0];
echo "Item ID: $itemId\n";

package main

import (
	"bytes"
	"encoding/json"
	"fmt"
	"mime/multipart"
	"net/http"
	"net/url"
	"os"
)

const (
	publisherID = "your_publisher_id"
	producerID  = "rag-pipeline-part1-building-stronger-communities"
)

func main() {
	apiKey := os.Getenv("GLOO_API_KEY")
	token := apiKey // Used by parts 2 and 3

	// Upload the file with a stable producer ID
	fileBytes, _ := os.ReadFile("building-stronger-communities.md")
	var buf bytes.Buffer
	writer := multipart.NewWriter(&buf)
	writer.WriteField("publisher_id", publisherID)
	part, _ := writer.CreateFormFile("files", "building-stronger-communities.md")
	part.Write(fileBytes)
	writer.Close()

	uploadURL := "https://platform.ai.gloo.com/ingestion/v2/files?producer_id=" +
		url.QueryEscape(producerID)
	req, _ := http.NewRequest("POST", uploadURL, &buf)
	req.Header.Set("Authorization", "Bearer "+token)
	req.Header.Set("Content-Type", writer.FormDataContentType())
	resp, err := http.DefaultClient.Do(req)
	if err != nil {
		panic(err)
	}
	defer resp.Body.Close()

	// A fresh upload returns the new item ID in "ingesting";
	// re-uploading the same content returns it in "duplicates".
	var result struct {
		Ingesting  []string `json:"ingesting"`
		Duplicates []string `json:"duplicates"`
	}
	json.NewDecoder(resp.Body).Decode(&result)
	itemID := append(result.Ingesting, result.Duplicates...)[0]
	fmt.Println("Item ID:", itemID)
}

import com.google.gson.Gson;
import com.google.gson.JsonObject;
import java.net.URI;
import java.net.URLEncoder;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.Base64;

public class UploadContent {
    static final String PUBLISHER_ID = "your_publisher_id";
    static final String PRODUCER_ID = "rag-pipeline-part1-building-stronger-communities";

    public static void main(String[] args) throws Exception {
        HttpClient http = HttpClient.newHttpClient();
        Gson gson = new Gson();

        String token = System.getenv("GLOO_API_KEY"); // Used by parts 2 and 3

        // Upload the file with a stable producer ID (manual multipart body)
        String boundary = "----GlooBoundary";
        var body = new java.io.ByteArrayOutputStream();
        body.write(("--" + boundary + "\r\n"
            + "Content-Disposition: form-data; name=\"publisher_id\"\r\n\r\n"
            + PUBLISHER_ID + "\r\n").getBytes(StandardCharsets.UTF_8));
        body.write(("--" + boundary + "\r\n"
            + "Content-Disposition: form-data; name=\"files\"; "
            + "filename=\"building-stronger-communities.md\"\r\n"
            + "Content-Type: text/markdown\r\n\r\n").getBytes(StandardCharsets.UTF_8));
        body.write(Files.readAllBytes(Path.of("building-stronger-communities.md")));
        body.write(("\r\n--" + boundary + "--\r\n").getBytes(StandardCharsets.UTF_8));

        HttpRequest uploadRequest = HttpRequest.newBuilder()
            .uri(URI.create("https://platform.ai.gloo.com/ingestion/v2/files?producer_id="
                + URLEncoder.encode(PRODUCER_ID, StandardCharsets.UTF_8)))
            .header("Authorization", "Bearer " + token)
            .header("Content-Type", "multipart/form-data; boundary=" + boundary)
            .POST(HttpRequest.BodyPublishers.ofByteArray(body.toByteArray()))
            .build();
        JsonObject result = gson
            .fromJson(http.send(uploadRequest, HttpResponse.BodyHandlers.ofString()).body(),
                JsonObject.class);

        // A fresh upload returns the new item ID in "ingesting";
        // re-uploading the same content returns it in "duplicates".
        var ingesting = result.getAsJsonArray("ingesting");
        String itemId = (ingesting.size() > 0
            ? ingesting
            : result.getAsJsonArray("duplicates")).get(0).getAsString();
        System.out.println("Item ID: " + itemId);
    }
}

What You’ll See

Item ID: 822308cc-72fc-478d-a2eb-fbdf01a6a15d

Keep this item ID: the next two steps use it, and Parts 2 and 3 operate on this same item.

For multi-file uploads, batch processing, and supported file types, see the Upload Files to Data Engine deep dive.

Step 3: Enrich with Metadata

Attach descriptive metadata with PATCH /engine/v2/item. Titles and authors appear in search results and source citations, and tags let you organize and bulk-manage content in Part 2.

metadata = {
    "publisher_id": PUBLISHER_ID,
    "item_id": item_id,
    "item_title": "Building Stronger Communities Through Service",
    "item_summary": "Practical guidance for starting and sustaining community service efforts.",
    "author": ["Gloo AI Docs Team"],
    "item_tags": ["community", "service", "rag-pipeline-series"],
}
response = requests.patch(
    "https://platform.ai.gloo.com/engine/v2/item",
    headers={"Authorization": f"Bearer {token}", "Content-Type": "application/json"},
    json=metadata,
)
response.raise_for_status()
print("Metadata set")

const metadata = {
  publisher_id: PUBLISHER_ID,
  item_id: itemId,
  item_title: 'Building Stronger Communities Through Service',
  item_summary:
    'Practical guidance for starting and sustaining community service efforts.',
  author: ['Gloo AI Docs Team'],
  item_tags: ['community', 'service', 'rag-pipeline-series'],
};
const metadataResponse = await fetch(
  'https://platform.ai.gloo.com/engine/v2/item',
  {
    method: 'PATCH',
    headers: {
      Authorization: `Bearer ${token}`,
      'Content-Type': 'application/json',
    },
    body: JSON.stringify(metadata),
  },
);
if (!metadataResponse.ok) throw new Error(`HTTP ${metadataResponse.status}`);
console.log('Metadata set');

const metadata = {
  publisher_id: PUBLISHER_ID,
  item_id: itemId,
  item_title: 'Building Stronger Communities Through Service',
  item_summary:
    'Practical guidance for starting and sustaining community service efforts.',
  author: ['Gloo AI Docs Team'],
  item_tags: ['community', 'service', 'rag-pipeline-series'],
};
const metadataResponse = await fetch(
  'https://platform.ai.gloo.com/engine/v2/item',
  {
    method: 'PATCH',
    headers: {
      Authorization: `Bearer ${token}`,
      'Content-Type': 'application/json',
    },
    body: JSON.stringify(metadata),
  },
);
if (!metadataResponse.ok) throw new Error(`HTTP ${metadataResponse.status}`);
console.log('Metadata set');

$metadata = [
    'publisher_id' => $publisherId,
    'item_id' => $itemId,
    'item_title' => 'Building Stronger Communities Through Service',
    'item_summary' => 'Practical guidance for starting and sustaining community service efforts.',
    'author' => ['Gloo AI Docs Team'],
    'item_tags' => ['community', 'service', 'rag-pipeline-series'],
];
$ch = curl_init('https://platform.ai.gloo.com/engine/v2/item');
curl_setopt_array($ch, [
    CURLOPT_RETURNTRANSFER => true,
    CURLOPT_CUSTOMREQUEST => 'PATCH',
    CURLOPT_HTTPHEADER => [
        "Authorization: Bearer $token",
        'Content-Type: application/json',
    ],
    CURLOPT_POSTFIELDS => json_encode($metadata),
]);
curl_exec($ch);
curl_close($ch);
echo "Metadata set\n";

metadata := map[string]any{
	"publisher_id": publisherID,
	"item_id":      itemID,
	"item_title":   "Building Stronger Communities Through Service",
	"item_summary": "Practical guidance for starting and sustaining community service efforts.",
	"author":       []string{"Gloo AI Docs Team"},
	"item_tags":    []string{"community", "service", "rag-pipeline-series"},
}
payload, _ := json.Marshal(metadata)
req, _ = http.NewRequest("PATCH", "https://platform.ai.gloo.com/engine/v2/item",
	bytes.NewReader(payload))
req.Header.Set("Authorization", "Bearer "+token)
req.Header.Set("Content-Type", "application/json")
resp, err = http.DefaultClient.Do(req)
if err != nil {
	panic(err)
}
resp.Body.Close()
fmt.Println("Metadata set")

String metadataJson = """
    {
      "publisher_id": "%s",
      "item_id": "%s",
      "item_title": "Building Stronger Communities Through Service",
      "item_summary": "Practical guidance for starting and sustaining community service efforts.",
      "author": ["Gloo AI Docs Team"],
      "item_tags": ["community", "service", "rag-pipeline-series"]
    }
    """.formatted(PUBLISHER_ID, itemId);
HttpRequest metadataRequest = HttpRequest.newBuilder()
    .uri(URI.create("https://platform.ai.gloo.com/engine/v2/item"))
    .header("Authorization", "Bearer " + token)
    .header("Content-Type", "application/json")
    .method("PATCH", HttpRequest.BodyPublishers.ofString(metadataJson))
    .build();
http.send(metadataRequest, HttpResponse.BodyHandlers.ofString());
System.out.println("Metadata set");

Metadata can be set as soon as the item ID exists. You don’t need to wait for ingestion to finish. You can also target a single item by producer_id instead of item_id.

Step 4: Verify Indexing

Ingestion is asynchronous: the upload response means your file is queued, not searchable. Poll GET /engine/v2/items/{item_id} until status reaches COMPLETED, typically a few minutes for a small file. While processing you’ll see intermediate states such as CHUNKING (see How Chunking Works for what happens during this stage).

import time

POLL_INTERVAL_SECONDS = 15
POLL_TIMEOUT_SECONDS = 600

deadline = time.time() + POLL_TIMEOUT_SECONDS
while time.time() < deadline:
    item = requests.get(
        f"https://platform.ai.gloo.com/engine/v2/items/{item_id}",
        headers={"Authorization": f"Bearer {token}"},
    ).json()
    status = item.get("status", "unknown")
    print(f"Status: {status}")

    if status.upper() == "COMPLETED":
        print(f"Indexed: {item['item_title']} (tags: {', '.join(item['item_tags'])})")
        break
    if status.upper() in ("FAILED", "ERROR"):
        raise RuntimeError(f"Ingestion failed with status: {status}")

    time.sleep(POLL_INTERVAL_SECONDS)
else:
    raise TimeoutError(f"Not indexed within {POLL_TIMEOUT_SECONDS}s")

const POLL_INTERVAL_MS = 15_000;
const POLL_TIMEOUT_MS = 600_000;
const sleep = (ms) => new Promise((resolve) => setTimeout(resolve, ms));

const deadline = Date.now() + POLL_TIMEOUT_MS;
let indexed = false;
while (Date.now() < deadline) {
  const itemResponse = await fetch(
    `https://platform.ai.gloo.com/engine/v2/items/${itemId}`,
    { headers: { Authorization: `Bearer ${token}` } },
  );
  const item = await itemResponse.json();
  const status = item.status ?? 'unknown';
  console.log(`Status: ${status}`);

  if (status.toUpperCase() === 'COMPLETED') {
    console.log(
      `Indexed: ${item.item_title} (tags: ${item.item_tags.join(', ')})`,
    );
    indexed = true;
    break;
  }
  if (['FAILED', 'ERROR'].includes(status.toUpperCase())) {
    throw new Error(`Ingestion failed with status: ${status}`);
  }

  await sleep(POLL_INTERVAL_MS);
}
if (!indexed) throw new Error(`Not indexed within ${POLL_TIMEOUT_MS / 1000}s`);

const POLL_INTERVAL_MS = 15_000;
const POLL_TIMEOUT_MS = 600_000;
const sleep = (ms: number): Promise<void> =>
  new Promise((resolve) => setTimeout(resolve, ms));

interface ItemMetadata {
  status: string;
  item_title: string | null;
  item_tags: string[];
}

const deadline = Date.now() + POLL_TIMEOUT_MS;
let indexed = false;
while (Date.now() < deadline) {
  const itemResponse = await fetch(
    `https://platform.ai.gloo.com/engine/v2/items/${itemId}`,
    { headers: { Authorization: `Bearer ${token}` } },
  );
  const item = (await itemResponse.json()) as ItemMetadata;
  const status = item.status ?? 'unknown';
  console.log(`Status: ${status}`);

  if (status.toUpperCase() === 'COMPLETED') {
    console.log(
      `Indexed: ${item.item_title} (tags: ${item.item_tags.join(', ')})`,
    );
    indexed = true;
    break;
  }
  if (['FAILED', 'ERROR'].includes(status.toUpperCase())) {
    throw new Error(`Ingestion failed with status: ${status}`);
  }

  await sleep(POLL_INTERVAL_MS);
}
if (!indexed) throw new Error(`Not indexed within ${POLL_TIMEOUT_MS / 1000}s`);

const POLL_INTERVAL_SECONDS = 15;
const POLL_TIMEOUT_SECONDS = 600;

$deadline = time() + POLL_TIMEOUT_SECONDS;
$indexed = false;
while (time() < $deadline) {
    $ch = curl_init("https://platform.ai.gloo.com/engine/v2/items/$itemId");
    curl_setopt_array($ch, [
        CURLOPT_RETURNTRANSFER => true,
        CURLOPT_HTTPHEADER => ["Authorization: Bearer $token"],
    ]);
    $item = json_decode(curl_exec($ch), true);
    curl_close($ch);

    $status = $item['status'] ?? 'unknown';
    echo "Status: $status\n";

    if (strtoupper($status) === 'COMPLETED') {
        echo "Indexed: {$item['item_title']} (tags: " . implode(', ', $item['item_tags']) . ")\n";
        $indexed = true;
        break;
    }
    if (in_array(strtoupper($status), ['FAILED', 'ERROR'], true)) {
        throw new RuntimeException("Ingestion failed with status: $status");
    }

    sleep(POLL_INTERVAL_SECONDS);
}
if (!$indexed) {
    throw new RuntimeException('Not indexed within ' . POLL_TIMEOUT_SECONDS . 's');
}

const (
	pollInterval = 15 * time.Second
	pollTimeout  = 600 * time.Second
)

deadline := time.Now().Add(pollTimeout)
indexed := false
for time.Now().Before(deadline) {
	req, _ := http.NewRequest("GET",
		"https://platform.ai.gloo.com/engine/v2/items/"+itemID, nil)
	req.Header.Set("Authorization", "Bearer "+token)
	resp, err := http.DefaultClient.Do(req)
	if err != nil {
		panic(err)
	}
	var item struct {
		Status    string   `json:"status"`
		ItemTitle string   `json:"item_title"`
		ItemTags  []string `json:"item_tags"`
	}
	json.NewDecoder(resp.Body).Decode(&item)
	resp.Body.Close()
	fmt.Println("Status:", item.Status)

	if strings.EqualFold(item.Status, "COMPLETED") {
		fmt.Printf("Indexed: %s (tags: %s)\n", item.ItemTitle, strings.Join(item.ItemTags, ", "))
		indexed = true
		break
	}
	if strings.EqualFold(item.Status, "FAILED") || strings.EqualFold(item.Status, "ERROR") {
		panic("Ingestion failed with status: " + item.Status)
	}

	time.Sleep(pollInterval)
}
if !indexed {
	panic("Not indexed within timeout")
}

Duration pollInterval = Duration.ofSeconds(15);
Duration pollTimeout = Duration.ofSeconds(600);

Instant deadline = Instant.now().plus(pollTimeout);
boolean indexed = false;
while (Instant.now().isBefore(deadline)) {
    HttpRequest statusRequest = HttpRequest.newBuilder()
        .uri(URI.create("https://platform.ai.gloo.com/engine/v2/items/" + itemId))
        .header("Authorization", "Bearer " + token)
        .GET()
        .build();
    JsonObject item = gson
        .fromJson(http.send(statusRequest, HttpResponse.BodyHandlers.ofString()).body(),
            JsonObject.class);
    String status = item.has("status") ? item.get("status").getAsString() : "unknown";
    System.out.println("Status: " + status);

    if (status.equalsIgnoreCase("COMPLETED")) {
        System.out.println("Indexed: " + item.get("item_title").getAsString());
        indexed = true;
        break;
    }
    if (status.equalsIgnoreCase("FAILED") || status.equalsIgnoreCase("ERROR")) {
        throw new IllegalStateException("Ingestion failed with status: " + status);
    }

    Thread.sleep(pollInterval.toMillis());
}
if (!indexed) {
    throw new IllegalStateException("Not indexed within timeout");
}

What You’ll See

For a fresh upload (about 6 minutes for the sample file):

Status: QUEUED
Status: CHUNKING
...
Status: COMPLETED
Indexed: Building Stronger Communities Through Service (tags: community, service, rag-pipeline-series)

If you re-run against already-indexed content, the first poll returns COMPLETED immediately. The response also returns the title, author, and tags from Step 3: proof they’re attached to the indexed item.

Run the Complete Example

The cookbook contains the full pipeline (upload, metadata, and polling with error handling) as one runnable program in all six languages. From the cookbook repository, install dependencies, copy .env.example to .env and add your credentials, then run it:

cd rag-pipeline-part-1/python
python3 -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate
pip install -r requirements.txt
cp .env.example .env       # then add your API key and Publisher ID
python main.py

cd rag-pipeline-part-1/javascript
npm install
cp .env.example .env       # then add your API key and Publisher ID
npm start

cd rag-pipeline-part-1/typescript
npm install
cp .env.example .env       # then add your API key and Publisher ID
npm start

cd rag-pipeline-part-1/php
composer install
cp .env.example .env       # then add your API key and Publisher ID
php index.php

cd rag-pipeline-part-1/go
go mod tidy
cp .env.example .env       # then add your API key and Publisher ID
go run main.go

cd rag-pipeline-part-1/java
mvn compile
cp .env.example .env       # then add your API key and Publisher ID
mvn -q exec:java

You’ll see:

Step 1: Uploading sample content...
  Queued for ingestion: 822308cc-72fc-478d-a2eb-fbdf01a6a15d

Step 2: Setting item metadata...
  Metadata set: title, summary, author, 3 tags

Step 3: Verifying indexing (polling)...
  Status: QUEUED
  Status: COMPLETED

Pipeline content is indexed and ready.
  Item ID:  822308cc-72fc-478d-a2eb-fbdf01a6a15d
  Title:    Building Stronger Communities Through Service
  Author:   Gloo AI Docs Team
  Tags:     community, service, rag-pipeline-series
  Status:   COMPLETED

Working Code Sample

View Complete Code

Clone or browse the complete working examples for all 6 languages (JavaScript, TypeScript, Python, PHP, Go, Java) with setup instructions and the sample content file.

The code snippets above are simplified and self-contained, designed for readability and easy copy-paste. The cookbook examples add duplicate handling and structured error handling. Both implement the same APIs and patterns.

Troubleshooting

Error: 401 Unauthorized

Your API key is missing or invalid. Verify the GLOO_API_KEY environment variable is set correctly. See the Authentication Tutorial.

Error: 400 publisher_not_found

The publisher_id doesn’t exist or isn’t accessible with your credentials. Copy the Publisher ID (a UUID) from Studio > Data Engine > Publishers.

Error: 400 too_many_files

producer_id applies to a single file. When uploading multiple files in one request, omit it.

Polling times out

Larger files take longer to process. Increase the timeout, or check Studio > Ingestion Analytics to see whether the publisher is still processing.

Next Steps

Your pipeline now has indexed, metadata-rich content. Wire it into retrieval:

Building Custom Search: query this content semantically with the Search API
Grounded Completions: answer questions from this content with source citations (set rag_publisher to your publisher name, not the Publisher ID)
Part 2: Content Lifecycle: update, bulk-edit, and delete the items you created here
Part 3: Verification, Error Handling & Resilience: production-grade retry and verification patterns

To learn the concepts behind this part, read Building a Knowledge Base in the Learning Center: knowledge sources, chunking and metadata.

Get Started

Responses & Completions

Production

Content

Search & Recommendations

Embeddings

Legacy

Part 1: Set Up the Pipeline

The Full Journey

Pipeline at a Glance

Prerequisites

Step 1: Create Your Publisher

Step 2: Upload Content with a Producer ID

What You’ll See

Step 3: Enrich with Metadata

Step 4: Verify Indexing

What You’ll See

Run the Complete Example

Working Code Sample

View Complete Code

Troubleshooting

Error: 401 Unauthorized

Error: 400 publisher_not_found

Error: 400 too_many_files

Polling times out

Next Steps

​The Full Journey

​Pipeline at a Glance

​Prerequisites

​Step 1: Create Your Publisher

​Step 2: Upload Content with a Producer ID

​What You’ll See

​Step 3: Enrich with Metadata

​Step 4: Verify Indexing

​What You’ll See

​Run the Complete Example

​Working Code Sample

View Complete Code

​Troubleshooting

​Error: 401 Unauthorized

​Error: 400 publisher_not_found

​Error: 400 too_many_files

​Polling times out

​Next Steps

The Full Journey

Pipeline at a Glance

Prerequisites

Step 1: Create Your Publisher

Step 2: Upload Content with a Producer ID

What You’ll See

Step 3: Enrich with Metadata

Step 4: Verify Indexing

What You’ll See

Run the Complete Example

Working Code Sample

Troubleshooting

Error: 401 Unauthorized

Error: 400 publisher_not_found

Error: 400 too_many_files

Polling times out

Next Steps