Skip to main content
Organizations bring content into an AI system so people can ask questions in plain language and get answers grounded in that content, instead of digging through a wiki, CMS, or shared drive by hand. That only works if the content is discoverable, accurately attributed, and trustworthy. Gloo’s Data Engine handles the AI-readiness side: fetching content, transcribing audio or video, normalizing it into clean text, chunking it into passages, embedding, and indexing for search and grounded completions. But it can’t recover context you never gave it. That is the job of a content preparation layer: the decisions you make about your source content before Gloo sees it. How you extract it, what shape you normalize it into, what stable IDs you assign, and what metadata you attach determine whether the system stays manageable for years or turns into broken citations and un-refreshable content. This guide describes that preparation layer as one repeatable, source-agnostic pattern — the source system is the least important part of the design.
This is a design guide, not a runnable walkthrough. For a complete, runnable pipeline in six languages, follow the Build an End-to-End RAG Pipeline series — everything downstream of the adapter stage here is exactly what that series implements.

Do You Need a Custom Pipeline?

Before building anything, check whether Studio’s managed ingestion already covers your source. Gloo AI Studio provides an interface with four no-code connectors:
  • Web Scraping — point Gloo at a website or sitemap URL and it crawls and ingests the pages. The right path for public sites you don’t have API access to.
  • Files — drag-and-drop upload across all supported formats.
  • YouTube — ingest a channel or videos; Gloo handles transcription.
  • Feeds — subscribe to RSS/content feeds without writing a poller.
Use Studio managed ingestion when point-and-go is enough: a public website, a YouTube channel, a standard feed, or occasional file drops, especially when the person doing the ingesting isn’t a developer. Build a custom pipeline — the subject of this guide — when you need control the managed connectors don’t give you: stable producer_id schemes for programmatic updates and deletes, metadata attached at scale from your source system, incremental sync against a private CMS/API/database, or content that lives behind authentication.
Audio and video content (podcasts, sermon recordings, YouTube videos) is handled for you via the YouTube connector and audio/video file uploads, which transcribe it automatically so you don’t need a transcript first.The preparation advice in this guide targets text content; if you already have text transcripts, they follow the same pattern as any other document.

The Pattern

Every ingestion integration, regardless of source, decomposes into the same five stages:
Ingestion pipeline: Adapter and Normalize, then Upload, Edit, and VerifyIngestion pipeline: Adapter and Normalize, then Upload, Edit, and Verify
Adapter — source-specific code that extracts content items and their native metadata from wherever they live: paginate a CMS API, query database rows, list bucket objects, read a feed. It’s the only stage that changes with the source, so keep it thin: its job is to emit items in a common shape for the rest of the pipeline. Normalize — convert each item to clean text (Markdown preferred), strip boilerplate, deduplicate within the batch, detect the content type, and assign a stable ID. Most quality problems are won or lost here — see Pre-Ingestion Preparation below. Upload — send each item to POST /ingestion/v2/files with a producer_id, your own stable ID for finding the item later. Re-running the whole pipeline is safe for unchanged content: Gloo recognizes content it already has and returns it in the response’s duplicates list instead of creating a copy. Changed content becomes a new item unless you replace the existing one with the update parameter. Edit Metadata — attach metadata (title, summary, author, tags, canonical URL) via PATCH /engine/v2/item. You can target the item by producer_id, so this stage needs no state from the upload stage. Verify — ingestion is asynchronous: the upload response means queued, not searchable. Poll GET /engine/v2/items/{item_id} until status reaches COMPLETED (you may see intermediate states such as CHUNKING along the way). The key property: everything to the right of the adapter is identical for every source. Write those stages once; a new content source needs only a new adapter.

Pre-Ingestion Preparation

These are the decisions to make up front — each is cheap to decide before launch and expensive to retrofit after.

Detect the content type early

Classify each item — article, transcript, report, Q&A, devotional — as it comes out of the adapter, before normalization. The type drives everything downstream: how you clean the text, whether the item has natural boundaries worth preserving, and what metadata makes it findable. Most source systems already know the type (a CMS post type, a database column, a file extension); carry it through rather than inferring it later.

Normalize to clean text

Gloo accepts a wide range of formats and detects each automatically: Markdown, plain text, rich text, reStructuredText, ORG, HTML, PDF, Word, PowerPoint, EPUB, CSV, TSV, Excel, JSON, XML — plus audio (aac, flac, m4a, mp3, ogg, wav) and video (avi, flv, mov, mp4, webm, wmv), which Gloo transcribes for you. When you control the conversion, prefer Markdown: it preserves heading structure (which helps chunking find natural boundaries) while staying free of layout noise. Whatever you strip, strip it consistently: the goal is that two items about the same topic differ only in their content, not in whatever noise their source format happened to leave behind.

Deduplicate — your half and Gloo’s half

  • You dedup within a run. Source systems often expose the same item more than once — a post that appears in two categories, a document reachable at two paths, a feed that repeats entries. Dedup on the source system’s native ID inside your adapter, before anything is uploaded.
  • Gloo dedups identical content across runs. Uploading a file whose bytes are identical to one already in the publisher’s library doesn’t create a copy — the existing item comes back in the response’s duplicates list. This is what makes “just re-run the pipeline” a safe recovery strategy. Gloo compares exact file bytes, so keep your normalization deterministic (no generated timestamps), or re-runs will create new items.
Gloo matches on content, not on producer_id. When a source item changes, re-uploading it under the same producer_id creates a second item; replace the existing item with the update parameter instead, or delete it and upload again. If you dedup within each batch and use stable producer_ids, you never need to store Gloo’s item_ids yourself: unchanged items are skipped, and you can look up the item_id for a changed item from its producer_id. If several items share a producer_id, the lookup returns the newest one. This works for source items whose content differs: two source items with identical bytes share one Gloo item under the first upload’s producer_id, and the second upload returns that item in duplicates, so they can’t have separate metadata or lifecycles.

Choose a producer_id scheme before the first upload

This is the single highest-leverage decision in the pipeline. The producer_id is your stable handle on an item: it lets you find the item, edit its metadata, and look up Gloo’s item_id without storing it, and it lets a sync process map source-system changes onto Gloo items years later. The recommended scheme is {source-system}-{native-id}:
Do:
  • Derive it from the source system’s own primary key — the one identifier that survives edits, renames, and moves.
  • Prefix with the source system so IDs stay unique when a second source later feeds the same publisher.
  • Treat it as permanent: the same source item must always map to the same producer_id.
Don’t:
  • Derive it from the title, URL slug, or file path if those can change — a renamed item would look like a brand-new one, leaving the old version orphaned in your publisher.
  • Derive it from position or ordering (row number, feed index).
  • Upload changed content under an existing producer_id without the update parameter — Gloo doesn’t replace items by producer_id, so you’d end up with two items sharing one ID. Replace the existing item with update instead.
producer_id applies to single-file uploads only. A multi-file upload that includes it is rejected with a 400 too_many_files error. Pipelines should upload one file per request so every item gets its stable handle.

Pre-chunk vs. let Gloo chunk

Default: let Gloo chunk. After upload, Gloo automatically splits your content into retrieval-sized chunks, embeds them, and indexes them. The defaults need no configuration, and you can change the chunk size and overlap for an upload to POST /ingestion/v2/files. For most content types (articles, reports, transcripts, long-form prose) automatic chunking is the right answer. Split content yourself before upload only when it has hard semantic boundaries that must not be crossed — units that would be wrong to retrieve partially or merged with a neighbor:
  • Q&A content — each question-and-answer pair is one unit; upload each pair as its own item.
  • Devotionals or daily entries — each entry stands alone; a chunk spanning two days’ entries would mislead retrieval.
  • Episodic collections — if each segment has its own title, date, and canonical URL, it deserves its own item and metadata.
When you split, each piece becomes its own upload with its own producer_id (e.g. faq-item-291, devotional-2026-03-14) and its own metadata. This decides item granularity — what counts as one document — not chunk size; within each item, Gloo still chunks automatically. For what automatic chunking does, with per-content-type guidance, see How Chunking Works. For why chunk size and boundaries matter to retrieval in the first place, see Building a Knowledge Base.

Metadata: attach at upload time vs. edit later

The item metadata fields — item_title, item_summary, item_url, item_image, publication_date, author, item_tags, and friends — can be set as soon as the item exists; you don’t need to wait for ingestion to finish. The decision rule:
  • Attach immediately what the source system already knows. Title, author, publication date, canonical URL, and native tags are already in the adapter’s output — PATCH them right after each upload, as part of the pipeline.
  • Edit later what has to be derived. Editorial summaries, curated tags, and classifications that need review can land in a separate edit pass — days or weeks later — targeting items by producer_id.
The canonical URL (item_url) matters most: it lets applications link users back to the original content when a search result or grounded completion cites the item. Capture it in the adapter; it’s often hard to reconstruct later.
Metadata edits are eventually consistent on the read path: a GET immediately after a PATCH may return the old values. When your pipeline verifies edits, poll rather than asserting on the first read. See Part 3 of the RAG pipeline series for the polling pattern.

Source Adapters

An adapter’s contract is small: yield items in a common shape, setting exactly one of text (Markdown you extracted and normalized) or data (raw bytes of a file already in a supported format). Define that shape once:
Every adapter below feeds the same normalize → upload → edit metadata → verify code. The sketches are illustrative, not runnable: they show each adapter’s shape, leaving out pagination, auth, and error handling.
The pattern is language-neutral. The sketches use Python and TypeScript to match the RAG pipeline series; port the same structure to Go, Java, PHP, or anything else.

CMS (WordPress, Contentful, Sanity, Drupal)

Page through the CMS’s content API, convert each entry’s HTML body to Markdown, and use the CMS’s own post ID as the native ID.

API-based tools (Notion, Confluence, custom JSON)

Traverse the tool’s page or space hierarchy, render each page’s block tree to Markdown, and use the page UUID as the native ID.

Database (knowledge-base tables, blog posts, CMS rows)

Query the rows, render each record as a small Markdown document (heading plus labeled fields), and use the primary key as the native ID.

File library (S3, GCS, SharePoint)

List the objects, filter to supported formats, and use the object key as the native ID. Files can pass through as-is — Gloo detects supported formats automatically — so this adapter sets data (raw bytes) rather than text. Scanned or layout-heavy PDFs still benefit from the extraction pass described under normalization; if you do extract text yourself, set text instead.

Feeds and batch exports (RSS, weekly CMS export)

Feeds are the incremental case: track the last GUID or timestamp you’ve seen and yield only new or changed entries. This same shape handles a weekly CMS export — diff the export against the last run’s state.
Because Gloo skips content it already has, incremental sync degrades gracefully: if your sync state is lost, a full re-run re-uploads everything, and unchanged entries come back in duplicates instead of being copied. A changed entry needs its existing item replaced: look up the item ID for its producer_id, then upload with update set to that ID and the same producer_id (see Replace an Item’s Content). Uploading it without update creates a second item. When source items are removed, delete the corresponding Gloo items with DELETE /engine/v2/items, scoped to exactly those item IDs — see Part 2 of the RAG pipeline series for the scoped-delete pattern.
The POST /engine/v2/publisher/{publisher_id}/items/by-producer lookup (producer ID → item ID) is cached for up to about five minutes. Uploads, replacements and deletes clear the cache, but a lookup made right after one can still return the previous mapping for a short time. Treat the upload response — or a GET on the item — as the source of truth in sync code.

Decision Matrix

What Gloo Does for You

Don’t build these yourself; Gloo handles them after upload:
  • Format detection and parsing — Markdown, text, rich text, reStructuredText, ORG, HTML, PDF, Word, PowerPoint, EPUB, CSV, TSV, Excel, JSON, and XML files are detected and processed without any format flags.
  • Transcription — audio (aac, flac, m4a, mp3, ogg, wav) and video (avi, flv, mov, mp4, webm, wmv) files are transcribed automatically, whether uploaded as files or ingested via the Studio YouTube connector.
  • Chunking — content is split into retrieval-sized chunks automatically; the defaults can be changed for an upload.
  • Embedding — chunks are embedded for semantic search; for uploaded content, you never manage models or vectors. To build your own vector store, see the Embeddings API.
  • Vector storage and retrieval indexing — no vector database to run, no index to maintain.
  • Cross-run deduplication — re-uploading content that’s already in the publisher’s library is skipped and returned in the response’s duplicates list, so re-running a pipeline doesn’t create copies. To replace an item’s content, upload with the update parameter.
Your pipeline’s job ends at clean text, stable IDs, good metadata, and verified indexing. Everything between upload and search-readiness is Gloo’s job, which is why the verify stage polls a status instead of orchestrating anything.

Next steps

For API-level reference on the upload and metadata endpoints, see Upload Content and Edit Content.

Build an End-to-End RAG Pipeline, Part 1

The runnable version of upload → edit metadata → verify, in six languages.

Part 2: Manage the Content Lifecycle

Updates, bulk edits with filters, and safely scoped deletes for keeping Gloo in sync with your source.

Part 3: Verification, Errors & Resilience

Error handling, retries with backoff, and health-check patterns for production pipelines.

How Chunking Works

What happens to your content after upload: parsing, chunking, embedding, and indexing, plus per-content-type guidance.