Skip to main content
This API is deprecated. Follow the Upload Files to Data Engine tutorial instead.
This recipe builds an automated content ingestion pipeline on the Realtime API. It watches for new files, extracts metadata, and uploads content to the Gloo AI platform with error handling and monitoring, so the content becomes available for search and AI interaction.

Prerequisites

You need:
The Realtime Ingestion API authenticates with your API key; see the Authentication Tutorial.

V1 Realtime Upload [Deprecated]

Key Features

  • Real-time Processing: Content is processed upon upload
  • Rich Metadata: Support for comprehensive content categorization
  • Flexible Content Types: Articles, documents, media, and structured content
  • Automatic Indexing: Content becomes searchable instantly

Step 1: Understanding the Realtime API

The Realtime Ingestion API processes uploaded content and makes it available for search and AI interaction. The primary endpoint is: POST /ingestion/v1/real_time_upload

Key Features

  • Real-time Processing: Content is processed upon upload
  • Rich Metadata: Support for comprehensive content categorization
  • Flexible Content Types: Articles, documents, media, and structured content
  • Automatic Indexing: Content becomes searchable instantly upon file process completion.

Required Fields

Optional Metadata Fields

  • Content Details: author, publication_date, item_title, item_subtitle, item_summary
  • Categorization: type, pub_type, denomination, item_tags
  • Media: item_image, item_url, hosted_url
  • Access Control: drm, evergreen
  • Hierarchical Structure: h2_title, h3_title for document sections

Step 2: Basic Content Upload

This example shows the core API call with authentication and error handling.

API Request Structure

The kallm DRM value shown in the examples below is deprecated and no longer supported. Use aspen only.

Expected Response


Step 3: Verifying Content Upload

Check upload progress in Gloo AI Studio:
  1. Log in to Gloo AI Studio.
  2. Navigate to the Data Engine section from the main Studio sidebar.
  3. Click on Your Data.
Your Data in Studio

Step 4: Setting Up File Monitoring

For automated ingestion, monitor directories for new files and process them as they arrive.

File Watching Strategy

  1. Monitor Target Directory: Watch for new files or changes
  2. Extract Metadata: Parse filename and content for metadata
  3. Validate Content: Ensure required fields are present
  4. Upload with Retry: Handle failures gracefully
  5. Log Results: Track successful and failed uploads

Step 5: Batch Processing Pipeline

For many files or large volumes of content, batch processing gives better performance and resource management.

Batch Processing Benefits

  • Rate Limiting: Control API call frequency to stay within your rate limits
  • Progress Tracking: Monitor processing status
  • Error Recovery: Retry failed uploads
  • Resource Management: Efficient memory and network usage

Step 6: Production Considerations

Error Handling

Before you run this pipeline in production, handle authentication failures, back off on rate limits, retry network errors with a bounded budget, and validate content before upload. See Error Handling and Retries.

Monitoring and Logging

  • Success Metrics: Upload counts and timing
  • Error Tracking: Failed uploads with reasons
  • Performance Monitoring: API response times
  • Health Checks: System status monitoring

Complete Examples

These examples combine authentication, file processing, and error handling into a complete solution for each language. First, set up your environment variables in a .env file:
Or export them in your shell for Go and Java:

Testing Your Implementation

To test any of the complete examples:

1. Environment Setup

Create a .env file with your Gloo AI credentials:

2. Install Dependencies

Check each language’s README in the examples for its setup requirements.

3. Test Single File Upload

4. Test Directory Monitoring

5. Test Batch Processing

The system will:
  • Authenticate using your API key
  • Extract metadata from filenames and content
  • Upload content with proper error handling
  • Display success/failure status for each operation
  • Provide structured JSON responses from the API

Production Deployment Considerations

Error Handling

The examples report authentication errors, add delays between API calls, and retry network issues with configurable timeouts; validate files before you upload them. For which errors to retry and how to bound retries, see Error Handling and Retries.

Monitoring

  • Logging: Structured logs for all operations
  • Metrics: Success rates, processing times, error counts
  • Alerting: Failed upload notifications
  • Health Checks: System status monitoring

Scaling

  • Concurrent Processing: Parallel file processing capabilities
  • Queue Management: Async processing for high-volume scenarios
  • Resource Management: Memory and CPU optimization
  • Load Balancing: Multiple instance coordination

Working Code Sample

View Complete Code

Clone or browse the complete working examples for all 6 languages (JavaScript, TypeScript, Python, PHP, Go, Java) with setup instructions.

Next Steps

Explore next:
  1. Search API - Query your ingested content
  2. Chat Integration - Use ingested content in conversations
  3. Content Management - Organize and categorize content
  4. Advanced Metadata - Rich content classification and tagging
  5. Custom Pipelines - The source-agnostic pattern for integrating with a CMS, API, database, file library, or RSS feed