> ## Documentation Index > Fetch the complete documentation index at: https://docs.gloo.com/llms.txt > Use this file to discover all available pages before exploring further. # Upload Files to Data Engine > Learn how to upload files to the Gloo AI Data Engine for processing and AI-powered search. This recipe shows how to upload files directly to the Gloo AI Data Engine using the Upload Files API. You'll learn to upload single files, batch multiple files, associate custom IDs, and add metadata to your content. **Upload once. Search instantly. Build faster.** The Upload Files API lets you send any file directly to Gloo for real-time processing and indexing. Within minutes, your content becomes searchable, context-aware, and ready for AI-driven interactions. ## Prerequisites Before starting, ensure you have: * A Gloo AI Studio account * Your Client ID and Client Secret from the [API Credentials page](/studio/manage-api-credentials) * Your Publisher ID from your Gloo AI Studio account * **Authentication setup** - Complete the [Authentication Tutorial](/tutorials/authentication) first The Upload Files API requires Bearer token authentication. If you haven't set up authentication yet, follow the [Authentication Tutorial](/tutorials/authentication) to learn how to exchange your credentials for access tokens and manage token expiration. *** ## Understanding the Upload Files API The Upload Files API allows you to upload documents that get processed and made available for search and AI interaction. The primary endpoint is: **POST** `/ingestion/v2/files` ### Key Features * **Direct File Upload**: Send files via multipart/form-data * **Multi-File Support**: Upload multiple files in a single request * **Automatic Processing**: Files are parsed, indexed, and made searchable * **Format Detection**: Automatic detection and processing of various file types * **Duplicate Detection**: Identifies and reports duplicate content ### Required Fields The Upload Files API requires two form fields: * `publisher_id`: Your publisher ID from Gloo AI Studio * `files`: The file(s) to upload (use the field name `files`, not `file`) ### Supported File Types * PDF documents * Microsoft Word (.doc, .docx) * Plain text (.txt) * Markdown (.md) * And more *** ## Step 1: Basic Single File Upload Let's start with uploading a single file. This demonstrates the core API call with proper authentication. ```bash Shell theme={null} curl -X POST https://platform.ai.gloo.com/ingestion/v2/files \ -H "Authorization: Bearer $GLOO_ACCESS_TOKEN" \ -F "publisher_id=$GLOO_PUBLISHER_ID" \ -F "files=@/path/to/your/document.pdf" ``` ```python Python theme={null} import requests import os from dotenv import load_dotenv load_dotenv() # Get access token (see Authentication tutorial) ACCESS_TOKEN = os.getenv("GLOO_ACCESS_TOKEN") PUBLISHER_ID = os.getenv("GLOO_PUBLISHER_ID") API_URL = "https://platform.ai.gloo.com/ingestion/v2/files" def upload_single_file(file_path): """Upload a single file to the Data Engine.""" headers = { "Authorization": f"Bearer {ACCESS_TOKEN}" } with open(file_path, 'rb') as f: files = {"files": (os.path.basename(file_path), f)} data = {"publisher_id": PUBLISHER_ID} response = requests.post(API_URL, headers=headers, files=files, data=data) response.raise_for_status() return response.json() # Example usage if __name__ == "__main__": result = upload_single_file("/path/to/your/document.pdf") print(f"Upload result: {result}") ``` ```javascript JavaScript theme={null} const fs = require('fs'); const path = require('path'); const FormData = require('form-data'); const axios = require('axios'); require('dotenv').config(); const ACCESS_TOKEN = process.env.GLOO_ACCESS_TOKEN; const PUBLISHER_ID = process.env.GLOO_PUBLISHER_ID; const API_URL = "https://platform.ai.gloo.com/ingestion/v2/files"; async function uploadSingleFile(filePath) { const formData = new FormData(); formData.append('files', fs.createReadStream(filePath)); formData.append('publisher_id', PUBLISHER_ID); const response = await axios.post(API_URL, formData, { headers: { 'Authorization': `Bearer ${ACCESS_TOKEN}`, ...formData.getHeaders() } }); return response.data; } // Example usage uploadSingleFile('/path/to/your/document.pdf') .then(result => console.log('Upload result:', result)) .catch(error => console.error('Upload failed:', error.message)); ``` ```typescript TypeScript theme={null} import * as fs from 'fs'; import * as path from 'path'; import FormData from 'form-data'; import axios from 'axios'; import * as dotenv from 'dotenv'; dotenv.config(); const ACCESS_TOKEN = process.env.GLOO_ACCESS_TOKEN; const PUBLISHER_ID = process.env.GLOO_PUBLISHER_ID; const API_URL = "https://platform.ai.gloo.com/ingestion/v2/files"; interface UploadResponse { success: boolean; message: string; ingesting: string[]; duplicates: string[]; } async function uploadSingleFile(filePath: string): Promise { const formData = new FormData(); formData.append('files', fs.createReadStream(filePath)); formData.append('publisher_id', PUBLISHER_ID); const response = await axios.post(API_URL, formData, { headers: { 'Authorization': `Bearer ${ACCESS_TOKEN}`, ...formData.getHeaders() } }); return response.data; } // Example usage uploadSingleFile('/path/to/your/document.pdf') .then(result => console.log('Upload result:', result)) .catch(error => console.error('Upload failed:', error.message)); ``` ```php PHP theme={null} load(); $ACCESS_TOKEN = $_ENV['GLOO_ACCESS_TOKEN'] ?? null; $PUBLISHER_ID = $_ENV['GLOO_PUBLISHER_ID'] ?? null; $API_URL = 'https://platform.ai.gloo.com/ingestion/v2/files'; if (!$ACCESS_TOKEN) { die("Error: GLOO_ACCESS_TOKEN environment variable is required\n"); } function uploadSingleFile(string $filePath): array { global $ACCESS_TOKEN, $PUBLISHER_ID, $API_URL; $ch = curl_init(); $postFields = [ 'files' => new CURLFile($filePath, mime_content_type($filePath), basename($filePath)), 'publisher_id' => $PUBLISHER_ID ]; curl_setopt($ch, CURLOPT_URL, $API_URL); curl_setopt($ch, CURLOPT_RETURNTRANSFER, true); curl_setopt($ch, CURLOPT_POST, true); curl_setopt($ch, CURLOPT_POSTFIELDS, $postFields); curl_setopt($ch, CURLOPT_HTTPHEADER, [ 'Authorization: Bearer ' . $ACCESS_TOKEN ]); $result = curl_exec($ch); if (curl_errno($ch)) { throw new Exception('Upload failed: ' . curl_error($ch)); } curl_close($ch); return json_decode($result, true); } // Example usage $result = uploadSingleFile('/path/to/your/document.pdf'); echo "Upload result: " . json_encode($result, JSON_PRETTY_PRINT) . "\n"; ?> ``` ```go Go theme={null} package main import ( "bytes" "encoding/json" "fmt" "io" "mime/multipart" "net/http" "os" "path/filepath" "time" "github.com/joho/godotenv" ) var ( accessToken string publisherID string apiURL = "https://platform.ai.gloo.com/ingestion/v2/files" ) type UploadResponse struct { Success bool `json:"success"` Message string `json:"message"` Ingesting []string `json:"ingesting"` Duplicates []string `json:"duplicates"` } func init() { godotenv.Load() accessToken = os.Getenv("GLOO_ACCESS_TOKEN") publisherID = os.Getenv("GLOO_PUBLISHER_ID") if accessToken == "" { fmt.Println("Error: GLOO_ACCESS_TOKEN environment variable is required") os.Exit(1) } } func uploadSingleFile(filePath string) (*UploadResponse, error) { file, err := os.Open(filePath) if err != nil { return nil, err } defer file.Close() var body bytes.Buffer writer := multipart.NewWriter(&body) part, err := writer.CreateFormFile("files", filepath.Base(filePath)) if err != nil { return nil, err } if _, err := io.Copy(part, file); err != nil { return nil, err } // Add publisher_id field if err := writer.WriteField("publisher_id", publisherID); err != nil { return nil, err } writer.Close() req, err := http.NewRequest("POST", apiURL, &body) if err != nil { return nil, err } req.Header.Set("Authorization", "Bearer "+accessToken) req.Header.Set("Content-Type", writer.FormDataContentType()) client := &http.Client{Timeout: 60 * time.Second} resp, err := client.Do(req) if err != nil { return nil, err } defer resp.Body.Close() respBody, err := io.ReadAll(resp.Body) if err != nil { return nil, err } if resp.StatusCode >= 300 { return nil, fmt.Errorf("upload failed: %s", string(respBody)) } var result UploadResponse json.Unmarshal(respBody, &result) return &result, nil } func main() { result, err := uploadSingleFile("/path/to/your/document.pdf") if err != nil { fmt.Printf("Upload failed: %v\n", err) return } fmt.Printf("Upload result: %+v\n", result) } ``` ```java Java theme={null} import io.github.cdimascio.dotenv.Dotenv; import java.io.ByteArrayOutputStream; import java.io.IOException; import java.net.URI; import java.net.http.HttpClient; import java.net.http.HttpRequest; import java.net.http.HttpResponse; import java.nio.charset.StandardCharsets; import java.nio.file.Files; import java.nio.file.Path; import java.util.UUID; public class FileUploader { private static final Dotenv dotenv = Dotenv.configure().ignoreIfMissing().load(); private static final String ACCESS_TOKEN = dotenv.get("GLOO_ACCESS_TOKEN"); private static final String PUBLISHER_ID = dotenv.get("GLOO_PUBLISHER_ID"); private static final String API_URL = "https://platform.ai.gloo.com/ingestion/v2/files"; private static final HttpClient httpClient = HttpClient.newHttpClient(); public static String uploadSingleFile(String filePath) throws IOException, InterruptedException { if (ACCESS_TOKEN == null || ACCESS_TOKEN.isEmpty()) { throw new IllegalStateException("GLOO_ACCESS_TOKEN environment variable is required"); } Path path = Path.of(filePath); String boundary = UUID.randomUUID().toString(); byte[] body = buildMultipartBody(boundary, path); HttpRequest request = HttpRequest.newBuilder() .uri(URI.create(API_URL)) .header("Authorization", "Bearer " + ACCESS_TOKEN) .header("Content-Type", "multipart/form-data; boundary=" + boundary) .POST(HttpRequest.BodyPublishers.ofByteArray(body)) .build(); HttpResponse response = httpClient.send(request, HttpResponse.BodyHandlers.ofString()); if (response.statusCode() >= 300) { throw new IOException("Upload failed: " + response.body()); } return response.body(); } private static byte[] buildMultipartBody(String boundary, Path filePath) throws IOException { String fileName = filePath.getFileName().toString(); byte[] fileContent = Files.readAllBytes(filePath); ByteArrayOutputStream out = new ByteArrayOutputStream(); // Add publisher_id field out.write(("--" + boundary + "\r\n").getBytes(StandardCharsets.UTF_8)); out.write("Content-Disposition: form-data; name=\"publisher_id\"\r\n\r\n".getBytes(StandardCharsets.UTF_8)); out.write((PUBLISHER_ID + "\r\n").getBytes(StandardCharsets.UTF_8)); // Add file field out.write(("--" + boundary + "\r\n").getBytes(StandardCharsets.UTF_8)); out.write(("Content-Disposition: form-data; name=\"files\"; filename=\"" + fileName + "\"\r\n").getBytes(StandardCharsets.UTF_8)); out.write("Content-Type: application/octet-stream\r\n\r\n".getBytes(StandardCharsets.UTF_8)); out.write(fileContent); out.write(("\r\n--" + boundary + "--\r\n").getBytes(StandardCharsets.UTF_8)); return out.toByteArray(); } public static void main(String[] args) { try { String result = uploadSingleFile("/path/to/your/document.pdf"); System.out.println("Upload result: " + result); } catch (Exception e) { System.err.println("Upload failed: " + e.getMessage()); } } } ``` ### Expected Response ```json theme={null} { "success": true, "message": "File processing started in background.", "ingesting": [ "c999008e-de60-495c-8c9f-6a4b59cdb04b" ], "duplicates": [] } ``` *** ## Step 2: Multi-File Upload You can upload multiple files in a single request for better efficiency. You can mix file types freely - the endpoint automatically detects and processes each format. ```bash Shell theme={null} curl -X POST https://platform.ai.gloo.com/ingestion/v2/files \ -H "Authorization: Bearer $GLOO_ACCESS_TOKEN" \ -F "publisher_id=$GLOO_PUBLISHER_ID" \ -F "files=@/path/to/document1.pdf" \ -F "files=@/path/to/document2.pdf" \ -F "files=@/path/to/document3.docx" ``` ```python Python theme={null} def upload_multiple_files(file_paths): """Upload multiple files to the Data Engine.""" headers = { "Authorization": f"Bearer {ACCESS_TOKEN}" } files = [] open_files = [] try: for file_path in file_paths: f = open(file_path, 'rb') open_files.append(f) files.append(('files', (os.path.basename(file_path), f))) data = {"publisher_id": PUBLISHER_ID} response = requests.post(API_URL, headers=headers, files=files, data=data) response.raise_for_status() return response.json() finally: for f in open_files: f.close() # Example usage if __name__ == "__main__": files_to_upload = [ "/path/to/document1.pdf", "/path/to/document2.pdf", "/path/to/document3.docx" ] result = upload_multiple_files(files_to_upload) print(f"Uploaded {len(result.get('ingesting', []))} files") if result.get('duplicates'): print(f"Duplicates found: {len(result['duplicates'])}") ``` ```javascript JavaScript theme={null} async function uploadMultipleFiles(filePaths) { const formData = new FormData(); for (const filePath of filePaths) { formData.append('files', fs.createReadStream(filePath)); } formData.append('publisher_id', PUBLISHER_ID); const response = await axios.post(API_URL, formData, { headers: { 'Authorization': `Bearer ${ACCESS_TOKEN}`, ...formData.getHeaders() } }); return response.data; } // Example usage const filesToUpload = [ '/path/to/document1.pdf', '/path/to/document2.pdf', '/path/to/document3.docx' ]; uploadMultipleFiles(filesToUpload) .then(result => { console.log(`Uploaded ${result.ingesting?.length || 0} files`); if (result.duplicates?.length) { console.log(`Duplicates found: ${result.duplicates.length}`); } }) .catch(error => console.error('Upload failed:', error.message)); ``` ```typescript TypeScript theme={null} async function uploadMultipleFiles(filePaths: string[]): Promise { const formData = new FormData(); for (const filePath of filePaths) { formData.append('files', fs.createReadStream(filePath)); } formData.append('publisher_id', PUBLISHER_ID); const response = await axios.post(API_URL, formData, { headers: { 'Authorization': `Bearer ${ACCESS_TOKEN}`, ...formData.getHeaders() } }); return response.data; } // Example usage const filesToUpload = [ '/path/to/document1.pdf', '/path/to/document2.pdf', '/path/to/document3.docx' ]; uploadMultipleFiles(filesToUpload) .then(result => { console.log(`Uploaded ${result.ingesting?.length || 0} files`); if (result.duplicates?.length) { console.log(`Duplicates found: ${result.duplicates.length}`); } }) .catch(error => console.error('Upload failed:', error.message)); ``` ```php PHP theme={null} function uploadMultipleFiles(array $filePaths): array { global $ACCESS_TOKEN, $PUBLISHER_ID, $API_URL; $ch = curl_init(); $postFields = ['publisher_id' => $PUBLISHER_ID]; foreach ($filePaths as $index => $filePath) { $postFields["files[$index]"] = new CURLFile( $filePath, mime_content_type($filePath), basename($filePath) ); } curl_setopt($ch, CURLOPT_URL, $API_URL); curl_setopt($ch, CURLOPT_RETURNTRANSFER, true); curl_setopt($ch, CURLOPT_POST, true); curl_setopt($ch, CURLOPT_POSTFIELDS, $postFields); curl_setopt($ch, CURLOPT_HTTPHEADER, [ 'Authorization: Bearer ' . $ACCESS_TOKEN ]); $result = curl_exec($ch); if (curl_errno($ch)) { throw new Exception('Upload failed: ' . curl_error($ch)); } curl_close($ch); return json_decode($result, true); } // Example usage $filesToUpload = [ '/path/to/document1.pdf', '/path/to/document2.pdf', '/path/to/document3.docx' ]; $result = uploadMultipleFiles($filesToUpload); echo "Uploaded " . count($result['ingesting'] ?? []) . " files\n"; if (!empty($result['duplicates'])) { echo "Duplicates found: " . count($result['duplicates']) . "\n"; } ``` ```go Go theme={null} func uploadMultipleFiles(filePaths []string) (*UploadResponse, error) { var body bytes.Buffer writer := multipart.NewWriter(&body) for _, filePath := range filePaths { file, err := os.Open(filePath) if err != nil { return nil, err } part, err := writer.CreateFormFile("files", filepath.Base(filePath)) if err != nil { file.Close() return nil, err } if _, err := io.Copy(part, file); err != nil { file.Close() return nil, err } file.Close() } // Add publisher_id field if err := writer.WriteField("publisher_id", publisherID); err != nil { return nil, err } writer.Close() req, err := http.NewRequest("POST", apiURL, &body) if err != nil { return nil, err } req.Header.Set("Authorization", "Bearer "+accessToken) req.Header.Set("Content-Type", writer.FormDataContentType()) client := &http.Client{Timeout: 120 * time.Second} resp, err := client.Do(req) if err != nil { return nil, err } defer resp.Body.Close() respBody, err := io.ReadAll(resp.Body) if err != nil { return nil, err } if resp.StatusCode >= 300 { return nil, fmt.Errorf("upload failed: %s", string(respBody)) } var result UploadResponse json.Unmarshal(respBody, &result) return &result, nil } func main() { filesToUpload := []string{ "/path/to/document1.pdf", "/path/to/document2.pdf", "/path/to/document3.docx", } result, err := uploadMultipleFiles(filesToUpload) if err != nil { fmt.Printf("Upload failed: %v\n", err) return } fmt.Printf("Uploaded %d files\n", len(result.Ingesting)) if len(result.Duplicates) > 0 { fmt.Printf("Duplicates found: %d\n", len(result.Duplicates)) } } ``` ```java Java theme={null} public static String uploadMultipleFiles(String[] filePaths) throws IOException, InterruptedException { if (ACCESS_TOKEN == null || ACCESS_TOKEN.isEmpty()) { throw new IllegalStateException("GLOO_ACCESS_TOKEN environment variable is required"); } String boundary = UUID.randomUUID().toString(); StringBuilder body = new StringBuilder(); // Add publisher_id field body.append("--").append(boundary).append("\r\n"); body.append("Content-Disposition: form-data; name=\"publisher_id\"\r\n\r\n"); body.append(PUBLISHER_ID).append("\r\n"); for (String filePath : filePaths) { Path path = Path.of(filePath); String fileName = path.getFileName().toString(); byte[] fileContent = Files.readAllBytes(path); body.append("--").append(boundary).append("\r\n"); body.append("Content-Disposition: form-data; name=\"files\"; filename=\"") .append(fileName).append("\"\r\n"); body.append("Content-Type: application/octet-stream\r\n\r\n"); body.append(new String(fileContent)); body.append("\r\n"); } body.append("--").append(boundary).append("--\r\n"); HttpRequest request = HttpRequest.newBuilder() .uri(URI.create(API_URL)) .header("Authorization", "Bearer " + ACCESS_TOKEN) .header("Content-Type", "multipart/form-data; boundary=" + boundary) .POST(HttpRequest.BodyPublishers.ofString(body.toString())) .build(); HttpResponse response = httpClient.send(request, HttpResponse.BodyHandlers.ofString()); if (response.statusCode() >= 300) { throw new IOException("Upload failed: " + response.body()); } return response.body(); } public static void main(String[] args) { try { String[] filesToUpload = { "/path/to/document1.pdf", "/path/to/document2.pdf", "/path/to/document3.docx" }; String result = uploadMultipleFiles(filesToUpload); System.out.println("Upload result: " + result); } catch (Exception e) { System.err.println("Upload failed: " + e.getMessage()); } } ``` ### Expected Response ```json theme={null} { "success": true, "message": "File processing started in background.", "ingesting": [ "c999008e-de60-495c-8c9f-6a4b59cdb04b", "b10e85d8-243d-46e2-9504-d93874a9ebcb", "b45058a8-2f8c-4a88-8aba-7adb4afcd38d" ], "duplicates": [] } ``` *** ## Step 3: Using Producer ID The `producer_id` query parameter lets you associate your internal ID with an uploaded file. This is useful for tracking and updating content from your system. If the `producer_id` field is supplied with a multi-file upload, the ID will be ignored. Producer IDs must have a one-to-one relationship with a file. ```bash Shell theme={null} curl -X POST "https://platform.ai.gloo.com/ingestion/v2/files?producer_id=my-internal-id-12345" \ -H "Authorization: Bearer $GLOO_ACCESS_TOKEN" \ -F "publisher_id=$GLOO_PUBLISHER_ID" \ -F "files=@/path/to/your/document.pdf" ``` ```python Python theme={null} def upload_file_with_producer_id(file_path, producer_id): """Upload a file with a custom producer ID.""" headers = { "Authorization": f"Bearer {ACCESS_TOKEN}" } params = { "producer_id": producer_id } with open(file_path, 'rb') as f: files = {"files": (os.path.basename(file_path), f)} data = {"publisher_id": PUBLISHER_ID} response = requests.post(API_URL, headers=headers, files=files, data=data, params=params) response.raise_for_status() return response.json() # Example usage if __name__ == "__main__": result = upload_file_with_producer_id( "/path/to/document.pdf", "my-internal-id-12345" ) print(f"Upload result: {result}") ``` ```javascript JavaScript theme={null} async function uploadFileWithProducerId(filePath, producerId) { const formData = new FormData(); formData.append('files', fs.createReadStream(filePath)); formData.append('publisher_id', PUBLISHER_ID); const response = await axios.post(API_URL, formData, { headers: { 'Authorization': `Bearer ${ACCESS_TOKEN}`, ...formData.getHeaders() }, params: { producer_id: producerId } }); return response.data; } // Example usage uploadFileWithProducerId('/path/to/document.pdf', 'my-internal-id-12345') .then(result => console.log('Upload result:', result)) .catch(error => console.error('Upload failed:', error.message)); ``` ```typescript TypeScript theme={null} async function uploadFileWithProducerId(filePath: string, producerId: string): Promise { const formData = new FormData(); formData.append('files', fs.createReadStream(filePath)); formData.append('publisher_id', PUBLISHER_ID); const response = await axios.post(API_URL, formData, { headers: { 'Authorization': `Bearer ${ACCESS_TOKEN}`, ...formData.getHeaders() }, params: { producer_id: producerId } }); return response.data; } // Example usage uploadFileWithProducerId('/path/to/document.pdf', 'my-internal-id-12345') .then(result => console.log('Upload result:', result)) .catch(error => console.error('Upload failed:', error.message)); ``` ```php PHP theme={null} function uploadFileWithProducerId(string $filePath, string $producerId): array { global $ACCESS_TOKEN, $PUBLISHER_ID, $API_URL; $url = $API_URL . '?' . http_build_query(['producer_id' => $producerId]); $ch = curl_init(); $postFields = [ 'files' => new CURLFile($filePath, mime_content_type($filePath), basename($filePath)), 'publisher_id' => $PUBLISHER_ID ]; curl_setopt($ch, CURLOPT_URL, $url); curl_setopt($ch, CURLOPT_RETURNTRANSFER, true); curl_setopt($ch, CURLOPT_POST, true); curl_setopt($ch, CURLOPT_POSTFIELDS, $postFields); curl_setopt($ch, CURLOPT_HTTPHEADER, [ 'Authorization: Bearer ' . $ACCESS_TOKEN ]); $result = curl_exec($ch); if (curl_errno($ch)) { throw new Exception('Upload failed: ' . curl_error($ch)); } curl_close($ch); return json_decode($result, true); } // Example usage $result = uploadFileWithProducerId('/path/to/document.pdf', 'my-internal-id-12345'); echo "Upload result: " . json_encode($result, JSON_PRETTY_PRINT) . "\n"; ``` ```go Go theme={null} func uploadFileWithProducerId(filePath, producerId string) (*UploadResponse, error) { file, err := os.Open(filePath) if err != nil { return nil, err } defer file.Close() var body bytes.Buffer writer := multipart.NewWriter(&body) part, err := writer.CreateFormFile("files", filepath.Base(filePath)) if err != nil { return nil, err } if _, err := io.Copy(part, file); err != nil { return nil, err } // Add publisher_id field if err := writer.WriteField("publisher_id", publisherID); err != nil { return nil, err } writer.Close() url := fmt.Sprintf("%s?producer_id=%s", apiURL, producerId) req, err := http.NewRequest("POST", url, &body) if err != nil { return nil, err } req.Header.Set("Authorization", "Bearer "+accessToken) req.Header.Set("Content-Type", writer.FormDataContentType()) client := &http.Client{Timeout: 60 * time.Second} resp, err := client.Do(req) if err != nil { return nil, err } defer resp.Body.Close() respBody, err := io.ReadAll(resp.Body) if err != nil { return nil, err } if resp.StatusCode >= 300 { return nil, fmt.Errorf("upload failed: %s", string(respBody)) } var result UploadResponse json.Unmarshal(respBody, &result) return &result, nil } func main() { result, err := uploadFileWithProducerId("/path/to/document.pdf", "my-internal-id-12345") if err != nil { fmt.Printf("Upload failed: %v\n", err) return } fmt.Printf("Upload result: %+v\n", result) } ``` ```java Java theme={null} public static String uploadFileWithProducerId(String filePath, String producerId) throws IOException, InterruptedException { if (ACCESS_TOKEN == null || ACCESS_TOKEN.isEmpty()) { throw new IllegalStateException("GLOO_ACCESS_TOKEN environment variable is required"); } Path path = Path.of(filePath); String boundary = UUID.randomUUID().toString(); String body = buildMultipartBodyWithPublisher(boundary, path); String url = API_URL + "?producer_id=" + producerId; HttpRequest request = HttpRequest.newBuilder() .uri(URI.create(url)) .header("Authorization", "Bearer " + ACCESS_TOKEN) .header("Content-Type", "multipart/form-data; boundary=" + boundary) .POST(HttpRequest.BodyPublishers.ofString(body)) .build(); HttpResponse response = httpClient.send(request, HttpResponse.BodyHandlers.ofString()); if (response.statusCode() >= 300) { throw new IOException("Upload failed: " + response.body()); } return response.body(); } // Note: buildMultipartBodyWithPublisher includes publisher_id and uses "files" field name // See the complete example in the sandbox code for full implementation public static void main(String[] args) { try { String result = uploadFileWithProducerId("/path/to/document.pdf", "my-internal-id-12345"); System.out.println("Upload result: " + result); } catch (Exception e) { System.err.println("Upload failed: " + e.getMessage()); } } ``` *** ## Step 4: Adding Metadata to Uploaded Content After uploading files, you can add or update metadata using the Update Item Metadata endpoint. Identify content by either Gloo's `item_id` (returned from the upload) or your `producer_id`. Only the fields you include are updated; omitted fields remain unchanged. ```bash Shell theme={null} curl -X POST https://platform.ai.gloo.com/engine/v2/item \ -H "Authorization: Bearer $GLOO_ACCESS_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "publisher_id": "your-publisher-id", "item_id": "c999008e-de60-495c-8c9f-6a4b59cdb04b", "item_title": "Document Title", "item_subtitle": "A brief subtitle", "author": ["John Doe", "Jane Smith"], "publication_date": "2025-01-15", "item_tags": ["category1", "category2"] }' ``` ```python Python theme={null} METADATA_URL = "https://platform.ai.gloo.com/engine/v2/item" def update_item_metadata(item_id=None, producer_id=None, publisher_id=None, **metadata): """Update metadata for an uploaded item.""" headers = { "Authorization": f"Bearer {ACCESS_TOKEN}", "Content-Type": "application/json" } data = { "publisher_id": publisher_id } if item_id: data["item_id"] = item_id if producer_id: data["producer_id"] = producer_id data.update(metadata) response = requests.post(METADATA_URL, headers=headers, json=data) response.raise_for_status() return response.json() # Example usage if __name__ == "__main__": result = update_item_metadata( item_id="c999008e-de60-495c-8c9f-6a4b59cdb04b", publisher_id="your-publisher-id", item_title="Document Title", item_subtitle="A brief subtitle", author=["John Doe", "Jane Smith"], publication_date="2025-01-15", item_tags=["category1", "category2"] ) print(f"Metadata update result: {result}") ``` ```javascript JavaScript theme={null} const METADATA_URL = "https://platform.ai.gloo.com/engine/v2/item"; async function updateItemMetadata({ itemId, producerId, publisherId, ...metadata }) { const data = { publisher_id: publisherId, ...(itemId && { item_id: itemId }), ...(producerId && { producer_id: producerId }), ...metadata }; const response = await axios.post(METADATA_URL, data, { headers: { 'Authorization': `Bearer ${ACCESS_TOKEN}`, 'Content-Type': 'application/json' } }); return response.data; } // Example usage updateItemMetadata({ itemId: 'c999008e-de60-495c-8c9f-6a4b59cdb04b', publisherId: 'your-publisher-id', item_title: 'Document Title', item_subtitle: 'A brief subtitle', author: ['John Doe', 'Jane Smith'], publication_date: '2025-01-15', item_tags: ['category1', 'category2'] }) .then(result => console.log('Metadata update result:', result)) .catch(error => console.error('Update failed:', error.message)); ``` ```typescript TypeScript theme={null} const METADATA_URL = "https://platform.ai.gloo.com/engine/v2/item"; interface ItemMetadata { itemId?: string; producerId?: string; publisherId: string; item_title?: string; item_subtitle?: string; author?: string[]; publication_date?: string; item_tags?: string[]; item_summary?: string; item_url?: string; item_image?: string; } interface MetadataResponse { success: boolean; message: string; } async function updateItemMetadata(metadata: ItemMetadata): Promise { const { itemId, producerId, publisherId, ...rest } = metadata; const data = { publisher_id: publisherId, ...(itemId && { item_id: itemId }), ...(producerId && { producer_id: producerId }), ...rest }; const response = await axios.post(METADATA_URL, data, { headers: { 'Authorization': `Bearer ${ACCESS_TOKEN}`, 'Content-Type': 'application/json' } }); return response.data; } // Example usage updateItemMetadata({ itemId: 'c999008e-de60-495c-8c9f-6a4b59cdb04b', publisherId: 'your-publisher-id', item_title: 'Document Title', item_subtitle: 'A brief subtitle', author: ['John Doe', 'Jane Smith'], publication_date: '2025-01-15', item_tags: ['category1', 'category2'] }) .then(result => console.log('Metadata update result:', result)) .catch(error => console.error('Update failed:', error.message)); ``` ```php PHP theme={null} $METADATA_URL = 'https://platform.ai.gloo.com/engine/v2/item'; function updateItemMetadata(array $params): array { global $ACCESS_TOKEN, $METADATA_URL; $ch = curl_init(); curl_setopt($ch, CURLOPT_URL, $METADATA_URL); curl_setopt($ch, CURLOPT_RETURNTRANSFER, true); curl_setopt($ch, CURLOPT_POST, true); curl_setopt($ch, CURLOPT_POSTFIELDS, json_encode($params)); curl_setopt($ch, CURLOPT_HTTPHEADER, [ 'Authorization: Bearer ' . $ACCESS_TOKEN, 'Content-Type: application/json' ]); $result = curl_exec($ch); if (curl_errno($ch)) { throw new Exception('Update failed: ' . curl_error($ch)); } curl_close($ch); return json_decode($result, true); } // Example usage $result = updateItemMetadata([ 'publisher_id' => 'your-publisher-id', 'item_id' => 'c999008e-de60-495c-8c9f-6a4b59cdb04b', 'item_title' => 'Document Title', 'item_subtitle' => 'A brief subtitle', 'author' => ['John Doe', 'Jane Smith'], 'publication_date' => '2025-01-15', 'item_tags' => ['category1', 'category2'] ]); echo "Metadata update result: " . json_encode($result, JSON_PRETTY_PRINT) . "\n"; ``` ```go Go theme={null} var metadataURL = "https://platform.ai.gloo.com/engine/v2/item" type ItemMetadata struct { PublisherID string `json:"publisher_id"` ItemID string `json:"item_id,omitempty"` ProducerID string `json:"producer_id,omitempty"` ItemTitle string `json:"item_title,omitempty"` ItemSubtitle string `json:"item_subtitle,omitempty"` Author []string `json:"author,omitempty"` PublicationDate string `json:"publication_date,omitempty"` ItemTags []string `json:"item_tags,omitempty"` } type MetadataResponse struct { Success bool `json:"success"` Message string `json:"message"` } func updateItemMetadata(metadata ItemMetadata) (*MetadataResponse, error) { jsonData, err := json.Marshal(metadata) if err != nil { return nil, err } req, err := http.NewRequest("POST", metadataURL, bytes.NewBuffer(jsonData)) if err != nil { return nil, err } req.Header.Set("Authorization", "Bearer "+accessToken) req.Header.Set("Content-Type", "application/json") client := &http.Client{Timeout: 30 * time.Second} resp, err := client.Do(req) if err != nil { return nil, err } defer resp.Body.Close() respBody, err := io.ReadAll(resp.Body) if err != nil { return nil, err } if resp.StatusCode >= 300 { return nil, fmt.Errorf("update failed: %s", string(respBody)) } var result MetadataResponse json.Unmarshal(respBody, &result) return &result, nil } func main() { result, err := updateItemMetadata(ItemMetadata{ PublisherID: "your-publisher-id", ItemID: "c999008e-de60-495c-8c9f-6a4b59cdb04b", ItemTitle: "Document Title", ItemSubtitle: "A brief subtitle", Author: []string{"John Doe", "Jane Smith"}, PublicationDate: "2025-01-15", ItemTags: []string{"category1", "category2"}, }) if err != nil { fmt.Printf("Update failed: %v\n", err) return } fmt.Printf("Metadata update result: %+v\n", result) } ``` ```java Java theme={null} private static final String METADATA_URL = "https://platform.ai.gloo.com/engine/v2/item"; public static String updateItemMetadata(String jsonPayload) throws IOException, InterruptedException { if (ACCESS_TOKEN == null || ACCESS_TOKEN.isEmpty()) { throw new IllegalStateException("GLOO_ACCESS_TOKEN environment variable is required"); } HttpRequest request = HttpRequest.newBuilder() .uri(URI.create(METADATA_URL)) .header("Authorization", "Bearer " + ACCESS_TOKEN) .header("Content-Type", "application/json") .POST(HttpRequest.BodyPublishers.ofString(jsonPayload)) .build(); HttpResponse response = httpClient.send(request, HttpResponse.BodyHandlers.ofString()); if (response.statusCode() >= 300) { throw new IOException("Update failed: " + response.body()); } return response.body(); } public static void main(String[] args) { try { String payload = """ { "publisher_id": "your-publisher-id", "item_id": "c999008e-de60-495c-8c9f-6a4b59cdb04b", "item_title": "Document Title", "item_subtitle": "A brief subtitle", "author": ["John Doe", "Jane Smith"], "publication_date": "2025-01-15", "item_tags": ["category1", "category2"] } """; String result = updateItemMetadata(payload); System.out.println("Metadata update result: " + result); } catch (Exception e) { System.err.println("Update failed: " + e.getMessage()); } } ``` ### Available Metadata Fields | Field | Type | Description | | ------------------ | --------- | ------------------------------------- | | `publisher_id` | string | **Required.** Your publisher ID | | `item_id` | string | Gloo's item ID (from upload response) | | `producer_id` | string | Your internal ID | | `item_title` | string | Document title | | `item_subtitle` | string | Document subtitle | | `file_name` | string | Original filename | | `publication_date` | string | Publication date (YYYY-MM-DD) | | `item_image` | string | Image URL | | `item_url` | string | Source URL | | `item_summary` | string | Brief summary | | `author` | string\[] | List of authors | | `item_tags` | string\[] | Categorization tags | If you provide **both** `item_id` and `producer_id`, the service will **prefer `item_id`** when loading the target item. This allows for the `producer_id` to be updated. *** ## Step 5: Verifying Your Upload After uploading content, you can verify it in Gloo AI Studio: 1. Log in to [Gloo AI Studio](https://studio.ai.gloo.com/) 2. Navigate to the **Data Engine** section from the main sidebar 3. Click on **Your Data** You'll see your uploaded files with their processing status and metadata. *** ## Complete Example Here's a complete example that combines authentication, file upload, and metadata update: ```python Python theme={null} """ Complete file upload workflow for Gloo AI Data Engine. """ import requests import time import os from dotenv import load_dotenv load_dotenv() # Configuration CLIENT_ID = os.getenv("GLOO_CLIENT_ID") CLIENT_SECRET = os.getenv("GLOO_CLIENT_SECRET") PUBLISHER_ID = os.getenv("GLOO_PUBLISHER_ID") TOKEN_URL = "https://platform.ai.gloo.com/oauth2/token" UPLOAD_URL = "https://platform.ai.gloo.com/ingestion/v2/files" METADATA_URL = "https://platform.ai.gloo.com/engine/v2/item" # Token management access_token_info = {} def get_access_token(): """Retrieve a new access token.""" headers = {"Content-Type": "application/x-www-form-urlencoded"} data = {"grant_type": "client_credentials", "scope": "api/access"} response = requests.post(TOKEN_URL, headers=headers, data=data, auth=(CLIENT_ID, CLIENT_SECRET)) response.raise_for_status() token_data = response.json() token_data['expires_at'] = int(time.time()) + token_data['expires_in'] return token_data def ensure_valid_token(): """Ensure we have a valid access token.""" global access_token_info if not access_token_info or time.time() > (access_token_info.get('expires_at', 0) - 60): print("Fetching new access token...") access_token_info = get_access_token() return access_token_info['access_token'] def upload_file(file_path, producer_id=None): """Upload a file to the Data Engine.""" token = ensure_valid_token() headers = {"Authorization": f"Bearer {token}"} params = {"producer_id": producer_id} if producer_id else {} with open(file_path, 'rb') as f: files = {"files": (os.path.basename(file_path), f)} data = {"publisher_id": PUBLISHER_ID} response = requests.post(UPLOAD_URL, headers=headers, files=files, data=data, params=params) response.raise_for_status() return response.json() def update_metadata(item_id, **metadata): """Update metadata for an uploaded item.""" token = ensure_valid_token() headers = { "Authorization": f"Bearer {token}", "Content-Type": "application/json" } data = {"publisher_id": PUBLISHER_ID, "item_id": item_id, **metadata} response = requests.post(METADATA_URL, headers=headers, json=data) response.raise_for_status() return response.json() def main(): """Main workflow: upload file and add metadata.""" file_path = "/path/to/your/document.pdf" # Step 1: Upload the file print(f"Uploading {file_path}...") upload_result = upload_file(file_path, producer_id="my-doc-001") print(f"Upload result: {upload_result}") if upload_result.get('ingesting'): item_id = upload_result['ingesting'][0] # Step 2: Add metadata print(f"Adding metadata to item {item_id}...") metadata_result = update_metadata( item_id, item_title="My Document Title", author=["Author Name"], publication_date="2025-01-15", item_tags=["documentation", "api"] ) print(f"Metadata result: {metadata_result}") if __name__ == "__main__": main() ``` ```javascript JavaScript theme={null} /** * Complete file upload workflow for Gloo AI Data Engine. */ const fs = require('fs'); const path = require('path'); const FormData = require('form-data'); const axios = require('axios'); require('dotenv').config(); // Configuration const CLIENT_ID = process.env.GLOO_CLIENT_ID; const CLIENT_SECRET = process.env.GLOO_CLIENT_SECRET; const PUBLISHER_ID = process.env.GLOO_PUBLISHER_ID; const TOKEN_URL = "https://platform.ai.gloo.com/oauth2/token"; const UPLOAD_URL = "https://platform.ai.gloo.com/ingestion/v2/files"; const METADATA_URL = "https://platform.ai.gloo.com/engine/v2/item"; // Token management let tokenInfo = {}; async function getAccessToken() { const body = 'grant_type=client_credentials&scope=api/access'; const response = await axios.post(TOKEN_URL, body, { headers: { 'Content-Type': 'application/x-www-form-urlencoded' }, auth: { username: CLIENT_ID, password: CLIENT_SECRET } }); const tokenData = response.data; tokenData.expires_at = Math.floor(Date.now() / 1000) + tokenData.expires_in; return tokenData; } async function ensureValidToken() { if (!tokenInfo.access_token || Date.now() / 1000 > (tokenInfo.expires_at - 60)) { console.log("Fetching new access token..."); tokenInfo = await getAccessToken(); } return tokenInfo.access_token; } async function uploadFile(filePath, producerId = null) { const token = await ensureValidToken(); const formData = new FormData(); formData.append('files', fs.createReadStream(filePath)); formData.append('publisher_id', PUBLISHER_ID); const config = { headers: { 'Authorization': `Bearer ${token}`, ...formData.getHeaders() } }; if (producerId) { config.params = { producer_id: producerId }; } const response = await axios.post(UPLOAD_URL, formData, config); return response.data; } async function updateMetadata(itemId, metadata) { const token = await ensureValidToken(); const data = { publisher_id: PUBLISHER_ID, item_id: itemId, ...metadata }; const response = await axios.post(METADATA_URL, data, { headers: { 'Authorization': `Bearer ${token}`, 'Content-Type': 'application/json' } }); return response.data; } async function main() { const filePath = '/path/to/your/document.pdf'; try { // Step 1: Upload the file console.log(`Uploading ${filePath}...`); const uploadResult = await uploadFile(filePath, 'my-doc-001'); console.log('Upload result:', uploadResult); if (uploadResult.ingesting?.length > 0) { const itemId = uploadResult.ingesting[0]; // Step 2: Add metadata console.log(`Adding metadata to item ${itemId}...`); const metadataResult = await updateMetadata(itemId, { item_title: 'My Document Title', author: ['Author Name'], publication_date: '2025-01-15', item_tags: ['documentation', 'api'] }); console.log('Metadata result:', metadataResult); } } catch (error) { console.error('Error:', error.response?.data || error.message); } } main(); ``` ```typescript TypeScript theme={null} /** * Complete file upload workflow for Gloo AI Data Engine. */ import * as fs from 'fs'; import * as path from 'path'; import FormData from 'form-data'; import axios from 'axios'; import * as dotenv from 'dotenv'; dotenv.config(); // Configuration const CLIENT_ID = process.env.GLOO_CLIENT_ID!; const CLIENT_SECRET = process.env.GLOO_CLIENT_SECRET!; const PUBLISHER_ID = process.env.GLOO_PUBLISHER_ID!; const TOKEN_URL = "https://platform.ai.gloo.com/oauth2/token"; const UPLOAD_URL = "https://platform.ai.gloo.com/ingestion/v2/files"; const METADATA_URL = "https://platform.ai.gloo.com/engine/v2/item"; // Types interface TokenInfo { access_token: string; expires_in: number; expires_at: number; token_type: string; } interface UploadResponse { success: boolean; message: string; ingesting: string[]; duplicates: string[]; } interface MetadataResponse { success: boolean; message: string; } // Token management let tokenInfo: Partial = {}; async function getAccessToken(): Promise { const body = 'grant_type=client_credentials&scope=api/access'; const response = await axios.post(TOKEN_URL, body, { headers: { 'Content-Type': 'application/x-www-form-urlencoded' }, auth: { username: CLIENT_ID, password: CLIENT_SECRET } }); const tokenData = response.data; tokenData.expires_at = Math.floor(Date.now() / 1000) + tokenData.expires_in; return tokenData; } async function ensureValidToken(): Promise { if (!tokenInfo.access_token || Date.now() / 1000 > ((tokenInfo.expires_at || 0) - 60)) { console.log("Fetching new access token..."); tokenInfo = await getAccessToken(); } return tokenInfo.access_token!; } async function uploadFile(filePath: string, producerId?: string): Promise { const token = await ensureValidToken(); const formData = new FormData(); formData.append('files', fs.createReadStream(filePath)); formData.append('publisher_id', PUBLISHER_ID); const config: any = { headers: { 'Authorization': `Bearer ${token}`, ...formData.getHeaders() } }; if (producerId) { config.params = { producer_id: producerId }; } const response = await axios.post(UPLOAD_URL, formData, config); return response.data; } async function updateMetadata(itemId: string, metadata: Record): Promise { const token = await ensureValidToken(); const data = { publisher_id: PUBLISHER_ID, item_id: itemId, ...metadata }; const response = await axios.post(METADATA_URL, data, { headers: { 'Authorization': `Bearer ${token}`, 'Content-Type': 'application/json' } }); return response.data; } async function main(): Promise { const filePath = '/path/to/your/document.pdf'; try { // Step 1: Upload the file console.log(`Uploading ${filePath}...`); const uploadResult = await uploadFile(filePath, 'my-doc-001'); console.log('Upload result:', uploadResult); if (uploadResult.ingesting?.length > 0) { const itemId = uploadResult.ingesting[0]; // Step 2: Add metadata console.log(`Adding metadata to item ${itemId}...`); const metadataResult = await updateMetadata(itemId, { item_title: 'My Document Title', author: ['Author Name'], publication_date: '2025-01-15', item_tags: ['documentation', 'api'] }); console.log('Metadata result:', metadataResult); } } catch (error: any) { console.error('Error:', error.response?.data || error.message); } } main(); ``` ```php PHP theme={null} safeLoad(); // Configuration $CLIENT_ID = $_ENV['GLOO_CLIENT_ID'] ?? ''; $CLIENT_SECRET = $_ENV['GLOO_CLIENT_SECRET'] ?? ''; $PUBLISHER_ID = $_ENV['GLOO_PUBLISHER_ID'] ?? ''; $TOKEN_URL = 'https://platform.ai.gloo.com/oauth2/token'; $UPLOAD_URL = 'https://platform.ai.gloo.com/ingestion/v2/files'; $METADATA_URL = 'https://platform.ai.gloo.com/engine/v2/item'; // Token management $tokenInfo = []; function getAccessToken(): array { global $CLIENT_ID, $CLIENT_SECRET, $TOKEN_URL; $ch = curl_init(); curl_setopt($ch, CURLOPT_URL, $TOKEN_URL); curl_setopt($ch, CURLOPT_RETURNTRANSFER, true); curl_setopt($ch, CURLOPT_POST, true); curl_setopt($ch, CURLOPT_POSTFIELDS, 'grant_type=client_credentials&scope=api/access'); curl_setopt($ch, CURLOPT_USERPWD, $CLIENT_ID . ':' . $CLIENT_SECRET); curl_setopt($ch, CURLOPT_HTTPHEADER, ['Content-Type: application/x-www-form-urlencoded']); $result = curl_exec($ch); curl_close($ch); $tokenData = json_decode($result, true); $tokenData['expires_at'] = time() + $tokenData['expires_in']; return $tokenData; } function ensureValidToken(): string { global $tokenInfo; if (empty($tokenInfo) || time() > ($tokenInfo['expires_at'] ?? 0) - 60) { echo "Fetching new access token...\n"; $tokenInfo = getAccessToken(); } return $tokenInfo['access_token']; } function uploadFile(string $filePath, ?string $producerId = null): array { global $UPLOAD_URL, $PUBLISHER_ID; $token = ensureValidToken(); $url = $UPLOAD_URL; if ($producerId) { $url .= '?' . http_build_query(['producer_id' => $producerId]); } $ch = curl_init(); curl_setopt($ch, CURLOPT_URL, $url); curl_setopt($ch, CURLOPT_RETURNTRANSFER, true); curl_setopt($ch, CURLOPT_POST, true); curl_setopt($ch, CURLOPT_POSTFIELDS, [ 'files' => new CURLFile($filePath, mime_content_type($filePath), basename($filePath)), 'publisher_id' => $PUBLISHER_ID ]); curl_setopt($ch, CURLOPT_HTTPHEADER, ['Authorization: Bearer ' . $token]); $result = curl_exec($ch); curl_close($ch); return json_decode($result, true); } function updateMetadata(string $itemId, array $metadata): array { global $METADATA_URL, $PUBLISHER_ID; $token = ensureValidToken(); $data = array_merge(['publisher_id' => $PUBLISHER_ID, 'item_id' => $itemId], $metadata); $ch = curl_init(); curl_setopt($ch, CURLOPT_URL, $METADATA_URL); curl_setopt($ch, CURLOPT_RETURNTRANSFER, true); curl_setopt($ch, CURLOPT_POST, true); curl_setopt($ch, CURLOPT_POSTFIELDS, json_encode($data)); curl_setopt($ch, CURLOPT_HTTPHEADER, [ 'Authorization: Bearer ' . $token, 'Content-Type: application/json' ]); $result = curl_exec($ch); curl_close($ch); return json_decode($result, true); } // Main workflow $filePath = '/path/to/your/document.pdf'; echo "Uploading $filePath...\n"; $uploadResult = uploadFile($filePath, 'my-doc-001'); echo "Upload result: " . json_encode($uploadResult) . "\n"; if (!empty($uploadResult['ingesting'])) { $itemId = $uploadResult['ingesting'][0]; echo "Adding metadata to item $itemId...\n"; $metadataResult = updateMetadata($itemId, [ 'item_title' => 'My Document Title', 'author' => ['Author Name'], 'publication_date' => '2025-01-15', 'item_tags' => ['documentation', 'api'] ]); echo "Metadata result: " . json_encode($metadataResult) . "\n"; } ?> ``` ```go Go theme={null} // Complete file upload workflow for Gloo AI Data Engine. package main import ( "bytes" "encoding/json" "fmt" "io" "mime/multipart" "net/http" "net/url" "os" "path/filepath" "strings" "time" "github.com/joho/godotenv" ) // Configuration var ( clientID string clientSecret string publisherID string tokenURL = "https://platform.ai.gloo.com/oauth2/token" uploadURL = "https://platform.ai.gloo.com/ingestion/v2/files" metadataURL = "https://platform.ai.gloo.com/engine/v2/item" ) // Types type TokenInfo struct { AccessToken string `json:"access_token"` ExpiresIn int `json:"expires_in"` ExpiresAt int64 `json:"expires_at"` } type UploadResponse struct { Success bool `json:"success"` Message string `json:"message"` Ingesting []string `json:"ingesting"` Duplicates []string `json:"duplicates"` } type MetadataResponse struct { Success bool `json:"success"` Message string `json:"message"` } // Token management var tokenInfo *TokenInfo func init() { godotenv.Load() clientID = os.Getenv("GLOO_CLIENT_ID") clientSecret = os.Getenv("GLOO_CLIENT_SECRET") publisherID = os.Getenv("GLOO_PUBLISHER_ID") } func getAccessToken() (*TokenInfo, error) { data := strings.NewReader("grant_type=client_credentials&scope=api/access") req, _ := http.NewRequest("POST", tokenURL, data) req.SetBasicAuth(clientID, clientSecret) req.Header.Set("Content-Type", "application/x-www-form-urlencoded") resp, err := http.DefaultClient.Do(req) if err != nil { return nil, err } defer resp.Body.Close() body, _ := io.ReadAll(resp.Body) var token TokenInfo json.Unmarshal(body, &token) token.ExpiresAt = time.Now().Unix() + int64(token.ExpiresIn) return &token, nil } func ensureValidToken() (string, error) { if tokenInfo == nil || time.Now().Unix() > (tokenInfo.ExpiresAt-60) { fmt.Println("Fetching new access token...") var err error tokenInfo, err = getAccessToken() if err != nil { return "", err } } return tokenInfo.AccessToken, nil } func uploadFile(filePath string, producerID string) (*UploadResponse, error) { token, _ := ensureValidToken() file, _ := os.Open(filePath) defer file.Close() var body bytes.Buffer writer := multipart.NewWriter(&body) part, _ := writer.CreateFormFile("files", filepath.Base(filePath)) io.Copy(part, file) writer.WriteField("publisher_id", publisherID) writer.Close() targetURL := uploadURL if producerID != "" { u, _ := url.Parse(uploadURL) q := u.Query() q.Set("producer_id", producerID) u.RawQuery = q.Encode() targetURL = u.String() } req, _ := http.NewRequest("POST", targetURL, &body) req.Header.Set("Authorization", "Bearer "+token) req.Header.Set("Content-Type", writer.FormDataContentType()) resp, _ := http.DefaultClient.Do(req) defer resp.Body.Close() respBody, _ := io.ReadAll(resp.Body) var result UploadResponse json.Unmarshal(respBody, &result) return &result, nil } func updateMetadata(itemID string, metadata map[string]interface{}) (*MetadataResponse, error) { token, _ := ensureValidToken() metadata["publisher_id"] = publisherID metadata["item_id"] = itemID jsonData, _ := json.Marshal(metadata) req, _ := http.NewRequest("POST", metadataURL, bytes.NewBuffer(jsonData)) req.Header.Set("Authorization", "Bearer "+token) req.Header.Set("Content-Type", "application/json") resp, _ := http.DefaultClient.Do(req) defer resp.Body.Close() respBody, _ := io.ReadAll(resp.Body) var result MetadataResponse json.Unmarshal(respBody, &result) return &result, nil } func main() { filePath := "/path/to/your/document.pdf" fmt.Printf("Uploading %s...\n", filePath) uploadResult, _ := uploadFile(filePath, "my-doc-001") fmt.Printf("Upload result: %+v\n", uploadResult) if len(uploadResult.Ingesting) > 0 { itemID := uploadResult.Ingesting[0] fmt.Printf("Adding metadata to item %s...\n", itemID) metadataResult, _ := updateMetadata(itemID, map[string]interface{}{ "item_title": "My Document Title", "author": []string{"Author Name"}, "publication_date": "2025-01-15", "item_tags": []string{"documentation", "api"}, }) fmt.Printf("Metadata result: %+v\n", metadataResult) } } ``` ```java Java theme={null} /** * Complete file upload workflow for Gloo AI Data Engine. */ package com.gloo.tutorial.uploadfiles; import io.github.cdimascio.dotenv.Dotenv; import com.google.gson.Gson; import com.google.gson.JsonObject; import java.io.*; import java.net.URI; import java.net.http.HttpClient; import java.net.http.HttpRequest; import java.net.http.HttpResponse; import java.nio.charset.StandardCharsets; import java.nio.file.*; import java.time.Instant; import java.util.*; public class Main { private static final Dotenv dotenv = Dotenv.configure().ignoreIfMissing().load(); private static final String CLIENT_ID = dotenv.get("GLOO_CLIENT_ID"); private static final String CLIENT_SECRET = dotenv.get("GLOO_CLIENT_SECRET"); private static final String PUBLISHER_ID = dotenv.get("GLOO_PUBLISHER_ID"); private static final String TOKEN_URL = "https://platform.ai.gloo.com/oauth2/token"; private static final String UPLOAD_URL = "https://platform.ai.gloo.com/ingestion/v2/files"; private static final String METADATA_URL = "https://platform.ai.gloo.com/engine/v2/item"; private static final HttpClient httpClient = HttpClient.newHttpClient(); private static final Gson gson = new Gson(); private static Map tokenInfo = new HashMap<>(); private static Map getAccessToken() throws Exception { String auth = CLIENT_ID + ":" + CLIENT_SECRET; String encodedAuth = Base64.getEncoder().encodeToString(auth.getBytes(StandardCharsets.UTF_8)); HttpRequest request = HttpRequest.newBuilder() .uri(URI.create(TOKEN_URL)) .header("Content-Type", "application/x-www-form-urlencoded") .header("Authorization", "Basic " + encodedAuth) .POST(HttpRequest.BodyPublishers.ofString("grant_type=client_credentials&scope=api/access")) .build(); HttpResponse response = httpClient.send(request, HttpResponse.BodyHandlers.ofString()); Map token = gson.fromJson(response.body(), Map.class); token.put("expires_at", Instant.now().getEpochSecond() + ((Double) token.get("expires_in")).longValue()); return token; } private static String ensureValidToken() throws Exception { long expiresAt = tokenInfo.containsKey("expires_at") ? ((Double) tokenInfo.get("expires_at")).longValue() : 0; if (tokenInfo.isEmpty() || Instant.now().getEpochSecond() > (expiresAt - 60)) { System.out.println("Fetching new access token..."); tokenInfo = getAccessToken(); } return (String) tokenInfo.get("access_token"); } private static Map uploadFile(String filePath, String producerId) throws Exception { String token = ensureValidToken(); Path path = Path.of(filePath); String boundary = UUID.randomUUID().toString(); ByteArrayOutputStream baos = new ByteArrayOutputStream(); PrintWriter writer = new PrintWriter(new OutputStreamWriter(baos, StandardCharsets.UTF_8), true); // Add publisher_id writer.append("--").append(boundary).append("\r\n"); writer.append("Content-Disposition: form-data; name=\"publisher_id\"\r\n\r\n"); writer.append(PUBLISHER_ID).append("\r\n"); // Add file writer.append("--").append(boundary).append("\r\n"); writer.append("Content-Disposition: form-data; name=\"files\"; filename=\"") .append(path.getFileName().toString()).append("\"\r\n"); writer.append("Content-Type: application/octet-stream\r\n\r\n"); writer.flush(); baos.write(Files.readAllBytes(path)); writer.append("\r\n--").append(boundary).append("--\r\n"); writer.flush(); String url = producerId != null ? UPLOAD_URL + "?producer_id=" + producerId : UPLOAD_URL; HttpRequest request = HttpRequest.newBuilder() .uri(URI.create(url)) .header("Authorization", "Bearer " + token) .header("Content-Type", "multipart/form-data; boundary=" + boundary) .POST(HttpRequest.BodyPublishers.ofByteArray(baos.toByteArray())) .build(); HttpResponse response = httpClient.send(request, HttpResponse.BodyHandlers.ofString()); return gson.fromJson(response.body(), Map.class); } private static Map updateMetadata(String itemId, Map metadata) throws Exception { String token = ensureValidToken(); JsonObject data = new JsonObject(); data.addProperty("publisher_id", PUBLISHER_ID); data.addProperty("item_id", itemId); for (Map.Entry entry : metadata.entrySet()) { if (entry.getValue() instanceof String) { data.addProperty(entry.getKey(), (String) entry.getValue()); } else if (entry.getValue() instanceof List) { data.add(entry.getKey(), gson.toJsonTree(entry.getValue())); } } HttpRequest request = HttpRequest.newBuilder() .uri(URI.create(METADATA_URL)) .header("Authorization", "Bearer " + token) .header("Content-Type", "application/json") .POST(HttpRequest.BodyPublishers.ofString(gson.toJson(data))) .build(); HttpResponse response = httpClient.send(request, HttpResponse.BodyHandlers.ofString()); return gson.fromJson(response.body(), Map.class); } public static void main(String[] args) { try { String filePath = "/path/to/your/document.pdf"; System.out.println("Uploading " + filePath + "..."); Map uploadResult = uploadFile(filePath, "my-doc-001"); System.out.println("Upload result: " + uploadResult); List ingesting = (List) uploadResult.get("ingesting"); if (ingesting != null && !ingesting.isEmpty()) { String itemId = ingesting.get(0); System.out.println("Adding metadata to item " + itemId + "..."); Map metadata = new HashMap<>(); metadata.put("item_title", "My Document Title"); metadata.put("author", List.of("Author Name")); metadata.put("publication_date", "2025-01-15"); metadata.put("item_tags", List.of("documentation", "api")); Map metadataResult = updateMetadata(itemId, metadata); System.out.println("Metadata result: " + metadataResult); } } catch (Exception e) { System.err.println("Error: " + e.getMessage()); } } } ``` *** ## Troubleshooting ### Error: 401 Unauthorized **Cause**: Token expired or invalid credentials. **Solution**: Refresh your access token and verify your Client ID and Secret. ### Error: 403 Forbidden **Cause**: Insufficient permissions for the publisher. **Solution**: Verify you have access to the specified publisher in Gloo AI Studio. ### Error: 413 Request Entity Too Large **Cause**: File size exceeds the limit. **Solution**: Check the [API limits](/api-reference/general/limits) and consider splitting large files. ### Error: Duplicate detected **Cause**: A file with identical content already exists. **Solution**: This is informational - the file won't be processed again. Check the `duplicates` array in the response. *** ## Working Code Sample Clone or browse the complete working examples for all 6 languages (JavaScript, TypeScript, Python, PHP, Go, Java) with setup instructions. ## Next Steps Now that you can upload files to the Data Engine, explore: 1. **[Search API](/api-guides/search)** - Query your uploaded content 2. **[Chat Integration](/tutorials/chat)** - Use uploaded content in conversations 3. **[Manage Content](/api-guides/manage-content)** - Manage and update your content 4. **[Upload Files API](/api-reference/ingestion/v2)** - For additional API information 5. **[Update Item Metadata API](/api-reference/ingestion/v2)** - For additional API information