Add documents to a collection — Collections, RAG & Files

Updated November 2, 2026. This lesson reflects xai-sdk 1.21.0 and xAI documentation: await calls require the asynchronous AsyncClient, indexing status compares against the collections_pb2.DOCUMENT_STATUS_PROCESSED constant (not a string), and bulk ingestion uses upload_document, which combines both steps into one call. Sources: Collections via the API · xai-sdk-python.

Why adding happens in two steps

You might expect to drop a PDF directly into a collection, like dragging a file into a folder. That’s not how the xAI API works: the document first goes through the Files API, which stores it and assigns a file_id, then through the Management API, which links that identifier to the collection. Two calls, two keys, two steps.

This separation has a direct and useful consequence: the file exists only once on the storage side, but can be linked to as many collections as needed. The same technical report can feed the product team’s database and the support team’s without being uploaded twice, without consuming quota twice, and without risking divergence between two copies.

Step 1: Upload the file

Nothing new here, this is the endpoint you’ve mastered since the first lesson:

curl -X POST https://api.x.ai/v1/files \
  -H "Authorization: Bearer $XAI_API_KEY" \
  -F "[email protected]" \
  -F "purpose=assistants"

Retrieve the file_id from the response: it is the key for the next step.

Step 2: Attach to the collection

Adding takes the form of a POST to a URL that combines the two identifiers, the collection ID and the file ID, and requires no request body:

curl -X POST https://management-api.x.ai/v1/collections/col_xyz789/documents/file_abc123 \
  -H "Authorization: Bearer $XAI_MANAGEMENT_API_KEY"

Once this call is accepted, xAI starts indexing the content. The document then passes through several processing states before it is actually available for search.

Wait for indexing to complete

This is a common pitfall in the first pipeline: the add request returns success, you immediately follow up with a search, and the collection responds that it finds nothing. The document exists, but it isn’t indexed yet. The status to wait for is DOCUMENT_STATUS_PROCESSED, and the SDK lets you poll it properly:

import asyncio
import os

from xai_sdk import AsyncClient
from xai_sdk.proto import collections_pb2

client = AsyncClient(
    api_key=os.getenv("XAI_API_KEY"),
    management_api_key=os.getenv("XAI_MANAGEMENT_API_KEY"),
)


async def wait_for_indexing(file_id, collection_id):
    doc = await client.collections.get_document(file_id, collection_id)
    while doc.status != collections_pb2.DOCUMENT_STATUS_PROCESSED:
        if doc.status == collections_pb2.DOCUMENT_STATUS_FAILED:
            raise RuntimeError(f"Indexing failed: {doc.error_message}")
        await asyncio.sleep(3)
        doc = await client.collections.get_document(file_id, collection_id)
    print("Document indexed and ready for search")


asyncio.run(wait_for_indexing("file_abc123", "col_xyz789"))

The delay depends directly on the size: a PDF of a few pages is indexed in a few seconds, while a 100 MB document may take several minutes. For overnight ingestion, this loop with a three-second pause is sufficient; in an interface where the user uploads a file and waits, provide visual feedback rather than a frozen screen.

Remove a document without losing it

Detaching a file from a collection is a separate operation from deleting it:

curl -X DELETE \
  https://management-api.x.ai/v1/collections/col_xyz789/documents/file_abc123 \
  -H "Authorization: Bearer $XAI_MANAGEMENT_API_KEY"

The file disappears from this collection’s index but remains available via the Files API, and can be linked elsewhere. This is exactly what you should do when a product sheet becomes obsolete for support while retaining its archive value: you remove it from the operational database without destroying the document.

Populate a collection in bulk

When initializing a database from an entire folder, write the loop once; it will serve you well for a long time:

import asyncio
import os
from pathlib import Path

from xai_sdk import AsyncClient

client = AsyncClient(
    api_key=os.getenv("XAI_API_KEY"),
    management_api_key=os.getenv("XAI_MANAGEMENT_API_KEY"),
)

files = [
    "docs/guide-installation.pdf",
    "docs/reference-api.pdf",
    "docs/faq.md",
    "docs/changelog.txt",
]


async def populate(collection_id):
    for file_path in map(Path, files):
        # Steps 1 and 2 in a single call: upload via Files API, then attach
        await client.collections.upload_document(
            collection_id=collection_id,
            name=file_path.name,
            data=file_path.read_bytes(),
        )
        print(f"Added: {file_path}")


asyncio.run(populate("col_xyz789"))

Notice the mix of formats in the list—PDF, Markdown, and plain text coexist without issue in the same collection, which saves you from standardizing your sources before ingestion.

Two keys, two scopes

This process uses two access identifiers that you shouldn’t confuse: the standard API key (XAI_API_KEY) for the step 1 upload, and the Management key (XAI_MANAGEMENT_API_KEY) for the step 2 attachment. The SDK receives both at client creation, and upload_document uses them one after the other. Verify that the latter has the AddFileToCollection permission: without it, your upload will succeed, your add will fail with a 403 error, and you will accumulate files correctly stored but absent from any collection.

Key takeaways

Adding to a collection happens in two steps: upload then add
A file can belong to multiple collections simultaneously
The document must reach the DOCUMENT_STATUS_PROCESSED status before it is searchable
Removing a document from a collection does not delete the underlying file
Two distinct API keys are required: standard for upload, Management for the collection