Metadata and Custom Fields

Context the document doesn’t carry

A technical report doesn’t always include the author’s name, the year it was written, or its confidentiality level in the body text. Yet these details make the difference between a search that finds the right document and one that returns three irrelevant ones. The collection metadata system serves exactly this purpose: to associate structured information with each document that search can exploit and you can filter by.

Attaching fields to a document

When adding a document, the fields parameter accepts key-value pairs that you define freely:

await client.collections.upload_document(
    collection_id=collection_id,
    name="technical-report-q1",
    data=pdf_content,
    fields={
        "author": "Marie Dupont",
        "year": "2026",
        "department": "R&D",
        "document_type": "report",
        "confidentiality": "internal"
    }
)

No schema is enforced at this level: field names are yours to choose. This freedom feels comfortable for the first document and becomes a problem by the hundredth, when author, auteur, and Author coexist in the same collection without any filter working.

Setting rules at the collection level

This is where field_definitions, declared on the collection, come in. They turn an oral convention into a constraint verified by the API. Three settings exist:

A definition combines these settings field by field:

{
  "field_definitions": [
    {
      "name": "author",
      "required": true,
      "inject_into_chunk": true
    },
    {
      "name": "document_id",
      "required": true,
      "unique": true
    },
    {
      "name": "year",
      "required": false,
      "inject_into_chunk": true
    }
  ]
}

Read this example as a document policy: every document has an author and an ID, that ID belongs to only one document—which blocks duplicates at ingestion—and the year remains optional to avoid rejecting undated items.

The option that changes relevance

Of these three settings, inject_into_chunk is the only one that affects answer quality, and it deserves a closer look. During indexing, a document is split into fragments—chunks—and enabling this option copies the field value into each of them.

Concretely, with author: "Marie Dupont" and inject_into_chunk: true, every indexed chunk carries this name. A question like “What did Marie Dupont write about network latency?” then stands a chance of hitting the mark, since the name appears in the searched content. Without injection, the same question relies only on the report text, where the author might appear only on the cover page.

So enable injection for context-bearing fields—author, department, year, category—and leave it disabled for administrative fields like an internal ID, upload date, or validation status, which would only add noise.

Three proven schemas

Useful metadata isn’t invented in the abstract: it stems from the questions asked of the database. A technical documentation base organizes around product and version, since a user’s first question is almost always “which version?”:

{
  "fields": {
    "product": "Grok API",
    "version": "v1",
    "type": "reference",
    "language": "en"
  }
}

An HR knowledge base, however, reasons by category and effective date, since the wrong answer there is most often a repealed policy:

{
  "fields": {
    "category": "hr-policy",
    "effective_date": "2026-01-01",
    "department": "all",
    "status": "active"
  }
}

Finally, a legal base structures around parties, contract type, and expiration, allowing you to pull agreements nearing expiration in a single query:

{
  "fields": {
    "contract_type": "vendor",
    "party": "Acme Corp",
    "expiration": "2027-06-30",
    "jurisdiction": "France"
  }
}

Keeping a consistent schema

Consistency is always lost through the same channels, and a few habits suffice to maintain it. Standardize field name formats—snake_case, in English, without mixing languages. Limit their count: five to ten fields cover most needs; beyond that, maintenance costs outweigh the relevance gains. Handle inject_into_chunk with discernment, since every injected field grows the chunks and thus storage costs. Finally, document the expected schema for each collection in a file accessible to the team: once multiple people feed it, that file, not the creator’s memory, guarantees consistency.

Key takeaways

• fields associates key-value metadata with each document
• field_definitions at the collection level define rules: required, unique, inject_into_chunk
• The inject_into_chunk option improves search relevance by adding metadata to the indexed content
• Standardize your metadata schemas to maintain collection consistency