What Is an Embedding: The Intuition

Objectives

• Understand what an embedding is and why it matters
• Grasp the geometric intuition behind word vectors
• Discover the primary use cases

The Problem: Machines Don’t Understand Text

Language models work with numbers, not words. Before a machine can compare, search, or reason about text, you must convert that text into a usable numerical representation. The key question is: which representation?

The Naive Approach: Bag of Words

The simplest method counts the occurrences of each word. It’s straightforward, but it completely ignores meaning. Take “The cat eats the mouse” and “The mouse eats the cat”: both sentences contain exactly the same words in the same proportions, so they produce the exact same representation, even though they describe opposite scenes. The reverse problem exists too: “Automobile” and “Car” refer to the same thing, but because they are two distinct character strings, the bag-of-words approach treats them as unrelated concepts. Word order and synonyms—two pillars of language—escape this method entirely.

Embeddings: Capturing Meaning

An embedding is a dense vector—a list of floating-point numbers—that captures the semantic meaning of a text. Two texts with similar meanings will have vectors that sit close together in space, regardless of the specific words used. Here’s how you generate one:

from openai import OpenAI

client = OpenAI()

# Generate an embedding
response = client.embeddings.create(
    input="Artificial intelligence is transforming the industry",
    model="text-embedding-3-large"
)

vector = response.data[0].embedding
print(f"Vector dimension: {len(vector)}")  # 3072
print(f"First elements: {vector[:5]}")

This vector of 3,072 numbers encodes the meaning of the sentence, with each dimension capturing a different aspect of the semantics. You don’t need to know what dimension 417 represents: what matters is that the overall configuration remains stable and comparable across texts.

The Geometric Intuition

Imagine a three-dimensional space where every word or phrase occupies a point. “King” and “Queen” end up side by side because they share most of their meaning. “Apple” and “Orange” form another neighborhood, far away from the first. “Dog” and “Cat” make up yet another cluster. This spatial organization is what makes comparison possible: measuring the distance between two points is equivalent to measuring the similarity between two meanings.

In reality, modern embeddings use thousands of dimensions—3,072 for text-embedding-3-large—allowing them to capture nuances far finer than our simple three-axis mental model.

The Famous Analogy

Embeddings also capture regular semantic relationships, to the point where you can manipulate them using arithmetic. The classic example looks like this:

vector("King") - vector("Man") + vector("Woman") ≈ vector("Queen")

In other words, subtracting the “male” component and adding the “female” component shifts the point from “King” toward “Queen”. This property shows that abstract concepts like gender, royalty, or size are encoded in a structured way, not scattered at random.

What Are Embeddings Used For?

The first use case is semantic search: instead of looking for exact keywords, you compare the meaning of the query to the meaning of the documents. A visitor who types “How do I feed my puppy?” will find an article titled “Feeding the Young Dog,” even though the two phrases share no common words.

The second is classification: because texts on the same topic naturally cluster together in space, you can automatically sort support tickets, emails, or articles without writing a single manual rule. Recommendation relies on the same mechanism viewed from the other end: to suggest content to a user, you look for vectors closest to what they’ve already viewed.

Finally, there’s RAG (Retrieval-Augmented Generation), which enriches an LLM’s response with relevant documents found via semantic similarity. This is the flagship use case of this course, and it combines all the others: indexing, comparison, and ranking.

What Changed in 2025–2026

OpenAI’s Embeddings API now offers two main models:

These models support dimensionality reduction: you can request a shorter vector, for example 256 dimensions, to save on storage with a controlled loss in quality. A corpus of one million chunks thus drops from several tens of gigabytes to just a few, fundamentally changing the infrastructure requirements.

Summary

• An embedding converts text into a dense numerical vector
• Semantically similar texts have vectors that are close together
• Primary use cases include search, classification, recommendation, and RAG
• OpenAI offers text-embedding-3-large (3,072 dim) and text-embedding-3-small (1,536 dim)