Text classification with embeddings

Objectives

• Classify text without training a model
• Implement zero-shot and few-shot classification using embeddings
• Compare performance against a lightweight supervised classifier

Zero-shot classification

Semantic search finds documents similar to a query among thousands of candidates. Classification flips this setup without changing the underlying mechanics: your “documents” are now your categories (five or ten of them), and the “query” is the text you want to classify. The label whose embedding is closest becomes the prediction. No training, no annotated data, just a single API call.

Two implementation details in the code below deserve attention. First, the text and the category names go into a single embeddings.create call, which cuts latency by the number of labels. Second, applying softmax to the similarity scores converts raw values (typically ranging from 0.3 to 0.5) into probabilities that sum to 1. You can display these probabilities to a user or compare them against a confidence threshold.

from openai import OpenAI
import numpy as np

client = OpenAI()

def classifier_zero_shot(
    texte: str,
    categories: list[str],
    model: str = "text-embedding-3-large"
) -> dict:
    """Classifies text into categories without training."""
    # Embeddings for categories
    all_inputs = [texte] + categories
    response = client.embeddings.create(
        input=all_inputs,
        model=model
    )
    embs = [d.embedding for d in sorted(response.data, key=lambda x: x.index)]
    texte_emb = np.array(embs[0])
    cat_embs = np.array(embs[1:])

    # Similarity with each category
    scores = cat_embs @ texte_emb

    # Softmax to get probabilities
    exp_scores = np.exp(scores - np.max(scores))
    probas = exp_scores / exp_scores.sum()

    resultats = sorted(
        zip(categories, probas),
        key=lambda x: x[1], reverse=True
    )

    return {
        "prediction": resultats[0][0],
        "confiance": float(resultats[0][1]),
        "scores": {cat: float(p) for cat, p in resultats}
    }

# Usage
texte = "Le cours de l'action a chuté de 15% après l'annonce"
categories = ["Finance", "Sport", "Technologie", "Politique", "Santé"]

resultat = classifier_zero_shot(texte, categories)
print(f"Prédiction : {resultat['prediction']} "
      f"({resultat['confiance']:.1%})")
print("Scores :", resultat["scores"])

In this example, “the stock price dropped 15% after the announcement” unsurprisingly lands in Finance. The approach hits a limit when categories are internal labels whose names reveal nothing about their content. A label like “P1” or “level 2 escalation” carries no exploitable semantic meaning, making the classification essentially random.

Few-shot classification

This is exactly what few-shot classification fixes. Instead of representing a category by its name, you represent it by the centroid of a few real examples. The “bug” category is no longer the word “bug” but the average of embeddings for “the app crashes on startup,” “500 error on the login page,” and “the button doesn’t work on mobile.” Three to five examples per class are usually enough, and the cost remains that of a single API call since all texts are sent together.

def classifier_few_shot(
    texte: str,
    exemples: dict[str, list[str]],
    model: str = "text-embedding-3-large"
) -> dict:
    """Classifies by similarity to labeled examples.

    Args:
        exemples: {"category": ["example1", "example2", ...]}
    """
    # Prepare all texts
    tous_textes = [texte]
    mapping = []
    for cat, txts in exemples.items():
        for t in txts:
            tous_textes.append(t)
            mapping.append(cat)

    # Embeddings in a single call
    response = client.embeddings.create(
        input=tous_textes, model=model
    )
    embs = [d.embedding for d in sorted(response.data, key=lambda x: x.index)]
    texte_emb = np.array(embs[0])

    # Average embeddings per category
    cat_embeddings = {}
    for i, cat in enumerate(mapping):
        emb = np.array(embs[i + 1])
        if cat not in cat_embeddings:
            cat_embeddings[cat] = []
        cat_embeddings[cat].append(emb)

    cat_moyennes = {
        cat: np.mean(embs_list, axis=0)
        for cat, embs_list in cat_embeddings.items()
    }

    # Similarity
    scores = {
        cat: float(np.dot(texte_emb, emb_moy) /
                    (np.linalg.norm(texte_emb) * np.linalg.norm(emb_moy)))
        for cat, emb_moy in cat_moyennes.items()
    }

    prediction = max(scores, key=scores.get)
    return {"prediction": prediction, "scores": scores}

# Usage
exemples = {
    "bug": [
        "L'application crash au démarrage",
        "Erreur 500 sur la page de connexion",
        "Le bouton ne fonctionne pas sur mobile",
    ],
    "feature": [
        "Pourriez-vous ajouter un mode sombre ?",
        "Il serait utile d'exporter en PDF",
        "Suggestion : intégrer un calendrier",
    ],
    "question": [
        "Comment changer mon mot de passe ?",
        "Où trouver les paramètres de notification ?",
        "Quelle est la limite de stockage ?",
    ],
}

texte = "L'export CSV ne contient pas toutes les colonnes"
resultat = classifier_few_shot(texte, exemples)
print(f"Catégorie : {resultat['prediction']}")

The test ticket—“the CSV export is missing columns”—contains neither the word “bug” nor “error,” but it resembles the three malfunction examples provided, so that’s where it lands. Note that averaging embeddings assumes a homogeneous category. If your “question” examples mix billing inquiries with API questions, their centroid will fall into semantic no man’s land. In that case, it’s better to split the category.

Lightweight supervised classifier

When you have several dozen examples per class, averaging becomes a waste of information: it flattens the shape of the point cloud. A linear classifier trained on the same embeddings learns a decision boundary instead. The embedding handles language understanding, while LogisticRegression simply draws a geometric line in that space. This results in training times of a few seconds on a standard CPU, whereas fine-tuning a language model would require a GPU.

Five-fold cross-validation isn’t just for show. It gives you the standard deviation of accuracy, providing a sense of how reliable your estimate is. A score of 0.91 ± 0.02 is usable; 0.91 ± 0.15 signals a dataset that is too small or imbalanced.

from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
import numpy as np

def entrainer_classifieur(
    textes: list[str],
    labels: list[str],
    model: str = "text-embedding-3-large"
) -> LogisticRegression:
    """Trains a classifier on embeddings."""
    # Generate embeddings
    response = client.embeddings.create(
        input=textes, model=model
    )
    X = np.array([d.embedding for d in sorted(response.data, key=lambda x: x.index)])

    # Train a classifier
    clf = LogisticRegression(max_iter=1000)
    
    # Cross-validation
    scores = cross_val_score(clf, X, labels, cv=5)
    print(f"Accuracy (5-fold CV) : {scores.mean():.3f} ± {scores.std():.3f}")

    # Train on the full dataset
    clf.fit(X, labels)
    return clf

def predire(clf, texte: str, model: str = "text-embedding-3-large") -> str:
    """Predicts the category of a text."""
    emb = client.embeddings.create(
        input=texte, model=model
    ).data[0].embedding
    return clf.predict([emb])[0]

Comparing approaches

Read in order, these three lines describe a progression you can follow throughout the product lifecycle. Start with zero-shot on day one, with no data at all. Once you’ve manually corrected twenty or so predictions, recycle them as few-shot examples. When annotation volume exceeds fifty per class, train the supervised classifier. Keep in mind that the “latency” row reminds you that even the supervised model stays fast; local inference adds negligible overhead compared to the embedding API call.

Summary

• Zero-shot classification compares text to labels via embeddings
• Few-shot uses examples to define each category
• A supervised classifier (LogisticRegression) achieves the best performance
• Start with zero-shot, then add examples if accuracy is insufficient