LLMs and Architecture

Updated November 2, 2026 — Le Chat is now Vibe. As of May 28, 2026, Le Chat is now Vibe (still at chat.mistral.ai): a single agent for work and code, carrying over your conversations, settings, and subscriptions. The legacy Le Chat experience remains accessible in Chat mode; in Work mode, agents are now skills (invoked with @), memories are the knowledge base, and deep search is the Deep Research skill. Plans are now called Mistral Free, Pro, Team, and Enterprise. This lesson now uses the name Vibe; some menus described here may have changed names or locations.

How a Language Model Works

You use a language model every time you ask a question on Vibe. But what actually happens between the moment you press Enter and the moment the answer appears? This lesson gives you the keys to understand the underlying architecture without diving into mathematical details. This isn’t just free general knowledge: understanding the mechanism helps you anticipate what the model does well, what it does poorly, and why.

What Is an LLM?

An LLM (Large Language Model) is a computer program trained on massive amounts of text to learn to predict the next word in a sentence. This principle seems deceptively simple—and yet, applied at scale, it produces surprisingly coherent and useful results. Concretely, when you ask Mistral Small 4 a question, the model proceeds as follows:

1. It transforms your text into tokens
2. It processes these tokens through billions of calculations
3. It predicts the most probable token that should follow
4. It repeats the operation token by token until it completes its response

This explains a phenomenon you observed from your first exchange: the response appears progressively on the screen (streaming) because the model effectively generates its response word by word. What you see scrolling isn’t a display artifact—it’s the generation itself, in real time.

Model Parameters

When we talk about a “7 billion parameter” or “14 billion parameter” model, we refer to the number of adjustable digital connections the model contains. These parameters are the model’s weights. To use an analogy: if the model were a brain, the parameters would be the synapses—the connections between neurons. The more parameters, the more nuances and complex relationships in language the model can capture.

However, be careful not to turn this figure into the sole criterion: a larger model is not automatically better. Efficiency depends just as much on the quality of the training data—clean and diverse data produces better results—the model architecture, meaning how the parameters are organized, the training techniques used to adjust these parameters, and fine-tuning, the post-training refinement for specific tasks. This is precisely what allows Mistral to compete with much larger models: an efficient architecture and high-quality training data.

The Transformer Architecture

All modern LLMs rely on the Transformer architecture, introduced in 2017. Two ideas are enough to grasp the essentials.

The first is the attention mechanism, the heart of the Transformer. It allows the model to “look” at the entire input text to understand relationships between words, even when they are distant in the sentence. Take “The cat that was sleeping on the sofa woke up”: thanks to attention, the model understands that “woke up” refers to the cat, not the sofa. This ability to link distant elements is what distinguishes Transformers from previous generations of language models.

The second idea is the stacking of layers. A Transformer consists of multiple stacked layers, and each refines the understanding of the text: the early layers capture simple grammatical relationships, while the deep layers grasp abstract concepts and complex semantic relationships. Understanding is therefore built through successive refinements, much like a reading that moves from syntax to meaning.

Mixture of Experts (MoE) — Mistral’s Specialty

One of Mistral AI’s major innovations is the use of the Mixture of Experts (MoE) architecture, popularized with Mixtral 8x7B. The principle breaks with the classic approach: instead of a single massive neural network that processes everything, MoE divides the model into several specialized “experts,” and a routing mechanism selects, for each processed token, the most relevant experts.

The benefit is twofold. In terms of efficiency, only a few experts are activated for each token, which significantly reduces computation time. In terms of performance, the model benefits from the total capacity of all experts combined, while only paying the computation cost for those that actually work—each expert can also specialize in a type of content or task. In practice, a 47 billion parameter MoE model that activates only 12 billion per token can offer performance comparable to a dense 47 billion model, while being as fast as a 12 billion model. This explains why Mistral models offer a performance-to-speed ratio often superior to the competition.

Model Weights

Weights are the numerical values that define the model’s behavior—these are the famous parameters, once training is complete. When Mistral releases a model as “Open,” it means these weights are downloadable. And owning the weights changes everything: you can host the model on your own servers, run it without an internet connection, modify or fine-tune it, and integrate it into embedded systems. Without the weights, you depend on the provider’s API. This is the fundamental difference between the Open and Premier models you encountered in previous lessons—you now understand what lies technically behind it.

Two-Phase Training

How is a model like Mistral Small 4 created? In two major steps. The first is pre-training: the model is exposed to massive amounts of text—books, articles, source code, web pages—and learns language structures. This phase requires thousands of GPUs for weeks and costs millions of euros; it gives the model its culture and language mastery.

The second step is fine-tuning and alignment: the pre-trained model is refined to follow instructions, be useful, and avoid producing dangerous content. This phase relies on human-annotated data and techniques like RLHF (Reinforcement Learning from Human Feedback). This is what transforms a raw text predictor into an assistant capable of dialog—the difference between a model that completes phrases and a model that answers you.

Key Takeaways

• An LLM predicts the next token, repeated thousands of times to produce a complete response
• Parameters (weights) are the digital connections that encode the model’s knowledge
• The Transformer architecture and its attention mechanism allow understanding relationships in text
• Mixture of Experts (MoE) is Mistral’s specialty: performance of a large model, speed of a small one
• Models are created in two phases: massive pre-training then targeted fine-tuning
• Open weights allow self-hosting and customization