Deploying with vLLM

vLLM: The Recommended Inference Engine

vLLM is the most popular Python inference engine for LLMs. Mistral AI recommends it for self-hosting because it’s easy to use, performs well, and offers an OpenAI-compatible API. We’ll walk through installation and configuration, then cover its two usage modes: batch (offline) mode and server mode. These modes aren’t mutually exclusive—many teams use the first to prepare data and the second to serve applications.

What You’ll Need

Downloading models happens through Hugging Face, which requires a token with READ permission:

export HF_TOKEN="hf_your_token_here"

Before your first download, accept the terms of use on the model’s Hugging Face page (for example: huggingface.co/mistralai/Mistral-Nemo-Instruct-2407). This is a one-time, manual step per account. Forgetting it causes download failures that can take an hour to diagnose because they look like authentication problems.

For the Python environment, use a dedicated virtualenv to avoid version conflicts with your other projects:

# Create a virtual environment
python -m venv vllm-env
source vllm-env/bin/activate

# Install vLLM (version >= 0.6.1.post1)
pip install vllm

Then there’s hardware, which determines what you can actually serve. Mistral 7B and Nemo fit on a single 24 GB GPU, typically an A10G, RTX 4090, or L4. Mixtral 8x7B requires two 80 GB GPUs, in practice A100-80GB cards. Mistral Large 2 in its open-weight version, with its 123 billion parameters, needs at least four A100-80GB GPUs. In all cases, install NVIDIA drivers and CUDA 12.x before vLLM.

Offline Mode: Processing Batches

Offline mode handles batches of requests without an HTTP server. You load the model into memory, process your requests, and retrieve results. Note the three arguments tokenizer_mode, load_format, and config_format: we’ll come back to these—they’re not optional.

from vllm import LLM
from vllm.sampling_params import SamplingParams

# Load the model with Mistral flags
llm = LLM(
    model="mistralai/Mistral-Nemo-Instruct-2407",
    tokenizer_mode="mistral",
    load_format="mistral",
    config_format="mistral"
)

# Generation parameters
sampling = SamplingParams(
    max_tokens=8192,
    temperature=0.3
)

# Process a batch of requests
messages_batch = [
    [{"role": "user", "content": "What is PagedAttention?"}],
    [{"role": "user", "content": "Explain tensor parallelism."}],
    [{"role": "user", "content": "How does the KV cache work?"}],
]

results = llm.chat(messages=messages_batch[0], sampling_params=sampling)
for output in results:
    print(output.outputs[0].text)

This mode shines for batch document processing—summarization, classification—model evaluation against benchmarks, synthetic data generation, and test or prototyping scripts. Whenever no one is waiting for an immediate response, you benefit from skipping the HTTP layer.

Server Mode: Exposing an API

Server mode launches an HTTP endpoint compatible with the OpenAI API:

vllm serve mistralai/Mistral-Nemo-Instruct-2407 \
    --tokenizer_mode mistral \
    --config_format mistral \
    --load_format mistral \
    --host 0.0.0.0 \
    --port 8000

The server exposes the API on port 8000. The practical consequence is significant: an application written for the OpenAI API switches to your self-hosted model by changing two lines, base_url and api_key. Your libraries, agent frameworks, and testing tools keep working.

from openai import OpenAI

# Connect to the local vLLM server
client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="not-needed"  # No auth by default
)

response = client.chat.completions.create(
    model="mistralai/Mistral-Nemo-Instruct-2407",
    messages=[
        {"role": "system", "content": "You are a DevOps expert."},
        {"role": "user", "content": "How do I monitor a vLLM server in production?"}
    ],
    temperature=0.3,
    max_tokens=1024
)
print(response.choices[0].message.content)

The comment # No auth by default deserves attention: a vLLM server launched this way authenticates no one. On a development machine, that’s convenient. On a network-accessible machine, it’s an open door. A quick command-line test confirms everything responds before moving on:

curl http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "mistralai/Mistral-Nemo-Instruct-2407",
        "messages": [{"role": "user", "content": "Hello, is this working?"}],
        "temperature": 0.3
    }'

Docker Deployment

For reproducible deployment, the official image avoids reinstalling on every machine:

docker run --runtime nvidia --gpus all \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HUGGING_FACE_HUB_TOKEN=${HF_TOKEN}" \
    -p 8000:8000 \
    --ipc=host \
    vllm/vllm-openai:latest \
    --model mistralai/Mistral-Nemo-Instruct-2407 \
    --tokenizer_mode mistral \
    --load_format mistral \
    --config_format mistral

Three options in this command deserve explanation. --runtime nvidia --gpus all gives the container access to NVIDIA GPUs; without it, the model would try running on CPU. The mount -v ~/.cache/huggingface:... shares the model cache with the host, saving you from redownloading tens of gigabytes every time the container restarts. --ipc=host grants the shared memory needed for inter-process communication between GPUs; omitting it causes hard-to-diagnose crashes as soon as you move to multiple cards.

The Three Essential Mistral Flags

Without these flags, the model may run but with degraded performance or incorrect results. This is the most common self-hosting Mistral pitfall: nothing crashes, the server starts, responses arrive—they’re just worse than they should be, and you risk wrongly concluding the model is disappointing.

Tuning Performance

When a model doesn’t fit on a single card, tensor parallelism distributes its layers across multiple GPUs. The value passed corresponds to the number of cards used:

vllm serve mistralai/Mixtral-8X7B-Instruct-v0.1 \
    --tokenizer_mode mistral \
    --config_format mistral \
    --load_format mistral \
    --tensor-parallel-size 2

The second lever concerns the proportion of GPU memory vLLM reserves:

vllm serve mistralai/Mistral-Nemo-Instruct-2407 \
    --tokenizer_mode mistral \
    --config_format mistral \
    --load_format mistral \
    --gpu-memory-utilization 0.9

The default value (0.9) works in most cases. Reduce it if other processes use the GPU—a vision service, a forgotten notebook—otherwise the first process takes all the memory and the second fails to start.

Finally, two calls suffice to verify a server is alive, making them good probes for your orchestrator:

# List available models
curl http://localhost:8000/v1/models

# Check server health
curl http://localhost:8000/health

Key Takeaways

• vLLM exposes an OpenAI-compatible API—you can use the openai SDK to communicate
• The three flags --tokenizer_mode mistral, --config_format mistral, --load_format mistral are mandatory
• Offline mode suits batch processing; server mode suits real-time applications
• Docker simplifies deployment and ensures reproducibility
• Tensor parallelism lets you distribute large models across multiple GPUs