Deploy Devstral Locally with vLLM

Lesson updated November 2, 2026 based on the Using offline models page in the Vibe documentation. Devstral has been removed from the Mistral API, but its weights remain published: Devstral Small 2 and Mistral Small 4 are the two recommended open models for local use. Vibe does not bundle a local model: you connect it to an OpenAI-compatible server via a preset in config.toml, then switch using /model. Source: Using offline models.

Prepare the Deployment

Deploying Devstral locally means running an inference server on your machine that exposes an OpenAI-compatible API. Vibe then connects to it exactly as if it were talking to the Mistral cloud. This interface compatibility makes the switch seamless; your tooling has no reason to know where the responses are coming from. The recommended framework for this role is vLLM, a high-performance inference engine developed by the open-source community.

What Your Hardware Allows

Before installing anything, check your GPU. It determines the usable precision and, consequently, the context window size you will have available.

Important note: The CPU-only configuration is functional but extremely slow for a 24B model. It is suitable only for testing, not for daily use. In other words, it lets you verify that the chain works, but not that you can work with it.

Install vLLM

The software prerequisite is minimal: Python 3.10 or higher, followed by a pip install.

# Python 3.10+ required
python3 --version

# Install vLLM
pip install vllm

Next, verify that the NVIDIA drivers and CUDA are installed and operational. This command is the best end-to-end test: if it displays your GPU and a CUDA version, the low-level layer is healthy, and any subsequent errors will come from elsewhere.

nvidia-smi
# Must display your GPU and the CUDA version

Start the Server

The startup command fits in four arguments, two of which are essential for working with Vibe.

vllm serve mistralai/Devstral-Small-2-24B-Instruct-2512 \
  --tool-call-parser mistral \
  --enable-auto-tool-choice \
  --port 8080

Let’s break down the flags:

• mistralai/Devstral-Small-2-24B-Instruct-2512: The Hugging Face model to load
• --tool-call-parser mistral: Enables parsing of tool calls in Mistral format
• --enable-auto-tool-choice: Allows the model to automatically choose its tools
• --port 8080: The server’s listening port

The two flags related to tools deserve attention. Without them, the server would answer your questions, but the model could no longer read a file or run a command. Vibe would become a simple chatbot, which is not the goal.

Adapt the Command to Your Machine

If you have multiple GPUs, tensor parallelism distributes the model across all cards. The combined VRAM then allows for a wider context window.

vllm serve mistralai/Devstral-Small-2-24B-Instruct-2512 \
  --tool-call-parser mistral \
  --enable-auto-tool-choice \
  --tensor-parallel-size 2 \
  --port 8080

On a more modest configuration, you need to lower the precision. An H100 or A100 handles FP8; an RTX 4090 will use 4-bit quantization with AWQ, at the cost of a slight loss in quality that is largely compensated by the fact that the model finally fits in memory.

# FP8 on H100/A100
vllm serve mistralai/Devstral-Small-2-24B-Instruct-2512 \
  --tool-call-parser mistral \
  --enable-auto-tool-choice \
  --dtype float16 \
  --port 8080

# 4-bit on RTX 4090
vllm serve mistralai/Devstral-Small-2-24B-Instruct-2512 \
  --tool-call-parser mistral \
  --enable-auto-tool-choice \
  --quantization awq \
  --port 8080

The last lever when VRAM remains insufficient is to reduce the maximum context length. The attention cache consumes memory proportionally to the context, and dropping to 16,384 tokens often frees up what was missing to start.

vllm serve mistralai/Devstral-Small-2-24B-Instruct-2512 \
  --tool-call-parser mistral \
  --enable-auto-tool-choice \
  --max-model-len 16384 \
  --port 8080

Connect Vibe and Verify

The server is running; now you need to tell Vibe to use it. Vibe does not bundle a local model: declare a provider and a model preset in config.toml.

# ~/.vibe/config.toml
active_model = "devstral-local"

[[providers]]
name = "local"
api_base = "http://localhost:8080/v1"
api_style = "openai"
backend = "generic"

[[models]]
name = "mistralai/Devstral-Small-2-24B-Instruct-2512"
provider = "local"
alias = "devstral-local"

Keep active_model above the first [[...]] table, otherwise TOML will attach it to that table. Set max_context_length in the preset to indicate your server’s context, which must also return usage in tokens in streaming responses (stream_options.include_usage). Shorter variant: Vibe provides a predefined provider, llamacpp, which points to http://127.0.0.1:8080/v1 without a key; in that case, write provider = "llamacpp" in the preset, without a [[providers]] block. In a session, /model switches between the local model and hosted models.

Test the chain at two levels: first the raw API, which confirms that the model is loaded, then a real request in Vibe to validate that tool calls work.

# Direct API test
curl http://localhost:8080/v1/models

# Test in Vibe (the ask agent requests before each tool)
vibe --agent ask
> List files in the current directory

For daily use, manually restarting the command quickly becomes tedious. A systemd service starts the server at boot and restarts it in case of a crash.

# Create a systemd service
sudo tee /etc/systemd/system/vllm-devstral.service << EOF
[Unit]
Description=vLLM Devstral Server
After=network.target

[Service]
Type=simple
User=$USER
ExecStart=/usr/local/bin/vllm serve mistralai/Devstral-Small-2-24B-Instruct-2512 --tool-call-parser mistral --enable-auto-tool-choice --port 8080
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl enable --now vllm-devstral

The Three Problems You Will Encounter

A CUDA out of memory error means the model does not fit: reduce --max-model-len or switch to 4-bit quantization. Slowness on the very first launch is normal and not a symptom—the initial startup downloads and loads the model, about 14 GB, and subsequent launches benefit from the cache. Finally, if Vibe cannot find the local server, simply check that port 8080 is listening before looking further.

curl http://localhost:8080/health

Key Takeaways

• vLLM is the recommended framework for serving Devstral locally
• The flags --tool-call-parser mistral and --enable-auto-tool-choice are mandatory
• An H100/A100 offers FP8 and 128K context; an RTX 4090 offers 4-bit and 32K
• Reduce --max-model-len if VRAM is insufficient
• Connect Vibe via a provider and model preset in config.toml (or the predefined llamacpp provider), then use /model
• Run vLLM as a systemd service for daily use