Host Devstral Locally

Lesson updated November 2, 2026 based on the Using offline models page in the Vibe documentation: Devstral has been removed from the API, but Devstral Small 2 remains, alongside Mistral Small 4, as one of the two recommended open models for local use. Unsourced cost estimates have been removed, and connecting Vibe to a local server now follows the official config.toml syntax. Source: Using offline models.

Why host Devstral locally?

Devstral Small 2 is released with open weights under the Apache 2.0 license (Devstral 2, its 123-billion-parameter big brother, uses a modified MIT license). Although removed from the Mistral API on March 31, 2026, it hasn’t disappeared: you can deploy it on your own servers without sending a single line of code to an external API, and the Vibe documentation still recommends it for this purpose.

The motivations often overlap. Privacy comes first: your code never leaves your infrastructure, cutting short any debates about a provider’s terms of use. Latency follows, since no network round-trip to the Mistral API sits between your keystrokes and the suggestion. The cost structure changes: you pay a fixed price for a GPU instead of usage-based billing, which becomes advantageous as soon as an entire team queries the model all day. Finally, there’s compliance, required by certain regulations and sectors like banking or defense, and availability, since no third-party service can interrupt your work.

Recommended models

The Vibe documentation recommends two open models for local use:

• Devstral Small 2 (mistralai/Devstral-Small-2-24B-Instruct-2512 on Hugging Face): 24 billion parameters, dense, trained for agentic tasks and coding, with the lightest hardware footprint;
• Mistral Small 4: 119 billion parameters in a mixture-of-experts architecture, with 6.5 billion active, a hybrid reasoning and coding model with a 256,000-token context, under the Apache 2.0 license.

Hardware prerequisites

For decent performance on long contexts, the documentation recommends an FP8 deployment with a context of about 128,000 tokens:

These figures apply to Devstral Small 2. Mistral Small 4 requires much more video memory to load its weights: count on at least one 80 GB H100 in FP8, and more for long contexts. Then adjust precision and context length to your hardware.

Deployment with vLLM

vLLM is the inference framework recommended by the Vibe documentation for serving Devstral. Installation requires just one command.

pip install vllm

The server then starts in a single line, with the two options that make the model usable in agent mode.

vllm serve mistralai/Devstral-Small-2-24B-Instruct-2512 \
  --tool-call-parser mistral \
  --enable-auto-tool-choice \
  --port 8080

Three variants cover most real-world situations: splitting the model across multiple GPUs, reducing precision when VRAM is tight, or trimming the context for the same reason.

# Multi-GPU (if you have 2+ GPUs)
vllm serve mistralai/Devstral-Small-2-24B-Instruct-2512 \
  --tool-call-parser mistral \
  --enable-auto-tool-choice \
  --tensor-parallel-size 2 \
  --port 8080

# Reduced precision (for GPUs with less VRAM)
vllm serve mistralai/Devstral-Small-2-24B-Instruct-2512 \
  --tool-call-parser mistral \
  --enable-auto-tool-choice \
  --dtype float16 \
  --port 8080

# Reduced context (to save VRAM)
vllm serve mistralai/Devstral-Small-2-24B-Instruct-2512 \
  --tool-call-parser mistral \
  --enable-auto-tool-choice \
  --max-model-len 32768 \
  --port 8080

Here’s what each flag controls:

• --tool-call-parser mistral — enables parsing of tool calls in Mistral format
• --enable-auto-tool-choice — allows the model to decide when to call a tool
• --tensor-parallel-size — splits the model across multiple GPUs
• --dtype — controls precision (float16, bfloat16, float32)
• --max-model-len — limits context size to save VRAM

The first two deserve special attention: omitting them gives you a server that works perfectly for conversation but fails silently as soon as an agent tries to call a tool. This is the most common cause of a local deployment being judged “disappointing.”

Using the local server

Once vLLM is running, port 8080 exposes an OpenAI-compatible API. The Mistral SDK connects to it by simply changing the server URL; the key is no longer needed.

from mistralai.client import Mistral

# Point to the local server
client = Mistral(
    api_key="dummy",  # No key needed locally
    server_url="http://localhost:8080",
)

response = client.chat.complete(
    model="mistralai/Devstral-Small-2-24B-Instruct-2512",
    messages=[
        {"role": "user", "content": "Analyze this code and suggest improvements."}
    ],
)

The OpenAI SDK works just as well, letting you switch existing code to local Devstral without rewriting it.

from openai import OpenAI

client = OpenAI(
    api_key="dummy",
    base_url="http://localhost:8080/v1",
)

response = client.chat.completions.create(
    model="mistralai/Devstral-Small-2-24B-Instruct-2512",
    messages=[
        {"role": "user", "content": "Explain this code."}
    ],
)

Alternative frameworks

vLLM isn’t the only path: Devstral works with any inference framework that exposes an OpenAI-compatible API. The Vibe documentation cites llama.cpp, suited for low-VRAM setups thanks to its GGUF quantized models, LM Studio, which offers a graphical interface without the command line, and Ollama. Check the exact model name and quantization in each tool before comparing results.

Connecting Vibe to the local server

Vibe doesn’t bundle a local model: once your server is running, you declare a provider and a model preset in config.toml. For an OpenAI-compatible server on port 8080:

active_model = "devstral-local"

[[providers]]
name = "local"
api_base = "http://localhost:8080/v1"
api_style = "openai"
backend = "generic"

[[models]]
name = "mistralai/Devstral-Small-2-24B-Instruct-2512"
provider = "local"
alias = "devstral-local"

Three precautions from the documentation: keep active_model above the first [[...]] array, or TOML will attach it to that array; set max_context_length in the preset to indicate your server’s context; and verify that the server returns token usage in streaming responses (stream_options.include_usage). Vibe also provides a predefined provider, llamacpp, which points to http://127.0.0.1:8080/v1 without a key: in that case, just write provider = "llamacpp" in the preset. The /model command then switches between the local model and hosted models.

For fully offline operation, also disable telemetry and update checks:

enable_telemetry = false
enable_update_checks = false

With these settings and a local provider, the CLI no longer contacts mistral.ai. Skills, MCP servers, and connectors that reach external systems must be disabled separately.

Key takeaways

• Devstral is removed from the API, but Devstral Small 2 (24B, Apache 2.0) remains recommended for local use, along with Mistral Small 4
• vLLM is the recommended framework; --tool-call-parser mistral and --enable-auto-tool-choice are essential for an agent
• H100 or A100 in FP8 with ~128k context; a 24 GB GPU in 4-bit with ~32k
• The server exposes an OpenAI-compatible API, usable with the Mistral SDK or the OpenAI SDK
• Vibe connects to the server via a preset in config.toml (or the llamacpp provider); disable telemetry and updates for offline use