Infrastructure and GPUs

Sizing Your GPU Infrastructure

Choosing the right GPU, memory capacity, and orchestration tool is a critical step in self-hosting. An undersized model doesn’t degrade gracefully: it refuses to start, or it crashes under the first real load. We’ll cover the hardware requirements for Mistral models, quantization techniques that reduce memory needs, and two orchestration tools: SkyPilot and Cerebrium.

What Each Model Requires

The following table lists FP16 requirements without quantization—the baseline for any sizing calculation.

You don’t need this table to estimate a model that isn’t listed. In FP16, budget roughly 2 GB of VRAM per billion parameters: for a 7B model, 7 x 2 = 14 GB minimum, plus overhead for the KV cache and inference engine—which is why the table shows 16 GB. This margin explains why a 16 GB card barely handles Mistral 7B, and why it won’t suffice once you serve long contexts to multiple concurrent users.

Quantization: Trading Precision for Memory

Quantization reduces the precision of model weights to lower memory consumption, often with minimal impact on quality. In FP8 (eight bits per weight), memory usage is cut in half compared to FP16, and activation fits with a simple argument:

# Serve a model in FP8 with vLLM
vllm serve mistralai/Mistral-Nemo-Instruct-2407 \
    --tokenizer_mode mistral \
    --config_format mistral \
    --load_format mistral \
    --dtype float8_e4m3fn

The practical consequence is clear: a Mixtral 8x7B that required two A100-80GB cards in FP16 fits on a single card in FP8, cutting your hourly bill in half. INT4 (four bits) goes even further, but the impact on quality becomes noticeable; it’s enabled in TGI via a dedicated argument:

# AWQ quantization with TGI
docker run --gpus all --shm-size 1g \
    -p 8080:80 \
    -e HUGGING_FACE_HUB_TOKEN=$HF_TOKEN \
    ghcr.io/huggingface/text-generation-inference:2.0.3 \
    --model-id mistralai/Mistral-7B-Instruct-v0.3 \
    --quantize awq

In practice, we recommend FP8 for production—the quality loss is negligible. INT4 works for prototyping or memory-constrained environments, typically a demo on a workstation with a consumer-grade GPU.

SkyPilot: Multi-Cloud Orchestration

SkyPilot is an open-source framework for deploying LLMs on any cloud—AWS, GCP, Azure, Lambda Labs, and others. It automatically selects the cheapest GPU and handles provisioning, so you don’t have to manually compare spot prices across three providers before every launch. Mistral provides ready-to-use YAML files whose structure describes the machine, setup, and serve command:

# mistral-7b.yaml
resources:
  accelerators: A10G:1
  cloud: aws

setup: |
  pip install vllm

run: |
  vllm serve mistralai/Mistral-7B-Instruct-v0.3 \
      --tokenizer_mode mistral \
      --config_format mistral \
      --load_format mistral

The full lifecycle of a cluster then comes down to four commands:

# Install SkyPilot
pip install "skypilot[aws,gcp,azure]"

# Launch the cluster
sky launch -c mistral-7b mistral-7b.yaml --region us-east-1

# Check status
sky status

# Stop (to save money)
sky stop mistral-7b

# Delete
sky down mistral-7b

Those last two commands are the ones people forget. An A10G cluster left running over a weekend costs as much as a month of API usage for moderate workloads, which is why knowing the ballpark figures matters:

One final point deserves your attention: SkyPilot creates VMs with ports open by default. In production, restrict network access with Security Groups or Firewall Rules, place a load balancer with authentication in front of the service, and never leave port 8000 exposed to the internet. An unauthenticated vLLM server reachable publicly is a GPU you’re paying for on someone else’s behalf.

Cerebrium: Serverless GPU

Cerebrium takes the opposite approach to SkyPilot: instead of managing machines, you describe a service and pay only for the compute you actually use, with auto-scaling. Getting started takes three commands:

pip install cerebrium
cerebrium login
cerebrium init mistral-vllm

The entire deployment is then described in a cerebrium.toml file:

[cerebrium.deployment]
name = "mistral-nemo"
python_version = "3.11"

[cerebrium.hardware]
gpu = "AMPERE_A10"
cpu = 4
memory = 16

[cerebrium.scaling]
min_replicas = 0
max_replicas = 5
cooldown = 60

[cerebrium.dependencies.pip]
vllm = "latest"

The min_replicas = 0 setting is the key advantage of this approach: when no one calls the service, it costs nothing. The trade-off is that the first request after an idle period must wait for an instance to start. The cerebrium deploy command is all you need to publish; the platform handles GPU provisioning, auto-scaling from zero to N replicas, load balancing, and monitoring.

Handling Production Load

To increase throughput, don’t look for a bigger GPU—multiply instances behind a load balancer, each serving the same model on its own card:

                    ┌─── vLLM instance 1 (GPU A)
Load Balancer ──────┼─── vLLM instance 2 (GPU B)
                    └─── vLLM instance 3 (GPU C)

Five metrics are enough to manage the whole setup: tokens per second measures generation throughput; TTFB (Time To First Byte) gives the latency before the first token, which is what the user perceives; GPU utilization should exceed 80% under load, otherwise you’re paying for idle hardware; monitor GPU memory to anticipate OOM errors; and queue depth, the number of pending requests, signals saturation before users complain. On a single machine, a simple continuous refresh already gives you the essentials:

# Monitor GPU utilization
watch -n 1 nvidia-smi

Key Takeaways

• Budget ~2 GB VRAM per billion parameters in FP16
• FP8 quantization halves memory usage with negligible quality impact
• SkyPilot simplifies multi-cloud deployment with ready-to-use YAML configs
• Cerebrium offers serverless GPU with auto-scaling (pay-per-compute)
• In production, add a load balancer and monitor GPU metrics continuously