TensorRT-LLM and TGI

Alternatives to vLLM

While vLLM is the recommended default engine, two alternatives deserve your attention: NVIDIA’s TensorRT-LLM for maximum performance, and Hugging Face’s TGI (Text Generation Inference) for its integrated ecosystem. Both serve the same Mistral models, but with opposing philosophies—one compiles, the other executes. This choice commits your organization well beyond the initial deployment.

TensorRT-LLM

TensorRT-LLM is NVIDIA’s inference optimization framework. It does more than simply load a model: it compiles it into an “engine” optimized specifically for your GPU, delivering the best possible performance on NVIDIA hardware. The workflow therefore consists of two distinct steps: a build phase that produces the TensorRT engine, followed by a serve phase that deploys it via Triton Inference Server. This separation is key to everything that follows: you gain performance at build time, not at runtime.

# Step 1: Clone the TensorRT-LLM repository
git clone https://github.com/NVIDIA/TensorRT-LLM.git
cd TensorRT-LLM

# Step 2: Build the engine (Mistral 7B example)
# Mistral models use the LLaMA example from the repository
python examples/llama/convert_checkpoint.py \
    --model_dir mistralai/Mistral-7B-Instruct-v0.3 \
    --output_dir ./checkpoint \
    --dtype float16

python -m tensorrt_llm.commands.build \
    --checkpoint_dir ./checkpoint \
    --output_dir ./engine \
    --gemm_plugin float16

The gains are measurable: depending on the use case, TensorRT-LLM proves 30% to 50% faster than vLLM. It applies kernel optimizations specific to each GPU architecture—Ampere, Hopper, Blackwell—offers advanced quantization in FP8, INT8, and INT4 with automatic calibration, and handles inflight batching to maximize throughput.

The trade-off is real. Each model must be compiled for your specific GPU, and this compilation takes anywhere from thirty minutes to several hours depending on the model. Changing models requires a new build, making the process painful if you are still testing different options. Furthermore, it only works on NVIDIA hardware: there is no portability to AMD or other platforms. In organizational terms, this means a team iterating on its model every week will spend more time compiling than serving.

Once the engine is built, Triton takes over and exposes the service:

# Launch Triton Inference Server with the engine
docker run --rm --gpus all \
    -v ./engine:/models/mistral/1 \
    -p 8000:8000 \
    nvcr.io/nvidia/tritonserver:latest \
    tritonserver --model-repository=/models

TGI (Text Generation Inference)

TGI is Hugging Face’s inference toolkit. It occupies a middle ground: less optimized than TensorRT-LLM, but simpler to set up and perfectly integrated into the Hugging Face ecosystem from which you likely already pull your weights. Deployment requires just a single Docker command:

docker run --gpus all --shm-size 1g \
    -p 8080:80 \
    -e HUGGING_FACE_HUB_TOKEN=$HF_TOKEN \
    ghcr.io/huggingface/text-generation-inference:2.0.3 \
    --model-id mistralai/Mistral-7B-Instruct-v0.3

The server becomes accessible on port 8080 once the model loads. It comes with built-in quantization (GPTQ, AWQ, bitsandbytes), tensor parallelism to distribute the model across multiple GPUs, token-by-token streaming, and continuous batching, which automatically groups concurrent requests instead of processing them one after another.

In production, a few additional settings help constrain memory consumption:

docker run --gpus all --shm-size 1g \
    -p 8080:80 \
    -e HUGGING_FACE_HUB_TOKEN=$HF_TOKEN \
    ghcr.io/huggingface/text-generation-inference:2.0.3 \
    --model-id mistralai/Mistral-7B-Instruct-v0.3 \
    --max-input-length 4096 \
    --max-total-tokens 8192 \
    --max-batch-prefill-tokens 4096 \
    --quantize awq

--max-input-length sets the maximum prompt length, --max-total-tokens caps the combined prompt and generation, and --quantize enables quantization to reduce the GPU footprint. Tuning these values precisely protects you from an unusually long request that could saturate memory and bring the service down for everyone.

On the client side, TGI exposes an API compatible with both the Mistral client and the OpenAI client, allowing you to reuse existing code:

# Using the OpenAI client
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8080/v1",
    api_key="not-needed"
)

response = client.chat.completions.create(
    model="mistralai/Mistral-7B-Instruct-v0.3",
    messages=[{"role": "user", "content": "Explain continuous batching."}],
    max_tokens=512
)
print(response.choices[0].message.content)

A native API remains available for integrations that do not use an SDK:

# Using curl (TGI native API)
curl http://localhost:8080/generate \
    -H "Content-Type: application/json" \
    -d '{
        "inputs": "Explain continuous batching.",
        "parameters": {"max_new_tokens": 512}
    }'

Comparing the Three Engines

Choosing Between the Three

vLLM remains the default choice, and for good reason: you get up and running quickly, you have an OpenAI-compatible API, and you can switch models without rebuilding anything. A team still in the exploration phase has no interest in imposing anything else on itself.

TensorRT-LLM is justified in a specific scenario: a fixed model served in production at very high throughput, on exclusively NVIDIA hardware, with an MLOps team capable of managing the build pipeline. If any one of these four elements is missing, the 30% to 50% performance gain will be eaten up by the time spent maintaining the compilation.

Finally, TGI is the natural choice if you already live in the Hugging Face ecosystem, want quantization without additional configuration, and are looking for a simple Docker solution that a team without inference specialization can adopt.

Key Takeaways

• TensorRT-LLM offers the best performance but requires a specific build per GPU
• TGI is the Hugging Face toolkit with built-in quantization, tensor parallelism, and batching
• vLLM remains the best compromise between performance and simplicity for most use cases
• All three engines expose an OpenAI-compatible API (directly or via Triton)
• Your choice depends on your volume, hardware, and tolerance for complexity