Complete Voice Pipeline

Course updated November 3, 2026 for Voxtral TTS 3, based on the Migrate from Voxtral 2 guide and Mistral’s Text to Speech pages: voxtral-mini-tts-2603, deprecated October 31, 2026 and retired April 30, 2027, is replaced in examples by voxtral-mini-tts-3-0. Voxtral 2 predefined voices (gb_jane_neutral…) remain supported; Voxtral 3 adds [emotion=…] and [pace=…] tags in text, the expressiveness setting, custom pronunciation (phoneme_hints), and 35 languages. Mistral recommends re-measuring latency after migration. Sources: Migrate from Voxtral 2 · Speech generation.

Lesson corrected November 2, 2026 based on Mistral’s audio documentation: the transcription method is client.audio.transcriptions.complete (not create), with the file passed as {"file_name": ..., "content": ...}, and the current model is voxtral-mini-transcribe-3. Source: Transcribe a file.

From Voice to Voice: The STT → LLM → TTS Pipeline

You know how to create voices, generate speech, and manage latency. This final lesson assembles these building blocks into a complete system: the user speaks, the machine listens, understands, reasons, and responds in a natural voice.

The architecture consists of three links. First, speech recognition converts speech to text; Mistral offers its own model, but OpenAI’s Whisper or another solution works just as well. This text then goes to a large language model, which interprets the request and drafts a response—Mistral Large 3, Claude, GPT-5.6, or Llama; the pipeline makes no assumptions about the provider. Finally, the response returns to Voxtral TTS, which renders it as speech sent via streaming.

A Functional Pipeline

The code below chains the three steps together. Each is isolated in its own function, allowing you to swap the STT or LLM model without touching the rest. Note that the conversation history is passed to and enriched by generer_reponse(): this is what gives coherence to successive exchanges.

import base64
from pathlib import Path
from mistralai.client import Mistral

client = Mistral(api_key="your-api-key")

# Configuration
VOICE_ID = "your-voice-id"
LLM_MODEL = "mistral-large-latest"
TTS_MODEL = "voxtral-mini-tts-3-0"

def transcribe_audio(audio_path: str) -> str:
    """Step 1: Convert audio to text (STT)."""
    path = Path(audio_path)
    with path.open("rb") as audio:
        response = client.audio.transcriptions.complete(
            model="voxtral-mini-transcribe-3",
            file={"file_name": path.name, "content": audio},
        )
    return response.text

def generate_response(question: str, history: list) -> str:
    """Step 2: Generate a response with the LLM."""
    history.append({"role": "user", "content": question})

    response = client.chat.complete(
        model=LLM_MODEL,
        messages=history,
    )

    answer = response.choices[0].message.content
    history.append({"role": "assistant", "content": answer})
    return answer

def synthesize_speech(output_file: str, fichier_sortie: str):
    """Step 3: Convert text to speech (TTS)."""
    audio_chunks = []

    with client.audio.speech.complete(
        model=TTS_MODEL,
        input=output_file,
        voice_id=VOICE_ID,
        response_format="pcm",
        stream=True,
    ) as stream:
        for event in stream:
            if event.event == "speech.audio.delta":
                audio_chunks.append(
                    base64.b64decode(event.data.audio_data)
                )

    Path(fichier_sortie).write_bytes(b"".join(audio_chunks))

def pipeline_vocal(audio_path: str):
    """Complete pipeline: STT → LLM → TTS."""
    history = [
        {
            "role": "system",
            "content": (
                "You are a professional voice assistant. "
                "Respond concisely and naturally, "
                "as in an oral conversation."
            ),
        }
    ]

    # Step 1: Transcribe
    print("Transcribing...")
    question = transcribe_audio(audio_path)
    print(f"Question: {question}")

    # Step 2: Reason
    print("Generating response...")
    answer = generate_response(question, history)
    print(f"Response: {answer}")

    # Step 3: Synthesize
    print("Synthesizing speech...")
    synthesize_speech(answer, "answer.pcm")
    print("Pipeline complete — file answer.pcm generated")

# Run the pipeline
pipeline_vocal("question_utilisateur.wav")

The system prompt deserves a word: explicitly requesting concise, spoken responses isn’t just a nicety. An unconstrained LLM willingly produces structured paragraphs with headings, which are disastrous when read aloud—and take even longer to synthesize.

Stop Waiting for Reasoning to Finish

The previous pipeline is sequential: synthesis doesn’t start until the LLM has fully written the response. For a thirty-word answer, this adds several seconds of silence that the user can’t explain. The solution is to overlap steps 2 and 3: the LLM generates in streaming, and as soon as a complete sentence is available—that is, as soon as a token ends with a period, exclamation mark, or question mark—it goes to synthesis while the model continues drafting the rest.

def streaming_pipeline(question_text: str):
    """Optimized pipeline: TTS starts as soon as the first
    LLM tokens arrive."""

    # LLM generates in streaming
    partial_response = ""
    phrases_to_synthesize = []

    llm_stream = client.chat.stream(
        model=LLM_MODEL,
        messages=[
            {"role": "system", "content": "Respond briefly."},
            {"role": "user", "content": question_text},
        ],
    )

    for chunk in llm_stream:
        token = chunk.data.choices[0].delta.content or ""
        partial_response += token

        # As soon as a complete sentence is available
        if token.endswith((".", "!", "?")):
            phrases_to_synthesize.append(partial_response)
            partial_response = ""

    # Synthesize each sentence in streaming
    for phrase in phrases_to_synthesize:
        with client.audio.speech.complete(
            model=TTS_MODEL,
            input=phrase,
            voice_id=VOICE_ID,
            response_format="pcm",
            stream=True,
        ) as stream:
            for event in stream:
                if event.event == "speech.audio.delta":
                    yield base64.b64decode(event.data.audio_data)

The Conversational Loop

A true assistant doesn’t handle an isolated question: it chains turns of speech while keeping the thread. The loop below records, transcribes, checks if the user wants to stop, then responds—reinjecting the same historique at each turn, which grows throughout the exchange.

def voice_assistant():
    """Complete conversational loop."""
    history = [
        {
            "role": "system",
            "content": "You are a concise voice assistant.",
        }
    ]

    print("Voice assistant ready. Speak!")

    while True:
        # Record the question (via microphone)
        audio_question = record_microphone(duration=10)

        # STT → LLM → TTS pipeline
        question = transcribe_audio(audio_question)
        if question.lower() in ("stop", "arrête", "au revoir"):
            synthesize_speech("Goodbye!", "sortie.pcm")
            break

        answer = generate_response(question, history)
        synthesize_speech(answer, "sortie.pcm")
        play_audio("sortie.pcm")

What Separates This Prototype from Production

The duree=10 in the recording illustrates the road ahead: a fixed delay forces a talkative user to cut themselves off and makes someone answering “yes” wait ten seconds. A voice activity detector (VAD) solves this by actually detecting the end of speech. You then need to handle interruptions, since a user will readily cut off an assistant that talks too long: your code must be able to stop synthesis in progress, which streaming makes possible but this prototype doesn’t do.

Three other points condition deployment. The conversation history, already present here, must be bounded and persisted if sessions last. A fallback is essential—if STT or TTS fails, display the text rather than leaving unexplained silence. Finally, instrument each step in production: it’s the only way to know whether that extra second comes from transcription, the LLM, or synthesis.

Next Steps

You now have everything you need to leverage Voxtral TTS in your projects. Several directions open up depending on your constraints: host Voxtral locally if latency or data control is paramount, integrate it into an existing framework like LangChain, LlamaIndex, or Haystack, or build a complete web application with WebRTC for bidirectional audio. You can also explore Mistral’s multimodal models, which combine text, image, and audio, and join the Mistral community on Discord to compare your builds with others.

Key Takeaways

• A complete voice assistant chains three steps: STT → LLM → TTS
• Parallelizing steps (generating TTS while the LLM continues) drastically reduces latency
• PCM format in streaming provides the responsiveness needed for natural conversation
• In production, handle end-of-speech detection, interruptions, and text fallback
• Voxtral TTS integrates into any voice pipeline thanks to the Mistral Python SDK

Test Your Knowledge

Synthesis, voice, latency: final check before the complete pipeline.

1. What does the Voices API allow?

Answer: Manage available voices and create custom ones—the voice becomes a configured asset, reusable across all your generations.

2. What ethical rules govern voice cloning?

Answer: Explicit consent from the cloned person and transparent, legitimate use—cloning a voice without authorization is off the table; technically possible doesn’t mean permitted.

3. How do you control the pronunciation of text?

Answer: By preparing the text: phonetically spell ambiguous terms, handle acronyms and numbers, punctuate for rhythm—the TTS reads what you write.

4. What levers reduce latency?

Answer: Real-time streaming (audio produced as text arrives), adapted audio formats, and segmented text—the voice starts before generation ends.

5. What does a complete voice pipeline look like?

Answer: User transcription (Voxtral) → model processing → response synthesis (TTS), streaming end-to-end: the loop of a voice assistant.

The loop is closed: from incoming voice to outgoing voice—the course’s final pipeline assembles everything you’ve just validated.