On-the-Fly Voice Cloning

Course updated November 3, 2026 for Voxtral TTS 3, based on the Migrate from Voxtral 2 guide and Mistral’s Text to Speech pages: voxtral-mini-tts-2603, deprecated October 31, 2026 and retired April 30, 2027, is replaced in examples by voxtral-mini-tts-3-0. Voxtral 2 predefined voices (gb_jane_neutral…) remain accepted; Voxtral 3 adds [emotion=…] and [pace=…] tags in text, the expressiveness setting, custom pronunciation (phoneme_hints), and 35 languages. Mistral recommends re-measuring latency after migration. Sources: Migrate from Voxtral 2 · Speech generation.

Clone Without Saving

The previous lesson relied on a saved voice, created beforehand via the Voices API and referenced by its voice_id. Voxtral offers a second path: on-the-fly cloning, where you provide the reference audio directly in the generation request. No object is created in your account: the sample serves only that specific request, then it is discarded.

In practice, you replace voice_id with ref_audio, which expects the audio sample encoded in base64. The rest of the call remains unchanged.

import base64
from pathlib import Path
from mistralai.client import Mistral

client = Mistral(api_key="your-api-key")

# Encode the reference audio
ref_audio_b64 = base64.b64encode(
    Path("reference.mp3").read_bytes()
).decode()

# Generate with on-the-fly cloning
response = client.audio.speech.complete(
    model="voxtral-mini-tts-3-0",
    input="This speech will resemble the reference voice.",
    ref_audio=ref_audio_b64,
    response_format="wav",
)

audio_bytes = base64.b64decode(response.audio_data)
Path("cloned-output.wav").write_bytes(audio_bytes)
print("Audio generated with on-the-fly cloning!")

Three Scenarios Where This Approach Shines

The first is prototyping. When you are searching for the right reference sample, you might try ten, fifteen, or even more. Creating that many voices in your account means you will later have to find and delete them one by one. With ref_audio, your test loop leaves no trace on the server, allowing you to compare renders simply by listening to the produced files.

# Quickly test multiple samples
samples = ["sample1.mp3", "sample2.mp3", "sample3.mp3"]

for file in samples:
    ref_b64 = base64.b64encode(Path(file).read_bytes()).decode()
    response = client.audio.speech.complete(
        model="voxtral-mini-tts-3-0",
        input="This is a voice quality test.",
        ref_audio=ref_b64,
        response_format="mp3",
    )
    output = f"test-{Path(file).stem}.mp3"
    Path(output).write_bytes(base64.b64decode(response.audio_data))
    print(f"Generated: {output}")

The second scenario is occasional use. A personalized birthday message using a loved one’s voice, or a client demo prepared for a specific meeting: if the voice will be used only once, there is no reason to install it permanently in your account.

The third is confidentiality. Reference audio sent via ref_audio is not stored on the server. If you are handling the voice of a patient, a witness, or an executive, this difference is not cosmetic: it changes what you must declare in your processing registry and what you need to erase in the event of a request.

voice_id vs. ref_audio: The Differences

The decision rule boils down to frequency of use. A voice used every day—such as an application narrator or a brand spokesperson—deserves to be created with client.audio.voices.create(): you save the bandwidth of sending a sample with every call, and you gain the processing time of the reference. A voice you are testing or will use only once should go through ref_audio, leaving you with nothing to clean up.

Combining Both Approaches

These two methods are not mutually exclusive; the most effective workflow puts them in sequence. First, find the right sample using ref_audio, listening to each result. Once a candidate satisfies you, reuse that same base64 string—already in memory—to create the final voice, complete with its name, languages, and gender.

# Phase 1: Quick test with ref_audio
ref_b64 = base64.b64encode(Path("my-voice.mp3").read_bytes()).decode()
response = client.audio.speech.complete(
    model="voxtral-mini-tts-3-0",
    input="Quality test.",
    ref_audio=ref_b64,
    response_format="mp3",
)
# Listen to the result...

# Phase 2: Satisfied? Save the voice
voice = client.audio.voices.create(
    name="my-validated-voice",
    sample_audio=ref_b64,
    sample_filename="my-voice.mp3",
    languages=["fr"],
    gender="male",
)
print(f"Voice saved: {voice.id}")

Practice with three recordings of your own voice, made under three different conditions—headset microphone, phone, reverberant room. Generate the same phrase with each via ref_audio, listen, and then save only the winner. In ten minutes, you will learn what distinguishes a good sample from a bad one, without leaving three useless voices behind.

Key Takeaways

• The ref_audio parameter lets you clone a voice without saving it
• Reference audio is base64-encoded and sent directly in the request
• On-the-fly cloning is ideal for testing, occasional use, and confidentiality
• For regular use, prefer a saved voice (lower latency, less bandwidth)
• You can test with ref_audio and then save the voice once satisfied