Audio formats and optimization

Choose the right format for each context

Audio format is one of those settings you rush through at the start and regret later. Yet it determines perceived quality, the bandwidth every listen consumes, and compatibility with your target platform. A podcast delivered as raw PCM blows up your distribution costs, and a phone stream at 24,000 Hz gets transcoded along the way, with all the degradation that comes with it. This lesson reviews the available formats and how to decide between them.

Supported codecs

MP3 covers most consumer distribution. WAV is for audio that still needs editing or must be kept lossless. PCM is the exchange format of real-time pipelines, including the Voice Agent, because it costs nothing to decode. Mu-law and A-law exist only for telephony and have no reason to show up anywhere else.

Sample rates

The sample rate sets how fine-grained the signal is: the higher it is, the better high frequencies come through, and the bigger the file.

In practice, three cases settle it. Telephony requires 8,000 Hz, no discussion: that’s what G.711 demands. Web and mobile do fine with the default 24,000 Hz, the best balance between quality and bandwidth for voice. Production work, such as an audiobook or a video soundtrack, justifies 44,100 or 48,000 Hz, because the file will be mixed, normalized, possibly re-encoded, and every step eats away a little quality.

MP3 bit rates

With the MP3 codec, the bit rate sets the compression and therefore the final file size.

Speech only occupies a small part of the spectrum: 128 kbps already reproduces it very well, and going up to 192 kbps makes your catalog half again as heavy without any listener hearing the difference on spoken content. Save that top tier for premium productions or musical passages.

Convert audio in Python

The Voice Agent delivers raw, base64-encoded PCM, and two conversions come up in almost every project. The first turns that stream into a numpy array normalized between -1 and 1, the format most signal-processing libraries expect, and goes the other way to send audio back to the API.

import base64
import numpy as np

def base64_to_audio(base64_audio: str) -> np.ndarray:
    audio_bytes = base64.b64decode(base64_audio)
    audio_int16 = np.frombuffer(audio_bytes, dtype=np.int16)
    return audio_int16.astype(np.float32) / 32768.0

def audio_to_base64(audio_data: np.ndarray) -> str:
    audio_int16 = (audio_data * 32767).astype(np.int16)
    return base64.b64encode(audio_int16.tobytes()).decode("utf-8")

The second wraps PCM in a WAV container. Without that header, an audio player knows neither how many channels nor which sample rate to assume, and it will either refuse the file or play it at the wrong speed.

import wave

def pcm_to_wav(pcm_data: bytes, sample_rate: int = 24000) -> bytes:
    import io
    buffer = io.BytesIO()
    with wave.open(buffer, "wb") as wf:
        wf.setnchannels(1)
        wf.setsampwidth(2)  # 16-bit
        wf.setframerate(sample_rate)
        wf.writeframes(pcm_data)
    return buffer.getvalue()

Optimize for the destination

For a web app, stick with 128 kbps MP3 at 24,000 Hz for TTS and 24,000 Hz PCM for the Voice Agent, which bidirectional streaming requires. Convert your static audio assets, such as jingles and prerecorded messages, to MP3 once and for all instead of serving WAV to every visitor.

For telephony, stay strictly with G.711 at 8,000 Hz and configure the Voice Agent directly in that format: any real-time transcoding adds latency where you have none to spare.

For storage, keep archiving and distribution separate: the master stays in 24,000 Hz WAV, ready to be reworked, and you distribute 128 kbps MP3, which is much lighter. Same content, two files, two purposes.

Manage memory in the browser

During streaming playback, every URL.createObjectURL call reserves memory that the browser doesn’t free on its own. Over a long session, these URLs pile up until the tab slows to a crawl. Revoke the previous one before creating a new one:

let currentBlobUrl = null;

function playAudioChunk(base64Data) {
  // Free the previous URL
  if (currentBlobUrl) {
    URL.revokeObjectURL(currentBlobUrl);
  }

  const audioBuffer = Uint8Array.from(atob(base64Data), c => c.charCodeAt(0));
  const blob = new Blob([audioBuffer], { type: "audio/pcm" });
  currentBlobUrl = URL.createObjectURL(blob);
  // Play the audio...
}

Interpret TTS error codes

400 errors are fixed in your code and won’t go away on their own: they come from an incompatible parameter, typically a sample rate the chosen codec doesn’t accept. 429 and 5xx errors, on the other hand, call for the same response: exponential backoff, with delays that double on each attempt. An immediate retry loop only makes things worse by hammering the quota you just exceeded.

Key takeaways

• Five codecs are available: MP3, WAV, PCM, mu-law, and A-law, each suited to a specific context
• The default sample rate is 24,000 Hz; use 8,000 Hz for telephony and 44,100 Hz or higher for production
• 128 kbps MP3 offers the best quality-to-size trade-off for spoken voice
• Manage Blob URLs in the browser to avoid memory leaks
• Implement exponential backoff to handle 429 (rate limiting) and 5xx errors

Check your understanding

Voice Agent and TTS: voice from end to end, in five questions.

Session, tools, expressiveness, security, formats: the complete voice assistant in this course brings all five together, and yours will follow the same plan.