Audio formats: PCM and G.711

Updated November 3, 2026, based on the xAI Speech to Speech page: the API accepts a fourth codec, audio/opus (raw Opus packets, 24 kHz mono), and a second transport mode, binary, which sends audio as binary WebSocket frames instead of base64 inside JSON. Source: Speech to Speech, Audio and Audio Transport.

Supported audio formats

The Voice Agent API supports four audio formats for input and output: PCM, G.711 μ-law, G.711 A-law, and Opus. Which one you choose depends on your use case: a web app, telephony, or professional audio processing.

PCM (Pulse Code Modulation)

PCM is the default and most versatile format. It encodes audio as uncompressed 16-bit signed integers (Linear16) in little-endian order.

Available sample rates

The API supports seven sample rates for PCM:

• 8000 Hz: minimal telephone quality
• 16000 Hz: a good trade-off for voice (wideband)
• 22050 Hz: intermediate quality
• 24000 Hz: the default, recommended for most cases
• 32000 Hz: higher quality
• 44100 Hz: CD quality
• 48000 Hz: studio quality

For a standard voice app, 24000 Hz offers the best balance between quality and bandwidth. Going higher brings no noticeable improvement for speech.

Configuration in session.update

{
  "type": "session.update",
  "session": {
    "audio": {
      "input": {
        "format": {"type": "audio/pcm", "rate": 24000}
      },
      "output": {
        "format": {"type": "audio/pcm", "rate": 24000}
      }
    }
  }
}

G.711: mu-law and A-law

The G.711 codecs are the standards of traditional telephony. They use logarithmic compression to shrink audio data while preserving voice quality within the telephone band (300–3400 Hz).

mu-law (audio/pcmu)

Used mainly in North America and Japan, it’s the native codec of North American phone networks. If you’re integrating the Voice Agent API with a US-based SIP system, audio/pcmu is the natural choice.

A-law (audio/pcma)

Used in Europe and the rest of the world. For an integration with a European phone carrier, go with audio/pcma.

G.711 limitations

Both G.711 codecs are limited to 8000 Hz, which means reduced audio bandwidth. The voice quality is fine for telephony but lower than 24000 Hz PCM. Use them only when your phone infrastructure requires it.

Different input and output formats

The API lets you configure different formats for input and output, which is useful in hybrid architectures:

{
  "type": "session.update",
  "session": {
    "audio": {
      "input": {
        "format": {"type": "audio/pcmu", "rate": 8000}
      },
      "output": {
        "format": {"type": "audio/pcm", "rate": 24000}
      }
    }
  }
}

In this example, you receive G.711 mu-law from a phone system and send high-quality PCM back to a broadcast system.

Opus

The fourth codec, audio/opus, carries raw Opus packets at 24,000 Hz mono, one packet per payload, with no container header. It’s a compressed codec: it cuts bandwidth sharply compared with PCM, which matters on a mobile connection, as long as your client can encode and decode Opus.

Transport: JSON or binary

The codec (format) says how the audio is encoded; the transport (transport) says how those bytes travel over the WebSocket. By default ("json"), audio is base64-encoded inside input_audio_buffer.append and response.output_audio.delta. With "binary", it goes out as binary WebSocket frames, with no base64 and no header; life-cycle events (response.created, response.done, transcripts) stay in JSON.

await ws.send(json.dumps({
    "type": "session.update",
    "session": {
        "audio": {
            "input": {"format": {"type": "audio/pcm", "rate": 24000}, "transport": "binary"},
            "output": {"format": {"type": "audio/pcm", "rate": 24000}, "transport": "binary"},
        }
    },
}))

await ws.send(pcm16_bytes)  # binary frame: raw 16-bit PCM, no base64

async for message in ws:
    if isinstance(message, bytes):
        player.write(message)           # the assistant's audio
    else:
        event = json.loads(message)     # JSON event

The documentation sets two rules. On input, once the format is configured, the server accepts both paths at the same time: input.transport only states your preferred path. On output, audio arrives only through output.transport, and a change mid-session only applies from the next response, so a single utterance never mixes JSON and binary. Binary avoids the base64 overhead, which adds about a third to the data volume.

Key takeaways

• The default format is 24000 Hz PCM, ideal for web apps
• G.711 mu-law (pcmu) is the North American telephony standard
• G.711 A-law (pcma) is the European telephony standard
• G.711 codecs are limited to 8000 Hz
• You can use different formats for input and output
• audio/opus (24 kHz mono) reduces bandwidth; transport: "binary" sends audio as binary frames, without base64
• For voice, 24000 Hz is the best trade-off between quality and bandwidth