Configure a voice session

Updated November 3, 2026, based on the xAI Speech to Speech page and voice prompting guide. Sources: Speech to Speech · Prompting Guide.

The message that sets everything: session.update

Once the WebSocket connection is open, nothing happens yet. The session.update message is what gives the agent its character: the voice it uses, the instructions it follows, how it detects turns, and which tools it may call. Think of it as the session’s birth certificate. Everything else in the conversation follows from it.

{
  "type": "session.update",
  "session": {
    "instructions": "You are a professional voice assistant for an e-commerce company. Keep your answers short and help customers with their orders.",
    "voice": "Ara",
    "turn_detection": {
      "type": "server_vad"
    },
    "audio": {
      "input": { "format": { "type": "audio/pcm", "rate": 24000 } },
      "output": { "format": { "type": "audio/pcm", "rate": 24000 } }
    },
    "tools": [
      { "type": "web_search" }
    ]
  }
}

The server acknowledges it with a session.updated event. You don’t have to send this message only once. You can send it again mid-conversation to change a setting on the fly, for example to switch voices when the call moves from sales to technical support, without ever dropping the connection.

Write instructions for the ear, not the eye

The instructions field plays the role of the system prompt. Use it to set the agent’s personality (formal, casual, technical), the topics it may cover, the constraints on its answers (length, format, language), and the business context it needs: your company name, product range, and internal procedures.

The catch is that what makes a good written prompt often makes a poor voice agent. Left to itself, a model writes ten-line answers. Read aloud, they drag on, and the user tunes out before the end. So ask explicitly for short, direct answers. If your app works in a single language, say which one, or the agent may switch to match the caller’s accent. And tell it what to do when it doesn’t understand: an instruction like “ask a clarifying question instead of guessing” keeps a misheard order number from triggering a pointless search. The xAI voice prompting guide even suggests a five-part structure (role and persona, goal, conversation flow, guardrails and escalation, voice style) and recommends one or two short sentences per turn, unless the caller asks for more detail.

Turn detection

The turn_detection block decides who speaks and when, and the xAI documentation provides only two settings for it. The first is type. In "server_vad" mode, the server detects the end of each utterance on its own, commits the audio, starts the response, and handles interruptions: all you do is send audio with input_audio_buffer.append. The second setting, idle_timeout_ms, is optional. If the user stays silent for that many milliseconds after a response, the server prompts them, and does it again after each new silence of the same length.

{
  "turn_detection": { "type": "server_vad", "idle_timeout_ms": 8000 }
}

There’s no sensitivity threshold or silence duration to tune. The threshold, silence_duration_ms, and prefix_padding_ms parameters you’ll see in examples written for the OpenAI Realtime API don’t appear in the xAI documentation.

If the automatic behavior doesn’t suit you, set the type to null and take control. It’s then up to you to buffer the audio with input_audio_buffer.append, commit the turn explicitly with input_audio_buffer.commit, and trigger generation with response.create. Manual mode is the right choice for a push-to-talk interface, or whenever you want precise control over when the agent speaks.

Choose the right audio format

The audio block sets the input and output formats independently:

{
  "audio": {
    "input": { "format": { "type": "audio/pcm", "rate": 24000 } },
    "output": { "format": { "type": "audio/pcm", "rate": 24000 } }
  }
}

Four types are accepted. If you don’t specify one, you get PCM at 24,000 Hz:

Your delivery channel decides. A web app in a browser does fine with 24,000 Hz PCM, a good balance between perceived quality and bandwidth. A SIP phone integration has no choice: the network carries G.711 at 8,000 Hz, and forcing a higher-quality format on it only adds a pointless conversion.

Give the agent tools

The tools array lists what the agent may call during the conversation, and you can mix different kinds in the same session:

{
  "tools": [
    { "type": "web_search" },
    { "type": "x_search", "allowed_x_handles": ["gaborit_music"] },
    {
      "type": "function",
      "name": "get_order_status",
      "parameters": {
        "type": "object",
        "properties": {
          "order_id": { "type": "string" }
        },
        "required": ["order_id"]
      }
    }
  ]
}

You provide the toolbox; the agent picks the tool. As the conversation unfolds, it decides that a question about the news calls for a web search, while an order number calls for get_order_status. So your job is less about orchestrating calls and more about describing each tool clearly, so the agent makes the right choice.

Key takeaways

• The session.update message configures the voice, instructions, VAD, and tools
• Server-side VAD (server_vad) detects turns and handles interruptions; idle_timeout_ms is the only setting
• Manual mode (turn_detection.type set to null) gives you precise control, for example with push-to-talk
• PCM, G.711, and Opus audio formats cover both web and telephony use cases
• You can change the configuration mid-session without dropping the connection