VAD: automatic speech detection

Updated November 3, 2026, based on the xAI Speech to Speech page and Voice API reference: turn detection is set with turn_detection.type ("server_vad" or null) and, optionally, turn_detection.idle_timeout_ms. Sources: Speech to Speech · Voice API.

How turn detection works

In a voice conversation, the system has to know when the user starts speaking and when they stop. That’s the job of VAD (voice activity detection). The xAI Voice Agent API provides server-side VAD, server_vad: the server analyzes the audio stream, spots the end of each utterance, and triggers the response itself.

Two modes, one field

You choose the mode in the turn_detection object of session.update:

{
  "type": "session.update",
  "session": {
    "turn_detection": { "type": "server_vad" }
  }
}

In server_vad mode, the server splits the turns, commits the audio, and starts the response. Interruptions are handled automatically too: xAI recommends this mode for natural turn-taking, when the user cuts the agent off mid-sentence. The documentation doesn’t offer any sensitivity or silence-duration setting, though: splitting turns is the server’s business.

VAD events

With server_vad, the server reports every transition:

• input_audio_buffer.speech_started: VAD detected the start of speech;
• input_audio_buffer.speech_stopped: it detected the end;
• input_audio_buffer.committed: the audio was committed as the user’s message.

ws.on("message", (data) => {
  const event = JSON.parse(data);
  switch (event.type) {
    case "input_audio_buffer.speech_started":
      stopPlayback();       // the user is speaking: stop the agent audio already received
      break;
    case "input_audio_buffer.speech_stopped":
      showThinking();       // end of speech detected: the response is coming
      break;
  }
});

On the server side, interruption is automatic; on the client side, audio you’ve already received would keep playing if you didn’t stop it. speech_started is the right moment to empty your playback queue.

Prompt a silent user: idle_timeout_ms

The only documented setting around VAD, idle_timeout_ms, makes the agent pick the conversation back up when the user goes quiet after a response:

{
  "type": "session.update",
  "session": {
    "turn_detection": { "type": "server_vad", "idle_timeout_ms": 8000 }
  }
}

If no speech is detected for that long after a response ends, the server emits input_audio_buffer.timeout_triggered, commits a silent user turn, and generates a check-in. The timer resets after each response, so it fires again every idle_timeout_ms for as long as the user says nothing. It’s off by default (null). On a phone switchboard, a check-in after a few seconds keeps silences from dragging on; write the follow-up phrase the agent should use into your instructions.

Manual mode: turn_detection.type set to null

For a push-to-talk interface, or whenever you want to decide yourself when a turn ends, set the type to null:

{
  "type": "session.update",
  "session": {
    "turn_detection": { "type": null }
  }
}

You then send audio with input_audio_buffer.append and, when the user releases the button, send input_audio_buffer.commit to commit the turn and response.create to request the response. input_audio_buffer.commit is only available in manual mode. input_audio_buffer.clear discards audio that was sent but not yet committed, and response.cancel stops a response in progress, since interruption is no longer automatic in manual mode.

Choose your mode

• Free-flowing conversation (assistant, phone switchboard, kiosk): server_vad, with idle_timeout_ms to prompt silent users if needed
• Walkie-talkie, dictation, very noisy environments: manual mode, where the end of the turn follows the button and background noise no longer triggers anything

Key takeaways

• turn_detection.type is either "server_vad" (the server splits turns and handles interruptions) or null (manual mode)
• The xAI documentation offers no sensitivity threshold or silence duration to tune
• speech_started, speech_stopped, and committed report VAD transitions
• idle_timeout_ms prompts a silent user, with the input_audio_buffer.timeout_triggered event
• In manual mode: append, then commit and response.create