Server-to-client events

The messages you receive

The server sends many events to keep you informed about the state of the conversation, deliver the response audio, and report errors. Understanding these events is essential to building a responsive, reliable app.

Session and conversation events

These events arrive when the connection starts and after every configuration change:

• session.created: the session is open; the event contains the session configuration
• session.updated: the configuration was updated successfully
• conversation.created: the conversation channel is set up

Audio buffer events

These events tell you about the voice activity VAD detects:

• input_audio_buffer.speech_started: VAD detects that the user has started speaking
• input_audio_buffer.speech_stopped: VAD detects the end of speech
• input_audio_buffer.committed: the audio buffer was committed as a message

ws.on("message", (data) => {
  const event = JSON.parse(data);

  switch (event.type) {
    case "input_audio_buffer.speech_started":
      // The user is speaking: show a visual indicator
      showSpeakingIndicator(true);
      break;

    case "input_audio_buffer.speech_stopped":
      // The user is done: hide the indicator
      showSpeakingIndicator(false);
      break;

    case "input_audio_buffer.committed":
      // The audio message is being processed
      showProcessingIndicator(true);
      break;
  }
});

Response audio events

The response audio streams in through two events.

response.output_audio.delta

Each delta contains a base64-encoded chunk of audio. Collect them and play them in order:

{
  "type": "response.output_audio.delta",
  "delta": "SGVsbG8gV29ybGQ...",
  "item_id": "item_resp_001",
  "output_index": 0,
  "content_index": 0
}

case "response.output_audio.delta":
  const audioChunk = base64ToArrayBuffer(event.delta);
  audioPlayer.enqueue(audioChunk);
  break;

response.output_audio.done

Signals that all the response audio has been sent:

{
  "type": "response.output_audio.done",
  "item_id": "item_resp_001"
}

Transcription events

The API can transcribe both the user’s audio and the response audio:

• conversation.item.input_audio_transcription.completed: a transcript of what the user said
• response.output_audio_transcript.delta: a streamed transcript of the agent’s response

case "conversation.item.input_audio_transcription.completed":
  console.log("User said:", event.transcript);
  displayUserMessage(event.transcript);
  break;

case "response.output_audio_transcript.delta":
  appendAgentTranscript(event.delta);
  break;

Text events

If the agent generates text (in addition to or instead of audio), you receive:

• response.text.delta: a streamed fragment of text

{
  "type": "response.text.delta",
  "delta": "Your order has been"
}

Function calling events

When the agent decides to call a function, two events are emitted:

• response.function_call_arguments.delta: the function arguments, streamed
• response.function_call_arguments.done: the complete function call with all its arguments

Lesson 13, on voice function calling, covers these events in detail.

response.done

This event marks the complete end of a response. It contains a summary of the response, including usage statistics:

{
  "type": "response.done",
  "response": {
    "id": "resp_001",
    "status": "completed",
    "output": [...]
  }
}

The error event

This event reports errors. According to the xAI reference, most errors are recoverable and the WebSocket connection stays open:

{
  "type": "error",
  "error": {
    "type": "invalid_request_error",
    "message": "Invalid audio format specified"
  }
}

Your app should log these errors and, depending on how serious they are, show the user a message or try to fix the problem automatically.

Key takeaways

• The server sends events for every step: voice detection, audio, transcription, text, and errors
• Response audio arrives as base64 chunks through response.output_audio.delta
• Transcripts are available for both user input and agent output
• response.done marks the end of a complete response
• Most errors reported through the error event are recoverable: the connection stays open