Master WebSocket events — Grok Voice & Audio

A conversation made of JSON events

Under the hood, a voice exchange with the Voice Agent is just a stream of JSON messages flowing both ways over the WebSocket connection. Every message has a type field that says what it is. Until you know this vocabulary, your client works by luck. Once you do, you know exactly where to step in to cut off a response, display a transcript, or recover from an error.

Events fall into two families, depending on which way they travel. The ones you send drive the session; the ones you receive tell you what’s happening on the server.

What your app sends

Your outgoing messages cover four needs: configuring the session, feeding the audio buffer, managing the conversation history, and controlling response generation.

Event	Purpose
`session.update`	Configures or reconfigures the session (voice, instructions, tools, VAD, audio format)
`input_audio_buffer.append`	Sends a base64-encoded audio chunk, continuously while the user speaks
`input_audio_buffer.commit`	Commits the current buffer contents as the user’s message; only needed in manual mode
`input_audio_buffer.clear`	Empties the audio buffer without processing it
`conversation.item.create`	Adds an item to the conversation: a text message, earlier history, or the result of a function call
`conversation.item.delete`	Deletes a conversation item by its ID
`response.create`	Asks the model to generate a response; automatic in VAD mode, explicit in manual mode
`response.cancel`	Cancels a response that is being generated

Two of these messages deserve a closer look. conversation.item.create is how you inject context: it’s what you use to restore the history of an earlier call at the start of a new session. As for input_audio_buffer.clear, it comes in handy when the user clearly said nothing worth processing (a throat-clear picked up by an overly sensitive mic) and you’d rather start again from a clean buffer than let the model interpret noise.

What the server sends back

The incoming stream is busier, because it gives a running commentary on every processing step.

Event	Meaning
`session.created`	The session is open
`session.updated`	A configuration update has been applied
`conversation.created`	The conversation is set up
`input_audio_buffer.speech_started`	VAD detects that the user has started speaking
`input_audio_buffer.speech_stopped`	VAD detects that the user has stopped speaking
`input_audio_buffer.committed`	The audio buffer has been committed, by VAD or manually
`response.created`	A response is being generated
`response.output_audio.delta`	A chunk of response audio, in base64
`response.output_audio.done`	The response audio is complete
`response.output_audio_transcript.delta`	The text transcript of the audio, streamed
`response.done`	The response is completely finished
`conversation.item.input_audio_transcription.completed`	The transcript of the user’s message is available
`response.text.delta`	Streamed text, for non-voice responses
`response.function_call_arguments.delta`	The arguments of a function call, streamed
`response.function_call_arguments.done`	The function call is complete and ready to run
`error`	An error occurred; most errors are recoverable

That last one needs a word of warning. Receiving error doesn’t mean the session is lost. According to the xAI reference, most errors are recoverable and the session stays open, so a client that closes the connection at the first error breaks a conversation that could have continued. Log it, tell the user if needed, but keep the socket open.

The skeleton of a client

In practice, everything comes down to a switch on msg.type. Here’s the minimal shape of a JavaScript client:

ws.onmessage = (event) => {
  const msg = JSON.parse(event.data);

  switch (msg.type) {
    case "session.created":
      // Send session.update with the configuration
      break;

    case "input_audio_buffer.speech_started":
      // The user is speaking: stop the response audio that is playing
      break;

    case "response.output_audio.delta":
      // Decode the base64 and play the audio
      playAudioChunk(msg.delta);
      break;

    case "response.function_call_arguments.done":
      // Run the function and send back the result
      handleFunctionCall(msg);
      break;

    case "error":
      console.error("Voice Agent error:", msg.error);
      break;
  }
};

Four branches are enough to run an agent: configure when the session opens, react when the user starts speaking, play incoming audio, and run functions. The other events enrich the interface and can wait for a second iteration.

The tricky part: interruptions

Barge-in, when the user starts talking while the agent is still answering, is where a sloppy implementation shows. The server cancels the current response on its own and starts generating a new one from what it just heard. But while the server has stopped sending, your app still holds several seconds of already-received audio in its output buffer.

If you do nothing, the user keeps hearing the sentence they just interrupted while talking over it, and the agent comes across as not listening. So as soon as you receive speech_started, stop the audio that is playing, flush the output buffer so you don’t play stale sound, and let the server process the new input. Those two steps take a few lines of code and make all the difference between an agent you can interrupt and one that talks into the void.

Key takeaways

Client events cover configuration, sending audio, managing the conversation, and controlling responses
Server events report speech detection, audio chunks, transcripts, and function calls
VAD handles interruptions (barge-in) automatically
Most error events are recoverable: don’t close the connection, handle the error
In manual mode, you must call commit and then response.create explicitly