Receive and play the response audio

Streaming playback

When the voice agent generates a response, the audio arrives in base64-encoded pieces (deltas). Your app has to decode them, queue them up, and play them smoothly so the user hears one continuous response with no gaps.

Audio player architecture

The principle is simple: each audio delta you receive is decoded and added to a queue. A playback process consumes the queue and sends the data to the system’s audio output:

class AudioStreamPlayer {
  constructor(sampleRate = 24000) {
    this.audioContext = new AudioContext({ sampleRate });
    this.queue = [];
    this.isPlaying = false;
    this.nextPlayTime = 0;
  }

  enqueue(pcm16ArrayBuffer) {
    // Convert PCM16 to Float32 for the Web Audio API
    const pcm16 = new Int16Array(pcm16ArrayBuffer);
    const float32 = new Float32Array(pcm16.length);
    for (let i = 0; i < pcm16.length; i++) {
      float32[i] = pcm16[i] / 32768.0;
    }

    const buffer = this.audioContext.createBuffer(
      1, float32.length, this.audioContext.sampleRate
    );
    buffer.getChannelData(0).set(float32);

    this.queue.push(buffer);
    if (!this.isPlaying) this.playNext();
  }

  playNext() {
    if (this.queue.length === 0) {
      this.isPlaying = false;
      return;
    }

    this.isPlaying = true;
    const buffer = this.queue.shift();
    const source = this.audioContext.createBufferSource();
    source.buffer = buffer;
    source.connect(this.audioContext.destination);

    const now = this.audioContext.currentTime;
    const startTime = Math.max(now, this.nextPlayTime);
    source.start(startTime);
    this.nextPlayTime = startTime + buffer.duration;

    source.onended = () => this.playNext();
  }

  stop() {
    this.queue = [];
    this.isPlaying = false;
    this.nextPlayTime = 0;
  }
}

Wire it to the WebSocket events

Here’s how to connect the audio player to the API’s events:

const player = new AudioStreamPlayer(24000);

ws.on("message", (data) => {
  const event = JSON.parse(data);

  switch (event.type) {
    case "response.output_audio.delta":
      const audioBytes = base64ToArrayBuffer(event.delta);
      player.enqueue(audioBytes);
      break;

    case "response.output_audio.done":
      console.log("Audio response complete");
      break;

    case "input_audio_buffer.speech_started":
      // The user interrupts the agent
      player.stop();
      ws.send(JSON.stringify({ type: "response.cancel" }));
      break;
  }
});

function base64ToArrayBuffer(base64) {
  const binary = atob(base64);
  const bytes = new Uint8Array(binary.length);
  for (let i = 0; i < binary.length; i++) {
    bytes[i] = binary.charCodeAt(i);
  }
  return bytes.buffer;
}

Handling interruptions

In a natural conversation, the user can interrupt the agent by talking over its response. Your app has to handle that:

1. Detect the interruption: the speech_started event arrives while audio is playing
2. Stop playback: empty the audio queue and stop the sound
3. Cancel the response: send response.cancel so the server stops generating
4. Listen: the user has the floor, and the cycle starts again

This “barge-in” mechanism is essential for a fluid conversation. Without it, users would have to wait for the agent to finish talking before they could answer.

Audio player in Python

For a Python app, you can use pyaudio for playback:

import pyaudio
import base64
import asyncio
from collections import deque

class AudioPlayer:
    def __init__(self, rate=24000):
        self.p = pyaudio.PyAudio()
        self.stream = self.p.open(
            format=pyaudio.paInt16,
            channels=1,
            rate=rate,
            output=True
        )
        self.queue = deque()

    def enqueue(self, base64_audio):
        audio_bytes = base64.b64decode(base64_audio)
        self.queue.append(audio_bytes)

    def play_all(self):
        while self.queue:
            chunk = self.queue.popleft()
            self.stream.write(chunk)

    def stop(self):
        self.queue.clear()

Show the transcript alongside

While the audio plays, you can display the response transcript in real time with the response.output_audio_transcript.delta events. Users can then read what the agent is saying, which helps in noisy environments and for accessibility:

let currentTranscript = "";

case "response.output_audio_transcript.delta":
  currentTranscript += event.delta;
  updateTranscriptDisplay(currentTranscript);
  break;

Key takeaways

• Response audio arrives as base64 chunks through response.output_audio.delta
• Use an audio queue to guarantee continuous playback with no gaps
• Handle interruptions (barge-in) by stopping playback and canceling the response
• Show the transcript alongside the audio to improve accessibility
• The browser’s Web Audio API needs a base64 to PCM16 to Float32 conversion