Send audio with input_audio_buffer.append

Stream audio to the server

Once the session is configured, you can start streaming audio to the server. The input_audio_buffer.append event sends base64-encoded chunks of audio.

Event structure

{
  "type": "input_audio_buffer.append",
  "audio": "SGVsbG8gV29ybGQ..."
}

The audio field contains raw audio data encoded in base64. Its format must match the one configured in session.update (by default, 16-bit little-endian PCM at 24000 Hz).

Capture audio in the browser

The browser’s Web Audio API lets you capture the microphone and send the data in real time:

async function startCapture(ws) {
  const stream = await navigator.mediaDevices.getUserMedia({
    audio: {
      sampleRate: 24000,
      channelCount: 1,
      echoCancellation: true,
      noiseSuppression: true
    }
  });

  const audioContext = new AudioContext({ sampleRate: 24000 });
  const source = audioContext.createMediaStreamSource(stream);
  const processor = audioContext.createScriptProcessor(4096, 1, 1);

  processor.onaudioprocess = (event) => {
    const inputData = event.inputBuffer.getChannelData(0);
    const pcm16 = float32ToPCM16(inputData);
    const base64 = arrayBufferToBase64(pcm16.buffer);

    ws.send(JSON.stringify({
      type: "input_audio_buffer.append",
      audio: base64
    }));
  };

  source.connect(processor);
  processor.connect(audioContext.destination);
}

function float32ToPCM16(float32Array) {
  const pcm16 = new Int16Array(float32Array.length);
  for (let i = 0; i < float32Array.length; i++) {
    const s = Math.max(-1, Math.min(1, float32Array[i]));
    pcm16[i] = s < 0 ? s * 0x8000 : s * 0x7FFF;
  }
  return pcm16;
}

function arrayBufferToBase64(buffer) {
  const bytes = new Uint8Array(buffer);
  let binary = "";
  for (let i = 0; i < bytes.byteLength; i++) {
    binary += String.fromCharCode(bytes[i]);
  }
  return btoa(binary);
}

Capture audio in Python

With the pyaudio library, you can capture the microphone and send the chunks:

import pyaudio
import base64
import json

RATE = 24000
CHUNK = 4096
FORMAT = pyaudio.paInt16
CHANNELS = 1

p = pyaudio.PyAudio()
stream = p.open(
    format=FORMAT,
    channels=CHANNELS,
    rate=RATE,
    input=True,
    frames_per_buffer=CHUNK
)

async def send_audio(ws):
    while True:
        data = stream.read(CHUNK, exception_on_overflow=False)
        audio_b64 = base64.b64encode(data).decode("utf-8")
        await ws.send(json.dumps({
            "type": "input_audio_buffer.append",
            "audio": audio_b64
        }))

Chunk size and frequency

Chunk size affects latency and performance:

• Small chunks (1024 samples / ~42 ms): minimal latency, but more WebSocket messages
• Medium chunks (4096 samples / ~170 ms): a good trade-off
• Large chunks (8192 samples / ~341 ms): fewer messages, but noticeable latency

For a natural conversation, send a chunk every 100 to 200 ms. The server’s VAD collects the data in an internal buffer and processes it continuously.

Echo cancellation

If your app plays the response audio through speakers while the microphone is on, you need echo cancellation. Without it, the agent would hear itself speak and could end up in a conversation loop with itself.

Your options:

• Use the Web Audio API’s echoCancellation and noiseSuppression options
• Mute the microphone while the agent is speaking
• Use a headset to isolate input from output

Key takeaways

• input_audio_buffer.append sends base64-encoded audio chunks
• The data format must match the session configuration (16-bit PCM by default)
• Send a chunk every 100 to 200 ms for a good balance between latency and performance
• Browser audio capture uses the Web Audio API with a Float32 to PCM16 conversion
• Handle echo cancellation to avoid audio loops