Build an interactive voice assistant

From prototype to production voice assistant

You know how to configure a Voice Agent session and produce audio with TTS. What’s left is assembling those building blocks into a product that holds up in front of real users. Three architectures come up again and again (the browser, mobile, and the interactive kiosk), and each one imposes technical constraints long before the quality of the model comes into play.

The web assistant in the browser

This is the most common starting point: a mic button in an existing app, the user speaks, and the assistant answers out loud. The technical cycle goes like this:

1. The user clicks the button: you start capturing audio with MediaRecorder or AudioWorklet
2. The captured audio is converted to base64 PCM and sent to the Voice Agent over WebSocket
3. Server-side VAD detects the end of speech and generates a response
4. The response audio chunks arrive through response.output_audio.delta
5. You decode them and play them with AudioContext

The trouble rarely comes from this loop; it comes from the browser itself. Safari requires the AudioContext to be created inside a user event handler. Create it on the first click of the mic button, not when the page loads, or you’ll get a suspended context and total silence. Autoplay policy points the same way: no audio playback starts without a prior user interaction, which rules out a greeting that plays automatically when someone lands on the page. For output formats, stick to PCM or MP3, since browsers can’t play mulaw and alaw natively. And the API key never goes into client code: it stays on your server, and the browser uses an ephemeral token instead (lesson 13).

Capture needs tuning too:

const stream = await navigator.mediaDevices.getUserMedia({
  audio: {
    sampleRate: 24000,
    channelCount: 1,
    echoCancellation: true,
    noiseSuppression: true
  }
});

Echo cancellation and noise suppression aren’t extras. Without them, the mic picks up the assistant’s reply coming out of the speakers, VAD takes it for the user speaking, and the agent keeps interrupting itself. Always test without headphones: that’s the setup where feedback shows up.

The mobile assistant

With React Native or Flutter, the principle stays the same, but the environment gets tougher. Use the native capture APIs (AVAudioEngine on iOS, AudioRecord on Android) rather than a generic abstraction: the quality of the input signal directly affects recognition. Build in automatic reconnection, because the WebSocket drops as soon as the app goes to the background, which happens the moment a notification arrives during a conversation.

Two things often slip through in-house testing. The first is audio routing: speaker or earpiece, the expected behavior differs depending on whether the user holds the phone to their ear or sets it on a table. The second is Bluetooth, whose own latency stacks on top of the network’s and the model’s; a conversation that flows smoothly with the built-in mic sometimes turns choppy with wireless earbuds.

The interactive kiosk

For a kiosk in a store, a hotel front desk, or a museum gallery, the dominant constraint becomes the sound environment. A cardioid, directional mic limits pickup to the person standing in front of the kiosk. Don’t count on a sensitivity setting to filter out the crowd: the Voice Agent’s turn detection only exposes its type and idle_timeout_ms, with no threshold to raise. When the venue is really noisy, such as a train station concourse or a trade show, push-to-talk with a physical button is more reliable than any automatic detection.

Finally, remember to close sessions after two to three minutes of inactivity. Otherwise, a visitor who walks away mid-conversation leaves a session open that consumes your resources and takes up one of your concurrent slots.

Patterns shared by all three architectures

The Voice Agent keeps the conversation history for the whole session. You can prime it before the first word by injecting useful context, such as the signed-in customer’s identity or an order in progress, with conversation.item.create:

{
  "type": "conversation.item.create",
  "item": {
    "type": "message",
    "role": "system",
    "content": [
      {
        "type": "text",
        "text": "The signed-in customer is Sarah Miller, customer number C-4521. She has an order in progress, CMD-8834."
      }
    ]
  }
}

The agent then stops asking users for information your app already has, which shaves a good ten seconds off every conversation.

To have the assistant speak first, insert an initial message and then trigger the response:

{
  "type": "conversation.item.create",
  "item": {
    "type": "message",
    "role": "user",
    "content": [{ "type": "text", "text": "Hello" }]
  }
}

A { "type": "response.create" } sent right after triggers the greeting.

That leaves visual feedback, which usually gets handled last even though it weighs heavily on how the product feels. Show an indicator while the agent is speaking (a sound wave, an animated avatar) and another one while you’re capturing audio. Without that signal, users can’t tell whether they should speak, whether they’re being heard, or whether the app has crashed, and they end up talking over the agent.

Key takeaways

• Three main architectures: web, mobile, and kiosk, each with its own constraints
• Audio in the browser needs care (Safari’s AudioContext, autoplay policy)
• Prefill the conversation context with conversation.item.create to personalize the experience
• A greeting and visual feedback significantly improve the user experience