Interruptions and Turn Management

Updated November 2, 2026 — Audio models. Several models in this course are deprecated: tts-1, tts-1-hd, and gpt-4o-mini-tts will stop working on January 6, 2027 (replacement: gpt-realtime-2.1-mini, which the OpenAI TTS guide states is not supported by /v1/audio/speech: speech synthesis migration moves to the Realtime API); gpt-realtime, gpt-4o-realtime, and legacy audio models on January 20, 2027 (gpt-realtime-2.1, gpt-realtime-2.1-mini, gpt-audio-1.5); whisper-1, gpt-4o-transcribe, and /v1/audio/translations on February 26, 2027 (gpt-transcribe, gpt-live-transcribe, v1/realtime/translations). For full-duplex voice, GPT-Live 1 has been available in the API since September 10, 2026.

Corrected November 3, 2026. session.update events previously placed turn_detection at the session root, a format from the old beta: with the current Realtime API, the setting lives in session.audio.input.turn_detection, and the session carries "type": "realtime". Interruption handling previously sent response.cancel, but with VAD the server cancels the response itself, and the code used an undefined variable. This lesson now covers semantic_vad, create_response, and interrupt_response, and distinguishes between WebSocket and WebRTC. Event shapes verified against openai SDK types 3.42.1. Sources: Voice activity detection · Realtime conversations.

Interruptions and Turn Management

In natural conversation, people interrupt each other, talk over one another, and cut each other off. A voice agent that cannot handle these situations will feel robotic and frustrating. The Realtime API provides two mechanisms for managing turn-taking: automatic Voice Activity Detection (VAD) and manual control.

Voice Activity Detection (VAD)

VAD is enabled by default in a voice conversation session, operating in server_vad mode: the server continuously analyzes incoming audio, segments based on silence, and signals the start and end of user speech. You configure it in session.audio.input.turn_detection:

await ws.send(json.dumps({
    "type": "session.update",
    "session": {
        "type": "realtime",
        "audio": {
            "input": {
                "turn_detection": {
                    "type": "server_vad",
                    "threshold": 0.5,
                    "prefix_padding_ms": 300,
                    "silence_duration_ms": 500,
                    "create_response": True,      # respond immediately when turn ends
                    "interrupt_response": True,   # cut off agent if user speaks
                }
            }
        },
    },
}))

VAD Parameters

A silence_duration_ms that is too short will trigger a response during natural pauses in the user’s speech; too long, and the agent will seem unresponsive. OpenAI does not publish language-specific defaults: start with the documentation example (500 ms) and adjust based on recordings of your users. A higher threshold requires louder sound to trigger, which helps in noisy environments.

Semantic VAD

The second mode, semantic_vad, does not rely on silence but on words: a classifier estimates whether the user has finished their sentence. A sentence left hanging buys more time than a clear statement, and the model interrupts less often. Its eagerness setting can be low (let the user take their time), medium, high (respond as soon as possible), or auto, the default value, which is equivalent to medium:

await ws.send(json.dumps({
    "type": "session.update",
    "session": {
        "type": "realtime",
        "audio": {"input": {"turn_detection": {"type": "semantic_vad", "eagerness": "low"}}},
    },
}))

Handling Interruptions

When the user speaks while the agent is responding, the server emits input_audio_buffer.speech_started. With VAD and interrupt_response enabled, the server cancels the current response: it then sends response.done with status cancelled. You still need to do two things on your end, on a WebSocket connection where you are playing the audio:

1. Stop playback immediately and discard any queued audio, noting how many milliseconds the user actually heard
2. Truncate the agent’s item to that duration using conversation.item.truncate

Do not wait for response.done to do either.

async def handle_interruption(ws, event: dict, player, item_id: str | None, ms_played: int) -> None:
    """Handles user speech while the agent is speaking (WebSocket)."""
    if event["type"] != "input_audio_buffer.speech_started":
        return

    # 1. Stop playback and clear the queued audio buffer
    player.stop_stream()

    # 2. Remove the part the user didn't hear from history
    if item_id:
        await ws.send(json.dumps({
            "type": "conversation.item.truncate",
            "item_id": item_id,
            "content_index": 0,
            "audio_end_ms": ms_played,
        }))

    # 3. Reopen playback for the next response
    player.start_stream()

Calculate ms_played based on what you actually sent to the speaker: in 16-bit mono PCM at 24 kHz, 48,000 bytes equal one second. In WebRTC or SIP, the server manages the output buffer itself, knows what has been played, and truncates automatically: you do not need to send anything.

Why truncate the response?

Without truncation, the conversation history would contain the model’s full response, even though the user only heard part of it. If the user asks, “What was the last thing you said?”, the model would think it had said it all. Truncating aligns the history with what the user actually heard. The documentation notes a limitation: truncation cuts the audio and removes the transcription of the unplayed portion, but it does not provide a word-for-word truncated transcription.

Manual Control Mode

For certain use cases (IVR menus, dictation, quizzes), automatic VAD is not suitable. Manual mode gives you full control:

# Disable VAD
await ws.send(json.dumps({
    "type": "session.update",
    "session": {"type": "realtime", "audio": {"input": {"turn_detection": None}}},
}))

# Button pressed: if agent is still speaking, cancel its response and truncate unplayed audio
await ws.send(json.dumps({"type": "response.cancel"}))

# Button released: audio was sent via input_audio_buffer.append, commit the turn...
await ws.send(json.dumps({"type": "input_audio_buffer.commit"}))

# ...then trigger the response
await ws.send(json.dumps({"type": "response.create"}))

Without VAD, nothing is automatic: this is where response.cancel comes in, when a response is in progress as the user takes back control. Between turns, input_audio_buffer.clear empties the input buffer.

A middle ground exists: keep VAD enabled so it continues to segment turns and emit events, but set create_response and interrupt_response to False. No response is sent without your explicit response.create, giving you time to verify or enrich the input (moderation, document retrieval), at the cost of some latency.

Manual Mode Use Cases

• IVR Menu: Wait for a DTMF tone or specific keyword before responding
• Dictation: Accumulate long audio segments without interruption
• Confirmation: Wait for an explicit “yes” or “no”
• Push-to-talk: User presses a button to speak

State Management Architecture

For robust turn management, maintain a state machine:

from enum import Enum

class ConversationState(Enum):
    IDLE = "idle"                    # Waiting
    USER_SPEAKING = "user_speaking"  # User is speaking
    PROCESSING = "processing"        # Model is processing
    AGENT_SPEAKING = "agent_speaking" # Agent is responding
    INTERRUPTED = "interrupted"       # Interruption in progress

class TurnManager:
    def __init__(self):
        self.state = ConversationState.IDLE
        self.current_response_item_id = None
        self.played_audio_ms = 0

    def on_event(self, event_type: str):
        if event_type == "input_audio_buffer.speech_started":
            if self.state == ConversationState.AGENT_SPEAKING:
                self.state = ConversationState.INTERRUPTED
            else:
                self.state = ConversationState.USER_SPEAKING

        elif event_type == "input_audio_buffer.speech_stopped":
            self.state = ConversationState.PROCESSING

        elif event_type == "response.output_audio.delta":
            self.state = ConversationState.AGENT_SPEAKING

        elif event_type == "response.done":
            self.state = ConversationState.IDLE

Key Takeaways

• VAD (server_vad by default) is configured in session.audio.input.turn_detection
• semantic_vad segments based on meaning; eagerness controls its patience
• With VAD, the server cancels the interrupted response; on WebSocket, stop playback and truncate with conversation.item.truncate (automatic in WebRTC and SIP)
• Manual mode (turn_detection set to None) suits push-to-talk, IVR menus, and dictation
• An explicit state machine prevents inconsistent behavior in edge cases