Measure and analyze latency

Why latency is your number-one metric

When you deploy an LLM-powered application to production, the latency your users perceive directly determines adoption. A response that takes 8 seconds to appear drives away 70% of users: they close the tab, retry the request, or conclude the tool doesn’t work. No quality improvement can compensate for this loss, because it happens before the user even reads the answer.

The challenge in production is that “it’s slow” isn’t a diagnosis. The same eight-second duration could stem from an oversized system prompt, an overly ambitious reasoning setting, a poorly placed application server, or post-processing that waits for the full response before displaying anything. Each cause requires a different fix, and fixing the wrong one changes nothing. This lesson teaches you to measure, decompose, and diagnose end-to-end latency so you can turn a complaint into an actionable number.

Latency components

Total API call latency breaks down into four segments you need to separate. Network latency is the transit time between your server and the OpenAI API; it’s invisible in your application logs but can exceed 100 milliseconds if your server is far from the endpoint. Time to First Token (TTFT) measures the delay before the first token is generated: this is the segment users feel as “waiting,” since the screen stays blank until it completes. Time Per Output Token (TPOT) then describes the generation speed for each subsequent token; multiplied by the response length, it gives the text scroll duration. Finally, application latency covers everything your own server does before and after the call: fetching documents, assembling the prompt, validating output JSON, and writing to the database.

This decomposition isn’t academic. A team that discovers its TTFT is fine but its application latency adds two seconds won’t switch models: it will examine its context-fetching SQL query.

Measuring with Python

Here’s how to instrument a call to the Responses API to capture each segment:

import time
import openai

client = openai.OpenAI()

def measure_latency(prompt: str, model: str = "gpt-5.6-terra") -> dict:
    """Measure decomposed latency of an API call."""
    metrics = {}

    start_total = time.perf_counter()

    # Stream the call to measure TTFT
    stream = client.responses.create(
        model=model,
        input=prompt,
        stream=True,
    )

    first_token = True
    output_tokens = 0

    for event in stream:
        if event.type == "response.output_text.delta":
            if first_token:
                metrics["ttft"] = time.perf_counter() - start_total
                start_generation = time.perf_counter()
                first_token = False
            output_tokens += 1

    metrics["total_duration"] = time.perf_counter() - start_total
    metrics["output_tokens"] = output_tokens
    if output_tokens > 1:
        metrics["tpot"] = (
            (metrics["total_duration"] - metrics["ttft"]) / (output_tokens - 1)
        )

    return metrics

result = measure_latency("Explain Bayes' theorem in 3 sentences.")
for key, value in result.items():
    print(f"{key}: {value:.4f}" if isinstance(value, float) else f"{key}: {value}")

The key takeaway from this code is using stream=True. Without streaming, you can’t isolate TTFT: the API returns only after the full response, so you measure only a total. The timer starts before the call, stops at the first text delta, and the remaining duration divided by the number of remaining tokens gives TPOT. Run the same prompt against two models or two reasoning settings and, within minutes, you’ll have the quantitative comparison that replaces your team’s guesses.

Diagnosing bottlenecks

A high TTFT, above two seconds, points to three causes. The most common is an overly long system prompt that isn’t cached: the model must ingest several thousand tokens before starting to write, on every call. Next is a high reasoning setting—high, xhigh, or max on GPT-5.6—which deliberately makes the model think before producing any visible output; this isn’t a bug, it’s expected behavior, but it should be a conscious choice, not a setting inherited from a prototype. Finally, there could be an OpenAI-side load spike, which you can’t control: that’s precisely why you need continuous measurement rather than a one-off check, to distinguish it from the other two.

A high TPOT, above 50 ms per token, reads differently. A very high max_output_tokens pushes the model to produce long sequences, and the total duration mechanically increases even if each token arrives at normal speed. Using complex function calling with many declared tools also burdens generation. Concretely, a team that left fifteen tools in its schema when half are no longer used will pay that overhead on every call.

Monitoring tools

Measuring once isn’t enough: you need to track the distribution over time, because the tail of the distribution generates support tickets. For continuous production monitoring, integrate these metrics into your observability stack:

import logging
from dataclasses import dataclass, field

@dataclass
class LatencyMetrics:
    """Aggregates latency metrics over a sliding window."""
    measurements: list[dict] = field(default_factory=list)

    def add(self, measurement: dict) -> None:
        self.measurements.append(measurement)
        if len(self.measurements) > 1000:
            self.measurements = self.measurements[-1000:]

    @property
    def ttft_p50(self) -> float:
        values = sorted(m["ttft"] for m in self.measurements if "ttft" in m)
        if not values:
            return 0.0
        return values[len(values) // 2]

    @property
    def ttft_p99(self) -> float:
        values = sorted(m["ttft"] for m in self.measurements if "ttft" in m)
        if not values:
            return 0.0
        idx = int(len(values) * 0.99)
        return values[min(idx, len(values) - 1)]

    def report(self) -> str:
        return (
            f"TTFT p50={self.ttft_p50:.3f}s p99={self.ttft_p99:.3f}s | "
            f"Samples: {len(self.measurements)}"
        )

The sliding window of one thousand measurements prevents unbounded memory growth while keeping a representative sample of recent traffic. Crucially, this class exposes two percentiles, not an average: a p50 of 700 ms with a p99 of 6 seconds describes a service most users find fast and a minority find unusable. An average would have masked both realities. Hook report() into your metrics tool and set an alert threshold on p99 before touching the rest of the system.

Key takeaways

• Latency decomposes into TTFT + generation (TPOT × tokens) + network
• Always measure with streaming to get real TTFT
• Track p50 and p99 percentiles, not just the average
• Model choice and reasoning settings directly impact latency: GPT-5.6 Terra with low reasoning responds faster than GPT-5.6 Sol with high reasoning