Latency

Latency is the waiting time from sending a question until the answer comes back. Speed is viewed in two parts. One is the time from sending until the first character appears. The other is the speed after the answer starts flowing, expressed as the number of tokens produced per second.

What Anthropic's guide explains separately are the overall waiting time until the answer is finished and the time until the first token appears. The speed of the flow is commonly viewed separately from these, as the number of tokens per second. When the answer is shown bit by bit as it is generated (streaming), the time to the first character largely decides how fast it feels.

Anthropic says latency changes with the size of the model, the complexity of the prompt, the infrastructure that runs the model, and similar factors. The main factors are as follows.

• Model size (larger models tend to be slower)
• The amount of thinking in a reasoning model (the first character is delayed by the thinking)
• Input and output length (longer documents and longer answers take more time)
• How busy the service is (even the same model varies by time of day)

OpenAI's guide lists principles for making responses faster, such as generating fewer tokens, sending fewer input tokens, and making fewer calls.

In uses where a person waits, such as conversation, the time to the first character matters. In uses that process many documents in bulk, overall throughput and price matter more than the wait for each call.

The pages for choosing by use list models from the highest rating down and narrow them by price and conditions (such as the length a model can read at once and tool use). Speed is part of neither the order nor the filters. When speed matters, send your own questions several times, measure the wait, and compare.