TTFT, TPOT and Tokens per Second: How to Read LLM Inference Benchmarks

When a cloud provider or chip vendor says its platform serves a language model "faster", the claim usually rests on a handful of metrics: time to first token (TTFT), time per output token (TPOT), tokens per second and requests per second. They sound interchangeable, but they measure different things, and NVIDIA's own benchmarking documentation warns that tool implementations vary, so results should only be compared when the definitions line up.

Getting these numbers right matters for both cost and user experience. The industry's main public benchmark, MLPerf Inference from MLCommons, now sets separate latency targets for chat-style and highly interactive workloads. For Llama 2 70B, for example, the interactive category requires a TTFT of 450 milliseconds and a TPOT of 40 milliseconds. This guide explains each metric, how they trade off, and how to read a benchmark without being misled.

Key facts at a glance

• TTFT is the time from submitting a request to receiving the first token. It includes queuing, prefill (processing the prompt) and network time.
• TPOT, or inter-token latency (ITL), is the average gap between tokens after the first one. Some tools treat the two terms as the same; others weight them differently.
• End-to-end latency equals TTFT plus generation time, the full wait for a complete answer.
• System tokens per second rises with more concurrent requests until the hardware saturates, while tokens per second per user falls as load grows.
• MLPerf Inference checks LLM latency at the 99th percentile in its Server and Interactive scenarios, so occasional slow requests count against a result.

Who this guide is for

This explainer is for developers choosing a model API or GPU cloud, platform teams sizing self-hosted inference, and buyers comparing vendor benchmark slides. It is also useful for product managers who need to set realistic response-time targets for an AI feature.

You do not need to run benchmarks yourself to benefit. Knowing which metric a vendor quotes, and under what load, is often enough to spot a comparison that is not apples to apples.

The latency metrics, step by step

Every LLM request has two phases. In the prefill phase, the model reads the whole prompt and builds a key-value (KV) cache, its short-term memory for the request. In the decode phase, it generates output one token at a time.

Time to first token

TTFT measures how long a user waits before anything appears. NVIDIA's documentation notes that longer prompts increase TTFT, because attention uses the full input sequence to build the KV cache before generation begins. Anyscale's documentation adds that when a server is busy, queuing delay also adds to TTFT, and this can vary with autoscaling and request prioritisation.

TTFT is the metric that matters most for chat. It is also why streaming exists: Hugging Face's Text Generation Inference documentation gives the example of a system generating 100 tokens per second. For a 1,000-token answer, a non-streaming user waits 10 seconds for anything, while a streaming user sees the start immediately and half the answer after five seconds.

Time per output token and inter-token latency

Once text starts flowing, TPOT describes how smoothly it arrives. NVIDIA's AIPerf tool defines inter-token latency as end-to-end latency minus TTFT, divided by the number of output tokens minus one, so only the decode phase is measured.

The subtle part is averaging across many requests. Anyscale's documentation for Ray Serve, which runs on the vLLM serving engine, explains that average TPOT weights every request equally while average ITL weights every token equally. Its worked example makes the gap clear.

Same traffic, two very different headline numbers. Always check which average a benchmark reports.

End-to-end latency

End-to-end latency is TTFT plus generation time. NVIDIA notes it includes queuing, batching and network latency. It matters most for jobs where the full answer is needed before anything else can happen, such as summarisation, code generation or batch processing, as Anyscale points out.

Throughput, cost and the trade-off

Throughput tells you how much work a deployment does, which drives cost per token.

• System TPS. NVIDIA defines total tokens per second as total output tokens divided by the time between the first request and the last response. It grows with concurrency until GPU compute is saturated, and can then fall.
• Per-user TPS. Output length divided by end-to-end latency for each request. As concurrency rises, system TPS goes up while per-user TPS goes down.
• Requests per second (RPS). The average number of requests completed successfully each second.

This is the central tension in inference cost. Packing more requests onto a GPU improves cost efficiency but makes each user wait longer. Anyscale notes that batching settings shift the balance: a higher maximum number of sequences raises RPS under load, while a larger token budget per batch can favour token throughput over request completion speed.

Goodput: the metric that joins the two

Raw throughput can hide poor service. Goodput is the percentage of requests that meet all of your service level objectives (SLOs). Anyscale's example SLO is a TTFT under 500 ms, a TPOT under 15 ms and end-to-end latency under 2 seconds. As load increases, throughput may keep rising while goodput drops, which is exactly the point where a cheaper configuration stops being acceptable.

How MLPerf sets the bar

MLPerf Inference is useful because its rules are public and the same for every submitter. MLCommons runs a Closed division, which requires the same model as the reference implementation so hardware can be compared directly, and an Open division, which allows different models or retraining. Results are also labelled Available, Preview or Research, Development and Internal, depending on whether the system can be bought or rented.

The rules define scenarios that mimic real traffic. In the Server and Interactive scenarios, queries arrive following a Poisson distribution, a model of random arrivals, and the latency target applies at the 99th percentile. In the Offline scenario, all samples are sent at once and only throughput is measured.

These targets give buyers a reference. A vendor quoting throughput for an interactive chat product should be able to say whether it held up under interactive-class latency limits, not just offline batch conditions.

How to read a benchmark claim

Use this short checklist before trusting a speed comparison.

1. Ask which metric. Is "fast" TTFT, TPOT, system TPS or per-user TPS? They can move in opposite directions.
2. Ask under what load. Throughput at one request at a time says little about a busy production service.
3. Check the averaging. Per-request TPOT and per-token ITL can differ by a factor of about three on the same traffic.
4. Look for percentiles. A median hides tail latency; MLPerf uses the 99th percentile for server-style tests.
5. Match prompt and output lengths to your workload, since longer prompts raise TTFT and longer outputs grow the KV cache.
6. Prefer goodput against your own SLOs over raw throughput.

NVIDIA's benchmarking guidance states the underlying rule plainly: compare results only when definitions align. That applies equally to vendor slides, independent reviews and your own internal tests.

Conclusion

LLM inference speed is not one number. TTFT shapes how responsive a product feels, TPOT shapes how smoothly answers stream, and throughput shapes what each token costs.

Key takeaways:

• Set SLOs first, for example a TTFT and TPOT limit, then measure goodput against them.
• Benchmark at realistic concurrency, because per-user speed falls as system throughput rises.
• Confirm the definitions behind any TPOT or ITL figure before comparing vendors.
• Use MLPerf's published targets as a sanity check for interactive versus batch claims.
• Stream responses for chat features to cut perceived waiting time.