index

Inference performance is a resource-allocation problem across prefill, decode, cache storage, and scheduling. Each phase responds to different changes. A long prompt benefits from faster prefill or reuse of its prefix; a long response benefits from lower decode cost; a busy service benefits from keeping useful work on the accelerator without pushing requests past their deadlines.

Separating these phases turns optimization into a sequence of measurable decisions. The workload determines the objective, the profile identifies the limiting resource, and a configuration sweep finds the operating point. The result is a latency and capacity curve for the actual service.

Latency accounting

A generated response has an initial wait followed by a sequence of token arrivals. For N output tokens:

end-to-end latency = TTFT + sum of the remaining inter-token gaps
mean TPOT = (completion time - first-token time) / (N - 1)

Time to first token, or TTFT, includes queueing, prompt processing, and transport when measured at the client. Time per output token, or TPOT, describes the generation phase. A one-token response has no subsequent token gap, so TPOT is undefined under this formulation.

The arithmetic identifies where a change pays off. A request with 400 ms TTFT and 199 gaps averaging 25 ms takes 5.375 seconds. Halving TTFT saves 200 ms. Reducing each gap by 5 ms saves 995 ms. A one-token classification request has the opposite allocation: nearly all its latency is in the path to the first token.

For agents, this accounting extends across calls. Ten sequential model steps accumulate ten queue waits and ten prefills, as well as tool execution. Per-call improvements matter in proportion to their contribution to the trajectory.

Workload shape selects the optimization

The same model can occupy several rows depending on the application. A document assistant and an autocomplete service may share weights while needing different replica configurations. Prompt and output length distributions, arrival rates, and shared-prefix structure define the workload more precisely than the model name.

Quantization reduces the bytes used for weights and can reduce memory traffic. Its effect on completed work also depends on output quality and response length. A faster token stream that requires more retries may consume more total resources per successful request.

Prefix caching removes repeated prefill

An attention cache records state for the preceding tokens. When another request begins with the same token prefix, that state can be reused. Automatic prefix caching therefore saves computation on shared instructions, documents, or earlier conversation turns.¹

The position of changing content matters. A request-specific timestamp near the beginning breaks the shared prefix before the reusable document. Placing stable instructions and common context first exposes a larger reusable region where the application semantics allow it.

The saving applies to repeated prefill. Every newly generated token still requires decode work. A service dominated by long outputs therefore sees a different gain from a service repeatedly asking short questions about the same long document.

Cache residency connects reuse to capacity. Frequently reused blocks are valuable only while they remain available. A workload with many unrelated long prefixes can evict them, so hit rate depends on the traffic mix and memory budget as well as prompt construction.

Paged storage increases useful cache capacity

The KV cache grows as a sequence generates tokens and shrinks when the request completes. Reserving contiguous space for the maximum possible length wastes memory and makes sharing difficult. PagedAttention addresses this with logical blocks mapped to physical cache storage.²

The block table lets a sequence grow without a single large contiguous allocation. Shared prefixes can refer to the same physical blocks, while later tokens occupy separate blocks. This increases the number of live sequences that fit in a fixed memory budget.

For a conventional full-attention decoder, approximate cache storage is:

KV bytes = 2 × layers × cached tokens
             × KV heads × head dimension × bytes per element

The factor of two accounts for keys and values. Grouped-query attention uses the number of KV heads in this expression, rather than the larger number of query heads. Sliding-window and compressed-cache architectures change the storage model.

For 32 layers, 8 KV heads, head dimension 128, and two-byte elements, each cached token occupies 128 KiB. An 8,192-token sequence needs 1 GiB, and sixteen such sequences need about 16 GiB before weights, temporary buffers, and allocation overhead. Weight quantization and cache quantization are separate choices.

This is why cache management can improve throughput without changing the speed of an individual matrix multiplication. More of the memory holds active work, so more sequences can share the cost of reading weights.

Chunked prefill controls interference

A long prefill and a one-token decode step have different execution shapes. Running the entire prefill at once can delay requests that are already streaming. Chunking divides the prompt into smaller scheduling units so the engine can interleave the two kinds of work.

SARATHI develops this combination of chunked prefills and decode work.³ The chunk size controls the tradeoff. Larger chunks give prompt processing more work per scheduling step. Smaller chunks give ongoing decodes more frequent opportunities to run.

The scheduler’s token budget links TTFT, TPOT, and throughput. A configuration that finishes one large prompt sooner may lengthen the token gaps of every other request. vLLM’s tuning documentation describes the corresponding scheduling and memory controls.⁴ Their useful values depend on the distribution of prompt lengths and the service objective.

Speculation changes the amount of work per target call

Speculative decoding uses a cheaper model to propose several tokens, then has the target model verify them together. Exact speculative sampling includes a correction procedure that preserves the target distribution within numerical limits.⁵

Its gain comes from completing several accepted tokens during a verification pass whose cost is lower than producing those tokens through separate target calls. The draft model adds work, so acceptance rate and draft latency jointly determine the result.

A small draft with moderate acceptance can outperform a larger, more accurate draft if it is substantially cheaper. Longer proposals can increase accepted tokens per pass while also increasing wasted work after rejection. The useful measurements are draft time, verification time, accepted tokens, and end-to-end generation time.

Speculation is especially attractive when the target has spare parallel compute during small-batch decoding. At high utilization, the additional work can interact differently with batching. It belongs in a workload-specific sweep alongside the other serving parameters.

Goodput identifies the operating point

Throughput counts completed work. Goodput counts work that meets the service’s latency and quality requirements. As offered load increases, throughput can continue rising after deadline attainment has begun to fall. The inference SLO guide develops this distinction in more detail.

Consider an illustrative comparison at a fixed arrival rate with a 700 ms TTFT limit:

B has the highest throughput. C delivers the most requests within the stated limits. The objective therefore selects C, subject to the same quality and cost requirements applied to the other configurations.

Client-side timestamps also expose buffering and streaming stalls. Mean TPOT can be acceptable while a response contains a long pause, so the distribution of token gaps complements request-level latency.

Capacity and arrival-rate experiments

A closed-loop generator maintains a fixed number of outstanding requests, replacing each as it completes. It measures throughput and latency at that concurrency. An open-loop generator schedules arrivals independently of completions, revealing how a queue grows when demand continues during a slowdown.

Both experiments are useful. A concurrency sweep locates capacity and saturation; an arrival-rate sweep establishes deadline attainment under production-shaped demand. Cold caches, repeated prefixes, bursts, and mixed prompt lengths exercise different parts of the system.

The request record contains arrival time, first-token time, completion time, token counts, cache status, and errors. Failed and rejected requests remain part of the offered workload. This keeps apparent latency gains from being confused with dropping difficult work.

Communication in a distributed replica

Splitting a model across GPUs reduces local matrix work while adding transfers. Tensor parallelism introduces collectives around partitioned layers; expert parallelism introduces token dispatch and combination. The exposed part of that communication determines the gain from another device.

A larger parallel group that saves 20 ms of compute and adds 30 ms of communication increases latency. Placement, transfer size, and overlap therefore belong in the same experiment as batch size and cache allocation. The ParallelKittens article examines those boundaries at the kernel level.

An inference configuration is a balance among these resources. Phase-level measurements explain why a change works, while goodput identifies where that change becomes useful capacity for the application.

References

Footnotes

1. vLLM, Automatic Prefix Caching. Reuse conditions and limitations. ↩

2. Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention, SOSP 2023. KV-cache allocation and serving experiments. ↩

3. Agrawal et al., SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills, 2023. ↩

4. vLLM, Optimization and Tuning. Scheduling and memory settings are version-dependent; record the version used. ↩

5. Chen et al., Accelerating Large Language Model Decoding with Speculative Sampling, 2023. Distribution-preserving verification and measured decoding results. ↩

Toy stream: five tokens arrive at 120, 140, 160, 180, and 200 milliseconds after the request starts.

Wait until the first token

120 ms

Four remaining decode gaps

80 ms

End-to-end time is 120 + 4 × 20 = 200 ms. There are four decode gaps, not five.

Illustrative timings, not a measured benchmark. The initial wait can include queueing and prefill.

A cache lookup precedes the target model computation it may save.

1. Identify prefix

   Match the token sequence and the runtime-specific cache identity.

2. Reuse valid state

   Recover the compatible cached KV blocks.

3. Process suffix

   Run the uncached input through the target model.

4. Decode output

   Continue autoregressive generation with the complete request state.

Prefix reuse removes matched prefill work; it does not remove all decoding or scheduling cost.