Sizing GPUs for LLM inference

The first question in any self-hosted LLM project is which GPUs to buy or rent, and how many. The usual answer, "it depends", hides a calculation that is simple enough to do on a whiteboard. GPU memory for inference goes to two things: the model weights, which are fixed, and the KV cache, which grows with every concurrent request and every token of context. Everything else is overhead.

Get those two numbers right and most sizing decisions follow.

Weight memory

Weight memory is the parameter count times the bytes per parameter:

Nobody serves in FP32. BF16 is the default most models are released in. FP8 runs natively on Hopper GPUs such as the H100 and on Ada cards such as the L40S. Four-bit formats add a little for scales and zero points, which is why they land slightly above half a byte per parameter.

This table already rules out options. A 70B model in BF16 does not fit on a single 80 GB GPU at all, before a single request is served. It needs two 80 GB GPUs with tensor parallelism, or quantization.

KV cache memory

During generation, the model stores the attention keys and values for every token in every layer, so it does not recompute them for the next token. That store is the KV cache, and it is usually what limits throughput.

Per token, it costs:

bytes per token = 2 × layers × KV heads × head dimension × bytes per value
                  (keys and values)

The number of KV heads matters a lot. Models using grouped-query attention share each key and value head across several query heads, which shrinks the cache. For the Llama 3.1 family, with 8 KV heads and a head dimension of 128, in BF16:

8B:   2 × 32 layers × 8 × 128 × 2 bytes = 128 KiB per token
70B:  2 × 80 layers × 8 × 128 × 2 bytes = 320 KiB per token
 
70B, one sequence of 8,192 tokens:  8,192 × 320 KiB ≈ 2.5 GiB
70B, 32 sequences of 8,192 tokens:  ≈ 80 GiB

That last line is the one that surprises people. Thirty-two concurrent requests at 8K context need more memory for the cache than the whole model needs for its weights in FP8. Read the layer count, KV head count and head dimension from the model's config.json rather than assuming them; older models without grouped-query attention can need four to eight times more cache per token.

Putting it together

Total memory is roughly:

GPU memory ≈ weights + KV cache + activations and runtime overhead
 
KV cache = per-token cost × context length × concurrent sequences

Overhead covers activations, CUDA graphs and the serving framework itself. Reserve a margin for it rather than calculating it; ten percent of the card is a reasonable starting point, then measure.

Working through one case: Llama 3.1 70B on two H100 80 GB GPUs with tensor parallelism, BF16 weights.

• Total memory: 160 GB.
• Weights: 140 GB, split across both cards.
• After a margin for overhead, only a few gigabytes remain for the KV cache.
• At 320 KiB per token, that is on the order of 10,000 tokens in flight across all requests.

That is one long-context request, or a handful of short ones. Moving the weights to FP8 frees about 70 GB, which goes straight to the cache and multiplies concurrency.

The trade-offs

Once weights are fixed, the KV cache budget is shared between three things you control:

• Context length. A model that supports 128K tokens does not have to be served at 128K. Capping the maximum length to what your prompts actually need is the cheapest capacity win available.
• Batch size. More concurrent sequences means higher total throughput, because the GPU does more useful work per weight read. Latency per request rises somewhat as batches grow.
• Cache precision. Storing the KV cache in FP8 halves its size, with a small quality cost that you should measure on your own prompts.

Serving engines with paged attention, such as vLLM, allocate the cache in small blocks as sequences grow rather than reserving the maximum length up front. This is why they fit far more concurrent requests than a naive implementation. It does not change the arithmetic: when the cache is full, new requests wait or older ones are preempted.

vllm serve meta-llama/Llama-3.1-70B-Instruct \
  --tensor-parallel-size 2 \
  --max-model-len 16384 \
  --gpu-memory-utilization 0.90 \
  --kv-cache-dtype fp8 \
  --max-num-seqs 64

At startup, vLLM logs how many KV cache blocks it could allocate and the maximum concurrency that implies for the configured length. Read that line. It is the real capacity of the deployment, and it often differs from the estimate.

Quantization

Quantization reduces weight memory and, because decoding is mostly limited by memory bandwidth, often improves token throughput as well. The options differ:

• FP8 weights on Hopper and Ada GPUs are close to BF16 in quality for most large models and need no calibration beyond what the published checkpoints already include.
• INT8 works on older hardware such as the A100, with similar savings.
• INT4 methods such as AWQ and GPTQ fit a 70B model on a single 48 GB or 80 GB card. Quality loss is more noticeable, especially on smaller models, and on reasoning, code and non-English text.

Evaluate a quantized model on your own tasks before committing. Generic benchmark scores do not tell you whether it still follows your output format or extracts the right fields.

A sizing checklist

1. Write down the target: model, maximum context length actually used, peak concurrent requests and acceptable latency.
2. Compute weight memory for each precision you would accept.
3. Read the model config and compute KV cache per token.
4. Multiply by context length and concurrency to get the cache budget.
5. Add a margin for overhead, then pick the smallest GPU count and type that fits.
6. Deploy, read the engine's reported cache capacity, and load test with realistic prompt and output lengths before buying more.