What takes VRAM when you run an LLM

"Will this model fit on my GPU?" is usually answered with the size of the model file. That is only the first part: the memory a model needs keeps growing with every token of context and every request served at the same time.

This explainer splits the memory into its parts, with the numbers of real open models and real GPUs.

Step by step

1. The weights

   Every parameter of the model is held in GPU memory. At 16 bits (BF16, the format most open models are published in) that is 2 bytes per parameter, so a 7.6-billion-parameter model needs about 15 GB for its weights alone.

2. Quantization shrinks the weights

   FP8 stores a weight in 8 bits and INT4 in about 4.25 bits with its scales. The weights of the same model drop to a half or less than a third. The embedding and output layers usually stay at 16 bits, which is why the estimate is a range.

3. The KV cache grows with the context

   For every token in the context, each layer keeps a key and a value vector so it does not recompute them: 2 × layers × KV heads × head size × bytes per token. Quantizing the weights does not shrink it. Going from a 4K to a 32K context multiplies it by eight.

4. And with every concurrent request

   A server holds a KV cache for each request it serves at the same time. Sixteen concurrent requests with a long context can take more memory than the weights themselves: this is what limits how many users one GPU serves.

5. Overhead, and whether it fits

   The runtime adds its own memory: the CUDA context, the framework and activations. Add the three parts and compare the total with a GPU's memory. The upper end of the range is the safe one to plan with.

What the diagram shows

The chart is a bar of GPU memory in GB, split into weights, KV cache and overhead, with lines at the memory of three reference GPUs. Below it, each part and the total are listed as ranges, and each GPU is marked as fitting the upper end of the total or not.

The controls choose the model, the quantization (BF16, FP8, INT4), the context length (4K, 8K, 32K tokens) and the number of concurrent requests (1, 4, 16). Quantization changes only the weights; context and concurrent requests change only the KV cache.

Try it in the tool

Open Open WebUI

What this does not cover

Dense models only: mixture-of-experts models need a different model of memory. The KV cache is at 16 bits, and the overhead is a declared assumption, not a measurement; engines that page or quantize the KV cache use less. Throughput is not covered.

Related pages

• Stack Sizer
• Methodology

Sources

• Qwen/Qwen2.5-7B-Instruct: config.json
• Qwen/Qwen2.5-7B-Instruct: model card
• Qwen/Qwen2.5-14B-Instruct: config.json
• Qwen/Qwen2.5-14B-Instruct: model card
• Qwen/Qwen2.5-32B-Instruct: config.json
• Qwen/Qwen2.5-32B-Instruct: model card
• deepseek-ai/DeepSeek-R1-Distill-Llama-70B: config.json
• deepseek-ai/DeepSeek-R1-Distill-Llama-70B: model card
• NVIDIA Ada GPU Architecture, Appendix A: GeForce RTX 4090 Full Specifications
• NVIDIA L40S GPU Specifications
• NVIDIA H100 Tensor Core GPU: Product Specifications