עב
עב

GPU Planner: which GPU fits your LLM

Pick an open model or type its figures, the weight format, the context and how many requests run at once. You get the GPU memory it needs, which GPUs it fits on and how many of each, a rough throughput and a command to serve it. Every result is an estimate from vendor datasheets and model configs.

Model
Each model's figures come from its official config.json and model card; or enter a model's figures yourself.
Workload
Lower formats need less memory and decode faster, with some loss of quality.
fp8 halves the KV cache; the engine must support it.
Prompt + output one request holds; the KV cache is reserved for all of it.
Concurrent users or requests decoded together; each holds its own KV cache.
Serving
The command below is written for it.
Suggest: the fewest GPUs, then the least memory per GPU, among those it fits on.

Estimate Estimates from datasheet and config figures and the stated assumptions, not measurements: memory is a range, and the throughput is a rough scale, not a benchmark.

Where it runs

Estimate Suggested: 1 × NVIDIA H100 SXM: Fits Even the cautious estimate fits in the usable memory.

Result

GPU memory on one GPU
51.67 GB to 56.43 GB
GPUs
1
Memory per GPU
51.67 GB to 56.43 GB
Usable per GPU
72 GB
Output throughput, all requests
312 tokens/s to 518 tokens/s
Output throughput, one request
39 tokens/s to 65 tokens/s
Weights
32.76 GB to 34.32 GB
KV cache
17.18 GB to 17.18 GB
Runtime overhead
1.73 GB to 4.93 GB
KV cache per token
0.2621 MB
GPUs in the dataset it fits on (up to 8 each)
10
Breakdown
LineValueAssumptions used
Weights32.76 GB to 34.32 GB
  • Bits per weight in fp8 (Assumption)
  • Bits per embedding weight, cautious (Assumption)
KV cache: sequences × context × bytes per token17.18 GB to 17.18 GB
    Runtime overhead1.73 GB to 4.93 GB
    • Bits per weight in fp8 (Assumption)
    • Bits per embedding weight, cautious (Assumption)
    • Runtime overhead per GPU, fixed, optimistic (Assumption)
    • Runtime overhead per GPU, fixed, cautious (Assumption)
    • Runtime overhead, share of weights, optimistic (Assumption)
    • Runtime overhead, share of weights, cautious (Assumption)
    Total on one GPU51.67 GB to 56.43 GB
    • Bits per weight in fp8 (Assumption)
    • Bits per embedding weight, cautious (Assumption)
    • Runtime overhead per GPU, fixed, optimistic (Assumption)
    • Runtime overhead per GPU, fixed, cautious (Assumption)
    • Runtime overhead, share of weights, optimistic (Assumption)
    • Runtime overhead, share of weights, cautious (Assumption)
    Per GPU on 1 × NVIDIA H100 SXM51.67 GB to 56.43 GB
    • Bits per weight in fp8 (Assumption)
    • Bits per embedding weight, cautious (Assumption)
    • Runtime overhead per GPU, fixed, optimistic (Assumption)
    • Runtime overhead per GPU, fixed, cautious (Assumption)
    • Runtime overhead, share of weights, optimistic (Assumption)
    • Runtime overhead, share of weights, cautious (Assumption)
    Usable on one NVIDIA H100 SXM72 GB
    • Share of GPU memory the server may use
    Output throughput, all sequences312 tokens/s to 518 tokens/s
    • Bits per weight in fp8 (Assumption)
    • Bits per embedding weight, cautious (Assumption)
    • Memory bandwidth reached, cautious (Assumption)
    • Memory bandwidth reached, optimistic (Assumption)
    • Compute reached, cautious (Assumption)
    • Compute reached, optimistic (Assumption)
    • Average share of the context filled (Assumption)

    Calculator gpu-planner 1.0.0 · engine @anaroq/calc 1.4.0 · data snapshot 2026.10.4 · How the calculators work

    Assumptions

    Every number the result uses that you didn't enter. Change any of them to match your case.

    • %

      vLLM's default gpu_memory_utilization; the rest is left to the driver and other processes.

      Source ·
    • Bits per weight in fp8Assumption
      8 bits (Fixed definition)

      8 bits a weight; the per-channel or per-block scales add under 0.1 bit.

      No source: this is our own assumption. Change it if you know your case.
    • Bits per embedding weight, cautiousAssumption
      16 bits (Fixed definition)

      Published quantized checkpoints keep the embedding and output matrices at 16 bits; the cautious end counts them so.

      No source: this is our own assumption. Change it if you know your case.
    • Runtime overhead per GPU, fixed, optimisticAssumption
      0.75 GB (Fixed definition)

      CUDA context, the serving framework and activation workspace on each GPU: a fixed part and a part that grows with the weights it holds. A declared range, not a measurement.

      No source: this is our own assumption. Change it if you know your case.
    • Runtime overhead per GPU, fixed, cautiousAssumption
      1.5 GB (Fixed definition)

      CUDA context, the serving framework and activation workspace on each GPU: a fixed part and a part that grows with the weights it holds. A declared range, not a measurement.

      No source: this is our own assumption. Change it if you know your case.
    • Runtime overhead, share of weights, optimisticAssumption
      0.03 (Fixed definition)

      CUDA context, the serving framework and activation workspace on each GPU: a fixed part and a part that grows with the weights it holds. A declared range, not a measurement.

      No source: this is our own assumption. Change it if you know your case.
    • Runtime overhead, share of weights, cautiousAssumption
      0.1 (Fixed definition)

      CUDA context, the serving framework and activation workspace on each GPU: a fixed part and a part that grows with the weights it holds. A declared range, not a measurement.

      No source: this is our own assumption. Change it if you know your case.
    • Memory bandwidth reached, cautiousAssumption
      0.5 (Fixed definition)

      Share of the datasheet memory bandwidth real decode kernels reach: each generated token reads all the weights once.

      No source: this is our own assumption. Change it if you know your case.
    • Memory bandwidth reached, optimisticAssumption
      0.8 (Fixed definition)

      Share of the datasheet memory bandwidth real decode kernels reach: each generated token reads all the weights once.

      No source: this is our own assumption. Change it if you know your case.
    • Compute reached, cautiousAssumption
      0.3 (Fixed definition)

      Share of the peak tensor throughput reached when many sequences make decode compute-bound (2 FLOPs per parameter per token).

      No source: this is our own assumption. Change it if you know your case.
    • Compute reached, optimisticAssumption
      0.5 (Fixed definition)

      Share of the peak tensor throughput reached when many sequences make decode compute-bound (2 FLOPs per parameter per token).

      No source: this is our own assumption. Change it if you know your case.
    • Average share of the context filledAssumption
      0.5 (Fixed definition)

      Each decode step also reads the sequences' KV cache; sequences are taken at half their context on average.

      No source: this is our own assumption. Change it if you know your case.

    What you enter and the assumptions you change are kept in this page's link, after the #.

    Fit on every GPU in the dataset

    Each GPU's memory from its official datasheet, with the day it was read; a figure read more than 180 days ago is marked stale. Status is on one GPU; GPUs needed is the smallest tensor-parallel degree (1, 2, 4 or 8) that fits.

    GPUMemoryOn one GPU (estimate)GPUs needed, cautiousGPUs needed, optimistic
    NVIDIA B200180 GBDatasheet · Fits11
    NVIDIA H200 SXM141 GBDatasheet · Fits11
    NVIDIA H100 SXM80 GBDatasheet · Fits11
    NVIDIA H100 NVL94 GBDatasheet · Fits11
    NVIDIA A100 80GB SXM80 GBDatasheet · Fits11
    NVIDIA A100 80GB PCIe80 GBDatasheet · Fits11
    NVIDIA L40S48 GBDatasheet · Does not fit22
    NVIDIA L424 GBDatasheet · Does not fit44
    NVIDIA GeForce RTX 509032 GBDatasheet · Does not fit42
    NVIDIA GeForce RTX 409024 GBDatasheet · Does not fit44

    Example command

    A starting point with the engine's documented flags for this context, these requests and this format. Check it against the engine's documentation for your version:

    serve.sh
    vllm serve Qwen/Qwen3-32B \
  --tensor-parallel-size 1 \
  --max-model-len 8192 \
  --max-num-seqs 8 \
  --gpu-memory-utilization 0.9 \
  --quantization fp8

      Engine documentation

      Data and versions

      GPU engine @anaroq/gpu 1.0.0 · GPU data 2026.10.0 · model data 2026.10.0

      Take the result with you

      The command, and the result as a Markdown file: what you entered, the memory breakdown, the fit on every GPU, every assumption, and the link that reproduces it.

      gpu-planner.md
      # GPU Planner: which GPU fits your LLM

An estimate from the assumptions listed below, not a measurement or a guarantee.

No source: this is our own assumption. Change it if you know your case.

## What you entered

- Open model: Qwen3 32B
- Weight format: fp8
- KV cache format: bf16 (default)
- Context per request (tokens): 8,192
- Requests at once: 8
- Serving engine: vLLM
- GPU: Suggest a GPU

## Result

- GPU memory on one GPU: 51.67 GB to 56.43 GB
- GPUs: 1
- Memory per GPU: 51.67 GB to 56.43 GB
- Usable per GPU: 72 GB
- Output throughput, all requests: 312 tokens/s to 518 tokens/s
- Output throughput, one request: 39 tokens/s to 65 tokens/s
- Weights: 32.76 GB to 34.32 GB
- KV cache: 17.18 GB to 17.18 GB
- Runtime overhead: 1.73 GB to 4.93 GB
- KV cache per token: 0.2621 MB
- GPUs in the dataset it fits on (up to 8 each): 10

> Estimate: Estimates from datasheet and config figures and the stated assumptions, not measurements: memory is a range, and the throughput is a rough scale, not a benchmark.

## Where it runs

Suggested: 1 × NVIDIA H100 SXM: Fits (Estimate)

## Fit on every GPU in the dataset

Each GPU's memory from its official datasheet, with the day it was read; a figure read more than 180 days ago is marked stale. Status is on one GPU; GPUs needed is the smallest tensor-parallel degree (1, 2, 4 or 8) that fits.

| GPU | Memory | On one GPU (estimate) | GPUs needed, cautious | GPUs needed, optimistic |
| --- | --- | --- | --- | --- |
| NVIDIA B200 | 180 GB ([Datasheet](https://www.nvidia.com/en-us/data-center/hgx/), 2026-10-13) | Fits | 1 | 1 |
| NVIDIA H200 SXM | 141 GB ([Datasheet](https://www.nvidia.com/en-us/data-center/h200/), 2026-10-13) | Fits | 1 | 1 |
| NVIDIA H100 SXM | 80 GB ([Datasheet](https://www.nvidia.com/en-us/data-center/h100/), 2026-10-13) | Fits | 1 | 1 |
| NVIDIA H100 NVL | 94 GB ([Datasheet](https://www.nvidia.com/en-us/data-center/h100/), 2026-10-13) | Fits | 1 | 1 |
| NVIDIA A100 80GB SXM | 80 GB ([Datasheet](https://www.nvidia.com/en-us/data-center/a100/), 2026-10-13) | Fits | 1 | 1 |
| NVIDIA A100 80GB PCIe | 80 GB ([Datasheet](https://www.nvidia.com/en-us/data-center/a100/), 2026-10-13) | Fits | 1 | 1 |
| NVIDIA L40S | 48 GB ([Datasheet](https://www.nvidia.com/en-us/data-center/l40s/), 2026-10-13) | Does not fit | 2 | 2 |
| NVIDIA L4 | 24 GB ([Datasheet](https://www.nvidia.com/en-us/data-center/l4/), 2026-10-13) | Does not fit | 4 | 4 |
| NVIDIA GeForce RTX 5090 | 32 GB ([Datasheet](https://images.nvidia.com/aem-dam/Solutions/geforce/blackwell/nvidia-rtx-blackwell-gpu-architecture.pdf), 2026-10-13) | Does not fit | 4 | 2 |
| NVIDIA GeForce RTX 4090 | 24 GB ([Datasheet](https://images.nvidia.com/aem-dam/Solutions/geforce/ada/nvidia-ada-gpu-architecture.pdf), 2026-10-13) | Does not fit | 4 | 4 |

## Example command

```sh
vllm serve Qwen/Qwen3-32B \
  --tensor-parallel-size 1 \
  --max-model-len 8192 \
  --max-num-seqs 8 \
  --gpu-memory-utilization 0.9 \
  --quantization fp8
```


## Data and versions

- GPU engine @anaroq/gpu 1.0.0 · GPU data 2026.10.0 · model data 2026.10.0
- Qwen/Qwen3-32B: config.json: https://huggingface.co/Qwen/Qwen3-32B/blob/9216db5781bf21249d130ec9da846c4624c16137/config.json (2026-10-13)
- Qwen/Qwen3-32B: model card: https://huggingface.co/Qwen/Qwen3-32B (2026-10-13)
- vLLM: Engine Arguments: https://docs.vllm.ai/en/stable/configuration/engine_args.html (2026-10-16)
- vLLM: FP8 W8A8: https://docs.vllm.ai/en/stable/features/quantization/fp8.html (2026-10-16)
- vLLM: Quantized KV Cache: https://docs.vllm.ai/en/stable/features/quantization/quantized_kvcache.html (2026-10-16)

## Breakdown

| Line | Value | Assumptions used |
| --- | --- | --- |
| Weights | 32.76 GB to 34.32 GB | Bits per weight in fp8, Bits per embedding weight, cautious |
| KV cache: sequences × context × bytes per token | 17.18 GB to 17.18 GB | – |
| Runtime overhead | 1.73 GB to 4.93 GB | Bits per weight in fp8, Bits per embedding weight, cautious, Runtime overhead per GPU, fixed, optimistic, Runtime overhead per GPU, fixed, cautious, Runtime overhead, share of weights, optimistic, Runtime overhead, share of weights, cautious |
| Total on one GPU | 51.67 GB to 56.43 GB | Bits per weight in fp8, Bits per embedding weight, cautious, Runtime overhead per GPU, fixed, optimistic, Runtime overhead per GPU, fixed, cautious, Runtime overhead, share of weights, optimistic, Runtime overhead, share of weights, cautious |
| Per GPU on 1 × NVIDIA H100 SXM | 51.67 GB to 56.43 GB | Bits per weight in fp8, Bits per embedding weight, cautious, Runtime overhead per GPU, fixed, optimistic, Runtime overhead per GPU, fixed, cautious, Runtime overhead, share of weights, optimistic, Runtime overhead, share of weights, cautious |
| Usable on one NVIDIA H100 SXM | 72 GB | Share of GPU memory the server may use |
| Output throughput, all sequences | 312 tokens/s to 518 tokens/s | Bits per weight in fp8, Bits per embedding weight, cautious, Memory bandwidth reached, cautious, Memory bandwidth reached, optimistic, Compute reached, cautious, Compute reached, optimistic, Average share of the context filled |

## Assumptions

- Share of GPU memory the server may use: 90%. vLLM's default gpu_memory_utilization; the rest is left to the driver and other processes. Source: https://docs.vllm.ai/en/stable/configuration/engine_args.html (read 16 Oct 2026)
- Bits per weight in fp8: 8 bits. 8 bits a weight; the per-channel or per-block scales add under 0.1 bit. **Assumption**
- Bits per embedding weight, cautious: 16 bits. Published quantized checkpoints keep the embedding and output matrices at 16 bits; the cautious end counts them so. **Assumption**
- Runtime overhead per GPU, fixed, optimistic: 0.75 GB. CUDA context, the serving framework and activation workspace on each GPU: a fixed part and a part that grows with the weights it holds. A declared range, not a measurement. **Assumption**
- Runtime overhead per GPU, fixed, cautious: 1.5 GB. CUDA context, the serving framework and activation workspace on each GPU: a fixed part and a part that grows with the weights it holds. A declared range, not a measurement. **Assumption**
- Runtime overhead, share of weights, optimistic: 0.03. CUDA context, the serving framework and activation workspace on each GPU: a fixed part and a part that grows with the weights it holds. A declared range, not a measurement. **Assumption**
- Runtime overhead, share of weights, cautious: 0.1. CUDA context, the serving framework and activation workspace on each GPU: a fixed part and a part that grows with the weights it holds. A declared range, not a measurement. **Assumption**
- Memory bandwidth reached, cautious: 0.5. Share of the datasheet memory bandwidth real decode kernels reach: each generated token reads all the weights once. **Assumption**
- Memory bandwidth reached, optimistic: 0.8. Share of the datasheet memory bandwidth real decode kernels reach: each generated token reads all the weights once. **Assumption**
- Compute reached, cautious: 0.3. Share of the peak tensor throughput reached when many sequences make decode compute-bound (2 FLOPs per parameter per token). **Assumption**
- Compute reached, optimistic: 0.5. Share of the peak tensor throughput reached when many sequences make decode compute-bound (2 FLOPs per parameter per token). **Assumption**
- Average share of the context filled: 0.5. Each decode step also reads the sequences' KV cache; sequences are taken at half their context on average. **Assumption**

Calculator gpu-planner 1.0.0 · engine @anaroq/calc 1.4.0 · data snapshot 2026.10.4

Open this calculation: https://anaroq.com/tools/gpu-planner#k=eyJhc3N1bXB0aW9ucyI6e30sImNhbGMiOiJncHUtcGxhbm5lciIsImlucHV0Ijp7ImJhdGNoIjo4LCJjb250ZXh0VG9rZW5zIjo4MTkyLCJlbmdpbmUiOiJ2bGxtIiwiZ3B1IjoiYXV0byIsImt2RHR5cGUiOiJiZjE2IiwibWFudWFsQXR0ZW50aW9uSGVhZHMiOjMyLCJtYW51YWxIZWFkRGltIjoxMjgsIm1hbnVhbEhpZGRlblNpemUiOjQwOTYsIm1hbnVhbEt2SGVhZHMiOjgsIm1hbnVhbExheWVycyI6MzIsIm1hbnVhbFBhcmFtZXRlcnNCIjo4LjAzLCJtYW51YWxWb2NhYlNpemUiOjEyODI1NiwibW9kZWwiOiJxd2VuMy0zMmIiLCJxdWFudGl6YXRpb24iOiJmcDgifSwidiI6MX0
How the calculators work: https://anaroq.com/methodology#calculators

      What it does

      It works out the GPU memory an open LLM needs for a weight format, a context and a number of requests served at once, and checks it against every GPU in the dataset: fits, tight or does not fit, and how many GPUs with tensor parallelism. For the GPU you pick, or the one it suggests, it adds a rough decode throughput and a command for vLLM or Ollama. The figures come from the site's GPU engine (@anaroq/gpu), its GPU datasheets and the models' own config files, each with its version and source.

      How it works

      1. Weights = parameters × bits per weight ÷ 8. The cautious end keeps the embedding and output matrices at 16 bits, as published quantized checkpoints do; int4 counts 4.25 bits for its scales.
      2. KV cache = 2 × layers × KV heads × head dimension × bytes (2, or 1 in fp8) × context × requests at once: the worst case a server reserves.
      3. Runtime overhead per GPU = 0.75 to 1.5 GB + 3 to 10% of the weights it holds.
      4. Fit: the cautious total fits in 90% of the GPU's memory (vLLM's default share), tight when only the optimistic one does. With tensor parallelism over 1, 2, 4 or 8 GPUs the weights split evenly and the KV cache splits up to the number of KV heads; the degree must divide the attention heads.
      5. Suggest: among the GPUs it fits on, the fewest GPUs, then one with FP8 support when the weights are fp8, then the least memory per GPU. No price decides it.
      6. Throughput: every generated token reads all weights and the requests' KV cache, at 50 to 80% of the datasheet memory bandwidth, unless 30 to 50% of the peak tensor compute (2 FLOPs per parameter per token) is lower. A rough scale, not a benchmark.

      Example

      Qwen3 32B in fp8, 8 requests at once of 8,192 tokens each, served with vLLM, with the GPU suggested:

      Qwen3 32B · fp8 · KV bf16 · 8 × 8,192 tokens · vLLM · suggest a GPU

      Estimate

      GPU memory on one GPU
      51.67 GB to 56.43 GB
      GPUs
      1
      Memory per GPU
      51.67 GB to 56.43 GB
      Usable per GPU
      72 GB
      Output throughput, all requests
      312 tokens/s to 518 tokens/s
      Output throughput, one request
      39 tokens/s to 65 tokens/s
      Weights
      32.76 GB to 34.32 GB
      KV cache
      17.18 GB to 17.18 GB
      Runtime overhead
      1.73 GB to 4.93 GB
      KV cache per token
      0.2621 MB
      GPUs in the dataset it fits on (up to 8 each)
      10

      Limits

      • An estimate, not a measurement: real memory and throughput depend on the engine, its version, the kernels and the traffic. Measure on your hardware before you rely on it.
      • Dense models only: mixture-of-experts models need active-parameter modelling that this version does not have.
      • Tensor parallelism inside one server, up to 8 GPUs; the communication between GPUs is not counted in the throughput.
      • The dataset holds vendor datasheets of NVIDIA GPUs only; no price and no offer is part of it.
      • The Ollama command needs the model's tag or GGUF repository, which only you can choose.

      Methodology: how the numbers are worked out

      Sources

      1. ANAROQ calculator methodology: assumptions, units and rounding
      2. NVIDIA H100 Tensor Core GPU: Product Specifications
      3. vLLM: Engine Arguments
      4. Ollama: FAQ

      Updated

      Next step

      LLM cost calculator: self-host vs API

      A free public tool by ANAROQ, a company building compute infrastructure. What ANAROQ is building