עב
עב

NVIDIA GeForce RTX 4090: specs and which LLMs fit

The NVIDIA GeForce RTX 4090 as its official source states it, which open models fit on it, and how it compares with the GPUs next to it in memory. Every figure links to its source and the day it was read.

Specifications

From the vendor's own source. Tensor figures are dense, without sparsity.

Memory
24 GB GDDR6XSource ·
Memory bandwidth
1,008 GB/sSource ·
FP16 Tensor, dense
165.2 TFLOPSSource ·
BF16 Tensor, dense
165.2 TFLOPSSource ·
FP8 Tensor, dense
330.3 TFLOPSSource ·
INT8 Tensor, dense
660.6 TOPSSource ·
Max board power
450 WSource ·
Architecture
Ada LovelaceSource ·
Segment
Consumer (GeForce)Source ·
Form factor
PCIe cardSource ·

How the figures were stored

The whitepaper lists dense and sparse figures; the dense ones are here, FP16, BF16 and FP8 with FP32 accumulate, comparable to data-center figures. Board power is the total graphics power (TGP).

Which open models fit

On one NVIDIA GeForce RTX 4090, 4 of the 10 models fit in bf16 as published and 6 in int4, with this workload.

Estimate An estimate from the GPU Planner's calculator: one request of 8,192 tokens, KV cache in bf16, served with vLLM, with 90% of each GPU's memory usable. Each cell is how many of this GPU the model needs with tensor parallelism; tight means only the optimistic estimate fits.

Open models and the GPUs they need, by weight format
Modelbf16 (as published)fp8int4
Qwen2.5 7B InstructFits on 1Fits on 1Fits on 1
Qwen2.5 14B InstructFits on 2Fits on 1Fits on 1
Qwen2.5 32B InstructFits on 4Fits on 2Fits on 2
Qwen2.5 72B InstructTight on 8Fits on 8Fits on 4
Qwen3 8BFits on 1Fits on 1Fits on 1
Qwen3 32BFits on 4Fits on 2Fits on 2
DeepSeek-R1-Distill-Llama 8BFits on 1Fits on 1Fits on 1
DeepSeek-R1-Distill-Llama 70BFits on 8Fits on 8Fits on 4
Phi-4Fits on 2Fits on 1Fits on 1
Mistral 7B Instruct v0.3Fits on 1Fits on 1Fits on 1

Assumptions

Every number the result uses that you didn't enter. Change any of them to match your case.

  • %

    vLLM's default gpu_memory_utilization; the rest is left to the driver and other processes.

    Source ·
  • Bits per weight in bf16Assumption
    16 bits (Fixed definition)

    16 bits a weight: the format the models are published in.

    No source: this is our own assumption. Change it if you know your case.
  • Bits per weight in fp8Assumption
    8 bits (Fixed definition)

    8 bits a weight; the per-channel or per-block scales add under 0.1 bit.

    No source: this is our own assumption. Change it if you know your case.
  • Bits per weight in int4Assumption
    4.25 bits (Fixed definition)

    4 bits a weight plus a 16-bit scale and a 4-bit zero per 128 weights (AWQ / GPTQ), about 4.16, rounded up to 4.25 for packing.

    No source: this is our own assumption. Change it if you know your case.
  • Bits per embedding weight, cautiousAssumption
    16 bits (Fixed definition)

    Published quantized checkpoints keep the embedding and output matrices at 16 bits; the cautious end counts them so.

    No source: this is our own assumption. Change it if you know your case.
  • Runtime overhead per GPU, fixed, optimisticAssumption
    0.75 GB (Fixed definition)

    CUDA context, the serving framework and activation workspace on each GPU: a fixed part and a part that grows with the weights it holds. A declared range, not a measurement.

    No source: this is our own assumption. Change it if you know your case.
  • Runtime overhead per GPU, fixed, cautiousAssumption
    1.5 GB (Fixed definition)

    CUDA context, the serving framework and activation workspace on each GPU: a fixed part and a part that grows with the weights it holds. A declared range, not a measurement.

    No source: this is our own assumption. Change it if you know your case.
  • Runtime overhead, share of weights, optimisticAssumption
    0.03 (Fixed definition)

    CUDA context, the serving framework and activation workspace on each GPU: a fixed part and a part that grows with the weights it holds. A declared range, not a measurement.

    No source: this is our own assumption. Change it if you know your case.
  • Runtime overhead, share of weights, cautiousAssumption
    0.1 (Fixed definition)

    CUDA context, the serving framework and activation workspace on each GPU: a fixed part and a part that grows with the weights it holds. A declared range, not a measurement.

    No source: this is our own assumption. Change it if you know your case.

What you enter and the assumptions you change are kept in this page's link, after the #.

Compared with the GPUs next to it

The nearest GPUs in the dataset by memory, with the same figures from their own sources.

Compared with the GPUs next to it
GPUMemoryBandwidthBF16, denseFP8, denseBoard powerSource
NVIDIA L424 GB GDDR6300 GB/s121 TFLOPS242.5 TFLOPS72 WSource ·
NVIDIA GeForce RTX 4090 This GPU24 GB GDDR6X1,008 GB/s165.2 TFLOPS330.3 TFLOPS450 WSource ·
NVIDIA GeForce RTX 509032 GB GDDR71,792 GB/s209.5 TFLOPS419 TFLOPS575 WSource ·

Plan your own case

More requests at once, a longer context or another model change the answer. The GPU Planner works it out with this GPU already picked, and writes the vLLM or Ollama command.

Open the GPU Planner with the NVIDIA GeForce RTX 4090 picked

Limits

  • The figures are the vendor's, not measurements; the fit is an estimate of the site's GPU engine, not a benchmark.
  • The fit table covers dense open models of the dataset only; mixture-of-experts models are not in it.
  • No price, availability or offer is part of this page.

Sources

Model configurations

GPU engine @anaroq/gpu 1.0.0 · GPU data 2026.10.0 · model data 2026.10.0

How the estimates are worked out: Methodology

All GPUs compared