עב
עב

NVIDIA L40S: specs and which LLMs fit

The NVIDIA L40S as its official source states it, which open models fit on it, and how it compares with the GPUs next to it in memory. Every figure links to its source and the day it was read.

Specifications

From the vendor's own source. Tensor figures are dense, without sparsity.

Memory
48 GB GDDR6Source ·
Memory bandwidth
864 GB/sSource ·
FP16 Tensor, dense
362.05 TFLOPSSource ·
BF16 Tensor, dense
362.05 TFLOPSSource ·
FP8 Tensor, dense
733 TFLOPSSource ·
INT8 Tensor, dense
733 TOPSSource ·
Max board power
350 WSource ·
Architecture
Ada LovelaceSource ·
Segment
Data centerSource ·
Form factor
PCIe cardSource ·

How the figures were stored

The datasheet lists dense and sparse figures; the dense ones are here.

Which open models fit

On one NVIDIA L40S, 6 of the 10 models fit in bf16 as published and 8 in int4, with this workload.

Estimate An estimate from the GPU Planner's calculator: one request of 8,192 tokens, KV cache in bf16, served with vLLM, with 90% of each GPU's memory usable. Each cell is how many of this GPU the model needs with tensor parallelism; tight means only the optimistic estimate fits.

Open models and the GPUs they need, by weight format
Modelbf16 (as published)fp8int4
Qwen2.5 7B InstructFits on 1Fits on 1Fits on 1
Qwen2.5 14B InstructFits on 1Fits on 1Fits on 1
Qwen2.5 32B InstructFits on 2Fits on 1Fits on 1
Qwen2.5 72B InstructFits on 4Fits on 4Fits on 2
Qwen3 8BFits on 1Fits on 1Fits on 1
Qwen3 32BFits on 2Fits on 1Fits on 1
DeepSeek-R1-Distill-Llama 8BFits on 1Fits on 1Fits on 1
DeepSeek-R1-Distill-Llama 70BFits on 4Fits on 2Fits on 2
Phi-4Fits on 1Fits on 1Fits on 1
Mistral 7B Instruct v0.3Fits on 1Fits on 1Fits on 1

Assumptions

Every number the result uses that you didn't enter. Change any of them to match your case.

  • %

    vLLM's default gpu_memory_utilization; the rest is left to the driver and other processes.

    Source ·
  • Bits per weight in bf16Assumption
    16 bits (Fixed definition)

    16 bits a weight: the format the models are published in.

    No source: this is our own assumption. Change it if you know your case.
  • Bits per weight in fp8Assumption
    8 bits (Fixed definition)

    8 bits a weight; the per-channel or per-block scales add under 0.1 bit.

    No source: this is our own assumption. Change it if you know your case.
  • Bits per weight in int4Assumption
    4.25 bits (Fixed definition)

    4 bits a weight plus a 16-bit scale and a 4-bit zero per 128 weights (AWQ / GPTQ), about 4.16, rounded up to 4.25 for packing.

    No source: this is our own assumption. Change it if you know your case.
  • Bits per embedding weight, cautiousAssumption
    16 bits (Fixed definition)

    Published quantized checkpoints keep the embedding and output matrices at 16 bits; the cautious end counts them so.

    No source: this is our own assumption. Change it if you know your case.
  • Runtime overhead per GPU, fixed, optimisticAssumption
    0.75 GB (Fixed definition)

    CUDA context, the serving framework and activation workspace on each GPU: a fixed part and a part that grows with the weights it holds. A declared range, not a measurement.

    No source: this is our own assumption. Change it if you know your case.
  • Runtime overhead per GPU, fixed, cautiousAssumption
    1.5 GB (Fixed definition)

    CUDA context, the serving framework and activation workspace on each GPU: a fixed part and a part that grows with the weights it holds. A declared range, not a measurement.

    No source: this is our own assumption. Change it if you know your case.
  • Runtime overhead, share of weights, optimisticAssumption
    0.03 (Fixed definition)

    CUDA context, the serving framework and activation workspace on each GPU: a fixed part and a part that grows with the weights it holds. A declared range, not a measurement.

    No source: this is our own assumption. Change it if you know your case.
  • Runtime overhead, share of weights, cautiousAssumption
    0.1 (Fixed definition)

    CUDA context, the serving framework and activation workspace on each GPU: a fixed part and a part that grows with the weights it holds. A declared range, not a measurement.

    No source: this is our own assumption. Change it if you know your case.

What you enter and the assumptions you change are kept in this page's link, after the #.

Compared with the GPUs next to it

The nearest GPUs in the dataset by memory, with the same figures from their own sources.

Compared with the GPUs next to it
GPUMemoryBandwidthBF16, denseFP8, denseBoard powerSource
NVIDIA GeForce RTX 509032 GB GDDR71,792 GB/s209.5 TFLOPS419 TFLOPS575 WSource ·
NVIDIA L40S This GPU48 GB GDDR6864 GB/s362.05 TFLOPS733 TFLOPS350 WSource ·
NVIDIA A100 80GB PCIe80 GB HBM2e1,935 GB/s312 TFLOPSNo FP8 Tensor Cores300 WSource ·

Plan your own case

More requests at once, a longer context or another model change the answer. The GPU Planner works it out with this GPU already picked, and writes the vLLM or Ollama command.

Open the GPU Planner with the NVIDIA L40S picked

Limits

  • The figures are the vendor's, not measurements; the fit is an estimate of the site's GPU engine, not a benchmark.
  • The fit table covers dense open models of the dataset only; mixture-of-experts models are not in it.
  • No price, availability or offer is part of this page.

Sources

Model configurations

GPU engine @anaroq/gpu 1.0.0 · GPU data 2026.10.0 · model data 2026.10.0

How the estimates are worked out: Methodology

All GPUs compared