עב
עב

NVIDIA A100 80GB SXM: specs and which LLMs fit

The NVIDIA A100 80GB SXM as its official source states it, which open models fit on it, and how it compares with the GPUs next to it in memory. Every figure links to its source and the day it was read.

Specifications

From the vendor's own source. Tensor figures are dense, without sparsity.

Memory
80 GB HBM2eSource ·
Memory bandwidth
2,039 GB/sSource ·
FP16 Tensor, dense
312 TFLOPSSource ·
BF16 Tensor, dense
312 TFLOPSSource ·
FP8 Tensor, dense
No FP8 Tensor CoresSource ·
INT8 Tensor, dense
624 TOPSSource ·
Max board power
400 WSource ·
Architecture
AmpereSource ·
Segment
Data centerSource ·
Form factor
SXM moduleSource ·

How the figures were stored

The datasheet lists dense and sparse figures; the dense ones are here. Ampere has no FP8 Tensor Cores. Board power is the standard configuration (a CTS version goes up to 500 W).

Which open models fit

On one NVIDIA A100 80GB SXM, 6 of the 10 models fit in bf16 as published and 10 in int4, with this workload.

Estimate An estimate from the GPU Planner's calculator: one request of 8,192 tokens, KV cache in bf16, served with vLLM, with 90% of each GPU's memory usable. Each cell is how many of this GPU the model needs with tensor parallelism; tight means only the optimistic estimate fits.

Open models and the GPUs they need, by weight format
Modelbf16 (as published)fp8int4
Qwen2.5 7B InstructFits on 1Fits on 1Fits on 1
Qwen2.5 14B InstructFits on 1Fits on 1Fits on 1
Qwen2.5 32B InstructFits on 2Fits on 1Fits on 1
Qwen2.5 72B InstructFits on 4Fits on 2Fits on 1
Qwen3 8BFits on 1Fits on 1Fits on 1
Qwen3 32BFits on 2Fits on 1Fits on 1
DeepSeek-R1-Distill-Llama 8BFits on 1Fits on 1Fits on 1
DeepSeek-R1-Distill-Llama 70BFits on 4Fits on 2Fits on 1
Phi-4Fits on 1Fits on 1Fits on 1
Mistral 7B Instruct v0.3Fits on 1Fits on 1Fits on 1

Assumptions

Every number the result uses that you didn't enter. Change any of them to match your case.

  • %

    vLLM's default gpu_memory_utilization; the rest is left to the driver and other processes.

    Source ·
  • Bits per weight in bf16Assumption
    16 bits (Fixed definition)

    16 bits a weight: the format the models are published in.

    No source: this is our own assumption. Change it if you know your case.
  • Bits per weight in fp8Assumption
    8 bits (Fixed definition)

    8 bits a weight; the per-channel or per-block scales add under 0.1 bit.

    No source: this is our own assumption. Change it if you know your case.
  • Bits per weight in int4Assumption
    4.25 bits (Fixed definition)

    4 bits a weight plus a 16-bit scale and a 4-bit zero per 128 weights (AWQ / GPTQ), about 4.16, rounded up to 4.25 for packing.

    No source: this is our own assumption. Change it if you know your case.
  • Bits per embedding weight, cautiousAssumption
    16 bits (Fixed definition)

    Published quantized checkpoints keep the embedding and output matrices at 16 bits; the cautious end counts them so.

    No source: this is our own assumption. Change it if you know your case.
  • Runtime overhead per GPU, fixed, optimisticAssumption
    0.75 GB (Fixed definition)

    CUDA context, the serving framework and activation workspace on each GPU: a fixed part and a part that grows with the weights it holds. A declared range, not a measurement.

    No source: this is our own assumption. Change it if you know your case.
  • Runtime overhead per GPU, fixed, cautiousAssumption
    1.5 GB (Fixed definition)

    CUDA context, the serving framework and activation workspace on each GPU: a fixed part and a part that grows with the weights it holds. A declared range, not a measurement.

    No source: this is our own assumption. Change it if you know your case.
  • Runtime overhead, share of weights, optimisticAssumption
    0.03 (Fixed definition)

    CUDA context, the serving framework and activation workspace on each GPU: a fixed part and a part that grows with the weights it holds. A declared range, not a measurement.

    No source: this is our own assumption. Change it if you know your case.
  • Runtime overhead, share of weights, cautiousAssumption
    0.1 (Fixed definition)

    CUDA context, the serving framework and activation workspace on each GPU: a fixed part and a part that grows with the weights it holds. A declared range, not a measurement.

    No source: this is our own assumption. Change it if you know your case.

What you enter and the assumptions you change are kept in this page's link, after the #.

Compared with the GPUs next to it

The nearest GPUs in the dataset by memory, with the same figures from their own sources.

Compared with the GPUs next to it
GPUMemoryBandwidthBF16, denseFP8, denseBoard powerSource
NVIDIA A100 80GB PCIe80 GB HBM2e1,935 GB/s312 TFLOPSNo FP8 Tensor Cores300 WSource ·
NVIDIA A100 80GB SXM This GPU80 GB HBM2e2,039 GB/s312 TFLOPSNo FP8 Tensor Cores400 WSource ·
NVIDIA H100 SXM80 GB HBM33,350 GB/s989.5 TFLOPS1,979 TFLOPS700 WSource ·

Plan your own case

More requests at once, a longer context or another model change the answer. The GPU Planner works it out with this GPU already picked, and writes the vLLM or Ollama command.

Open the GPU Planner with the NVIDIA A100 80GB SXM picked

Limits

  • The figures are the vendor's, not measurements; the fit is an estimate of the site's GPU engine, not a benchmark.
  • The fit table covers dense open models of the dataset only; mixture-of-experts models are not in it.
  • No price, availability or offer is part of this page.

Sources

Model configurations

GPU engine @anaroq/gpu 1.0.0 · GPU data 2026.10.0 · model data 2026.10.0

How the estimates are worked out: Methodology

All GPUs compared