עב
עב

NVIDIA H200 SXM: specs and which LLMs fit

The NVIDIA H200 SXM as its official source states it, which open models fit on it, and how it compares with the GPUs next to it in memory. Every figure links to its source and the day it was read.

Specifications

From the vendor's own source. Tensor figures are dense, without sparsity.

Memory
141 GB HBM3eSource ·
Memory bandwidth
4,800 GB/sSource ·
FP16 Tensor, dense
989.5 TFLOPSSource ·
BF16 Tensor, dense
989.5 TFLOPSSource ·
FP8 Tensor, dense
1,979 TFLOPSSource ·
INT8 Tensor, dense
1,979 TOPSSource ·
Max board power
700 WSource ·
Architecture
HopperSource ·
Segment
Data centerSource ·
Form factor
SXM moduleSource ·

How the figures were stored

The datasheet's Tensor figures are with sparsity; the figures here are half of them (dense).

Which open models fit

On one NVIDIA H200 SXM, 8 of the 10 models fit in bf16 as published and 10 in int4, with this workload.

Estimate An estimate from the GPU Planner's calculator: one request of 8,192 tokens, KV cache in bf16, served with vLLM, with 90% of each GPU's memory usable. Each cell is how many of this GPU the model needs with tensor parallelism; tight means only the optimistic estimate fits.

Open models and the GPUs they need, by weight format
Modelbf16 (as published)fp8int4
Qwen2.5 7B InstructFits on 1Fits on 1Fits on 1
Qwen2.5 14B InstructFits on 1Fits on 1Fits on 1
Qwen2.5 32B InstructFits on 1Fits on 1Fits on 1
Qwen2.5 72B InstructFits on 2Fits on 1Fits on 1
Qwen3 8BFits on 1Fits on 1Fits on 1
Qwen3 32BFits on 1Fits on 1Fits on 1
DeepSeek-R1-Distill-Llama 8BFits on 1Fits on 1Fits on 1
DeepSeek-R1-Distill-Llama 70BFits on 2Fits on 1Fits on 1
Phi-4Fits on 1Fits on 1Fits on 1
Mistral 7B Instruct v0.3Fits on 1Fits on 1Fits on 1

Assumptions

Every number the result uses that you didn't enter. Change any of them to match your case.

  • %

    vLLM's default gpu_memory_utilization; the rest is left to the driver and other processes.

    Source ·
  • Bits per weight in bf16Assumption
    16 bits (Fixed definition)

    16 bits a weight: the format the models are published in.

    No source: this is our own assumption. Change it if you know your case.
  • Bits per weight in fp8Assumption
    8 bits (Fixed definition)

    8 bits a weight; the per-channel or per-block scales add under 0.1 bit.

    No source: this is our own assumption. Change it if you know your case.
  • Bits per weight in int4Assumption
    4.25 bits (Fixed definition)

    4 bits a weight plus a 16-bit scale and a 4-bit zero per 128 weights (AWQ / GPTQ), about 4.16, rounded up to 4.25 for packing.

    No source: this is our own assumption. Change it if you know your case.
  • Bits per embedding weight, cautiousAssumption
    16 bits (Fixed definition)

    Published quantized checkpoints keep the embedding and output matrices at 16 bits; the cautious end counts them so.

    No source: this is our own assumption. Change it if you know your case.
  • Runtime overhead per GPU, fixed, optimisticAssumption
    0.75 GB (Fixed definition)

    CUDA context, the serving framework and activation workspace on each GPU: a fixed part and a part that grows with the weights it holds. A declared range, not a measurement.

    No source: this is our own assumption. Change it if you know your case.
  • Runtime overhead per GPU, fixed, cautiousAssumption
    1.5 GB (Fixed definition)

    CUDA context, the serving framework and activation workspace on each GPU: a fixed part and a part that grows with the weights it holds. A declared range, not a measurement.

    No source: this is our own assumption. Change it if you know your case.
  • Runtime overhead, share of weights, optimisticAssumption
    0.03 (Fixed definition)

    CUDA context, the serving framework and activation workspace on each GPU: a fixed part and a part that grows with the weights it holds. A declared range, not a measurement.

    No source: this is our own assumption. Change it if you know your case.
  • Runtime overhead, share of weights, cautiousAssumption
    0.1 (Fixed definition)

    CUDA context, the serving framework and activation workspace on each GPU: a fixed part and a part that grows with the weights it holds. A declared range, not a measurement.

    No source: this is our own assumption. Change it if you know your case.

What you enter and the assumptions you change are kept in this page's link, after the #.

Compared with the GPUs next to it

The nearest GPUs in the dataset by memory, with the same figures from their own sources.

Compared with the GPUs next to it
GPUMemoryBandwidthBF16, denseFP8, denseBoard powerSource
NVIDIA H100 NVL94 GB HBM33,900 GB/s835.5 TFLOPS1,670.5 TFLOPS400 WSource ·
NVIDIA H200 SXM This GPU141 GB HBM3e4,800 GB/s989.5 TFLOPS1,979 TFLOPS700 WSource ·
NVIDIA B200180 GB HBM3e8,000 GB/s2,250 TFLOPS4,500 TFLOPSNot published per GPUSource ·

Plan your own case

More requests at once, a longer context or another model change the answer. The GPU Planner works it out with this GPU already picked, and writes the vLLM or Ollama command.

Open the GPU Planner with the NVIDIA H200 SXM picked

Limits

  • The figures are the vendor's, not measurements; the fit is an estimate of the site's GPU engine, not a benchmark.
  • The fit table covers dense open models of the dataset only; mixture-of-experts models are not in it.
  • No price, availability or offer is part of this page.

Sources

Model configurations

GPU engine @anaroq/gpu 1.0.0 · GPU data 2026.10.0 · model data 2026.10.0

How the estimates are worked out: Methodology

All GPUs compared