AI on your own GPU
What it takes to run open AI models on a GPU you control: memory, speed and the setup around them.
Running a model yourself starts with one question: does it fit? The answer depends on the size of the model, the precision its weights are stored in and how much context it has to keep in memory, and it decides which GPU, and how many, the model needs.
This topic collects the tools and pages that answer that question from published datasheets and model configurations, with every assumption shown, so the estimate can be checked before any hardware is rented or bought.
A recommended path
- Compare the GPUs
See each GPU's memory, bandwidth and Tensor throughput from its datasheet, and which open models fit on it.
- Check the fit
Estimate the GPU memory for the weights and the KV cache, and whether one GPU is enough.
- Check the assumptions
Read how the estimates are worked out, and what they leave out, before you rely on them.
Tools and pages
| Page | Type | About |
|---|---|---|
| GPUs for running LLMs: specs compared | Data | The NVIDIA GPUs in the site's GPU dataset side by side: memory, memory bandwidth, dense Tensor throughput and board power from each vendor datasheet, with the day it was read, and a page per GPU with the open models that fit on it. |
| NVIDIA B200: specs and which LLMs fit | Data | NVIDIA B200 figures from the vendor datasheet, each with its source and read date, which open LLMs fit on it in bf16, fp8 and int4 as an estimate, how it compares with the GPUs next to it, and a link to the GPU Planner with it picked. |
| NVIDIA H200 SXM: specs and which LLMs fit | Data | NVIDIA H200 SXM figures from the vendor datasheet, each with its source and read date, which open LLMs fit on it in bf16, fp8 and int4 as an estimate, how it compares with the GPUs next to it, and a link to the GPU Planner with it picked. |
| NVIDIA H100 SXM: specs and which LLMs fit | Data | NVIDIA H100 SXM figures from the vendor datasheet, each with its source and read date, which open LLMs fit on it in bf16, fp8 and int4 as an estimate, how it compares with the GPUs next to it, and a link to the GPU Planner with it picked. |
| NVIDIA H100 NVL: specs and which LLMs fit | Data | NVIDIA H100 NVL figures from the vendor datasheet, each with its source and read date, which open LLMs fit on it in bf16, fp8 and int4 as an estimate, how it compares with the GPUs next to it, and a link to the GPU Planner with it picked. |
| NVIDIA A100 80GB SXM: specs and which LLMs fit | Data | NVIDIA A100 80GB SXM figures from the vendor datasheet, each with its source and read date, which open LLMs fit on it in bf16, fp8 and int4 as an estimate, how it compares with the GPUs next to it, and a link to the GPU Planner with it picked. |
| NVIDIA A100 80GB PCIe: specs and which LLMs fit | Data | NVIDIA A100 80GB PCIe figures from the vendor datasheet, each with its source and read date, which open LLMs fit on it in bf16, fp8 and int4 as an estimate, how it compares with the GPUs next to it, and a link to the GPU Planner with it picked. |
| NVIDIA L40S: specs and which LLMs fit | Data | NVIDIA L40S figures from the vendor datasheet, each with its source and read date, which open LLMs fit on it in bf16, fp8 and int4 as an estimate, how it compares with the GPUs next to it, and a link to the GPU Planner with it picked. |
| NVIDIA L4: specs and which LLMs fit | Data | NVIDIA L4 figures from the vendor datasheet, each with its source and read date, which open LLMs fit on it in bf16, fp8 and int4 as an estimate, how it compares with the GPUs next to it, and a link to the GPU Planner with it picked. |
| NVIDIA GeForce RTX 5090: specs and which LLMs fit | Data | NVIDIA GeForce RTX 5090 figures from the vendor datasheet, each with its source and read date, which open LLMs fit on it in bf16, fp8 and int4 as an estimate, how it compares with the GPUs next to it, and a link to the GPU Planner with it picked. |
| NVIDIA GeForce RTX 4090: specs and which LLMs fit | Data | NVIDIA GeForce RTX 4090 figures from the vendor datasheet, each with its source and read date, which open LLMs fit on it in bf16, fp8 and int4 as an estimate, how it compares with the GPUs next to it, and a link to the GPU Planner with it picked. |
See also: Methodology · Free tools