Best GPU for LLM Inference: L4 to B200 and MI300X Compared

8 min read

The best GPU for LLM inference is the cheapest one that holds your model, its KV cache at your real concurrency, and enough memory bandwidth to hit the speed your users expect. In practice that narrows to a short list: an L4 or RTX-class card for small models and a few users, an L40S for mid-size models, an H100 or A100 80GB for 70B at 4-bit, and an H200, B200 or AMD MI300X once you serve big models to many people at once.

Two numbers decide almost everything: memory capacity (does it fit) and memory bandwidth (how fast it generates). Compute throughput matters less than most spec sheets suggest, because generating text one token at a time is limited by how fast the GPU can read the model's weights from memory, not by how fast it can multiply.

This post compares the cards people actually deploy, then gives a decision table by model size and number of users. If you have not yet worked out how much memory your model needs, start with GPU memory for LLM inference and come back.

The specs that matter for inference

GPU Memory Memory bandwidth FP8 Where you usually get it
RTX 4090 24 GB GDDR6X 1,008 GB/s Yes Workstations
RTX 5090 32 GB GDDR7 1,792 GB/s Yes Workstations
L4 24 GB 300 GB/s Yes AWS G6, many GPU clouds
L40S 48 GB GDDR6 864 GB/s Yes AWS G6e, servers you buy
A100 80GB 80 GB HBM2e 2,039 GB/s (SXM), 1,935 GB/s (PCIe) No Older cloud fleets, used market
H100 80 GB (SXM), 94 GB (NVL) 3.35 TB/s (SXM), 3.9 TB/s (NVL) Yes AWS P5, most GPU clouds
H200 141 GB HBM3e 4.8 TB/s Yes AWS P5e and P5en
B200 180 GB HBM3e (HGX) Up to 8 TB/s Yes, plus FP4 AWS P6-B200, 8-GPU systems
AMD MI300X 192 GB HBM3 5.3 TB/s Yes Azure ND MI300X v5

Specs are from the vendors' product pages and datasheets linked above. Cloud instance families are from the AWS accelerated computing page and Azure's ND family docs. We have left prices out because they change monthly and vary by region and commitment.

A few things the table does not show:

  • FP8 support matters because FP8 weights and an FP8 KV cache halve memory against 16-bit with a small quality cost. The A100 predates FP8 tensor cores, so it runs FP8 models slower or not at all, depending on the inference engine. It still runs 4-bit and 8-bit integer formats well.
  • NVLink lets GPUs in one server share a model quickly. SXM parts (A100 SXM, H100 SXM, H200, B200) and the H100 NVL bridge have it. The L40S, L4 and consumer RTX cards talk over PCIe, which is fine for one GPU per model and slower once you split a model across cards.
  • Licensing. NVIDIA's GeForce driver license has a "No Datacenter Deployment" clause. An RTX 4090 or 5090 in a developer's workstation is normal; a rack of them in a data center is a legal question to put to your counsel, not a technical one.

Why bandwidth sets your speed

During generation, the GPU reads every weight once per token (for a dense model). So a single user's tokens per second has a hard ceiling:

max tokens/s (one stream) ~ memory bandwidth / bytes of weights

For an 8B model at 8-bit, about 8.5 GB of weights:

GPU Bandwidth Ceiling, one stream
L4 300 GB/s ~35 tokens/s
L40S 864 GB/s ~100 tokens/s
RTX 4090 1,008 GB/s ~120 tokens/s
H100 SXM 3,350 GB/s ~390 tokens/s

Real numbers land below these ceilings, but the ranking holds. That is why an L4 feels slow for anything above 8B even though the model fits: 24 GB is plenty of room, 300 GB/s is not much speed. It is a good card for small models, embeddings and many low-traffic endpoints at low power (72 W), and a poor card for a 32B chat model.

Batching changes the picture for throughput. When 20 users share one pass over the weights, total tokens per second rises sharply while each user's speed falls only a little. That is where compute and KV cache capacity come back in, and why bigger-memory cards serve far more users than their bandwidth advantage alone suggests. Batching only helps if your inference server does it: continuous batching servers such as vLLM do, single-user tools mostly do not.

Mixture-of-experts models such as DeepSeek R1 or Qwen3-30B-A3B read only the active experts per token, so they generate faster than their total size suggests, but every expert still has to sit in memory. Size them by total parameters for memory and by active parameters for speed.

Best GPU for LLM inference by model size and concurrency

"Users" here means simultaneous requests at roughly an 8K-token context, not total headcount. Memory figures assume 4-bit or 8-bit weights; see the LLM quantization guide for what you trade at each level.

Model 1-5 concurrent users 10-30 concurrent users 50+ concurrent users
7-8B L4, RTX 4090 L40S H100
14B RTX 4090 (4-bit), L40S L40S (8-bit) H100
32B RTX 5090 (4-bit), L40S (8-bit) H100 (8-bit) H200
70B H100 or A100 80GB (4-bit) H200, or 2x H100 2-4x H100 or H200
100B+ MoE, 671B Multi-GPU node 8x H200, B200 or MI300X 8x H200, B200 or MI300X

How to read it:

  • Pick the column, then the cheapest card in it. If two cards fit, the one with more bandwidth gives faster answers; the one with more memory gives more users or longer context.
  • Moving right means the KV cache dominates. At 30 users and 8K context, Llama 3.1 8B needs about 30 GB of cache on top of its weights. The model has not grown; the number of conversations has.
  • One big card beats two small ones when the model fits on the big one. Splitting a model across PCIe cards costs throughput and leaves each card holding duplicate buffers. A single H100 serving 70B at 4-bit is simpler and usually faster than two L40S cards doing the same job. The Llama 70B VRAM requirements post has the per-configuration numbers.
  • The full DeepSeek R1 needs a whole node. At FP8 the weights alone are roughly 700 GB, which fits an 8x H200 server (1,128 GB total) and does not fit 8x H100 (640 GB). The DeepSeek R1 hardware requirements post covers the distilled models that run on one card.

H100 vs A100 for inference

The A100 80GB is still widely available and often cheaper. It has the same memory size as the H100 SXM, so anything that fits one fits the other. The H100 SXM has about 1.6x the bandwidth (3.35 vs 2.04 TB/s) and native FP8, so it generates faster and serves FP8 checkpoints the A100 cannot run efficiently.

Choose the A100 when you serve 4-bit or 8-bit integer quantizations (GGUF, AWQ, GPTQ), your latency target is loose, and the price gap is large. Choose the H100 when you want FP8 weights or KV cache, or when per-user speed matters.

H200, B200 and MI300X

These are the cards for large models at real concurrency. The H200 is an H100 with 141 GB of faster memory, so it runs the same software and fits a 70B model at FP8 with room for a large KV cache on one GPU. The B200 roughly doubles bandwidth again and adds FP4, but it ships mainly in 8-GPU systems, which is more than most single-model deployments need.

The MI300X has the most memory per GPU in this list (192 GB) and 5.3 TB/s of bandwidth. The hardware case is strong. The software case depends on your stack: vLLM supports AMD GPUs through ROCm and llama.cpp has a ROCm (HIP) backend, but fewer tested configurations and community answers exist than for NVIDIA. Confirm that your exact model, quantization format and inference engine run on ROCm before you buy.

Buy or rent

Renting makes sense while you are still learning your real concurrency and context lengths. Buying makes sense once a GPU will be busy most of the day for a year or more. The self-hosted LLM cost post has a break-even model you can fill in with your own quotes.

Whichever you choose, benchmark before you commit to a fleet. Load your model at the quantization you plan to serve, replay real prompts at your expected peak concurrency, and measure time to first token and tokens per second at the 95th percentile. Spec sheets tell you what is possible; your prompts tell you what you will get.

Where AI Refinery fits

AI Refinery, a self-hosted LLM platform, runs on GPUs you provide, in your cloud account, your data center or an air-gapped network. Every model variant in its catalog is benchmarked before it is listed, with its size on disk, the GPU memory it needs (weights plus KV cache at the full context window) and measured tokens per second. That gives you a starting point for sizing before you rent or buy anything. Plans are sized by model, up to 14B, up to 70B or unlimited, and are described on the AI Refinery pricing section.

If you plan to run on AMD hardware, ask us at [email protected] before buying so we can confirm support for your setup.

GPU specs change between product revisions. Check the vendor's current datasheet before you order hardware.