gpt-oss Hardware Requirements: VRAM for 20B and 120B
The gpt-oss hardware requirements are easier to meet than the parameter counts suggest. gpt-oss-20b needs about 14 GB of GPU memory for one user at an 8K-token context, so it runs on a 16 GB card. gpt-oss-120b needs about 70 GB, so it fits on one 80 GB GPU such as an H100 or A100 80GB. Both ship with their expert weights already in 4-bit MXFP4, which is why the files are small and why further quantization saves very little.
These are OpenAI's open-weight models, released in August 2025 under the Apache 2.0 license. This post works through where the memory goes, using the published config.json files and file sizes, so you can size hardware before you buy it.
gpt-oss hardware requirements at a glance
Totals add the KV cache for one user and 10% for working buffers, the same method as our GPU memory for LLM guide:
| Model | Total / active parameters | Weights (MXFP4 GGUF) | Total, 1 user at 8K | Total, 1 user at 128K |
|---|---|---|---|---|
| gpt-oss-20b | 21B / 3.6B | ~12.1 GB | ~13.5 GB | ~17 GB |
| gpt-oss-120b | 117B / 5.1B | ~63.4 GB | ~70 GB | ~75 GB |
The parameter counts come from the gpt-oss-120b model card, which states that the 120B model fits on "a single 80GB GPU (like NVIDIA H100 or AMD MI300X)" and that the 20B model runs "within 16GB of memory." The weight sizes are the MXFP4 files published by the llama.cpp project (20b, 120b). The original safetensors checkpoints are about the same size: 13.8 GB and 65.3 GB.
Why the weights are already 4-bit
Most open models are released in 16-bit, and you quantize them yourself. gpt-oss went the other way. OpenAI post-trained the models with the mixture-of-experts (MoE) weights in MXFP4, a 4-bit floating-point format, and ran all of its published evaluations on those 4-bit weights. You can see this in the quantization_config block of each model's config.json: the attention layers, router, embeddings and output head stay at 16-bit, and everything else is MXFP4.
That has a practical consequence for GGUF users. The expert weights are the vast majority of the parameters, and they are already at about 4.25 bits each. Re-quantizing to Q4_K_M or even Q2_K only touches the small 16-bit remainder. Look at the file list of Unsloth's gpt-oss-120b GGUF builds:
| GGUF variant | File size |
|---|---|
| F16 (MXFP4 experts kept) | ~65.4 GB |
| Q8_0 | ~63.4 GB |
| Q4_K_M | ~62.8 GB |
| Q2_K | ~62.6 GB |
Going from F16 to Q2_K saves under 3 GB, about 4%, and costs quality in the attention layers. For gpt-oss, use the MXFP4 file and spend your effort elsewhere. The general trade-offs between levels are in our LLM quantization guide; this model family is the exception to them.
The KV cache is unusually small
Both models alternate two kinds of attention layer. Half the layers use a sliding window of 128 tokens, so their cache never grows past 128 tokens. The other half attend to the full context. Only the full-attention layers grow with context, and each has 8 key-value heads with a head size of 64:
gpt-oss-120b: 2 x 18 full layers x 8 KV heads x 64 head size x 2 bytes = 36,864 bytes per token (36 KB)
gpt-oss-20b: 2 x 12 full layers x 8 KV heads x 64 head size x 2 bytes = 24,576 bytes per token (24 KB)
For comparison, Llama 3.3 70B needs 320 KB per token (see Llama 70B VRAM requirements). gpt-oss-120b needs about a ninth of that, despite having more parameters.
| Context length | gpt-oss-20b, 1 user | gpt-oss-120b, 1 user |
|---|---|---|
| 8K tokens | ~0.2 GB | ~0.3 GB |
| 32K tokens | ~0.8 GB | ~1.2 GB |
| 128K tokens | ~3.2 GB | ~4.8 GB |
Both models support a context of 131,072 tokens (max_position_embeddings in the config). These figures assume your inference engine allocates the sliding-window layers at their window size. An engine that reserves full-length cache for every layer doubles the per-token figure, which is still small. llama.cpp, for example, uses the reduced cache by default and has a --swa-full flag to turn it off, documented in its server README.
Reasoning tokens count too
gpt-oss models are reasoning models. They write a chain of thought before the answer, and you set how much with a reasoning level of low, medium or high in the system prompt ("Reasoning: high"). Those reasoning tokens live in the KV cache like any others. A short question at high effort can produce thousands of tokens before the answer starts.
Because the cache is cheap for these models, the effect on memory is modest. The effect on latency is not: more reasoning tokens means a longer wait for the first visible word. When you compare reasoning levels, measure time to the final answer as well as time to first token.
Which GPUs fit gpt-oss
gpt-oss-20b
- 16 GB card (RTX 4080, RTX 5070 Ti, A4000-class workstation cards): one user up to about 64K tokens of context. Full 128K is slightly over the line once buffers are counted.
- 24 GB card (RTX 4090, RTX 3090, L4, A10): full 128K context, or around 40 users at 8K.
- 48 GB card (L40S, RTX 6000 Ada, A6000): room for heavy concurrency. At this size, consider whether the larger model would serve you better.
gpt-oss-120b
- One 80 GB card (H100, A100 80GB): about 70 GB for weights and buffers leaves roughly 10 GB for cache. That is around 30 users at 8K, or one user at the full 128K plus a few short conversations. A 141 GB H200 removes the constraint.
- Two 48 GB cards (96 GB total): works with tensor parallelism, with more room for users than a single 80 GB card. The link between the cards affects speed.
- One 48 GB card: does not fit on the GPU alone. llama.cpp can keep some or all expert weights in system RAM with
--n-cpu-moe Nor--cpu-moe. Because only 5.1B parameters are active per token, this is usable for one or two people, but it is far slower than a GPU-only setup and not suited to a shared service.
Apple Silicon is different because the GPU shares system memory. By the same arithmetic, gpt-oss-20b fits a Mac with 24 GB or more, and gpt-oss-120b needs a machine with 96 GB or more.
Running gpt-oss locally
The model card lists the supported routes. The three most common:
# Ollama: single user, quickest start
ollama run gpt-oss:20b
# llama.cpp server with the MXFP4 GGUF
llama-server -hf ggml-org/gpt-oss-20b-GGUF --ctx-size 32768
# vLLM: concurrent serving with an OpenAI-compatible API
vllm serve openai/gpt-oss-120b
Both models were trained on OpenAI's harmony response format, and the model card says they "will not work correctly otherwise." The chat templates shipped with the model and with these engines apply it for you. If you send raw prompts to a completion endpoint, you have to apply it yourself. vLLM keeps an up-to-date gpt-oss recipe with the flags and GPU generations it currently supports; read it before deploying, because MXFP4 kernel support varies by GPU and version. For the difference between these engines, see vLLM vs Ollama.
License and use terms
Both models are released under Apache 2.0, which allows commercial use, modification and redistribution. The repository also includes a short usage policy asking users to comply with applicable law. Read both before deploying, as you would for any model; the open-source LLM for enterprise post has a license checklist.
Where gpt-oss falls in AI Refinery's plans
AI Refinery, a self-hosted LLM platform, sizes its plans by total parameter count, not active parameters. gpt-oss-20b, at 21B, is above Basic's 14B limit and falls under Professional (up to 70B). gpt-oss-120b, at 117B, falls under Enterprise, which has no size limit. Which models are in the catalog at any time is shown in the console, not in this post. Plan details are in the pricing section.
Each catalog variant lists measured GPU memory with the KV cache counted at the model's full context window, so expect the console figure to be closer to the 128K column above than the 8K one. The customer provides the GPUs. For the rest of the platform, see the private LLM platform overview.
Figures are estimates from public configs and file sizes, checked in November 2026. Measure on your own hardware before buying GPUs.