GGUF vs AWQ vs GPTQ: Which Quantization Format to Use
In a GGUF vs AWQ choice, the format usually follows the engine, not the other way round. GGUF is the file format of llama.cpp and the tools built on it, and it runs almost anywhere: NVIDIA, AMD, Apple Silicon and plain CPUs. AWQ and GPTQ are 4-bit methods for GPU servers such as vLLM and SGLang, where they load as standard Hugging Face checkpoints and use fast GPU kernels. If you run on recent NVIDIA data center GPUs, FP8 is often the better option than any 4-bit format.
All three shrink a model to roughly a quarter of its 16-bit size, and at 4 bits their quality is close enough that your engine and hardware should decide. This post covers how each format is produced, which engines load which, what the files look like on disk, and a decision table at the end. If you want the basics first (what 4-bit means, what Q4_K_M stands for), read LLM quantization explained.
What each format is
GGUF is a single-file container defined by the llama.cpp project. One .gguf file holds the quantized weights, the tokenizer, the chat template and the model metadata. Its quantization types range from 2-bit to 8-bit (Q2_K up to Q8_0), and the k-quant types spend extra bits on the most sensitive tensors. You can quantize without any calibration data, or pass an importance matrix (llama-imatrix) computed from sample text to protect the weights that matter most.
GPTQ comes from the 2022 paper GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers. It quantizes one layer at a time and uses approximate second-order information to correct the error each rounding step introduces. The authors report quantizing a 175B model to 3 or 4 bits in about four GPU hours with little accuracy loss. It needs a calibration dataset, typically a few hundred text samples.
AWQ (Activation-aware Weight Quantization) comes from Lin et al., 2023, which won the MLSys 2024 best paper award. Its observation is that about 1% of weights matter far more than the rest, and that you find them by looking at activations, not weight values. It scales those channels before quantizing so they lose less precision. It also needs calibration data, but no backpropagation, and the paper reports that it outperforms earlier methods, including GPTQ, on its language modeling and domain benchmarks.
Two more you will meet:
- FP8 stores weights (and usually activations) as 8-bit floating point. vLLM can quantize a 16-bit model to FP8 at load time with
--quantization fp8, no calibration needed. Full FP8 compute needs NVIDIA compute capability 8.9 or higher (Ada Lovelace, Hopper and newer), according to the vLLM FP8 docs; older GPUs fall back to weight-only FP8. - bitsandbytes quantizes on the fly when Hugging Face Transformers loads a model (
load_in_4bit=True, usually with the NF4 type from QLoRA). The Transformers docs recommend NF4 for training 4-bit models. It is the standard way to fine-tune with QLoRA, but it is not a serving format you should pick for throughput.
What the files look like on disk
Qwen publishes official GGUF, AWQ and GPTQ versions of Qwen2.5-7B-Instruct, which makes a fair comparison (sizes from the Hugging Face repos, November 2026):
| Version | Files | Size on disk |
|---|---|---|
| GGUF Q4_K_M | .gguf (split in 2 parts) |
4.7 GB |
| AWQ 4-bit | safetensors + tokenizer files | 5.6 GB |
| GPTQ Int4 | safetensors + tokenizer files | 5.6 GB |
| GGUF Q8_0 | .gguf (split in 3 parts) |
8.1 GB |
The AWQ and GPTQ files are almost a gigabyte larger than Q4_K_M at nominally the same 4 bits. The gap comes mostly from the embedding and output layers, which these checkpoints keep at 16-bit while GGUF quantizes them too. Qwen2.5-7B has a 152,000-token vocabulary, so those two layers alone are about 2 GB at 16-bit. Bits per weight in the name is only an approximation of the file you download.
Memory at runtime is the file size plus the KV cache, covered in GPU memory for LLM inference.
Which engines load which format
This is the part that usually decides the question. Support changes quickly, so the table reflects each project's documentation as of November 2026:
| Engine | GGUF | AWQ | GPTQ | FP8 |
|---|---|---|---|---|
| llama.cpp, Ollama | Native | No | No | No |
| vLLM | Experimental | Yes | Yes | Yes |
| SGLang | Yes (CUDA) | Via Marlin kernels | Via Marlin kernels | Yes |
| Hugging Face Transformers | Loads by dequantizing | Yes | Yes | Yes |
A few details behind those cells:
- vLLM's own GGUF page calls its support "highly experimental and under-optimized" and says it loads only single-file GGUF models. Split files like the Qwen ones above must be merged with
gguf-splitfirst, and the docs suggest passing the original model's tokenizer with--tokenizer. - vLLM reads AWQ and GPTQ configuration from the checkpoint, so
vllm serve Qwen/Qwen2.5-7B-Instruct-AWQneeds no extra flag. Its quantization hardware table lists which GPU generations support each method; AWQ, for example, is not listed for Volta. - SGLang's quantization docs now route AWQ and GPTQ checkpoints through
awq_marlinandgptq_marlinon CUDA, and list GGUF as supported on CUDA. - Hugging Face Text Generation Inference (TGI) also served AWQ and GPTQ, but its documentation now says the project is in maintenance mode. Do not start a new deployment on it.
On Apple Silicon, GGUF through llama.cpp's Metal backend is the practical choice; vLLM and SGLang target NVIDIA and AMD GPUs. On a CPU-only server, GGUF is the only one of the three that runs well at all.
How each one is produced
GGUF takes two commands from a llama.cpp checkout:
python convert_hf_to_gguf.py ./Qwen2.5-7B-Instruct --outtype f16 --outfile qwen-7b-f16.gguf
./llama-quantize qwen-7b-f16.gguf qwen-7b-Q4_K_M.gguf Q4_K_M
A 7B model converts in minutes on a laptop. Adding an importance matrix (llama-imatrix on a few hundred kilobytes of representative text, then --imatrix on llama-quantize) improves the low-bit types noticeably and costs a little more time.
AWQ and GPTQ need a GPU, the full 16-bit model in memory and a calibration set. The original AutoAWQ project is archived, and vLLM's AutoAWQ page points to llm-compressor, which now produces AWQ, GPTQ and FP8 checkpoints. Calibration data matters: quantize a model for legal drafting with generic web text and you protect the wrong weights. Use samples that look like your real prompts.
In practice most teams never quantize anything. Model publishers and the community post GGUF, AWQ and FP8 versions of popular models within days of release. Check that the uploader is the model's original publisher or a source you trust, because a quantized file is a new binary that nobody has reviewed.
Quality and speed: what the evidence says
Quality. At 4 bits, all three formats land close to each other and close to the original on general benchmarks. The AWQ paper reports it beating GPTQ in its tests, and GGUF's importance matrix applies the same idea of protecting the weights that matter most. These differences are smaller than the gap between 4-bit and 8-bit, and much smaller than the gap between a 14B and a 32B model. If quality is tight, move up to 8-bit or FP8 before you agonize over 4-bit methods.
Speed. This depends on the engine far more than on the format:
- One user, one request at a time: llama.cpp with a GGUF file is fast and simple. Token generation is limited by memory bandwidth, and a smaller file means fewer bytes per token.
- Many concurrent users on NVIDIA GPUs: vLLM or SGLang with AWQ, GPTQ or FP8 will usually serve more total tokens per second, because their batching and Marlin kernels are built for that workload. vLLM vs Ollama explains why continuous batching makes the difference.
- Running GGUF inside vLLM gives you vLLM's batching with a less optimized kernel path. It is useful when you only have a GGUF file, not as a first choice for throughput.
Run your own test before committing hardware. Measure time to first token and tokens per second at your expected concurrency, using real prompts, as described in vLLM vs SGLang.
GGUF vs AWQ vs GPTQ: decision table
| Your situation | Use |
|---|---|
| Laptop, Mac, CPU-only server or mixed hardware | GGUF |
| Ollama or llama.cpp already in place | GGUF |
| vLLM or SGLang on Ampere-class NVIDIA GPUs, 4-bit needed to fit | AWQ (or GPTQ if that is what is published) |
| H100, L40S or newer NVIDIA GPUs with memory to spare | FP8 |
| Fine-tuning with QLoRA | bitsandbytes NF4 for training, then export to a serving format |
| Quality is the top priority and it fits | 8-bit (GGUF Q8_0 or FP8) |
Where AI Refinery fits
AI Refinery, a self-hosted LLM platform, ships its catalog models as GGUF files in several quantization variants. Every variant is benchmarked before it reaches the catalog, so you see the size on disk, the GPU memory needed at the full context window and the measured tokens per second, then launch an inference server with one click and call it through an OpenAI-compatible endpoint. That removes the format question for teams that want a working private endpoint on their own GPUs.
It is not the right tool if you specifically need AWQ, GPTQ or FP8 checkpoints, for example to squeeze maximum batched throughput out of a fleet of H100s with a tuned engine build. In that case, run vLLM or SGLang directly with the format that suits your GPUs. The private model catalog page shows what the platform covers.
Engine support for each format changes with every release. Check the current documentation for the exact version you deploy.