---
title: "What is AWQ? Activation-aware weight quantization explained"
description: "AWQ is a 4-bit weight quantization method that uses activation statistics to protect the weights that matter most. How it works and where it is used."
url: https://broadrouter.com/glossary/awq/
lang: en
updated: 2026-11-16
publisher: BroadRouter (a BroadNet site); API by BroadNet Inference, https://inference.broadnet.ai
---

# AWQ (activation-aware weight quantization)

Updated 2026-11-16

AWQ (activation-aware weight quantization) is a post-training method that compresses an LLM's weights to low precision, usually 4-bit integers. It uses activation statistics from a small calibration set to find the most important weight channels and scales them so they lose less accuracy. It needs no backpropagation.

Also called: AWQ, activation-aware weight quantization, AWQ quantization

A 4-bit model is about a quarter the size of its 16-bit original. AWQ is one of the most used ways to get there without losing much quality.

## How it works

Plain [quantization](https://broadrouter.com/glossary/quantization/) rounds every weight to a small set of values. Some weights matter more than others, and rounding them hurts the most. The AWQ paper’s key finding is that “Protecting only 1% salient weights can greatly reduce quantization error”, and that to find those weights “we should refer to the activation distribution, not weights.”

The steps, in outline:

1.  Run a small calibration set through the full-precision model and record activation sizes per channel.
2.  Scale up the salient weight channels with what the paper calls “an equivalent transformation”, so rounding costs them less while the layer’s output stays the same. This avoids “hardware-inefficient mix-precision quantization”.
3.  Round all weights to low precision, typically 4-bit integers with a scale per group of weights.

The paper stresses that “AWQ does not rely on any backpropagation or reconstruction”, so it “generalizes to different domains and modalities” without overfitting the calibration set. It won the MLSys 2024 Best Paper Award.

In vLLM, a pre-quantized AWQ checkpoint loads with one argument:

```python
from vllm import LLM
llm = LLM(model="TheBloke/Llama-2-7b-Chat-AWQ", quantization="auto_awq")
```

vLLM’s page describes the format as reducing weights “from BF16/FP16 to INT4” and warns that “The AutoAWQ library is deprecated”; new AWQ models are made with vLLM’s llm-compressor.

## Why it matters

-   **Memory and cost.** Smaller weights fit on fewer or cheaper GPUs and leave more memory for the [KV cache](https://broadrouter.com/glossary/kv-cache/), which means more requests at once.
-   **Speed.** Decode reads every weight for each token, so smaller weights can mean faster output. The paper reports its TinyChat runtime gives “more than 3x speedup over the Huggingface FP16 implementation”.
-   **Same weights, different results.** Two hosts serving “the same model” may use different quantization. Ask which format a provider serves when you compare quality.

AWQ is weight-only. It is different from [FP8](https://broadrouter.com/glossary/fp8/), which can quantize activations too, and from [GGUF](https://broadrouter.com/glossary/gguf/), a file format with its own quant types for llama.cpp.

## On BroadNet Inference

BroadNet reports that it serves [Gemma 4 VL 26B](https://broadrouter.com/models/gemma-4-vl-26b/) with 4-bit AWQ weights it quantized itself, together with multi-token-prediction speculative decoding (k=2) and an fp8 KV cache. Per its model page, it tested NVFP4 and retired it after a head-to-head. BroadNet also reports that the same Gemma 4 weights served by a hosted provider scored 0.13 lower on average than its own serving (BroadNet does not say on which metric or scale); the [GPU fleet and serving post](https://broadrouter.com/blog/broadnet-gpu-fleet-and-serving/) has the details. BroadNet does not publish the quantization format of its other API models. If you plan to run AWQ weights yourself, the [self-hosting vs API cost guide](https://broadrouter.com/blog/self-hosting-llm-vs-api-cost/) covers the arithmetic.

## Questions

### Is AWQ the same as INT4?

AWQ is a method; INT4 is a number format. AWQ usually produces 4-bit integer weights, so an AWQ model is one kind of INT4 model. Other methods, such as GPTQ or GGUF quant types, can also produce 4-bit weights.

### Does AWQ reduce quality?

Any low-bit quantization can. The AWQ paper reports that protecting about 1% of salient weights greatly reduces quantization error, but you should test the quantized model on your own prompts.

### Can I run AWQ models in vLLM?

Yes. vLLM's docs show loading a pre-quantized AWQ checkpoint. They also note that the AutoAWQ library is deprecated and point to vLLM's llm-compressor project for making new AWQ models.

## See also

-   [Quantization](https://broadrouter.com/glossary/quantization/) (term)
-   [INT4](https://broadrouter.com/glossary/int4/) (term)
-   [FP8](https://broadrouter.com/glossary/fp8/) (term)
-   [GGUF](https://broadrouter.com/glossary/gguf/) (term)
-   [vLLM](https://broadrouter.com/glossary/vllm/) (term)
-   [Open-weight model](https://broadrouter.com/glossary/open-weight-model/) (term)
-   [BroadNet's GPU hardware and serving approach, as BroadNet describes it](https://broadrouter.com/blog/broadnet-gpu-fleet-and-serving/) (article)
-   [Self-hosting an LLM vs an API: the real cost and break-even](https://broadrouter.com/blog/self-hosting-llm-vs-api-cost/) (article)

Try it on your real workload. New accounts start with $5 of credit for Gemma and transcription. DeepSeek, GLM and Qwen3.8 need a top-up.

[Get started](https://inference.broadnet.ai/account/?utm_source=broadrouter&utm_medium=referral&utm_campaign=glossary-awq&utm_content=page-cta) ·

**Sources**

-   [Lin et al., AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration, arXiv 2306.00978 (checked 2026-11-15)](https://arxiv.org/abs/2306.00978)
-   [vLLM docs, AutoAWQ quantization (checked 2026-11-15)](https://docs.vllm.ai/en/latest/features/quantization/auto_awq.html)
-   [BroadNet, Gemma 4 VL 26B model page: serving configuration (checked 2026-11-15)](https://www.broadnet.ai/inference/models/gemma-4-vl-26b)
