BroadRouter
Get started

AWQ (activation-aware weight quantization)

Updated

AWQ (activation-aware weight quantization) is a post-training method that compresses an LLM's weights to low precision, usually 4-bit integers. It uses activation statistics from a small calibration set to find the most important weight channels and scales them so they lose less accuracy. It needs no backpropagation.

Also called: AWQ, activation-aware weight quantization, AWQ quantization

A 4-bit model is about a quarter the size of its 16-bit original. AWQ is one of the most used ways to get there without losing much quality.

How it works

Plain quantization rounds every weight to a small set of values. Some weights matter more than others, and rounding them hurts the most. The AWQ paper’s key finding is that “Protecting only 1% salient weights can greatly reduce quantization error”, and that to find those weights “we should refer to the activation distribution, not weights.”

The steps, in outline:

  1. Run a small calibration set through the full-precision model and record activation sizes per channel.
  2. Scale up the salient weight channels with what the paper calls “an equivalent transformation”, so rounding costs them less while the layer’s output stays the same. This avoids “hardware-inefficient mix-precision quantization”.
  3. Round all weights to low precision, typically 4-bit integers with a scale per group of weights.

The paper stresses that “AWQ does not rely on any backpropagation or reconstruction”, so it “generalizes to different domains and modalities” without overfitting the calibration set. It won the MLSys 2024 Best Paper Award.

In vLLM, a pre-quantized AWQ checkpoint loads with one argument:

from vllm import LLM
llm = LLM(model="TheBloke/Llama-2-7b-Chat-AWQ", quantization="auto_awq")

vLLM’s page describes the format as reducing weights “from BF16/FP16 to INT4” and warns that “The AutoAWQ library is deprecated”; new AWQ models are made with vLLM’s llm-compressor.

Why it matters

  • Memory and cost. Smaller weights fit on fewer or cheaper GPUs and leave more memory for the KV cache, which means more requests at once.
  • Speed. Decode reads every weight for each token, so smaller weights can mean faster output. The paper reports its TinyChat runtime gives “more than 3x speedup over the Huggingface FP16 implementation”.
  • Same weights, different results. Two hosts serving “the same model” may use different quantization. Ask which format a provider serves when you compare quality.

AWQ is weight-only. It is different from FP8, which can quantize activations too, and from GGUF, a file format with its own quant types for llama.cpp.

On BroadNet Inference

BroadNet reports that it serves Gemma 4 VL 26B with 4-bit AWQ weights it quantized itself, together with multi-token-prediction speculative decoding (k=2) and an fp8 KV cache. Per its model page, it tested NVFP4 and retired it after a head-to-head. BroadNet also reports that the same Gemma 4 weights served by a hosted provider scored 0.13 lower on average than its own serving (BroadNet does not say on which metric or scale); the GPU fleet and serving post has the details. BroadNet does not publish the quantization format of its other API models. If you plan to run AWQ weights yourself, the self-hosting vs API cost guide covers the arithmetic.

Questions

Is AWQ the same as INT4?

AWQ is a method; INT4 is a number format. AWQ usually produces 4-bit integer weights, so an AWQ model is one kind of INT4 model. Other methods, such as GPTQ or GGUF quant types, can also produce 4-bit weights.

Does AWQ reduce quality?

Any low-bit quantization can. The AWQ paper reports that protecting about 1% of salient weights greatly reduces quantization error, but you should test the quantized model on your own prompts.

Can I run AWQ models in vLLM?

Yes. vLLM's docs show loading a pre-quantized AWQ checkpoint. They also note that the AutoAWQ library is deprecated and point to vLLM's llm-compressor project for making new AWQ models.

Read as MarkdownAsk ChatGPTAsk ClaudeAsk Perplexity