---
title: "What is GGUF? The llama.cpp model file format explained"
description: "GGUF is the single-file model format used by llama.cpp and tools like Ollama and LM Studio. What is inside a GGUF file and how to read quant names like Q4_K."
url: https://broadrouter.com/glossary/gguf/
lang: en
updated: 2026-11-15
publisher: BroadRouter (a BroadNet site); API by BroadNet Inference, https://inference.broadnet.ai
---

# GGUF

Updated 2026-11-15

GGUF is a binary file format for storing language models for inference with GGML-based runtimes such as llama.cpp. One GGUF file holds the weights, often quantized, plus the metadata needed to load the model, including tokenizer and architecture details, and it can be memory-mapped for fast loading.

Also called: GGUF file, GGUF format, llama.cpp format

If you have downloaded a model to run on a laptop, it was probably a file ending in `.gguf`.

## How it works

The ggml specification describes GGUF as a binary format for storing models for inference with GGML and GGML-based executors, designed for fast loading and saving. It replaced the older GGML, GGMF and GGJT formats. Its stated goals include:

-   **Single file.** A model can be shared and loaded with no extra files.
-   **mmap compatible.** The runtime can memory-map the weights instead of copying them.
-   **Extensible.** New metadata can be added without breaking older models.
-   **Full information.** Everything needed to load the model is in the file.

A GGUF file has a header (a magic number, the format version, tensor and metadata counts), typed metadata key-value pairs, a table of tensor names, shapes, types and offsets, padding to an alignment boundary (32 bytes by default), then the tensor data.

File names follow a convention. The spec’s example `Mixtral-8x7B-v0.1-KQ2.gguf` breaks down into base name, size label (eight experts of 7B, see [mixture of experts](https://broadrouter.com/glossary/mixture-of-experts/)), version and encoding.

Most GGUF files are [quantized](https://broadrouter.com/glossary/quantization/). Hugging Face’s table lists the common types:

| Type | Bits per weight | Notes |
| --- | --- | --- |
| Q8\_0 | 8 | Legacy round-to-nearest, 32-weight blocks |
| Q6\_K | 6.5625 | k-quant super-blocks |
| Q5\_K | 5.5 | k-quant super-blocks |
| Q4\_K | 4.5 | k-quant super-blocks |
| IQ4\_XS | 4.25 | Uses an importance matrix |
| Q2\_K | 2.625 | k-quant super-blocks |

## Why it matters

-   **Local and edge use.** Hugging Face lists llama.cpp, LM Studio, GPT4All and Ollama as tools that use GGUF. It is the usual route to run an [open-weight model](https://broadrouter.com/glossary/open-weight-model/) on a CPU, a Mac or a single consumer GPU.
-   **The tokenizer travels with the weights.** Because metadata is in the file, a wrong or missing [chat template](https://broadrouter.com/glossary/chat-template/) or [tokenizer](https://broadrouter.com/glossary/tokenizer/) setting in a conversion can change output quality.
-   **Quality depends on the quant type.** A Q2 file and a Q8 file of the same model are not the same model in practice. Compare like with like.
-   **Different from GPU serving formats.** Server engines such as [vLLM](https://broadrouter.com/glossary/vllm/) list GGUF among supported formats, but production GPU serving more often uses formats such as [AWQ](https://broadrouter.com/glossary/awq/) or FP8.

## On BroadNet Inference

BroadNet’s API serves models over HTTP; it does not hand out GGUF files, and you never load weights yourself. BroadNet’s catalogue reports that in its lab it has run [Qwen3.8 27B](https://broadrouter.com/models/qwen3.8-27b/) as GGUF Q4\_K\_M with a vision projector and multi-token-prediction drafter on llama.cpp, alongside an AWQ INT4 build on vLLM for AMD ROCm. It does not say which format serves the public API. If you are weighing a local GGUF setup against an API, the [self-hosting vs API cost guide](https://broadrouter.com/blog/self-hosting-llm-vs-api-cost/) and [open-weight vs closed models](https://broadrouter.com/blog/open-weight-vs-closed-models/) cover the trade-offs.

## Questions

### What does a name like Q4\_K mean?

It is the quantization type. Hugging Face's table describes Q4\_K as 4-bit quantization in super-blocks of 8 blocks of 32 weights, about 4.5 bits per weight. Fewer bits use less memory and usually cost more quality.

### Can I call a GGUF model through the OpenAI API format?

Not the file itself. You load it into a runtime such as llama.cpp's server, Ollama or LM Studio, and that runtime serves it. Check the runtime's docs for the API it exposes.

### Is GGUF the same as safetensors?

No. Hugging Face describes safetensors as a tensor-only format, while GGUF encodes both the tensors and a standardized set of metadata in one file.

## See also

-   [Quantization](https://broadrouter.com/glossary/quantization/) (term)
-   [AWQ (activation-aware weight quantization)](https://broadrouter.com/glossary/awq/) (term)
-   [INT4](https://broadrouter.com/glossary/int4/) (term)
-   [Open-weight model](https://broadrouter.com/glossary/open-weight-model/) (term)
-   [Tokenizer](https://broadrouter.com/glossary/tokenizer/) (term)
-   [vLLM](https://broadrouter.com/glossary/vllm/) (term)
-   [Self-hosting an LLM vs an API: the real cost and break-even](https://broadrouter.com/blog/self-hosting-llm-vs-api-cost/) (article)
-   [Open-weight vs closed AI models for business: cost, control, risk](https://broadrouter.com/blog/open-weight-vs-closed-models/) (article)

Try it on your real workload. New accounts start with $5 of credit for Gemma and transcription. DeepSeek, GLM and Qwen3.8 need a top-up.

[Get started](https://inference.broadnet.ai/account/?utm_source=broadrouter&utm_medium=referral&utm_campaign=glossary-gguf&utm_content=page-cta) ·

**Sources**

-   [ggml project, GGUF specification, docs/gguf.md (checked 2026-11-15)](https://github.com/ggml-org/ggml/blob/master/docs/gguf.md)
-   [Hugging Face Hub docs, GGUF (quantization types table) (checked 2026-11-15)](https://huggingface.co/docs/hub/gguf)
-   [BroadNet model catalogue: lab serving configurations (checked 2026-11-15)](https://www.broadnet.ai/inference/models)
