If you have downloaded a model to run on a laptop, it was probably a file ending in .gguf.
How it works
The ggml specification describes GGUF as a binary format for storing models for inference with GGML and GGML-based executors, designed for fast loading and saving. It replaced the older GGML, GGMF and GGJT formats. Its stated goals include:
- Single file. A model can be shared and loaded with no extra files.
- mmap compatible. The runtime can memory-map the weights instead of copying them.
- Extensible. New metadata can be added without breaking older models.
- Full information. Everything needed to load the model is in the file.
A GGUF file has a header (a magic number, the format version, tensor and metadata counts), typed metadata key-value pairs, a table of tensor names, shapes, types and offsets, padding to an alignment boundary (32 bytes by default), then the tensor data.
File names follow a convention. The spec’s example Mixtral-8x7B-v0.1-KQ2.gguf breaks down into base name, size label (eight experts of 7B, see mixture of experts), version and encoding.
Most GGUF files are quantized. Hugging Face’s table lists the common types:
| Type | Bits per weight | Notes |
|---|---|---|
| Q8_0 | 8 | Legacy round-to-nearest, 32-weight blocks |
| Q6_K | 6.5625 | k-quant super-blocks |
| Q5_K | 5.5 | k-quant super-blocks |
| Q4_K | 4.5 | k-quant super-blocks |
| IQ4_XS | 4.25 | Uses an importance matrix |
| Q2_K | 2.625 | k-quant super-blocks |
Why it matters
- Local and edge use. Hugging Face lists llama.cpp, LM Studio, GPT4All and Ollama as tools that use GGUF. It is the usual route to run an open-weight model on a CPU, a Mac or a single consumer GPU.
- The tokenizer travels with the weights. Because metadata is in the file, a wrong or missing chat template or tokenizer setting in a conversion can change output quality.
- Quality depends on the quant type. A Q2 file and a Q8 file of the same model are not the same model in practice. Compare like with like.
- Different from GPU serving formats. Server engines such as vLLM list GGUF among supported formats, but production GPU serving more often uses formats such as AWQ or FP8.
On BroadNet Inference
BroadNet’s API serves models over HTTP; it does not hand out GGUF files, and you never load weights yourself. BroadNet’s catalogue reports that in its lab it has run Qwen3.8 27B as GGUF Q4_K_M with a vision projector and multi-token-prediction drafter on llama.cpp, alongside an AWQ INT4 build on vLLM for AMD ROCm. It does not say which format serves the public API. If you are weighing a local GGUF setup against an API, the self-hosting vs API cost guide and open-weight vs closed models cover the trade-offs.