When people self-host an open-weight model on a GPU server, vLLM is one of the first engines they try.
How it works
The project’s docs describe vLLM as “a fast and easy-to-use library for LLM inference and serving”, “Originally developed in the Sky Computing Lab at UC Berkeley”. Its core ideas:
- PagedAttention. The 2023 paper behind vLLM manages KV cache memory with a method “inspired by the classical virtual memory and paging techniques in operating systems”. It reports “near-zero waste in KV cache memory” and 2 to 4 times the throughput of FasterTransformer and Orca at similar latency.
- Continuous batching. Incoming requests are batched dynamically instead of waiting for a fixed batch. The docs list chunked prefill and prefix caching alongside it; see prefill and decode.
- Many formats. The docs list FP8, INT8, INT4, GPTQ/AWQ, GGUF and others for quantization.
- Hardware. NVIDIA and AMD GPUs and several CPU families, with plugins for other accelerators.
- An OpenAI-compatible server. Start it with
vllm serveand call it like an OpenAI-compatible API.
A minimal client call to your own server:
from openai import OpenAI
client = OpenAI(base_url="http://YOUR_VLLM_HOST/v1", api_key="unused")
r = client.chat.completions.create(
model="your-model",
messages=[{"role": "user", "content": "Hello"}])
The server needs a chat template for the model. The serving docs warn: “Without a chat template, the server will not be able to process chat and all chat requests will error.” You can pass one with --chat-template.
Why it matters
- Self-hosting baseline. If you compare an API price with running your own GPUs, vLLM is a common engine to measure with. Our self-hosting vs API cost guide suggests testing your real prompts at real concurrency with vLLM or SGLang.
- Serving changes results. The same weights can be faster or slower, and score differently, depending on engine version, quantization and settings.
- A shared vocabulary. Terms such as PagedAttention, chunked prefill and speculative decoding are easier to follow once you know where vLLM uses them.
On BroadNet Inference
BroadNet does not say which engine serves its public API, so this page does not claim one. What BroadNet has published: a September 2026 preprint, Uno in vLLM, that adds IFM’s Uno self-drafting adapter to vLLM and reports 2.58 times plain-vLLM throughput at one stream and 1.74 times at 32 on an H100 with Qwen3-8B. Its code is public in BroadNet’s vllm-uno repository. BroadNet’s catalogue also lists an AWQ INT4 build of Qwen3.8 27B on its patched vLLM for AMD ROCm among lab configurations. For how BroadNet describes its fleet, see its GPU hardware and serving post.