BroadRouter
Get started

vLLM

Updated

vLLM is an open-source library for LLM inference and serving, first developed at UC Berkeley's Sky Computing Lab. It manages KV cache memory with PagedAttention, batches incoming requests continuously, supports many quantization formats and exposes an OpenAI-compatible HTTP server.

Also called: vLLM server, vLLM serve

When people self-host an open-weight model on a GPU server, vLLM is one of the first engines they try.

How it works

The project’s docs describe vLLM as “a fast and easy-to-use library for LLM inference and serving”, “Originally developed in the Sky Computing Lab at UC Berkeley”. Its core ideas:

  • PagedAttention. The 2023 paper behind vLLM manages KV cache memory with a method “inspired by the classical virtual memory and paging techniques in operating systems”. It reports “near-zero waste in KV cache memory” and 2 to 4 times the throughput of FasterTransformer and Orca at similar latency.
  • Continuous batching. Incoming requests are batched dynamically instead of waiting for a fixed batch. The docs list chunked prefill and prefix caching alongside it; see prefill and decode.
  • Many formats. The docs list FP8, INT8, INT4, GPTQ/AWQ, GGUF and others for quantization.
  • Hardware. NVIDIA and AMD GPUs and several CPU families, with plugins for other accelerators.
  • An OpenAI-compatible server. Start it with vllm serve and call it like an OpenAI-compatible API.

A minimal client call to your own server:

from openai import OpenAI
client = OpenAI(base_url="http://YOUR_VLLM_HOST/v1", api_key="unused")
r = client.chat.completions.create(
    model="your-model",
    messages=[{"role": "user", "content": "Hello"}])

The server needs a chat template for the model. The serving docs warn: “Without a chat template, the server will not be able to process chat and all chat requests will error.” You can pass one with --chat-template.

Why it matters

  • Self-hosting baseline. If you compare an API price with running your own GPUs, vLLM is a common engine to measure with. Our self-hosting vs API cost guide suggests testing your real prompts at real concurrency with vLLM or SGLang.
  • Serving changes results. The same weights can be faster or slower, and score differently, depending on engine version, quantization and settings.
  • A shared vocabulary. Terms such as PagedAttention, chunked prefill and speculative decoding are easier to follow once you know where vLLM uses them.

On BroadNet Inference

BroadNet does not say which engine serves its public API, so this page does not claim one. What BroadNet has published: a September 2026 preprint, Uno in vLLM, that adds IFM’s Uno self-drafting adapter to vLLM and reports 2.58 times plain-vLLM throughput at one stream and 1.74 times at 32 on an H100 with Qwen3-8B. Its code is public in BroadNet’s vllm-uno repository. BroadNet’s catalogue also lists an AWQ INT4 build of Qwen3.8 27B on its patched vLLM for AMD ROCm among lab configurations. For how BroadNet describes its fleet, see its GPU hardware and serving post.

Questions

Is vLLM an API provider?

No. vLLM is software you run on your own hardware. Using it directly means you operate the GPUs, scaling and upgrades yourself.

Can I point the OpenAI SDK at a vLLM server?

Yes. vLLM exposes an OpenAI-compatible server, so you set the SDK's base URL to your server and send chat completion requests. The model needs a chat template, or chat requests fail.

Does BroadNet Inference run on vLLM?

BroadNet does not say which engine serves its public API. It has published a vLLM research implementation (Uno in vLLM) and lists a patched vLLM build for AMD ROCm among its lab configurations.

Read as MarkdownAsk ChatGPTAsk ClaudeAsk Perplexity