---
title: "What is an LLM router? Aggregators, gateways and routing"
description: "An LLM router picks which model or provider serves each request. How hosted aggregators, self-hosted gateways and model-choice routers work, and their trade-offs."
url: https://broadrouter.com/glossary/llm-router/
lang: en
updated: 2026-11-16
publisher: BroadRouter (a BroadNet site); API by BroadNet Inference, https://inference.broadnet.ai
---

# LLM router

Updated 2026-11-16

An LLM router is software or a service that sits between an application and one or more language model APIs and decides, for each request, which model or provider handles it, using rules such as cost, latency, availability or the type of task.

Also called: AI gateway, LLM gateway, model router, LLM aggregator

An application that talks to one model needs no router. Routers appear when a team uses several models or providers and wants one place to choose between them.

## How it works

The application sends an OpenAI-style request to the router. The router applies its rules, forwards the request to a backend, and returns the answer in the same format. There are three common kinds:

-   **Hosted aggregators.** OpenRouter, for example, offers “hundreds of AI models through a single API endpoint” at `https://openrouter.ai/api/v1`, with automatic fallbacks. You hold one key and one bill; the aggregator holds accounts with the providers.
-   **Self-hosted gateways.** The LiteLLM Router runs in your own infrastructure. It spreads traffic across deployments (simple shuffle by default, or latency-, usage- or cost-based), retries rate-limit errors with exponential backoff, and puts a deployment in cooldown after a 429 or a high failure rate.
-   **Model-choice routers.** Research such as RouteLLM trains a classifier that sends easy prompts to a cheaper model and hard ones to a stronger model, to cut cost without losing quality.

A self-hosted gateway is configured as a list of backends. This LiteLLM entry adds one [OpenAI-compatible](https://broadrouter.com/glossary/openai-compatible-api/) provider:

```yaml
model_list:
  - model_name: flash
    litellm_params:
      model: openai/broadnet/deepseek-v4.1-flash
      api_base: https://inference.broadnet.ai/v1
      api_key: os.environ/BROADNET_API_KEY
```

## Why it matters

-   **Reliability.** A router can fail over when one provider returns 429 or 5xx errors. See [rate limits](https://broadrouter.com/glossary/rate-limit/).
-   **Cost.** Sending simple prompts to a cheaper model can lower the bill.
-   **One integration.** Code talks to one [base URL](https://broadrouter.com/glossary/base-url/) and one key format.

The trade-offs are real. A router adds a network hop. A hosted router is one more party that sees your prompts, so retention depends on both the router and the provider it picks. Models also differ in tokenizers, reasoning behavior and parameter support, so a silent fallback can change answers.

## On BroadNet Inference

BroadNet Inference is not a router. It is a single provider with one short [model list](https://broadrouter.com/models/) at `https://inference.broadnet.ai/v1`; BroadNet’s Terms say its models run on BroadNet-owned or leased US hardware or with US inference partners. There is no automatic routing or fallback across providers. You choose the model in each request, for example `broadnet/deepseek-v4.1-flash` or `broadnet/glm-5.3`.

Because the API is OpenAI-compatible, it can sit behind a router as one backend. The [LiteLLM proxy guide](https://broadrouter.com/guides/litellm-proxy-custom-provider/) shows the setup. For how a router and a flat-rate provider compare, read [OpenRouter alternative for agents](https://broadrouter.com/blog/openrouter-alternative-flat-rate/). BroadNet does not retain prompts or outputs; if you add a hosted router in front, check its [data retention](https://broadrouter.com/glossary/zero-data-retention/) too.

## Common mistakes

-   Retrying a 401 or 403 on another provider. Those are key or policy errors; fix them first. See [errors and retries](https://broadrouter.com/guides/errors-and-retries/).
-   Treating model names as interchangeable. Test each fallback model on your own prompts.

## Questions

### Is an LLM router the same as a load balancer?

A load balancer is one part of a router. A router can also pick a different model for a task, fall back to another provider after an error, and translate between API formats. A plain load balancer only spreads traffic across identical servers.

### Does a router add latency?

Usually a little, because the request makes one extra network hop and the router may inspect it. Fallbacks can add more, since a failed attempt runs before the retry. Measure time to first token with and without the router.

### Is BroadNet Inference an LLM router?

No. Despite this site's name, BroadNet Inference is a single provider; BroadNet's Terms say its models run on BroadNet-owned or leased US hardware or with US inference partners. You pick the model in each request. You can place it behind a router as one of the providers.

## See also

-   [OpenAI-compatible API](https://broadrouter.com/glossary/openai-compatible-api/) (term)
-   [Base URL](https://broadrouter.com/glossary/base-url/) (term)
-   [Rate limit](https://broadrouter.com/glossary/rate-limit/) (term)
-   [Zero data retention](https://broadrouter.com/glossary/zero-data-retention/) (term)
-   [Inference provider](https://broadrouter.com/glossary/inference-provider/) (term)
-   [Model fallback](https://broadrouter.com/glossary/model-fallback/) (term)
-   [Route LiteLLM to BroadNet Inference (SDK and proxy)](https://broadrouter.com/guides/litellm-proxy-custom-provider/) (guide)
-   [BroadNet Inference error codes and when to retry](https://broadrouter.com/guides/errors-and-retries/) (guide)
-   [OpenRouter vs a flat-rate plan for AI agents: which costs less?](https://broadrouter.com/blog/openrouter-alternative-flat-rate/) (article)
-   [Switching to an OpenAI-compatible API: a migration checklist](https://broadrouter.com/blog/openai-compatible-api-migration-checklist/) (article)

Try it on your real workload. New accounts start with $5 of credit for Gemma and transcription. DeepSeek, GLM and Qwen3.8 need a top-up.

[Get started](https://inference.broadnet.ai/account/?utm_source=broadrouter&utm_medium=referral&utm_campaign=glossary-llm-router&utm_content=page-cta) ·

**Sources**

-   [OpenRouter quickstart (one endpoint for many models)](https://openrouter.ai/docs/quickstart)
-   [LiteLLM Router docs, load balancing, fallbacks and cooldowns](https://docs.litellm.ai/docs/routing)
-   [RouteLLM: Learning to Route LLMs with Preference Data (arXiv:2406.18665)](https://arxiv.org/abs/2406.18665)
-   [BroadNet Inference API reference](https://inference.broadnet.ai/docs?utm_source=broadrouter&utm_medium=referral&utm_campaign=glossary-llm-router&utm_content=sources)
-   [BroadNet Terms of Service, §2 hosting (checked 2026-11-16)](https://www.broadnet.ai/terms)
