BroadRouter
Get started

Inference provider

Updated

An inference provider is a company that runs machine learning models, often open-weight LLMs, on GPUs it owns, leases or rents, and sells access to them through an API, usually priced per token, per request or per unit of time. It handles serving, scaling and hardware.

Also called: LLM inference provider, LLM API provider, model hosting provider, AI inference provider

When an open-weight model is released, someone still has to run it. Inference providers are the companies that do.

How it works

A provider loads model weights onto GPUs, runs serving software that batches requests and manages memory, and puts an API in front. Most expose an OpenAI-compatible API, so you switch providers by changing the base URL, key and model ID.

The term covers a few business shapes:

  • Model labs that serve their own models.
  • Independent hosts that serve open-weight models from many labs.
  • Clouds that rent GPUs or managed endpoints, sometimes billed per GPU-hour.
  • Aggregators. Hugging Face’s Inference Providers give access to “hundreds of machine learning models, powered by world-class inference providers” through one token. An LLM router works the same way.

Because the API shape is shared, the same code runs on any of them. Only three values change:

import os
from openai import OpenAI

msgs = [{"role": "user", "content": "Hello"}]
client = OpenAI(
    base_url="https://inference.broadnet.ai/v1",          # the provider
    api_key=os.environ["BROADNET_API_KEY"],  # its key
)
client.chat.completions.create(model="broadnet/deepseek-v4.1-flash", messages=msgs)  # its model ID

Pricing differs too: per token, per audio minute, per hour, or flat per session.

Why it matters

Picking a provider is mostly picking trade-offs you cannot see in the model name:

  • Data handling. Retention, logging and training use vary. Look for a written zero data retention policy.
  • Where it runs. Region and subprocessors matter for compliance.
  • Limits. Concurrency, rate limits and context length can be lower than the model’s maximum.
  • Serving choices. Precision and serving settings affect speed and sometimes output.
  • Price shape. The right shape depends on your traffic; see the provider pricing comparison.
  • Price level. The same open model can cost several times more at one provider than another. See same model, different host.

On BroadNet Inference

BroadNet Inference is an inference provider. Its privacy statement says its own models and the open-weight models it hosts “run on BroadNet-owned or leased hardware in the United States, with partner capacity as backup”, and that models served by partners run under zero-data-retention terms with US inference partners. Request and response content is not retained.

It offers chat models, including DeepSeek V4.1 Flash and GLM 5.3, and two speech-to-text models through an OpenAI-compatible API at https://inference.broadnet.ai/v1. Billing is pay-as-you-go per token or per audio minute, an hourly key, or flat plans for DeepSeek V4.1 Flash. Use the provider checklist to compare it with others on your own criteria.

Questions

What is the difference between an inference provider and an LLM router?

A provider runs the models itself, on hardware it owns, leases or rents. A router forwards your request to one of several providers through one API and one bill. Some companies do both.

Do two providers serving the same open model give the same answers?

Not always. Precision, context limits, default settings and the serving software can differ, so outputs can differ. Test on your own prompts before switching.

How do I switch inference providers?

With an OpenAI-compatible API, change three values: the base URL, the API key and the model ID. Then rerun your eval, because the same model can behave differently on another provider.

Read as MarkdownAsk ChatGPTAsk ClaudeAsk Perplexity