When an open-weight model is released, someone still has to run it. Inference providers are the companies that do.
How it works
A provider loads model weights onto GPUs, runs serving software that batches requests and manages memory, and puts an API in front. Most expose an OpenAI-compatible API, so you switch providers by changing the base URL, key and model ID.
The term covers a few business shapes:
- Model labs that serve their own models.
- Independent hosts that serve open-weight models from many labs.
- Clouds that rent GPUs or managed endpoints, sometimes billed per GPU-hour.
- Aggregators. Hugging Face’s Inference Providers give access to “hundreds of machine learning models, powered by world-class inference providers” through one token. An LLM router works the same way.
Because the API shape is shared, the same code runs on any of them. Only three values change:
import os
from openai import OpenAI
msgs = [{"role": "user", "content": "Hello"}]
client = OpenAI(
base_url="https://inference.broadnet.ai/v1", # the provider
api_key=os.environ["BROADNET_API_KEY"], # its key
)
client.chat.completions.create(model="broadnet/deepseek-v4.1-flash", messages=msgs) # its model ID
Pricing differs too: per token, per audio minute, per hour, or flat per session.
Why it matters
Picking a provider is mostly picking trade-offs you cannot see in the model name:
- Data handling. Retention, logging and training use vary. Look for a written zero data retention policy.
- Where it runs. Region and subprocessors matter for compliance.
- Limits. Concurrency, rate limits and context length can be lower than the model’s maximum.
- Serving choices. Precision and serving settings affect speed and sometimes output.
- Price shape. The right shape depends on your traffic; see the provider pricing comparison.
- Price level. The same open model can cost several times more at one provider than another. See same model, different host.
On BroadNet Inference
BroadNet Inference is an inference provider. Its privacy statement says its own models and the open-weight models it hosts “run on BroadNet-owned or leased hardware in the United States, with partner capacity as backup”, and that models served by partners run under zero-data-retention terms with US inference partners. Request and response content is not retained.
It offers chat models, including DeepSeek V4.1 Flash and GLM 5.3, and two speech-to-text models through an OpenAI-compatible API at https://inference.broadnet.ai/v1. Billing is pay-as-you-go per token or per audio minute, an hourly key, or flat plans for DeepSeek V4.1 Flash. Use the provider checklist to compare it with others on your own criteria.