BroadRouter
Get started

A private endpoint starts with a clear workload

By the BroadNet Inference team · 7 min read · Published

Before you ask for a private endpoint, write down the workload: requests per day and at peak, tokens in and out, the concurrency that implies, the context you need, the processing region, the model and its licence, and a budget. Agree region and limits in writing, then smoke-test the issued endpoint before real data.

“We need a dedicated endpoint” is the start of a conversation, not a specification. The provider can’t size, place or price capacity without your numbers, and you can’t judge their answer without them either. This post lists what to define before you ask, using a worked invoice-extraction example, and ends with a script that checks the endpoint you are given.

It applies to any provider. Where it describes BroadNet’s offer, it sticks to what BroadNet says publicly, checked on 15 November 2026. Anything beyond that is a question for BroadNet.

When does a private endpoint make sense?

Usually for one of four reasons:

  • Traffic above self-serve limits. BroadNet’s API reference publishes 30 requests per minute and 4 requests in flight for a pay-as-you-go key. A steady workload above that is a capacity question, not a retry question. (Handling 429s covers short bursts.)
  • A model that isn’t on the public list. BroadNet’s catalogue marks some models “available on request”, including Qwen3-VL 30B-A3B, Qwen3-VL 8B and a BroadNet fine-tune of Cohere Transcribe Arabic. Reading a model catalogue explains the difference.
  • A processing location you must control. Data-residency rules, such as those in the Saudi and UAE PDPL checklist, may require a region you have agreed in writing.
  • Predictable latency. Shared capacity varies with other users’ load. Reserved capacity removes that variable, though not network distance or your own prompt sizes.

If none of these apply, a self-serve key is simpler. And if your volume is very high and steady, compare against running it yourself: self-hosting vs API cost has the method.

What should you define about the workload?

Write a one-page brief. These are the numbers a provider needs:

Item What to write down Example
Task What the model does, in one sentence Read invoice photos, return JSON fields
Volume Requests per day, and the share in the busiest hour 40,000 a day, 15% in the peak hour
Tokens Average and 95th-percentile input and output tokens 1,800 in (one page image + prompt), 350 out
Context Largest single request, prompt plus answer 4,000 tokens
Latency Acceptable time to first token and to full answer Full answer within 10 seconds
Streaming Needed or not Not needed (batch-style)
Inputs Text, images, audio; formats and sizes JPEG and PNG, one page each
Languages Which, and the share of each Arabic and English, 60/40
Growth Expected volume in 6 and 12 months Double in 12 months

Measure the token counts on real samples, not guesses. Arabic often needs more tokens than English for the same text; the Arabic token counter and real cost by language show by how much on different tokenizers. For images, the image token calculator gives a first estimate.

How do you turn volume into tokens per second and concurrency?

Peak requests per second is the busiest-hour volume divided by 3,600. Concurrency, the number of requests in flight at once, follows from Little’s law: the average number in a system equals the arrival rate times the average time each one spends there. For an API, that is requests per second × seconds per request.

import math

def size(requests_per_day: int, peak_share_of_day_in_busiest_hour: float,
         in_tokens: int, out_tokens: int, out_tok_per_s_per_stream: float) -> dict:
    """Rough peak numbers to put in a capacity enquiry. Measure your own inputs; these are not provider specs."""
    peak_rps = requests_per_day * peak_share_of_day_in_busiest_hour / 3600
    seconds_per_request = out_tokens / out_tok_per_s_per_stream        # ignores queueing and prompt time
    concurrency = peak_rps * seconds_per_request                      # Little's law: L = λ × W
    return {
        "peak_requests_per_second": round(peak_rps, 2),
        "seconds_per_request": round(seconds_per_request, 1),
        "concurrent_requests_at_peak": math.ceil(concurrency),
        "peak_input_tokens_per_second": round(peak_rps * in_tokens),
        "peak_output_tokens_per_second": round(peak_rps * out_tokens),
        "monthly_tokens_millions": round(requests_per_day * 30 * (in_tokens + out_tokens) / 1e6, 1),
    }

# 40,000 invoice extractions a day, 15% of them in the busiest hour,
# 1,800 tokens in, 350 out, 60 tokens/s per stream measured on a trial.
print(size(40_000, 0.15, 1_800, 350, 60))

It printed 1.67 requests per second at peak, about 5.8 seconds per request, 10 concurrent requests, 3,000 input and 583 output tokens per second, and 2,580 million tokens a month. The 60 tokens per second is an assumption for the example; measure your own on the model you will use, because speed varies by model, prompt length and load. The latency, TTFT and throughput post explains what to time.

Two cautions. The formula ignores prompt processing time, so long prompts or large images make real requests slower and push concurrency up. And averages hide bursts: give the provider your 95th-percentile request size and a short burst profile, not only the mean. This example’s peak (about 100 requests a minute and 10 in flight) is well above the published self-serve limits for one pay-as-you-go key, which is exactly when a capacity conversation is worth having.

What about region, residency and limits?

Agree them explicitly, in writing, before any regulated data moves. BroadNet’s Terms (§7.3a) say: “Contact us to agree processing locations for your deployment before sending data with residency requirements,” and that partner-hosted and backup processing involves other processors, with a current subprocessor list available to customers on request. Its privacy statement says BroadNet’s own and hosted open-weight models run on BroadNet-owned or leased hardware in the United States, with partner capacity as backup, and its overview page says regional processing in MENA or Europe is available on request.

So the questions for any provider are:

  1. Which region processes requests, and which region holds backup capacity?
  2. Which subprocessors can touch the traffic, including for failover or overflow?
  3. What are the endpoint’s rate, concurrency and token limits, and what happens above them?
  4. What availability and support terms apply, and where are they written? BroadNet’s Terms (§7.4) say self-serve has no availability commitment, and that private endpoints and reserved capacity follow your separate written agreement. To see what written SLAs from other providers cover, read LLM API SLAs compared.
  5. What is retained? For BroadNet’s inference API, the published position is: inference content is not retained; necessary account, billing and security metadata is retained as described in its privacy statement. Confirm the same applies to your endpoint.

Which model, and under which licence?

Name the exact model and version, and check its licence permits your use and API serving. BroadNet’s model pages list Apache-2.0 for Gemma 4 26B, Qwen3-VL 30B-A3B and the Cohere Transcribe base weights, with the BroadNet fine-tune of Cohere Transcribe Arabic “under contract”. Its Terms say models are governed by their licences, and that DeepSeek V4.1 Flash’s licence still applies when partner-hosted.

Also agree the context window. Context limits depend on the model and the deployment, not only on the base model’s maximum: BroadNet lists 32K tokens for Gemma 4 VL 26B on its public API and 16K for Qwen3-VL 30B-A3B, which is on request. Use the context window calculator against your largest request, not your average one.

How should you set a budget?

BroadNet publishes no private-endpoint price; its overview page says “Reserved GPU, priced to your traffic. Ask for a quote.” So set your own ceiling first. Price the same workload on self-serve with the LLM API cost calculator, using the monthly token count from the sizing script, and decide what predictable capacity, a region or a model that isn’t public is worth to you on top. A quote below the self-serve figure is a bonus; a quote far above it needs a reason from your list of requirements.

How do you send the enquiry?

BroadNet’s API reference says: “Private endpoints, our Arabic fine-tunes and other models on request are reached with a private ID issued with the endpoint.” Its overview page asks for “your model, languages and traffic requirements”. Send your one-page brief through the contact page, without API keys or customer data. Include the sizing output, the region, the model and licence, the latency target, your budget ceiling and your timeline.

How do you validate the endpoint you’re given?

Before production traffic, test the endpoint against what was agreed. This script uses only the Python standard library. It checks that the private model ID works, that streaming ends with the data: [DONE] marker BroadNet’s docs describe, that the agreed concurrency runs without 429s, and that a wrong model ID is rejected.

import json, os, sys, time, urllib.error, urllib.request
from concurrent.futures import ThreadPoolExecutor

BASE = os.environ["ENDPOINT_BASE_URL"].rstrip("/")    # from your agreement, e.g. https://.../v1
KEY = os.environ["ENDPOINT_API_KEY"]
MODEL = os.environ["ENDPOINT_MODEL_ID"]                # the private ID issued with the endpoint
AGREED_CONCURRENCY = int(os.environ.get("AGREED_CONCURRENCY", "4"))

def post(payload: dict, stream=False):
    req = urllib.request.Request(f"{BASE}/chat/completions", data=json.dumps(payload).encode(),
        headers={"Authorization": f"Bearer {KEY}", "Content-Type": "application/json"})
    t0 = time.monotonic()
    try:
        with urllib.request.urlopen(req, timeout=120) as r:
            body = r.read().decode()
            return r.status, r.headers.get("X-Request-ID"), body, time.monotonic() - t0
    except urllib.error.HTTPError as e:
        return e.code, e.headers.get("X-Request-ID"), e.read().decode(), time.monotonic() - t0

def main() -> int:
    msg = [{"role": "user", "content": "Reply with the word OK."}]
    results = {}
    code, rid, body, secs = post({"model": MODEL, "messages": msg, "max_tokens": 1000})
    ok = code == 200 and "usage" in json.loads(body)
    results["model_id_accepted"] = (ok, f"HTTP {code}, request ID {rid}, {secs:.2f}s")
    code, _, body, _ = post({"model": MODEL, "messages": msg, "stream": True,
                             "stream_options": {"include_usage": True}})
    results["stream_ends_with_DONE"] = (code == 200 and body.rstrip().endswith("data: [DONE]"), f"HTTP {code}")
    with ThreadPoolExecutor(AGREED_CONCURRENCY) as pool:
        codes = list(pool.map(lambda _: post({"model": MODEL, "messages": msg, "max_tokens": 1000})[0],
                              range(AGREED_CONCURRENCY)))
    results["agreed_concurrency_no_429"] = (all(c == 200 for c in codes), f"codes {sorted(codes)}")
    code, _, _, _ = post({"model": MODEL + "-typo", "messages": msg})
    results["unknown_model_rejected"] = (code in (400, 403, 404), f"HTTP {code}")
    for name, (passed, note) in results.items():
        print(f"{'PASS' if passed else 'FAIL'}  {name}: {note}")
    return 0 if all(p for p, _ in results.values()) else 1

if __name__ == "__main__":
    sys.exit(main())

We ran it against a local mock server (no real endpoint) that allows 8 requests in flight. With AGREED_CONCURRENCY=8 all four checks passed. With 12, the concurrency check failed with eight 200s and four 429s, which is the result you want to see before launch rather than after. max_tokens=1000 leaves room for reasoning models, which spend part of the budget before answering.

After the smoke test, run a short load test at your measured peak with real-sized prompts, and record the endpoint’s time to first token and full-answer latency. Keep the request IDs; BroadNet’s docs ask for the X-Request-ID, timestamp, route and status in support requests.

Where BroadNet Inference fits

BroadRouter is BroadNet’s own site. BroadNet Inference sells self-serve access to public models by the token, by audio minute, by session plan and by hourly key; prices are on the pricing page. Private endpoints, regional processing, fine-tunes and models listed “on request” are not self-serve: they are enquiries that BroadNet quotes and agrees in writing. This post can’t tell you what BroadNet will offer for your workload. A clear brief is how you find out.

Questions

What is a private LLM endpoint?

An API endpoint backed by capacity reserved for one customer, rather than shared with every self-serve user. BroadNet describes its version as a reserved GPU or node with the model you choose, your own key and limits, and the region you need, reached with a private model ID.

How do I estimate the concurrency I need?

Use Little's law: concurrent requests = requests per second at peak × seconds per request. 1.67 requests per second at about 5.8 seconds each means about 10 requests in flight. Measure the seconds per request on the real model with your real prompts.

Can I buy a BroadNet private endpoint self-service?

No. BroadNet presents private endpoints, regional processing and unlisted models as enquiries: you describe the workload and BroadNet quotes. Its Terms say private endpoints and reserved capacity follow a separate written agreement.

Does a private endpoint come with an SLA?

Only if your written agreement says so. BroadNet's Terms state that self-serve has no availability commitment and that service levels exist only in a separate written agreement. Ask for the exact terms before you rely on them.

Where will my data be processed?

Agree it before you send data. BroadNet's Terms ask customers with residency requirements to agree processing locations first, and say partner-hosted and backup processing involves other processors, with a subprocessor list available to customers on request.

Read as MarkdownAsk ChatGPTAsk ClaudeAsk Perplexity