---
title: "Private LLM endpoint planning: what to define first | BroadRouter"
description: "What to define before asking for a private LLM endpoint: workload, tokens per second, concurrency, context, region, licence and budget, then how to test it."
url: https://broadrouter.com/blog/private-endpoint-planning/
lang: en
updated: 2026-11-15
publisher: BroadRouter (a BroadNet site); API by BroadNet Inference, https://inference.broadnet.ai
---

# A private endpoint starts with a clear workload

By the BroadNet Inference team · 7 min read · Published 2026-11-15

Before you ask for a private endpoint, write down the workload: requests per day and at peak, tokens in and out, the concurrency that implies, the context you need, the processing region, the model and its licence, and a budget. Agree region and limits in writing, then smoke-test the issued endpoint before real data.

## Key takeaways

-   A capacity request is only as good as its numbers: peak requests per second, tokens in and out, and the latency you can accept.
-   Concurrency follows from Little's law: requests in flight = arrival rate × time per request. Measure time per request on a trial first.
-   BroadNet publicly describes private endpoints as reserved GPU capacity priced to your traffic, with your own key, limits and region, reached with a private model ID. Everything else is agreed with BroadNet.
-   BroadNet's Terms say self-serve has no availability commitment, and private endpoints and reserved capacity follow a separate written agreement.
-   Test the issued endpoint before production: model ID, streaming, the agreed concurrency and the error for a wrong model ID.

**On this page**

1.  When does a private endpoint make sense?
2.  What should you define about the workload?
3.  How do you turn volume into tokens per second and concurrency?
4.  What about region, residency and limits?
5.  Which model, and under which licence?
6.  How should you set a budget?
7.  How do you send the enquiry?
8.  How do you validate the endpoint you’re given?
9.  Where BroadNet Inference fits

“We need a dedicated endpoint” is the start of a conversation, not a specification. The provider can’t size, place or price capacity without your numbers, and you can’t judge their answer without them either. This post lists what to define before you ask, using a worked invoice-extraction example, and ends with a script that checks the endpoint you are given.

It applies to any provider. Where it describes BroadNet’s offer, it sticks to what BroadNet says publicly, checked on 15 November 2026. Anything beyond that is a question for BroadNet.

## When does a private endpoint make sense?

Usually for one of four reasons:

-   **Traffic above self-serve limits.** BroadNet’s API reference publishes 30 requests per minute and 4 requests in flight for a pay-as-you-go key. A steady workload above that is a capacity question, not a retry question. ([Handling 429s](https://broadrouter.com/blog/handle-llm-api-429-rate-limits/) covers short bursts.)
-   **A model that isn’t on the public list.** BroadNet’s catalogue marks some models “available on request”, including Qwen3-VL 30B-A3B, Qwen3-VL 8B and a BroadNet fine-tune of Cohere Transcribe Arabic. [Reading a model catalogue](https://broadrouter.com/blog/understand-model-status/) explains the difference.
-   **A processing location you must control.** Data-residency rules, such as those in the [Saudi and UAE PDPL checklist](https://broadrouter.com/blog/saudi-uae-pdpl-llm-api-checklist/), may require a region you have agreed in writing.
-   **Predictable latency.** Shared capacity varies with other users’ load. Reserved capacity removes that variable, though not network distance or your own prompt sizes.

If none of these apply, a self-serve key is simpler. And if your volume is very high and steady, compare against running it yourself: [self-hosting vs API cost](https://broadrouter.com/blog/self-hosting-llm-vs-api-cost/) has the method.

## What should you define about the workload?

Write a one-page brief. These are the numbers a provider needs:

| Item | What to write down | Example |
| --- | --- | --- |
| Task | What the model does, in one sentence | Read invoice photos, return JSON fields |
| Volume | Requests per day, and the share in the busiest hour | 40,000 a day, 15% in the peak hour |
| Tokens | Average and 95th-percentile input and output tokens | 1,800 in (one page image + prompt), 350 out |
| Context | Largest single request, prompt plus answer | 4,000 tokens |
| Latency | Acceptable time to first token and to full answer | Full answer within 10 seconds |
| Streaming | Needed or not | Not needed (batch-style) |
| Inputs | Text, images, audio; formats and sizes | JPEG and PNG, one page each |
| Languages | Which, and the share of each | Arabic and English, 60/40 |
| Growth | Expected volume in 6 and 12 months | Double in 12 months |

Measure the token counts on real samples, not guesses. Arabic often needs more tokens than English for the same text; the [Arabic token counter](https://broadrouter.com/tools/arabic-token-counter/) and [real cost by language](https://broadrouter.com/data/real-cost-by-language/) show by how much on different tokenizers. For images, the [image token calculator](https://broadrouter.com/tools/image-token-calculator/) gives a first estimate.

## How do you turn volume into tokens per second and concurrency?

Peak requests per second is the busiest-hour volume divided by 3,600. Concurrency, the number of requests in flight at once, follows from [Little’s law](https://en.wikipedia.org/wiki/Little%27s_law): the average number in a system equals the arrival rate times the average time each one spends there. For an API, that is requests per second × seconds per request.

```python
import math

def size(requests_per_day: int, peak_share_of_day_in_busiest_hour: float,
         in_tokens: int, out_tokens: int, out_tok_per_s_per_stream: float) -> dict:
    """Rough peak numbers to put in a capacity enquiry. Measure your own inputs; these are not provider specs."""
    peak_rps = requests_per_day * peak_share_of_day_in_busiest_hour / 3600
    seconds_per_request = out_tokens / out_tok_per_s_per_stream        # ignores queueing and prompt time
    concurrency = peak_rps * seconds_per_request                      # Little's law: L = λ × W
    return {
        "peak_requests_per_second": round(peak_rps, 2),
        "seconds_per_request": round(seconds_per_request, 1),
        "concurrent_requests_at_peak": math.ceil(concurrency),
        "peak_input_tokens_per_second": round(peak_rps * in_tokens),
        "peak_output_tokens_per_second": round(peak_rps * out_tokens),
        "monthly_tokens_millions": round(requests_per_day * 30 * (in_tokens + out_tokens) / 1e6, 1),
    }

# 40,000 invoice extractions a day, 15% of them in the busiest hour,
# 1,800 tokens in, 350 out, 60 tokens/s per stream measured on a trial.
print(size(40_000, 0.15, 1_800, 350, 60))
```

It printed 1.67 requests per second at peak, about 5.8 seconds per request, 10 concurrent requests, 3,000 input and 583 output tokens per second, and 2,580 million tokens a month. The 60 tokens per second is an assumption for the example; measure your own on the model you will use, because speed varies by model, prompt length and load. The [latency, TTFT and throughput](https://broadrouter.com/blog/llm-latency-ttft-throughput/) post explains what to time.

Two cautions. The formula ignores prompt processing time, so long prompts or large images make real requests slower and push concurrency up. And averages hide bursts: give the provider your 95th-percentile request size and a short burst profile, not only the mean. This example’s peak (about 100 requests a minute and 10 in flight) is well above the published self-serve limits for one pay-as-you-go key, which is exactly when a capacity conversation is worth having.

## What about region, residency and limits?

Agree them explicitly, in writing, before any regulated data moves. BroadNet’s Terms (§7.3a) say: “Contact us to agree processing locations for your deployment before sending data with residency requirements,” and that partner-hosted and backup processing involves other processors, with a current subprocessor list available to customers on request. Its privacy statement says BroadNet’s own and hosted open-weight models run on BroadNet-owned or leased hardware in the United States, with partner capacity as backup, and its overview page says regional processing in MENA or Europe is available on request.

So the questions for any provider are:

1.  Which region processes requests, and which region holds backup capacity?
2.  Which subprocessors can touch the traffic, including for failover or overflow?
3.  What are the endpoint’s rate, concurrency and token limits, and what happens above them?
4.  What availability and support terms apply, and where are they written? BroadNet’s Terms (§7.4) say self-serve has no availability commitment, and that private endpoints and reserved capacity follow your separate written agreement. To see what written SLAs from other providers cover, read [LLM API SLAs compared](https://broadrouter.com/blog/llm-api-sla-uptime-compared/).
5.  What is retained? For BroadNet’s inference API, the published position is: inference content is not retained; necessary account, billing and security metadata is retained as described in its [privacy statement](https://www.broadnet.ai/inference-privacy). Confirm the same applies to your endpoint.

## Which model, and under which licence?

Name the exact model and version, and check its licence permits your use and API serving. BroadNet’s model pages list Apache-2.0 for Gemma 4 26B, Qwen3-VL 30B-A3B and the Cohere Transcribe base weights, with the BroadNet fine-tune of Cohere Transcribe Arabic “under contract”. Its Terms say models are governed by their licences, and that DeepSeek V4.1 Flash’s licence still applies when partner-hosted.

Also agree the context window. Context limits depend on the model and the deployment, not only on the base model’s maximum: BroadNet lists 32K tokens for Gemma 4 VL 26B on its public API and 16K for Qwen3-VL 30B-A3B, which is on request. Use the [context window calculator](https://broadrouter.com/tools/context-window-calculator/) against your largest request, not your average one.

## How should you set a budget?

BroadNet publishes no private-endpoint price; its overview page says “Reserved GPU, priced to your traffic. Ask for a quote.” So set your own ceiling first. Price the same workload on self-serve with the [LLM API cost calculator](https://broadrouter.com/tools/llm-api-cost-calculator/), using the monthly token count from the sizing script, and decide what predictable capacity, a region or a model that isn’t public is worth to you on top. A quote below the self-serve figure is a bonus; a quote far above it needs a reason from your list of requirements.

## How do you send the enquiry?

BroadNet’s API reference says: “Private endpoints, our Arabic fine-tunes and other models on request are reached with a private ID issued with the endpoint.” Its overview page asks for “your model, languages and traffic requirements”. Send your one-page brief through the [contact page](https://broadrouter.com/contact/#private), without API keys or customer data. Include the sizing output, the region, the model and licence, the latency target, your budget ceiling and your timeline.

## How do you validate the endpoint you’re given?

Before production traffic, test the endpoint against what was agreed. This script uses only the Python standard library. It checks that the private model ID works, that streaming ends with the `data: [DONE]` marker BroadNet’s docs describe, that the agreed concurrency runs without 429s, and that a wrong model ID is rejected.

```python
import json, os, sys, time, urllib.error, urllib.request
from concurrent.futures import ThreadPoolExecutor

BASE = os.environ["ENDPOINT_BASE_URL"].rstrip("/")    # from your agreement, e.g. https://.../v1
KEY = os.environ["ENDPOINT_API_KEY"]
MODEL = os.environ["ENDPOINT_MODEL_ID"]                # the private ID issued with the endpoint
AGREED_CONCURRENCY = int(os.environ.get("AGREED_CONCURRENCY", "4"))

def post(payload: dict, stream=False):
    req = urllib.request.Request(f"{BASE}/chat/completions", data=json.dumps(payload).encode(),
        headers={"Authorization": f"Bearer {KEY}", "Content-Type": "application/json"})
    t0 = time.monotonic()
    try:
        with urllib.request.urlopen(req, timeout=120) as r:
            body = r.read().decode()
            return r.status, r.headers.get("X-Request-ID"), body, time.monotonic() - t0
    except urllib.error.HTTPError as e:
        return e.code, e.headers.get("X-Request-ID"), e.read().decode(), time.monotonic() - t0

def main() -> int:
    msg = [{"role": "user", "content": "Reply with the word OK."}]
    results = {}
    code, rid, body, secs = post({"model": MODEL, "messages": msg, "max_tokens": 1000})
    ok = code == 200 and "usage" in json.loads(body)
    results["model_id_accepted"] = (ok, f"HTTP {code}, request ID {rid}, {secs:.2f}s")
    code, _, body, _ = post({"model": MODEL, "messages": msg, "stream": True,
                             "stream_options": {"include_usage": True}})
    results["stream_ends_with_DONE"] = (code == 200 and body.rstrip().endswith("data: [DONE]"), f"HTTP {code}")
    with ThreadPoolExecutor(AGREED_CONCURRENCY) as pool:
        codes = list(pool.map(lambda _: post({"model": MODEL, "messages": msg, "max_tokens": 1000})[0],
                              range(AGREED_CONCURRENCY)))
    results["agreed_concurrency_no_429"] = (all(c == 200 for c in codes), f"codes {sorted(codes)}")
    code, _, _, _ = post({"model": MODEL + "-typo", "messages": msg})
    results["unknown_model_rejected"] = (code in (400, 403, 404), f"HTTP {code}")
    for name, (passed, note) in results.items():
        print(f"{'PASS' if passed else 'FAIL'}  {name}: {note}")
    return 0 if all(p for p, _ in results.values()) else 1

if __name__ == "__main__":
    sys.exit(main())
```

We ran it against a local mock server (no real endpoint) that allows 8 requests in flight. With `AGREED_CONCURRENCY=8` all four checks passed. With 12, the concurrency check failed with eight 200s and four 429s, which is the result you want to see before launch rather than after. `max_tokens=1000` leaves room for reasoning models, which spend part of the budget before answering.

After the smoke test, run a short load test at your measured peak with real-sized prompts, and record the endpoint’s time to first token and full-answer latency. Keep the request IDs; BroadNet’s docs ask for the `X-Request-ID`, timestamp, route and status in support requests.

## Where BroadNet Inference fits

BroadRouter is BroadNet’s own site. BroadNet Inference sells self-serve access to public models by the token, by audio minute, by session plan and by hourly key; prices are on the [pricing page](https://broadrouter.com/pricing/). Private endpoints, regional processing, fine-tunes and models listed “on request” are not self-serve: they are enquiries that BroadNet quotes and agrees in writing. This post can’t tell you what BroadNet will offer for your workload. A clear brief is how you find out.

Free tools for this

-   [OpenAI to BroadNet migration helper](https://broadrouter.com/tools/openai-to-broadnet-migration/) Paste OpenAI SDK code; get base URL, key and model edits, a diff and the checks that remain.
-   [max\_tokens calculator for reasoning models](https://broadrouter.com/tools/reasoning-token-budget/) Answer length + reasoning budget + margin = a max\_tokens that does not cut the answer off.
-   [JSON schema generator for LLM structured output](https://broadrouter.com/tools/json-schema-generator/) Sample JSON to strict schema, Pydantic, Zod and a ready request.
-   [LLM JSON output validator](https://broadrouter.com/tools/llm-json-output-validator/) Parse errors by line and column, safe repairs, cut-off detection and JSON Schema checks for model output.

[All free tools](https://broadrouter.com/tools/)

## Terms in this article

[Token](https://broadrouter.com/glossary/token/)

A token is the unit of text a language model reads and writes: a word, part of a word, a punctuation mark or a byte sequence produced by the model's tokenizer. LLM APIs count, limit and usually bill usage in tokens.

[Concurrency](https://broadrouter.com/glossary/concurrency/)

Concurrency is the number of requests a client has in flight at the same time. LLM APIs often cap it per key or plan, separately from requests per minute, because each open request holds GPU capacity until its response has finished.

[Latency](https://broadrouter.com/glossary/latency/)

Latency, for a language model API, is the time a request takes from being sent to the final output token being received. It is the sum of time to first token and generation time, and is usually reported as percentiles such as p50 and p95 rather than as an average.

[Context window](https://broadrouter.com/glossary/context-window/)

A context window is the maximum number of tokens a language model can work with in a single request. It covers the prompt, conversation history, tool definitions, images and the generated output, including any reasoning tokens. Anything that does not fit must be cut, summarised or split across several requests.

[Pay-as-you-go](https://broadrouter.com/glossary/pay-as-you-go/)

Pay-as-you-go is a pricing model in which you pay only for what you use, metered per unit such as tokens, requests or audio minutes, with no subscription or minimum commitment. Most LLM APIs bill this way, from a prepaid balance or a periodic invoice.

[Throughput](https://broadrouter.com/glossary/throughput/)

Throughput, in LLM serving, is the total amount of work a system completes per unit of time across all concurrent requests, usually measured as output tokens per second or requests per second. It differs from per-stream speed, which is how fast a single request receives its tokens.

[Tool calling](https://broadrouter.com/glossary/tool-calling/)

Tool calling, also called function calling, is an LLM API feature in which the model returns a structured request to run a function the developer has defined, with JSON arguments. The application runs the function and sends the result back; the model never runs code itself.

[Input and output tokens](https://broadrouter.com/glossary/input-and-output-tokens/)

Input tokens are the tokens a language model reads in a request, including the system prompt, conversation history, tool definitions and images. Output tokens are the tokens it generates in reply, including any reasoning. LLM APIs usually price them separately, and output usually costs more.

## Questions

### What is a private LLM endpoint?

An API endpoint backed by capacity reserved for one customer, rather than shared with every self-serve user. BroadNet describes its version as a reserved GPU or node with the model you choose, your own key and limits, and the region you need, reached with a private model ID.

### How do I estimate the concurrency I need?

Use Little's law: concurrent requests = requests per second at peak × seconds per request. 1.67 requests per second at about 5.8 seconds each means about 10 requests in flight. Measure the seconds per request on the real model with your real prompts.

### Can I buy a BroadNet private endpoint self-service?

No. BroadNet presents private endpoints, regional processing and unlisted models as enquiries: you describe the workload and BroadNet quotes. Its Terms say private endpoints and reserved capacity follow a separate written agreement.

### Does a private endpoint come with an SLA?

Only if your written agreement says so. BroadNet's Terms state that self-serve has no availability commitment and that service levels exist only in a separate written agreement. Ask for the exact terms before you rely on them.

### Where will my data be processed?

Agree it before you send data. BroadNet's Terms ask customers with residency requirements to agree processing locations first, and say partner-hosted and backup processing involves other processors, with a subprocessor list available to customers on request.

## Keep reading

-   [Self-hosting an LLM vs an API: the real cost and break-even](https://broadrouter.com/blog/self-hosting-llm-vs-api-cost/)
-   [LLM latency: TTFT vs tokens per second vs end-to-end time](https://broadrouter.com/blog/llm-latency-ttft-throughput/)
-   [Saudi and UAE PDPL and LLM APIs: a cross-border checklist](https://broadrouter.com/blog/saudi-uae-pdpl-llm-api-checklist/)
-   [How to choose an LLM API provider: a 20-point checklist](https://broadrouter.com/blog/choose-llm-api-provider-checklist/)
-   [How to estimate LLM API cost before you launch a feature](https://broadrouter.com/blog/estimate-llm-api-cost-before-launch/)
-   [How to handle LLM API 429 errors: backoff, jitter and limiters](https://broadrouter.com/blog/handle-llm-api-429-rate-limits/)
-   [Live, on request, research: read a model catalogue clearly](https://broadrouter.com/blog/understand-model-status/)
-   [Before fine-tuning: define the problem to solve](https://broadrouter.com/blog/fine-tuning-readiness/)
-   [BroadNet's GPU hardware and serving approach, as BroadNet describes it](https://broadrouter.com/blog/broadnet-gpu-fleet-and-serving/)
-   [Guide: BroadNet Inference error codes and when to retry](https://broadrouter.com/guides/errors-and-retries/)
-   [Guide: Lock API keys to your servers with IP allow-lists](https://broadrouter.com/guides/api-keys-ip-allow-lists/)

Try it on your real workload. New accounts start with $5 of credit for Gemma and transcription. DeepSeek, GLM and Qwen3.8 need a top-up.

[Get started](https://inference.broadnet.ai/account/?utm_source=broadrouter&utm_medium=referral&utm_campaign=blog-private-endpoint-planning&utm_content=page-cta) ·

**Sources**

-   [BroadNet Inference overview: private endpoint "Reserved GPU, priced to your traffic", regional processing on request (checked 2026-11-15)](https://www.broadnet.ai/inference)
-   [BroadNet model catalogue: available on request, multi-GPU private endpoints sized per customer (checked 2026-11-15)](https://www.broadnet.ai/inference/models)
-   [BroadNet model page, Qwen3-VL 30B-A3B Instruct: on request, private ID issued per customer, 16K context (checked 2026-11-15)](https://www.broadnet.ai/inference/models/qwen3-vl-30b-a3b)
-   [BroadNet model page, Gemma 4 VL 26B: private endpoint with your own key, limits and region (checked 2026-11-15)](https://www.broadnet.ai/inference/models/gemma-4-vl-26b)
-   [BroadNet Terms of Service, §7.3a hosting and processing locations and §7.4 availability, last updated 23 September 2026 (checked 2026-11-15)](https://www.broadnet.ai/terms)
-   [BroadNet Inference API reference: private IDs, pay-as-you-go limits, streaming, errors (checked 2026-11-15)](https://inference.broadnet.ai/docs?utm_source=broadrouter&utm_medium=referral&utm_campaign=blog-private-endpoint-planning&utm_content=sources)
-   [BroadNet Inference privacy and data retention statement: where processing happens (checked 2026-11-15)](https://www.broadnet.ai/inference-privacy)
-   [Little's law, L = λW (Wikipedia, checked 2026-11-15)](https://en.wikipedia.org/wiki/Little%27s_law)
