BroadNet’s model catalogue lists every model it says it has run, benchmarked, quantised or fine-tuned on its own hardware, and marks the ones it rejected with a reason. About 30 models and methods carry the RULED OUT label. The figures below were measured in BroadNet’s lab in September 2026; they are BroadNet’s results, not ours.
The catalogue uses six labels: PROD (carried live traffic), SERVED (in the catalogue’s words, “ran on our fleet”), BENCH (benchmarked), QUANT (BroadNet produced the quantisation), FT (BroadNet fine-tuned it) and RULED OUT.
Which chat models did BroadNet rule out?
| Model | What BroadNet did | Why it was ruled out |
|---|---|---|
| Qwen3.5 35B A3B (AWQ 4-bit) | Served as a speed tier, about 164 tokens/s | Failed BroadNet’s credential-safety battery for security roles |
| GLM 4.7 Flash (AWQ) | Served experimentally | Labelled SERVED, not RULED OUT: BroadNet says it was ruled out for security roles by its credential-safety battery and held below production |
| Nemotron 3 Nano 30B A3B (NVFP4) | Served with a custom reasoning parser | Labelled SERVED, not RULED OUT: BroadNet says it was ruled out for security roles by its credential-safety battery and held below production |
| Qwen3.6 35B A3B | Served on RTX 5090 and GB10 | Lost the chatbot slot to Gemma 4 26B |
| Gemma 4 E4B (AWQ INT4) | Served on GB10 as a co-tenant, about 94 tokens/s | Lost the chatbot slot to Gemma 4 26B |
| Gemma 4 26B A4B NVFP4 (three publishers) | Quantisation evaluated | Retired in favour of AWQ after a head-to-head on sm_120 and sm_121 GPUs |
| EAGLE3 speculative decoding | Tested for saturated serving | Cost 69% of throughput at 32 concurrent requests; multi-token-prediction drafters kept |
The pattern matters for anyone choosing an open model. Three of the rejections are about safety, not quality: models that were fast enough but repeated credentials. That is the failure mode BroadNet’s Safety Map study found most often in small-active mixture-of-experts models. The model that won the chatbot slot, Gemma 4 26B A4B, is the one BroadNet sells as Gemma 4 VL 26B.
Which speech models were ruled out?
| Model | Why it was ruled out |
|---|---|
| Gemma 4 E2B, Gemma 4 12B (as transcribers) | The E4B audio variant was kept instead |
| Omnilingual ASR 300M, 1B, 3B and CTC variants (Meta) | The 7B variant earned the slot |
| Voxtral Mini 3B / 4B Realtime (Mistral) | Ruled out for Arabic |
| ArTSTv3 (MBZUAI lineage) | Ruled out for the Arabic slot; Cohere Transcribe Arabic kept |
| FireRedVAD, DeepFilterNet, BNR (front-ends) | Silero voice activity detection adopted after a head-to-head |
BroadNet reports that the model it kept, Cohere Transcribe Arabic 07-2026, scores 21.1 word error rate on MASC dialect speech, against 22.3 for ElevenLabs Scribe v2 and 31.5 for Whisper large-v3, on its own test. It serves that model as BroadNet Transcribe Arabic. The catalogue also says Whisper large-v3 was benchmarked as a fallback.
Which document, embedding and image models lost?
OCR and document parsing
- LlamaParse, the incumbent API, scored 16.6 on BroadNet’s OmniDocBench subset. BroadNet ruled it out because “it loses tables, and self-hosting is 98% cheaper per 1,000 pages”.
- olmOCR-2, HunyuanOCR, GOT-OCR 2.0 and Nanonets OCR2 were evaluated and ruled out; Nanonets on licence terms. dots.mocr was kept as the production pick because it both parses and describes figures, even though PaddleOCR-VL 1.6 scored higher (94.74 against 86.81).
Embeddings
- gemini-embedding-2 was ruled out for Arabic after it regressed against gemini-embedding-001 (0.767 against 0.824 on BroadNet’s retrieval benchmark).
- nomic-embed-text v1.5, the former incumbent, scored 0.468 overall and 0.130 on Arabic, and was retired.
- Octen 8B, Harrier 0.6B and Nemotron Embed 1B were served and ruled out on real Arabic traffic, “where our podium held despite their public-board wins”.
BroadNet adds one finding that outranked model choice: a workload-specific query instruction moved retrieval quality (nDCG) by 0.048, more than moving from a 4B to an 8B embedding model.
Image generation and safety
- Cosmos3 Super 64B and HunyuanImage 3.0 80B were ruled out on hardware fit and licence terms.
- openai/privacy-filter 1.5B caught 100% of credentials but 0% of salary figures in BroadNet’s leak corpus, so it was ruled out as a firewall.
What can you learn from the rejections?
Four lessons hold beyond BroadNet’s own stack:
- Test the job, not the leaderboard. Several embedding models with public-board wins lost on real Arabic traffic.
- Safety is a separate gate. A fast, capable model can still fail a credential test. Run one before a model sees secrets; BroadNet’s evaluation method is a starting point.
- Licences decide what can be served. Non-commercial models such as Voxtral 4B TTS and Higgs Audio v3 appear in the catalogue only as quality ceilings.
- Serving choices are measured too. Quantisation formats and drafting methods were ruled out on head-to-head numbers, not by default. See BroadNet’s GPU hardware and serving approach.
Every number here comes from one company’s lab and traffic. Use it as a shortlist of what to test, then evaluate on your own data before you switch. A ruled-out or benchmarked row is also not something you can call; reading a model catalogue explains how to tell live, on-request and research entries apart.
