Word error rate (WER) is the number of substituted, deleted and inserted words divided by the number of words in the correct transcript. Lower is better. Arabic WER is easy to inflate: the same correct transcript can be written several ways, and a naive comparison counts every spelling choice, diacritic and dialect spelling as a mistake.
This post explains what WER measures, why Arabic makes it slippery and how to compute it so your numbers mean something. To build the test set itself, see Arabic dialect speech recognition; to choose a provider, see the Arabic speech-to-text API buyer’s guide.
What is word error rate?
Word error rate compares a machine transcript (the hypothesis) with a correct human transcript (the reference). It finds the minimum number of word edits that turn one into the other:
- S, substitutions: a word replaced by a different word.
- D, deletions: a reference word missing from the hypothesis.
- I, insertions: an extra word in the hypothesis.
Then:
WER = (S + D + I) / N, where N is the number of words in the reference.
A WER of 0.10 (often written as 10) means about one error per ten reference words. Lower is better. Because insertions count, WER can go above 100% for a transcript full of invented words.
Character error rate (CER) is the same formula over characters. It is useful for Arabic, where a single wrong letter in a prefix makes the whole word count as wrong.
Why is Arabic WER hard to compare?
1. Diacritics
Arabic short vowels are written as optional marks (tashkeel) above and below letters: fatha, damma, kasra, sukun, shadda and tanween. Everyday text leaves them out. If your reference has “مَدْرَسَةٌ” and the model writes “مدرسة”, a strict comparison counts a substitution, even though any Arabic reader sees the same word.
2. Spelling variants
Several letters have common, accepted variants:
| Variation | Example | Typical normalisation |
|---|---|---|
| Hamza on alef | أحمد / احمد, إلى / الى | Map أ إ آ to ا |
| Ta marbuta vs ha | مدرسة / مدرسه | Map ة to ه |
| Alef maqsura vs ya | على / علي (in loose writing) | Map ى to ي |
| Tanween with alef | صباحاً / صباحا | Strip the diacritic |
| Tatweel (stretching) | جـمـيـل / جميل | Remove ـ |
3. Clitics and word boundaries
Arabic attaches conjunctions, prepositions and pronouns to words: “وبمدرستهم” is “and in their school” in one written word. If the model writes “و بمدرستهم” with a space, the word count changes and WER punishes it twice. Some evaluations tokenise words into morphemes first; most do not. Know which you are reading.
4. Dialects
There is no standard spelling for dialect speech, unlike Modern Standard Arabic. An Egyptian speaker saying “I want” might be transcribed “عايز” or “عاوز”; a Gulf speaker’s “what” might be “وش” or “ويش”. Some transcribers also “correct” dialect into MSA. Each of these is a WER error even when the meaning is perfect. That is a large part of why dialect WER is so much higher than read-speech WER.
5. Numbers and code-switching
Speakers often switch into English mid-sentence (code-switching). Is it “٢٥”, “25” or “خمسة وعشرين”? Is an English brand name written in Latin script or in Arabic letters? Decide a convention, apply it to the reference, and normalise the hypothesis to it.
How much does normalisation change the score?
Reference: ذهبت إلى المدرسة صباحاً (“I went to school in the morning”)
Hypothesis: ذهبت الى المدرسه صباحا
Without normalisation, three of four words differ (إلى/الى, المدرسة/المدرسه, صباحاً/صباحا): WER 75%. After normalising both texts, they are identical: WER 0%. Same transcript, same quality, and the number moved from terrible to perfect. This is why a WER figure without its normalisation rules tells you little.
How do you compute Arabic WER in Python?
import re
import jiwer # pip install jiwer
DIACRITICS = re.compile(r"[ً-ْٰ]") # tanween, harakat, shadda, sukun, dagger alef
PUNCT = re.compile(r"[^\w\s]")
def normalise_ar(text: str) -> str:
text = DIACRITICS.sub("", text)
text = text.replace("ـ", "") # tatweel
text = re.sub("[أإآٱ]", "ا", text)
text = text.replace("ى", "ي").replace("ة", "ه")
text = text.translate(str.maketrans("٠١٢٣٤٥٦٧٨٩", "0123456789"))
text = PUNCT.sub(" ", text)
return " ".join(text.split())
refs = ["ذهبت إلى المدرسة صباحاً"]
hyps = ["ذهبت الى المدرسه صباحا"]
print("raw WER:", jiwer.wer(refs, hyps))
print("normalised WER:", jiwer.wer([normalise_ar(r) for r in refs], [normalise_ar(h) for h in hyps]))
print("normalised CER:", jiwer.cer([normalise_ar(r) for r in refs], [normalise_ar(h) for h in hyps]))
Apply exactly the same function to references and hypotheses, for every model you compare. Publish it next to your numbers.
What WER does BroadNet report for Arabic?
BroadNet publishes lab measurements for BroadNet Transcribe Arabic in its model catalogue:
| Test set | What it is | WER reported by BroadNet |
|---|---|---|
| FLEURS-ar | Clean read sentences | 3.8 |
| MASC-ar | Mixed real-world Arabic speech | 21.1 |
| BroadNet dialect set | Dialect speech | 23.4 |
BroadNet also reports the model at #2 of 37 (its September 2026 figure; the live board listed it #4 of 39 on 15 November 2026) on the Open Universal Arabic ASR Leaderboard, and #1 among models whose licences permit serving through an API. These are BroadNet’s figures, not independent audits. The leaderboard applies one evaluation setup to every model, which makes its ranks more comparable than numbers from separate vendor pages.
Read the spread, not just the best number. The jump from 3.8 to 23.4 is the gap between read speech and spontaneous dialect for one model, so test every candidate on both kinds of audio. Your own audio, whether call-centre lines, WhatsApp voice notes or meetings, will land somewhere in that range, and only a test tells you where.
How do you measure WER on your own audio?
- Sample by condition. Group clips by dialect and channel (phone, voice note, studio). The dialect test-set method covers sample sizes and spelling conventions.
- Transcribe with each candidate. For BroadNet, send
language=ar; without it, Arabic can come back transliterated into Latin letters. The Arabic speech-to-text guide has the code, and the $5 starting credit covers about 1,190 minutes of audio. - Normalise and score. Use the function above, report WER and CER per group.
- Read the errors. Sort substitutions by frequency. If the top errors are names, amounts or product codes, the business impact is bigger than the score suggests.
What should you measure besides WER?
WER treats every word equally, but your product does not. A support bot cares about intent; an invoicing workflow cares about numbers; a compliance team cares about specific phrases. Add a task metric next to WER: intent accuracy after an LLM reads the transcript, or exact match on extracted amounts. A model with slightly higher WER can still win if its errors fall on words that do not matter.
For the cost side, put your monthly audio hours into the free speech-to-text cost calculator, and see how the Arabic and 14-language models differ in BroadNet Transcribe vs Transcribe Arabic.
