BroadRouter
Get started

Character error rate (CER)

Updated

Character error rate (CER) measures transcription or OCR accuracy at the character level: the substitutions, deletions and insertions needed to turn the output into the reference text, divided by the number of characters in the reference. Lower is better, and 0 is a perfect match.

Also called: CER, character error rate

WER asks how many words were wrong. CER asks how many letters were wrong, which is often the fairer question for Arabic.

How it works

CER uses the same edit-distance idea as word error rate, applied to characters. The Hugging Face metric card gives the formula:

CER = (S + D + I) / N, where S is substitutions, D deletions, I insertions and N the number of characters in the reference (N = S + D + C, with C the correct characters).

A worked Arabic example. The reference is “وبالكتاب” (“and with the book”, one word, 8 letters). The transcript is “وبلكتاب”, missing one alef:

Metric Errors Reference units Score
WER 1 wrong word 1 word 1.0 (100%)
CER 1 deleted letter 8 characters 0.125 (12.5%)

One missing letter makes the whole word wrong for WER. CER shows the transcript was almost right.

In Python, the jiwer package computes it with jiwer.cer(reference, hypothesis), and jiwer.process_characters also returns the alignment. Normalise both sides first, as for WER: punctuation, spacing, and for Arabic, diacritics and alef or yaa forms. See Arabic text normalization.

Why it matters

  • Arabic morphology. Arabic attaches conjunctions, prepositions and pronouns to words as clitics, so one word can carry what English writes as three or four. A single wrong prefix letter costs a full word in WER. CER weighs it as one character.
  • Languages without spaces. WER needs word boundaries. The Hugging Face card says CER is useful “especially for multilingual datasets where WER is not suitable given the diversity of languages.”
  • Error type. A model with high WER but low CER is getting words nearly right: spelling, hamza or clitic slips that normalisation or a correction step may fix. High CER means the model is missing whole words or content.
  • OCR too. The same metric scores text read from images, so it is useful when you compare OCR pipelines.

The metric card also warns that CER does not tell you why errors happen. Read sample transcripts alongside the score.

On BroadNet Inference

BroadNet publishes WER, not CER, for BroadNet Transcribe Arabic and BroadNet Transcribe. To get CER, transcribe a test set of your own recordings and score it yourself. The Arabic WER explained post has a normalisation function and a jiwer script that prints both WER and CER, and the Arabic speech-to-text guide has the request code. Always send language=ar: without it, Arabic can come back transliterated into Latin letters, which wrecks both scores.

Common mistakes

  • Counting spaces inconsistently. Decide whether spaces count as characters and apply the same rule to every model you compare.
  • Comparing CER to someone else’s WER. They are different scales. Compare like with like.

Questions

Can CER be higher than 100%?

Yes. The Hugging Face metric card notes that CER is not always between 0 and 1, especially when a transcript has many insertions, because the denominator counts only reference characters.

Should I report CER or WER for Arabic?

Report both. WER is what leaderboards and vendors quote, so you need it to compare. CER shows whether the errors are whole wrong words or a letter or clitic here and there, which WER hides.

Does BroadNet publish CER for its transcription models?

No. BroadNet publishes WER only. To get CER, score your own test set with a tool such as jiwer.

Read as MarkdownAsk ChatGPTAsk ClaudeAsk Perplexity