Arabic lets the same word be written several ways. A user types احمد; the document says أَحْمَد. A search index that compares exact characters misses that match, and so does the keyword half of a RAG pipeline. Normalization fixes this by mapping variants to one form before you compare them. Done carelessly, it also merges words that mean different things. This post goes rule by rule: what each one does, when to apply it, what it breaks, and how the main libraries differ. It ends with tested Python.
For the full Arabic RAG pipeline (chunking, hybrid search, cited answers), see Arabic RAG that works. This post goes deeper on the normalization step.
Does normalization actually help retrieval?
Yes, for keyword search, and the evidence is old but clear. Larkey, Ballesteros and Connell (SIGIR 2002) tested Arabic retrieval on the TREC-2001 newswire collection. Before stemming, they normalized corpus and queries: removed diacritics and punctuation, unified alef forms, and mapped word-final ى and ة. Their monolingual results, with unexpanded queries:
| Processing | Average precision | Change vs raw (from the rounded figures) |
|---|---|---|
| Raw text | .194 | — |
| Normalized | .238 | +22.7% |
| Normalized + light stemming (light8) | .376 | +93.8% |
Two cautions. This is 2002 keyword retrieval on news, not embedding search. And the largest gain came from light stemming (removing prefixes such as و and ال and common suffixes), which is a separate step from normalization. Lucene’s ArabicAnalyzer and Elasticsearch’s arabic analyzer apply both: a normalizer, then a light stemmer.
What should you normalize, and what does each rule cost?
| Rule | Example | Search index | Risk |
|---|---|---|---|
| Unicode NFKC | ﻻ → لا, ا + ٔ → أ | Yes, always first | Rewrites compatibility characters in other scripts too |
| Remove invisible marks | zero-width joiners, RLM/LRM, BOM | Yes | None for Arabic; ZWNJ matters in Persian |
| Strip tashkeel | أَحْمَدُ → أحمد | Yes | Merges words told apart only by vowels |
| Remove tatweel (ـ) | الـــمدينة → المدينة | Yes | None |
| Unify alef | أ إ آ ٱ → ا | Yes | Merges a few pairs: سأل (asked) and سال (flowed) |
| Alef maksura | ى → ي | Usually | Merges على (on) and علي (Ali) |
| Ta marbuta | ة → ه | Usually | Merges كتابة (writing) and كتابه (his book) |
| Hamza seats | ؤ ئ → ء | Optional | Merges مسؤول and مسئول (good) but also some distinct words |
| Persian letters | ی → ي, ک → ك | Yes, if users have Persian keyboards | None for Arabic text |
| Digits | ٢٠٢٦ and ۲۰۲۶ → 2026 | Yes | None; keep the original for display |
The first column is for the index key and the query. The text you store, show and pass to the model stays as written.
Why should Unicode normalization come first?
The same visible letter can be stored in different ways. أ can be one code point (U+0623) or bare alef followed by a combining hamza (U+0627 U+0654). Older PDFs and some keyboards produce Arabic presentation forms (U+FB50 to U+FEFF), such as the single-character lam-alef ﻻ. Under Unicode normalization, NFC composes the hamza onto the alef, and NFKC also unpacks presentation forms and ligatures. In Python, unicodedata.normalize("NFKC", "ﻻ") returns لا, and ﷲ becomes الله.
Run this before anything else. Otherwise a regex that replaces أ misses the decomposed form. There is a second trap: CAMeL Tools’ dediac_ar removes the characters in its AR_DIAC_CHARSET (U+064B to U+0652 and U+0670). That set leaves out the combining hamza and madda marks (U+0653 to U+0655), so on decomposed text they survive dediacritization. NFKC composes them first.
NFKC is a compatibility mapping. It also changes characters outside Arabic, for example full-width Latin letters. That is usually what you want in a search key, and one more reason to keep the original beside it.
What should you not normalize?
- The text you display, cite or quote. Normalized Arabic looks misspelled (المدينه). Store the original next to the key.
- What you send to the LLM. Models read hamza and tashkeel fine, and exact text keeps names and quotes correct. Stripping tashkeel saves some tokens; check how many with the Arabic token counter before you trade accuracy for it.
- Text where marks carry meaning: Quran, hadith, poetry, children’s books, pronunciation and diacritization tasks (see adding diacritics with an LLM).
- Spelling or grammar correction: normalising first erases the hamza and taa marbuta errors you want to fix; see Arabic grammar and spelling correction with an LLM.
- Identifiers. Product codes, emails, URLs and Latin text must not go through Arabic rules. Apply normalization to Arabic-script fields only, or to Arabic tokens only.
- Fields people search exactly. For legal names or ID-like fields, index a raw copy as well, so an exact match can still rank first.
How do CAMeL Tools, PyArabic and Lucene differ?
They don’t implement the same rules, and mixing them gives an index and queries that never match. We read each library’s documentation and source, and ran PyArabic 0.6.15:
| Function | What it actually does |
|---|---|
CAMeL normalize_unicode |
NFKC (or NFC with compatibility=False); also expands ﷼ to ريال and ﷽ to its full phrase |
CAMeL normalize_alef_ar |
إ أ ٱ آ → ا |
CAMeL normalize_alef_maksura_ar |
ى → ي |
CAMeL normalize_teh_marbuta_ar |
ة → ه |
CAMeL dediac_ar |
Removes U+064B to U+0652 and U+0670; leaves tatweel |
PyArabic strip_tashkeel |
Removes all harakat including shadda |
PyArabic strip_tatweel |
Removes ـ |
PyArabic normalize_alef |
Every alef form and ى → ا, so إلى becomes الا |
PyArabic normalize_hamza |
Default method maps every hamza form to ء, including أ, so أحمد becomes ءحمد |
Lucene ArabicNormalizer |
Hamza-on-alef forms → ا, ة → ه, ى → ي, removes harakat and tatweel |
The PyArabic rows surprise people. Its normalize_alef docstring says alef maksura is an exception, but in 0.6.15 the ALEFAT set includes ى, and the function maps it to ا. Its normalize_hamza is built for hamza-agnostic matching, not general search. Order matters too: run alef normalization first, and normalize_hamza then only touches ؤ, ئ and ء, so مسؤول and مسئول both become مسءول. Its tasheel method gives مسوول and مسيول instead, which don’t match each other.
Elasticsearch’s arabic_normalization filter uses the Lucene normalizer, so a Python pipeline that wants to match an Elasticsearch index should copy Lucene’s rules exactly. That means ى → ي, not ى → ا.
What does a tested normalizer look like?
This function follows Lucene’s choices, adds Unicode, digit and Persian-letter handling, and makes the risky rules switchable. It uses only the standard library.
import re
import unicodedata
DIACRITICS = re.compile("[ً-ٰٟ]") # tashkeel, extra marks, superscript alef
INVISIBLE = re.compile("[---]") # zero-width, bidi
ALEF = re.compile("[آأإٱ]") # آ أ إ ٱ -> ا
DIGITS = str.maketrans("٠١٢٣٤٥٦٧٨٩۰۱۲۳۴۵۶۷۸۹", "0123456789" * 2)
FARSI = str.maketrans({"ی": "ي", "ک": "ك"}) # ی -> ي, ک -> ك
def normalize(text, *, alef=True, alef_maksura=True, ta_marbuta=True,
hamza_seats=False, digits=True):
text = unicodedata.normalize("NFKC", text) # compose marks, unpack ligatures
text = INVISIBLE.sub("", text)
text = DIACRITICS.sub("", text).replace("ـ", "") # tashkeel, tatweel
text = text.translate(FARSI)
if alef:
text = ALEF.sub("ا", text)
if alef_maksura:
text = text.replace("ى", "ي") # ى -> ي
if ta_marbuta:
text = text.replace("ة", "ه") # ة -> ه
if hamza_seats:
text = re.sub("[ؤئ]", "ء", text) # ؤ ئ -> ء
if digits:
text = text.translate(DIGITS)
return text
What it returned on our test strings:
| Input | Output | Note |
|---|---|---|
| أَحْمَدُ | احمد | Tashkeel and hamza gone |
| الـــمدينة | المدينه | Tatweel and ta marbuta |
| إلى | الي | Alef and alef maksura |
| ا + U+0654 + مل | امل | Decomposed hamza composed, then unified |
| ﻻبد | لابد | Presentation form unpacked |
| ٢٠٢٦ / ۱۴۰۵ | 2026 / 1405 | Arabic-Indic and Persian digits |
| مسؤول / مسئول | Both مسءول with hamza_seats=True |
Unchanged by default |
| فارسی | فارسي | Persian ya |
Use it in two places: when you build the index key and when you read a query. Store original and normalize(original) side by side, search on the second and return the first. The free Arabic text normalizer lets you paste text and see each rule’s effect before you choose your settings.
How does normalization fit embeddings and RAG?
A RAG pipeline usually has two retrievers: keyword search (BM25) and embeddings. Normalization clearly helps the keyword side. For the embedding side, the evidence above doesn’t apply: embedding models are trained on raw text, and many already place احمد and أحمد close together. Normalizing before embedding can help, do nothing, or hurt. Build 50 to 100 real queries with known answers, run them with raw and normalized input, and keep whichever recalls more. Whatever you pick, apply it identically to documents and queries.
Normalization also lowers token counts slightly, because diacritics and tatweel are extra characters for the tokenizer. Measured ratios are in the tokenizer efficiency dataset and tokens per word in Arabic. The same normalization is used to score speech recognition fairly; see Arabic WER explained.
Arabizi (Arabic in Latin letters) needs a different approach altogether: how LLMs handle Arabizi.
Where BroadNet Inference fits
Normalization runs in your own code, so no API is needed for it. BroadNet Inference comes in after retrieval. It does not sell embeddings, so pair a retriever and an embeddings provider of your choice with a BroadNet chat model for the answer step. GLM 5.3 and DeepSeek V4.1 Flash take 1,048,576 tokens of context, so you can pass many retrieved passages in their original, unnormalized Arabic. For choosing a model on Arabic quality, see choosing an LLM for Arabic. If you only need keyword search, Elasticsearch’s built-in arabic analyzer may be all you need.
