BroadRouter
Get started

Arabic text normalization for search and RAG: what to change, what not

By the BroadNet Inference team · 7 min read · Published · Updated

For search and RAG indexes, normalize both the stored copy and every query the same way: Unicode NFKC, strip tashkeel and tatweel, unify alef forms, map ى to ي and ة to ه, and convert digits. Keep the original text for display and for the LLM. Treat hamza seats and ta marbuta as tunable: they raise recall but merge some distinct words.

Arabic lets the same word be written several ways. A user types احمد; the document says أَحْمَد. A search index that compares exact characters misses that match, and so does the keyword half of a RAG pipeline. Normalization fixes this by mapping variants to one form before you compare them. Done carelessly, it also merges words that mean different things. This post goes rule by rule: what each one does, when to apply it, what it breaks, and how the main libraries differ. It ends with tested Python.

For the full Arabic RAG pipeline (chunking, hybrid search, cited answers), see Arabic RAG that works. This post goes deeper on the normalization step.

Does normalization actually help retrieval?

Yes, for keyword search, and the evidence is old but clear. Larkey, Ballesteros and Connell (SIGIR 2002) tested Arabic retrieval on the TREC-2001 newswire collection. Before stemming, they normalized corpus and queries: removed diacritics and punctuation, unified alef forms, and mapped word-final ى and ة. Their monolingual results, with unexpanded queries:

Processing Average precision Change vs raw (from the rounded figures)
Raw text .194 —
Normalized .238 +22.7%
Normalized + light stemming (light8) .376 +93.8%

Two cautions. This is 2002 keyword retrieval on news, not embedding search. And the largest gain came from light stemming (removing prefixes such as و and ال and common suffixes), which is a separate step from normalization. Lucene’s ArabicAnalyzer and Elasticsearch’s arabic analyzer apply both: a normalizer, then a light stemmer.

What should you normalize, and what does each rule cost?

Rule Example Search index Risk
Unicode NFKC ﻻ → لا, ا + ٔ → أ Yes, always first Rewrites compatibility characters in other scripts too
Remove invisible marks zero-width joiners, RLM/LRM, BOM Yes None for Arabic; ZWNJ matters in Persian
Strip tashkeel أَحْمَدُ → أحمد Yes Merges words told apart only by vowels
Remove tatweel (ـ) الـــمدينة → المدينة Yes None
Unify alef أ إ آ ٱ → ا Yes Merges a few pairs: سأل (asked) and سال (flowed)
Alef maksura ى → ي Usually Merges على (on) and علي (Ali)
Ta marbuta ة → ه Usually Merges كتابة (writing) and كتابه (his book)
Hamza seats ؤ ئ → ء Optional Merges مسؤول and مسئول (good) but also some distinct words
Persian letters ی → ي, ک → ك Yes, if users have Persian keyboards None for Arabic text
Digits ٢٠٢٦ and ۲۰۲۶ → 2026 Yes None; keep the original for display

The first column is for the index key and the query. The text you store, show and pass to the model stays as written.

Why should Unicode normalization come first?

The same visible letter can be stored in different ways. أ can be one code point (U+0623) or bare alef followed by a combining hamza (U+0627 U+0654). Older PDFs and some keyboards produce Arabic presentation forms (U+FB50 to U+FEFF), such as the single-character lam-alef ﻻ. Under Unicode normalization, NFC composes the hamza onto the alef, and NFKC also unpacks presentation forms and ligatures. In Python, unicodedata.normalize("NFKC", "ﻻ") returns لا, and ﷲ becomes الله.

Run this before anything else. Otherwise a regex that replaces أ misses the decomposed form. There is a second trap: CAMeL Tools’ dediac_ar removes the characters in its AR_DIAC_CHARSET (U+064B to U+0652 and U+0670). That set leaves out the combining hamza and madda marks (U+0653 to U+0655), so on decomposed text they survive dediacritization. NFKC composes them first.

NFKC is a compatibility mapping. It also changes characters outside Arabic, for example full-width Latin letters. That is usually what you want in a search key, and one more reason to keep the original beside it.

What should you not normalize?

  • The text you display, cite or quote. Normalized Arabic looks misspelled (المدينه). Store the original next to the key.
  • What you send to the LLM. Models read hamza and tashkeel fine, and exact text keeps names and quotes correct. Stripping tashkeel saves some tokens; check how many with the Arabic token counter before you trade accuracy for it.
  • Text where marks carry meaning: Quran, hadith, poetry, children’s books, pronunciation and diacritization tasks (see adding diacritics with an LLM).
  • Spelling or grammar correction: normalising first erases the hamza and taa marbuta errors you want to fix; see Arabic grammar and spelling correction with an LLM.
  • Identifiers. Product codes, emails, URLs and Latin text must not go through Arabic rules. Apply normalization to Arabic-script fields only, or to Arabic tokens only.
  • Fields people search exactly. For legal names or ID-like fields, index a raw copy as well, so an exact match can still rank first.

How do CAMeL Tools, PyArabic and Lucene differ?

They don’t implement the same rules, and mixing them gives an index and queries that never match. We read each library’s documentation and source, and ran PyArabic 0.6.15:

Function What it actually does
CAMeL normalize_unicode NFKC (or NFC with compatibility=False); also expands ﷼ to ريال and ﷽ to its full phrase
CAMeL normalize_alef_ar إ أ ٱ آ → ا
CAMeL normalize_alef_maksura_ar ى → ي
CAMeL normalize_teh_marbuta_ar ة → ه
CAMeL dediac_ar Removes U+064B to U+0652 and U+0670; leaves tatweel
PyArabic strip_tashkeel Removes all harakat including shadda
PyArabic strip_tatweel Removes ـ
PyArabic normalize_alef Every alef form and ى → ا, so إلى becomes الا
PyArabic normalize_hamza Default method maps every hamza form to ء, including أ, so أحمد becomes ءحمد
Lucene ArabicNormalizer Hamza-on-alef forms → ا, ة → ه, ى → ي, removes harakat and tatweel

The PyArabic rows surprise people. Its normalize_alef docstring says alef maksura is an exception, but in 0.6.15 the ALEFAT set includes ى, and the function maps it to ا. Its normalize_hamza is built for hamza-agnostic matching, not general search. Order matters too: run alef normalization first, and normalize_hamza then only touches ؤ, ئ and ء, so مسؤول and مسئول both become مسءول. Its tasheel method gives مسوول and مسيول instead, which don’t match each other.

Elasticsearch’s arabic_normalization filter uses the Lucene normalizer, so a Python pipeline that wants to match an Elasticsearch index should copy Lucene’s rules exactly. That means ى → ي, not ى → ا.

What does a tested normalizer look like?

This function follows Lucene’s choices, adds Unicode, digit and Persian-letter handling, and makes the risky rules switchable. It uses only the standard library.

import re
import unicodedata

DIACRITICS = re.compile("[ً-ٰٟ]")  # tashkeel, extra marks, superscript alef
INVISIBLE = re.compile("[​-‏‪-‮⁦-⁩]")  # zero-width, bidi
ALEF = re.compile("[آأإٱ]")    # آ أ إ ٱ -> ا
DIGITS = str.maketrans("٠١٢٣٤٥٦٧٨٩۰۱۲۳۴۵۶۷۸۹", "0123456789" * 2)
FARSI = str.maketrans({"ی": "ي", "ک": "ك"})  # ی -> ي, ک -> ك

def normalize(text, *, alef=True, alef_maksura=True, ta_marbuta=True,
              hamza_seats=False, digits=True):
    text = unicodedata.normalize("NFKC", text)     # compose marks, unpack ligatures
    text = INVISIBLE.sub("", text)
    text = DIACRITICS.sub("", text).replace("ـ", "")   # tashkeel, tatweel
    text = text.translate(FARSI)
    if alef:
        text = ALEF.sub("ا", text)
    if alef_maksura:
        text = text.replace("ى", "ي")    # ى -> ي
    if ta_marbuta:
        text = text.replace("ة", "ه")    # ة -> ه
    if hamza_seats:
        text = re.sub("[ؤئ]", "ء", text)  # ؤ ئ -> ء
    if digits:
        text = text.translate(DIGITS)
    return text

What it returned on our test strings:

Input Output Note
أَحْمَدُ احمد Tashkeel and hamza gone
الـــمدينة المدينه Tatweel and ta marbuta
إلى الي Alef and alef maksura
ا + U+0654 + مل امل Decomposed hamza composed, then unified
ﻻبد لابد Presentation form unpacked
٢٠٢٦ / ۱۴۰۵ 2026 / 1405 Arabic-Indic and Persian digits
مسؤول / مسئول Both مسءول with hamza_seats=True Unchanged by default
فارسی فارسي Persian ya

Use it in two places: when you build the index key and when you read a query. Store original and normalize(original) side by side, search on the second and return the first. The free Arabic text normalizer lets you paste text and see each rule’s effect before you choose your settings.

How does normalization fit embeddings and RAG?

A RAG pipeline usually has two retrievers: keyword search (BM25) and embeddings. Normalization clearly helps the keyword side. For the embedding side, the evidence above doesn’t apply: embedding models are trained on raw text, and many already place احمد and أحمد close together. Normalizing before embedding can help, do nothing, or hurt. Build 50 to 100 real queries with known answers, run them with raw and normalized input, and keep whichever recalls more. Whatever you pick, apply it identically to documents and queries.

Normalization also lowers token counts slightly, because diacritics and tatweel are extra characters for the tokenizer. Measured ratios are in the tokenizer efficiency dataset and tokens per word in Arabic. The same normalization is used to score speech recognition fairly; see Arabic WER explained.

Arabizi (Arabic in Latin letters) needs a different approach altogether: how LLMs handle Arabizi.

Where BroadNet Inference fits

Normalization runs in your own code, so no API is needed for it. BroadNet Inference comes in after retrieval. It does not sell embeddings, so pair a retriever and an embeddings provider of your choice with a BroadNet chat model for the answer step. GLM 5.3 and DeepSeek V4.1 Flash take 1,048,576 tokens of context, so you can pass many retrieved passages in their original, unnormalized Arabic. For choosing a model on Arabic quality, see choosing an LLM for Arabic. If you only need keyword search, Elasticsearch’s built-in arabic analyzer may be all you need.

Questions

Should I remove tashkeel (diacritics) before indexing Arabic text?

For search, yes: most queries and most modern text have none, so diacritised documents would otherwise fail to match. Strip them from the index key and the query, keep the original text for display, and keep them for Quran, poetry or pronunciation tasks where they carry meaning.

Should ta marbuta (ة) be normalized to ha (ه)?

Usually for search, because many people type ه at the end of a word. It does merge some pairs, such as كتابة (writing) and كتابه (his book). If those collisions matter in your domain, keep ة and test both settings.

What does Arabic normalization do in Elasticsearch?

The arabic_normalization token filter uses Lucene's ArabicNormalizer: hamza-on-alef forms to bare alef, ta marbuta to ha, alef maksura to ya, and removal of harakat and tatweel. The built-in arabic analyzer adds stop words and a stemmer that removes attached articles, conjunctions, prepositions and common suffixes.

Does normalizing Arabic help embedding search?

It is not established. The measured gains come from keyword retrieval. Embedding models learn from raw text and often handle spelling variants, so build a small set of real queries and compare recall with and without normalization before you commit.

Should I normalize Arabic text before sending it to an LLM?

Generally no. Send the original text so quotes, names and citations stay exact. Stripping tashkeel saves some tokens, but only do it when the task doesn't need the marks.

Read as MarkdownAsk ChatGPTAsk ClaudeAsk Perplexity