---
title: "Arabic text normalization for search and RAG | BroadRouter"
description: "Tashkeel, alef forms, ta marbuta, hamza seats, Persian letters, digits and Unicode: which Arabic normalization rules help search, which hurt, and tested Python code."
url: https://broadrouter.com/blog/arabic-text-normalization-search/
lang: en
updated: 2026-11-15
publisher: BroadRouter (a BroadNet site); API by BroadNet Inference, https://inference.broadnet.ai
---

# Arabic text normalization for search and RAG: what to change, what not

By the BroadNet Inference team · 7 min read · Published 2026-11-15 · Updated 2026-11-16

For search and RAG indexes, normalize both the stored copy and every query the same way: Unicode NFKC, strip tashkeel and tatweel, unify alef forms, map ى to ي and ة to ه, and convert digits. Keep the original text for display and for the LLM. Treat hamza seats and ta marbuta as tunable: they raise recall but merge some distinct words.

## Key takeaways

-   Normalize the search key, never the text you show or cite. Store the original beside it.
-   In a classic study on TREC-2001 Arabic newswire, normalization alone raised average precision from .194 to .238, and light stemming on top reached .376.
-   Run Unicode NFKC first. It composes decomposed hamza, unpacks presentation forms and ligatures, and makes the later rules reliable.
-   Libraries disagree. PyArabic's normalize\_alef maps ى to ا; CAMeL Tools maps ى to ي. Pick one rule set and use it everywhere.
-   For embedding search, test normalized and raw input on your own queries. Normalization rules were measured on keyword search.

**On this page**

1.  Does normalization actually help retrieval?
2.  What should you normalize, and what does each rule cost?
3.  Why should Unicode normalization come first?
4.  What should you not normalize?
5.  How do CAMeL Tools, PyArabic and Lucene differ?
6.  What does a tested normalizer look like?
7.  How does normalization fit embeddings and RAG?
8.  Where BroadNet Inference fits

Arabic lets the same word be written several ways. A user types احمد; the document says أَحْمَد. A search index that compares exact characters misses that match, and so does the keyword half of a RAG pipeline. Normalization fixes this by mapping variants to one form before you compare them. Done carelessly, it also merges words that mean different things. This post goes rule by rule: what each one does, when to apply it, what it breaks, and how the main libraries differ. It ends with tested Python.

For the full Arabic RAG pipeline (chunking, hybrid search, cited answers), see [Arabic RAG that works](https://broadrouter.com/blog/arabic-rag-search/). This post goes deeper on the normalization step.

## Does normalization actually help retrieval?

Yes, for keyword search, and the evidence is old but clear. Larkey, Ballesteros and Connell (SIGIR 2002) tested Arabic retrieval on the TREC-2001 newswire collection. Before stemming, they normalized corpus and queries: removed diacritics and punctuation, unified alef forms, and mapped word-final ى and ة. Their monolingual results, with unexpanded queries:

| Processing | Average precision | Change vs raw (from the rounded figures) |
| --- | --- | --- |
| Raw text | .194 | — |
| Normalized | .238 | +22.7% |
| Normalized + light stemming (light8) | .376 | +93.8% |

Two cautions. This is 2002 keyword retrieval on news, not embedding search. And the largest gain came from light stemming (removing prefixes such as و and ال and common suffixes), which is a separate step from normalization. Lucene’s `ArabicAnalyzer` and Elasticsearch’s `arabic` analyzer apply both: a normalizer, then a light stemmer.

## What should you normalize, and what does each rule cost?

| Rule | Example | Search index | Risk |
| --- | --- | --- | --- |
| Unicode NFKC | ﻻ → لا, ا + ٔ → أ | Yes, always first | Rewrites compatibility characters in other scripts too |
| Remove invisible marks | zero-width joiners, RLM/LRM, BOM | Yes | None for Arabic; ZWNJ matters in Persian |
| Strip tashkeel | أَحْمَدُ → أحمد | Yes | Merges words told apart only by vowels |
| Remove tatweel (ـ) | الـــمدينة → المدينة | Yes | None |
| Unify alef | أ إ آ ٱ → ا | Yes | Merges a few pairs: سأل (asked) and سال (flowed) |
| Alef maksura | ى → ي | Usually | Merges على (on) and علي (Ali) |
| Ta marbuta | ة → ه | Usually | Merges كتابة (writing) and كتابه (his book) |
| Hamza seats | ؤ ئ → ء | Optional | Merges مسؤول and مسئول (good) but also some distinct words |
| Persian letters | ی → ي, ک → ك | Yes, if users have Persian keyboards | None for Arabic text |
| Digits | ٢٠٢٦ and ۲۰۲۶ → 2026 | Yes | None; keep the original for display |

The first column is for the index key and the query. The text you store, show and pass to the model stays as written.

## Why should Unicode normalization come first?

The same visible letter can be stored in different ways. أ can be one code point (U+0623) or bare alef followed by a combining hamza (U+0627 U+0654). Older PDFs and some keyboards produce Arabic presentation forms (U+FB50 to U+FEFF), such as the single-character lam-alef ﻻ. Under [Unicode normalization](https://unicode.org/reports/tr15/), NFC composes the hamza onto the alef, and NFKC also unpacks presentation forms and ligatures. In Python, `unicodedata.normalize("NFKC", "ﻻ")` returns لا, and `ﷲ` becomes الله.

Run this before anything else. Otherwise a regex that replaces أ misses the decomposed form. There is a second trap: CAMeL Tools’ `dediac_ar` removes the characters in its `AR_DIAC_CHARSET` (U+064B to U+0652 and U+0670). That set leaves out the combining hamza and madda marks (U+0653 to U+0655), so on decomposed text they survive dediacritization. NFKC composes them first.

NFKC is a compatibility mapping. It also changes characters outside Arabic, for example full-width Latin letters. That is usually what you want in a search key, and one more reason to keep the original beside it.

## What should you not normalize?

-   **The text you display, cite or quote.** Normalized Arabic looks misspelled (المدينه). Store the original next to the key.
-   **What you send to the LLM.** Models read hamza and tashkeel fine, and exact text keeps names and quotes correct. Stripping tashkeel saves some tokens; check how many with the [Arabic token counter](https://broadrouter.com/tools/arabic-token-counter/) before you trade accuracy for it.
-   **Text where marks carry meaning:** Quran, hadith, poetry, children’s books, pronunciation and diacritization tasks (see [adding diacritics with an LLM](https://broadrouter.com/blog/arabic-diacritization-llm/)).
-   **Spelling or grammar correction:** normalising first erases the hamza and taa marbuta errors you want to fix; see [Arabic grammar and spelling correction with an LLM](https://broadrouter.com/blog/arabic-grammar-spelling-correction-llm/).
-   **Identifiers.** Product codes, emails, URLs and Latin text must not go through Arabic rules. Apply normalization to Arabic-script fields only, or to Arabic tokens only.
-   **Fields people search exactly.** For legal names or ID-like fields, index a raw copy as well, so an exact match can still rank first.

## How do CAMeL Tools, PyArabic and Lucene differ?

They don’t implement the same rules, and mixing them gives an index and queries that never match. We read each library’s documentation and source, and ran PyArabic 0.6.15:

| Function | What it actually does |
| --- | --- |
| CAMeL `normalize_unicode` | NFKC (or NFC with `compatibility=False`); also expands ﷼ to ريال and ﷽ to its full phrase |
| CAMeL `normalize_alef_ar` | إ أ ٱ آ → ا |
| CAMeL `normalize_alef_maksura_ar` | ى → ي |
| CAMeL `normalize_teh_marbuta_ar` | ة → ه |
| CAMeL `dediac_ar` | Removes U+064B to U+0652 and U+0670; leaves tatweel |
| PyArabic `strip_tashkeel` | Removes all harakat including shadda |
| PyArabic `strip_tatweel` | Removes ـ |
| PyArabic `normalize_alef` | Every alef form **and ى** → ا, so إلى becomes الا |
| PyArabic `normalize_hamza` | Default method maps every hamza form to ء, including أ, so أحمد becomes ءحمد |
| Lucene `ArabicNormalizer` | Hamza-on-alef forms → ا, ة → ه, ى → ي, removes harakat and tatweel |

The PyArabic rows surprise people. Its `normalize_alef` docstring says alef maksura is an exception, but in 0.6.15 the `ALEFAT` set includes ى, and the function maps it to ا. Its `normalize_hamza` is built for hamza-agnostic matching, not general search. Order matters too: run alef normalization first, and `normalize_hamza` then only touches ؤ, ئ and ء, so مسؤول and مسئول both become مسءول. Its `tasheel` method gives مسوول and مسيول instead, which don’t match each other.

Elasticsearch’s `arabic_normalization` filter uses the Lucene normalizer, so a Python pipeline that wants to match an Elasticsearch index should copy Lucene’s rules exactly. That means ى → ي, not ى → ا.

## What does a tested normalizer look like?

This function follows Lucene’s choices, adds Unicode, digit and Persian-letter handling, and makes the risky rules switchable. It uses only the standard library.

```python
import re
import unicodedata

DIACRITICS = re.compile("[ً-ٰٟ]")  # tashkeel, extra marks, superscript alef
INVISIBLE = re.compile("[​-‏‪-‮⁦-⁩﻿]")  # zero-width, bidi
ALEF = re.compile("[آأإٱ]")    # آ أ إ ٱ -> ا
DIGITS = str.maketrans("٠١٢٣٤٥٦٧٨٩۰۱۲۳۴۵۶۷۸۹", "0123456789" * 2)
FARSI = str.maketrans({"ی": "ي", "ک": "ك"})  # ی -> ي, ک -> ك

def normalize(text, *, alef=True, alef_maksura=True, ta_marbuta=True,
              hamza_seats=False, digits=True):
    text = unicodedata.normalize("NFKC", text)     # compose marks, unpack ligatures
    text = INVISIBLE.sub("", text)
    text = DIACRITICS.sub("", text).replace("ـ", "")   # tashkeel, tatweel
    text = text.translate(FARSI)
    if alef:
        text = ALEF.sub("ا", text)
    if alef_maksura:
        text = text.replace("ى", "ي")    # ى -> ي
    if ta_marbuta:
        text = text.replace("ة", "ه")    # ة -> ه
    if hamza_seats:
        text = re.sub("[ؤئ]", "ء", text)  # ؤ ئ -> ء
    if digits:
        text = text.translate(DIGITS)
    return text
```

What it returned on our test strings:

| Input | Output | Note |
| --- | --- | --- |
| أَحْمَدُ | احمد | Tashkeel and hamza gone |
| الـــمدينة | المدينه | Tatweel and ta marbuta |
| إلى | الي | Alef and alef maksura |
| ا + U+0654 + مل | امل | Decomposed hamza composed, then unified |
| ﻻبد | لابد | Presentation form unpacked |
| ٢٠٢٦ / ۱۴۰۵ | 2026 / 1405 | Arabic-Indic and Persian digits |
| مسؤول / مسئول | Both مسءول with `hamza_seats=True` | Unchanged by default |
| فارسی | فارسي | Persian ya |

Use it in two places: when you build the index key and when you read a query. Store `original` and `normalize(original)` side by side, search on the second and return the first. The free [Arabic text normalizer](https://broadrouter.com/tools/arabic-text-normalizer/) lets you paste text and see each rule’s effect before you choose your settings.

## How does normalization fit embeddings and RAG?

A RAG pipeline usually has two retrievers: keyword search (BM25) and [embeddings](https://broadrouter.com/glossary/embeddings/). Normalization clearly helps the keyword side. For the embedding side, the evidence above doesn’t apply: embedding models are trained on raw text, and many already place احمد and أحمد close together. Normalizing before embedding can help, do nothing, or hurt. Build 50 to 100 real queries with known answers, run them with raw and normalized input, and keep whichever recalls more. Whatever you pick, apply it identically to documents and queries.

Normalization also lowers token counts slightly, because diacritics and tatweel are extra characters for the [tokenizer](https://broadrouter.com/glossary/tokenizer/). Measured ratios are in the [tokenizer efficiency dataset](https://broadrouter.com/data/tokenizer-efficiency/) and [tokens per word in Arabic](https://broadrouter.com/blog/tokens-per-word-arabic-english/). The same normalization is used to score speech recognition fairly; see [Arabic WER explained](https://broadrouter.com/blog/arabic-asr-wer-explained/).

Arabizi (Arabic in Latin letters) needs a different approach altogether: [how LLMs handle Arabizi](https://broadrouter.com/blog/arabizi-llm-handling/).

## Where BroadNet Inference fits

Normalization runs in your own code, so no API is needed for it. BroadNet Inference comes in after retrieval. It does not sell embeddings, so pair a retriever and an embeddings provider of your choice with a BroadNet chat model for the answer step. [GLM 5.3](https://broadrouter.com/models/glm-5.3/) and DeepSeek V4.1 Flash take 1,048,576 tokens of context, so you can pass many retrieved passages in their original, unnormalized Arabic. For choosing a model on Arabic quality, see [choosing an LLM for Arabic](https://broadrouter.com/blog/choosing-an-llm-for-arabic/). If you only need keyword search, Elasticsearch’s built-in `arabic` analyzer may be all you need.

Free tools for this

-   [Arabic token counter](https://broadrouter.com/tools/arabic-token-counter/) Arabic vs English token counts per tokenizer: the "Arabic tax" in tokens and money.
-   [Arabic text normalizer](https://broadrouter.com/tools/arabic-text-normalizer/) Tashkeel, tatweel, alef, ya, ta marbuta, digits and invisible characters, with a matching Python function.
-   [Speech-to-text cost calculator](https://broadrouter.com/tools/speech-to-text-cost-calculator/) Hours, calls or voice notes to a monthly transcription bill, against your current provider.

[All free tools](https://broadrouter.com/tools/)

## Terms in this article

[Arabic diacritics (tashkeel)](https://broadrouter.com/glossary/arabic-diacritics/)

Arabic diacritics, or tashkeel, are small marks written above or below letters to show short vowels, doubled consonants and the absence of a vowel. They are optional: most Arabic text, including news, omits them, so readers and software infer the vowels from context.

[Token](https://broadrouter.com/glossary/token/)

A token is the unit of text a language model reads and writes: a word, part of a word, a punctuation mark or a byte sequence produced by the model's tokenizer. LLM APIs count, limit and usually bill usage in tokens.

[Hamza](https://broadrouter.com/glossary/hamza/)

Hamza (ء) is the Arabic letter for the glottal stop. It is written on its own or on a seat: alef (أ إ), waw (ؤ) or yeh (ئ), and آ combines alef with madda. Because typed text is inconsistent about it, search engines and speech scoring often fold the alef forms into ا.

[Unicode normalization](https://broadrouter.com/glossary/unicode-normalization/)

Unicode normalization converts text to a standard form so that strings that look the same use the same code points. The four forms are NFD, NFC, NFKD and NFKC; the K forms also fold compatibility characters, such as Arabic presentation forms and ligatures, into ordinary letters.

[Arabic text normalization](https://broadrouter.com/glossary/arabic-text-normalization/)

Arabic text normalization is the step that maps spelling variants of the same word to one form before matching: removing diacritics and tatweel, unifying alef forms, and often mapping ى to ي and ة to ه. It is applied to search keys and evaluation, not to text you display.

[Embeddings](https://broadrouter.com/glossary/embeddings/)

Embeddings are lists of floating-point numbers (vectors) that a model produces to represent a piece of text, an image or other input. Inputs with similar meaning get vectors that are close together, so distance between vectors can rank search results, cluster documents or find related items.

[Tool calling](https://broadrouter.com/glossary/tool-calling/)

Tool calling, also called function calling, is an LLM API feature in which the model returns a structured request to run a function the developer has defined, with JSON arguments. The application runs the function and sends the result back; the model never runs code itself.

[RAG (retrieval-augmented generation)](https://broadrouter.com/glossary/rag/)

RAG (retrieval-augmented generation) is a pattern in which an application first retrieves passages relevant to a question from an external store, such as a search or vector index, and then adds them to the prompt so a language model can answer from that text rather than from memory alone.

## Questions

### Should I remove tashkeel (diacritics) before indexing Arabic text?

For search, yes: most queries and most modern text have none, so diacritised documents would otherwise fail to match. Strip them from the index key and the query, keep the original text for display, and keep them for Quran, poetry or pronunciation tasks where they carry meaning.

### Should ta marbuta (ة) be normalized to ha (ه)?

Usually for search, because many people type ه at the end of a word. It does merge some pairs, such as كتابة (writing) and كتابه (his book). If those collisions matter in your domain, keep ة and test both settings.

### What does Arabic normalization do in Elasticsearch?

The arabic\_normalization token filter uses Lucene's ArabicNormalizer: hamza-on-alef forms to bare alef, ta marbuta to ha, alef maksura to ya, and removal of harakat and tatweel. The built-in arabic analyzer adds stop words and a stemmer that removes attached articles, conjunctions, prepositions and common suffixes.

### Does normalizing Arabic help embedding search?

It is not established. The measured gains come from keyword retrieval. Embedding models learn from raw text and often handle spelling variants, so build a small set of real queries and compare recall with and without normalization before you commit.

### Should I normalize Arabic text before sending it to an LLM?

Generally no. Send the original text so quotes, names and citations stay exact. Stripping tashkeel saves some tokens, but only do it when the task doesn't need the marks.

## Keep reading

-   [Arabic RAG: normalisation, chunking and retrieval that works](https://broadrouter.com/blog/arabic-rag-search/)
-   [Tokens per word in Arabic vs English: why Arabic costs more](https://broadrouter.com/blog/tokens-per-word-arabic-english/)
-   [Arabic ASR word error rate (WER) explained](https://broadrouter.com/blog/arabic-asr-wer-explained/)
-   [Arabizi and LLMs: handling Franco-Arabic with numbers like 3 and 7](https://broadrouter.com/blog/arabizi-llm-handling/)
-   [Choosing an LLM API for Arabic: what to test](https://broadrouter.com/blog/choosing-an-llm-for-arabic/)
-   [Arabic named entity extraction with an LLM: JSON, dates and checks](https://broadrouter.com/blog/arabic-named-entity-extraction-llm/)
-   [Add diacritics to Arabic text with an LLM: prompt, checks, DER](https://broadrouter.com/blog/arabic-diacritization-llm/)
-   [Arabic grammar and spelling correction with an LLM: a how-to](https://broadrouter.com/blog/arabic-grammar-spelling-correction-llm/)
-   [Arabic meeting minutes from recordings: an API pipeline](https://broadrouter.com/blog/arabic-meeting-minutes-from-recordings/)
-   [Arabic OCR API comparison: Google, Azure, Textract, Mistral, LLMs](https://broadrouter.com/blog/arabic-ocr-api-comparison/)
-   [Guide: Use the OpenAI Python SDK with BroadNet Inference](https://broadrouter.com/guides/openai-python-sdk/)

Try it on your real workload. New accounts start with $5 of credit for Gemma and transcription. DeepSeek, GLM and Qwen3.8 need a top-up.

[Get started](https://inference.broadnet.ai/account/?utm_source=broadrouter&utm_medium=referral&utm_campaign=blog-arabic-text-normalization-search&utm_content=page-cta) ·

**Sources**

-   [Larkey, Ballesteros and Connell, Improving Stemming for Arabic Information Retrieval: Light Stemming and Co-occurrence Analysis, SIGIR 2002 (checked 2026-11-15)](https://ciir.cs.umass.edu/pubfiles/ir-249.pdf)
-   [CAMeL Tools 1.6.0 documentation, camel\_tools.utils.normalize (checked 2026-11-15)](https://camel-tools.readthedocs.io/en/latest/api/utils/normalize.html)
-   [CAMeL Tools source, charsets.py: the AR\_DIAC\_CHARSET used by dediac\_ar (checked 2026-11-15)](https://github.com/CAMeL-Lab/camel_tools/blob/master/camel_tools/utils/charsets.py)
-   [PyArabic 0.6.15 on GitHub, araby module (checked 2026-11-15)](https://github.com/linuxscout/pyarabic)
-   [Apache Lucene 9.12 ArabicNormalizer documentation (checked 2026-11-15)](https://lucene.apache.org/core/9_12_0/analysis/common/org/apache/lucene/analysis/ar/ArabicNormalizer.html)
-   [Elasticsearch normalization token filters: arabic\_normalization (checked 2026-11-15)](https://www.elastic.co/docs/reference/text-analysis/analysis-normalization-tokenfilter)
-   [Unicode Standard Annex #15, Unicode Normalization Forms (checked 2026-11-15)](https://unicode.org/reports/tr15/)
-   [Unicode Arabic code chart, U+0600 to U+06FF (checked 2026-11-15)](https://www.unicode.org/charts/PDF/U0600.pdf)
