Named entity extraction turns a paragraph into records your code can use: who, which organisation, where and when. Arabic adds four problems English pipelines rarely meet: optional diacritics, several spellings of the same letter, names written in Latin letters (Arabizi), and dates in two calendars. This post builds an extraction step on an OpenAI-compatible LLM that handles each one in code, and a scoring script to tell you whether it works on your text.
For classifying whole messages rather than pulling spans out of them, see Arabic text classification. For the JSON request options themselves, see JSON mode vs json_schema.
How well do LLMs do Arabic NER?
The most relevant public measurement we found is LAraBench (Abdelali et al., EACL 2024), which ran GPT-3.5 and GPT-4 on Arabic NER sets as token-by-token tagging. Macro-F1, higher is better:
| Dataset | GPT-3.5 zero-shot | GPT-4 zero-shot | GPT-4 3-shot | Fine-tuned system cited |
|---|---|---|---|---|
| ANERcorp (news) | 0.210 | 0.355 | 0.420 | 0.886 |
| AQMAR (Wikipedia) | 0.230 | 0.365 | 0.390 | 0.690 |
| QASR (transcripts) | 0.208 | 0.504 | n/a | 0.698 |
More examples helped: GPT-4 on ANERcorp rose from 0.355 with no examples to 0.451 with ten. The authors also had to remap tags the model wrote in its own format, such as PER-B for B-PER.
Read this with care. These are 2023 models doing a strict tagging task on benchmark label sets. A current model asked for a short list of people and companies in a support ticket may do much better, or not. The table tells you two things that still hold: an LLM is not automatically a good NER system, and you need your own measurement.
Which Arabic NER datasets exist?
| Dataset | Text | Size and labels |
|---|---|---|
| ANERcorp | News | About 150,000 tokens; person, location, organisation, miscellaneous. CAMeL Lab publishes a standard train/test split |
| CLEANANERCorp | ANERcorp, corrected | Fixes annotation errors, missing labels and inconsistencies found in the original |
| AQMAR | Arabic Wikipedia | 74,000 tokens from 28 articles |
| Wojood | MSA and dialect | About 550,000 tokens, 21 entity types, nested; used in the WojoodNER 2023 shared task |
| CANERCorpus | Classical Arabic hadith | More than 7,000 hadiths, 20 types on Islamic topics |
Use them to sanity-check a prompt, not as your acceptance test. Your entity types (customer, product, branch, contract number) will differ from theirs, and their licences differ too. Check each one before commercial use.
What should the prompt ask for?
Three rules make the output checkable:
- Copy the span exactly. No translation, no transliteration, no “completing” a first name into a full name. If you need an English spelling, ask for it in a separate field and treat it as unverified.
- A closed type list. PERSON, ORG, LOCATION, DATE, or your own set. Free-form types can’t be scored.
- Dates as parts, not conversions. Ask for year, month and day as written, plus the calendar. A trailing هـ marks a Hijri date; م marks Gregorian.
Write the instructions in English or Arabic; Arabic prompt engineering covers the trade-off. Put the JSON shape in the system prompt and keep it identical across requests so it can be cached.
How do you check entities against the source?
The model may return a name the text never mentions, or copy a name correctly but without its diacritics. A plain substring test can’t tell those apart. Normalise both sides first: remove tashkeel and tatweel, unify alef forms, and join عبد ال so عبد الله matches عبدالله. The rules are the same as in Arabic text normalization for search, applied to matching only. Keep the original span for display.
import json, os, re
from hijridate import Hijri
from openai import OpenAI
client = OpenAI(base_url="https://inference.broadnet.ai/v1", api_key=os.environ["BROADNET_API_KEY"])
TYPES = {"PERSON", "ORG", "LOCATION", "DATE"}
SYSTEM = """Extract named entities from the Arabic text. Reply with JSON only:
{"entities": [{"text": "...", "type": "PERSON|ORG|LOCATION|DATE"}],
"dates": [{"text": "...", "calendar": "hijri|gregorian", "year": 0, "month": 0, "day": 0}]}
Rules: copy "text" exactly as it appears in the input, including any diacritics.
Do not translate, transliterate or complete names. A date followed by هـ is Hijri.
If there are no entities, return empty lists."""
DIACRITICS = re.compile(r"[ً-ٰٟـ]") # tashkeel, dagger alef, tatweel
def norm(s: str) -> str:
s = DIACRITICS.sub("", s)
s = re.sub("[إأآٱ]", "ا", s).replace("ى", "ي").replace("عبد ال", "عبدال")
return " ".join(s.split())
def validate(source: str, data: dict) -> tuple[list, list]:
"""Keep entities that really occur in the source; report everything else."""
src, kept, problems = norm(source), [], []
for e in data.get("entities", []):
if e.get("type") not in TYPES:
problems.append(f"unknown type: {e}")
elif norm(e.get("text", "")) not in src:
problems.append(f"not in source: {e['text']}")
else:
kept.append(e)
for d in data.get("dates", []):
if d.get("calendar") == "hijri":
try:
d["iso"] = Hijri(d["year"], d["month"], d["day"]).to_gregorian().isoformat()
except (ValueError, OverflowError) as err:
problems.append(f"bad Hijri date {d['text']}: {err}")
return kept, problems
def extract(text: str, model="broadnet/qwen3.8-27b") -> dict:
r = client.chat.completions.create(
model=model, max_tokens=2000,
messages=[{"role": "system", "content": SYSTEM}, {"role": "user", "content": text}])
data = json.loads(r.choices[0].message.content)
data["entities"], data["problems"] = validate(text, data)
return data
text = "التقى خالِد بن سَعِيد مديرةَ شركة النخيل للتقنية في جدة يوم ١٥ جمادى الآخرة ١٤٤٥هـ."
print(json.dumps(extract(text), ensure_ascii=False, indent=1))
We ran this against a local mock server (no live API call) whose canned reply added one invented person and one impossible date (month 13). The output kept خالد بن سعيد (the reply dropped the diacritics; the normalised match still found it), شركة النخيل للتقنية and جدة. It converted ١٥ جمادى الآخرة ١٤٤٥هـ to 2023-12-28, and listed two problems: the invented name was “not in source”, and month 13 raised “month must be in 1-12”. The names and company are fictional.
Notes on the code:
max_tokens=2000leaves room for reasoning. BroadNet’s docs say DeepSeek V4.1 Flash, GLM 5.3 and Qwen3.8 27B reason before answering, and that reasoning counts towardmax_tokens. See empty responses from reasoning models.json.loadscan fail. Wrap it, and on failure resend once with the error, as in the repair loop. On GLM 5.3 and Qwen3.8 27B, which BroadNet documents with structured output, add aresponse_formatschema too. Validation stays either way.- Substring matching is deliberately loose. It confirms the span exists; it does not confirm the type. Types are what the scoring step below measures.
How do you handle Hijri dates?
Arabic text writes dates many ways: ١٥/٦/١٤٤٥هـ, ١٥ جمادى الآخرة ١٤٤٥, or a Gregorian date with month names that vary by region (نوفمبر in Egypt and the Gulf, تشرين الثاني in the Levant). A model can usually split these into parts, and the code checks the parts. Converting between calendars is arithmetic on a published table, and a model has no reason to get it right.
hijridate, formerly hijri-converter, converts using the Umm al-Qura calendar. Its documented range is 1343 to 1500 AH (1 August 1924 to 16 November 2077 CE). Out of range it raises OverflowError; an invalid month or day raises ValueError, which is why the code catches both. Its docs say it is not intended for religious purposes where crescent sighting is preferred, so a date set by sighting can differ from the converted value. Store the date as written next to the converted one.
Eastern Arabic digits need no special handling here: the model returns integers. If you parse dates yourself, map ٠١٢٣٤٥٦٧٨٩ (and the Persian ۰–۹) to 0–9 first.
What about names in Arabizi and mixed text?
Customer messages often write names in Latin letters: “ana 7a2ool la Mo7ammad bokra”. The exact-span rule still works: the entity text is “Mo7ammad”, and the substring check passes. What fails is matching that record to محمد in your CRM. Do that in a second step: convert the span to Arabic script, as in handling Arabizi, then match against your records with the same normalisation. Keep a confidence flag on any record that came through a conversion.
The same applies to English company names inside Arabic sentences, and to Latin spellings of Arabic names (Mohammed, Muhammad, Mohamed). Don’t ask the extraction step to pick one spelling.
How do you score the extractor?
Label 100 to 200 real samples by hand, with the same type list. Then compare predictions with entity-level exact match on the normalised text and type, and report precision, recall and F1 per type:
from collections import Counter
def score(gold: list[list[dict]], pred: list[list[dict]]) -> dict:
"""Entity-level exact match on (normalised text, type), micro-averaged, plus per type."""
tp, fp, fn = Counter(), Counter(), Counter()
for g, p in zip(gold, pred):
G = Counter((norm(e["text"]), e["type"]) for e in g)
P = Counter((norm(e["text"]), e["type"]) for e in p)
for (t, ty), n in (G | P).items():
hit = min(G[(t, ty)], P[(t, ty)])
tp[ty] += hit; fp[ty] += P[(t, ty)] - hit; fn[ty] += G[(t, ty)] - hit
def prf(a, b, c):
p = a / (a + b) if a + b else 0.0
r = a / (a + c) if a + c else 0.0
return {"p": round(p, 3), "r": round(r, 3), "f1": round(2 * p * r / (p + r), 3) if p + r else 0.0}
out = {ty: prf(tp[ty], fp[ty], fn[ty]) for ty in sorted(tp | fp | fn)}
out["micro"] = prf(sum(tp.values()), sum(fp.values()), sum(fn.values()))
return out
On a two-sentence toy set (three gold entities; the prediction typed جدة as ORG instead of LOCATION and added one extra place), it printed micro precision 0.5, recall 0.667 and F1 0.571, and LOCATION F1 0. A per-type view shows you where to spend effort: a weak ORG score usually means your type definitions need examples, not a bigger model.
Decide in advance what counts as a match. Exact match is strict: “شركة النخيل” against a gold “شركة النخيل للتقنية” scores as one false positive and one false negative. If partial spans are acceptable for your use, report a second, overlap-based score, but keep the strict one as the headline.
Which BroadNet model should you use?
All prices are per 1M tokens, from BroadNet’s pricing page.
| Model | Why | Input / output |
|---|---|---|
| Qwen3.8 27B | Structured output documented; BroadNet reports it tested in Arabic, including Gulf and Egyptian dialect understanding; reads images | $0.175 / $2.17 |
| GLM 5.3 | Structured output documented; 1,048,576-token context for very long documents | $0.78 / $3.46 |
| Gemma 4 VL 26B | Runs on the $5 starting credit; no structured output documented, so validate prompt-only JSON; 32,768-token context | $0.10 / $0.22 |
Run the scoring script on your sample with each candidate and pick on F1 per type and cost, not on the vendor’s description. Arabic usually takes more tokens than English for the same content; tokens per word in Arabic has measured ratios. For entities inside scanned forms or photos, extract from the image directly, as in reading Arabic invoices with vision.
Where BroadNet Inference fits
BroadRouter is BroadNet’s own site. BroadNet Inference gives you OpenAI-compatible access to the models above with one key, so the code runs with the OpenAI Python SDK after changing the base URL. Its docs say prompts and answers are not retained after the request, which matters when the text is full of people’s names. It has no dedicated NER endpoint and publishes no Arabic NER benchmark of its own, so measure on your data. If you need a self-hosted tagger for high volume, a fine-tuned model trained on Wojood or your own labels may be cheaper and more consistent than any LLM; the published gap above is the reason to check.
