BroadRouter
Get started

Arabic text summarization with an LLM: prompts, length and scoring

By the BroadNet Inference team · 7 min read · Published · Updated

Name the register (usually Modern Standard Arabic, even for dialect input) and a word range in the prompt. Then check in code: count the words, confirm every number in the summary appears in the source, and retry once with the problem stated. Score with an Arabic-aware ROUGE tokenizer plus human review; default rouge-score gives Arabic 0.

Summarising Arabic text with an LLM takes one request. Getting summaries you can ship takes three decisions the request doesn’t make for you: which register to write in, how long the summary must be, and how you will know it is right. This post covers written text: articles, reports, tickets and long message threads. For spoken meetings, the Arabic meeting minutes pipeline adds the transcription step and section-by-section merging.

What should an Arabic summarization prompt say?

Five lines carry most of the result:

  1. Language and register. “Modern Standard Arabic” (الفصحى), or a named dialect. Without it, a model summarising Egyptian or Gulf input may answer in a mix.
  2. A word range. “40 to 60 words”. Ranges are easier to hit than exact counts.
  3. Facts as written. Numbers, dates and names copied from the text, not rounded or converted.
  4. Nothing new. No background the text does not contain.
  5. Shape. One paragraph, or three bullet points, or JSON with fixed fields.

Instructions can be in Arabic or English; either way, state the output language and register explicitly. Arabic prompt engineering covers register, numerals and mixed-script text in more depth. Here is the prompt used in the code below:

لخّص النص التالي بالعربية الفصحى في {lo} إلى {hi} كلمة.
- اذكر الأرقام والتواريخ والأسماء كما وردت في النص فقط.
- لا تضف معلومات غير موجودة في النص.
- اكتب فقرة واحدة دون عناوين أو نقاط.

How do you summarise dialect text?

Most business text in Arabic is not MSA: customer reviews, WhatsApp threads and social posts are in dialect, often mixed with English or Arabizi. Three choices help:

  • Summarise into MSA by default. It is neutral across regions and easier to search. If the summary goes back to the same customer, match their dialect instead.
  • Keep key quotes verbatim. Ask for one short quote in the original wording when tone matters, such as a complaint. Paraphrasing dialect into MSA may soften it.
  • Test on your dialects. Dialect understanding varies between models; see how LLMs cope with Gulf and Egyptian Arabic.

Public data won’t do that testing for you. The large Arabic summarization sets are MSA news, and we did not find a public Gulf or Egyptian summarization benchmark to point you to.

How do you control summary length?

Ask in words, then measure in words. Don’t trust the model’s own count, and don’t use a token budget as the unit: tokens per Arabic word vary by model. We measured a 96-word Arabic news paragraph at 166 tokens with the official Qwen3.8 27B tokenizer, and its 58-word summary at 114 tokens. Across the tokenizers of BroadNet’s four chat models, counted with the Arabic token counter’s tokenizer files, the same paragraph ranged from 166 to 224 tokens. Tokens per word in Arabic has the larger measurement.

Splitting on whitespace is the right count for most uses, because it is what readers see. Arabic attaches و, ب, ل and ال to the word, so a 50-word Arabic summary carries more than 50 English words of content. Set your range from Arabic examples you like, not from an English target.

Two more rules on length:

  • max_tokens is not a length control. It cuts the reply off mid-sentence. Set it well above the summary size, especially on reasoning models: BroadNet’s docs say DeepSeek V4.1 Flash, GLM 5.3 and Qwen3.8 27B reason before they answer, and that reasoning counts toward max_tokens. See empty responses from reasoning models.
  • Retry once, with the number. “Your summary is 17 words; I asked for 40 to 60” fixes more than repeating the instruction.

How do you catch invented details?

Abstractive summarizers invent things. Maynez et al. (ACL 2020) found “substantial amounts of hallucinated content” in all the model summaries their annotators reviewed; that was before today’s LLMs, but no model is immune. You can’t check meaning cheaply in code, but you can check numbers: every number in the summary should appear in the source. Map Eastern Arabic digits (٠–٩) and Persian digits (۰–۹) to 0–9 first, or ١٢ and 12 will look different.

import os, re
from openai import OpenAI

client = OpenAI(base_url="https://inference.broadnet.ai/v1", api_key=os.environ["BROADNET_API_KEY"])

PROMPT = """لخّص النص التالي بالعربية الفصحى في {lo} إلى {hi} كلمة.
- اذكر الأرقام والتواريخ والأسماء كما وردت في النص فقط.
- لا تضف معلومات غير موجودة في النص.
- اكتب فقرة واحدة دون عناوين أو نقاط.

النص:
{text}"""

DIGITS = str.maketrans("٠١٢٣٤٥٦٧٨٩۰۱۲۳۴۵۶۷۸۹", "01234567890123456789")
def numbers(s: str) -> set[str]:
    return set(re.findall(r"\d+(?:[.,٫]\d+)?", s.translate(DIGITS)))

def words(s: str) -> int:
    return len(s.split())          # clitics (و، ال، ب) stay attached: count what readers see

def check(source: str, summary: str, lo: int, hi: int) -> list[str]:
    issues = []
    n = words(summary)
    if not lo <= n <= hi:
        issues.append(f"الطول {n} كلمة، والمطلوب من {lo} إلى {hi}.")
    extra = numbers(summary) - numbers(source)
    if extra:
        issues.append("أرقام غير موجودة في النص: " + "، ".join(sorted(extra)))
    return issues

def summarize(text: str, lo=40, hi=60, model="broadnet/qwen3.8-27b", attempts=2) -> dict:
    messages = [{"role": "user", "content": PROMPT.format(lo=lo, hi=hi, text=text)}]
    for i in range(attempts):
        r = client.chat.completions.create(model=model, messages=messages, max_tokens=2000)
        summary = (r.choices[0].message.content or "").strip()
        issues = check(text, summary, lo, hi)
        if not issues:
            return {"summary": summary, "attempts": i + 1, "issues": []}
        messages += [{"role": "assistant", "content": summary},
                     {"role": "user", "content": "صحّح الملخص: " + " ".join(issues)}]
    return {"summary": summary, "attempts": attempts, "issues": issues}   # send to review

We tested this against a local mock server (no live API call) with a synthetic 96-word article about a city bus plan. The mock’s first reply was 17 words and said ٣٠٠ buses where the article said ٢٤٠. check returned both problems: the measured length, and “numbers not in the text: 300”. The second, corrected reply passed at 58 words, so the function returned it with attempts: 2.

The number check misses things: a wrong name, a reversed decision, a claim with no number in it. It is a filter, not a verdict. For names, reuse the span check from Arabic named entity extraction: extract entities from the summary and confirm each one appears in the source.

How do you summarise long Arabic documents?

If the text fits the context window with room to spare, send it whole. Qwen3.8 27B takes 262,144 tokens per request and GLM 5.3 1,048,576, prompt and answer together; Gemma 4 VL 26B takes 32,768. Long inputs have a known weakness, though: models use facts in the middle of a long context less reliably (lost in the middle). For a long report, summarise each section, then summarise the summaries, and keep section headings in the final prompt. The meeting post shows the merge step in code; long context vs RAG covers when to retrieve instead.

How do you evaluate Arabic summaries?

Use three layers: an automatic overlap score to catch regressions, a semantic score for a second opinion, and people for the decision.

ROUGE, with the right tokenizer. Google’s rouge-score package (version 0.1.2, still the current release on PyPI on 16 November 2026) lowercases text and replaces every character outside a–z and 0–9 with a space. Arabic disappears. We checked:

import re
from rouge_score import rouge_scorer

ref = "أعلنت البلدية خطة لتوسعة النقل العام تضيف ١٢ خطاً للحافلات"
hyp = "البلدية تعلن خطة توسعة للنقل العام بإضافة ١٢ خطا للحافلات"

print(rouge_scorer.RougeScorer(["rouge1"]).score(ref, hyp)["rouge1"].fmeasure)   # 0.0

class ArabicTokenizer:
    """Whitespace tokens, no diacritics, unified alef, optional light prefix stripping."""
    def __init__(self, strip_prefixes=False):
        self.strip = strip_prefixes
    def tokenize(self, text):
        text = re.sub(r"[ً-ٰٟـ]", "", text)
        text = re.sub("[إأآ]", "ا", text).replace("ى", "ي").replace("ة", "ه")
        toks = re.findall(r"\w+", text)
        if self.strip:   # crude: drop و/ب/ل/ال prefixes on longer words
            toks = [re.sub(r"^(وال|بال|لل|ال|و|ب|ل)(?=\w{3,})", "", t) for t in toks]
        return toks

for strip in (False, True):
    s = rouge_scorer.RougeScorer(["rouge1"], tokenizer=ArabicTokenizer(strip))
    print(strip, round(s.score(ref, hyp)["rouge1"].fmeasure, 3))   # False 0.6, True 0.8

The same two sentences score 0, 0.6 or 0.8 depending only on tokenization. That is the real lesson: Arabic morphology (للنقل against النقل, بإضافة against تضيف) moves ROUGE a lot, so a score means nothing without its tokenizer. The XL-Sum authors publish a multilingual ROUGE fork with language-specific stemming, including Arabic; use it, or your own tokenizer, and keep it fixed across every run you compare.

BERTScore compares meaning through embeddings, so it tolerates paraphrase. For languages other than English and Chinese its default model is bert-base-multilingual-cased. Treat it as a second signal, not a replacement for reading.

People, with a rubric. Have an Arabic reader score 50 to 100 summaries on three questions: is anything important missing, is anything stated that the source doesn’t say, and is the register right. An LLM judge can pre-screen at volume once its scores agree with your readers on a sample. Evaluating an LLM before switching covers building that set.

For public reference data:

Dataset What it holds Notes
XL-Sum Arabic 46,897 BBC Arabic article and summary pairs (37,519 / 4,689 / 4,689) CC BY-NC-SA 4.0: non-commercial research only
EASC 153 articles, five human extractive summaries each Small; Wikipedia and two newspapers
LANS 8.4 million articles with summaries from 22 Arab newspapers, 1999 to 2019 Summaries come from site metadata

All three are MSA news. Your support tickets and reports are not, so a sample of your own text, summarised by a person, is the test that counts.

Which BroadNet model should you use, and what does it cost?

For a pilot, Gemma 4 VL 26B runs on the $5 starting credit at $0.10 input and $0.22 output per 1M tokens; its 32,768-token window fits most single articles. Qwen3.8 27B, which BroadNet reports testing for fluent Arabic writing and Gulf and Egyptian dialect understanding, costs $0.175 input and $2.17 output after a top-up. Use GLM 5.3 when one document is longer than Qwen’s window.

The arithmetic for 10,000 articles of 1,000 tokens each, with 150-token summaries: 10 million input tokens and 1.5 million output tokens, so on Qwen3.8 27B that is 10 × $0.175 plus 1.5 × $2.17. Add reasoning tokens, which bill as output, and any retries; measure both on a sample from the usage block. If the prompt is the same for every article, put it first so it can be billed at the cached input rate.

Where BroadNet Inference fits

BroadRouter is BroadNet’s own site. BroadNet Inference serves the summarization step through an OpenAI-compatible API, so the code above runs with the OpenAI Python SDK. Its docs say prompts and answers are not retained after the request. It publishes no Arabic summarization benchmark of its own, and nothing here measures summary quality on its models; run the checks above on your own texts. For spoken input, pair it with BroadNet Transcribe Arabic as in the meeting minutes post.

Questions

Can an LLM summarise Arabic text well?

Yes, but fluent is not the same as accurate. Check lengths and numbers in code, and have an Arabic reader score a sample for missing points and invented details before you rely on it.

Should a summary of dialect text be in dialect or MSA?

Usually MSA, because it reads as neutral and searchable across regions. Keep short quotes in the original dialect when the wording matters, such as a complaint. Say which you want in the prompt; models otherwise drift between the two.

Why does ROUGE give 0 for my Arabic summaries?

The widely used rouge-score Python package replaces characters outside a-z and 0-9 with spaces by default, so Arabic text produces no tokens. Pass your own tokenizer, or use the multilingual ROUGE fork from the XL-Sum authors.

How do I control the length of an Arabic summary?

Ask for a word range, not a token count, and measure the reply by splitting on whitespace. If it misses, resend once with the measured length. Remember that Arabic words often carry attached prefixes, so a 50-word Arabic summary holds more than 50 English words of meaning.

Which dataset can I use to test Arabic summarization?

XL-Sum has 46,897 Arabic BBC article and summary pairs (non-commercial licence). EASC has 153 articles with five human extractive summaries each. Both are MSA news, so add a sample of your own texts, especially dialect ones.

Read as MarkdownAsk ChatGPTAsk ClaudeAsk Perplexity