BroadRouter

Lost in the middle

Updated

Lost in the middle is the observed tendency of language models to use relevant information best when it appears near the beginning or end of a long input and worst when it sits in the middle. It was named in a 2023 study by Liu et al.

Also called: lost-in-the-middle effect, middle-of-context problem, long-context position bias, U-shaped context performance

Put the answer in the middle of a long prompt and many models are more likely to miss it than if it sits at the start or end.

How it works

Liu et al. (2023) tested models on two tasks. In multi-document question answering, the model got a question and several retrieved documents, only one of which held the answer. In key-value retrieval, it had to find one value in a long list of pairs. The researchers varied where the relevant item sat.

Their finding: performance “is often highest when relevant information occurs at the beginning or end of the input context, and significantly degrades when models must access relevant information in the middle of long contexts, even for explicitly long-context models.” Plotted by position, accuracy forms a U shape.

The practical reading is simple. A long context window tells you how much a model accepts, not how evenly it uses it.

Why it matters

  • RAG ordering. In retrieval-augmented generation, the retriever ranks passages. If you place the best one in the middle of 20, you waste the ranking. Put it first, or first and last.
  • Long documents. Asking about a clause on page 40 of 80 is harder than asking about page 1. Point the model to the section, or extract it first.
  • Agents. In a long agent history, early instructions and the latest turn get attention; mid-run tool output is the part most likely to be underused.

Lost in the middle is one piece of a wider effect called context rot, where accuracy drops with length and distractors, not only position.

On BroadNet Inference

BroadNet’s DeepSeek V4.1 Flash and GLM 5.3 accept up to 1,048,576 tokens per request. That makes position more important, not less, because there is more middle. A layout that works with both position effects and cached input:

[stable system prompt and instructions]  -> cached prefix
[documents, best-ranked first]
[the question, restated in full]         -> end of prompt

The 1M-token context guide and prompt structure for caching go further.

Questions

Do newer long-context models still get lost in the middle?

The original study found the drop even in models built for long contexts. Later work such as Chroma's 2025 context rot report still finds accuracy falling with length. Test your own model and prompt layout.

Where should I put the most important document in a prompt?

Near the start or the end, not buried in the middle. For RAG, put the top-ranked passage first and restate the question after the documents.

Does this conflict with prompt caching?

It can. Caching favours a stable prefix, while position effects favour the most relevant text near the edges. Keep stable instructions first, and place the question and key passages at the end.

Read as MarkdownAsk ChatGPTAsk ClaudeAsk Perplexity