---
title: "Prompt structure for caching - stable prefix design | BroadRouter"
description: "How to lay out LLM prompts so the cached prefix stays identical: system prompt, tool schemas, documents, append-only history, plus a test that proves it."
url: https://broadrouter.com/blog/prompt-structure-for-caching/
lang: en
updated: 2026-11-14
publisher: BroadRouter (a BroadNet site); API by BroadNet Inference, https://inference.broadnet.ai
---

# How to structure prompts so the prefix caches

By the BroadNet Inference team · 4 min read · Published 2026-11-14 · Updated 2026-11-16

Order every request from most stable to least stable: system prompt, tool definitions, reference documents, conversation history, then the new message. Make each layer byte-identical between calls: no timestamps or IDs at the top, tools sorted, JSON serialised deterministically, and history only ever appended. Then measure cached\_tokens to confirm the prefix is actually reused.

## Key takeaways

-   Caches match an exact prefix. One changed byte invalidates everything after it, so the order of layers matters more than their size.
-   Four layers, most stable first: system prompt, tools, documents, history. The new user turn goes last.
-   Serialise tools and JSON documents with sorted keys and fixed separators; dictionary order is a silent cache breaker.
-   Restrict tools per call with tool\_choice instead of removing tool definitions.
-   Test it: send the same prefix twice and check that prompt\_tokens\_details.cached\_tokens rises on the second call.

**On this page**

1.  Why does prompt order decide the cache hit rate?
2.  What is the right layer order?
3.  What breaks the prefix without you noticing?
4.  How do you build a cache-friendly request in code?
5.  How do you prove the prefix is cached?
6.  How does this change for agents and long context?
7.  Where BroadNet Inference fits

[Cached input tokens](https://broadrouter.com/glossary/cached-input-tokens/) are cheap, and on BroadNet Inference DeepSeek V4.1 Flash bills them at $0.0055 per 1M instead of $0.175. The [cached tokens explainer](https://broadrouter.com/blog/cached-input-tokens-explained/) covers the price and how to read the usage field. This post is about the design work: how to lay out a prompt so that the cache actually hits, and how to prove it does.

## Why does prompt order decide the cache hit rate?

Every major provider caches by **prefix**. OpenAI’s guide says reuse requires the entire rendered prefix to match. DeepSeek’s docs describe the same rule: a request hits only if it fully matches a stored prefix, and one that diverges partway through does not. Anthropic builds its cache in the order tools, system, messages.

Under the hood the server reuses the [KV cache](https://broadrouter.com/glossary/kv-cache/) it built for that prefix. So the cache is a straight line from the first token. The first byte that differs from the last call ends the reusable part. A one-line timestamp at the top of a 50,000-token prompt costs you all 50,000 tokens at full price.

## What is the right layer order?

| Layer | Changes | Example | Rule |
| --- | --- | --- | --- |
| 1\. System prompt | Per deploy | Role, rules, output format | No dates, IDs, user names |
| 2\. Tool definitions | Per deploy | Function schemas | Sorted, deterministic JSON |
| 3\. Reference material | Per session or task | Repo files, contracts, product docs | Fixed order, stable labels |
| 4\. Conversation history | Per turn, append-only | Earlier user and assistant turns | Never edit; only add |
| 5\. New message | Every call | The question, plus the current time if needed | Volatile data goes here |

Layer 5 is where anything volatile belongs: the current date, the user’s locale, a request ID, retrieved snippets that differ per question.

## What breaks the prefix without you noticing?

1.  **Dictionary order.** Frameworks often build the `tools` array from a dict or registry. If plugins load in a different order, the bytes change.
2.  **JSON formatting.** `json.dumps(x)` and `json.dumps(x, indent=2)` are different prefixes. So are two serialisers with different key orders.
3.  **Template whitespace.** A trailing space or a `\r\n` from one code path and `\n` from another.
4.  **Dynamic tool lists.** Removing tools a user cannot use changes the prefix. OpenAI’s guide recommends keeping definitions stable and restricting use with `tool_choice` or an allowed-tools setting.
5.  **Summarising history in place.** Rewriting old turns into a summary resets everything after the system prompt. Summarise rarely and in large steps.
6.  **Retrieval before instructions.** RAG snippets placed between the system prompt and the tools differ per query and break the tools layer too.

## How do you build a cache-friendly request in code?

Make each layer a function of stable inputs only, and serialise deterministically:

```python
import json, os
from openai import OpenAI

client = OpenAI(base_url="https://inference.broadnet.ai/v1", api_key=os.environ["BROADNET_API_KEY"])

SYSTEM = open("prompts/system.md").read()  # versioned file, no templating

def stable_tools(registry: dict) -> list:
    # Sort by name, sort keys inside each schema.
    return [json.loads(json.dumps(registry[name], sort_keys=True)) for name in sorted(registry)]

def docs_block(paths: list[str]) -> str:
    parts = [f"=== {p} ===\n{open(p).read()}" for p in sorted(paths)]
    return "\n\n".join(parts)

def build_messages(history: list, user_text: str, now_iso: str, doc_paths: list[str]):
    return (
        [{"role": "system", "content": SYSTEM},
         {"role": "user", "content": docs_block(doc_paths)}]
        + history                                   # append-only
        + [{"role": "user", "content": f"(now: {now_iso})\n{user_text}"}]
    )

resp = client.chat.completions.create(
    model="broadnet/deepseek-v4.1-flash",
    messages=build_messages(history, question, now_iso, files),
    tools=stable_tools(TOOLS),
)
history += [{"role": "user", "content": question}, resp.choices[0].message.model_dump(exclude_none=True)]
```

Two details. The current time goes in the last message, not the system prompt. And the history stores the assistant message exactly as returned, so the next call resends identical bytes.

## How do you prove the prefix is cached?

Run a two-call test in CI or a notebook:

1.  Build a request with your real system prompt, tools and documents.
2.  Send it. Note `usage.prompt_tokens`.
3.  Change only the final user message and send again.
4.  Read `usage.prompt_tokens_details.cached_tokens` on the second call. It should cover nearly all of the shared prefix.
5.  If it is zero or small, diff the serialised request bodies byte by byte. The first difference is your cache breaker.

Caching is best-effort on most providers, and DeepSeek’s docs say so directly, so a single miss is not proof of a bug. A consistent miss is. Track the ratio of cached to prompt tokens per request in your [usage logs](https://broadrouter.com/blog/llm-api-usage-logging/), and use the [prompt caching calculator](https://broadrouter.com/tools/prompt-caching-calculator/) to see what a higher hit rate saves.

## How does this change for agents and long context?

Agents benefit most because they resend the same large prefix dozens of times per task. Two habits matter:

-   **One conversation per task.** Starting a fresh conversation for each sub-step throws the prefix away.
-   **Read files once, near the top.** If the agent re-reads a file, append the new copy rather than replacing the old one, unless the file changed.

With a [1M-token context window](https://broadrouter.com/blog/million-token-context-window-guide/), layer 3 can hold a whole repository. That is affordable on a per-token key only when it caches, which is why the layer order matters more as prompts grow. Cached prefixes also tend to reduce time to first token; see [TTFT vs throughput](https://broadrouter.com/blog/llm-latency-ttft-throughput/).

## Where BroadNet Inference fits

On BroadNet Inference the usage block reports `prompt_tokens_details.cached_tokens` when the model reports it, and cached input on [DeepSeek V4.1 Flash](https://broadrouter.com/models/deepseek-v4.1-flash/), [GLM 5.3](https://broadrouter.com/models/glm-5.3/) and [Qwen3.8 27B](https://broadrouter.com/models/qwen3.8-27b/) is billed at the lower rate listed on the [pricing page](https://broadrouter.com/pricing/). BroadNet does not offer explicit cache breakpoints or a cache key parameter in its docs, so the layout above is the whole technique. On an unlimited plan the cache does not change the bill, because sessions have no token meter, but the same structure still helps latency.

Free tools for this

-   [OpenAI to BroadNet migration helper](https://broadrouter.com/tools/openai-to-broadnet-migration/) Paste OpenAI SDK code; get base URL, key and model edits, a diff and the checks that remain.
-   [max\_tokens calculator for reasoning models](https://broadrouter.com/tools/reasoning-token-budget/) Answer length + reasoning budget + margin = a max\_tokens that does not cut the answer off.
-   [JSON schema generator for LLM structured output](https://broadrouter.com/tools/json-schema-generator/) Sample JSON to strict schema, Pydantic, Zod and a ready request.
-   [LLM JSON output validator](https://broadrouter.com/tools/llm-json-output-validator/) Parse errors by line and column, safe repairs, cut-off detection and JSON Schema checks for model output.

[All free tools](https://broadrouter.com/tools/)

## Terms in this article

[Token](https://broadrouter.com/glossary/token/)

A token is the unit of text a language model reads and writes: a word, part of a word, a punctuation mark or a byte sequence produced by the model's tokenizer. LLM APIs count, limit and usually bill usage in tokens.

[Tool calling](https://broadrouter.com/glossary/tool-calling/)

Tool calling, also called function calling, is an LLM API feature in which the model returns a structured request to run a function the developer has defined, with JSON arguments. The application runs the function and sends the result back; the model never runs code itself.

[System prompt](https://broadrouter.com/glossary/system-prompt/)

A system prompt is the instruction text an application sends to a language model ahead of the conversation, usually as a message with the system or developer role. It sets the model's role, rules, tone and output format, and the model is meant to follow it over conflicting user messages.

[Cached input tokens](https://broadrouter.com/glossary/cached-input-tokens/)

Cached input tokens are prompt tokens that an LLM provider has recently processed and can reuse from a cache instead of computing again, usually a repeated prefix such as a system prompt or document. Providers bill them at a lower rate than new input tokens.

[Input and output tokens](https://broadrouter.com/glossary/input-and-output-tokens/)

Input tokens are the tokens a language model reads in a request, including the system prompt, conversation history, tool definitions and images. Output tokens are the tokens it generates in reply, including any reasoning. LLM APIs usually price them separately, and output usually costs more.

[Context window](https://broadrouter.com/glossary/context-window/)

A context window is the maximum number of tokens a language model can work with in a single request. It covers the prompt, conversation history, tool definitions, images and the generated output, including any reasoning tokens. Anything that does not fit must be cut, summarised or split across several requests.

[KV cache](https://broadrouter.com/glossary/kv-cache/)

A KV cache is the memory where a transformer language model stores the attention keys and values computed for every token it has already processed, so each new token reuses them instead of recomputing the whole sequence. It grows with context length and often limits how many requests a GPU can serve.

[Latency](https://broadrouter.com/glossary/latency/)

Latency, for a language model API, is the time a request takes from being sent to the final output token being received. It is the sum of time to first token and generation time, and is usually reported as percentiles such as p50 and p95 rather than as an average.

## Questions

### Why is my prompt cache hit rate low?

Something near the start of the prompt changes on every call. The usual causes are a timestamp or request ID in the system prompt, tools serialised in a different order, retrieved documents placed before the instructions, or history that gets summarised and rewritten.

### Should the user question go before or after long documents?

After. Put the documents first and the question last. The documents then form a reusable prefix, and many models also answer better when the question follows the material.

### Do tool definitions break prompt caching?

Only if they change. Tools are part of the prefix, so keep the same set in the same order with the same JSON. To limit tools for one call, use tool\_choice rather than editing the list.

### Does few-shot example order matter for caching?

Yes. Shuffled examples produce a different prefix each time. Fix the order, or place examples that change per request after the stable part.

## Keep reading

-   [Cached input tokens: how to cut an LLM API bill](https://broadrouter.com/blog/cached-input-tokens-explained/)
-   [When a 1M-token context window is worth it (and when not)](https://broadrouter.com/blog/million-token-context-window-guide/)
-   [Flat-rate vs pay-per-token LLM API: the break-even math](https://broadrouter.com/blog/flat-rate-vs-pay-per-token-llm-api/)
-   [A private endpoint starts with a clear workload](https://broadrouter.com/blog/private-endpoint-planning/)
-   [Before fine-tuning: define the problem to solve](https://broadrouter.com/blog/fine-tuning-readiness/)
-   [Guide: Stream chat completions and handle dropped connections](https://broadrouter.com/guides/streaming-chat-completions/)
-   [Guide: Use the OpenAI Python SDK with BroadNet Inference](https://broadrouter.com/guides/openai-python-sdk/)

Try it on your real workload. New accounts start with $5 of credit for Gemma and transcription. DeepSeek, GLM and Qwen3.8 need a top-up.

[Get started](https://inference.broadnet.ai/account/?utm_source=broadrouter&utm_medium=referral&utm_campaign=blog-prompt-structure-for-caching&utm_content=page-cta) ·

**Sources**

-   [BroadNet Inference API reference, cached tokens in usage (checked 2026-11-16)](https://inference.broadnet.ai/docs?utm_source=broadrouter&utm_medium=referral&utm_campaign=blog-prompt-structure-for-caching&utm_content=sources)
-   [OpenAI prompt caching guide, prefix matching and structuring advice (fetched 2026-11-14)](https://developers.openai.com/api/docs/guides/prompt-caching)
-   [Anthropic prompt caching docs, cache order tools then system then messages (checked 2026-11-16)](https://platform.claude.com/docs/en/build-with-claude/prompt-caching)
-   [DeepSeek API docs, context caching (fetched 2026-11-14)](https://api-docs.deepseek.com/guides/kv_cache)
