Cached input tokens are cheap, and on BroadNet Inference DeepSeek V4.1 Flash bills them at $0.0055 per 1M instead of $0.175. The cached tokens explainer covers the price and how to read the usage field. This post is about the design work: how to lay out a prompt so that the cache actually hits, and how to prove it does.
Why does prompt order decide the cache hit rate?
Every major provider caches by prefix. OpenAI’s guide says reuse requires the entire rendered prefix to match. DeepSeek’s docs describe the same rule: a request hits only if it fully matches a stored prefix, and one that diverges partway through does not. Anthropic builds its cache in the order tools, system, messages.
Under the hood the server reuses the KV cache it built for that prefix. So the cache is a straight line from the first token. The first byte that differs from the last call ends the reusable part. A one-line timestamp at the top of a 50,000-token prompt costs you all 50,000 tokens at full price.
What is the right layer order?
| Layer | Changes | Example | Rule |
|---|---|---|---|
| 1. System prompt | Per deploy | Role, rules, output format | No dates, IDs, user names |
| 2. Tool definitions | Per deploy | Function schemas | Sorted, deterministic JSON |
| 3. Reference material | Per session or task | Repo files, contracts, product docs | Fixed order, stable labels |
| 4. Conversation history | Per turn, append-only | Earlier user and assistant turns | Never edit; only add |
| 5. New message | Every call | The question, plus the current time if needed | Volatile data goes here |
Layer 5 is where anything volatile belongs: the current date, the user’s locale, a request ID, retrieved snippets that differ per question.
What breaks the prefix without you noticing?
- Dictionary order. Frameworks often build the
toolsarray from a dict or registry. If plugins load in a different order, the bytes change. - JSON formatting.
json.dumps(x)andjson.dumps(x, indent=2)are different prefixes. So are two serialisers with different key orders. - Template whitespace. A trailing space or a
\r\nfrom one code path and\nfrom another. - Dynamic tool lists. Removing tools a user cannot use changes the prefix. OpenAI’s guide recommends keeping definitions stable and restricting use with
tool_choiceor an allowed-tools setting. - Summarising history in place. Rewriting old turns into a summary resets everything after the system prompt. Summarise rarely and in large steps.
- Retrieval before instructions. RAG snippets placed between the system prompt and the tools differ per query and break the tools layer too.
How do you build a cache-friendly request in code?
Make each layer a function of stable inputs only, and serialise deterministically:
import json, os
from openai import OpenAI
client = OpenAI(base_url="https://inference.broadnet.ai/v1", api_key=os.environ["BROADNET_API_KEY"])
SYSTEM = open("prompts/system.md").read() # versioned file, no templating
def stable_tools(registry: dict) -> list:
# Sort by name, sort keys inside each schema.
return [json.loads(json.dumps(registry[name], sort_keys=True)) for name in sorted(registry)]
def docs_block(paths: list[str]) -> str:
parts = [f"=== {p} ===\n{open(p).read()}" for p in sorted(paths)]
return "\n\n".join(parts)
def build_messages(history: list, user_text: str, now_iso: str, doc_paths: list[str]):
return (
[{"role": "system", "content": SYSTEM},
{"role": "user", "content": docs_block(doc_paths)}]
+ history # append-only
+ [{"role": "user", "content": f"(now: {now_iso})\n{user_text}"}]
)
resp = client.chat.completions.create(
model="broadnet/deepseek-v4.1-flash",
messages=build_messages(history, question, now_iso, files),
tools=stable_tools(TOOLS),
)
history += [{"role": "user", "content": question}, resp.choices[0].message.model_dump(exclude_none=True)]
Two details. The current time goes in the last message, not the system prompt. And the history stores the assistant message exactly as returned, so the next call resends identical bytes.
How do you prove the prefix is cached?
Run a two-call test in CI or a notebook:
- Build a request with your real system prompt, tools and documents.
- Send it. Note
usage.prompt_tokens. - Change only the final user message and send again.
- Read
usage.prompt_tokens_details.cached_tokenson the second call. It should cover nearly all of the shared prefix. - If it is zero or small, diff the serialised request bodies byte by byte. The first difference is your cache breaker.
Caching is best-effort on most providers, and DeepSeek’s docs say so directly, so a single miss is not proof of a bug. A consistent miss is. Track the ratio of cached to prompt tokens per request in your usage logs, and use the prompt caching calculator to see what a higher hit rate saves.
How does this change for agents and long context?
Agents benefit most because they resend the same large prefix dozens of times per task. Two habits matter:
- One conversation per task. Starting a fresh conversation for each sub-step throws the prefix away.
- Read files once, near the top. If the agent re-reads a file, append the new copy rather than replacing the old one, unless the file changed.
With a 1M-token context window, layer 3 can hold a whole repository. That is affordable on a per-token key only when it caches, which is why the layer order matters more as prompts grow. Cached prefixes also tend to reduce time to first token; see TTFT vs throughput.
Where BroadNet Inference fits
On BroadNet Inference the usage block reports prompt_tokens_details.cached_tokens when the model reports it, and cached input on DeepSeek V4.1 Flash, GLM 5.3 and Qwen3.8 27B is billed at the lower rate listed on the pricing page. BroadNet does not offer explicit cache breakpoints or a cache key parameter in its docs, so the layout above is the whole technique. On an unlimited plan the cache does not change the bill, because sessions have no token meter, but the same structure still helps latency.
