BroadRouter

Cache LLM API responses to cut cost: an exact-match cache in Python

By the BroadNet Inference team · 8 min read · Published

Hash the full request (model, messages, temperature, max_tokens, response_format, tools, reasoning settings and your prompt version) into a key, and store the answer under it with an expiry. Serve a stored answer only for deterministic, side-effect-free requests. Measure how often identical requests repeat in your logs first: that share is the most an exact cache can save.

Every identical LLM request you send twice is paid for twice. A response cache stores the answer the first time and returns it the next time, with no tokens billed and no wait. The idea is simple. The hard parts are deciding what makes two requests “identical”, which requests must never be served from storage, and whether your traffic repeats enough to matter.

This post builds a small exact-match cache in Python on SQLite, tested against a mock OpenAI-compatible server, and shows how to measure the hit rate from your logs before you write any of it.

Is response caching the same as prompt caching?

No. They save different things, and you can use both.

Prompt caching (provider) Response caching (you)
What is reused The repeated start of the prompt The whole answer
Who runs it The API provider Your code
Model still runs? Yes, writes a fresh answer No
What it saves Part of the input cost Input, output and reasoning tokens, plus latency
When it hits Shared prefix, different questions Exactly the same request

Cached input tokens explains the provider side, and prompt structure for caching shows how to lay out a prompt so the prefix is reused. The prompt caching calculator prices it. A response cache sits in front of all that: on a hit, no request leaves your server.

Will a cache actually save you money?

Measure before you build. The cost-saving percentages in many caching tutorials come from someone else’s traffic. Your number depends on one thing: how often the same request arrives again within your expiry window.

If you already log each call, as in LLM API usage logging, add one field: request_hash, the cache key described below. It stores no prompt text, so it fits a log that keeps content out. Then count repeats:

import json
from collections import Counter

def potential_hit_rate(log_path: str) -> float:
    """Share of calls whose request_hash was already seen: an upper bound for an exact cache."""
    hashes = [json.loads(line)["request_hash"] for line in open(log_path)]
    seen, repeats = set(), 0
    for h in hashes:
        repeats += h in seen
        seen.add(h)
    print("top repeated requests:", Counter(hashes).most_common(3))
    return repeats / len(hashes) if hashes else 0.0

print(f"potential hit rate: {potential_hit_rate('calls.jsonl'):.1%}")

On a synthetic log of 20 calls, one request five times, another three times and twelve unique, it printed a potential hit rate of 30.0%. That is an upper bound: it ignores expiry, and it assumes every repeat is safe to serve from storage.

Then turn the rate into money:

monthly saving = hit rate × requests per month
                 × (input tokens × input price + output tokens × output price) ÷ 1,000,000

Output tokens include reasoning. BroadNet’s docs say DeepSeek V4.1 Flash, GLM 5.3 and Qwen3.8 27B reason before answering and that reasoning counts toward max_tokens, so a hit saves tokens you never see in the reply. Take completion_tokens from the logged usage, not the length of the answer.

A worked example in tokens: 100,000 requests a month at 1,200 input and 150 output tokens each, with a 30% hit rate, avoid 36 million input tokens and 4.5 million output tokens. On DeepSeek V4.1 Flash those are billed at $0.175 and $0.52 per 1M respectively; plug your own figures into the LLM API cost calculator. If most of that input was already a provider cache hit, the input saving is at the lower cached rate ($0.0055 per 1M on the same model), so the output side is often what matters.

What goes in the cache key?

Everything that can change the answer, and nothing else.

  • Include: model, the full messages list (system prompt included), temperature, top_p, max_tokens, seed, response_format, tools, tool_choice and reasoning_effort.
  • Add a prompt version you control, such as faq-v3. Bump it when you change prompt templates, retrieval data or post-processing that the request body does not show.
  • Leave out anything that varies per call without changing the answer: request IDs, timestamps, trace headers, a user field you only send for abuse tracking.

Serialise the key material the same way every time. Python’s json.dumps with sort_keys=True and separators=(",", ":") gives one canonical string regardless of dictionary order; ensure_ascii=False keeps Arabic and other non-Latin text as UTF-8 instead of escapes. Hash it with SHA-256.

A missing field is the dangerous mistake. If max_tokens is not in the key, a reply cut short under a small limit will be served to a request that allowed a long one.

The cache: SQLite, TTL and a skip policy

import hashlib, json, sqlite3, time

# Every field that changes the answer belongs in the key.
KEY_FIELDS = ("model", "messages", "temperature", "top_p", "max_tokens", "seed",
              "response_format", "tools", "tool_choice", "reasoning_effort")

def cache_key(params: dict, prompt_version: str) -> str:
    material = {k: params[k] for k in KEY_FIELDS if k in params}
    material["prompt_version"] = prompt_version
    canon = json.dumps(material, sort_keys=True, separators=(",", ":"), ensure_ascii=False)
    return hashlib.sha256(canon.encode("utf-8")).hexdigest()

class ResponseCache:
    def __init__(self, path="llm_cache.sqlite", ttl_s=7 * 86400):
        self.db, self.ttl = sqlite3.connect(path), ttl_s
        self.db.execute("CREATE TABLE IF NOT EXISTS cache (k TEXT PRIMARY KEY, created REAL,"
                        " body TEXT, prompt_tokens INT, completion_tokens INT)")
        self.hits = self.misses = self.skips = self.saved_in = self.saved_out = 0

    def create(self, client, prompt_version: str, **params):
        if params.get("stream") or params.get("temperature", 1) > 0:
            self.skips += 1                       # not cacheable: call the model directly
            return client.chat.completions.create(**params)
        k = cache_key(params, prompt_version)
        row = self.db.execute("SELECT body, created, prompt_tokens, completion_tokens"
                              " FROM cache WHERE k = ?", (k,)).fetchone()
        if row and time.time() - row[1] < self.ttl:
            self.hits += 1; self.saved_in += row[2]; self.saved_out += row[3]
            return json.loads(row[0])             # a plain dict, not an SDK object
        self.misses += 1
        r = client.chat.completions.create(**params)
        choice = r.choices[0]
        if choice.finish_reason == "stop" and not choice.message.tool_calls:
            self.db.execute("INSERT OR REPLACE INTO cache VALUES (?, ?, ?, ?, ?)",
                            (k, time.time(), r.model_dump_json(),
                             r.usage.prompt_tokens, r.usage.completion_tokens))
            self.db.commit()
        return r.model_dump()

And a run against it:

import os
from openai import OpenAI

client = OpenAI(base_url="https://inference.broadnet.ai/v1", api_key=os.environ["BROADNET_API_KEY"])
cache = ResponseCache("demo.sqlite", ttl_s=86400)
SYSTEM = {"role": "system", "content": "Answer questions about our refund policy in one sentence."}
questions = ["How long do refunds take?", "Can I refund a gift card?",
             "How long do refunds take?", "How long do refunds take?", "Can I refund a gift card?"]
for q in questions:
    r = cache.create(client, "faq-v3", model="broadnet/deepseek-v4.1-flash", temperature=0,
                     max_tokens=1200, messages=[SYSTEM, {"role": "user", "content": q}])
    print(r["choices"][0]["message"]["content"])
# a personalised, sampled request is never cached
cache.create(client, "faq-v3", model="broadnet/deepseek-v4.1-flash", temperature=0.7,
             messages=[SYSTEM, {"role": "user", "content": "Write me a friendly reminder"}])
# a new prompt version is a new key
cache.create(client, "faq-v4", model="broadnet/deepseek-v4.1-flash", temperature=0,
             max_tokens=1200, messages=[SYSTEM, {"role": "user", "content": "How long do refunds take?"}])
print(f"hits={cache.hits} misses={cache.misses} skipped={cache.skips} "
      f"saved_input={cache.saved_in} saved_output={cache.saved_out}")

We ran this with the OpenAI Python SDK 3.51.0 against a local mock server that counted upstream calls and reported 1,200 prompt and 150 completion tokens per reply (no live API call). Seven requests made four upstream calls. The script printed hits=3 misses=3 skipped=1 saved_input=3600 saved_output=450. Run again on the same database file, all six cacheable requests were hits (hits=6 misses=0): SQLite keeps the cache across restarts. Keys built from the same fields in a different order matched.

Notes on the code:

  • max_tokens=1200 leaves room for reasoning before the answer. With a small limit a reasoning model can return an empty reply (why). Those replies are not stored, because finish_reason is length, not stop.
  • Hits return a dict. Misses also return model_dump(), so callers see one shape. If your code expects SDK objects, wrap both.
  • A missing temperature counts as sampled. The server then applies its own default, which may be above 0, so the wrapper skips caching unless you set 0 explicitly.
  • SQLite suits one process or a few. For several servers, put the same key and TTL logic in Redis or your database.

What should never be served from a cache?

  • Sampled replies (temperature above 0). You chose variety. Caching removes it, and a user who presses “regenerate” gets the same text.
  • Personalised answers unless the user ID is in the key. Two users sending “summarise my account” have different accounts. Better: cache only requests whose messages contain no per-user data.
  • Tool calls. A tool call is an instruction to act. Replaying “refund order 1182” from storage can repeat a side effect, and the right next step depends on current state. The wrapper refuses to store any reply with tool_calls.
  • Streamed replies. The wrapper bypasses the cache for stream=True. Caching a stream means buffering it, checking it finished, and replaying it, which is rarely worth it for chat UIs.
  • Truncated or failed replies. Store only finish_reason == "stop". A cut-off answer or an error cached for a week is a bug that looks like a model problem.
  • Answers that depend on time. “What is today’s exchange rate?” needs a short TTL or no cache.

What does cache well: classification of a fixed set of inputs, extraction from documents you reprocess, FAQ answers from a fixed prompt, test suites and evals you rerun, and development loops where you send the same request again and again.

When should cached answers expire?

Use three kinds of invalidation:

  1. TTL. A time limit per entry: days for stable FAQ answers, minutes for anything that reads live data.
  2. Version in the key. Change the prompt version when the system prompt, retrieval index or output schema changes. Old entries stop matching and age out.
  3. Model in the key. It is already there, so moving from one model to another starts a fresh cache, which is what you want while you evaluate a switch.

Delete expired rows on a schedule (DELETE FROM cache WHERE created < ?). Otherwise the file only grows.

There is a side benefit. Serving engines do not promise identical output for identical input: vLLM’s documentation says it “does not guarantee the reproducibility of the results by default”. A cache gives every user the same answer to the same question for the life of the entry.

What about semantic caching?

A semantic cache embeds each question and returns a stored answer when a new question is close enough, so “how long is a refund?” can hit “how long do refunds take?”. GPTCache is the best-known open-source example. Its README says it uses a vector store for similarity search and a similarity evaluator to decide a hit, warns that “you may encounter false positives during cache hits and false negatives during cache misses”, and notes that disabling embeddings turns it into a keyword-matching cache.

Two practical points. First, it needs an embeddings model and a vector store. BroadNet Inference does not offer embeddings, so you would use another provider or a local model for that layer; the embedding cost calculator prices the options. Second, a false hit returns a confident answer to a different question. Use a strict similarity threshold, test it on real near-miss pairs, and keep semantic caching for low-stakes answers. Put the exact cache in front either way: it is free to check and never wrong about what was asked.

What does a cache mean for privacy?

A cache is a store of prompts and answers. That changes where your data lives.

BroadNet’s inference privacy statement, checked 2026-11-15, says its gateway keeps no prompts, messages, responses or transcripts, while content-free traffic and usage records (token counts, cost, timings) are kept. Inference content is not retained. Necessary account, billing and security metadata is retained as described in that statement. Once you add a response cache, your system holds the content the provider does not. Treat it like any other store of user data:

  • Cache only requests without personal data where you can, which also raises the hit rate.
  • Set a TTL short enough for your retention policy, and delete on schedule.
  • Encrypt the file or table at rest and limit who can read it.
  • Include the cache in data-deletion requests if it can hold user text.

Sending sensitive data to an LLM API covers the rest of that checklist.

How does caching fit with other cost controls?

Caching removes repeated calls. It does nothing for unique ones, so it sits next to the other controls rather than replacing them: shorter prompts, provider prompt caching, a max_tokens cap per request, and budgets and loop limits from LLM API spend control. For large one-off jobs, deduplicate inputs before you send them; that is the same idea as a cache, done once. Processing a large job without a Batch API shows the job side.

On a flat-rate plan the maths changes. BroadNet’s DeepSeek sessions have no token meter, so a cache hit saves no money there. It still saves time: a session runs one request at a time, so every repeat you skip frees the session for new work.

Where BroadNet Inference fits

BroadRouter is BroadNet’s own site. BroadNet Inference exposes DeepSeek V4.1 Flash, GLM 5.3, Qwen3.8 27B and Gemma 4 VL 26B behind an OpenAI-compatible /v1/chat/completions endpoint, so the wrapper above works with the OpenAI Python SDK by changing the base URL. It has no response-caching feature of its own; the cache in this post is yours. Provider-side cache hits are reported in usage.prompt_tokens_details.cached_tokens and billed at the cached input price, per its API reference. Prices per model are on the pricing page.

Questions

What is the difference between prompt caching and response caching?

Prompt caching is done by the provider: a repeated prompt prefix is billed at a lower cached-input price, and the model still writes a new answer. Response caching is done by you: you store the whole answer and return it for an identical request without calling the model at all.

Should I cache LLM responses at temperature 0?

Yes, if the request has no side effects and the same answer is acceptable for everyone who sends it. Temperature 0 does not guarantee identical output from a serving engine anyway, so a cache also makes the answer consistent. Skip caching when temperature is above 0 or omitted.

How much does LLM response caching save?

The share of requests that are exact repeats within your expiry window, times the cost of those requests. Count repeated request hashes in your own logs to get that share. It can be near zero for open-ended chat and high for fixed FAQ or classification inputs.

Is semantic caching better than exact-match caching?

It hits more often, because similar questions match, but it can return an answer to a different question. It also needs an embeddings model and a vector store. Start with an exact cache, and add a semantic layer only where a wrong hit is cheap.

Can I cache streamed responses?

Not with the simple wrapper here. You would have to collect the full stream, check it finished normally, store it, and replay it as a stream later. Most teams cache only non-streamed calls such as classification, extraction and FAQ answers.

Read as MarkdownAsk ChatGPTAsk ClaudeAsk Perplexity