A coding agent that works for an afternoon builds up a lot of history: files read, commands run, test output, plans, corrections. Every turn sends all of it back to the model. How you manage that growing context decides three things at once: what each turn costs, how fast it answers, and whether the agent still remembers what it was doing.
A 1M-token window changes the trade-offs. It does not make them go away.
What does a 1M-token window buy a long-running agent?
On BroadNet Inference, DeepSeek V4.1 Flash and GLM 5.3 accept up to 1,048,576 tokens per request, prompt and answer together. Qwen3.8 27B accepts 262,144. With smaller windows, an agent has to summarise after an hour or two of work. With a million tokens, it can usually keep a full session’s raw history and only compact near the end of a very long run.
That is a real advantage: summaries are lossy, and every summary is a chance to forget the one detail that mattered. But three costs still grow with the prompt:
- Money, on a per-token key, because every turn resends the whole prompt.
- Latency, because the server must process every input token before the first output token.
- Attention. Quality can drop as inputs grow, an effect called context rot. The paper “Lost in the Middle” found that models tend to use information at the start and end of a long input more reliably than information in the middle. Newer models do better, but the habit of keeping important facts near the edges still pays.
Rule 1: why should history be append-only?
Prompt caching matches from the start of the prompt. If the first 300,000 tokens of this turn are identical to the last turn, they can be served from cache; the first changed token ends the match. BroadNet’s API reports this in usage.prompt_tokens_details.cached_tokens when the model reports it, and bills cached input at a lower rate: $0.0055 per 1M tokens on DeepSeek V4.1 Flash against $0.175 for new input.
So structure every request in the same order and only ever add to the end:
[ system prompt ] never changes during a session
[ tool definitions ] fixed order, never changes
[ project notes ] changes rarely, and only by appending
[ turn 1 ... turn N ] grows at the end
[ new message ]
The usual prefix breakers are a timestamp in the system prompt, tool definitions in unstable order, and re-rendering old tool results. For agents, add one more: “sliding window” trimming that drops the oldest turn every time, which shifts everything after it. Each of those turns a cached prompt into a new one; prompt structure for caching lists the rest, with a test that proves the prefix is cached. On a long session the difference is large: most of the prompt can be cached input instead of new input. How cached input is billed.
Rule 2: how do you measure the cache rate?
Log it per request. With the OpenAI SDK, three lines after the call do it:
import os
from openai import OpenAI
client = OpenAI(base_url="https://inference.broadnet.ai/v1", api_key=os.environ["BROADNET_API_KEY"])
resp = client.chat.completions.create(model="broadnet/deepseek-v4.1-flash", messages=history, max_tokens=8000)
u = resp.usage
cached = (u.prompt_tokens_details.cached_tokens if u.prompt_tokens_details else 0) or 0
print(f"turn prompt={u.prompt_tokens:,} cached={cached:,} ({cached / max(1, u.prompt_tokens):.0%})")
A healthy append-only agent shows a high cached share on every turn after the first. A sudden drop to near zero means something early in the prompt changed. Find it and pin it. The prompt caching calculator turns a cache rate into money per session.
Rule 3: what should you cut first when context grows?
When context does need to shrink, what you drop matters. In most agent transcripts the bulk is tool output: a 4,000-line test log, a whole file read to find one function, a directory listing. The valuable part is small: the plan, the decisions and why, the errors that were fixed.
Order of what to cut, first to last:
- Stale tool output. Replace the body of old tool results with a one-line stub: “ran pytest: 3 failures in test_api.py (fixed in turn 14)”.
- Superseded file contents. If the agent read
api.py, edited it, then read it again, the first read is dead weight. - Exploration that went nowhere. Searches and reads that did not lead to a change.
- Reasoning and plans. Keep these longest. They explain why the code looks the way it does.
Truncating tool output at the moment it is produced is even better. Cap each tool result (for example, the last 200 lines of a log plus a line count) and let the agent ask for more if it needs it. That keeps history append-only and small.
Rule 4: when should an agent compact its history?
Compaction (summarising older history into a shorter block) is the heavy tool. It rewrites the prefix, so the turn after a compaction pays full input price for everything after the system prompt. Compacting a little every few turns means paying that price over and over.
Better:
- Pick a high threshold. With a 1M window, compacting at, say, 60–70% full leaves room for the summary turn and a long tail of new work.
- Keep the recent tail verbatim. Summarise the old part and keep the last stretch of turns intact. opencode’s compaction docs describe this: it summarises the conversation and keeps the most recent part, about 15,000 tokens by default.
- Summarise with the full prompt. Sending the whole conversation to the summary request lets it reuse the cache from the previous turn.
- Compact once, then append again. After a compaction the new prefix is stable, and caching resumes from there.
Rule 5: what belongs in a notes file?
Some facts must survive any number of compactions: the goal, the constraints, the decisions taken, the commands that build and test the project. Keep them in a short notes file (a NOTES.md or a task file) that the agent appends to and rereads. It sits near the top of the prompt, it changes rarely, and it is the first thing a fresh session reads if the run restarts.
This is also the cheapest form of memory: a few thousand tokens that save the agent from rediscovering the same facts after every summary.
How do reasoning tokens affect the budget?
DeepSeek V4.1 Flash, GLM 5.3 and Qwen3.8 27B reason before they answer, and the reasoning counts toward max_tokens. A long session does not change that, but large prompts tempt people to lower max_tokens to “save room”. Do not. A tight output budget produces empty answers, which the agent then retries. Why answers come back empty.
Remember too that the window counts prompt and answer together. Near 1,048,576 tokens of prompt, there is no room left to answer. Compact before you get there; the context window calculator shows how much room a given prompt leaves.
Does this matter on a flat plan?
On a BroadNet plan session, DeepSeek V4.1 Flash has no token meter: a Full Moon session is $60 a month however long the prompts are. The money argument for caching disappears. The other two do not. A cached prefix is typically faster to process in many serving systems, and a well-managed context gives better answers. The habits are the same; only the reason changes. How plan sessions work.
Context management checklist
- System prompt and tool list fixed for the whole session.
- History only appended; old turns never reformatted.
- Cache rate logged per request.
- Tool output capped when produced; stale output stubbed later.
- Compaction at a high threshold, keeping the recent tail verbatim.
- A notes file with goal, constraints and decisions.
- Output budget large enough for reasoning.
For more on when a million-token window is worth using at all, see the 1M context window guide. For stopping sessions that grow without making progress, see agent loop cost control.
