BroadRouter
Get started

Agent token budget: where an agent loop's tokens go

By the BroadNet Inference team · 5 min read · Published

An agent loop resends its whole context on every call: system prompt, tool definitions, conversation history, file contents and tool output, plus the new step. Input usually outweighs output many times over. Log prompt_tokens, cached_tokens and completion_tokens per call, find the biggest repeating block, then trim tools, cap tool output and restart long sessions.

A chat costs what you type plus what the model writes. An agent costs far more, because it calls the model dozens of times and resends almost everything each time. If you want to cut that cost, or just stop hitting the context window, you need to know which part of each request is big. This post breaks an agent loop into five buckets, shows how to measure each from the API’s own usage data, and lists what to trim first.

Why does an agent resend everything?

Chat-completions APIs are stateless. The model remembers nothing between calls, so the agent sends the full context every step. A 40-step task where each call carries 30,000 tokens reads 1.2 million input tokens to produce maybe 40,000 output tokens. That 30-to-1 ratio is why input tokens and their cache price dominate agent bills.

Where do the tokens go?

Bucket What it is Repeats every call? Typical driver
System prompt Agent instructions, rules files, project notes Yes Long rules files
Tool schemas Name, description and JSON Schema of every tool Yes Built-in tools plus every MCP server
History Earlier messages, tool calls and results Yes, and grows Long sessions
File reads and tool output File contents, test logs, search results Yes, until dropped Reading whole files, verbose commands
Output Visible answer, tool calls and reasoning No Reasoning effort, rewriting whole files

The first two are fixed overhead. Cursor wrote in September 2026 that the system prompt and tool definitions “had become one of the largest sources of spend that we fully control”, and called file reads “another large source of token spend”. Its changes, including cutting about two-thirds of the system prompt and loading tools used in fewer than 20% of conversations only when needed, cut users’ token costs by 7%. GitHub’s VS Code team named the same two costs for Copilot: the repeated prompt prefix and tool-definition overhead.

How do you measure it?

You don’t need a tracing product. The usage block on each response is enough.

  1. Turn on usage. For streaming, add stream_options: {"include_usage": true}; usage arrives in the last chunk. See streaming.
  2. Log per call: step number, prompt_tokens, prompt_tokens_details.cached_tokens, completion_tokens and, when reported, completion_tokens_details.reasoning_tokens.
  3. Measure the fixed overhead once. Send one request with only the system prompt and tools, and the word “hi”. Its prompt_tokens is your floor.
  4. Measure a tool’s output. Count characters of each tool result before you append it. Dividing by four gives a rough English token count; paste a sample into the free token counter for the exact figure on your model’s tokenizer.
r = client.chat.completions.create(model=MODEL, messages=msgs, tools=tools)
u = r.usage
cached = getattr(u.prompt_tokens_details, "cached_tokens", 0) or 0
log.info("step=%d prompt=%d cached=%d out=%d", step, u.prompt_tokens, cached, u.completion_tokens)

Plot prompt_tokens by step. A steady slope means history is growing. A sudden jump means a large file or log entered the context. A high floor means too many tools.

In Aider, /tokens reports what’s in the current context, and /drop removes files from it.

What should you trim first?

Order by size times repetitions, not by what’s easiest:

  1. Unused tools. Every tool schema rides on every call. OpenAI’s guide says function definitions “are billed as input tokens” and suggests fewer than 20 functions at the start of a turn, as a soft rule. Disable MCP servers you don’t use in this task; opencode’s docs say plainly that MCP tools consume model context.
  2. Tool output. Cap command output (last 200 lines of a test log, not 20,000). Return search results as paths and line numbers, then read only what’s needed.
  3. File reads. Read the function, not the file. Aider’s repo map uses a default 1k-token budget (--map-tokens) instead of whole files.
  4. History. Start a new task per change. When a session must continue, replace old turns with a short summary.
  5. System prompt. Remove rules the model already follows. Keep the rest stable.
  6. Output. Ask for diffs, not whole files. Set reasoning_effort to low for routine steps. Reasoning tokens are billed as output.

Why keep the prefix stable?

Providers cache a repeated prefix and bill it at a lower rate. On BroadNet Inference, DeepSeek V4.1 Flash bills cached input at $0.0055 per 1M tokens and new input at $0.175. GLM 5.3 bills $0.173 against $0.78. A timestamp in the system prompt, or tools listed in a different order each call, breaks the cache from that point on. Put stable parts first, in a fixed order, and changing parts last. More in cached input tokens explained; the prompt caching calculator shows what a given hit rate saves.

Caching lowers the price of the prefix, but not its size. Cached tokens still fill the context window and still take time to process.

What does a worked agent budget look like?

Take the 40-step task above, with 80% of input cached:

Part Tokens Billed at
New input 240,000 Input price
Cached input 960,000 Cached input price
Output incl. reasoning 40,000 Output price

Cutting 5,000 tokens of unused tool schemas from each call removes 200,000 input tokens from this task, about a sixth of the total, with no change to the agent’s behaviour. For the cost levers beyond trimming, such as model choice, caching and flat plans, see 10 ways to cut coding agent LLM costs. For multi-agent setups, multiply by the number of agents; see what multi-agent setups cost. To model your own loop step by step, use the agent run cost estimator.

Where BroadNet Inference fits

BroadNet Inference returns prompt and completion tokens in the usage block, plus cached and reasoning tokens when the model reports them, so the measurement above works as written. Its low cached-input price on DeepSeek V4.1 Flash rewards a stable prefix. If trimming isn’t worth your time, the unlimited plans remove the token meter for DeepSeek V4.1 Flash at $60 a month per session; the context window still applies, and a lean context still runs faster. Plans are on inference.broadnet.ai/plans.

Questions

Why does my coding agent use so many input tokens?

Because the API is stateless. Each call resends the system prompt, every tool definition, the conversation so far and all file contents still in context. A long session multiplies that by the number of steps.

How do I see how many tokens each agent step uses?

Read the usage block of each response. With streaming, set stream_options include_usage to true and read the last chunk. Log prompt_tokens, prompt_tokens_details.cached_tokens and completion_tokens with a step number.

Do tool definitions count as input tokens?

Yes. The tools list is part of the prompt and is sent on every call. MCP servers add their tools to that list, which is why opencode's docs say to add only the servers you need.

Does prompt caching fix the problem?

It cuts the price of the repeated prefix, not its size. Cached input still fills the context window. On BroadNet, DeepSeek V4.1 Flash bills cached input at $0.0055 per 1M tokens against $0.175 for new input.

Are reasoning tokens part of the budget?

Yes, on the output side. Reasoning models count thinking toward max_tokens and bill it as output on pay-as-you-go. Set reasoning_effort to low for routine steps.

Read as MarkdownAsk ChatGPTAsk ClaudeAsk Perplexity