The context audit I run on every agent

One of the agents I’ve built, running in production, read 11,697 tokens before anyone typed a word: the system prompt, every tool’s definition, and the pages the conversation sat beside. Most turns needed a fraction of it.

The natural instinct is to shrink that start, and it is worth doing. But with prompt caching, the start of a run is mostly a cheap cache read. You pay full price for it only on a cold start. The expensive context is what piles up while the agent works: tool results that are sent again on every step after they arrive.

So the audit has two halves, one for each place context is wasted, and one rule that covers both.

How to run the audit

Open the transcript of one real conversation, ideally a long one, and count. Not from memory and not from the prompt file, but from what was actually sent to the model on each step. Everything below follows from that count.

The start of the run

1. Count it properly

Count section by section with your provider’s token-counting endpoint. The common “four characters per token” estimate came out 30% low on this agent’s text, which is enough to send you after the wrong section.

The count tells you where to aim. Here, tool descriptions were only 994 of the 8,517 tool tokens. The rest were input schemas the model needs to make a correct first call, so the savings had to come from somewhere else.

2. Ask “when is this needed?” of every prompt paragraph

Go through the prompt one paragraph at a time. Much of it only matters once a particular kind of work has started. Rules for writing up web research don’t matter until the first search. The check before editing a page from another conversation doesn’t matter until a search finds that page.

Move each of those sections into a guide that rides on the first answer of the tool that starts that work, once per run:

{
  "results": [ ... ],
  "guide": "Research rules for this run: a snippet is not a source,
            open a page before you cite it, never build a URL yourself,
            and write findings up as ..."
}

Keep in the prompt whatever decides a first move, plus one line telling the model that more guidance arrives with the tools. That line matters: a model told less must still know that it hasn’t been told everything.

3. Send long documents as outlines

A long page in view becomes its summary and an outline, with each section’s size and a note that the outline is not the page. The agent reads the block, section or range it needs. Short pages still go in whole, because a round trip costs more than it saves.

Summary: plan for the admin pages revamp, three phases, open question on roles.

Outline (this is not the page; read sections with page.read):
  [b12] Goals ........................ 410 chars
  [b31] Phase 1: navigation .......... 2,900 chars
  [b58] Phase 2: permissions ......... 3,400 chars
  [b90] Open questions ............... 700 chars

Apply the same rule to the agent’s own output. A page it wrote earlier in the turn should be named, not sent back to it in full.

4. Keep the tool list fixed

It is tempting to list tools by name and load each definition only when the agent picks it. Don’t. Providers cache a request from the start, and tools come first. Changing the tool list mid-run changes the beginning of the request and invalidates the cache for everything after it.

Instead, keep every tool available from the first step and make each definition its short form: what the tool is for, when to use it, and its input shape without the titles and repeated text schema libraries generate. The longer advice becomes the tool’s guide, delivered with its first answer.

Steps 1 to 4 took the start of every run from 12,418 to 10,272 tokens, while new features kept adding text to the prompt.

The middle of the run

5. Find the biggest tool result in the transcript

Sort every tool result in the transcript by size. In one real conversation on this agent, a single code search returned about 7,000 tokens. That result stayed in the transcript and was sent again on every step after it, in a run of 14 calls.

The start of a run is paid once. A result like that is paid on every step. This is where the larger share of the waste sits.

6. Shrink large results to a note and a handle

Past a set size, a tool result should be kept as a short note of what it held, plus a handle to fetch the full result again:

[code search result, ~7,000 tokens -> kept as a note]
Searched "admin" in the repository: 38 matches across 12 files.
Most relevant: admin/routes.py (permissions), admin/pages.tsx (layout).
Full result: repo.code(id="r-17") to read it again.

Nothing is lost. The agent can always read the whole result again. It just stops paying for it on every step. The same idea extends further:

• Send big searches to a subagent. It reads the large result in its own context and returns the few lines that matter.
• Don’t return the same file twice. If a file was already read in this run and hasn’t changed, point to the earlier read.
• Clear old reads. Once older code reads pass a size limit (40,000 tokens on this agent), drop them and keep a note of what mattered.
• Edit documents by section. Rewriting a whole page to change one section sends the whole page twice.

Before anything ships

7. Replay real conversations

Every context change is replayed on real conversations, on a copy of production data, through the same code path the app uses. The replay set includes questions that can only be answered from a detail the change deferred, each with a reference answer.

• Answers: a judge model grades each one, and a string check confirms the key facts.
• Behaviour: whether the agent fetched what it needed, how many steps it took, and what it did afterwards.
• Cost: per conversation and at the start of a run, reported beside quality, never instead of it.

Counting behaviour is the part most teams skip, and it matters. One version of the tool change here was 14.8% cheaper and scored the same with the judge, but after reading a page the agent offered next steps in 24 of 32 conversations instead of 31. The replay flagged it, and it shipped only once that was back to normal.

The rule: quality no worse, and cost no worse beyond the noise of the replay. Run it twice. A saving that disappears on the second run was noise.

Results so far

Each change was measured against the code as it stood at the time.

Closing thoughts

Most teams stop at the start of the run because that is the part they can see in a file. The bigger cost is in the middle, in results that ride along on every step. Count both, fix the start first because it is quick, then go after the middle, and replay everything before it ships.

Appendix: the audit as a checklist

THE START OF THE RUN
[ ] 1. Count it with your provider's token counter, section by section.
[ ] 2. Ask "when is this needed?" of every prompt paragraph.
       Move late-needed rules into the first answer of the tool that starts that work.
[ ] 3. Send long documents as a summary plus a sized outline.
[ ] 4. Keep the tool list fixed for the whole run; shorten definitions instead.

THE MIDDLE OF THE RUN
[ ] 5. Sort tool results in a real transcript by size. Find the biggest.
[ ] 6. Past a set size, keep a note plus a handle to fetch the full result.

BEFORE ANYTHING SHIPS
[ ] 7. Replay real conversations. Grade answers, count behaviour,
       report cost beside quality. Run it twice.