Why my coding agent re-read 162,000 tokens on every single turn

Some models take notes you can photocopy. Others keep a running mental state you cannot copy. An agent that takes sixty steps needs the photocopy.

That is the whole finding. Everything below is how I got to it, starting from an engineering task that ran forty-five minutes without finishing, and a line in the log I had read straight past at least three times.

I run a coding agent on my own hardware, two RTX 5090s with 64 GB of VRAM between them. I have written before about how the machine is put together and tuned, about stopping its inference server burning CPU while idle, and about the engine bake-off that decided what serves on it. That last post admitted it had two problems tangled together and only pulled one of them apart. This is the other one.

It is the worse of the two. An engine you can swap in an afternoon. This was the model, and it was a property of the model that no amount of tuning was going to fix.

Why is my agent slow on a local LLM? Every obvious answer was wrong

The task was ordinary. Read some files, run the tests, read the failure, edit, run them again. The kind of card that should take ten minutes. It ran past forty-five and never finished.

I reached for three explanations, in this order.

The GPUs are thermally throttling. They were hot. The top card in a stacked pair runs about twenty degrees warmer than the bottom one because it is starved of intake air, and mine sat in the high eighties with the fan pinned. That is a real problem and I fixed it separately by power-limiting both cards. It was not this problem.

The model is too slow. Also no, and the log said so plainly, which I will come back to.

The context is too long, so shrink the system prompt. This is the one that felt obviously right, and it is the one worth dwelling on, because it is wrong in an instructive way.

There genuinely was bloat. The agent was being handed roughly 185 tool definitions on every request, and measured from the router's own request log those schemas came to about 55,000 tokens. Curating them down to about 24 tools was worth doing on its own merits and I did it. But it did not touch the thing that was actually hurting.

Here is why. The context at the point of pain was about 162,000 tokens, and the injected system prompt prose was only three to four thousand of that. The rest was the transcript: every file the agent had read, every command it had run, every diff it had applied, accumulated over dozens of turns. That is not bloat you can trim. That is the work.

And the deeper point, which took me embarrassingly long to see:

Prompt slimming changes the size of the thing you re-read. It does not stop you re-reading it.

Halve the context and you halve a cost you are still paying on every single turn, in a loop whose defining feature is that the context grows. That is a discount, not a fix.

llama.cpp full prompt reprocessing every turn: two numbers that disagreed

The log had the answer in it the whole time. What it needed was for me to read the two halves against each other instead of separately.

Take those two speeds one at a time and neither looks alarming. 3,100 tokens per second of prompt processing is a reasonable rate. 140 tokens per second of generation is healthy. Nothing is broken. Nothing is throttling.

Now put them next to each other. A few hundred tokens of reply at 140 tokens per second is a handful of seconds. Reading the context at 3,100 tokens per second is 52 seconds. The machine was spending something like fourteen times longer reading its own notes than writing anything down.

Multiply that by the loop. Sixty turns at 52 seconds of re-reading each is 52 minutes of doing nothing but catching up, before a single useful token is generated. That is the forty-five minute card, entirely accounted for.

This is the diagnostic worth taking away, and it generalises past my hardware:

When generation is healthy and prompt processing is not, you do not have a speed problem. You have a caching problem. A slow GPU is slow at both. A model too big for the card is slow at both. Only a cache failure produces one healthy number and one terrible one, because only a cache failure lets the machine be fast at work it has to do and catastrophic at work it should not be doing at all.

It looked like a sliding window attention problem. It was not.

My first real hypothesis, after the three bad ones, was sliding window attention. That would have been good news, because it is the kind of thing a launch flag fixes.

Two lines killed it.

The first was n_swa = 0. No sliding window was configured, so there was no window to be fighting with.

The second was the log saying what it was doing and why, in as many words. Paraphrasing only the punctuation, it reported that it was forcing full prompt re-processing due to lack of cache data, and it named the reason in parentheses: hybrid/recurrent memory.

I want to be honest about how that felt, because it is the useful part. That line had been scrolling past me for days. It reads like noise. It is not noise. It is the server telling you, correctly and specifically, that it cannot cache this model's state, and that it is therefore going to start from the beginning every time.

If you are pasting log text into a search box right now, those are the two strings to search for: n_swa = 0 alongside anything about full prompt re-processing and hybrid or recurrent memory. Together they mean the problem is architectural, and no flag on the launch line will move it.

Gated DeltaNet KV cache: you cannot fork a rolling state

Here is the plain version first.

A standard attention model, as it reads, leaves behind a set of notes: one note per token, and each note stays put once written. That collection is the KV cache. Because the notes are separate and stable, you can photocopy the stack at any page, hand the copy to a new request, and it carries on from there. The copy is exact and taking it costs nothing much.

A hybrid, linear or recurrent model does not keep notes. It keeps a running summary in its head and updates it token by token. That is far cheaper as it goes, which is the entire attraction, and it is why these architectures post such good numbers on long context. But there is only ever one summary, it only reflects the exact sequence that produced it, and there is no page to photocopy. To get the state at token 100,000 you have to read the first 100,000 tokens. Again.

Now precisely.

Standard attention families, and this includes grouped-query attention, multi-head latent attention and sliding-window variants, materialise per-token key and value tensors. Those tensors are position-addressable and immutable once computed, so a serving engine can hash a block of them, store it, and hand it to any later request whose tokens hash the same. That is what cross-request prefix caching is, and it is why a shared prefix is nearly free the second time.

Gated DeltaNet, and linear and recurrent designs generally, carry a fixed-size recurrent state that is overwritten at every step. There is no per-token artefact to address and no block to hash. The state is a single moving object, and the operation that prefix caching fundamentally depends on, which is forking a conversation at an arbitrary point and continuing two ways from there, has nothing to fork.

That is the sentence to keep. Prefix caching is a fork operation, and a rolling state cannot be forked.

The model I was running, Qwen3-Coder-Next, is exactly this. It is an 80B model with about 3B active parameters, it scores extremely well on agentic coding benchmarks, and on a spec sheet it is close to ideal for a box like mine. It was the single worst choice available for the workload, and nothing on the model card says so.

Caching support by attention architecture

One row of that table deserves calling out on its own, because it is the trap I very nearly walked into a second time.

Qwen3.6-27B looks like the perfect escape hatch. About 18 GB, excellent coding scores, and described as dense, which sounds like it must be standard attention. It is not. It is a three-to-one Gated DeltaNet hybrid, and "dense" there refers to parameters-per-token, not to the attention mechanism. The word you are looking for on a model card is not dense. It is the attention architecture, named explicitly.

Qwen3-Next prefix caching and hybrid model prefix cache on vLLM: where this stands in September 2026

I made my decision in June 2026. This section is the part most likely to be stale, so I re-checked every claim in it against the public trackers on 2 September 2026 before publishing. All four issues below were open on that date.

It has moved. It has not arrived.

vLLM. The tracking issue for prefix caching on hybrid models, #26201, is still open. An align mode now exists and is, for Gated DeltaNet hybrids, the only prefix-caching mode those models support. It works by keeping one recurrent state checkpoint per request, at the last aligned block boundary before the prompt ends. A second pull request, #26807, would add an all mode that checkpoints at every block boundary. It is open, not merged, and its own description notes that align currently beats it on raw throughput and that cold-cache and warm-cache requests can produce slightly different output because of a float32 to bfloat16 truncation at block boundaries. Warm requests stay self-consistent. Still, an answer that changes depending on whether the cache was warm is a strange thing to introduce into an agent loop.

Two open bugs matter more than the feature status.

#40696 reports that on Mamba-hybrid models the attention block size is pinned to 528 tokens to match the recurrent page size, which creates a cliff. Below 528 tokens the reported hit rate is roughly zero. Above it, about 95%. The reporter measured throughput falling from 200 requests per second to under 100 by shortening prompts from about 560 tokens to about 480.

#45238, filed 11 June 2026 and still open with no fix merged, is the one I would want to know about before running any agent on this. Because align mode keeps only that single checkpoint at the last block boundary before the prompt ends, the checkpoint can land inside the tokens that are unique to the current request rather than inside the shared prefix. When it does, the recurrent component fails to match, and since the hybrid coordinator requires every component group to hit before it will count a hit, the whole request reports zero reuse. The attention blocks matched perfectly. It does not matter. The reporter measured 52 hits out of 64 with a 1,600-token shared prefix, and 0 out of 64 after shifting 100 tokens from the shared part into the unique part, with mean time-to-first-token going from 433 ms to 905 ms.

Read that failure shape against an agent loop. Every turn of an agent appends new, request-unique tokens to the end of a large shared prefix. That is not an edge case for this bug. That is the normal operating condition.

And the failure is silent. There is no error. The hit rate simply reads zero, and the issue's own suggested remedy includes, at minimum, adding a metric so you can see when the recurrent component vetoed an otherwise perfect match.

SGLang. The picture here is genuinely better than it was in June, and it is better in a structural way rather than a flag-shaped way. LMSYS published Unified Radix Cache on 11 August 2026. Instead of one cache implementation per attention type, it keeps a single token-keyed tree whose nodes carry a reuse rule per component: full attention keeps the KV for every token in the matched prefix, sliding-window attention keeps a contiguous trailing window, and the recurrent component keeps one checkpoint at the reusable frontier. Its published benchmarks include a hybrid run on Qwen3-Next-80B-A3B, which is the same family as the model that started all of this.

So the honest summary is not "hybrid models cannot be cached." It is narrower and more useful. Whether a hybrid caches is a property of your engine, not of the model, and the answer changes month to month. On vLLM the sharp edges are documented, open, and fall exactly where a long agent loop puts pressure. On SGLang they largely do not, as of August. If you are serving one-shot requests, much of this will not reach you. If you are running sixty-turn loops, read those four issues before you commit to vLLM.

Everything in this section is read from public trackers rather than measured by me, and it will keep moving. I know it will, because it moved underneath me while this series was being written, which is the last section of this post.

What I ran instead, in June

At the time, I stopped trying to make the trap work and took the boring option: a standard-attention model that prefix-caches cleanly and leaves room for the cache once it is loaded. Read this section as a snapshot of June, because it did not stay true.

That turned out to be GLM-4.7-Flash, a 30B model with about 3B active parameters using multi-head latent attention, quantised to 4-bit AWQ, served on vLLM across both cards. It occupies roughly 17 GB, which on a 64 GB box leaves a great deal of headroom, and headroom is not a nice-to-have here. The KV cache is the thing you are trying to keep.

The result, measured on a roughly 30,000-token shared prefix: 19.7 seconds on the cold turn, 0.10 seconds on every warm turn after it. About 190 times faster, at around an 80% hit rate on a real transcript.

Two honest caveats on those figures, both of which I made in more detail in the engine post. The 19.7 and the 52 are not a like-for-like comparison: different engine, different model, different context size. The number that transfers is the within-engine ratio between cold and warm, not the gap between the two stacks. And 80% is what a real agent transcript looks like, because a real transcript keeps appending tokens nothing has cached yet.

The shape is the argument, not the magnitude. With prefix caching the per-turn cost stays roughly flat as the conversation grows. Without it, the per-turn cost grows with the transcript, on a workload defined by a transcript that grows.

I also gave up some benchmark score to get there. Qwen3-Coder-Next scores higher on agentic coding than what I replaced it with. It also never finished the card.

The box runs a hybrid again

Here is the part that undercuts the tidy version of this post. Leaving it out would make the post more quotable and less true.

The machine no longer runs GLM-4.7-Flash. It runs Ornith-1.5-35B-A3B: MIT licensed, roughly 35B total parameters with about 3B active per token, 262,144 tokens of context, served on SGLang.

Ornith-1.5 is a Gated DeltaNet hybrid. Not a distant relative of the thing that cost me the forty-five minute card, but the same architecture class in the same three-to-one arrangement. Its config.json reports Qwen3_5MoeForConditionalGeneration with forty hidden layers and three linear-attention layers for every one full-attention layer, so ten of the forty carry standard attention and thirty do not. By the letter of the check I am about to recommend, I should have closed the tab.

Two things changed, and neither of them was the architecture.

The engine caught up. The unified radix cache described above is what makes this viable. The recurrent component stopped being a thing the cache could not express and became a component with its own reuse rule, one checkpoint held at the reusable frontier. That is not a workaround on top of the constraint. It is the constraint being answered.

And it is measurable on the box, which is the only part that settles anything. Serving a Gated DeltaNet hybrid on this machine on SGLang, the prefix-cache hit rate on resumed turns reads about 99.4%. It occupies 27.6 GB per card at tensor-parallel 2, starts in 60 to 150 seconds, and decodes at 56 to 68 tokens per second at 70,000 to 85,000 tokens of context. Compare that 99.4% against the number this whole post is about, which was effectively zero, forever, on every turn.

Three caveats, because this is exactly the kind of good news that gets over-claimed.

The serving figures above were measured on Ornith-1.0, the previous release in the same family with the same three-to-one hybrid layout, on this same box and this same engine. They are the closest first-party evidence I have and they are not a published like-for-like measurement of 1.5 itself.

This is SGLang. Every vLLM issue in the section above was still open when I checked on 2 September 2026. If you are on vLLM, none of this rescues you yet.

And the June trade quietly stopped being a trade. Ornith-1.5-35B reports 79 on SWE-bench Verified against Qwen3-Coder-Next's roughly 70 to 74. The architecture I abandoned for being uncacheable now has a descendant that caches and scores higher. That is timing, not vindication, and it is the strongest argument in this post for the thing I am about to tell you to do.

The check to run before you download 50 GB of weights

Architecture is a compatibility constraint, not a spec-sheet detail. It belongs in the same mental category as "does this fit in VRAM", and it deserves to be checked before the download rather than after.

Three steps, in increasing order of how much they cost you.

One. Read the architecture off the model card, and do not accept a summary. You are looking for the attention mechanism named explicitly. Grouped-query attention, multi-head latent attention and sliding-window attention are all fine. Gated DeltaNet, linear attention, Mamba, state-space and the word hybrid are all reasons to keep reading. Do not let "dense" reassure you, because as Qwen3.6-27B shows, it can describe the parameter count while the attention underneath is still hybrid. If the card is vague, open config.json and look at the layer types directly.

Two. If it is hybrid, go and read your engine's open issues for that architecture before you download anything. Not the release notes, which tell you a feature exists. The issue tracker, which tells you how it fails. For a linear attention agent loop the questions that matter are whether caching is on by default, what block size or alignment it imposes, and whether a miss in the recurrent component silently vetoes an otherwise good match. All three of those are open questions on vLLM as I write this, and all three are cheaper to read about than to discover.

Three. Run the two-turn test before you trust any of it. This is the one that actually settles it, it takes about a minute, and it works on any engine and any model. Send a long prompt and time the prompt-processing phase. Then send the same prompt with a little extra on the end and time it again. If the second turn costs about what the first one did, you have no reuse, whatever the documentation says. If it drops to near nothing, you do.

Step three is the only one of the three that cannot go stale. Steps one and two tell you what was true when someone wrote it down, and this series is a demonstration that both of them expire: the same architecture fails the test in June and passes it in September on a different engine. Run the timing test on the model you are actually about to serve, on the engine you are actually going to serve it with.

That third check is the one I wish I had run in week one. It would have cost me sixty seconds and it would have saved me the forty-five minute card, the thermal investigation, the prompt diet, and several days of believing I had a hardware problem.

The model was never slow. It was just being asked to read its notes from the beginning, sixty times in a row, because nothing in the stack knew how to keep them. The model on that box today has the same architecture and does not have the problem, which is the whole argument for testing rather than trusting.