On September 15, DeepSeek released V4.1-Flash, and the detail that matters here is not a benchmark score, it is the per-token KV cache footprint: roughly 890 bytes, about a quarter of what its predecessor needed and 437 times smaller than DeepSeek's original model. Getting there took cross-layer cache reuse and FP4 KV caching, stacked on top of the Multi-Head Latent Attention architecture DeepSeek introduced two years earlier. That is a direct answer to the single largest hidden cost in running AI agents in production: the memory that has to sit resident on a GPU for the entire lifetime of every active session.
For teams shipping single-turn chat features, KV cache size was never a line item worth worrying about. Agents changed that. An agent loop does not send one prompt and get one answer back, it holds a growing conversation, a set of tool schemas, and every tool result it has seen so far, all resident in memory for as long as the session runs. Industry estimates put typical agent workloads at input to output ratios around 100 to 1, which means the overwhelming majority of an agent's inference cost is driven by how much context has to be held and re-attended to, not by how many tokens it eventually generates. That is a memory-bound problem, and the KV cache is where that memory lives, a cost centre most teams running agents in production have never actually measured.
Every time a transformer model generates a token, it needs to attend back over every token that came before it. Recomputing that attention from scratch on every new token would make autoregressive generation unusably slow, so inference servers cache the key and value vectors for every token, in every layer, in every attention head, the first time they are computed, then reuse them for every subsequent token in the session. That cache is what makes fast, token-by-token generation possible. It is also a memory allocation that grows linearly with context length and has to sit on the GPU for as long as the session stays active, competing directly with every other concurrent request for the same finite pool of high-bandwidth memory.
Agent workloads load that cache in ways a simple chatbot never does. System prompts alone routinely run two thousand to ten thousand tokens once you include role definitions, formatting rules, and safety instructions. Tool and function schemas add another five thousand to fifty thousand tokens depending on how many capabilities the agent has access to, all of which has to sit in context before the model can even decide whether to call anything. Then the conversation itself grows by a few hundred to two thousand tokens with every turn, as tool calls, tool outputs, and intermediate reasoning accumulate. None of that is unusual for a single request. It is the accumulation across a long-running session, multiplied across every concurrent user, that turns the KV cache into the resource that actually caps how many agent sessions a given GPU fleet can serve at once.
This is also why naive serving setups waste so much of the memory they do allocate. Inference engines that reserve a fixed, contiguous memory block per sequence waste an estimated sixty to eighty percent of KV cache capacity to fragmentation, since they have to provision for the worst case rather than the length a session actually uses. vLLM's PagedAttention, which borrows the paging idea from operating system virtual memory, brought that waste down to under four percent, and is why it became the default serving choice for teams self-hosting open models.
In practice, the teams handling this well are combining four distinct levers, and they do not all live in the same layer of the stack.
Prefix caching reuses the key and value vectors for any prefix that is identical across requests, so the model does not recompute attention over content it has already seen, such as a stable system prompt or a fixed set of tool schemas. It is the cheapest lever available because it requires no model change and no new infrastructure, only prompt discipline, but the pricing and mechanics differ enough between providers to change how you should structure calls.
If you are self-hosting inference rather than calling a hosted API, the serving layer's memory manager matters more than almost anything else. PagedAttention-style serving, now the default in vLLM and most serious open-model deployments, allocates KV cache in small, non-contiguous blocks instead of one large reserved region per sequence. That single change accounts for most of the fragmentation waste reduction mentioned earlier, and typically unlocks a two to four times throughput gain on the same GPU fleet by letting more concurrent sessions share memory that used to sit reserved and idle.
Beyond memory management, you can shrink what each cached vector costs to store. Four-bit and FP4 KV quantization compress the cached key and value vectors themselves, and the difference is not subtle: on a 24GB GPU, FP16 caching fits roughly three concurrent agents at an 8,000-token context, while Q4 fits around twelve, and at 16,000 tokens and beyond, FP16 often cannot fit even a single multi-agent workflow where Q4 still can. DeepSeek's move to FP4 KV caching in V4.1-Flash applies the same idea at the model-provider level rather than in your own serving stack.
The most aggressive lever is not something you bolt on, it is a property of the model architecture itself. Grouped-query and multi-query attention, now common across most modern model families, reduce the number of key-value heads that need caching in the first place. DeepSeek's Multi-Head Latent Attention goes further, projecting keys and values into a shared low-rank latent space and caching only that compact representation, then reconstructing head-specific detail through lightweight projections at inference time. V4.1-Flash combines that with cross-layer cache reuse to reach its 890-byte-per-token footprint. For a consultancy client, this lever mostly shows up as a model-selection decision rather than an engineering task: choosing a model architecture designed for cache efficiency does more for your inference bill than any serving-layer optimization you could add afterward, unless you are training or fine-tuning your own models.
None of these levers are free of trade-offs, and the failure modes matter more for agents than for simple chat.
For most teams, the right sequence is not to reach for the most sophisticated lever first, it is to fix prompt structure and measurement before touching infrastructure.
KV cache economics have quietly become the dominant cost variable in production agent systems, the same way per-token pricing was the variable everyone tracked two years ago, and most teams running agents today have never measured it. Start with an audit of how your prompts are structured and whether your current caching setup produces a real hit rate, since that single check tends to reveal more wasted spend than any infrastructure change. If you are scoping a production AI agent deployment and want a second opinion on where the memory and cost trade-offs actually sit for your workload, that kind of architecture assessment is part of the work we do in our AI consultancy engagements. Measure the hit rate before you optimize anything else, it usually tells you where to start.
Before we start, please share a few details so we can follow up with you.
End this conversation? Your chat will be emailed to us.