The Agent Loop Is Now the Real AI Speed Limit

A developer waiting for an AI coding agent rarely experiences a model as a stream of tokens. They experience it as a chain of pauses: the agent reads files, decides what to inspect, waits for a tool, absorbs the result, tries again, and finally explains itself. OpenAI’s July 29, 2026 account of GPT-5.6 matters because it reframes the performance problem around that human experience. Faster generation helps. But once an agent makes many model calls in one task, the more consequential question becomes simpler: how often are you making the system redo work it has already done?

For several years, the public measure of AI speed has been tokens per second. It is easy to understand and easy to market. A model that writes 80 tokens per second feels quicker than one that writes 30. Yet this measure describes only one stretch of a much longer journey.

A user asks an agent to diagnose a production error. The agent may search the codebase, read deployment history, inspect logs, compare configurations, edit a file, run tests, review the test output, and write a response. Each of those acts can require another model request. Every request may rebuild context, tokenize text, prepare a GPU workload, generate tokens, call a tool, apply approval or safety policy, carry data across a network, and then start again.

OpenAI says a single Codex turn can involve many model and tool iterations, and notes the uncomfortable arithmetic: if a task needs 30 requests, one added second per request becomes 30 seconds of user-visible waiting.

That is the shift behind GPT-5.6. It is not an argument that the model itself has become irrelevant. OpenAI describes improvements in model efficiency, speculative decoding, routing, caching, kernels, and orchestration. But the deeper lesson is that an agent is not a chatbot with buttons attached. It is a loop. Once the loop becomes long, repeated overhead becomes the bottleneck.

The practical implication is both empowering and humbling. Many teams will not get their next meaningful latency improvement by changing models. They will get it by preserving a stable prompt prefix, shrinking tool output, keeping a connection alive, running independent tools concurrently, and refusing to send the model the same irrelevant context for the tenth time.

The wait a user actually feels

Imagine an on-call engineer named Maya. It is late, a deployment has failed, and she asks her internal agent: “Why did checkout errors rise after the last release, and can you prepare a safe fix?”

The agent does not answer from a single prompt. It has to do work.

First, it needs the operating instructions that govern its behavior. Then it needs Maya’s request, the recent conversation, allowed tools, tool schemas, repository metadata, permissions, deployment context, and perhaps a summary of previous investigations. It may ask to search error logs. The tool returns thousands of lines. The agent decides to inspect a changed service. It reads a file, identifies a likely null-value path, writes a patch, runs tests, sees an unrelated failure, searches the deployment manifest, and eventually produces a recommendation for Maya.

From her side of the screen, the frustrating moments are not only the moments when the model is “thinking.” They are the gaps between actions. A fast model can make those gaps more conspicuous.

This is why raw output rate can become a misleading comfort metric. Suppose decoding becomes dramatically faster. The model now emits its tool call almost instantly. That does not make a database query faster. It does not shorten a cold container start. It does not remove a round trip to a remote tool server. It does not make a safety approval instant. It does not eliminate the cost of repeatedly processing a 100,000-token context.

It simply moves the bottleneck.

OpenAI’s description of its GPT-5.6 stack is unusually direct on this point. The company says that context preparation, data transmission, inference, tools, and process startup all contribute to latency. It frames its Rust-based agentic harness as the layer that connects models, tools, and user environments, and therefore the layer where repeated work must be removed.

The point of this diagram is not that every agent follows precisely this order. Some systems combine stages. Some tool calls happen in parallel. Some safety checks occur before inference, some after. The point is that a user’s one request is often a small workflow engine in disguise.

That workflow has a repeated region. Every millisecond inside it deserves suspicion.

The two kinds of model work

To understand why prompt caching has become central, separate the work a model does into two broad phases.

The first is prefill. Before the model can write its next token, it must process the input context: instructions, messages, tool definitions, prior tool results, files, and other material. This is where the model builds an internal working state commonly called a key-value cache, or KV cache. In plainer language, it is the model’s prepared memory of everything it has read so far.

The second is decode. This is the familiar sequential act of producing output tokens. The model writes a token, incorporates it into its working state, then writes the next one.

Decode is visible because it looks like typing. Prefill is easier to overlook because it happens before the first visible word appears. But in agent systems with long, repeated histories, prefill can be the hidden bill in both time and money.

Consider an agent that has accumulated a long policy prompt, a detailed tool manifest, a conversation history, and several tool results. It sends essentially the same material to the model before each new step. If it does this ten times in a task, it may be paying to process the same stable context ten times.

Prompt caching changes that equation. It allows previously processed prompt material to be reused rather than recomputed. OpenAI says that when uncached input is processed, the model builds its KV cache in a compute-intensive pass. It also says cache availability affects how requests are routed and served.

That is why caching is not merely a pricing feature. It is an architectural feature. It can cut the amount of repeated GPU work, reduce time to first token, improve capacity, and change the economics of an agent loop.

The most important phrase is “identical blocks.” Caches do not reward conceptual similarity. They reward stable, compatible representations of prior computation.

A human can see that two JSON tool schemas mean the same thing even if their fields appear in another order. A cache usually cannot safely make that assumption. A human can see that an approval policy remains unchanged even if an application regenerates it with a new timestamp. A cache key sees a changed input.

In practice, a cache miss can be caused by apparently small implementation choices:

• A tool list is emitted in a different order.
• A schema generator changes whitespace or key ordering.
• A runtime policy is inserted in the middle of an otherwise stable system prompt.
• A developer edits an old message rather than appending a new one.
• A tool result is retroactively summarized into an earlier history position.
• A dynamic date, request ID, feature flag, or session-specific label enters the stable prefix.
• The application silently changes model settings or a cache configuration that participates in reuse rules.

OpenAI’s GPT-5.6 guidance is explicit about the preferred discipline. Its harness treats model-visible history as append-only, presents tools in deterministic order, and applies runtime settings such as approval policies during execution rather than embedding them inside tool definitions.

This is not elegant housekeeping. It is performance engineering.

Why append-only history wins

A conversation seems like a document. That intuition causes trouble.

Documents invite editing. We revise paragraphs, replace old wording, move information to where it seems most readable. An agent history needs a different mental model. It is closer to an accounting ledger or a database log. New facts belong at the end. Earlier records should remain stable unless there is a compelling reason to invalidate the cache and recompute.

Suppose an agent begins with a 20,000-token stable prefix:

1. System instructions.
2. A security policy.
3. A repository map.
4. A deterministic tool manifest.
5. Earlier user messages.
6. Earlier model outputs.
7. Earlier tool results.

The agent then receives a fresh 800-token log excerpt. The efficient move is to append the excerpt at the end. The model can reuse the prepared representation of the 20,000-token prefix and process only the new material.

Now imagine a well-meaning orchestration layer that inserts the log excerpt beside the earlier incident description “for readability.” The semantic story is clearer to a human reader. But the position of everything after that insertion changes. The cached prefix may no longer match. The system has turned a small addition into a broad recomputation.

This is why agent traces should be treated as immutable event streams. If an earlier fact needs correcting, append a correction. If tool output is too large, append a compact summary and preserve a reference to the original result. Do not casually rewrite old context.

There is a quality reason, too. Long histories are not only expensive. They can distract the model. OpenAI says that larger context windows can increase cost, distract the model, and prompt unnecessary reasoning. Its harness uses deferred discovery so tools, skills, plugins, and integrations appear only when needed, and caps default tool output at 10,000 tokens unless the model requests more.

The best context is not the most comprehensive context. It is the smallest context that makes the next decision reliable.

That principle changes how teams should think about tools. A giant manifest containing every company integration is not a sign of an advanced agent. It may be a sign that the agent has been asked to read a telephone directory before making every call.

Cache hits have economics, not just elegance

GPT-5.6 makes cache behavior especially worth measuring because OpenAI has made the pricing visible. The company says cache writes are billed at 1.25 times the normal uncached input rate, while cache reads receive a 90 percent cached-input discount. It also introduced explicit cache breakpoints and a minimum 30-minute cache life.

That means caching is not automatic savings in every case. A cache write is an investment. It costs more up front because the service is preparing reusable work. The investment pays off only when later calls reuse it.

OpenAI lists standard GPT-5.6 input prices of $5 per million tokens for Sol, $2.50 for Terra, and $1 for Luna. With the 90 percent cache-read discount, the comparable cached-input rates are $0.50, $0.25, and $0.10 per million tokens.

The arithmetic is straightforward enough to put into an agent dashboard.

Let:

• S be stable prefix tokens.
• D be dynamic tokens added on each call.
• N be the number of model requests in a task.
• P be the uncached input price per token.
• r be the cache-read price ratio, here 0.1.
• w be the cache-write price ratio, here 1.25.

Without useful reuse, a rough input cost is:

N × (S + D) × P

With a first cache write and later reads, a rough input cost is:

w × S × P + D × P + (N - 1) × [r × S × P + D × P]

The exact bill depends on API behavior, cached ranges, token counts, and request configuration. But the strategic point does not change. If the stable prefix is large and the agent makes many turns, reuse can dominate economics. If the agent is making one-off requests with constantly changing context, cache writes can be a poor bargain.

For latency, the analogous calculation is even more revealing. Let Tfixed represent connection, scheduling, serialization, safety, and tool overhead. Let Tprefill(S) be time spent processing stable context. Let Tdecode be generation time. A sequence of uncached calls roughly behaves like:

N × [Tfixed + Tprefill(S + D) + Tdecode]

With a reusable prefix, later turns increasingly resemble:

First turn: Tfixed + Tprefill(S + D) + Tdecode

Later turns: Tfixed + Tprefill(D) + Tdecode

The faster decode becomes, the more obvious Tfixed, tool waiting, and repeated prefill become. That is the agent-loop bottleneck in one line.

Explicit breakpoints are a contract

The cache is strongest when an application tells the platform where stability ends.

OpenAI’s model guidance says GPT-5.6 supports explicit prompt caching, allowing developers to mark reusable prompt prefixes. It recommends tracking cached_tokens and cache_write_tokens, and using explicit breakpoints or explicit cache mode to prevent unnecessary writes.

This should be understood as a contract between the orchestration layer and the model service.

Before the breakpoint, the application promises: this material is stable enough to reuse. It will preserve ordering, formatting, and relevant configuration. After the breakpoint, the application is free to append the volatile material that represents the current moment: a user request, a new tool result, a changed file, a fresh query, or a new approval outcome.

A practical prompt structure might look like this:

1. Stable system instructions.
2. Stable organization and safety rules.
3. Stable repository or domain summary.
4. Stable tool manifest, sorted deterministically.
5. Stable prior history.
6. Explicit cache breakpoint.
7. Latest user input.
8. Latest tool result.
9. Temporary runtime state.

This does not mean every item must remain forever. A stable prefix can be periodically rebuilt when a task changes phase. The important point is to rebuild deliberately, not accidentally.

For example, an agent investigating an incident may begin with a broad repository map. Once it isolates the failing service, it can create a new compact phase-specific prefix containing only the relevant module map, policies, and validated findings. The cache is then working on a smaller, more useful foundation.

This is the difference between memorization and preparation. The goal is not to make the agent carry its entire past. The goal is to make it carry the right past cheaply.

Faster inference changes the shape of the problem

OpenAI’s July 29 account describes several lower-level inference improvements. It says speculative decoding uses a smaller draft model to propose several tokens that the primary model can verify in parallel, reducing costly sequential generation when proposals are accepted. The company says improvements to its draft model increased token-generation efficiency by more than 15 percent. It also says kernel improvements reduced end-to-end serving costs by 20 percent.

These are genuine gains. They matter for every user and every application. But their greatest strategic effect may be to expose the pieces around them.

Think of a restaurant kitchen. If a chef becomes twice as fast, the restaurant does not automatically serve meals twice as fast. The bottleneck may move to the host stand, the oven, the dishwasher, or the line between kitchen and table. The sensible response is not to deny the chef’s improvement. It is to redesign the whole flow.

The same is true for agents.

When the model takes several seconds to produce a tool call, a 300-millisecond network overhead is easy to ignore. When the tool call arrives almost immediately, that same 300 milliseconds becomes visible. When prefill is reduced through caching, a slow tool response becomes the dominant pause. When tools are parallelized, a human approval gate may become the final constraint.

A persistent connection can help by avoiding repeated setup work for every request. Incremental transport can help by sending only newly needed data rather than replaying large payloads. OpenAI distinguishes that network concern from prompt caching: incremental transport changes what crosses the network, while caching changes what model computation can be avoided.

Those are different levers. They should be measured separately.

The mistake is to describe this as an unfortunate side effect. It is progress. A system can only optimize the bottleneck it can see. Faster models make hidden operational waste measurable.

Instrument the loop before choosing the fix

An agent team should resist the urge to begin with a vendor comparison. First, collect a trace for representative tasks.

For every model request, record:

• A task and turn identifier.
• A request sequence number.
• Input token count.
• Output token count.
• Cached tokens.
• Cache-write tokens.
• Time to first token.
• Total model time.
• Context assembly time.
• Tokenization and serialization time.
• Queue or scheduling delay, if available.
• Network request and response time.
• Tool dispatch time.
• Tool execution time.
• Tool-result size before and after normalization.
• Safety, policy, or approval time.
• Whether the tool call was independent of other calls.
• Whether the result was actually used in the final answer.

Then graph the p50, p95, and p99 latencies by stage. Averages are comforting and often useless. A median tool call may be fast while one slow dependency determines the experience of everyone handling difficult tasks.

Next, calculate three ratios.

First, cache reuse ratio:

cached input tokens / total input tokens

Second, context efficiency ratio:

tokens materially relevant to the next decision / total context tokens

This requires judgment, but even a rough review can reveal tool dumps and stale history that nobody needs.

Third, orchestration share:

non-model wall-clock time / total agent-turn wall-clock time

If orchestration share is high, a faster model will have limited effect. If decode dominates, model selection or inference tier may be the right lever. If prefill dominates, focus on caching and context. If tool time dominates, inspect tool design, concurrency, and data locality. If approvals dominate, the issue is workflow design rather than AI performance.

OpenAI’s own recommendation for tool-heavy workflows points in a similar direction. Its GPT-5.6 guidance says teams should benchmark representative tasks and compare task success, completeness, required evidence, total tokens, latency, and cost. Fewer calls are only improvements if the final answer still meets the required quality bar.

That last qualification matters. A system can feel fast because it stopped checking its work. That is not orchestration. It is carelessness with a stopwatch.

A decision framework for the next millisecond

Once the trace exists, the choices become clearer.

Choose a faster model or faster service tier when output generation is genuinely the largest portion of critical-path time, when the task needs long answers or intensive reasoning, and when latency gains justify the added cost. OpenAI’s Fast mode documentation says its GPT-5.6 Sol service tier targets more than 80 tokens per second for 99 percent of eligible requests, though actual experience still depends on context length and the rest of the workflow.

Choose a persistent connection or incremental transport when traces show repeated setup, handshakes, serialization, or retransmission of data that has not changed. This is especially useful for interactive agents that make many short sequential calls.

Choose parallel tools when calls are independent. Searching logs and inspecting a deployment record may happen concurrently. Reading a file and running a test may not, if the test depends on the edit. Concurrency should follow dependency structure, not enthusiasm.

Choose less context when long prefill, low cache reuse, or irrelevant history are the primary costs. Use deferred tool discovery. Summarize selectively. Cap output. Retain references to source material rather than repeatedly injecting entire documents.

Choose stable prefixes and explicit cache breakpoints when the agent repeatedly reads a substantial base of instructions, schemas, and history. Preserve exact ordering. Do not let volatile runtime state leak into the reusable region.

Choose better tool design when tools return unbounded text, expose excessive schemas, or hide slow operations behind one vague call. A tool should return the smallest trusted artifact needed for the next decision. Its manifest should be stable, concise, and deterministic.

And choose a redesigned human workflow when the slowest step is an approval that cannot safely be automated. A faster agent cannot bypass an organization’s legitimate need for accountability. It can, however, prepare a better approval packet and reduce the human time needed to assess it.

The future of agent performance will therefore look less like a race between model names and more like operations research. The winning systems will map dependencies, remove duplicate work, shorten queues, preserve reusable state, and place judgment where it belongs.

The human waiting at the screen will not care whether the improvement came from a kernel, a draft model, a cache hit, a sorted tool list, or a tool call that finally ran in parallel. They will care that the agent did not make them wait while it reread the same manual for the ninth time.