Fast Models Turn Agent Infrastructure Into the New Competitive Battleground

OpenAI said in its GPT-5.3-Codex-Spark announcement that the new real-time coding model can generate more than 1,000 tokens per second on specialized Cerebras hardware. That is a striking jump from the roughly 65 tokens per second that OpenAI says earlier flagship models delivered through the Responses API. But the more consequential development is not the token number alone. It is the admission that, at that speed, the API and agent runtime can become the bottleneck.

For AI application builders, this changes the competitive map. Model providers have spent years competing on intelligence, context windows and price. Agent vendors now face a different execution test: can their stack retain conversational state, relay tool calls efficiently, enforce safety checks without repeated work, and return useful results at the cadence a fast model makes possible?

The answer matters especially for coding agents. A request to fix a bug may require dozens of turns in which the model examines files, selects a tool, waits for code execution, receives the result and decides what to do next. Even modest fixed overhead, repeated across every turn, becomes visible to a user waiting for a completed task.

The new critical path

The conventional mental model of AI latency puts inference at the center. That made sense when GPU generation was slow enough to conceal nearly everything around it. A user sent an HTTP request, the system validated it, assembled context, tokenized input, routed the request, generated output and returned a response. If generation consumed most of the time, optimizing the surrounding layers produced limited product benefit.

At 1,000 tokens per second, that relationship changes. The model can generate a 500-token answer in about half a second. A sequence of session reconstruction, request validation, tokenization, safety processing, network hops and tool round trips can easily dominate the user’s perceived wait.

User inputrequest and session data

Client session assemblyassembles request context

API processingvalidates safety and renders context

Model inferencevariable latency

Tool executionclient or local tool

This distinction also clarifies why time-to-first-token, or TTFT, is not the same as time-to-useful-result. TTFT measures the first visible sign that the system is responding. It is essential for interactive experiences because it affects whether a product feels alive. Yet an agent can begin streaming quickly and still take a long time to complete a coding task if its tools are slow or if each subsequent turn repeats expensive setup work.

Time-to-useful-result is the better commercial metric for agent products. It captures the point at which a developer receives a usable patch, an analyst receives a validated answer, or an operator receives an action that can be reviewed. The distinction matters because a product can improve one metric while leaving the other largely unchanged.

Why a persistent connection changes the economics

OpenAI explained in its WebSockets engineering post that it approached the problem by caching rendered tokens and model configuration, removing intermediate network hops, accelerating parts of its safety stack and keeping a persistent WebSocket connection to the Responses API. OpenAI says those changes made agent loops 40% faster end-to-end.

The architectural shift is straightforward. In a sequence of independent HTTP requests, each follow-up can require the platform to reconstruct conversational state and process context that mostly has not changed. That creates a tax on every turn. The tax grows with longer conversations and more tool calls.

A persistent session reverses that pattern. The service keeps connection-scoped state in memory. Rather than resending and reprocessing the entire history, the client sends only new input or tool output. The system can retain prior response objects, tool definitions, namespaces, rendered tokens and model-routing decisions.

Send full history Validate full history Render tokens Route model Infer Return responseRepeated work, turn 1

new follow-up request

Send full history Validate full history Render tokens Route model Infer Return responseRepeated work, turn 2

new follow-up request

Send full history Validate full history Render tokens Route model Infer Return responseRepeated work, turn 3

Create connection Validate and render shared state InferCached state retained in the session

persistent session

Send only new tool result or user inputNew input crosses the connection

new delta

Validate delta Append cached tokensReuse cached state

validated context

Resume inferenceContinue from the cached context

Accent marks repeated work; alternate tone marks cached session state

The point is not that WebSockets are universally superior to HTTP. They are superior when an application has repeated, stateful exchanges whose reusable context can remain valid during the session. A one-shot request with a small prompt has little state to amortize. A long-running coding agent, by contrast, has exactly the conditions that reward persistence.

OpenAI reports that its changes reduced per client-server round-trip overhead by 80%, per-token overhead by 30% and TTFT by 50% for the Codex-Spark path. These figures are reductions, not interchangeable measures of total user latency, but together they show where the gains came from: less setup, less repeated work and less delay before the model begins producing visible output.

	Latency-overhead measure	Reported reduction
Client-server round-trip overhead, 80%	metric	reduction
Time to first token, 50%	metric	reduction
Per-token overhead, 30%	metric	reduction

Source: OpenAI, Introducing GPT-5.3-Codex-Spark