The most consequential part of OpenAI’s Jalapeño announcement is not the claim that a new chip is faster. It is the suggestion that the old choice between quick answers and efficient answers may be less fixed than the industry assumed. If that holds up in production, the question for buyers changes: not which accelerator has the biggest headline number, but where a real request spends its time, power and money.
A person waiting for an AI assistant is not waiting for a chip benchmark. They are waiting for a first useful word, then for the next words to arrive at a pace that feels like thought rather than delay.
That distinction matters more now that AI products are becoming active collaborators. A programmer using an agent does not merely send one prompt and receive one answer. The agent reads files, calls tools, revises a plan, asks for another permission, and continues. A researcher may submit a long document, wait for an initial summary, then ask a series of follow-up questions. A customer service system may need to retrieve records, reason over policy, draft a response and check it before a human ever sees it.
Every pause in that chain becomes part of the experience. Multiply a small delay by dozens of model calls and the system begins to feel slow even when its individual components look respectable on paper.
OpenAI’s August 25 Jalapeño results are interesting because they target this uncomfortable reality. The company says its first custom inference chip can deliver more AI work per watt while returning answers with lower latency than the comparison systems it tested. Across GPT-OSS 120B, DeepSeek R1 670B and Kimi K2.5 1T, OpenAI reported 1.5 to 1.9 times more work per watt at peak throughput, alongside 1.7 to 3.6 times lower end-to-end latency. For highly interactive workloads, it reported 2.1 to 4.1 times higher performance.
Those are striking claims. They are also exactly the kind that deserve careful reading.
For years, infrastructure teams have treated low latency and high utilization as opposing goals. Keep a machine ready to serve one user immediately, and much of its capacity may sit idle. Fill it with many users to improve efficiency, and each user waits longer in the queue or shares resources with everyone else. Inference has turned that balancing act into one of the central economic problems of AI.
Jalapeño does not abolish the underlying physics. It cannot make memory movement free, erase network delay, or turn a giant model into a small one. What it appears to offer is a different answer to a more precise question: can a system reduce the waste that makes the trade-off feel inevitable?
That is a question about the whole serving path. It reaches beyond the arithmetic engines on a chip, into memory capacity, memory bandwidth, network topology, model placement, batching policy, scheduling software and the definition of a fair benchmark.
The request is really two workloads
When people say an AI model is “generating an answer,” they flatten two very different jobs into one phrase.
The first is prefill. This happens after a user sends a prompt and before the model produces its first response token. The system reads the prompt, processes every input token through the model, and builds the state needed to continue the conversation. The longer the prompt, the more work prefill demands.
The second is decode. This starts when the first answer token is produced. The model then generates one token at a time, feeding each new token back into the process as context for the next. A reply that looks like a flowing paragraph is, at the system level, a sequence of small repeated steps.
That difference is the beginning of the inference bottleneck story.
Prefill tends to offer abundant parallel work. A model can process many prompt tokens in broad matrix operations, which is exactly the kind of task modern AI accelerators handle well. In the simplified picture, the machine can keep its arithmetic units busy because there are many calculations available at once.
Decode is different. To create the next token, the system repeatedly needs access to model weights and to the context state accumulated during the conversation. That context state is often called the KV cache. The task becomes less like moving a freight train at full capacity and more like repeatedly finding one specific book in a vast library, reading it quickly, putting it back and immediately finding the next one.
The bottleneck may no longer be raw arithmetic. It may be the rate at which data can travel from memory to compute and back again.
The two user-facing measurements follow naturally.
Time-to-first-token, often abbreviated TTFT, is the delay between a request arriving and the first visible answer token. It includes queueing, prompt processing, routing, any retrieval or tool preparation that occurs before generation, and the prefill phase itself.
Time-between-tokens, or TBT, is the interval between subsequent output tokens. It captures the cadence of the answer after it has started. If TBT is high or erratic, a response may begin promisingly but then crawl across the screen.
Neither measure is sufficient on its own. A system can achieve a superb TTFT by reserving capacity for each user, only to generate slowly afterward. Another can deliver excellent throughput by batching aggressively, while making a user wait long enough for the first token that the product feels unresponsive.
The important point is that these are not simply two ways to score the same machine. They expose different physical constraints and different scheduling choices.
OpenAI says Jalapeño was designed around this division. Its account describes prefill as compute-intensive, decode as more constrained by memory bandwidth, and multi-chip communication as another source of delay. The company says the system is intended to keep model state, including KV-cache data, explicitly placed and local while activating compute, memory and networking according to the phase of inference.
That is a systems claim, not merely a chip claim.
Why efficiency and responsiveness usually collide
Imagine a busy restaurant with a small kitchen.
If the chef makes one dish at a time the moment each order arrives, diners receive immediate attention. But the kitchen wastes opportunities to prepare similar ingredients together. It uses more labor and energy for each meal.
If the chef waits until many orders arrive, the kitchen can chop, cook and plate in efficient batches. But the first diner sits watching the door. The kitchen has optimized itself while making the customer less happy.
AI inference faces the same tension, only at much higher speed.
Batching is usually the tool that increases accelerator utilization. By putting many requests together, a serving system can spread overhead across more work and create larger parallel operations. That can lift total tokens produced per second and improve energy efficiency. It can also raise queueing time, especially when requests have different prompt lengths, different desired output lengths and different latency expectations.
The problem becomes sharper during decode. Every request wants another token soon, but the server may be handling thousands of active sequences. It must decide which ones receive attention in the next scheduling step. Serve a large group together and improve aggregate efficiency. Serve a small group and protect interactivity. Move data between chips to assemble the required model state, and the arithmetic units may wait for the network.
This is why a claimed breakthrough in both speed and efficiency should not be read as proof that the trade-off disappeared. It should be read as a question: which sources of waiting have been removed, and under what workload?
There are several places where waste can hide.
First, a system may be waiting to form batches. Second, it may be moving weights from high-bandwidth memory into compute units. Third, it may be reading and writing KV-cache data. Fourth, it may be coordinating chips across a server or rack. Fifth, it may be stalled by uneven work distribution, where one device finishes early and waits for the slowest participant.
A chip with enormous mathematical capacity can still perform poorly if it spends too much of the day waiting for data. Conversely, a system with a more modest arithmetic peak can look remarkably effective if it keeps its critical data close and its work consistently flowing.
The roofline question: what is the machine waiting for?
A useful way to reason about inference is a roofline-style model. It does not require an electrical engineering degree. It asks only one practical question: is the workload limited by calculation or by moving data?
Every accelerator has a maximum rate at which it can perform math. It also has a maximum rate at which it can move data from memory. The achievable performance of a given task is constrained by whichever limit it hits first.
If a task performs many operations for every byte it needs to read, it is likely compute-bound. More arithmetic capability can help.
If it performs relatively few operations for every byte moved, it is likely memory-bound. Faster math alone will not change much, because the arithmetic units are already waiting on data.
Prefill often sits closer to the first case. Decode often moves toward the second. But those are tendencies, not laws. The balance shifts with model architecture, quantization, context length, batch size, mixture-of-experts routing, cache policies and the number of chips required to hold the model.
The phrase “memory bandwidth” can feel abstract. In practice, it is the width of the road between storage and the compute engines. The wider the road, the more model data can arrive every second. But road width is only part of the story.
Distance matters. A cache entry in a nearby memory resource is different from one that must cross a board, a switch or several chips. Congestion matters. A theoretically fast interconnect can still create harmful tail latency if many requests compete for it. Organization matters. If a system can predict what data will be needed and arrange for it to arrive before the compute unit asks, it can avoid idle time.
This is why the memory hierarchy is not a supporting detail. It is part of the accelerator.
The same applies to the KV cache. At every layer of a transformer, the model stores representations of previous tokens. As conversations grow, that accumulated state grows. For a long-running agent, cache management can become a defining infrastructure issue. The system may need to keep frequently used context in the fastest memory, share common prompt prefixes across users, evict less valuable state, compress it, or move it to a lower tier without causing the next token to stumble.
A benchmark that uses one prompt followed by one answer can reveal important capabilities. It cannot, by itself, represent the messier life of production traffic, where users return, agents make many calls, context windows expand and common system prompts may or may not be reused effectively.
SemiAnalysis, which operates the InferenceX benchmark used for the Jalapeño tests, made this caveat explicitly. Its reporting said it observed benchmark runs in OpenAI’s lab but did not run the full suite itself. It also warned that a single-turn 8K prompt and 1K output workload does not fully test the cache behavior, routing, prefix caching and offload machinery that longer, multi-turn agent workloads stress.
That does not invalidate the result. It specifies its boundary.
One model, many chips, one more bottleneck
A sufficiently large model does not fit neatly on a single accelerator. It is divided across devices. That can happen by splitting layers, splitting tensors within layers, distributing experts in a mixture-of-experts model, or combining several methods.
Each strategy creates communication.
Some communication is predictable and can be carefully scheduled. Some is bursty. Some arrives on the critical path, meaning the next token cannot be produced until the exchange completes. In those moments, more compute is irrelevant. A device can have idle arithmetic units while it waits for a remote device to send a partial result.
The practical danger is not simply that communication takes time. It is that communication creates coordination points. A multi-chip system can become like a relay race in which every runner is fast, but the baton exchange is awkward.
OpenAI’s public description is notable for treating the network as integral to Jalapeño rather than as external plumbing. Its stated goal is to minimize data movement and communication delays, using explicit placement of model state and a large connected domain to keep the whole workload fast from beginning to end.
This is the broader lesson. The best chip in isolation may not create the best serving system. The important unit of competition is increasingly the chip plus its memory, interconnect, server design, rack topology, compiler, kernels and scheduler.
That has strategic implications. General-purpose GPUs retain a major advantage in flexibility, mature software and broad deployment. They are useful across training, inference and many workloads that have not yet been fully characterized. A custom ASIC can remove overhead for a narrower target, but it takes on the burden of maintaining a software ecosystem and adapting to rapidly evolving models.
The custom-chip wager becomes stronger when a company has an enormous and predictable internal workload. It can see its traffic patterns, control the serving stack, tune models for the hardware and use production telemetry to refine the next generation. That is precisely the feedback loop OpenAI is describing.
It also explains why a result from a first-party chip should not automatically translate to a universal recommendation. An enterprise running a different model, using a different runtime, with different prompt lengths and a different latency promise may encounter an entirely different bottleneck.
What “tokens per kilowatt” leaves out
Tokens per kilowatt is an attractive metric because power has become a hard constraint. A data center cannot always acquire another megawatt merely by signing a purchase order. Grid connections, transformers, cooling equipment and construction move on different timelines from accelerator purchases.
For an operator limited by available power, producing more useful tokens from the same electrical envelope can be enormously valuable. It can mean serving more customers without waiting for a new facility. It can also reduce the energy expense associated with each unit of output.
But the phrase contains three hidden questions: which tokens, which kilowatt and under which user experience?
OpenAI says it normalizes results using each accelerator’s published chip power rating. It rates Jalapeño at 700 watts, while saying its measured sustained power was at or below 550 watts on the workloads tested. That is useful disclosure, but it illustrates why readers should distinguish a published package power figure from observed sustained chip power, and both from the electricity drawn by a complete rack.
At rack level, power includes more than the accelerator package. There are CPUs, memory, network equipment, storage, power conversion losses and cooling. Some of those costs are shared across a large amount of work. Others grow with traffic. A chip-level number can be technically correct and still fail to predict operating cost accurately.
Then there is utilization. An accelerator can look excellent in a controlled test where demand is steady and configurations are tuned. A production fleet may spend meaningful time underutilized because traffic changes by hour, customers demand strict latency, failures require spare capacity, and some models or regions cannot share the same hardware pool.
Quantization matters, too. Reducing the precision used to store weights and perform some calculations can lower memory traffic and increase the amount of model capacity that fits on a device. But the chosen format affects quality, accuracy, kernel behavior and sometimes the model’s willingness to produce a correct answer. A vendor comparison must state the quantization format and verify that output quality is comparable.
Speculative decoding matters. In this technique, a smaller draft model proposes several likely next tokens, and a larger model verifies them. It can improve effective output speed, but the gain depends on how often the draft is right and how the serving stack handles the extra work. It should be declared, not treated as background detail.
The same is true of multi-token prediction, prompt caching, prefix sharing, disaggregated prefill and decode, and any routing choice that changes the work performed per user-visible token.
A token is not a standardized unit of economic value. Ten fast tokens that answer a question correctly may be worth far more than one hundred cheap tokens that send an agent down the wrong path.
A vendor comparison worth trusting
The purpose of a benchmark should not be to crown a permanent winner. It should help a buyer decide whether a system fits a workload. That requires enough detail that another team can understand what was measured and what was not.
A credible comparison should begin with a workload profile:
| Measurement | Why it changes the outcome |
|---|---|
| Model and version | Architecture, size, attention pattern and expert routing change compute, memory and communication needs. |
| Quantization and accuracy checks | Lower precision can improve speed but may alter quality or task success. |
| Prompt and output distribution | An 8K prompt and 1K answer does not represent every product. Long prompts, short replies and multi-turn agents behave differently. |
| Concurrency and arrival pattern | Steady saturation traffic differs from unpredictable interactive demand. |
| TTFT, median TBT and tail TBT | Averages can conceal the pauses users actually notice. |
| Throughput at matched latency | Higher throughput only matters if the agreed responsiveness target is preserved. |
| Power boundary | Package rating, measured chip power, server draw, rack draw and facility power are different measurements. |
| Hardware and software configuration | Network topology, host CPUs, runtime version, kernels and scheduler policies can materially affect results. |
| Cache policy | Prefix caching, KV-cache compression, offload and reuse can determine long-context economics. |
| Cost and availability | Purchase price, cloud rate, depreciation, support, utilization and deployment timeline matter to production cost. |
The phrase matched latency deserves special attention. It is one of the most meaningful ways to compare systems, provided the match is real.
Suppose System A can produce 100,000 tokens per second only when average TBT rises to 80 milliseconds. System B produces 70,000 tokens per second while holding TBT to 20 milliseconds. If the product requires 20 milliseconds, System A’s 100,000-token result is not relevant. The fair question is how much throughput each system achieves while meeting the same TTFT and TBT targets, including reasonable tail percentiles.
Likewise, matching only TBT is incomplete if TTFT gets worse. A customer reading a long prompt may care deeply about first response time. An autocomplete product may care overwhelmingly about the next-token cadence. An offline document pipeline may not care about either, provided total cost per completed job falls.
This is why OpenAI’s result is more useful as a performance frontier than as a single score. The company says Jalapeño performed across operating points from high-throughput serving to highly interactive use, rather than only at a favored point. That is the right ambition. But buyers still need the underlying curves, the exact latency criteria, the full power boundary and comparable workloads to decide how much of that frontier applies to them.
The production-cost test
The final question is not whether Jalapeño can generate a token efficiently. It is whether it lowers the cost of a successful product outcome.
That is a wider calculation.
A low-latency system may enable an agent to take more useful steps before a person abandons the task. It may shorten the feedback loop for a developer and make a coding assistant materially more valuable. It may reduce the capacity needed to preserve responsiveness at peak demand. In these cases, latency has commercial value, not merely aesthetic value.
A high-throughput system may reduce the cost of batch processing, offline summarization or document analysis. In those cases, user-perceived speed may matter little. The winning design could be one that maximizes work per watt, accepts more queueing and runs jobs when energy is cheaper.
A system that is excellent at both can be unusually flexible. It may move resources between interactive and batched workloads as demand shifts. It may avoid the traditional need to maintain separate capacity pools, one reserved for premium responsiveness and another filled to the limit for economy.
That is the promise behind Jalapeño. Not a magical chip that wins every test, but a more balanced architecture that wastes less time shifting between the phases of real inference.
Still, production economics can diverge sharply from benchmark economics. A chip can have a strong performance-per-watt result and still be expensive to deploy if supply is limited, racks require unusual infrastructure, software integration is immature or utilization remains low. It can also be technically fast but poorly suited to a company’s model portfolio.
TechCrunch reported that OpenAI’s initial deployment is expected in very small volumes by the end of 2026, with more substantial deployment planned for 2027. That timing matters. By the time a system reaches broad deployment, competing hardware and software stacks may have advanced, and the relevant comparison may no longer be the one in an August benchmark.
The opposite is also true. A comparison that seems modest today can matter greatly if it gives a company control over its own roadmap. Owning more of the stack can let an AI provider tune models, kernels, memory allocation and network behavior together. It can reduce dependency on any one vendor and turn operational experience into a faster feedback loop for the next design.
That is not just a semiconductor story. It is a business-model story.
The new question buyers should ask
The old question was simple: which accelerator is fastest?
The newer and better question is: what is our inference system actually waiting for?
If the answer is matrix compute during long-prompt prefill, prioritize useful arithmetic throughput and efficient large-batch execution.
If the answer is weight and KV-cache movement during decode, prioritize memory bandwidth, memory capacity, locality and the software that manages cache state.
If the answer is synchronization across devices, inspect the interconnect, topology, collective communication behavior and model partitioning strategy.
If the answer is queueing, focus on scheduling, batching policy, admission control and the latency targets the product truly requires.
If the answer is idle capacity, the problem may not be hardware at all. It may be traffic variability, fleet fragmentation or a product design that makes pooling difficult.
Jalapeño matters because it makes that diagnosis harder to avoid. OpenAI is arguing that inference performance emerges from coordinated choices across the chip, memory, network and serving software. The company may ultimately prove right, partly right or overly optimistic about how broadly its advantage travels. But the framing is correct.
In the next era of AI infrastructure, the bottleneck will not be a property stamped onto a processor. It will be the slowest and most expensive handoff in the path between a person’s request and a useful result.
