The Chip Behind the Answer Is Becoming Part of the Product

For years, the visible contest in artificial intelligence has been about models: who has the largest, smartest, most capable system. The next contest may look less glamorous, but it will determine how widely those systems can be used. It is about serving a model efficiently, request by request, token by token.

Jalapeño is OpenAI’s first custom inference chip, developed with Broadcom. In August, OpenAI’s engineering team published Jalapeño’s first benchmark results, arguing that the system can deliver more AI work for each unit of power while returning answers with lower latency than the commercial systems in its comparison. The company reported gains across GPT-OSS 120B, DeepSeek R1 670B, and Kimi K2.5 1T on SemiAnalysis’ public InferenceX benchmark.

“Jalapeño can serve more AI work per unit of power, while also returning responses more quickly. It’s very efficient to serve a lot of customers, but it can also be very low latency.”

That is a notable claim, but it should not be read as a simple scoreboard declaring a permanent victor over Nvidia. Jalapeño is not yet broadly deployed, and the competitive hardware landscape will move before it is. TechCrunch reported that OpenAI expected small-volume deployment by the end of 2026, with more significant deployment in 2027.

The deeper story is what Jalapeño represents: inference hardware is being designed around the lived reality of an AI request, rather than around a peak compute number on a specification sheet. That change could decide whether future agents feel like patient interns, instant collaborators, or expensive background processes reserved for the biggest companies.

A useful answer travels through several machines

A generated answer begins with a prompt, but a serving system sees a chain of very different jobs.

First comes prefill. The system ingests the user’s prompt, attached documents, prior messages, tool outputs, and system instructions. This is a compute-heavy phase because the model must process a large block of tokens at once. Then comes decode, the iterative phase in which the model produces one new token, then uses that token to produce the next. Decode leans much more heavily on memory bandwidth because the system must repeatedly access model data and its accumulated conversational state.

That accumulated state is the KV cache, shorthand for key-value cache. It stores the model’s representation of earlier tokens so it does not need to recompute the entire conversation every time it writes a new word. For a long-running agent that reads files, calls tools, and continues reasoning, keeping that state close to the compute matters as much as raw arithmetic speed.

A conventional accelerator can be extremely good at a portion of this journey and still leave performance on the table across the whole journey. Compute cores might wait for memory. One chip might need to retrieve KV-cache state from another. Network transfers can impose delays that look tiny in isolation but grow painful when an agent makes dozens of sequential decisions.

OpenAI says Jalapeño was designed to reduce those waits by explicitly placing and retaining model state locally, then coordinating compute, memory, and networking differently for each phase. Its architecture treats the network as an integral part of the system, not a separate afterthought connecting otherwise independent processors.

This is why a chip announcement is really a systems announcement. A fast arithmetic unit cannot solve an inference bottleneck caused by cache movement, queueing, or a workload that is poorly batched. The useful unit is not a floating-point operation. It is the completed request that reaches a person at the right moment.

The numbers worth watching are not peak FLOPS

Peak FLOPS, or theoretical floating-point operations per second, remains a useful engineering specification. But it is an incomplete guide for someone deciding how much an AI product will cost to run.

An interactive agent experiences latency in layers. There is time to first token, which determines whether a response feels immediate. There is time between tokens, which determines whether text and tool feedback arrive fluidly. There is full end-to-end latency, which includes prompt processing, generation, coordination, and any time spent waiting on the system. Then there is throughput, the number of useful tokens or requests the system can handle in aggregate.

OpenAI’s presentation focuses on a compound question: how much throughput can a system sustain while meeting a chosen user experience, and how much power does it require to do it? That is more revealing than a peak number because a provider cannot simply delay every user’s answer, batch enormous volumes of work, and call the resulting throughput a consumer-quality service.

The company reported peak mixed throughput per kilowatt figures for three public models. The comparison uses the same unit for every value, so it is a more coherent view than a chart mixing latency, watts, and token counts.

For GPT-OSS 120B, OpenAI reported 85,448 mixed tokens per second per kilowatt for Jalapeño, compared with 44,960 for the comparison system. For DeepSeek R1, it reported 19,641 versus 11,781. For Kimi K2.5, it reported 18,195 versus 11,862.

Yet performance per kilowatt deserves careful reading. OpenAI normalized comparisons using published accelerator package power ratings. It lists Jalapeño at a 700 watt rating, while saying its measured sustained draw stayed at or below 550 watts in these workloads. The comparison ratings in the published results were 1,200 watts for GB200 on GPT-OSS and 1,400 watts for GB300 on DeepSeek and Kimi.

That methodology is meaningful, but it is also a reason to resist treating the outcome as a final statement about complete deployed systems. A data center pays for more than accelerator package power. It pays for CPUs, memory, networking, cooling, power conversion, idle capacity, and operational redundancy. Published chip ratings and actual rack energy are not identical measures. A benchmark using one nominal 8k-input, 1k-output shape does not encompass every real workload either.

The critical question is not whether Jalapeño won its test. It is how far the test lets us generalize. A model serving long contexts, a short chatbot exchange, a coding agent that pauses for tools, and an overnight batch job stress the system in different ways.

Why a faster token can change the product itself

The business impact of lower latency becomes clearest in agents. A chatbot can tolerate a brief pause before writing a paragraph. An agent that must plan, retrieve data, call a tool, inspect the result, and repeat may accumulate latency at every step.

Consider a procurement assistant checking a proposed supplier. It reads a contract, searches an internal policy database, asks a forecasting model for a demand estimate, identifies a missing clause, and drafts an escalation note. If each turn through the system feels slow, the workflow becomes a string of interruptions. If each stage is fast enough, it becomes something closer to a conversation with a skilled colleague who already has the relevant documents open.

Read contract

contract text • ⏱ latency compounds

Extract obligations

obligation list • ⏱ latency compounds

Call policy database

policy results • ⏱ latency compounds

Ask forecasting model

demand estimate • ⏱ latency compounds

Identify missing clause

missing clause • ⏱ latency compounds

Draft escalation note

Lower time between tokens and tool steps makes multi-step work feel continuous

This is the human side of token economics. A token is not only a unit on a bill. It can be a fraction of attention. Slow responses cause people to switch tasks, second-guess the tool, retry prompts, or abandon a workflow. Faster responses can make the same model feel more reliable, even if its underlying reasoning is unchanged.

OpenAI frames the economics in those broader terms. In its full-stack strategy statement, the company argues that model quality, serving software, chips, memory, networks, data centers, and products should reinforce one another, with efficiency translating into lower cost for useful work rather than merely cheaper computation.

That framing is strategically important. A company that owns only the model must adapt it to rented infrastructure. A company that understands its most common prompts, its model architecture, its traffic pattern, and its product latency targets can tune the layers together. It may decide that a premium system belongs on difficult frontier tasks while a more efficient system serves high-volume work.

Batch jobs and live agents need different machines

The right hardware depends on what “good” means for the workload.

For latency-sensitive agents, buyers should focus on time to first token, time between tokens, tail latency at busy periods, tool-loop latency, and predictable performance under concurrent use. A system that has excellent average throughput but occasional long stalls can make an agent feel erratic.

For batch inference, the priority can shift toward aggregate throughput, energy use, high utilization, and cost per completed task. A company generating summaries overnight can accept more batching and a slower individual response if it completes a large queue cheaply.

For both, utilization is central. Buying powerful hardware that spends much of the day idle is expensive, regardless of its peak benchmark score. The ideal system is not necessarily the fastest chip. It is the system that keeps expensive resources productive while preserving the service level users actually notice.

A practical total-cost calculation starts with more than tokens per dollar. It includes energy, accelerator time, infrastructure overhead, software engineering, retries, failed tasks, and the human time spent supervising the result. It should also account for whether a more capable model produces a correct answer in one attempt instead of three.

OpenAI makes a similar argument about useful intelligence per dollar, saying that better models, routing, context management, software, and hardware can all reduce wasted work. The phrase matters because it redirects attention from the price of a single output token to the full cost of achieving an outcome a person can act on.

The strategic pattern is bigger than one chip

Jalapeño is not evidence that general-purpose GPUs are suddenly obsolete. OpenAI itself says it will continue deploying Nvidia accelerators and systems from other partners for training and inference. The company’s strategy is a portfolio, not a clean break.

But custom inference silicon is becoming a strategic capability. TechCrunch reported that Anthropic confirmed it was building a custom silicon team, with the goal of co-designing models and hardware for faster, more efficient operation. That does not mean every leading AI lab will manufacture every component itself. It does suggest that the largest providers increasingly see hardware understanding as inseparable from product control.

For customers, the lesson is not to pick a side in a benchmark race. It is to demand a fuller explanation. Ask what workload was measured, how power was counted, whether the results include networking and host overhead, what latency target was held constant, how performance changes with context length, and how the system behaves at peak demand.

The future AI experience may be shaped less by the dramatic moment a model learns a new capability than by the quiet engineering that makes that capability available at conversational speed. Jalapeño’s real significance is that it treats every generated token as a full-stack event. If that design philosophy spreads, AI will become not only more powerful, but more present: faster to trust, cheaper to use, and better woven into the ordinary decisions that fill a working day.