AI Inference Is Becoming a Software Problem Before It Is a Chip Problem

For the person waiting on an AI assistant to finish a proposal, write code, or answer a customer, the make of accelerator in a data center is invisible. The pause is not. That gap between silicon and experience is where the new contest in AI is being fought: in the inference layer that decides how a model uses compute, memory, networks and power.

A few years ago, buying the best available GPU looked like the clearest route to better AI. That logic still matters. Faster chips, more memory and better networking remain powerful advantages. But it is no longer enough to explain why one system delivers a response in a second while another takes ten, or why two operators running the same model face radically different costs.

The decisive question is increasingly not simply, “Which chip did you buy?” It is, “How well does your software turn that chip into useful answers?”

TechCrunch reported that ZML released LLMD to run large language models across Nvidia, AMD, Google TPU, Apple Metal and Intel Arc hardware. Its premise is disruptive precisely because it is mundane: a capable runtime can make existing hardware work much harder.

That is also the bet behind Kog’s low level GPU work, Microsoft’s Maia 200 and OpenAI’s Jalapeño. They point to a world in which inference, the act of generating an answer from a trained model, becomes a systems discipline. Models, compilers, kernels, memory placement, queues and networks must operate as one machine.

One prompt, two very different jobs

A chatbot response looks like a single action. Under the hood, it is usually two different kinds of work.

First comes prefill. The system reads the user’s prompt, perhaps a document, a conversation history and retrieved company data. It processes many input tokens at once and builds a working memory known as the KV cache. This stage tends to reward large, efficient compute operations.

Then comes decode. The model produces the answer one token at a time. Each new token depends on those already generated, while repeatedly consulting the KV cache. Decode is a more serial process, which often becomes constrained less by raw arithmetic and more by memory bandwidth, cache movement and delay between machines.

This distinction explains why headline chip numbers can confuse buyers. A system that excels at prefill can ingest giant documents quickly but still feel slow once it starts writing. Another can stream tokens at extraordinary speed for one user but deliver disappointing overall capacity when thousands arrive at once.

The best chip for an enterprise search product may not be the best chip for an autonomous coding agent. The first may require high throughput over long prompts. The second may make a developer painfully aware of every fraction of a second between generated tokens. A consumer assistant may need both, while also absorbing unpredictable peaks in demand.

The scheduler is now part of the product

The inference runtime is the traffic controller. It chooses which requests run together, where a model is placed, when a request should be split, what precision to use, whether a cache can be reused and how work moves among accelerators.

Batching illustrates the tradeoff. Putting many requests in one batch lets a system use the chip more fully. That improves throughput and usually lowers cost per token. But a request may wait in the queue for other work to arrive. For an interactive user, that wait can be more damaging than a lower aggregate capacity figure.

Continuous batching softens the problem by adding and removing requests while a batch is running. Smarter scheduling goes further. It groups compatible request lengths, separates prompt processing from token generation, reserves capacity for urgent requests and keeps cache data close to the accelerator that will need it.

This is why tokens per second is necessary but insufficient. There are at least three versions of the number:

• Per request tokens per second, which affects how quickly one response is written.
• Aggregate tokens per second, which measures the total output of a fleet.
• Tokens per user, which asks how much performance remains available when many people are active at once.

Kog has made the most provocative version of the single request argument. TechCrunch reported that Kog’s demonstration reached 3,000 per request tokens per second on its purpose built 2 billion parameter Laneformer model, while the company acknowledged that reproducing such gains on larger language models is the challenge that matters.

That caveat is important. A spectacular result on a small, specialized model is evidence of technical potential, not proof of production economics for a frontier model serving millions of varied requests.

Microsoft’s hardware announcement makes the complementary point. TechCrunch reported that Microsoft’s Maia 200 was specified at more than 10 petaflops at 4 bit precision and about 5 petaflops at 8 bit precision. Those are meaningful compute figures, but they cannot by themselves reveal an application’s user latency, cache behavior or cost per successful task.

Why portability leaves performance on the table

A hardware agnostic runtime promises freedom. It can let a cloud operator shift work among Nvidia GPUs, AMD GPUs, TPUs, Apple devices and newer accelerators as prices, availability or power constraints change. It weakens lock in and gives emerging chipmakers a route into real deployments.

But portability has a price because chips do not merely differ in speed. They differ in memory hierarchy, supported number formats, compiler toolchains, interconnects and the obscure behavior of their instruction sets. The fastest kernel for one accelerator can be mediocre on another. Even the optimal batch size can change.

A universal runtime therefore faces an uncomfortable choice. It can expose a common denominator that runs almost everywhere, or it can build specialized paths for each target. The first is easier to maintain. The second is where much of the value lies.

Kog’s approach makes the tradeoff unusually explicit. Its engineers reportedly spend weeks or months investigating each new GPU at a low level. That is costly and difficult to scale, but it reflects a hard truth: peak performance comes from knowing the peculiarities of the hardware, not pretending they do not exist.

The strategic answer may be a layered architecture. Keep the model API, routing policy and observability portable. Then allow chip specific kernels, memory allocators and collective communication libraries underneath. This preserves negotiating power without demanding that every workload accept average performance.

Measure useful output, not impressive output

A serious inference review starts with a user journey, not a vendor benchmark. Measure time to first token, then steady generation speed, then the time to complete a useful answer. Break those figures down by prompt size, output size, model, concurrent users and percentile. Average latency can hide a miserable experience for users caught in the busiest moments.

Then inspect utilization. Low accelerator utilization may mean insufficient batching, poor request mixing, or a bottleneck outside the chip. High utilization is not automatically a victory if queues are growing and first token latency is deteriorating.

Memory bandwidth is especially revealing during decode. If compute units are waiting while weights or KV cache data moves, another peak FLOPS number will not solve the problem. If requests span multiple machines, inspect network time and cache transfers. If the fleet appears busy but produces little, inspect scheduling overhead, failed retries and fragmentation in available memory.

Finally, tie energy to output. Watts per useful output is more honest than watts per chip. It asks how much energy was needed to produce a completed answer that a user did not immediately discard, regenerate or correct. OpenAI makes a similar argument in its account of the full stack behind Jalapeño, framing throughput, latency, energy efficiency and cost as system properties rather than isolated chip attributes.

Custom silicon changes the bargaining power

OpenAI’s Jalapeño is the clearest expression of this integrated strategy. TechCrunch reported that OpenAI’s benchmarks for Jalapeño highlighted both tokens per user and throughput per kilowatt, with the design focused on reducing prefill and communication delays and keeping KV cache state local.

The lesson is not that every AI company must build a chip. Most should not. Designing silicon is slow, expensive and risky. The lesson is that infrastructure advantage is accumulating at the seams between layers. A company that owns the model, runtime, memory strategy and network can optimize the whole loop. A company that rents every layer must work harder to keep its options open.

For buyers, that changes the purchasing checklist. Ask which bottleneck dominates your workload. Ask whether peak throughput survives at your request mix. Ask how much performance is lost when you move to another cloud or accelerator. Ask whether the runtime exposes cache, batching and queueing controls, or merely reports a flattering aggregate token count.

The fastest accelerator still matters. But the increasingly valuable capability is the system that knows when not to use it, how to feed it and how to keep every watt, byte and millisecond pointed toward work a person actually values.