A benchmark claim built around the whole system
SemiAnalysis said in an X post that co-design between vLLM, Inferact and NVIDIA had produced “7.2x” better performance per TCO than other AI chips. The post attributes the result to Wide Expert Parallelism, or WideEP, and to the architecture of the GB300 NVL72 backplane.
The claim describes more than a faster accelerator. It presents the GB300 NVL72 as a platform whose GPUs, memory, interconnect and serving software have been tuned to operate as one large inference machine. In practical terms, that could mean a user asking a model a question receives an answer more quickly, while a cloud provider serves more requests without adding equivalent hardware or electricity costs.
The comparison is significant because inference is becoming an increasingly large part of AI infrastructure spending. Training happens in concentrated bursts, but serving a model continues every time a customer opens a chatbot, an agent begins a task or an enterprise sends a document for analysis. Small efficiency gains can accumulate across millions of requests.
However, the X post does not itself provide the full benchmark context. It does not spell out the workload, competing systems, pricing model or exact definition of total cost of ownership. Those details are essential to judging whether the 7.2-fold figure represents a broad advantage or a narrower result on a workload particularly suited to the GB300 configuration.
What WideEP changes
The technical idea is aimed at mixture of experts models, which activate only a portion of their total parameters for each request. The model may be enormous, but an individual token generally needs only a subset of its experts. That design can reduce computation, although it also creates a difficult systems problem: the relevant expert weights must be available where and when they are needed.
NVIDIA explained in a technical blog post that WideEP shards mixture of experts weights across the GPUs in a GB300 NVL72 system. The company says this reduces the time spent loading expert weights and reports up to 2.7 times higher DeepSeek R1 throughput on the same hardware after software optimization.
That distinction matters. The reported 2.7 times throughput improvement is a same hardware comparison following software changes. The 7.2 times performance per TCO claim is a broader comparison against other AI chips. The two figures therefore describe different measurements and should not be treated as interchangeable.
The hardware arrangement is also central. A GB300 NVL72 system contains many accelerators connected through a high bandwidth fabric. WideEP is intended to use that internal communication structure instead of treating each GPU as an isolated device. The result is a more deliberate division of labor, with model experts distributed across the system and tokens routed to the locations where computation can happen efficiently.
Long context is the proving ground
The approach is especially relevant as AI applications handle longer prompts, documents and multi step tasks. Long context increases the amount of data that must move through the system, making memory access and communication just as important as raw mathematical throughput.
The NVIDIA and SGLang teams documented their GB300 work in a post focused on DeepSeek deployment. They described WideEP, DeepEP, prefill and decode disaggregation, and other optimizations for long context inference, while reporting workload specific gains.
Prefill processes the incoming prompt, while decode generates the response token by token. Separating those stages can allow each part of the workload to run on infrastructure suited to its behavior. For an enterprise assistant reading a large contract, or an agent maintaining a long history of actions, that separation could influence both responsiveness and operating cost.
The broader lesson is that model serving increasingly resembles an industrial scheduling problem. The system must decide where weights live, how requests are grouped, when data crosses the interconnect and which machines handle different stages of generation. A chip with strong standalone specifications may lose to a less obvious configuration if the surrounding software moves information more effectively.
Verification remains important
SemiAnalysis’s InferenceX benchmark platform says it compares realistic agentic inference across accelerator systems using reproducible GitHub Actions workflows, with public recipes, logs and artifacts. That framework could make claims like this easier to inspect, provided the relevant GB300 result, workload configuration and cost assumptions are available in sufficient detail.
The reactions supplied for this report were supportive. Inferact said its team had been optimizing vLLM on GB300 and argued that the resulting gains translate directly into better TCO, directing readers to the current InferenceX results. vLLM contributor Woosuk Kwon recommended vLLM for the referenced DSV41 workload, another positive response to the software stack connected with the result.
No opposing response was found in the supplied material. That means the claim has received support from participants and advocates associated with the optimization, but not yet a documented independent challenge or confirmation in these sources.
For buyers, the decision ahead will not be based only on accelerator counts. They will need to examine software maturity, model compatibility, networking behavior, utilization and the engineering cost of moving workloads. If the result survives broader testing, AI infrastructure may increasingly be purchased as a coordinated system rather than as a collection of chips.
Maya Lindqvist is not a person. No notebook, no deadlines, no face behind the name — just a byline this newsroom publishes under. Here is the production line underneath it, because a name beside a portrait reads like a journalist, and this one is not one.
The models. Writing: gpt-5.6-luna and qwen3-max. Out on the live web: gpt-5.6-luna and gpt-5.6-terra. Pictures: gpt-image-1 and gpt-image-1-mini. Swap one in the newsroom and this line swaps with it — it is read off the machines, not typed here.
How a story is made
- Research. The searching model reads around the story, pointed at primary sources — the filing, the post, the repository — rather than at somebody else's write-up of them.
- Writing. The writing model drafts it against what was found, at Maya Lindqvist's usual length and in Maya Lindqvist's usual register.
- The loop. A reviewer reads the draft and sends it back with notes. Then reads it again. A piece can go round several times before it leaves the building.
- Enrichment. A quotation has to appear word for word on the page it is taken from. A chart may only use figures that appear in the source it cites. Whatever fails is dropped, and the reason is kept.
- Fact check. A last pass hunts for claims the article makes and its sources do not.
- A human stop. Sensitive subjects are held for a person to read before publication, and a person can kill any of it at any point.
If that sounds less like a newsroom and more like a factory: quite. It is called Press Factory.
This article was generated using AI and published automatically without human pre-publication review.
How this article was made
The article was produced by the Grandmonts Media News Engine using automated research, drafting and verification workflows. No human editor reviewed the article before publication. Grandmonts Media remains responsible for the published content. Errors can be reported at office@grandmonts.cz.