The 80 GB Illusion: What gpt-oss Reveals About MoE Memory

A local model can look improbably small when its headline says “5.1B active parameters per token.” OpenAI’s gpt-oss-120b is the useful corrective. It activates a small fraction of its 116.8 billion parameters for each token, yet a single GPU still needs room for a 60.8 GiB checkpoint, working memory, and an ever growing cache of the conversation. The result is not a contradiction. It is a lesson in how easily capacity, computation, bandwidth, latency and context are collapsed into one misleading word: size.

The person deploying a local model rarely begins with a theoretical question about neural network architecture. They begin with a practical one.

There is an 80 GB GPU in a rack, or perhaps a workstation budget that can stretch to one. A new open weight model claims strong reasoning performance. Its headline says it has 116.8 billion total parameters, but activates only about 5.1 billion per token. The model card says it can run on one 80 GB GPU.

That sounds like a loophole in the economics of AI.

It is not. It is a carefully engineered compromise, and OpenAI’s gpt-oss-120b makes the compromise unusually visible. The model is a Mixture of Experts, or MoE, transformer with 36 layers, 128 experts in each MoE block, and a router that selects four experts for each token. Its expert weights are quantised in MXFP4, while its attention design uses grouped query attention to limit cache growth. It supports a context length of 131,072 tokens.

Every one of those choices attacks a different bill.

MoE reduces the computation needed for a token because most expert networks sit idle for that token. Quantisation reduces the storage required for model weights. Grouped query attention reduces the amount of key and value state stored for every prior token. Local attention windows reduce some attention work. But none of these is a universal discount.

The most important habit for anyone assessing local inference is to stop asking whether a model “fits.” Ask five separate questions instead:

1. Can the weights fit in GPU memory?
2. How much computation is active for each token?
3. How large is the checkpoint on disk and in memory?
4. How much memory traffic is needed to generate each new token?
5. How much key value cache will the context and concurrency target consume?

gpt-oss-120b is a compact case study because all five answers are dramatically different.

The number that matters depends on the moment

OpenAI calls the larger model gpt-oss-120b, but its exact total parameter count is 116.83 billion. Of those, 114.71 billion are in the MoE MLP blocks, 0.96 billion are in attention, and 1.16 billion belong to embedding and unembedding components. OpenAI lists 5.13 billion active parameters per token and a 60.8 GiB checkpoint size.

Those numbers describe different things. Treating them as interchangeable is how a deployment plan goes wrong.

Total parameters tell you how much learned capacity the model contains. They matter for quality, checkpoint storage and, in a conventional single GPU deployment, the amount of weight memory that must remain available. A router may select only four experts for the current token, but it could select any four experts for the next token. Unless the deployment accepts costly offloading or sharding, the complete collection of experts must be locally reachable.

Active parameters describe the subset of weights involved in a forward pass for one token. This is the number closest to compute cost. It helps explain why a 116.8B MoE can perform nearer to a much smaller dense model in arithmetic per generated token.

Checkpoint size is the practical size of the released weights after their storage format is taken into account. It is the opening entry in a GPU memory budget, not the final one.

Memory traffic is the amount of data the accelerator has to read and write while generating tokens. A model can fit perfectly and still feel slow if each token requires streaming large quantities of weights or reading a long cache through a bandwidth bottleneck.

KV cache is runtime state. It grows with prompt length, generated tokens and concurrent requests. It does not care that the model activates only four experts. It belongs primarily to attention.

A useful analogy is a hospital. Total parameters are the whole building and every specialist employed there. Active parameters are the team treating one patient. Checkpoint size is the floor space required to keep the hospital standing. Memory traffic is the movement of records, equipment and people through the corridors. The KV cache is every patient’s growing medical file. Hiring only four specialists for a particular case does not make the records disappear.

This distinction matters especially because gpt-oss-120b’s impressive “5.1B active” figure is true, but incomplete. It tells an engineer why the model can reduce active compute. It does not tell them how much VRAM remains after loading the model, how many long conversations can coexist, or what latency to expect after a 100,000 token prompt.

What the router actually does

Mixture of Experts sounds mystical until it is drawn as a traffic system.

At the start of an MoE block, a token is represented by a vector in the model’s residual stream. For gpt-oss, that residual stream has width 2,880. A small router projection looks at that vector and produces one score, or logit, for every available expert. In gpt-oss-120b, that means 128 router logits.

The router does not send the token through all 128 experts. It identifies the four highest scoring experts. It applies a softmax only across those selected four scores, turning them into weights that sum to one. Each chosen expert processes the token through its own gated SwiGLU MLP. The model multiplies each expert output by its router weight, adds the four outputs together, and returns the result to the residual stream.

In shorthand:

router logits = Router(token state)
selected experts = top 4 router logits
router weights = softmax(selected logits)
MoE output = sum(router weight × selected expert output)

The important detail is that the model is sparse in its expert MLP computation, not sparse in every operation. Attention still runs. Normalisation still runs. Routing still runs. The embedding and output components still run. And the four selected experts still need their weights fetched and applied.

This is why the model’s parameter marketing can be both accurate and easy to misread.

For any individual token, only four of 128 experts are executed in each MoE block. That is only 3.125 percent of the available experts. Yet the deployment cannot simply retain 3.125 percent of the MLP weights. The router’s choices vary by token, layer and input. A code completion may favour a different collection of experts from a legal summary, a Japanese translation or a mathematical proof. A production inference system needs the option to call any expert that the router selects.

There are ways to split that cost. In a multi GPU system, experts can be distributed across devices. In a CPU offload setup, unused experts may live in system memory and be transferred on demand. But those alternatives turn a capacity problem into a communication and latency problem. The whole point of OpenAI’s “single 80 GB GPU” framing is that, with the released quantisation, the full model checkpoint can remain on the accelerator.

The router saves arithmetic. It does not make the model’s learned knowledge free to store.

Four bit weights are a capacity tool, not a complete deployment plan

The central trick that makes gpt-oss-120b practical on a single large GPU is quantisation.

OpenAI says the MoE weights are post trained in MXFP4, a format using 4.25 bits per parameter. The MoE weights account for more than 90 percent of the total parameter count. OpenAI lists the full gpt-oss-120b checkpoint at 60.8 GiB and says this quantisation enables the model to fit in 80 GB of memory.

Without quantisation, the arithmetic gets forbidding quickly. A 116.83B parameter checkpoint represented uniformly in 16 bit precision would require roughly 218 GiB before allowing for any runtime memory. At fourish bits, the largest share of the model’s weights becomes far more manageable. The released checkpoint is still substantial, but it becomes plausible on a high memory accelerator.

That is a victory over weight capacity.

It is not automatically a victory over latency.

During generation, inference has two broad phases. First comes prefill, where the system processes the prompt tokens and constructs attention state. Then comes decode, where the system generates one new token at a time. In the decode phase, the GPU repeatedly consults the model weights and the accumulated key value cache. If the workload has a small batch, decode often becomes constrained by moving data rather than by peak mathematical throughput.

An MoE model adds another nuance. Each token runs only four experts, so it avoids calculating through all 128. But it must still access the selected experts efficiently. A strong GPU kernel, good expert packing and a stable routing pattern can help. Poor memory layout, expert sharding over slow links, or offloading experts across PCIe can erase much of the apparent efficiency.

Quantisation can help here too because fewer bits mean fewer bytes moved. But the implementation matters. Low precision weights may need scaling metadata, specialised kernels and dequantisation steps. A nominal bit count is not a measured tokens per second result.

That is why “this model fits on my card” is only the beginning of benchmarking. The real question is whether it fits with enough margin to run the expected context length, sampling configuration and number of simultaneous users, while still meeting the response time the product promises.

The cache bill hidden behind 131,072 tokens

The most revealing number in the gpt-oss model card may not be 116.8B or 5.1B. It may be 131,072.

That is the native context length OpenAI specifies for the dense attention layers, extended with YaRN positional scaling. The model has 64 query heads of dimension 64, but only eight key value heads because it uses grouped query attention with a group size of eight. Its attention layers alternate between dense attention and locally banded attention with a window of 128 tokens.

Grouped query attention is an important memory saving measure. In ordinary multi head attention, each query head typically has its own key and value heads. Here, eight query heads share each key value head. The model retains 64 query heads for expressive querying but stores only eight key value head pairs.

That changes the cache calculation dramatically.

For one token in one layer, the key value cache stores:

8 KV heads
× 64 values per head
× 2 tensors, one key and one value
× 2 bytes per value in 16 bit storage
= 2,048 bytes

gpt-oss-120b has 36 layers:

2,048 bytes per token per layer
× 36 layers
= 73,728 bytes per token
= 72 KiB per token

At the full 131,072 token context length:

72 KiB per token
× 131,072 tokens
= 9 GiB of KV cache for one sequence

This estimate assumes 16 bit cache storage and uses the dimensions disclosed in OpenAI’s model card. It is a model architecture calculation, not a promise that every serving engine will reserve exactly that amount. Engines differ in cache precision, allocation strategy, paging, padding, metadata and whether they use compression. But the order of magnitude is the lesson: full context is not free.

The cache has a brutal property that weight memory does not: it multiplies.

A single 131k token conversation needs roughly 9 GiB under this simple 16 bit estimate. Four such conversations need roughly 36 GiB. Eight would need roughly 72 GiB, before considering the 60.8 GiB checkpoint, temporary buffers and the runtime’s own overhead. That is impossible on a single 80 GB accelerator without cache compression, shorter contexts, sharding, swapping or a much more constrained concurrency arrangement.

The practical inference is straightforward. The model can support a maximum context of 131,072 tokens. That does not mean a particular one GPU deployment can support that maximum context at the same time as high concurrency.

Context length is a capability ceiling. It is not a free default setting.

There is another wrinkle. gpt-oss alternates dense attention layers with locally banded sparse layers. The local layers look only within a 128 token neighbourhood, reducing their attention computation relative to full dense attention. But the dense layers still need access to the long history. A serving engine may be able to optimise some local layer cache handling, yet the full context remains necessary wherever dense attention needs it. The local window does not turn a 131k conversation into a 128 token memory problem.

Why a model that fits can still be slow

Memory capacity asks whether the model can be loaded. Memory bandwidth asks whether it can answer quickly.

At short context, much of the decode cost can be understood as repeatedly reading selected weights, executing the active parts of the model and producing the next token. MoE is attractive because it dramatically lowers the number of expert parameters used in that calculation. OpenAI reports 5.13B active parameters per forward pass rather than 116.83B total.

At long context, attention imposes another repeated task. For a new token, each dense attention layer compares its query with prior keys and uses prior values to form an output. The system must read the relevant cache. The longer the existing sequence, the more cache traffic is involved in generating each successive token.

This is why a long prompt can produce a strange user experience. The model loads successfully. The first response begins after a large prefill delay. Then token generation begins at a reasonable pace, only to slow as the conversation becomes enormous or multiple long sessions compete for cache pages. The bottleneck has moved.

The MoE model has not failed. The operator has simply encountered a different bill.

An efficient local deployment therefore needs a workload policy, not merely a model choice. That policy should specify:

• A default context limit, lower than the model’s theoretical maximum.
• A maximum number of concurrent sequences at each context tier.
• A cache data type and a tested quality threshold for it.
• A batching policy that improves throughput without making interactive latency unacceptable.
• A routing and expert execution implementation suitable for the chosen GPU.
• A reserve for temporary tensors, graph capture, allocator fragmentation and server overhead.

The word “reserve” matters. A deployment built to use every available byte is not a deployment