Gemma 4 Turns Speculative Decoding Into a Practical Local AI Decision

For people running language models on a laptop, workstation, or private server, the uncomfortable truth is that a model can be smart enough and still feel too slow to use. Gemma 4’s multi-token-prediction drafters matter because they attack that everyday frustration directly. They make speculative decoding less like an obscure serving optimization and more like a design choice that can determine whether a local assistant feels conversational, whether a coding tool interrupts flow, and whether a production team needs another GPU.

A person using a local model rarely thinks in tokens. They think in pauses.

There is the pause after pressing Enter in a coding assistant. The pause while a customer support tool constructs a reply. The pause after dictating a sentence, waiting to see whether the machine understood what was said. A model can eventually produce a useful answer, yet still create the impression that it is thinking too slowly to be trusted with fast work.

That gap between capability and responsiveness is becoming one of the central practical problems in AI deployment. It is particularly acute for local systems. Cloud providers can throw large pools of accelerators, sophisticated batching, and carefully tuned serving stacks at the problem. A team running a model on a single workstation, or inside a private environment where data cannot leave the building, has fewer levers.

Google DeepMind’s Gemma 4 family was designed for local and edge deployment, spanning smaller efficient models and larger models aimed at personal computers and consumer GPUs. In May 2026, the company added multi-token-prediction, or MTP, drafters designed to accelerate Gemma 4 generation through speculative decoding. Ars Technica reported that Google claimed up to a threefold speed improvement in favorable cases, while stressing the role of lightweight drafters and shared active context.

The important phrase is “in favorable cases.”

Speculative decoding is not a button that turns every model into a faster model. It is a wager. A small fast system proposes several likely next tokens. The large accurate system checks that wager. If the proposal is good, several tokens can be committed with work that would otherwise have advanced the answer by only one token. If the proposal is poor, the verifier must reject part of it, repair the sequence, and the supposed shortcut becomes extra work.

That makes speculation an engineering question, not a slogan. The right question is not, “Does my server support it?” The right question is, “Will this particular model, on this hardware, serving this workload, produce enough accepted tokens to justify the cost of drafting and checking them?”

The slow rhythm of ordinary generation

A conventional decoder-only language model writes in a strict sequence. Given a prompt, it predicts one next token. That token is appended to the context. The model then predicts the next token using the prompt plus everything it has already generated. It repeats until it reaches a stop condition.

This is called autoregressive generation because each step depends on the output of the prior step.

That dependency is why a model cannot simply generate a whole paragraph in parallel in the way a spreadsheet recalculates cells. It has to decide whether the next item is “The,” then whether the following token is “answer,” then whether the next is “is,” and so on. Even very obvious connective words require a full decode step.

The phrase “full decode step” can sound more expensive than it is. Modern accelerators perform enormous amounts of arithmetic quickly. Yet generation is often constrained less by arithmetic than by moving model weights and cached context through memory.

At each token, the system needs to access the model’s parameters. For a large model, those parameters may be many gigabytes. On a high end accelerator, fast memory helps. On a consumer GPU, an integrated laptop chip, or a CPU running a quantized model, memory bandwidth is often the bottleneck. The compute units spend part of their time waiting for data.

This distinction matters because speculative decoding does not make the target model less intelligent or smaller. It tries to make every costly visit to the target model accomplish more useful output.

A useful analogy is a busy customs checkpoint. Ordinary generation lets one traveler approach the desk at a time. Speculative decoding asks a fast assistant to line up a small group whose paperwork is probably correct, then gives the senior officer enough information to inspect the group together. When the paperwork is consistent, the line moves faster. When it is not, the officer has to stop, correct the first error, and rebuild the group from there.

The senior officer is still in charge. That is the key to understanding quality.

A drafter proposes, a target decides

Speculative decoding splits generation into two roles.

The drafter is inexpensive and fast. It predicts a short run of future tokens. The target is the main model that defines the answer quality the user actually wants. It evaluates the proposed run and determines which tokens can be accepted.

Suppose the target has already generated:

“The deployment should be measured by”

A drafter may propose the next four tokens:

“latency, quality, and cost”

Instead of asking the large target model to generate “latency,” then “quality,” then “and,” then “cost” through four separate decode steps, the serving system provides the target with that candidate sequence as input. Because the candidate tokens are now known inputs, the target can process them in a more parallel fashion than it could have generated them serially.

If the target agrees with the proposal, the system accepts several tokens at once. If it accepts the first two but not the third, it retains the accepted prefix, samples or selects a target-approved correction at the rejection point, and begins a new speculative cycle.

There are two ideas hidden inside this apparently simple design.

First, verification is not the same as repeating the target model’s ordinary token-by-token decode loop. If it were, no speedup would be possible. The target evaluates a block of proposed tokens in a way that exposes its assessment of each position in the block. The serial dependency has been shifted into the drafter’s cheaper work, while the costly target is used to validate a bundle.

Second, acceptance is not necessarily literal agreement with one deterministic answer. In sampled generation, the target and drafter each define probability distributions over the next token. Correct speculative sampling uses an acceptance rule and a correction procedure so that the final output follows the target model’s distribution, not the drafter’s distribution. The drafter influences how efficiently the output is reached, but it does not get authority to change what the target distribution says.

This is why “no quality degradation” can be a technically meaningful claim. Proper speculative decoding can be distributionally equivalent to target-only sampling. In plain language, it can produce answers drawn from the same probability law the target would have used without speculation.

That statement has conditions. The serving implementation must perform the acceptance and correction logic correctly. Its sampling configuration must be supported. Features such as temperature, top-p filtering, repetition penalties, constrained decoding, tool grammars, stop rules, adapters, and custom logits processors may complicate equivalence. Some systems will deliberately use an approximate variant because it is easier or faster. The safe operational rule is simple: treat equivalence as something to validate in your own stack, not as a magic property of the word “speculative.”

Why MTP changes the tradeoff

Traditional speculative decoding often uses a separate small language model as the drafter. A 1 billion parameter model might propose a few tokens for a 7 billion or 30 billion parameter target. This can work very well, especially when the small model is trained to imitate the target.

But it creates practical burdens.

The separate model needs its own weights, runtime integration, tokenizer compatibility, and often its own key-value cache. It has to be deployed, monitored, upgraded, and matched to the target. It may also consume scarce memory that a local deployment would rather reserve for the main model or longer context.

MTP takes another route. Rather than relying only on a separately deployed miniature model, it equips the system with drafter components designed to predict multiple future positions. The core idea is that one representation of the current context can feed several prediction heads, each trained to estimate a different future offset.

One head predicts the immediate next token. Another predicts the token after that. Others reach further ahead. The model is no longer asking one head to simulate an entire future one token at a time. It is using parallel guesses about several positions.

That does not eliminate uncertainty. A future token depends on the tokens before it, which have not yet been committed. The farther the head looks ahead, the harder its task becomes. But natural language and code contain enough local regularity that these predictions can often be useful. Boilerplate code, punctuation, common phrases, JSON syntax, conventional API calls, and predictable formatting are fertile ground.

Google DeepMind has positioned Gemma 4’s MTP drafters as an inference acceleration feature for the model family, rather than leaving teams to find and operate a separate draft model themselves. The Gemma 4 page describes the broader family as intended for efficient local and edge deployment, from smaller edge variants to larger workstation-oriented models.

The distinction may sound architectural, but its business impact is straightforward. A local deployment that needs another full model in memory to speculate may have no spare memory. A drafter that can reuse the target’s already computed context has a chance to make speculation viable in places where a separate draft model is too cumbersome.

The cache is the hidden protagonist

Every serious discussion of speculative decoding eventually reaches the key-value cache, often abbreviated as the KV cache.

A transformer does not want to reread and fully recompute every prior token every time it writes one new token. During generation, it stores intermediate attention information for previous tokens. That stored context is the KV cache. It is the model’s working memory for the current request.

The cache makes long conversations and long documents practical. It also creates a large and growing memory object. For multiple simultaneous requests, cache management can become as important as model weights.

A separate draft model often needs a cache of its own because its internal layers differ from those of the target. That means two copies of active context, two systems advancing state, and potential synchronization work after each accepted or rejected speculative block.

An MTP design can be more elegant because its drafting mechanism is closer to the target’s own representation of the context. Ars Technica reported that Gemma 4’s drafters share the key-value cache, avoiding the need to recalculate context that the main model has already processed.

Sharing is not merely a convenience. It changes the costs that determine whether speculation wins.

For a local model, a small additional set of drafter weights may be acceptable. Duplicating a large cache may not be. For long-context tasks, the cache can become substantial. For concurrent serving, every active request has its own cache pages, so an architectural choice that looks minor in a single benchmark can determine how many sessions fit on a machine.

Still, cache sharing should not be reduced to a universal rule. It depends on the actual implementation. Model architecture, attention layout, quantization method, runtime, and cache format all matter. A team should confirm what is actually shared, what is copied, and what must be recomputed following a rejection. A product label that says “MTP” is not a substitute for a memory trace.

The math that matters is accepted length

The headline metric for speculative decoding is often acceptance rate. It is important, but incomplete.

Imagine a drafter proposes four tokens per round. In one workload, the target accepts an average of 3.4 tokens from each proposal. In another, it accepts 1.1. The first workload may see real acceleration. The second is living dangerously close to the cost of doing normal generation plus unnecessary drafting and verification.

The more useful metric is accepted tokens per speculative round, sometimes called accepted length.

If a proposal length is (k), and the expected number of accepted drafted tokens is (a), then the system is trying to obtain roughly (a), plus any target-generated correction token where applicable, for one target verification pass. The exact accounting depends on the algorithm and runtime, but the intuition is stable: long accepted prefixes are the source of the gain.

A rough latency model is:

speculative time per committed token = drafter time per round + target verification time per round + rejection overhead divided by expected committed tokens per round

Ordinary decoding is simpler:

ordinary time per committed token = target decode time per token

The speculative system wins only when its extra work, divided across enough committed tokens, is less than one ordinary target step.

This reveals why proposal length has a sweet spot.

A proposal of one token does not create much opportunity for parallel target verification. A proposal of 16 tokens might create enormous opportunity on paper, but prediction accuracy typically falls with distance. It can also increase buffer work, target verification cost, and the amount of discarded computation after an early rejection.

The best proposal length differs by model and workload. A coding assistant filling a familiar function may tolerate a longer proposal because syntax and conventions are predictable. A creative writing prompt, multilingual conversation, complex reasoning trace, or tool-using agent may have less predictable next-token behavior. The same target model can therefore benefit substantially in one setting and barely at all in another.

Low acceptance is not a mild disappointment. It can be worse than no speculation.

The system has spent time running the drafter, allocating proposal buffers, entering verification kernels, and perhaps rearranging cache state. If the target repeatedly rejects the first or second token, the target remains responsible for almost every meaningful decision while carrying the overhead of the scheme meant to help it.

A robust serving stack should therefore have an exit ramp. It should reduce proposal length, switch to ordinary decoding, or choose a different drafter when recent acceptance deteriorates. Static configuration is easy. Adaptive control is usually more valuable.

Latency is more than tokens per second

Tokens per second is useful because it measures the visible pace of a response after it begins. It is not the whole user experience.

For an interactive product, measure at least four things:

1. Time to first token: How long passes between the request and the first visible output?

2. Decode throughput: How quickly tokens arrive after the response starts.

3. Inter-token latency: Whether output arrives smoothly or in distracting bursts.

4. End-to-end task latency: How long it takes to complete the actual user action, including prompt processing, retrieval, tool calls, structured output checks, and network overhead where relevant.

Speculative decoding has its clearest impact on the decode phase. It may help time to first token a little, especially if the implementation starts drafting efficiently, but it does not erase a long prompt-prefill stage. A user who submits a 100,000-token document may be waiting mostly for prompt processing, not answer generation.

The experience can also become burstier. Ordinary decoding emits a token each step. Speculation may commit several tokens at a time. For a terminal or chat interface, that can look pleasantly fast. For streamed speech synthesis, live captions, or tightly synchronized interfaces, the application may need buffering rules that preserve steady delivery.

This is why a dashboard should separate prefill from decode and should report percentiles, not only averages. A mean speedup can hide a worst case that users notice every day.

Batching turns a local trick into a serving problem

A single user on a laptop is one environment. A production server handling hundreds of requests is another.

In ordinary batched generation, requests advance together in a scheduler. Some are producing their first token. Some are continuing a long answer. Some finish early. The serving system must keep the accelerator busy while managing different sequence lengths and cache pages.

Speculative decoding adds uneven proposal outcomes. One request may accept four drafted tokens. Another may reject at the first position. One may need a correction sample. Another may be constrained by a JSON grammar. The scheduler has more state to manage and more variation in per-request work.

At low concurrency, speculative decoding can be especially attractive because a large target model may be memory-bound and underutilized. A drafter uses otherwise idle compute resources to propose work, then the target validates a larger block. This is the setting where the basic story is most compelling.

At high concurrency, the picture becomes less obvious. Ordinary batching already improves hardware utilization because the accelerator processes many sequences together. The target model may no longer have much idle compute to exploit. Verification can still help, but the incremental gain may shrink, while the scheduler complexity grows.

There is also a fairness question. If one request generates speculative blocks efficiently while another is repeatedly rejected, a naive scheduler can give uneven latency. Production systems need policies that protect tail latency and avoid letting expensive repair cycles crowd out short, simple requests.

The practical lesson is not that speculation belongs only on laptops. It is that local and low-concurrency deployments often have a stronger first case for it. At larger scale, it must compete with the benefits of conventional batching, continuous batching, cache paging, prompt caching, model parallelism, and routing across model sizes.

Choosing a separate drafter, MTP, or nothing

There is no universal winner. The decision should follow the deployment constraint.

Choose a separate draft model when the target is large, the workload is stable and predictable, and there is enough memory to host both models. This approach can be powerful when the draft model has been trained or selected specifically to match the target. It also gives an operator freedom to upgrade the drafter independently.

Choose MTP heads or a model-native drafter when operational simplicity and memory efficiency matter. This is especially attractive for local deployments, where avoiding an additional full model and reusing active context can be more important than extracting the maximum theoretical speedup. Gemma 4’s MTP release makes this option more accessible to teams that want speculation without building a two-model system from scratch.

Choose no speculation when the target is already fast enough, the workload has low acceptance, the stack uses unsupported generation controls, or hardware bottlenecks lie elsewhere. A small quantized model on a modern GPU may be fast enough that drafter overhead only adds noise. A retrieval system spending most of its time fetching documents will not be transformed by a faster decoder. Neither will an agent whose dominant delay is waiting for external tools.

This is not a failure of the technology. It is disciplined system design. The cheapest token is the one you do not need to generate, and the fastest architecture is the one that addresses the actual bottleneck.

How to test it before committing

A useful experiment does not begin with a vendor speedup number. It begins with representative prompts.

Collect a workload sample that includes short requests, long requests, structured outputs, tool calls, code generation, multilingual inputs, and whatever users actually do. Do not benchmark only a clean set of easy completions. That is how teams accidentally optimize for a demo.

For each request, log:

• target model and quantization
• drafter type and size
• proposal length
• number of draft rounds
• accepted tokens per round
• first-token acceptance rate
• rejection position
• drafter time
• target verification time
• cache allocation and memory use
• time to first token
• inter-token latency
• total response time
• output token count
• concurrency level
• sampling and decoding settings

Then compare ordinary decoding and speculative decoding at the same quality settings. Use the same prompts, seed policy where possible, stop conditions, grammar constraints, and output length limits.

Do not trust only aggregate acceptance rate. Plot the distribution. If half of requests accept nearly every draft token and half reject instantly, the average can look respectable while the user experience is inconsistent. Segment results by task type. Often the answer is not “use speculation” or “do not use speculation.” It is “use it for code completion and templated support answers, but disable it for constrained JSON generation and long tool chains.”

Finally, measure cost in the unit that matters. For a local installation, that might be watt-hours, VRAM headroom, fan noise, or the ability to keep a second application open. For a production cluster, it might be requests per accelerator, p95 latency, or dollars per completed task.

The broader shift

Speculative decoding is easy to describe as a trick, but Gemma 4 suggests a more consequential interpretation.

For years, serving efficiency was largely the domain of infrastructure specialists. Model builders trained the intelligence. Serving teams tried to extract more output from expensive hardware after the fact. The appearance of model-native drafting moves some of that efficiency work into the model product itself.

That matters because the economics of AI are increasingly determined by the distance between a compelling model demo and a tolerable everyday interaction. People do not experience parameter counts. They experience whether an assistant responds before their attention shifts elsewhere.

Gemma 4 does not abolish the serial nature of language generation. The target model still has to set the standard. Verification still costs time. Rejections still happen. Memory traffic, batching, and hardware constraints still decide whether an elegant algorithm turns into a useful product.

But it does make an important proposition more practical: a model can be designed not only to know more, but also to make its own slowest habit less costly.

For anyone operating local or low-latency AI, that changes the planning conversation. Speculation is no longer merely a capability buried in a serving framework. It is a measurable option at the heart of the model deployment decision. The teams that benefit will not be the ones that repeat the largest speedup claim. They will be the ones that measure their accepted length, find their real bottleneck, and know when the fastest prediction is the one they never ask the machine to make.