Can Frontier Models Train Across Data Centers Without Staying in Lockstep?

The synchronization tax

Conventional large-scale training is built around a demanding bargain. Many accelerators each process different slices of data, calculate proposed changes to the model, exchange those changes, then begin the next step from the same shared model state.

That arrangement is powerful because it keeps every participant aligned. It is also unforgiving. If one group of machines is delayed by network congestion, hardware trouble or a slower batch, the others wait. This is manageable inside a single, purpose-built data center with extremely fast networking. It gets harder when computing capacity is spread across cities, regions or hardware generations.

The human consequence is mundane but expensive: teams acquire or reserve massive contiguous clusters because splitting work across available capacity can create too much coordination overhead. Compute sitting in another facility may be less useful than it looks if using it means slowing down the whole job.

Google DeepMind’s April 2026 announcement of Decoupled DiLoCo proposes treating this problem less like an exercise in keeping every musician on the same beat and more like an orchestra that rehearses in sections before reconvening. The company says the system divides a training job into independent learner units, or islands, that can continue making progress even when another island is unavailable.

The key idea is not that synchronization disappears. A model trained by isolated groups forever would drift into several incompatible versions. Instead, Decoupled DiLoCo moves synchronization to a less frequent, more tolerant layer of the system.

Each island trains locally for a meaningful stretch of work. It then sends an update, or fragment of its learned parameters, across the wide-area network. A synchronizing service merges enough incoming updates to form a new shared direction for the model. Islands can receive that newer state and keep going, but they do not need to freeze at every local step while waiting for all other islands.

That distinction changes the economics of distance. In ordinary data-parallel training, network delay is repeatedly inserted into the critical path. In a decoupled design, much of the communication can take place while an island is already doing more computation.

What the islands are actually doing

It is easy to hear “asynchronous training” and imagine a free-for-all in which every data center improvises. That is not the intended design.

A learner island still runs a disciplined local optimization process. It reads training tokens, calculates how the model should change, and applies many local updates. At intervals, it packages information about those changes for the synchronizer. The synchronizer combines the contributions it has received, then distributes a refreshed model state or correction back to the islands.

The practical change is that an island can be somewhat stale. It may be learning from a model version that is not the newest version held by the synchronizer. In distributed systems, staleness is normally treated as a warning sign. Here it is a deliberate trade: accept controlled delay in the shared state to avoid turning every transient problem into a global halt.

The synchronizer is therefore not a conventional step-by-step traffic controller. It is closer to an editor collecting drafts from several reporters. It does not need every draft at the same instant to begin assembling the next edition, but it does need enough reliable contributions to avoid publishing nonsense.

DeepMind describes a minimum-quorum approach: rather than wait for every learner unit, the system proceeds after receiving enough usable updates. That is the mechanism that makes fault tolerance operational rather than rhetorical. A disconnected or failed island becomes an absent contributor, not automatically the reason the entire run stops.

The company also describes reintegrating learner units after they return. That matters because failures in a very large training system are not exceptional. Drives, hosts, networking equipment and accelerators fail. Maintenance interrupts capacity. Whole racks can be removed from service. The question is not whether something will fail during a long training run. It is whether the architecture turns local failure into global idle time.

Why bandwidth becomes the headline number

Distance is costly in two ways. There is latency, meaning the time a message needs to travel. Then there is bandwidth, meaning how much information can be transferred per second. Frontier training has traditionally demanded unusually generous amounts of both.

DeepMind’s comparison says that, across eight data centers, conventional data-parallel training would require 198 gigabits per second of bandwidth, while Decoupled DiLoCo would require 0.84 gigabits per second. Those are dramatically different requirements, though the company notes that the bandwidth and failure goodput comparisons are based on simulated runs.

The scale of the gap is the reason the proposal deserves attention. A design that needs close to 200 Gbps of inter-data-center capacity effectively assumes specialized network infrastructure. A design that can work in the low single-digit Gbps range opens the possibility of using more ordinary wide-area links between facilities.

That does not mean bandwidth is free, or that an enterprise can simply join spare machines across the public internet and train a frontier model. The traffic still has to be secured, scheduled, monitored and paid for. Model updates remain large. Storage, checkpointing and data access become distributed-systems problems of their own.

But reduced bandwidth needs could alter a familiar strategic constraint: the belief that the only credible path to frontier training is one giant, tightly coupled campus-scale cluster.

DeepMind says it trained a 12-billion-parameter model across four U.S. regions using 2-5 Gbps wide-area networking, and reports that this was more than 20 times faster than conventional synchronization methods in that setting. The company attributes the speedup to allowing computation to continue through periods that would otherwise be blocked by communication.

That result is important, but it is also where readers should be most careful. “More than 20 times faster” is not a universal property of asynchronous learning. It is a comparison with a particular conventional synchronization setup under particular network conditions. If the baseline is forced to wait often across distant regions, a decoupled system should look spectacular. If all chips already live in one exceptionally well-connected cluster, the advantage may narrow sharply.

The quality question is harder than the networking question

Moving fewer bits is easy to celebrate. Producing a model that is just as good is much harder.

Delayed updates can damage learning because each island may be optimizing against an older view of the model. Different islands may encounter different data patterns at different times. Their locally useful updates can partially conflict when merged. If the merge process is careless, the global model may wobble, converge more slowly or settle at a weaker result.

This is the central scientific risk in Decoupled DiLoCo. Communication efficiency is valuable only if the saved time and infrastructure cost do not reappear as longer training schedules, unstable runs or diminished model capability.

DeepMind reports that Gemma 4 models trained using Decoupled DiLoCo matched the benchmarked machine-learning performance of conventionally trained models in its experiments. Its published figure lists average benchmark accuracy of 64.1% for Decoupled DiLoCo and 64.4% for data-parallel training.

That is encouraging, but it is not the final word on frontier-scale equivalence. Benchmark averages can hide important differences in reasoning, coding, multilingual behavior, safety behavior or rare capabilities. A training technique also needs to be tested across architectures, datasets, optimizer choices, model sizes and training durations before it becomes a default infrastructure choice.

The right interpretation is neither “the problem is solved” nor “the result does not count.” It is that DeepMind has supplied evidence that the quality penalty can be small under the conditions it tested. The remaining question is how wide that safe operating range really is.

Resilience is not the same as simplicity

The strongest case for decoupled training is not merely that it might use cheaper networking. It is that it changes what a failure means.

DeepMind says its chaos-engineering tests deliberately introduced hardware failures, including loss of entire learner units. Under high failure rates, its figure reports 88% goodput for Decoupled DiLoCo versus 27% for conventional data-parallel training. Goodput here means useful training work rather than time consumed by stalled or unproductive activity.

A system that keeps working through failure can be more valuable than one that is theoretically faster when nothing goes wrong. Large training runs are long enough that reliability compounds. Every dependency creates another opportunity for a pause, a restart or a costly intervention by an infrastructure team.

Still, fault tolerance shifts complexity rather than abolishing it. A decoupled system needs rules for late updates, duplicate updates, corrupted transfers, uneven data consumption, restart states and safe reentry after an island comes back. It needs observability good enough to distinguish healthy asynchrony from a training run quietly drifting off course.

It also demands a more nuanced operating culture. In a lockstep system, operators have a simple signal: everyone is at the same step, or the job is blocked. In a decoupled system, useful questions multiply. How stale is each island? Is one region repeatedly contributing weak or delayed updates? Is the quorum too permissive? Are faster islands dominating the learning process?

Those are solvable problems, but they are real costs. Organizations should not confuse “less dependent on a perfect network” with “easy to operate.”

Where this could matter first

The most suitable workloads are likely to be long-running pretraining jobs with enough local computation between exchanges, substantial geographic separation and a meaningful chance of infrastructure disruption. These are exactly the settings where waiting is expensive and spare capacity may be geographically scattered.

Mixed hardware is another compelling use case. DeepMind says its experiments combined TPU v6e and TPU v5p hardware, with chips operating at different speeds, while matching the machine-learning performance of single-generation training runs.

For operators, that suggests an alternative to the familiar upgrade cycle in which older accelerators become second-class citizens the moment a new generation arrives. If heterogeneous machines can contribute without forcing every faster participant into the pace of the slowest one, existing capacity may remain strategically useful for longer.

The least suitable work is likely to be highly sensitive to exact synchronization, very short training jobs where setup complexity dominates, or workloads already running inside a low-latency supercluster with abundant internal bandwidth. In those cases, the conventional method has a simple advantage: it is proven, predictable and easier to reason about.

The broader implication is not that the giant AI supercluster is about to disappear. Frontier models will still reward dense compute, fast interconnects and carefully engineered facilities. But Decoupled DiLoCo offers a credible challenge to the assumption that all useful compute must be gathered in one place before it can train one model.

For executives and infrastructure planners, the test is straightforward. Ask how often the system communicates, how large each exchange is, what happens when one participant fails, how stale an update may become, and whether quality holds on the evaluations that matter. A proposal that can answer those questions clearly may be more than a clever networking trick. It may be a way to turn fragmented global compute into a more resilient training machine.