DiLoCo vs Federated Learning: Choosing a Distributed Training Method

DiLoCo vs Federated Learning: The Core Difference

DiLoCo, short for Distributed Low-Communication training, lets every worker run hundreds of local optimization steps before synchronizing, then exchanges compact pseudo-gradients through an outer optimiser. Federated learning, in its classic FedAvg form, averages model updates from clients on a much tighter cadence, so communication cost scales with model size and round frequency. The practical consequence: FedAvg wants fast, reliable links between broadly similar clients, while DiLoCo tolerates slow home connections, mixed hardware, and workers that come and go. That is why DiLoCo has become the default pattern for decentralised AI training over the public internet, where you cannot assume datacenter interconnects. Both approaches produce a single shared model; the difference is the price of agreement. If your workers sit in one cluster, either works. If they sit in living rooms on different continents, the communication budget decides for you, and it decides for DiLoCo.

How Federated Averaging Works and Where It Strains

FedAvg was introduced by Google in 2016 for a specific setting: millions of phones, each holding private data that must never leave the device. Clients receive the current model, take a few local steps, and return their weight deltas; the server averages them into a new global model. It is elegant, battle-tested, and still the right tool for privacy-driven mobile workloads. It strains when the model is large and the network is the public internet. A weight delta for a multi-billion-parameter model is gigabytes, and FedAvg wants it every round. Stragglers stall aggregation or get dropped, biasing the result toward fast clients. Heterogeneous hardware amplifies both problems. Decentralised compute markets mostly sidestep FedAvg for exactly these reasons. For frontier-scale open training runs, the community needed something that treats bandwidth as the scarcest resource, which is the design premise DiLoCo starts from.

How DiLoCo Cuts Communication by Orders of Magnitude

DiLoCo splits optimization into an inner loop and an outer loop. Each worker runs its own inner optimiser, typically AdamW, for hundreds of steps on local data. Only then do workers synchronize: the difference between their current weights and the last agreed checkpoint forms a pseudo-gradient, which an outer optimiser with Nesterov momentum applies to the shared model. Because syncs are rare and pseudo-gradients compress and sparsify well, the bytes on the wire drop by orders of magnitude versus per-step gradient exchange. OpenDiLoCo demonstrated the pattern across continents on commodity links. QBC applies the same insight on a live chain: workers submit sparse, signed pseudo-gradients measured in kilobytes per round, small enough that anyone can contribute to training from home without fiber. The chain then handles what the paper does not: deciding, publicly and verifiably, whose updates actually improved the model.

Fault Tolerance and Heterogeneous Hardware

A distributed training method that assumes uniform, always-on workers fails the moment real volunteers show up. FedAvg rounds are only as fast as their slowest included client, and dropped clients skew the average. DiLoCo absorbs heterogeneity structurally: a slow worker simply completes fewer inner steps between syncs, and a missing worker just does not contribute a pseudo-gradient that round. Nothing blocks. This is what lets CPU and GPU contributors coexist in one run, and it is why on QBC both hardware classes earn: rewards follow measured contribution, not device type, so you can earn with a plain CPU beside someone on a gaming GPU. The open question heterogeneity raises is quality, not liveness. A flaky or malicious worker cannot stall DiLoCo, but it can submit a harmful update, which is why verification, not synchronization, becomes the hard problem once training leaves the datacenter.

Verification: The Dimension Neither Method Solves Alone

Neither DiLoCo nor FedAvg answers the question that matters most in open networks: how do you know a contribution is real and beneficial? Both papers assume honest participants. QBC bolts the missing layer on. Every pseudo-gradient is signed with a post-quantum Dilithium5 key, validators independently measure held-out loss on data the workers never saw, and a 2/3 quorum finalises each epoch on-chain before any reward pays out. Updates that poison or regress the model roll back and earn nothing. Across the attested series so far, held-out loss is down more than 75 percent, a number anyone can audit rather than take on faith. This turns a training protocol into verifiable AI on-chain: the method (DiLoCo) minimizes communication, and the chain maximizes accountability. Comparing frameworks purely on bandwidth misses this axis entirely, and it is the axis that decides whether strangers can train together.

Which Distributed Training Method Should You Choose?

Choose FedAvg-style federated learning when privacy is the point: cross-device settings, similar clients, data that legally cannot move, models small enough to ship every round. Choose DiLoCo when scale and geography are the point: large models, public-internet links, volunteer or marketplace hardware, tolerance for churn. If you also need trustless rewards and proof the training happened, you need a verification layer on top of either, which is the design space occupied by QBC and, differently, by projects positioned as a Gensyn alternative. QBC pairs DiLoCo with a MoE adapter fine-tuning target, so contributions update a compact 10-expert adapter rather than a frozen 7B base, keeping per-round payloads tiny. A hybrid path is also legitimate: some teams prototype with FedAvg inside one trusted cluster, then move to DiLoCo when the run opens to outside hardware, since both share the same averaging intuition and differ mainly in cadence and payload size. The honest summary: DiLoCo is winning the open-internet setting on communication cost, and the interesting competition now is over proving the work was useful.

Inside a Live Round: From Local Steps to Finalised Epoch

Theory aside, here is what one production round looks like on QBC today. A worker, which might be a GPU box or a plain CPU machine since one binary auto-detects hardware, pulls the current adapter state and grinds through its local inner steps. When the round closes, it sparsifies the weight delta into a kilobyte-scale pseudo-gradient, signs it with Dilithium5, and submits. Validators aggregate the round, measure held-out loss on data no worker has seen, and a 2/3 quorum finalises the epoch on-chain; regressing updates roll back and earn nothing. None of this is a diagram. The two-machine DiLoCo run documents the first cross-host round, the outer optimiser writeup explains how Nesterov momentum turns sparse deltas into stable global progress, and the multi-worker scale-up log records the jump from one trainer to a mixed fleet; the wider ecosystem validated the same low-communication family when INTELLECT-1 trained a 10B model across continents. QBC's loop has since crossed one million inner steps. To stand inside a round instead of reading about one, start with the network joining guide.

Economics and Expectations, Stated Honestly

Method choice decides bandwidth; it does not decide whether contribution is worth your electricity, so the economics deserve plain statement. QBC pays training rewards automatically per verified epoch, and they are small today, fractions of QBC per round, sized for a young network proving its mechanism rather than for income. No projections are published here or anywhere on this site, because a projected yield in this category is a promise nobody can keep. What the network offers instead is receipts: the first paid epoch writeup walks through an actual settlement, from worker submissions through quorum attestation to the on-chain payout, and every epoch since is equally auditable through the public audit trail. That standard, verify the work and the payment or assume neither, is the one to apply across the whole idle-compute income category. DiLoCo earns its place in this comparison by making participation cheap enough that honest, small rewards are not swallowed by bandwidth costs; what it cannot do is manufacture yield, and no training method can.