FedEnD: Communication-efficient Federated Learning for non-IID data via decentralized ensemble distillation

Federated Learning (FL) offers a paradigm for collaborative AI that mitigates raw data exposure, yet the statistical heterogeneity of client data severely constrains its practical application. This non-independent and identically distributed (non-IID) setting induces client drift, leading to unstable optimization and degraded generalization, particularly for under-represented classes. Existing solutions present a difficult trade-off: iterative, regularization-based methods suffer from high communication overhead and a centralized bottleneck, while knowledge-distillation-based approaches rely on impractical artifacts, such as shared public datasets. This work introduces FedEnD, a novel framework that addresses the previous challenge through an efficient, fully decentralized architecture. FedEnD employs a two-stage protocol that decouples local specialist training from a collaborative fusion stage. Following a communication-free training phase, clients perform a one-shot peer-to-peer broadcast that shares (optionally privatized) specialist parameters and lightweight class-count statistics. Each client then uses these statistics to construct a class-distribution-weighted teacher ensemble from the received specialists. Crucially, this ensemble’s knowledge is distilled into a robust global model on each client, using only their local data as unlabeled inputs, obviating the need for a central server or auxiliary data. Extensive experiments on MNIST, FashionMNIST, SVHN, and CIFAR-10 demonstrate that FedEnD outperforms baselines, surpassing robust methods such as SCAFFOLD by +5.7% on complex datasets in pathologically skewed settings. This superior accuracy is achieved while reducing communication bandwidth by 68.6% compared to standard iterative averaging, and by up to 84% compared to gradient-correction methods like SCAFFOLD, highlighting a favorable trade-off between accuracy and communication bandwidth in decentralized learning under severe non-IID partitions.