Decentralized Federated Learning with Multimodal Prototypes for Heterogeneous Data

Decentralized Federated Learning (DFL) enables collaborative machine learning across numerous devices while avoiding bottlenecks and reliance on a single trusted entity inherent to centralized architectures. However, its practical application is challenged by modern scenarios where data is increasingly multimodal. The key obstacles in such settings are severe data heterogeneity, characterized by non-Independent and Identically Distributed (non-IID) class distributions, and incomplete data, where modalities are often missing across clients. Existing solutions struggle with these challenges, either incurring high communication costs or lacking effective mechanisms for fusing partial information. To overcome these limitations, this work introduces Modalis, a novel framework for multimodal DFL that achieves superior model performance under data heterogeneity while minimizing network consumption. It pioneers a communication-efficient, prototype-centric protocol in which clients exchange compact, modality-aware class representations rather than high-dimensional model parameters. This process is guided by a multi-objective loss function enforcing inter-modality coherence and representation alignment for effective knowledge fusion. The framework integrates sophisticated architectural innovations, including contextual null embeddings for intelligent data imputation and robust multimodal fusion using adaptive gating and multi-way transformers. The approach is validated through theoretical analysis, providing formal convergence guarantees, and extensive experiments on standard multimodal benchmarks. These results demonstrate that Modalis achieves superior performance, improving F1 scores by up to 4% under high heterogeneity and reducing communication costs by over 40 times compared to state-of-the-art baselines, establishing it as a highly effective solution for collaborative AI.