Modalis: Prototype Exchange for Heterogeneous Multimodal DFL

When a federated client lacks one of the modalities used by its peers, sharing a full model is not always the most useful form of collaboration. The Modalis approach studies a narrower exchange: class-aware multimodal prototypes that carry representation knowledge while allowing each client to keep its examples and local training process private.

The method is evaluated in a decentralized setting with clients connected by a peer graph. It is designed for two simultaneous sources of difficulty: class distributions can be non-IID and some clients can miss one or more modalities. The protocol therefore has to decide not only how to average information, but also how to interpret an incomplete view.

From local embeddings to shared prototypes

Let $h_r(x_r)$ be the embedding produced by modality encoder $r$. A client can summarize the examples of class $c$ with a prototype:

$$p_{i,c}^{(r)} = \frac{1}{|D_{i,c}^{(r)}|} \sum_{x\in D_{i,c}^{(r)}} h_r(x).$$

The prototype is a semantic summary, not a copy of the raw data. It can be aligned with prototypes from other clients and used to regularize a local representation. If a modality is unavailable, the protocol needs an explicit substitute rather than pretending that an ordinary zero vector contains equivalent evidence.

Four mechanisms that work together

Null contextual embeddings

Modalis represents an absent modality with a learned contextual placeholder. The placeholder carries the fact that the modality is missing and gives the fusion block a stable input shape. It is not treated as an observation and should not be interpreted as a reconstructed sensor reading.

Adaptive gating

The contribution of each modality is adjusted according to what is available and what is useful for the current example. A simple abstraction is:

$$g_{i,r}=\frac{m_{i,r}\exp(a_{i,r})} {\sum_{s=1}^{R}m_{i,s}\exp(a_{i,s})+\varepsilon}, \qquad z_i=\sum_{r=1}^{R}g_{i,r}h_r(x_r).$$

Here $m_{i,r}$ is a presence mask and $a_{i,r}$ is a learned relevance score. The expression illustrates the role of the gate: unavailable modalities contribute no weight, while available modalities can be emphasized or down-weighted.

Multimodal fusion

Fusion maps the available representations into a shared space where local embeddings and exchanged prototypes can be compared. This is important for cross-modal transfer. A client that has only images should still be able to benefit from knowledge learned by a peer that also observed audio, but the transfer must pass through a compatible representation.

Alignment objectives

The local task loss is complemented by objectives that keep the representation close to useful peer prototypes. An illustrative loss is:

$$\mathcal{L}=\mathcal{L}_{task} +\lambda_p\mathcal{L}_{proto} +\lambda_a\mathcal{L}_{align},$$

where the weights control the balance between local prediction, prototype consistency and cross-client alignment. The exact implementation and hyperparameters belong to the experiment; the design principle is to make the collaboration objective explicit.

What the reported results mean

Across the evaluated datasets and heterogeneity settings, the method reports a relative F1 improvement of up to approximately 4.0% over the strongest competing method in the corresponding comparison. On AVMNIST, the image-only client reaches an F1 of 82.1%, a relative improvement of 19.9% over FedAvg in the reported configuration.

These are encouraging results for partial-view clients, but they should be read with the experimental boundaries: four benchmark datasets, 20 clients, controlled missing-modality probabilities and a specified peer graph. They show that prototype transfer can be useful; they do not establish zero-shot generalization to arbitrary sensors or domains.

Why the design is interesting

The contribution is not just a smaller message. It changes the unit of collaboration from a complete parameter vector to a class- and modality-aware representation. That choice can be valuable when:

• local data are skewed by class,
• modalities are missing asymmetrically,
• full-model exchange is expensive,
• a peer needs semantic knowledge rather than another optimizer state.

It also creates new responsibilities. Prototypes can become stale, poorly aligned or informative about the source data. A production system would need versioning, leakage analysis, secure exchange and a policy for low-support classes.

A cautious interpretation

Prototype-based multimodal DFL is best understood as a bridge between representation learning and decentralized coordination. It does not remove the need for robust aggregation, topology management or privacy analysis. It offers a compact object around which those mechanisms can be designed when clients see different parts of the same task.

This note is an original synthesis of the Modalis contribution in the doctoral thesis and its Information Fusion publication. It does not reproduce the source text.