Modalis: Prototype Exchange for Heterogeneous Multimodal DFL

When a federated client lacks one of the modalities used by its peers, sharing a full model is not always the most useful form of collaboration. The Modalis approach studies a narrower exchange: class-aware multimodal prototypes that carry representation knowledge while allowing each client to keep its examples and local training process private.

The method is evaluated in a decentralized setting with clients connected by a peer graph. It is designed for two simultaneous sources of difficulty: class distributions can be non-IID and some clients can miss one or more modalities. The protocol therefore has to decide not only how to average information, but also how to interpret an incomplete view.

From local embeddings to shared prototypes

Let $h_r(x_r)$ be the embedding produced by modality encoder $r$. A client can summarize the examples of class $c$ with a prototype:

$$p_{i,c}^{(r)} = \frac{1}{|D_{i,c}^{(r)}|} \sum_{x\in D_{i,c}^{(r)}} h_r(x).$$If the formula does not fit, focus it and use the left and right arrows, or scroll horizontally.

The prototype is a semantic summary, not a copy of the raw data. It can be aligned with prototypes from other clients and used to regularize a local representation. If a modality is unavailable, the protocol needs an explicit substitute rather than pretending that an ordinary zero vector contains equivalent evidence.

Four mechanisms that work together

Null contextual embeddings

Modalis represents an absent modality with a learned contextual placeholder. The placeholder carries the fact that the modality is missing and gives the fusion block a stable input shape. It is not treated as an observation and should not be interpreted as a reconstructed sensor reading.

Adaptive gating

The contribution of each modality is adjusted according to what is available and what is useful for the current example. A simple abstraction is:

$$g_{i,r}=\frac{m_{i,r}\exp(a_{i,r})} {\sum_{s=1}^{R}m_{i,s}\exp(a_{i,s})+\varepsilon}, \qquad z_i=\sum_{r=1}^{R}g_{i,r}h_r(x_r).$$If the formula does not fit, focus it and use the left and right arrows, or scroll horizontally.

Here $m_{i,r}$ is a presence mask and $a_{i,r}$ is a learned relevance score. The expression illustrates the role of the gate: unavailable modalities contribute no weight, while available modalities can be emphasized or down-weighted.

Multimodal fusion

Fusion maps the available representations into a shared space where local embeddings and exchanged prototypes can be compared. This is important for cross-modal transfer. A client that has only images should still be able to benefit from knowledge learned by a peer that also observed audio, but the transfer must pass through a compatible representation.

Alignment objectives

The local task loss is complemented by objectives that keep the representation close to useful peer prototypes. An illustrative loss is:

$$\mathcal{L}=\mathcal{L}_{task} +\lambda_p\mathcal{L}_{proto} +\lambda_a\mathcal{L}_{align},$$If the formula does not fit, focus it and use the left and right arrows, or scroll horizontally.

where the weights control the balance between local prediction, prototype consistency and cross-client alignment. The exact implementation and hyperparameters belong to the experiment; the design principle is to make the collaboration objective explicit.

What the reported results mean

The results in this section are reported in the Modalis paper.

Across the evaluated datasets and heterogeneity settings, the method reports a relative F1 improvement of up to approximately 4.0% over the strongest competing method in the corresponding comparison. On AVMNIST, the image-only client reaches an F1 of 82.1%, a relative improvement of 19.9% over FedAvg in the reported configuration.

These are encouraging results for partial-view clients, but they should be read with the experimental boundaries: four benchmark datasets, 20 clients, controlled missing-modality probabilities and a specified peer graph. They show that prototype transfer can be useful; they do not establish zero-shot generalization to arbitrary sensors or domains.

Why the design is interesting

The contribution is not just a smaller message. It changes the unit of collaboration from a complete parameter vector to a class- and modality-aware representation. That choice can be valuable when:

• local data are skewed by class,
• modalities are missing asymmetrically,
• full-model exchange is expensive,
• a peer needs semantic knowledge rather than another optimizer state.

It also creates new responsibilities. Prototypes can become stale, poorly aligned or informative about the source data. A production system would need versioning, leakage analysis, secure exchange and a policy for low-support classes.

A cautious interpretation

Prototype-based multimodal DFL is best understood as a bridge between representation learning and decentralized coordination. It does not remove the need for robust aggregation, topology management or privacy analysis. It offers a compact object around which those mechanisms can be designed when clients see different parts of the same task.

A prototype needs more than a vector

As a protocol extension, I propose attaching class, modality, encoder version, originating round and local support to the vector. The receiver must distinguish an absent prototype from one estimated using few examples. Exact support counts can also leak information, so their disclosure needs a privacy policy.

If encoders evolve differently, identical embedding dimensions do not guarantee compatibility. Evaluate alignment during training and after long pauses. A version identifier helps detect incompatibility but does not establish aligned spaces.

Ablations to understand the contribution

A useful experimental extension disables exchange, gating and alignment objectives separately under comparable conditions. Compare each variant by modality pattern and class support. Changing everything at once cannot reveal which component benefits a unimodal client.

The primary reference is Modalis, Information Fusion 132, 104251. This note's formulas are illustrative rather than replacements for its implementation. In particular, a strictly masked gate and a contextual placeholder can serve different functions depending on the fusion architecture.

When every sensor is missing

An operational system needs a state for no valid local evidence. A neural network accepting an input vector does not establish that sufficient information exists for prediction. I propose marking the output unevaluable or using a validated fallback, and measuring that condition separately.

The multimodal heterogeneity guide explains partial views. The communication-efficiency article details exchange accounting.