When Federated Clients See Different Modalities

Federated learning usually assumes that clients solve the same task with comparable inputs. That assumption becomes fragile when one participant has images, another has audio and a third has only inertial sensors. The problem is not merely that the datasets are non-IID. The available modalities are different as well.

This setting appears in many forms: an audiovisual device with a broken microphone, a vehicle that has telemetry but no camera, or a crisis-monitoring client that receives text without an image. A useful decentralized system should keep learning when local views are incomplete rather than treating every client as if it had the same sensor suite.

Two kinds of heterogeneity

Statistical heterogeneity changes the distribution of classes or features across clients. A convenient way to think about it is a concentration parameter $\beta$: smaller values mean that local data are more skewed and that the global label distribution is harder to reconstruct from any one participant.

Modality heterogeneity is different. A client may have a complete set of inputs, a subset or a modality that is temporarily unavailable. Let $m_{i,r}\in\{0,1\}$ indicate whether client $i$ can observe modality $r$. Its representation can be written as:

$$h_i(x)=\operatorname{Fuse}\left(\{m_{i,r}h_r(x_r)\}_{r=1}^{R},\;\{m_{i,r}\}_{r=1}^{R}\right).$$

The mask is important. Without it, a zero vector could mean a real observation, a missing sensor or a failed preprocessing step.

Why full-model averaging struggles

FedAvg-style averaging assumes that parameter updates are comparable enough to combine. With non-IID classes and missing modalities, a client can push the shared model towards a representation that is excellent for its local view but poorly aligned with the representations of other clients.

The failure may be hidden by the global mean. A well-connected multimodal client can dominate the average while a unimodal client receives a model whose internal features do not match its inputs. This is especially problematic when the client needs knowledge about a modality it cannot observe locally.

Prototype exchange as a different abstraction

Instead of sending all parameters, a client can summarize class-specific representations. For class $c$ and modality $r$, a local prototype may be expressed as:

$$p_{i,c}^{(r)} = \frac{1}{|D_{i,c}^{(r)}|} \sum_{x\in D_{i,c}^{(r)}} h_r(x),$$

provided that the local set is non-empty. Peers can then align prototypes, transfer information between available and unavailable modalities and keep the private examples local. This does not solve every privacy issue, but it gives the protocol a compact semantic object to exchange.

The evaluation design

The Modalis study evaluates this problem with AVMNIST, CREMA-D, UCI-HAR and CRISIS-MMD. The datasets cover audiovisual digits, audiovisual emotion, inertial activity recognition and image-text crisis events. Class skew is controlled with $\beta\in\{1.0,0.5,0.1\}$, while missing-modality configurations use $p_m\in\{0.0,0.25,0.5\}$.

The principal experiments use 20 clients connected by a static Erdős-Rényi graph with $p_{ER}=0.9$. This detail matters: the results describe a controlled graph and should not be read as evidence that every topology behaves identically.

The central question is not whether a multimodal model can beat a unimodal one in a centralized benchmark. It is whether decentralized clients with different views can benefit from each other's representations without requiring a central fusion service.

What a robust multimodal protocol needs

A practical protocol should include:

• a mask or explicit representation for unavailable modalities,
• a fusion rule that can change the weight of each available modality,
• an alignment objective that keeps local and exchanged representations compatible,
• a fallback when a prototype is stale or absent,
• evaluation per modality configuration, not only a single global score.

It should also report the cost of this machinery. Multimodal robustness is not free, and a method that requires every client to exchange a full model may be impossible on a constrained link.

The research lesson

Heterogeneous multimodal DFL is a coordination problem about partial views. The objective is not to pretend that every participant sees the same world. It is to make the differences explicit, summarize useful knowledge in a controlled representation and let each client benefit from information that is unavailable locally.

That perspective connects non-IID learning, missing sensors and communication efficiency. It also gives a clearer route to future work: dynamic modality availability, real topology changes, cross-dataset transfer and formal analysis of what a prototype can reveal.

This note is an original synthesis of the heterogeneous multimodal problem studied in the doctoral thesis. It does not reproduce the source article.