TY - GEN
T1 - Handling Missing Modalities in Multimodal Federated Learning for Healthcare Data Analytics
AU - Ngamsittipong, Vichayuth
AU - Jumratboonsom, Jakkaphat
AU - Thongsuk, Thannatorn
AU - Yipeng, Zhou
AU - Jatuviriyapornchai, Watthanan
AU - Sajjacholapunt, Petch
N1 - Publisher Copyright:
© The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd. 2026.
PY - 2026
Y1 - 2026
N2 - Accurate disease diagnosis increasingly depends on integrating diverse clinical data such as health records, medical images, and clinical notes. However, real-world medical datasets are often incomplete and distributed across hospitals under strict privacy regulations, which limiting the development of robust Machine Learning (ML) models. This study investigates multimodal Federated Learning (FL) as a privacy-preserving framework for collaborative model training without sharing raw patient data. Using a selected subset of 312 patients from the MIMIC-IV database containing complete EHR, medical imaging, and radiology note modalities, neural encoders were trained for each modality and distributed across simulated hospital nodes to simulate federated training. The predictive objective was to classify in-hospital mortality. Generative AI tools, including ChatGPT and Claude, were employed to synthesize missing radiology notes to address incomplete modalities. The results indicate that multimodal inputs yield higher predictive accuracy than unimodal input. Employing generative AI to synthesize missing modalities problem effectively restores model performance, while federated learning preserves patient privacy without compromising predictive accuracy.
AB - Accurate disease diagnosis increasingly depends on integrating diverse clinical data such as health records, medical images, and clinical notes. However, real-world medical datasets are often incomplete and distributed across hospitals under strict privacy regulations, which limiting the development of robust Machine Learning (ML) models. This study investigates multimodal Federated Learning (FL) as a privacy-preserving framework for collaborative model training without sharing raw patient data. Using a selected subset of 312 patients from the MIMIC-IV database containing complete EHR, medical imaging, and radiology note modalities, neural encoders were trained for each modality and distributed across simulated hospital nodes to simulate federated training. The predictive objective was to classify in-hospital mortality. Generative AI tools, including ChatGPT and Claude, were employed to synthesize missing radiology notes to address incomplete modalities. The results indicate that multimodal inputs yield higher predictive accuracy than unimodal input. Employing generative AI to synthesize missing modalities problem effectively restores model performance, while federated learning preserves patient privacy without compromising predictive accuracy.
KW - Data Privacy
KW - Generative Imputation
KW - Healthcare Analytics
KW - Missing Modalities
KW - Multimodal Federated Learning
UR - https://www.scopus.com/pages/publications/105044995436
U2 - 10.1007/978-981-95-9843-4_11
DO - 10.1007/978-981-95-9843-4_11
M3 - Conference contribution
AN - SCOPUS:105044995436
SN - 9789819598427
T3 - Communications in Computer and Information Science
SP - 124
EP - 139
BT - Ubi-Media Computing, Pervasive Systems, Algorithms and Networks - 14th International Conference on Ubi-Media Computing, Ubi-Media 2026, Proceedings
A2 - Hui, Lin
A2 - Hsu, Ching-Hsien
A2 - Sarban Singh, Ranjit Singh
PB - Springer Science and Business Media Deutschland GmbH
T2 - 14th International Conference on Ubi-Media Computing and Workshops, Ubi-Media 2026
Y2 - 18 January 2026 through 22 January 2026
ER -