Multimodal data fusion for systems improvement: A review

Abstract

In recent years, information available from multiple data modalities has become increasingly common for industrial engineering and operations research applications. There have been a number of research works combining these data in unsupervised, supervised, and semi-supervised fashions that have addressed various issues of combining heterogeneous data, as well as several existing open challenges that remain to be addressed. In this review paper, we provide an overview of some methods for the fusion of multimodal data. We provide detailed real-world examples in manufacturing and medicine, introduce early, late, and intermediate fusion, as well as discuss several approaches under decomposition-based and neural network fusion paradigms. We summarize the capabilities and limitations of these methods and conclude the review article by discussing the existing challenges and potential research opportunities.

Keywords:

• Multimodal data
• heterogeneous data
• data fusion

Data availability statement

There is no data set associated with this article.

Notes on contributors

Nathan Gaw is an Assistant Professor in the Department of Operational Sciences at Air Force Institute of Technology. He received his BS and MS in biomedical engineering and a PhD in industrial engineering from Arizona State University (ASU), Tempe, AZ, USA, in 2013, 2014, and 2019, respectively. Nathan's research focuses on multi-modality fusion in healthcare and military applications fusing imaging, genetics, and telemonitoring data. He is a member of IISE, INFORMS, and IEEE.

Safoora Yousefi is an applied scientist at Microsoft. Safoora received their BSc in computer science from University of Tehran, and their PhD in computer science from Emory University. Prior to Microsoft, they interned at Google Brain and Roche. Their research interests include machine learning, multi-task, adversarial, and self-supervised learning and their applications to real world scenarios such as natural language processing and cancer genomics.

Mostafa Reisi Gahrooei received the master's degree in computational science and engineering and the PhD degree (2019) in industrial and systems engineering from the Georgia Institute of Technology, Atlanta, GA, USA, and the MSc degrees in transportation engineering and applied mathematics from the Southern Illinois University Edwardsville, Edwardsville, IL, USA. He is currently an Assistant Professor with the Department of Industrial and Systems Engineering, the University of Florida, Gainesville, FL, USA. His research focuses on modeling, monitoring, and control of complex systems with multimodal, functional, and high-dimensional data. Dr. Reisi Gahrooei is a member of the Institute for Operations Research and the Management Sciences (INFORMS) and the Institute of Industrial and Systems Engineers (IISE).