What multimodal components, tools, dataset and focus of emotion are used in the current research of multimodal emotion: a systematic literature review

Abstract

Emotional engagement is essential in human communication, and the meaning of emotions often entails multimodal relationships. Besides language, multimodality and emotions are semiotic systems that have increasingly attracted the attention of researchers, especially concerning their contribution to the meaning-making process in communication. However, research trends that address how emotions and multimodal components (hereafter multimodal emotions) complement each other in the meaning-making process have not been extensively researched. Hence, this research aims to identify research trends on multimodality and emotions studied from 2018 to 2022. The method used in this research was systematic literature review (SLR) to review, identify, evaluate and interpret all existing research on this topic. Data were obtained from IEEE Xplore, Science Direct and Emerald journals which were then sorted based on the PRISMA method. This analysis acquired comprehensive information on the multimodal components of emotions, data acquisition technologies, datasets and emotional focus and trends. This SLR demonstrated that the research of emotions involving multimodality was conducted with a focus on audio-visual mode with the use of machines namely EEG and LSTM sourced from the IEMOCAP dataset and focused on positive and negative emotions, with anger being the largest focus. Moreover, the findings also showed the interconnectedness of each multimodal component of emotions. Based on these findings, this study suggests that multimodal emotion research should focus on identifying and investigating the meanings generated from emotions to produce good communication. Lack of subject privacy and unmanageable bias in research are other things to consider for future research directions.

IMPACT STATEMENT

Communication is a meaning-making endeavor comprised language and various semiotic systems, including multimodality and emotion. No research has yet identified how emotion and multimodal components complement each other. This study aims to identify research trends on multimodality and emotion from journal articles published in 2018–2022 to investigate multimodal components and other data sets used to explore the meaning-making process and to explore how emotions are comprehensively accessed. Using the PRISMA method, the data was analyzed to obtain comprehensive information regarding the multimodal components of emotions, data acquisition technologies, datasets and emotional focus with research trends. This SLR showed that emotion research involving multimodality was conducted with a focus and dominance on audio-visual mode with the use of machines namely EEG and LSTM sourced from the IEMOCAP dataset type. In addition, the dominant emotion focus in this study was positive and negative emotions, with anger being the largest focus.

Keywords:

• Multimodality
• multimodal emotions
• emotions
• systematic literature review

REVIEWING EDITOR:

• Guangchao Charles Feng, School of Communication, Hong Kong Baptist University, Hong Kong

Subjects:

• Interpersonal Communication
• Semiotics
• Sociolinguistics

1. Introduction

Emotions are enacted on the basis of each person’s subjective experience, whilst multimodal emotions are patterns of human emotions in interaction to convey meaning in communication. This concept has been developed for about two decades since it first appeared in the 1990s (Lim et al., 2022; Morton et al., 2020; Song et al., 2019). Initially, the studies of multimodal emotions were a part of research in the field of psychology before being developed into various research themes. Over the last five years, trends in multimodal emotion research have focused on recognizing emotions (Barrett et al., 2019; Keltner et al., 2019; Waterloo et al., 2018), identifying emotions (Davies et al., 2019; Lim et al., 2022; Swain et al., 2018) and conducting experiments on tools or technologies that can evoke emotions (Baig & Kavakli, 2019; Muszynski et al., 2021; Zhang et al., 2020). The Urgency of Multimodal Emotion Research in Communication: 1. Effective Communication: Multimodal emotions enable clearer and more effective message delivery. Combining facial expressions, tone of voice, body language and other elements can strengthen verbal messages and reduce misunderstandings. 2. Understanding Emotional Nuances: Multimodal emotions help us grasp emotional nuances that might not be conveyed solely through words. For instance, a hesitant tone of voice may indicate uncertainty even if the words sound positive. 3. Building Relationships: Multimodal emotional components can foster trust and better connections between individuals. Showing interest and attention multimodally can make others feel comfortable and valued. 4. Enhancing Social Skills: The ability to comprehend and utilize multimodal emotions is crucial for improving social skills. By recognizing multimodal emotional components, we can interact with others more appropriately and situationally.

In the development of this concept, multimodal emotions have been researched with regard to communication in general, such as interaction in daily life between people, in educational contexts between teachers and students, in film context between actors, in business context between sellers and buyers, and in a company context between employees and customers, or between superiors and subordinates. This coverage shows that multimodal emotions are a resources for making meaning and interpreting messages conveyed in communication (Fröhlich et al., 2019; Löffler et al., 2018; Mills & Unsworth, 2018). For example, research by Davies et al. (2019), Chou and Budenz (2020), Men and Yue (2019), Strandberg et al. (2020), Wirz (2018) and Amini et al. (2018) shows the importance of emotional meaning in communication. Based on the preceding explanation and literature review, no prior research has focused on the process of meaning-making involving multimodal emotions in communication. Existing studies have primarily concentrated on emotion recognition, emotion identification and the development of tools to elicit emotions. This research aims to identify common components of multimodal emotions to formulate more in-depth and comprehensive directions for future research, particularly in emotional meaning.

The increasingly broad context of communication requires a review of the literature to see how meaning can be provided in communication by involving multimodal emotions. This initial exploration of meaning-making processes focused on identifying common components of multimodal emotions to formulate more in-depth and comprehensive future research directions, especially in the area of emotional meaning. Multimodal research on human emotions is very important to study and it is directly beneficial to set right interaction context to create good and effective communication. The data from previous research on multimodal emotion studies were collected to determine the methods and the data collection technique, and were obtained from publications exploring multimodal emotions from 2018 to 2022. This period was chosen considering the significant impact of the Industrial Revolution version 4.0 (Germany, America, China, Singapore) on the increasing number of studies on multimodal emotions, seen from the large use of technology, such as datasets and several applications in identifying emotions that arise from participants. This research relies on a systematic literature review (SLR) to methodically review and identify journals that follow predetermined stages or protocols in each phase (see Mohamed Shaffril et al., 2021; van Dinter et al., 2021; Williams et al., 2021), and to avoid subjective identification in providing journal article identification procedures (van Dinter et al., 2021).

Conducted due to the absence of literature review examining the relationship between multimodality and emotions in terms of meaning formation, this SLR was intended to answer the following questions: 1) How are the different multimodal components analyzed and categorized? 2) What tools and technologies can be used to obtain multimodal emotion data? 3) What datasets are available to analyze multimodal emotions and their main features? and 4) What emotional focus is available to obtain better multimodal accuracy?

2. Theoretical framework

2.1. Previous research

The emergence of multimodal emotions is based on human recognition using diverse modalities such as sound, gaze, movement and other natural behaviors to characterize individuals’ sentiments from various modes, such as voices, writing, actions and others. Multimodal components refer to the various elements humans employ to express and communicate their emotions. These elements extend beyond mere sounds and encompass a broader range of nonverbal cues: 1) Gaze, the direction of one’s gaze, eye contact and facial expressions can convey a wide spectrum of emotions, including happiness, sadness, anger and confusion. 2) Movement, body movements, such as gestures, posture and facial expressions, provide additional information about an individual’s emotional state. 3) Natural behaviors and actions like smiling, frowning, or crying serve as nonverbal ways to express emotions. 4) Voice, the tone, intonation and volume of one’s voice can effectively convey one’s emotions. 5) Actions, physical actions, such as touching someone or giving a gift, can also communicate emotions. Many studies have characterized such recognition in terms of expressions of emotions (Barrett et al., 2019; Keltner et al., 2019; Millar & Lee, 2021; Uusberg et al., 2019; Van Kleef, 2021; Waterloo et al., 2018), identification of emotions (Davies et al., 2019; Lim et al., 2022; Swain et al., 2018), exploration of emotions (Jeon et al., 2018; Li, Ch’ng, et al., 2018; Li, Song, et al., 2018; Nabi et al., 2018) and experimentation of tools to capture the emergence of emotions (Askool et al., 2019; Athavipach et al., 2019; Mehta et al., 2018). The search for meanings from emotion evolves in tandem with the human need to identify emotional meanings, which tend to be abstract. Based on recent scientific development, emotions are studied not only between human interactions but also human interactions with computers and robots, and vice versa (Kim et al., 2022; Tsiourti et al., 2019).

The research of multimodal emotions has experienced rapid development in recent years. This is also influenced by the power of the 4.0 industrial revolution in which meanings can be interpreted concerning human language and semiotic systems and the assistance of digital tools. Primarily with the assistance of computers (Clayton & Ogunbode, 2023; Jiang et al., 2020; Marechal et al., 2019), the analysis of meanings produced by human emotions and other multimodal resources can be facilitated by digital tool for rendering data to yield enormous amounts of data and clustered data. In utilizing the available tool for rendering data, this study leverages existing datasets to help search for data previously gathered manually (Marshall & Wallace, 2019). Many scholars have conducted similar research to quickly derive findings from specific challenges while maintaining high data accuracy (Boehm et al., 2022; Sharma & Giannakos, 2020) like other research identifying consumer emotions toward the service of a company’s employees and the emotional influence of leaders on their subordinates (Keller et al., 2020; Kumari et al., 2022; Wang & Xie, 2020; Winasis et al., 2021).

Numerous studies have collaborated on numerous techniques employing computers to identify emotions to maintain reliable study outcomes and avoid research bias (Seeber et al., 2020; Zhang et al., 2020). However, the use of machines in recording one’s emotions still suffers from some drawbacks, such as lack of privacy of research subjects.

3. Method

3.1. Design

The design of the search followed the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) standards to gain relevant literature (Jamshidi et al., 2022; Xu et al., 2021). The notion of multimodal emotions in the context of the Industrial Revolution version 4.0 age has become widely known since 2018. This is based on the period of this year when there was a transformation in the merger between information and communication technology.

The identification of gaps and trends was systematically performed based on the number of articles collected, which was then selected based on certain criteria (Figure 1). Due to our lack of access to data, it was decided to select three databases that we considered relevant to the subject of this research.

Figure 1. PRISMA multimodal emotion stages of data collaction.

3.2. Search strategy and sample

Identification is the initial stage of the PRISMA standard. A literature search was conducted with the keywords or stings ‘Multi-modal OR Multimodal OR multi’ AND ‘Emotion’ in accordance with the title of this research. According to Papadakis (2021) and Toorajipour et al. (2021), literature searches should be based on inclusion and exclusion criteria. At this stage, 27,183 scientific publications were retrieved from the IEEE Xplore, Science Direct and Emerald databases. Furthermore, at the screening stage, article selection was carried out using the exclusion criteria by eliminating articles that only contained abstracts, were not written in English, and were a review, meta-analyses, newspapers, essays, reports or book chapters published from 2018 to 2022. At this stage, 277 articles were obtained. Then, double articles were removed, resulting in 147 articles. The next stage is the eligibility stage. In this stage, only full text articles were selected to be the data. Finally, the total number of articles met the criteria was 21 articles. This selection process was also based on the diversity of fields of science, countries, dataset components and emotions generated. In this last stage, each article was reviewed for its consistency of title, abstract, keywords and research methodology, techniques, the field of case studies and the distinguished strategies used. Indicators Aligning with the Research Focus on ‘The Meaning-Making Process Involving Multimodal Emotions in Communication’: 1) Presence of Multimodal Components, does the research under consideration analyze at least two multimodal components (e.g. facial expressions and tone of voice) to comprehend the communicated emotions? 2) Consistency of Multimodal Components, are the analyzed multimodal components (e.g. facial expressions and tone of voice) consistent in conveying the same emotions? Or are there inconsistencies that could affect the intended meaning? 3) Contribution of Multimodal Components, do the analyzed multimodal components contribute to understanding the communicated emotional meaning compared to using only one component? Evaluation of Research Data during the Sample Selection Process: 1) Literature Search, employing relevant keywords and credible database sources; 2) Inclusion and Exclusion Criteria, establishing clear criteria for selecting articles relevant to the research; 3) Phased Selection, applying the established criteria to filter and obtain more focused data.

3.3. Data analysis

After collecting all of the data, the researchers read the data in depth and inserted information from reading regarding categories of multimodal emotional components into a Ms. Excel table. These components included research focus, scientific field, research method and design, data collection tools, research subjects (dataset and age) and emotions (emotional focus, emotional stimulus, theory being applied and emotional experience). The next stage was to interpret the results of each category to find trends from multimodal emotion research over the past five years.

4. Results

4.1. Bibliographical information

This section presents 21 main journal articles (n = 12) and international proceedings articles (n = 9). More articles covering the topic of multimodal emotions were published in 2020 than in the two years before and after (see Figure 2). The most published articles were in 2020 (n = 8), while the least was in 2018 (n = 2), in 2019. There was an increasing trend of publications (n = 5). The trend of multimodal emotion studies decreased in 2021 as well as in 2022 (n = 3). However, this downward trend can be interpreted as an indication that publications in academic repositories related to this topic were experiencing saturation. This conjecture can be seen from the tendency of multimodal emotion studies to focus more on emotion recognition (n = 21) (see Table 1 for the bibliographical information in tables and pie charts).

Figure 2. Bibliographical information.

Table 1. Multimodal research in understanding emotion.

Authors	Participants	Multimodal components	Tools and technology
Arthanarisamy and Palaniswamy (2022), India	32	Audio + Visual + Text	EEG
Bhattacharya et al. (2023), Singapore	Not reported	Audio + Visual	Audio-Video Emotion Recognition (AVER)
Cai et al. (2020), China	10	Audio + Visual	CNN-LSTM model
Caihua (2019), China	10	Audio + Visual	SVM
Chudasama et al. (2022), India	13	Audio + Visual + Text	Multi-modal Fusion Network (M2FNet)
Dahmane et al. (2022), Canada	Not reported	Audio + Visual	CNN and LSTM
Garg et al. (2021), India	Not reported	Audio + Visual	EEG
Ghaleb et al. (2019a), Netherland	7442 clips from 91 actors (43 females and 48 males)	Audio + Visual	Radial basis function (RBF) based support vector machine (SVM)
Ghaleb et al. (2019b), Netherland	Not reported	Audio + Visual	Audio-Video Emotion Recognition (AVER)
Gordon et al. (2018), Australia	Four narrative videos	Audio + Visual	EEG
Guanghui and Xiaoping (2021), China	Not reported	Audio + Visual	Two-dimensional convolutional neural network (2D-CNN) and three-dimensional convolutional neural network (3D-CNN)
Hu et al. (2022), China	256-dimension facial feature vector and 512-dimension emotion feature speech emotion	Audio + Visual	MTCNN (Multi-task Cascaded Convolutional Networks)
Jo et al. (2020), USA	30	Audio + Visual	ROSbag-based
Lee et al. (2020), South Korea	64 Dimensions	Audio + Visual + Text	FFMPEG
Li, Ch’ng, et al. (2018); Li, Song, et al. (2018), China	9.13 m tweets	Audio + Visual + Text	Twitter’s streaming API
Liu and Tan (2020), China	777 Songs	Audio + Text	Late fusion method (LFSM)
Priyasad et al. (2020), Australia	Five sessions of utterances for 10 unique speakers	Audio + Visual	Deep Convolutional Neural Networks (DCNN) and Recurrent Neural Networks (RNN)
Song et al. (2019), China	28 Videos from 23 participants	Audio + Visual	A-LSTM, LSTM, SVM and KNN
Sun et al. (2020), China	291 Videos and 70 host speakers collected from YouTube.	Audio + Visual + Text	LSTM
Tung et al. (2019), Taiwan	528 (33 subjects × 16 videos)	Audio + Visual	EEG, ECG and GSR
Zhang et al. (2020), China	12 h of audiovisual data from 10 actors, including video, voice, facial motion capture and text transcription.	Audio + Visual + Text	LSTM, DenseNet and GloVe.

Table

Authors	Dataset	Focus of emotions
Arthanarisamy and Palaniswamy (2022), India	DEAP	Not reported
Bhattacharya et al. (2023), Singapore	CREMA-D and RAVDESS	Positive + negative
Cai et al. (2020), China	IEMOCAP	Excite, angry, aad and neutral.
Caihua (2019), China	Not reported	Anger, boredom, disgust, fear (Anxiety/Fear), happiness (sadness) and sadness, neutral.
Chudasama et al. (2022), India	MELD and IEMOCAP	(Anger, Disgust, Sadness, Joy, Surprise, Fear and Neutral).
Dahmane et al. (2022), Canada	One-Minute Gradual (OMG) 2018	Negative emotions
Garg et al. (2021), India	AMIGOS	Not reported
Ghaleb et al. (2019a), Netherland	eNTERFACE and CREMA-D	Anger, disgust, fear, happy, neutral and sad
Ghaleb et al. (2019b), Netherland	CREMA-D and RAVDESS.	Not reported
Gordon et al. (2018), Australia	Not reported	Not reported
Guanghui and Xiaoping (2021), China	RML, eNTERFACE05, BAUM-1	Not reported
Hu et al. (2022), China	Not reported	Angry, disgust, fear, happy, sad, surprise and neutral
Jo et al. (2020), USA	International Affective Pictures System (IAPS) and the International Affective Digitized Sounds (IADS)	Happiness, sadness, anger, fear, surprise, disgust and neutral
Lee et al. (2020), South Korea	TV series	Angry, disgust, fear, happy, neutral, sad and surprise
Li, Ch’ng, et al. (2018); Li, Song, et al. (2018), China	Twitter	Positive + negative
Liu and Tan (2020), China	Music Mood Classification Data Sets	Exuberance, anxious, contentment and depression
Priyasad et al. (2020), Australia	IEMOCAP	Anger, happiness, neutral and sadness
Song et al. (2019), China	Not reported	Positive emotions and negative emotions
Sun et al. (2020), China	MuSe-CaR	Arousal and valence
Tung et al. (2019), Taiwan	AMIGOS	Positive emotions and negative emotions
Zhang et al. (2020), China	IEMOCAP	Anger, excitement, happiness, neutrality and sadness

Table

Authors	Scientific field	Model of inquiry process
Arthanarisamy and Palaniswamy (2022), India	Psychology	Investigation
Bhattacharya et al. (2023), Singapore	Psychological	Explore
Cai et al. (2020), China	Computer	Development
Caihua (2019), China	Language	Identifying
Chudasama et al. (2022), India	Language	Development
Dahmane et al. (2022), Canada	Psychology	Investigation
Garg et al. (2021), India	Psychology	Investigation
Ghaleb et al. (2019a), Netherland	Education	Investigation
Ghaleb et al. (2019b), Netherland	Psychology	Investigation
Gordon et al. (2018), Australia	Psychology	Investigation
Guanghui and Xiaoping (2021), China	Psychology	Investigation
Hu et al. (2022), China	Computer	Investigation
Jo et al. (2020), USA	Human-computer interaction (HCI)	Investigation
Lee et al. (2020), South Korea	Psychology	Investigation
Li, Ch’ng, et al. (2018); Li, Song, et al. (2018), China	Psychology	Investigation
Liu and Tan (2020), China	Music	Investigation
Priyasad et al. (2020), Australia	Music	Investigation
Song et al. (2019), China	Psychology	Investigation
Sun et al. (2020), China	Psychology	Investigation
Tung et al. (2019), Taiwan	Psychology	Investigation
Zhang et al. (2020), China	Psychology	Identifying

4.2. Scientific information

This section reviews general information about the collected articles, namely the categories of research methods used, the inquiry process model applied, the research participants involved and the scientific fields widely researched. Of the 21 primary studies, the research design of all data used quantitative methods with a model of the inquiry process, namely investigation (n = 16), identifying (n = 3), exploration and research development, each amounting to one article (n = 1). In terms of research subjects, the types of data significantly influenced the choice of the research subjects. Of the 21 data, various amounts of those were found, namely: data from the videoed speech from ≤ 10 participants (n = 4), speech from >10 participants (n = 5), speech from 20 to 50 participants (n = 2), speech > 50 participants (n = 3) and data taken other than participants (n = 7), such as, from tweets, songs and time duration (hours). The collected multimodal emotion articles highlighted the recognition of emotions using machines as a tool to help analyze data derived from the datasets. Over the past five years, the study of multimodal emotions has spread across diverse disciplines. If seen specifically, this knowledge is spread over several fields of science (see Figure 3). The most dominant science is psychology (n = 13).

Figure 3. Scientific information.

Discussion of multimodal emotions is indeed a target in the field of psychology. This is related to the realm of emotion, which is closely related to psychology. Other fields that have been widely researched are music and language, as well as computers with the same amount of data (n = 2). Other scientific areas that are the focus of research on using multimodal emotions are the fields of human–computer interaction and education with the same number (n = 1) (see Figure 3 for the details of description).

4.3. Research locations

The distribution of research locations (see Figure 4) shows that the dominant multimodal emotions studies occurred in China (n = 8). This happened because of China’s connection as the source of the spread of COVID-19, in 2019. Meanwhile, research locations in other countries were evenly spread, resulting in disparity in one or two articles. The data show that the study of multimodal emotions has piqued the interest of researchers from China, India, the Netherlands, Australia, Canada, Singapore, South Korea, Taiwan and the USA.

Figure 4. Frequency of multimodal emotional research by research locations.

4.4. How can the different multimodal components be analyzed and categorized?

The literature indicates that the multimodal components of emotion used three modes: audio + visual + text. These three modes have become the research trends for the last five years. It has been identified that 14 out of 21 data (66.67%) used audio + visual modes. This trend of multimodal selection supports common ways of humans using multimodal means involving emotions (Bhattacharya et al., 2023; Cai et al., 2020; Caihua, 2019; Dahmane et al., 2022; Garg et al., 2021; Ghaleb et al., 2019a, 2019b; Guanghui & Xiaoping, 2021; Hu et al., 2022; Jo et al., 2020; Priyasad et al., 2020; Song et al., 2019; Tung et al., 2019).

The choice of audio + visual turns out to be dominant as one of the supporting tools to observe emotional expressions (see He et al., 2022; Kossaifi et al., 2019; Ma et al., 2019; Wang & Xie, 2020). Individuals’ empirical feelings can barely be identified, but emotional expressions can be observed through expressions (Ruba & Repacholi, 2020; Zerback & Wirz, 2021). Therefore, according to those studies, a tool or technology is needed to identify and carry out stages of multimodal components. In this stage, each of the 21 data was given a code. Next, the mode frequency or modality components are calculated to indicate how many of the 21 data used multimodal components (see Figure 5 for the visualization of the results). Based on the investigation of each of the data, the trend of multimodal components shows that the selection of audio + visual + text components was 28.57%, and the selection of the audio + text mode was 4.76%.

Figure 5. Frequency of multimodal emotions research by multimodal components.

4.5. What tools and technologies can be used to obtain multimodal emotions data?

The literature indicates that the researchers used machines to identify tools and technologies for data acquisition over the past five years. As analyzed in the research reports, the use of machines was influenced by the Industrial Revolution version 4.0, which began to touch the virtual world (Egger et al., 2019). More specifically, according to Egger, humans, machines and data are inseparable from making everything available everywhere, significantly influencing the meaning-making process involving multimodal emotions. Thus, the study of emotions has morphed into multimodal research (Azmi et al., 2022; Ninaus et al., 2019; Zhang et al., 2020). In exploiting the tools and technologies for conducting multimodal emotion research, the researchers used 6 out of 21 data (66.67%) from LSTM technology to identify emerging emotions (Cai et al., 2020; Dahmane et al., 2022; Li, Ch’ng, et al., 2018; Li, Song, et al., 2018; Song et al., 2019; Sun et al., 2020; Zhang et al., 2020). More specifically, emotion identification based on subsequent multimodal research using EEG tools totaled 5 data (23.81%), while other tools or technologies used in the emotions research included AVER, DCNN, FFMPEG, LFSM, M2 Net and SVM (see Figure 6).

Figure 6. Frequency of multimodal emotions research by tools and technologies.

4.6. What datasets are available for analyzing multimodal emotions and what are their main features?

The use of datasets in the form of raw data presented in tabular form that can be further processed in multimodal emotions research, has become a trend over the past five years (Bayoudh et al., 2022; Gandhi et al., 2022). In general, the dataset is owned by certain institutions or agencies granting permission to use it for scientific purposes (Deinsberger et al., 2020).

The third research question addresses the use of datasets in multimodal emotions research. The use of datasets is based on different research purposes. However, the literature analysis shows that the entire research (n = 21) used tools and technology affecting the use of data. Of the 21 data, a total of 17 data (80.95%) used the dataset as the research object. Four studies (19.05%) were identified to use the IEMOCAP dataset (Cai et al., 2020; Chudasama et al., 2022; Priyasad et al., 2020; Zhang et al., 2020). With the same percentage (19.05%), the researchers did not use the dataset. Next, the most frequently used datasets are CREMA-D and RAVDESS (14.29%). The rest used the related and adapted datasets to the research objectives, such as the use of datasets originating from TV series and Twitter (see Figure 7).

Figure 7. Frequency of multimodal emotions research by dataset.

4.7. What is the focus of emotions available to gain better multimodal accuracy?

The literature reveals the focus of emotions research, that is, researchers’ goal in examining emotions by using a multimodal approach. The investigation into the focus of emotions research was carried out to find out what emotions are being investigated and how often certain emotions become the focus of the researchers. Nine of the 21 articles (42.86%) made positive, negative and neutral emotions to be identified for research purposes (Cai et al., 2020; Caihua, 2019; Chudasama et al., 2022; Hu et al., 2022; Jo et al., 2020; Priyasad et al., 2020; Zhang et al., 2020). Researchers’ second most frequent focus was on positive and negative emotions (23.81%). The analysis of the literature indicates the same number of researchers as the second most frequent focus, which did not mention the focus of emotions (23.81%). Meanwhile, one research article (4.76%) was found to focus on negative emotions (Table 2).

Table 2. Frequency of multimodal emotions research by focus of emotions.

Focus of emotions	Frequency	Percentage (%)
No mentioned	5	23.81
Positive and negative emotions	4	19.05
Negative emotions	1	4.76
Excited, angry, sad and neutral.	1	4.76
Angry, bored, disgusted, scared, happy, sad and neutral	1	4.76
Angry, disgusted, sad, happy, shocked, scared and neutral	1	4.76
Angry, disgusted, scared, happy, neutral and sad	1	4.76
Angry, disgusted, scared, happy, sad, shocked and neutral	1	4.76
Happy, sad, angry, scared, shocked, disgusted and neutral	1	4.76
Angry, disgusted, scared, happy, neutral, sad and shocked	1	4.76
Happy, worried, satisfied and depressed	1	4.76
Angry, happy, neutral and sad	1	4.76
Arousal and valant	1	4.76
Angry, glad, happy, neutral and sad	1	4.76

See Table 3 for illustration and identification of emotions focus.

Table 3. Frequency of multimodal emotions research by emotions focus cluster.

Emotion focus cluster	Frequency	Percentage (%)
Arousal and valant	1	4.76
Negative emotions	1	4.76
Positive and negative emotions	5	23.81
Positive, negative and neutral emotions	9	42.86
No reported	5	23.81

Based on the review of the literature, as seen in Table 3, almost all of the researchers focused on positive, negative, and neutral emotions (50%) for the last five years. The researchers intended to study emotions thoroughly and comprehensively (see Li et al., 2022; Bashir et al., 2021). Some researchers did not mention the emotional focus of their research objectives (such as Arthanarisamy & Palaniswamy, 2022; Garg et al., 2021; Ghaleb et al., 2019a, 2019b; Gordon et al., 2018).

4.8. Summary of findings from systemic literature review

The results of bibliographical information on emotional research show that there has been quite a decrease in the percentage of approximately 24% from 2020 to 2021. However, it does not mean that the study of multimodal emotions is no longer of interest to researchers. This is because many studies on multimodal emotions are still being published in a wider scope of publications.

The theoretical analysis used in this study focused on a multimodal framework employing machines. This focus brings a consequence of tapping the same topics over and over again, notwithstanding the researchers’ revisions in some portions. Thus, the search leads to the observation of semiotic modes, which are significantly influenced by technological advancement (Singh, 2019). In this investigation, the researchers capitalized on the role of digital devices and machines in their multimodal study.

This analytical procedure was undertaken to provide a systematic and comprehensive review of multimodal emotions research. As previously described about multimodal components in examining emotions, audio + visual ranks first as the mode researchers used most frequently. This is because the majority of researchers used machines to analyze emotions. In addition, the audio + visual facilities in the video can evoke emotions naturally and in various ways (Bagila et al., 2019; Ismail et al., 2021; Pandeya & Lee, 2021). For example, if someone watches a film (audio + visual), at the same time, it will bring up emotions as a result of watching the film. The arising emotions are then captured by a machine or tool, which produces output in the form of whatever emotions arise.

It was found from the literature review that utilizing tools as expressions of emotion recognition experienced by humans has been most frequently used for multimodal emotional data acquisition over the past five years. This system is known to be more economical to use for certain purposes (Dzedzickis et al., 2020; Kim et al., 2019; Mittal et al., 2020). The desired results are achieved because low-cost equipment can easily capture the data needed to assess emotions (Laureanti et al., 2020). Furthermore, machines can also analyze large and varied amounts of data (Ngiam & Khor, 2019; Sarker, 2021).

In those studies, the analysis of the tools and technology used more frequently psychologically sensory nature to capture and identify emotions. For example, long short-term memory network (LSTM) was used as a tool to memorize a collection of information that has been stored for a long time, as well as delete information that is no longer relevant. LSTM is more efficient in processing, predicting, as well as classifying data based on a certain time sequence. Those studies also identified another technology: the EEG sensor, also known as Electroencephalography, which is a method for recording the brain’s electrical activity on the surface of the scalp (Beniczky & Schomer, 2020). The EEG records the fluctuations in the electrical potentials that occur as a result of the activity of brain cells. As research progresses and measurement tools become more sophisticated, EEG is increasingly being used in research on cognitive function. Although it provides the advantages of using machines in multimodal emotional research, it still shows drawbacks, such as lack of subject privacy and uncontrollable research bias (Bhattacharya et al., 2023).

As seen in Figure 7, most researchers focused on datasets containing recorded dialog datasets in the form of videos. The use of video included an audio + visual tool that supported the expression of emotions. for example, the The Interactive Emotional Dyadic Motion Capture (IEMOCAP) Dataset from the University of Southern California consists of >150 video recordings of male and female dialog, with two speakers per session, with a total of >300 sessions for each speaker. The researchers used IEMOCAP more frequently than other datasets because they found a more varied range of emotions than other datasets. This diversity of emotions was needed to identify the emergence of what emotions were generated based on gender and time. Systematically, this research also produced data that did not use datasets as research objects to identify emotions in multimodal emotions research. Four studies did not use a dataset because the researchers of the studies used adults as objects to identify the emotions that arose after being given an action. For example, someone who was watching a video simultaneously had an EEG device installed. Next, the EEG would analyze and bring up any visible emotions from that person. Those researchers only intended to identify the emotions generated after being given treatment. On the other hand, most of the researchers have only reached the identification stage, without any attempts to reach the emotional meaning stage. Meanwhile, the meaning of emotion is no less important to study, considering that emotion has an abstract meaning depending on the cause and effect of its appearance, context and even the speakers’ cultures.

Furthermore, in almost every emotion data, anger (90.48%) is always the focus of emotion and attention. The appearance of angry emotions as negative emotions in research can appear to research’s demand to understand the meaning of this emotion from many aspects, such as biological, emotional, intellectual, social and spiritual aspects (Jakovljević & Jakovljević, 2021). Other emotions rarely own the complexity of this aspect. In addition, anger is an acute emotional reaction that arises from several stimulating situations, such as threats, self-restraint, verbal attacks, disappointment and frustration (Muhajarah, 2022; Mufid et al., 2021).

Technological tools in the form of artificial intelligence (AI) and machine learning (ML) widely aid humans with their benefits in the workforce, where the tools can be used to help leaders understand their employees’ emotions regarding decisions on salary increases, job promotions and assignments. Furthermore, these technological tools can be used to come across a person’s tendency to commit suicide, detect drunk drivers, observe a person’s health when entering a country or help in the credit-granting system. AI and ML are also used by certain parties in the court decision-making process in selecting job applicants and assigning academics in the education system. This is, however, different from the use of AI and ML in the emotional realm. Although several studies reveal that AI and ML can read the emotions produced by a person, the result cannot be used as an absolute decision. The meaning of emotions must be based on the context that supports an emotion before and after the emotion is produced. This is what AI and ML cannot translate or define. When AI and ML are asked about when, how, and where certain emotions are generated, these cannot be answered and defined by AI and ML.

5. Conclusion

Multimodal emotion research has gained significance for its contribution to the meaning-making process (Illendula & Sheth, 2019; Marechal et al., 2019). As reviewed in this study, we focused on bibliographical information, methods and scientists, as well as general sections of multimodal emotions, i.e. multimodal components, data acquisition tools and technologies, datasets and emotion focus, to see the trend and directions of multimodal emotions research. Some highlighted points include a focus on the multimodal components; this literature review can affirm the choice of researchers, the majority of the audio + visual components in the research, influenced by research objectives and use of datasets, and data acquisition tools. Regarding the data acquisition tools and technology used in multimodal emotion research, all research (100%) used machines facilitated through easy access to digital tools in the Industrial Revolution 4.0, which began in 2018. Low costs, short processing time and large and varied quantities of data have triggered the use of machines. However, the use of machines shows some drawbacks, such as the absence of privacy for research subjects and the lack of research bias control.

In terms of datasets, the majority of multimodal research trends used datasets (80.95%), which is in line with the use of machines for data acquisition. In terms of emotional focus, many researchers focused on investigating positive, negative and neutral emotions (42.86%). Of all the emotions, anger gained the highest focus (90.48%). The focus on emotions only reached the identification stage. The findings can provide an initial reference to the prudent design of psychology and pedagogy and can be applied in a more appropriate strategy to optimize all modes and generate positive emotions in every learning process.

Based on the literature review, by taking into account the general components of emotional multimodal research, this study suggests that the researchers examine emotions using more modes. The more modes are used, the more comprehensive and higher quality the results will be. Further, more variety of modes can minimize bias in research (Bhattacharya et al., 2023). It is important to note that choosing a mode or component should also be adjusted to the research objectives and objects. If the subject’s conditions do not permit, the audiovisual mode is highly recommended (Bhattacharya et al., 2023; Cai et al., 2020). A Holistic Perspective on Multimodal Emotion Components: 1) Analyzed Components: Current research typically focuses on audiovisual components (sound and visuals) for emotion analysis. This choice is influenced by research objectives and the ease of access to tools and technologies driven by the Industrial Revolution 4.0 era. 2) Limitations of Components: Relying solely on audiovisual components has its drawbacks. This research suggests analyzing emotions using a wider range of multimodal components to achieve more comprehensive and high-quality results. 3) Component Variation: The greater the variety of multimodal components employed, the lower the likelihood of bias in research. Some studies suggest that component selection should be tailored to the research aims and subjects. 4) Recommended Components: If research subjects are unable to utilize multiple components, audiovisual components remain highly recommended due to their user-friendliness.

We acknowledge that this research has substantial limitations. Lack of access in obtaining databases, we only limit to three databases namely IEEE Xplore, Science Direct and Emerald. Therefore, future research is expected to add more and diverse database sources, such as Scopus Elsavier, PubMed and DOAJ. Deeper investigations on multimodal emotions can be conducted using larger results and interpretations from multiple databases.

This study also recommends that other researchers should be able to examine emotions not only on basic positive, negative, and neutral emotions but also on more derivative emotions. It is even better if multimodal emotions research examines the meaning of emotions reaching beyond the identification stage. Furthermore, researchers can examine multimodal emotions research with a focus on the construction of meaning generated by emotions from different regions or countries. This construction certainly influences cross-cultural knowledge from the countries/regions. Another implication is that future researchers can compare several data acquisition tools and technologies to find a more representative tool to analyze the resulting emotions.

Author details

Reny Rahmalina*

E-mail: [email protected]

ORCID ID: https://orcid.org/0000-0003-1016-7448

Wawan Gunawan

E-Mail: [email protected]

ORCID ID: https://orcid.org/0000-0003-1792-8350

Linguistics Study Program, Postgraduate School, Universitas Pendidikan Indonesia, Indonesia.

Correction

This article has been corrected with minor changes. These changes do not impact the academic content of the article.

Citation information

Cite this article as: What multimodal components, tools, dataset and focus of emotion are used in current research of multimodal emostion: A systematic literature review, Reny Rahmalina and Wawan Gunawan, Cogent Social Sciences.

Disclosure statement

The author(s) declared no potential conflicts of interest with respect to the research, authorship and/or publication of this article.

Additional information

Funding

The research is financed by Indonesia Endowment Funds for Education (LPDP), Center for Higher Education Funding (BPPT). BPI ID 202101121154.

Notes on contributors

Reny Rahmalina

Reny Rahmalina is a doctoral student in the linguistics program at the graduate school, Universitas Pendidikan Indonesia (Bandung, Indonesia). Her research interests include linguistics studies, innovation, education and teaching. Her works have been published in national and international journals, such as Atlantis press and others.

Wawan Gunawan

Wawan Gunawan, M.Ed., Ph.D., is a lecturer in the linguistics doctoral program at the graduate school, Universitas Pendidikan Indonesia. One of the fields he has been focusing on in recent years is multimodal studies. Based on the discovery of the close relationship between emotion and multimodality, we decided to investigate how emotion and multimodality components complement each other. We collaborated to accomplish this work. Future research is expected to focus on the meaning of the emotions generated. The use of data from datasets requires consideration of subject privacy and resulting bias.