Can Preschoolers Recognize the Facial Expressions of People Wearing Masks and Sunglasses? Effects of Adding Voice Information

ABSTRACT

Early childhood is marked by significant developmental changes in the ability to recognize facial expressions. However, since the COVID-19 outbreak, people have been wearing masks more frequently during social interactions which may hamper the recognition of facial expressions. This study examines whether preschoolers recognize the facial expressions of people with partially covered faces (wearing masks or sunglasses) or uncovered faces better, and whether recognition improves when additional voice information is provided. The participants included 27 Japanese preschoolers (11 boys and 16 girls) aged 3–5 years. The participants were presented with two groups of facial expressions: stimuli showing uncovered faces and those showing faces partially covered with a mask or sunglasses. A two-factor within-participant analysis of variance was conducted on the number of correct facial-expression responses in each trial. The children recognized the expressions of uncovered faces significantly better than those of faces with masks or sunglasses. When voice information was added, they recognized all facial expressions. Therefore, partially covered faces interfere with preschoolers’ recognition of facial expressions, and voice information aids facial expression recognition.

Introduction

In social interactions, people often try to read others’ thoughts and emotions, and convey their feelings to communicate smoothly. Facial expressions are particularly easy for children to read; they develop the ability to read these cues earlier than voices, situations, or body movements (Nelson & Russell, 2011; Quam & Swingley, 2012). Approximately half of all children aged 2 years can recognize the facial expressions of happiness, sadness, anger, and fear, and most can do so by the age of 5 years (Pons, Harris, & de Rosnay, 2004). Thus, early childhood (mainly 2–5 years of age) is marked by significant developmental changes in the ability to recognize facial expressions (Widen & Russell, 2008).

However, we cannot always see another person’s entire face. Since the COVID-19 outbreak, people have been wearing masks more frequently. Although mask-wearing is effective in preventing the transmission of infectious diseases, masks may hamper the recognition of facial expressions. When worn properly, a mask hides the lower part of the face, including the nose and mouth (60–70% of the entire face), making facial expression recognition difficult (Freud, Stajduhar, Rosenbaum, Avidan, & Ganel, 2020). Moreover, as facial expression recognition relies on information taken from the entire face (Farah, Wilson, Drain, & Tanaka, 1998; Maurer, Le Grand, & Mondloch, 2002), masks qualitatively alter how the partial face is recognized (Freud, Stajduhar, Rosenbaum, Avidan, & Ganel, 2020).

Therefore, how do people recognize facial expressions when only part of the face is visible? The eyes are the most noticeable facial features (Grossmann, 2017), and people initially recognize others’ facial expressions by looking at their eyes (Eisenbarth & Alpers, 2011; Scheller, Büchel, & Gamer, 2012). However, the eyes alone are insufficient, and attention is shifted to other features, such as the mouth and eyebrows, to obtain additional information (Leitzke & Pollak, 2016). Not all emotions are perceived similarly, and some parts of the face are more easily perceived depending on the type of emotion displayed (e.g., the eyes for sadness and anger and the mouth for happiness). However, each emotion is judged based on a synthesis of information obtained from various facial features (Bombari et al., 2013; Eisenbarth & Alpers, 2011). Thus, adults make judgments based on information gathered from the visible portions of a partially covered face; however, information regarding how this process manifests in preschoolers is scarce. Several studies have shown that children can identify basic emotions from the eyes alone by the age of 3 years (Franco et al., 2014) and from the eyes or mouth alone by 5–7 years of age (Gagnon, Gosselin, & Maassarani, 2014).

When a mask or shawl obscures the mouth, pleasure and fear, characteristically expressed via the mouth, become difficult to read (Bombari et al., 2013; Eisenbarth & Alpers, 2011). Ruba and Pollak (2020) examined the level of emotion recognition from faces without coverings, with masks, and with sunglasses in American children aged 7 to 13 years. No difference was found between the recognition of faces covered with masks and sunglasses; however, while the children responded to all conditions with above-chance accuracy, recognizing expressions on covered faces was more difficult than those on uncovered faces. A study conducted in Germany on adults aged 18–87 years showed similar results (Carbon, 2020). Thus, mask-wearing appears to impair facial expression recognition, regardless of age.

Based on these results, Gori, Schiatti, and Amadeo (2021) examined how well preschoolers who have developed the ability to recognize facial expressions can read them when the faces are covered with masks, and whether there are differences in the facial expression recognition of Italian toddlers (3–5 years), children (6–8 years), and adults (18–30 years) with and without masks. They found that masks decreased the percentage of correct responses for all ages. The results differed significantly for toddlers and revealed that facial expression recognition develops throughout childhood; preschoolers are more susceptible to the influence of masks on their ability to recognize facial expressions. The performance on the no mask condition was approximately 70% for toddlers and children, indicating that they could not recognize approximately 30% of the trials even without masks. Therefore, when they could not understand facial expressions in the trials with masks, it remains unclear whether the mask effect is significant or crucial in recognition. To focus on the mask effect, easy tasks are suitable for children.

Moreover, cultural differences affect distinguishing facial expressions. Yuki, Maddux, and Masuda (2007) found that Japanese people tend to interpret emotions based on others’ eyes, whereas American people do so based on others’ mouths. Therefore, when only part of the face is visible, there may be cultural differences in facial expression recognition. For example, Japanese people experience greater difficulty interpreting emotions from faces without eye information than American people. Jack, Blais, Scheepers, Schyns, and Caldara (2009) identified that East Asians use a culture-specific decoding strategy that is inadequate to distinguish the facial expressions of fear and disgust. East Asians persistently fixate on the eye region of other’s faces, whereas Western Caucasians fixate across the face.

Furthermore, children also exhibit cultural strategies of facial expression recognition. Senju, Vernetti, Kikuchi, Akechi, and Hasegawa (2013) reported that Japanese and British children aged 1–7 years show culture-specific eye movement patterns on face scanning. Geangu et al. (2016) found that Japanese and British 7-month-old infants rely on a cultural strategy to discriminate facial expressions. Moreover, Haensel, Ishikawa, Itakura, Smith, and Senju (2020) suggested that Japanese and British 10- and 16-month-old infants show similar cultural tendencies on face scanning as adults. Thus, cultural tendencies regarding facial expression recognition might be consistent across age groups.

Li, Liu, Li, Qian, and Dai (2020) reported that 63% of Japanese people wore face masks in public during the COVID-19 pandemic, whereas North Americans and Europeans discouraged healthy people from wearing face masks in public. Nakayachi, Ozaki, Shibata, and Yokoi (2020) suggested that, when wearing face masks, Japanese people feel that they are conforming to social norms and are less anxious. Therefore, more Japanese people wear face masks in public than those in the West. However, while the rate of wearing sunglasses in public in Japan is considerably low (Ng & Ikeda, 2011), sunglasses are the most used measures for sun protection among Americans and Europeans (Seité, Del Marmol, Moyal, & Friedman, 2017). Thus, these cultural differences in daily life and cognitive tendency might affect facial expression recognition with masks and sunglasses. Specifically, Japanese children might have more difficulty recognizing others’ facial expressions with sunglasses than with masks. Therefore, investigating how Japanese children recognize facial expressions of faces covered with masks and sunglasses is crucial.

Peoples’ emotions can still be read even if their facial expressions cannot be seen because emotions are not exclusively expressed through a single piece of information; facial expressions, voice, and posture play vital roles in emotion recognition (Planalp, Defrancisco, & Rutherford, 1996). When judging others’ emotions, we use varied information expressed by them (Schirmer & Adolphs, 2017). Visual and auditory information are frequently used, thus suggesting that voice information provides important cues for emotion recognition (Mehrabian, 1986). Vroomen and de Gelder (2000) found that people use both visual and auditory information, even when they are instructed to ignore such information, when it is inconsistent. Quam and Swingley (2012) found that children aged 4–5 years interpret exaggerated, stereotypically happy or sad pitch cues.

Regarding cultural differences, Japanese people tend to inhibit their facial expressions (Matsumoto, Takeuchi, Andayani, Kouznetsova, & Krupp, 1998; Yuki, Maddux, & Masuda, 2007) and focus on auditory information (Tanaka et al., 2010). Tanaka et al. (2010) found that the effect of to-be-ignored voice information on facial judgments was larger in Japanese samples than in Dutch samples. Ishii, Reyes, and Kitayama (2003) reported that Japanese people showed greater difficulty ignoring vocal emotional tone than verbal content, whereas American people showed greater difficulty ignoring verbal content than vocal emotional tone. Hence, Japanese people are more attuned to vocal processing in the multisensory perception of emotions.

Against this background, it is clear that masks and sunglasses affect facial expression recognition in Western countries (cf. Gori, Schiatti, & Amadeo, 2021; Ruba & Pollak, 2020) where people have cultural tendencies to focus on others’ mouths in facial expression recognition and are accustomed to wearing sunglasses in public. However, we have little knowledge regarding how masks and sunglasses affect facial expression recognition in countries where people have cultural tendencies to focus on others’ eyes in facial expression recognition and are accustomed to wearing masks in public. Moreover, the kind of information used to help us to recognize others’ emotions when we cannot use entire facial information is unclear. In this study, we recruited Japanese preschoolers as participants to investigate how masks and sunglasses affect their facial expression recognition. In addition, we presented vocal cues with facial stimuli to determine whether participants consider voice information as a cue to understand others’ emotions when they cannot use entire facial information. We hypothesized that Japanese preschoolers have greater difficulty recognizing the facial expressions of people wearing masks or sunglasses compared with those of people wearing neither (Hypothesis 1A). Regarding cultural background, we hypothesized that Japanese preschoolers can recognize the facial expressions of people wearing masks more easily than those of people wearing sunglasses (Hypothesis 1B). Further, based on Quam and Swingley (2012), we hypothesized that presenting voice information with facial expressions facilitates Japanese preschoolers’ recognition of the facial expressions of people wearing masks or sunglasses (Hypothesis 2).

Methods

Transparency and openness

In this section, we report our sample size determination, any data exclusions, manipulations, and measures, according to the Journal Article Reporting Standard (Kazak, 2018). Data were analyzed using SPSS version 28. The design and analysis of this study were not pre-registered.

Participants

Prior to data collection, we conducted a priori power analysis (G*Power; Faul, Erdfelder, Lang, & Buchner, 2007). It showed that the number of participants needed to achieve a medium to large effect size within factors using a repeated measures analysis of variance with f = 0.35, power of 1-β =.80, and a two-tailed α = .05, with an average correlation of .5 among repeated factors, was 25 per group. Therefore, we recruited 27 Japanese preschoolers (mean age 67.48 months, range 48–70 months; 11 boys and 16 girls) attending a university-affiliated kindergarten.

Materials

To examine the participants’ ability to recognize facial expressions, we prepared a facial expression and voice stimuli, presented on a laptop with Windows 10.

Facial expression stimuli

We used pictures of four Japanese adults, two men and two women, showing four emotions: happiness, sadness, anger, and surprise. These four emotions were assumed to be recognizable by preschoolers based on previous findings regarding children’s difficulty in reading expressions of fear (Ruba & Pollak, 2020). Stimuli from one person of each sex were used to confirm the recognition of the emotion words, and the remaining stimuli of the other person from each sex were used for the trial. We prepared 24 stimuli (four emotions in three types of coverings; uncovered, masked, and wearing sunglasses) for each sex. The masks, clothing, and backgrounds were presented in the same neutral white color. Furthermore, the photos were taken indoors under natural light. For the photos used in the mask trials, the models wore masks; for the sunglasses trials, sunglasses were added to the photos with no facial covering. All the photos were square-shaped, displaying the model from their head to neck.

Voice stimuli

For the voice stimuli, we used “ohayou” (“good morning” in Japanese) uttered by one male and one female university student, different from the models of the image stimuli, using the same four emotions as those depicted in the facial expression photos. This phrase was chosen for its neutrality and lack of emotional connotations. Moreover, it can be uttered with various emotions and is familiar to preschoolers. The recordings were made in a quiet environment where the models were asked to think of each emotion and record the audio, which was subsequently edited to 1.5 seconds before use.

For the facial expression photos and voices, a preliminary survey was conducted among 12 university students, and stimuli receiving more than 75% correct responses were used. Additionally, as the preliminary survey showed no difference between how the voice stimuli were heard depending on whether a mask was worn during the voice recording, voice recordings without a mask were used for all covering types.

The stimuli were created using PowerPoint to present photos of the four emotions as one set (Figure 1). When presenting each set of four photos, each photo was highlighted individually in succession. In the voice-added condition, the program was designed to play the voice stimulus corresponding to the facial expression simultaneously with the target image highlighted for each stimulus type. Additionally, the stimuli were presented randomly.

Figure 1. Facial expression stimuli used in the study.

Procedure

The surveys were conducted individually. The experimenter and the participant sat across the desk facing each other. After rapport building, the confirmation, practice, and main trials were conducted.

Confirmation trial

The confirmation trial was conducted to ensure that participants understood the emotion-related words and their meanings. On the computer screen, four pictures representing four unmasked facial expressions were presented individually, and each participant was asked to respond freely to the question “How do you think this person is feeling?” If a participant could not answer correctly, they would have been provided with a prompt. However, all participants answered correctly.

Practice trial

The practice trial was conducted to familiarize children with responding during the main trial. Four photos of emotionally expressive faces were presented on one screen, and the participants were told, “You will see four images of faces in order, sometimes with a voice. You will be asked to choose a face with a voice and a happy face, so watch and listen carefully.” Thereafter, the images were presented in order for two seconds each. When one of the four image stimuli was presented, a neutral voice saying “ohayou” was played. Afterward, the four photos were presented again, and the following questions were asked: (i) “From which face did you hear the voice?” and (ii) “Which face looks happy?” For each question, the children were asked to select one answer by pointing to it.

The same procedure was followed for all three trials (uncovered, mask, and sunglasses covering types); if there was no response or an incorrect response, the correct answer would have been revealed to inform participants how to answer. However, all participants answered correctly.

Main trial

The main trial was conducted similarly to the practice trial. Before presenting the stimuli in the main trial, participants were instructed, “You will see four faces individually, and when you have seen all four, you will choose a happy (sad, angry, or surprised) face, so watch carefully until the end.” In the voice-added condition, four voice stimuli of “ohayou” with four different emotional cues were used; that is, an image stimulus was presented together with a matching emotional voice stimulus. After the four stimuli were presented, the four photos were presented again, and the participants were asked to answer the question, “Which of these faces is a happy (sad, angry, or surprised) face?” The facial expression stimuli presented were matched to the participants’ sex. There were 12 voiceless trials (four emotions of three covering types) and 12 voice-added trials, presented in a random order. The cumulative score was on a scale of 0–4 for each covering type; 1 point was awarded for each correct response and 0 for each incorrect response.

Ethical considerations

Two of the authors obtained approval from the Ethics Committee of Research Involving Human Subjects at their respective institutions before conducting the study. We obtained informed consent from the person in charge of the kindergarten rather than from the participants’ parents because the university-affiliated kindergarten is also a research field for the institution, and the Ethics Committee of Research Involving Human Subjects at the authors’ institution specified that informed consent can only be obtained from the person in charge. Additionally, we obtained informed consent from the participants.

Results

We checked for a repetition effect because participants watched the same facial stimuli 24 times; however, no significant repetition or order effects were found (n.s.). Therefore, all participants were included in the final data analysis.

Figure 2 shows the average percentage of correct responses for each covering type × voice condition. A t-test for chance level (25%) was conducted for each of the other conditions; the percentage of correct responses was significantly higher than chance in all conditions (p < 0.001).

Figure 2. Average percentage of correct responses for stimulus type with and without voice.

Furthermore, a two-factor within-participant analysis of variance was conducted on the number of correct responses. The main effects of stimulus type (F (2, 52) = 13.04, p < 0.001, η_p² = .33) and voice (F (1, 26) = 18.09, p < 0.001, η_p² = .41) were significant. The interaction (F (2, 52) = 7.82, p = 0.01, η_p² = .23) was also significant. Multiple comparisons revealed that the number of correct responses was significantly higher in the uncovered than the sunglasses trial (p < 0.001), the uncovered than the mask trial (p = 0.03), and the mask than the sunglasses trial (p = 0.01). Regarding interaction, a simple main effect test revealed a significant difference for covering type only in the voiceless condition (F (2, 104) = 20.80, p < 0.001, η_p² = .44). Multiple comparisons revealed significant differences between covering types, and the number of correct responses was significantly higher in the uncovered (M = 3.93, SD = .07) than the sunglasses trial (M = 3.19, SD = .10) (p < 0.001), the uncovered than the mask trial (M = 3.70, SD = .12) (p = 0.02), and the mask than the sunglasses trial (p = 0.01). Contrastingly, no significant differences were observed (uncovered: M = 4.00, SD = .00, mask: M = 3.93, SD = .05, sunglasses: M = 3.78, SD = .10) in the voice-added condition (F (2, 104) = 1.84, p = 0.17, η_p² = .07). Furthermore, no differences were observed in the percentage of correct responses for each emotion.

Discussion

In this study, we compared Japanese children’s ability to recognize facial expressions when other peoples’ faces were uncovered, covered with masks, or covered with sunglasses. Additionally, we examined whether the paired presentation of voice stimuli facilitated facial expression recognition. In the voiceless condition, facial expression recognition was significantly lower when the model was wearing a mask compared with no face covering. Likewise, facial expression recognition was significantly lower when the model wore sunglasses compared with no face covering. Furthermore, participants could recognize emotions significantly more accurately when the model wore a mask compared with sunglasses. Thus, our results support Hypothesis 1A and suggest that preschoolers more accurately recognize facial expressions of people who are wearing masks compared with sunglasses (Hypothesis 1B).

Our results are consistent with those of previous studies (Carbon, 2020; Gori, Schiatti, & Amadeo, 2021; Ruba & Pollak, 2020) as facial expression recognition decreased when the other person wore a mask or sunglasses, thereby indicating that emotional judgments are made by observing multiple parts of the face.

Moreover, others’ emotions in daily life are not judged solely on static facial expressions, as in the facial photos used here; movements, sounds, situations, and contexts are also considered (Schirmer & Adolphs, 2017). In this study, no significant difference was observed in the performance of facial expression recognition among the uncovered, mask, and sunglasses conditions when voice stimuli were added (100% correct response rate), thus supporting Hypothesis 2. As multiple pieces of information are integrated to recognize others’ emotions (Schirmer & Adolphs, 2017), even if parts of the face are covered by a mask or sunglasses, visible and audible information is sufficient to compensate for and infer emotion. Several studies have shown that facial expressions and voice interact holistically in emotion recognition (de Gelder & Bertelson, 2003; de Gelder, Morris, & Dolan, 2005). Similarly, this study found that voice information affects Japanese children’s facial expression recognition.

We found that recognizing the facial expressions of those with masks was significantly more difficult than those without a covering. However, the correct response rate was approximately 95% even when the facial expressions were difficult to read. This may be due to differences in the experimental design compared with related studies (e.g., Gori, Schiatti, & Amadeo, 2021; Ruba & Pollak, 2020). In this study, participants selected the instructed face from the stimuli comprising four facial expressions, whereas when Ruba and Pollak (2020) showed facial expression stimuli to the children, they were prompted to identify the emotion on the face by selecting one of six labels. Gori, Schiatti, and Amadeo (2021) also showed facial expression stimuli to the children, who answered by selecting one of five labels. We used fewer labels than in previous related studies; therefore, it might have been easier for the participants in this study to recognize facial expressions, compared with previous studies.

Additionally, the higher performance of the participants in this study than in Ruba and Pollak (2020) might be due to a same-other-race effect – the facial expressions of people from one’s own race are easier to recognize than that of other races (Kelly et al., 2007). Kelly et al. (2007) suggested that the same-other-race effect on facial expression recognition emerges by 6 months of age. In this study, all participants were Japanese, and all the stimuli comprised Japanese faces, whereas the participants in Ruba and Pollak (2020) included Black, Caucasian, and multi-racial participants, and the stimuli comprised only Caucasian faces. Furthermore, cultural tendencies might affect the performance of participants. Cultural studies have suggested that Asian (including Japanese) people have a construal of the self as interdependent and characterize the Asian culture as collectivist (Markus & Kitayama, 1991; Matsumoto, Takeuchi, Andayani, Kouznetsova, & Krupp, 1998). Furthermore, Asian people have a cultural tendency to respect the values of others and value harmonious relationships. Therefore, Japanese people must control and suppress expressions of emotions to avoid imposing their emotions on others (Weiss, Thomas, Schick, Reyes, & Contractor, 2022). Moreover, Fogel, Toda, and Kawai (1988) found that Japanese mothers are more likely to punctuate their facial expressions than American mothers when communicating with infants. Thus, Japanese people should pay careful attention to others’ facial expressions because of the Japanese tendency to display subtle facial expressions in daily life. In this study, we used facial expression stimuli comprising exaggerated facial expressions. Owing to their exaggerated nature, it might have been easier for participants to discriminate between others’ facial expressions in the experimental situation than in their daily lives.

In this study, a comparison between partially covering the face stimuli with masks and sunglasses showed that reading facial expressions was more difficult with sunglasses. This result contradicts Ruba and Pollak’s (2020) study, which found no difference in facial expression recognition between the two conditions. Japanese people pay attention to others’ eyes when recognizing facial expressions (Grossmann, 2017; Jack, Blais, Scheepers, Schyns, & Caldara, 2009; Yuki, Maddux, & Masuda, 2007); therefore, recognizing others’ facial expressions when their eyes are covered by sunglasses is difficult. Moreover, daily life experience plays a crucial role in the development of face processing (Kelly et al., 2007). In this study, we collected the data in November 2021, when the participants had seen many people wearing masks in public owing to the COVID-19 pandemic. However, the participants had seen fewer people wearing sunglasses in public in Japan (e.g., Ng & Ikeda, 2011). Thus, Japanese participants in this study could recognize facial expressions with masks more easily than those with sunglasses.

In conclusion, although the Japanese preschoolers had a high percentage of correct responses when recognizing the facial expressions of people wearing masks and sunglasses, this result was lower than that for facial expression recognition of people with no covering. Furthermore, the additional voice information facilitated the understanding of facial expressions of people wearing masks and sunglasses.

The study has the following limitations. The facial expressions used may not have been representative of natural facial expressions expressed in everyday situations. The presented photo stimuli were created by asking the models to intentionally recall an emotion; they were not facial expressions expressed in natural situations. Therefore, the facial expressions in the stimuli could be exaggerated or different from those usually expressed. Furthermore, this study used only four emotions: happiness, sadness, anger, and surprise, to make tasks easy for focusing on the effect of facial covering. However, in daily life, we express more emotions, such as fear and disgust. Each emotion has different facial expression characteristics, and masks and sunglasses might affect these characteristics in different patterns. Therefore, a more detailed study examining the differences in facial expression recognition using photos taken in everyday situations or those with ambiguous or varied facial expressions should be conducted.

In this study, there were no substantial differences in participants’ ages, and even 3-year-olds could identify the facial expressions in the masked condition. However, masks may make it difficult for younger children to recognize facial expressions. Therefore, the effects of masks and sunglasses on younger children’s ability to recognize facial expressions must be examined.

Although previous research has focused on cultural differences in facial expression recognition (e.g., Yuki, Maddux, & Masuda, 2007), cultural differences in facial expression recognition with masks and sunglasses remain unclear. Furthermore, it remains unclear whether the effect of auditory information on facial expression recognition is specific to Japanese culture. Masuda et al. (2008) found that Japanese people observe emotions through others’ facial expressions, as inseparable from social context, whereas American people see them as individual feelings. Thus, Japanese people might be sensitive to not only others’ facial expressions but also other contextual information (including voice information). Therefore, future studies should conduct cross-cultural examinations regarding the voice effect on facial expression recognition with faces partially covered by masks and sunglasses.

Ethics approval

All procedures used in this research were approved by the Ethics Committee of Research Involving Human Subjects of Shizuoka University (approval number: 21–33).

Open Scholarship

This article has earned the Center for Open Science badge for Open Data. The data are openly accessible at https://doi.org/10.1080/15248372.2023.2207665

Acknowledgments

We would like to thank Editage (www.editage.com) for English language editing.

Disclosure statement

No potential conflict of interest was reported by the authors.

Data availability statement

All data have been made publicly available at the OSF and can be accessed at https://osf.io/gfbtk/. The analysis code and research materials for this study are available and can be accessed upon reasonable request by emailing the corresponding author.

Additional information

Funding

This work was supported by JSPS KAKENHI (Grant Number: 20K14155).