The reciprocal relationship between smiles and situational contexts

ABSTRACT

Smiles provide information about a social partner’s affect and intentions during social interaction. Although always encountered within a specific situation, the influence of contextual information on smile evaluation has not been widely investigated. Moreover, little is known about the reciprocal effect of smiles on evaluations of their accompanying situations. In this research, we assessed how different smile types and situational contexts affected participants’ social evaluations. In Study 1, 85 participants rated reward, affiliation, and dominance smiles embedded within either enjoyable, polite, or negative (unpleasant) situations. Context had a strong effect on smile ratings, such that smiles in enjoyable situations were rated as more genuine and joyful, as well as indicating less superiority than those in negative situations. In Study 2, 200 participants evaluated the situations that these smiles were perceived within (rather than the smiles themselves). Although situations paired with reward (vs. affiliation) smiles tended to be rated more positively, this effect was absent for negative situations. Ultimately, the findings point toward a reciprocal relationship between smiles and contexts, whereby the face influences evaluations of the situation and vice versa.

KEYWORDS:

• Smiles
• facial expression
• context
• emotion
• dynamic

Facial expressions in real life are inherently contextualised (Aviezer et al., 2017) in the sense that they are typically accompanied by additional information, such as the surrounding situation (Goodenough & Tinker, 1931), vocal cues (De Gelder et al., 2005; Tsankova et al., 2015), and body language (Aviezer et al., 2008). A wealth of research has shown that contextual information impacts emotion evaluations from faces (see Wieser & Brosch, 2012, for a review) and can even override facial information (Carroll & Russell, 1996; Wallbott, 1988b). However, facial expressions encountered in everyday life rarely match the distinct and mutually exclusive configurations associated with Ekman’s (1992) classic basic emotions (Matsumoto et al., 2009; Robertson et al., 2023), and may therefore be more amenable to socio-functional evaluations instead. In fact, most facial expressions convey social as well as emotional meaning (Fridlund, 1991; Kraut & Johnston, 1979).

The socio-functional approach (Carroll & Russell, 1996; Fridlund, 1994) proposes that facial expressions communicate information about the expresser’s social motives and intentions (e.g. approach-avoidance) rather than their underlying internal emotions per se. This confers the adaptive advantage of a more direct influence upon the social environment and the behaviours of others, which in turn enables the expresser to achieve their social goals more efficiently. Hence, facial expressions are conceived of as flexible tools for social influence. For instance, rather than reflecting the discrete underlying emotion of sadness (as suggested by Basic Emotions Theory; Ekman, 1992), pouting may instead communicate to the receiver that the expresser wants comfort and reassurance, and therefore has the ultimate purpose of recruiting the protection and support of a social partner.

In accordance with the socio-functional view, a recent model proposes that smiles can be classified in terms of how they affect people’s behaviour in the service of fundamental tasks of social living (The Simulation of Smiles Model, SIMS; Martin et al., 2017; Niedenthal et al., 2010). According to the SIMS model, reward smiles tend to involve activation of the sharp lip puller (AU13), dimpler (AU14), and eyebrow raisers (AU1 + 2), and reinforce desired behaviours by inducing positive affect in the perceiver (e.g. Shore & Heerey, 2011). Affiliation smiles aid the formation and maintenance of social bonds by communicating reassurance or an openness to positive relations. They normally involve activation of the lip pressor (AU24) and dimpler (AU14). Finally, dominance smiles are used to communicate and assert superiority over the perceiver in order to maintain high social status. These smiles are typically associated with asymmetrical activation of zygomaticus major muscle (AU12) which upturns just one corner of the mouth, as well as the upper lid raiser (AU5), nose wrinkler (AU9), cheek raiser (AU6), and upper lip raiser (AU10) (Rychlowska et al., 2017).

The varied morphological and socio-functional characteristics of smiles in general may make them more difficult to discern than other expressions. In fact, it is well-established that smile expressions are more ambiguous because they often co-occur with negative emotions or non-emotional states. This view has been previously shared by numerous scholars (Ekman & Friesen, 1982; Krumhuber & Kappas, 2005; Maringer et al., 2011; Namba et al., 2020; Niedenthal et al., 2013; Orlowska et al., 2018; Rychlowska et al., 2017). Consequently, people might rely on additional cues to disambiguate and infer their meaning. Hence, it is possible that contextual information influences the interpretation of a smile more than other expressions (Ito et al., 2013; Krumhuber, Hyniewska, et al., 2023; Mui et al., 2020; Namba et al., 2020; Wallbott, 1988a, 1988b).

Whilst extant research has shown that context does affect smile evaluations, this work has generally focused on ratings of emotion, genuineness, or authenticity (Gagnon et al., 2022; Krumhuber, Hyniewska, et al., 2023; Mui et al., 2020; Namba et al., 2020). For example, Namba et al. (2020) found that “happy” visual background contexts and vignettes both led to increased ratings of smile genuineness relative to when the same expressions were presented in isolation, whereas “polite” backgrounds and vignettes reduced genuineness ratings. Likewise, verbal descriptions of eliciting situations strongly affected evaluations of dynamic (Mui et al., 2020) and static smiles, such that compared to smiles in neutral contexts, smiles viewed in happy contexts were misremembered as having more distinctive cheek raising (Krumhuber, Hyniewska, et al., 2023) – traditionally implicated as a marker of smile genuineness (Ekman, 1992; Ekman et al., 1990; Frank et al., 1993).

Although past research has investigated context effects on smile evaluations, only one study to date has contrasted the effects of context across different smile types (Gagnon et al., 2022). Specifically, these authors found that smiles were rated as more sincere when displayed by smilers with a prior tendency to return favours in comparison to smilers who didn’t tend to return favours. Counter to their hypothesis, this effect was stronger for genuine (Duchenne) smiles than fake (non-Duchenne) ones. In contrast to the present study however, these authors manipulated dispositional rather than situational information and retained Duchenne’s (1862) classic smile typology. According to Duchenne, genuine smiles – supposedly elicited by the experience of a positive emotion – were characterised by the contraction of the orbicularis oculi muscle, which creates the distinctive cheek raising described previously. In contrast, fake smiles, which supposedly occur without the experience of an underlying positive emotion, were characterised by the absence of contraction in this muscle (Krumhuber & Kappas, 2022; Krumhuber & Manstead, 2009). To build upon Gagnon and colleagues work, here we instead investigated whether context impacts socio-functional evaluations of smiles (e.g. politeness/civility, enjoyment/reward, and superiority/condescension). The dependent variables were chosen because they aligned with the purported function of each of the three SIMS smile types. Nonetheless, we still retained a measure of genuineness to be consistent with previous literature. Furthermore, we tested whether these context effects varied across different smile types.

An additional aim of this paper was to increase the ecological validity of face and context stimuli. Most previous researchers have presented participants with static photographs of facial expressions, often displayed by artificially generated (e.g. Maringer et al., 2011) or disembodied faces (e.g. Righart & de Gelder, 2006, 2008). However, facial expressions viewed during social interactions typically unfold dynamically. As a result, static photographs lack certain cues that are present when observing an expression, such as the direction, quality, and speed of facial motion (see Krumhuber et al., 2013; Krumhuber, Skora, et al., 2023, for reviews). Consistent with this notion, dynamic facial expressions are perceived as more intense and realistic than static expressions (Biele & Grabowska, 2006; Cunningham & Wallraven, 2009; Weyers et al., 2006).

In a similar vein, the contextual stimuli used by researchers often lack much resemblance to situational contexts experienced in real life. The most common technique is to present participants with verbal vignettes (e.g. Krumhuber, Hyniewska, et al., 2023; Maringer et al., 2011; Mui et al., 2020), but this means that faces are seen in visual isolation. Furthermore, the vignette method requires an imaginative leap from participants and increases the transparency of the study’s aims, potentially encouraging demand characteristics. Whilst a handful of studies have assessed how visual background contexts affect facial expression ratings (e.g. Namba et al., 2020; Righart & de Gelder, 2006, 2008), these normally take the form of static photographs which don’t fit naturally with the faces. In fact, Reschke et al. (2018) explicitly declined to use such face-in-context stimuli in their own study because the combinations appeared unnatural.

This lack of ecological validity is particularly concerning given that the effects of context manipulations appear to vary depending upon their vividness and realism. Notably, Wallbott (1986, 1988a, 1988b) conducted a series of experiments which found a steady increase in contextual influence as the realism of the context improved. When expressions were presented with vignette scenarios, the face had almost twice as much of an impact on emotion intensity ratings as the contextual information. However, this gap between face and context influence narrowed when participants were presented with “candid picture” stimuli from magazines, and reversed when video clips from films and television were used as stimuli. Wallbott (1988b) suggested that the visual and dynamic presentation of context rendered this form of information more vivid, immediate, and readily accessible to raters, which facilitated the larger impact on emotion attributions. Consequently, Fernandez-Dols et al. (1991) proposed the use of more ecologically valid stimuli, including video clips of facial expressions and situations. Unfortunately, few studies to date have taken this approach, and no study investigating smiles has utilised both dynamic face and dynamic context stimuli.

Finally, although most previous work has assessed how context affects the evaluations of facial expressions, the relationship between context and facial expressions is likely reciprocal (Hess et al., 2020). Research has shown that facial expressions bias emotion categorisation of body language towards the emotion conveyed by the face (Kret et al., 2013; Lecker et al., 2020). Furthermore, the facial expressions of spectators influenced perceptions of a player’s success/failure in an ambiguous ball-game (Hess et al., 2020) and altered the perceived morality and politeness of unusual behaviours described in verbal vignettes (Hess et al., 2018). As facial expressions typically reflect a person’s evaluation of their current situation (e.g. Frijda, 1986; Scherer, 2001), they may provide information about the situation itself. For example, participants conclude that a social norm has been violated when they believe that another person is angry about an event (Hareli et al., 2013). However, no previous study has investigated whether smiles affect evaluations of situational context, and it remains unclear how these evaluations differ across smile types.

Present research

Given the above literature, the present study addresses two main research questions. In Study 1, we asked how socio-functional interpretations of different smile types are affected by dynamic visual contexts. In Study 2, we asked whether social interpretations of dynamic visual contexts are affected by the smile type expressed by the person situated within these contexts. To this end, we presented participants with vivid, dynamic, and realistic smile-in-context compound stimuli that represent a major advance in ecological validity compared to previous studies. In Study 1, participants rated the genuineness, reward/enjoyment, politeness/civility, and superiority/condescension of each smile. In Study 2, participants rated how positive, enjoyable/rewarding, polite/civil, and competitive/confrontational each situational context was. Although dominance smiles were presented, they were not included in the primary analyses because our pre-registered hypotheses only related to comparisons between affiliation and reward smiles (full pre-registration information available here: 10.17605/OSF.IO/UMVP3). We did not have specific hypotheses about dominance smiles, which were included for exploratory purposes.

In line with previous research (Gagnon et al., 2022; Mui et al., 2020; Namba et al., 2020), we expected that smile evaluations would be influenced by context in Study 1, such that smiles in enjoyment contexts would be evaluated as more genuine and more joyful than those presented in polite or negative contexts (Hypothesis 1), whereas smiles in polite contexts would be evaluated as more polite than those in enjoyment or negative contexts (Hypothesis 2). Furthermore, we hypothesised that smiles presented in negative contexts would be evaluated as less joyful, less polite, less genuine, and indicating higher levels of superiority than those presented in enjoyment or polite contexts (Hypothesis 3). Finally, in line with the hypotheses of Gagnon et al. (2022), we predicted that context effects would be stronger for affiliation smiles than reward smiles (Hypothesis 4) because they are more ambiguous and challenging to classify than enjoyment smiles (Johnson et al., 2010; Orlowska et al., 2018; Rychlowska et al., 2017). In Study 2, we predicted that contexts presented with reward smiles would be evaluated as more joyful and positive than those presented with affiliative smiles (Hypothesis 5), and contexts presented with affiliation smiles would be evaluated as more polite than those presented with reward smiles (Hypothesis 6).

Study 1

In Study 1, we aimed to assess whether socio-functional interpretations of different smile types would be affected by the dynamic visual contexts they are embedded within. Furthermore, we explored whether any of the three SIMS smile types were more or less susceptible to contextual influence.

Methods

Participants

To determine sample size, means and standard deviations from Namba et al. (2020) and Mui et al. (2020) were entered into an a priori power analysis using the ANOVA_power shiny app (Lakens & Caldwell, 2021). To detect an interaction between smile type and context category (η_p² = .45) with 90% power, we recruited 85 participants (58 women, 25 men, 2 non-binary) from the United Kingdom. The majority were Caucasian (70.5%; 16 Asian, 6 Multiple ethnic groups, 1 Black, 1 other, 1 prefer not to say), aged 18–70 years (M_age = 29.09, SD_age = 13.28). Participants were recruited via Prolific (www.prolific.co) and were paid £7.50 per hour. Power analyses for ANOVA were deemed most suitable given the lack of clear guidance for calculating power for LMMs. The study was approved by the university’s research ethics committee.

Materials

First, 36 smile videos were taken from the set developed by Rychlowska et al. (2017). We selected one video of each smile type (reward, affiliation, dominance) produced by 12 White actors (six male, six female; see Figure 1 for examples) in frontal view. Actors posed each smile type after being coached about its form and accompanying social motivations (see Martin et al., 2017; Rychlowska et al., 2017). In morphological terms (FACS, Ekman et al., 2002), reward smiles were characterised by symmetrical activation of the Lip Corner Puller (AU12), the Cheek Raiser (AU6), Lips Part (AU25) and/or Jaw Drop (AU26). Affiliation smiles were characterised by the Lip Corner Puller (AU12), the Chin Raiser (AU17), with or without Brow Raiser (AU1-2). Dominance smiles were characterised by asymmetrical activation of the Lip Corner Puller (AU12L or AU12R), with additional actions, such as Head Up (AU53), Upper Lip Raiser (AU10), and/or and Lips Part (AU25) (see Figure 1). Each video showed the face changing from non-expressive to peak emotional display. Videos ranged in duration from 1.8 s to 3.0 s (M = 2.40 s) and were 1280 × 720 pixels. All videos were in colour. We used Unscreen (www.Unscreen.com) to remove original video backgrounds.

Figure 1. Example screenshots from (A) Videos of reward, affiliation, and dominance smiles (B) Videos of positive, polite, and negative contexts, and (C) Videos of each smile type in each context.

Second, to create new backgrounds, thirty-six context videos were downloaded from Envato Elements (www.elements.envato.com); 12 of each situation type: enjoyment, polite, and negative. Videos were selected based on stimuli used in previous studies (e.g. Namba et al., 2020; Righart & de Gelder, 2008), and data describing situations in which certain smile types are likely to be produced (e.g. Martin et al., 2021). Enjoyment contexts depicted situations where people would typically express joy (e.g. beaches and parks). Polite contexts depicted situations where people would express politeness (e.g. offices and public transport). Negative contexts depicted situations where people would not normally smile (e.g. cemeteries and rubbish dumps). The type and number of contextual stimuli were evenly distributed. For enjoyment contexts, we had three sub-categories with the themes of “celebration”, “luxury”, and “park”. Within the polite contexts we had three sub-categories themed “shops”, “public transport”, and “workplace”. For negative contexts we had three sub-categories themed “fire”, “rubbish”, and “cemetery”. These sub-categories were balanced within and between blocks. Each sub-category consisted of four separate videos.

Finally, smile videos were superimposed over context videos with the editor tool from Kapwing (www.Kapwing.com). For each actor, nine smile-in-context videos were created such that each smile type was superimposed over the three context types (see Figure 1 for examples). Each of the face-in-context videos was 1980 × 1080 pixels.

Procedure

The experiment was programmed using Gorilla (www.Gorilla.sc). We specified that participants could only complete the study using a computer. Smartphones and tablets could not be used. After providing informed consent, participants first completed a block of baseline smile ratings. All 36 smile videos (without background context) were presented in a random order and could be watched only once. For each smile, participants made a series of ratings along 0–100 scales, starting with the genuineness of each smile (fake – real) and their confidence in this judgement (not confident – very confident). Participants then rated the extent to which the person was expressing enjoyment/reward, politeness/civility, and superiority/condescension (not at all – very much). Participants had unlimited time to respond. Questions were always presented in the same order.

Following this baseline rating block, participants completed a filler task which involved counting the number of squares in a complex figure. This was included as a distractor to ensure that ratings in the second block were not consciously influenced by ratings in the baseline block. They then completed a further rating block of smiles, this time embedded within a situational context. Participants were randomly allocated to one of three test blocks, consisting of the 36 smile stimuli presented once each, within either enjoyment, polite, and negative backgrounds (12 of each). Within each block, actor gender and smile type were fully balanced across context type. Furthermore, smile and context were balanced across the three test blocks for each actor, such that every smile video was presented in each context type (enjoyment, polite, and negative) across participants. The rating scales were the same as in the baseline (smile-alone) block. Finally, participants completed several questionnaires and answered additional questions which are not included in this paper. Upon completion of the experiment, all participants were fully debriefed.

Data analysis

Data analysis was conducted with R Studio version 2022.02.3 (RStudio Team, 2020) in R version 4.2.1 (R Core Team, 2022). Participant ratings were analysed using linear mixed models.

We followed the advice of Bates et al. (2015) when conducting our analysis. Bates et al proposed that one should start with a maximal model and remove the random-effect components which account for the smallest variance, one-by-one. In line with Bates et al’s recommendations, after each simplification, the reduced model was subjected to a Principal Components Analysis (PCA), using the rePCA function from the “lme4” package. The output of the rePCA function indicated whether there were any random effect terms in the model that did not explain any covariance.

In theory, this process should continue until each remaining random-effect term explains a considerable portion of variance. However, after each simplification, we also compared the goodness-of-fit of our reduced model to the more parameterised model with a likelihood ratio test (using the anova function from the “stats’ package). If goodness-of-fit did not significantly differ, the simplification was justified because parsimony is prioritised. However, if the goodness-of-fit differed significantly, we retained the more parameterised model and stopped the PCA-based model simplification process.

Random structure was identified prior to adding fixed effects to the model (Meteyard & Davies, 2020). Simplification of random effects continued until convergence. Satterthwaite’s F test determined inferential statistics.

For each rating (genuineness, enjoyment, politeness, superiority), smile type (two levels: reward vs affiliative) and context category (three levels: enjoyment vs polite vs negative) were entered as fixed effect predictors. Dominance smiles were not included in primary analyses as they were included in the study for exploratory purposes. Significant effects were clarified by tests of estimated marginal means (with the Holm–Bonferroni correction applied).

Data availability

We have reported how we determined our sample size, all data exclusions (if any), all manipulations, and all measures in the study, and we follow JARS (Kazak, 2018). All data, analysis code, and research materials are available at https://osf.io/d6y7p/?view_only=dfd19c81f0f945e7af70e0ea3bfc99bf. This study’s design and analysis were pre-registered; see 10.17605/OSF.IO/UMVP3.

Results and discussion

Manipulation checks

We conducted four one-way within-subjects ANOVAs on aggregated data from the baseline “no context” block to assess the influence of smile type (three levels: reward vs affiliation vs dominance) on each of the four main dependent variables. Where Mauchly’s Test indicated that assumption of sphericity was violated, the Greenhouse-Geisser correction was used. These analyses were performed to ensure that our smile stimuli had their intended meaning as ascribed by Rychlowska et al. (2017).

Genuineness

A significant main effect of smile type was observed, F(1.86, 155.85) = 258.92, p < .001, η²_G = .63. As expected, pairwise comparisons indicated that reward smiles (M = 71.59, SE = 1.84) were rated as more genuine than affiliation smiles (M = 39.26, SE = 1.67), p < .001, and dominance smiles (M = 33.63, SE = 1.65), p < .001.

Enjoyment

A significant main effect of smile type was observed, F(1.62, 135.93) = 253.94, p < .001, η²_G = .62. As expected, pairwise comparisons indicated that reward smiles (M = 72.12, SE = 1.48) were rated as more joyful than affiliation smiles (M = 37.03, SE = 1.73), p < .001, and dominance smiles (M = 35.46 SE = 2.04), p < .001.

Politeness

A significant main effect of smile type was observed, F(2, 168) = 61.19, p < .001, η²_G = .30. As expected, pairwise comparisons indicated that affiliation smiles (M = 67.26, SE = 1.59) were rated as more polite than reward smiles (M = 57.59, SE = 2.37), p < .001, and dominance smiles (M = 43.64, SE = 1.83), p < .001.

Superiority

A significant main effect of smile type was observed, F(1.77, 148.60) = 277.40, p < .001, η²_G = .59. As expected, pairwise comparisons indicated that dominance smiles (M = 60.14, SE = 2.03) were rated as more superior than affiliation smiles (M = 35.57, SE = 1.94), p < .001, and reward smiles (M = 18.21, SE = 1.75), p < .001.

Main analyses

LMMs were constructed to assess the effect of both smile type (two levels: reward vs affiliative) and context category (three levels: enjoyment vs polite vs negative) on each of the four main dependent variables during the smile-in-context block.

Genuineness

Results revealed a significant main effect of smile type, F(1, 14.51) = 53.52, p < .001, such that reward smiles (M = 70.85, 95% CI [64.12, 77.59]) were rated as more genuine than polite smiles (M = 39.26, 95% CI [32.36, 46.16]), t(14.51) = −7.32, p < .001. The main effect of context category was also significant, F(2, 1844.03) = 7.66, p < .001. Smiles in enjoyment contexts (M = 57.00, 95% CI [51.83, 62.16]) were rated significantly more genuine than smiles in negative contexts (M = 52.47, 95% CI [47.31, 57.64]), t(1844.03) = 3.80, p < .001. Similarly, smiles in polite contexts (M = 55.70, 95% CI [50.53, 60.86]) were rated as significantly more genuine than smiles in negative contexts, t(1844.03) = 2.71, p =  .013, although there were no genuineness differences between smiles presented in enjoyment compared to polite contexts (p = .27). Finally, we did not find an interaction between smile type and context category (p = .75). See Figure 2(a) for relevant means and CI’s.

Figure 2. Interactions between context category and smile type for ratings of smile genuineness (a), Enjoyment (b), Politeness (c), and Superiority (d) in Study 1. Error bars: 95% CI. Note: ***p < .001, ** p < .01, * p < .05, ns not significant.

Enjoyment

Results revealed a main effect of smile type, F(1, 16.54) = 84.65, p < .001, such that reward smiles (M = 72.09, 95% CI [66.40, 77.77]) were rated as showing more enjoyment than polite smiles (M = 36.85, 95% CI [29.82, 43.88]), t(16.54) = −9.20, p < .001. The main effect of context category was also significant, F(2, 1844.03) = 13.64, p < .001. Smiles in enjoyment contexts (M = 57.54, 95% CI [52.47, 62.61]) were rated as more joyful than smiles in both negative contexts (M = 51.89, 95% CI [46.83, 56.96]), t(1844.03) = 5.17, p < .001, and polite contexts (M = 53.98, 95% CI [48.91, 59.04]), t(1844.03) = 3.26, p = .002. There were no enjoyment differences between smiles presented in negative and polite contexts (p = .057). Finally, we did not find an interaction between smile type and context category (p = .84). See Figure 2(b) for relevant means and CI’s.

Politeness

Results revealed a main effect of smile type, F(1, 60.35) = 22.30, p < .001, such that affiliation smiles (M = 65.22, 95% CI [61.56, 68.88]) were rated as more polite than reward smiles (M = 53.76, 95% CI [49.23, 58.30]), t(60.36) =  4.72, p < .001. The main effect of context category was also significant, F(2, 1844.20) = 18.72, p < .001. Smiles in polite contexts (M = 63.18, 95% CI [59.67, 66.71]) were rated as more polite than smiles in both enjoyment contexts (M = 57.14, 95% CI [54.64, 61.68]), t(1844.19) = −4.75, p < .001, and negative contexts (M = 58.16, 95% CI [53.61, 60.65]), t(1844.19) = −5.72, p < .001. There were no politeness differences between smiles presented in enjoyment and negative contexts (p = .33). The interaction between smile type and context category was also significant, F(2, 1844.14) = 3.05, p = .047. For reward smiles, there were significant differences in politeness ratings between polite contexts and enjoyment contexts (p < .001), and between polite contexts and negative contexts (p < .001). However, for affiliation smiles, there were no significant differences between polite contexts and enjoyment contexts. Furthermore, although still significant (p = .039), the difference in politeness ratings between polite contexts and negative contexts was substantially smaller for affiliation smiles than for reward smiles. See Figure 2(c) for relevant means and CI’s.

Superiority

Results revealed a main effect of smile type, F(1, 14.8) = 26.22, p < .001, such that affiliation smiles (M = 33.29, 95% CI [27.55, 39.02]) were rated as displaying more superiority than reward smiles (M = 17.26, 95% CI [13.24, 21.29]), t(14.8) =  5.12, p < .001. The main effect of context category was also significant, F(2, 1844.06) = 4.59, p = .010. Smiles in negative contexts (M = 27.10, 95% CI [23.22, 30.99]) were rated as indicating greater superiority than smiles in enjoyment contexts (M = 23.97, 95% CI [20.08, 27.85]), t(1844.06) = −2.91, p = .011. However, there were no superiority differences between smiles presented in negative and polite contexts (M = 24.76, 95% CI [20.87, 28.64]) (p = .059). In addition, there were no superiority differences between smiles in enjoyment and polite contexts (p = .46). Finally, we did not find an interaction between smile type and context category (p = .55). See Figure 2(d) for relevant means and CI’s.

Exploratory analyses

Exploratory analyses were then performed on the whole dataset (with dominance smiles included). New random effects structures were established using the same procedures described above.

All main effects and interactions reported above were retained. Compared to reward smiles, dominance smiles were rated as less genuine (p < .001), less joyful (p < .001), and indicating greater superiority (p < .001), while politeness ratings did not differ significantly (p = .054). Compared to affiliation smiles they were rated as less polite (p < .001) and displaying more superiority (p < .001), while genuineness (p = .20) and enjoyment (p = .95) ratings did not differ.

The strengthened interaction between context category and smile type (p = .013) revealed that dominance smiles were rated as less polite than reward smiles in negative (p < .001) and polite contexts (p = .016), but there were no politeness differences between the two smile types in enjoyment contexts (p = .32). See Figure 3 for relevant means and CI’s.

Figure 3. Interaction between context category and smile type for ratings of smile politeness when dominance smiles are included in study 1. Error Bars: 95% CI.

Study 2

Compared to the first study which examined the effects of situational context on smile evaluations, Study 2 aimed to assess whether interpretations of dynamic visual contexts are affected by the smile type expressed by the person situated within these contexts. We predicted that contexts presented with reward smiles would be evaluated as more joyful and positive than those presented with affiliation smiles, and contexts presented with affiliation smiles would be evaluated as more polite than those presented with reward smiles. We had no clear reason to assume that these effects would differ across context categories.

Methods

Participants

We recruited 200 UK-based participants from Prolific (102 women, 96 men, 2 prefer not to say; 172 White/Caucasian, 20 Asian/Asian-British, 3 Mixed/Multiple ethnic groups, 3 Black/Black-British, 1 other, 1 prefer not to say), aged 19–79 years (M_age = 38.82, SD_age = 12.76). Participants were paid £7.50 per hour. Our sample size was deemed sufficient to find a face-on-context interaction effect that was half the size of the reverse context-on-face effect, in line with Kret et al. (2013). Hypothesised means and standard deviations were inputted into a power analysis using the ANOVA_power shiny app (Lakens & Caldwell, 2021), with the goal to obtain 90% power to detect a medium-sized interaction (η_p² = .23) between smile type and context category.

Materials

The materials used were the same as for Study 1.

Procedures

The procedures were almost the same as Study 1. However, participants rated situations rather than smiles. In the baseline block, situational context videos were presented alone (without smiles).

On each trial, participants first evaluated the valence of the situational context (negative – positive) and their confidence in this evaluation (not confident – very confident). Participants then rated the extent to which the situation was enjoyable/rewarding, polite/civil, and competitive/confrontational (not at all – very much).

Results

Manipulation checks

We conducted four one-way within-subjects ANOVAs on aggregated data from the baseline “no smile” block, to assess the influence of context category (three levels: enjoyment vs polite vs negative) on each of the dependent variables. These analyses were performed to ensure that our context stimuli were interpreted as intended. Where Mauchly’s Test indicated that assumption of sphericity was violated, the Greenhouse-Geisser correction was used.

Valence

A significant main effect of context category was found, F(1.90, 377.28) = 3447.54, p < .001, η²_G = .90. As expected, pairwise comparisons indicated that enjoyment contexts (M = 87.53, SE = 0.79) were rated as more positive than polite contexts (M = 57.22, SE = 0.97), p < .001, and negative contexts (M = 15.01, SE = 0.75), p < .001.

Enjoyableness

A significant main effect of context category was found, F(2, 398) = 2788.72, p < .001, η²_G = .88. As expected, pairwise comparisons indicated that enjoyment contexts (M = 85.53, SE = 0.85) were rated as more enjoyable than polite contexts (M = 48.36, SE = 1.21), p < .001, and negative contexts (M = 12.64, SE = 0.80), p < .001.

Politeness

A significant main effect of context category was found, F(1.62, 321.43) = 724.98, p < .001, η²_G = .58. Surprisingly, pairwise comparisons indicated that enjoyment contexts (M = 73.01, SE = 1.29) were rated as more polite than polite contexts (M = 65.22, SE = 1.24), p < .001. However, as anticipated, polite contexts were rated as more polite than negative contexts (M = 34.38, SE = 1.19), p < .001.

Competitiveness

A significant main effect of context category was found, F(1.62, 321.43) = 724.98, p < .001, η²_G = .58. As expected, pairwise comparisons indicated that negative contexts (M = 31.13, SE = 1.76) were rated as more competitive than polite contexts (M = 25.78, SE = 1.44), p = .002, and enjoyment contexts (M = 14.84, SE = 1.12), p < .001.

Main analyses

Once again, LMMs were constructed to assess the effect of both smile type (two levels: reward vs affiliative) and context category (three levels: enjoyment vs polite vs negative) on each of the four dependent variables. Significant effects were clarified by tests of estimated marginal means (Holm–Bonferroni correction applied).

Valence

Results revealed a main effect of context category, F(2, 19.9) = 159.90, p < .001, such that enjoyment contexts (M = 75.51, 95% CI [71.79, 79.22]) were rated as more positive than polite contexts (M = 61.66, 95% CI [57.80, 65.52]), t(24.61) =  11.72, p < .001, which in turn were rated as more positive than negative contexts (M = 24.88, 95% CI [18.62, 31.15]), t(13.58), p < .001. The main effect of smile type was also significant, F(1, 12.8) = 20.89, p < .001, such that contexts paired with reward smiles (M = 56.20, 95% CI [53.00, 59.40]) were rated as more positive than contexts paired with affiliation smiles (M = 51.84, 95% CI [49.02, 54.65]), t(12.76) =  −4.57, p < .001. The interaction between smile type and context category was also significant, F(2, 3959.9) = 13.50, p < .001. For both enjoyment and polite contexts, valence ratings were lower when paired with a polite smile compared to a reward smile (M_diff = −4.67, 95% CI [−7.06, −2.28], t(28.56) = −4.00, p < .001; M_diff = −7.22, 95% CI [−9.61, −4.83], t(28.56) = −6.19, p < .001, respectively). However, smile type did not affect valence ratings for negative contexts (p = .31). See Figure 4(a) for relevant means and CI’s.

Figure 4. Interactions between context category and smile type for ratings of context valence (a), Enjoyment (b), Politeness (c), and Competitiveness (d) in Study 2. Error bars: 95% CI.

Enjoyableness

Results revealed a main effect of context category, F(2, 20.1) = 178.09, p < .001, such that enjoyment contexts (M = 71.45, 95% CI [67.75, 75.14]) were rated as more enjoyable than polite contexts (M = 53.73, 95% CI [49.47, 58.00]), t(21.75) =  11.78, p < .001, which in turn were rated as more enjoyable than negative contexts (M = 21.19, 95% CI [16.21, 26.17]), t(13.91) =  −9.93, p < .001. The main effect of smile type was also significant, F(1, 12.9) = 17.85, p = .001, such that contexts paired with reward smiles (M = 51.00, 95% CI [47.76, 54.25]) were rated as more enjoyable than contexts paired with affiliation smiles (M = 46.58, 95% CI [43.88, 49.28]), t(12.9) =  −4.23, p = .001. The interaction between smile type and context category was also significant, F(2, 3960.2) = 6.67, p = .001. For both enjoyment and polite contexts, enjoyableness ratings were lower when paired with a polite smile compared to a reward smile (M_diff = −4.44, 95% CI [−7.01, −1.88], t(25.83) = −3.57, p = .001; M_diff = −6.55, 95% CI [−9.11, −3.99], t(25.83) = −5.26, p < .001, respectively). Once again, smile type did not affect valence ratings for negative contexts (p = .079). See Figure 4(b) for relevant means and CI’s.

Politeness

Results revealed a main effect of context category, F(2, 12.8) = 32.18, p < .001, such that enjoyment contexts (M = 71.11, 95% CI [66.99, 75.23]) were rated as more polite than polite contexts (M = 66.03, 95% CI [62.44, 69.61]), t(11.92) =  4.04, p < .001, which in turn were rated as more polite than negative contexts (M = 39.66, 95% CI [32.07, 47.26]), t(13.47) =  −6.36, p < .001. Intriguingly, the main effect of smile type was non-significant for politeness ratings, F(1, 13.1) = 4.03, p = .066. Nevertheless, the interaction between smile type and context category was significant, F(2, 3956.9) = 29.77, p < .001. For both enjoyment and polite contexts, politeness ratings were lower when paired with a polite smile compared to a reward smile (M_diff = −5.21, 95% CI [−8.20, −2.22], t(22.83) = −3.61, p = .002; M_diff = −5.30, 95% CI [−8.29, −2.32], t(22.83) = −3.68, p = .001, respectively). Once more, smile type did not affect valence ratings for negative contexts (p = .053). See Figure 4(c) for relevant means and CI’s.

Competitiveness

Results revealed a main effect of context category, F(2, 13.2) = 27.46, p < .001, such that negative contexts (M = 33.25, 95% CI [27.77, 38.72]) were rated as more competitive than polite contexts (M = 21.50, 95% CI [17.81, 25.18]), t(16.60) =  3.65, p = .003, which in turn were rated as more competitive than enjoyment contexts (M = 18.00, 95% CI [14.53, 21.48]), t(10.98) =  −4.24, p = .003. The main effect of smile type was also significant, F(1, 4164.8) = 13.11, p < .001, such that contexts paired with affiliation smiles (M = 25.18, 95% CI [22.16, 28.20]) were rated as more competitive than contexts paired with reward smiles (M = 23.32, 95% CI [20.30, 26.34]), t(4164.81) =  3.62, p < .001. The interaction between smile type and context category was also significant F(2, 4164.8) = 7.53, p < .001. For both enjoyment and polite contexts, competitiveness ratings were higher when paired with a polite smile compared to a reward smile (M_diff = 2.56, 95% CI [0.82, 4.31], t(4164.82) = 2.88, p = .004; M_diff = 3.88, 95% CI [2.13, 5.62], t(4164.81) = 4.35, p <.001, respectively). However, smile type did not affect valence ratings for negative contexts (p = .34). See Figure 4(d) for relevant means and CI’s.

Exploratory analyses

Exploratory analyses were conducted in the same way as Study 1.

All main effects and interactions from the primary analysis retained their significance when dominance smiles were included. As expected, tests of estimated marginal means showed that contexts paired with dominance smiles were rated as significantly less positive (p < .001), less enjoyable (p < .001), and more competitive (p < .001) than contexts paired with reward smiles. Likewise, they were rated as less positive (p < .001), less enjoyable (p < .001), and more confrontational (p < .001) than contexts paired with affiliation smiles.

For the politeness dependent variable, the main effect of smile type was now significant, F(2, 6561) = 182.30, p < .001, such that contexts paired with reward smiles (M = 60.19, 95% CI [56.88, 63.51]) were rated as more polite than contexts paired with affiliation smiles (M = 57.67, SE = 1.63, 95% CI [54.36, 60.98]), t(6561.04) = −4.84, p <.001, which in turn were rated as more polite than contexts paired with dominance smiles (M = 50.58, 95% CI [47.27, 53.90]), t(6561.04) = 13.58, p <.001. Finally, the strength of the interactions between smile type and context category seemed to be relatively unaffected by the addition of dominance smiles to the analysis. See Figure 5 for a representation of these interactions.

Figure 5. Interactions between context category and smile type for ratings of context valence (a), Enjoyment (b), Politeness (c), and Competitiveness (d), with Dominance smiles included in study 2. Error bars: 95% CI.

Comparison of smile and context effects

Unfortunately, due to the way that variance is partitioned in linear mixed models (e.g. Rights & Sterba, 2019), there is not an agreed upon method to calculate standard effect sizes for individual model terms such as main effects or interactions. The central problem is that the effect on the response scale need to be normalised by some estimate of variability (e.g. standard deviation). However, it is not clear which estimate to take in the case of a linear mixed model, because there are several, including within- and between-cluster variance.

Nonetheless, we decided to primarily employ linear mixed models in our analysis, because LMMs are vastly superior in controlling for Type I errors than alternative approaches and consequently results from mixed models are more likely to generalise to new observations (e.g. Barr et al., 2013; Judd et al., 2012). Whenever possible, we report unstandardised effect sizes, which is in line with general recommendation of how to report effect sizes (e.g. Pek & Flora, 2018).

Therefore, we used unstandardised estimated marginal mean contrasts to compare the magnitude of context effects in Study 1 with the magnitude of smile effects in Study 2. For Study 1, the size of each context effect was calculated by taking the largest mean difference between two context categories for each dependent variable, using the full dataset. For example, the effect size for the genuineness dependent variable (4.07) was the difference between mean genuineness ratings of smiles in enjoyment contexts (M = 50.07) and mean genuineness ratings of smiles in negative contexts (M = 46.00). Likewise, for Study 2, smile effect sizes were calculated by taking the largest mean difference between two smile types for each dependent variable, using the full dataset with dominance smiles included. This reporting is in line with general recommendations for how to report effect sizes (e.g. Pek & Flora, 2018).

Across all comparable dependent variables, the mean context effect in Study 1 was smaller than the comparable mean smile effect in Study 2. Descriptive statistics of these comparisons are displayed in Table 1.

Table 1. Effect size comparison of context (Study 1) and Smile (Study 2) effects across dependent variables.

Study 1 variable	Context effect size	Study 2 variable	Smile effect size
Genuine	4.07	Positive	9.31
Enjoyment/Reward	4.45	Enjoyable/Rewarding	8.63
Politeness/Civility	6.38	Polite/Civil	10.86
Superiority/Condescension	4.18	Competitive/Confrontational	9.75
Mean	4.77	Mean	9.64

Note: Effect size was operationalised as the largest difference between two groups for each dependent variable. For Study 1, we compared mean smile ratings across context categories. For Study 2, we compared mean context ratings across smile types.

General discussion

Our findings show that situational contexts influence socio-functional inferences from smiles. Further, this relationship is reciprocal such that evaluations of situational contexts are also influenced by smiles. As per Hypothesis 1, Study 1 revealed that smiles in enjoyment contexts were rated as more genuine and more joyful than smiles in negative contexts. In accordance with Hypothesis 2, smiles in polite contexts were rated as more polite than smiles in both enjoyment and negative contexts. As expected smiles in negative contexts were evaluated as less genuine, less enjoyable, and indicating greater superiority than smiles in enjoyment contexts, and less polite and less genuine than smiles in polite contexts (Hypothesis 3). However, contrary to Hypothesis 4, the magnitude of context effects was largely consistent across smile types. In fact, politeness evaluations of reward smiles were affected by polite contexts more than affiliation smiles were.

In Study 2, we found that smiles affected context evaluations. In line with Hypothesis 5, situations paired with reward smiles were rated as more positive, more enjoyable, and less competitive than situations paired with affiliation smiles. However, contrary to Hypothesis 6, situations paired with affiliation smiles were not rated as more polite than situations with reward smiles. Further, exploratory analyses showed that contexts were perceived as less positive, less enjoyable, less polite, and more confrontational when accompanied by a dominance smile. These effects were moderated by context, such that differences were reduced or eliminated in negative situations.

The effect of context on social inferences from smiles is consistent with a body of recent research showing that contextual information alters smile interpretations (Gagnon et al., 2022; Krumhuber, Hyniewska, et al., 2023; Maringer et al., 2011; Mui et al., 2020; Namba et al., 2020; Orlowska et al., 2023). However, our findings extend this research by showing that socio-functional evaluations of smiles are influenced by the surrounding situation, and that this effect occurs for both reward and affiliation smiles. Although contrary to our hypotheses, the failure to find enhanced context effects for affiliation smiles is consistent with Gagnon et al. (2022), who unexpectedly showed that evaluations of Duchenne smiles were more influenced by dispositional contextual information than evaluations of non-Duchenne smiles. These authors attributed this unexpected finding to methodological shortcomings which produced floor effects in their participants’ sincerity ratings. However, this methodological limitation does not apply in our study. Therefore, our findings provide stringer evidence that evaluations of affiliation smiles are not more susceptible to context than reward smiles. Nonetheless, Gagnon et al. (2022) further stressed that they wouldn’t classify non-Duchenne smiles as highly ambiguous stimuli (like neutral or blended expressions), and hence were reluctant to present their study as a valid test of the idea that context exerts a stronger influence on evaluations of ambiguous expressions.

Likewise, it is possible that the affiliation smiles in our research were not the most suitable stimuli to test this hypothesis because they have a well-defined social function. Thus, it is possible that our affiliation smiles are actually less ambiguous social signals than reward smiles, but only along the politeness dimension. This theory could explain the unexpected interaction between smile type and context category for ratings of politeness, whereby politeness evaluations of reward smiles were affected by polite contexts more than evaluations of affiliation smiles were.

The converse effect of smiles on context evaluations is in line with prior research showing that expressions may affect how body postures, behaviours, and situations are evaluated (Hess et al., 2018, 2020; Kret et al., 2013; Lecker et al., 2020). Building upon this work, we showed that facial expressions also affect evaluations of visual situational contexts. This effect is especially notable because the effects of smile type on context evaluations appear to be larger than the opposite effects of context category on smile evaluations. Although we are wary of drawing strong conclusions without standardised effect sizes, our findings are in sharp contrast with the results of Lecker et al. (2020), who observed an effect of body posture on facial emotion ratings that was twice as large as the converse effect of facial expression on body emotion ratings.

To explain this potential discrepancy, we suggest that the effect of facial expression on situation evaluations may be qualitatively distinct from the reciprocal effect of situational context on facial expression, which inflates its magnitude. Whilst substantial, the influence of situational context on facial expressions is inherently limited because the situation is not an active part of the facial expression. Consequently, the situation only contributes indirectly to face evaluations by indicating an increased likelihood that a person may be feeling or communicating a certain emotion or social motive. The face may be entirely unaffected by the context. In contrast, the face is a major part of the situation itself. A situation with a person smiling broadly is of a completely different nature to a situation with someone smirking mockingly, even if both situations occur in the same location. Therefore, smiles contribute directly to evaluations of the situational context. The situation cannot be separated from the facial expression. Such an asymmetry is not present in the relationship between body language and facial expression as studied by Kret et al. (2013) and Lecker et al. (2020) Hence, the more direct effect that smiles have on situational evaluations helps to overcome the natural advantage afforded by our regular practice in integrating context into the interpretation of facial expression (Lecker et al., 2020).

In Study 2, we included a novel comparison of the effect of different smile types across context categories. Importantly, the reduction or absence of a smile type effect in negative contexts demonstrates limits to the social power of smiles. On one hand, it is possible that the effects of different smile types become more difficult to distinguish in situations where smiles are incongruent or unexpected. This is somewhat in line with the findings of Carrera-Levillain and Fernandez-Dols (1994), who found that emotional evaluations of neutral faces were less affected by vignettes of uncommon and extreme situations than by common everyday scenarios. In both studies there appears to be a “congruency boundary” beyond which the presence of additional information is unable to affect ratings of the focal stimulus. It is perhaps at this abstract boundary where the two sources of information become dissociated and stop exerting influence over each other.

Alternatively, the absence of a smile type effect in negative situations may be caused by the nature of the negative situation itself rather than its incongruency with smile expressions. When encountering an aversive situation such as a fire or rubbish dump, it is possible that one’s attention to less arousing features of the situation (e.g. the facial expression) is reduced in favour of enhanced focus on the source of the aversion (Chajut & Algom, 2003; Hancock & Warm, 2003). Hence, processing of the subtle morphological differences necessary to distinguish between smile meanings may be reduced in negative scenes but not positive or neutral. Future research should test these two explanations.

Study 2 also produced the interesting and unexpected finding that “enjoyment” contexts (e.g. beaches, parks, fireworks) were rated as more polite/civil than contexts we had labelled as “polite” (e.g. offices, supermarkets, public transport) in the situation-alone block. Although this could raise questions about the validity of our polite stimuli, we note that – as expected – our enjoyment contexts were rated as more enjoyable (M = 85.52) than they were rated polite (M = 73.01), whilst our polite contexts were rated as more polite (M = 65.22) than they were rated enjoyable (M = 48.37). Hence, if we had used a rating system akin to Namba et al. (2020) where ratings across both response categories were complementary and had to add up to 100%, then this would not be an identifiable issue. Furthermore, even though the “enjoyment” contexts were perceived as more polite than “polite” contexts, the context categories are clearly conceptually distinct because they were rated significantly differently across all dependent variables in the situation-alone block. Therefore, rather than being a feature of our specific stimuli, we theorise that our enjoyment contexts may have rated as more polite/civil than our polite contexts because they receive generally higher ratings across all positively-valanced dependent variables, including politeness/civility. Nonetheless, the supposed politeness of these situations is not central to their typical lay conceptualisation, as evidenced by the higher ratings they received along the valence and enjoyableness dependent variables. In contrast, mundane everyday situations like offices and public transport are not strongly associated with any positive dependent variable, but the concept of politeness is more central to them than it is to our enjoyment situations.

Beyond our empirical results, this study contributes further to the study of social interaction by comparing the effects of different contexts across smile types, and vice versa. Apart from Gagnon et al. (2022), previous research has presented participants with just one type of smile, failing to consider the potential for interactions between social cues. However, overlooking these interactions fundamentally limits how we comprehend social processes. During everyday exchanges, interactants integrate a vast array of cues relevant to understanding the feelings and intentions of an interaction partner, including linguistic content, facial expression, tone of voice, body language, eye gaze, situation/setting, interaction history, physiological indicators, dispositional information, gender, age, ethnicity, social group membership, etc. Whilst these variables may have additive effects by combining to strengthen an impression, interactions between variables are inevitable. For example, if a person is smiling, it may indicate the experience of happiness. Nevertheless, if the smile is combined with blushing, stuttering, and gazing at their feet, the person is most likely not happy at all but is trying to mask embarrassment (Brunet et al., 2009). Likewise, relationships between these cues may be reciprocal, as social signals emitted are being constantly modified according to how they have been interpreted (Barrett, 2017).

Such complexity in cue integration has been largely ignored by social psychologists, but has been highlighted by the emerging computational field of Social Signal Processing (SSP – e.g. Brunet et al., 2009; 2012). SSP researchers argue that by focusing on the isolated contribution of single variables, researchers lack the information required to build more powerful and comprehensive models of social behaviour. Instead, they recommend that multiple cues are measured and recorded at the same time. The present study has attempted to make a limited start towards this goal by manipulating both facial expression and situation. However, far more needs to be done to capture the true complexity of social interaction.

Although an important contribution to the literature, there are limitations to the study. For example, it is possible that the results do not generalise beyond the specific set of UK-based participants. Previous research has demonstrated that culture affects both smile evaluations (e.g. Thibault et al., 2012) and the magnitude of context effects in smile ratings (e.g. Namba et al., 2020). It is important to note though that responses did not differ by ethnicity in our studies. However, the findings may not generalise to neuro-atypical groups such as autistic people, who typically struggle to integrate information during social interactions (e.g. Minshew & Goldstein, 1998). Future research could explore these processes in more neuro-diverse and culturally diverse samples.

Furthermore, as with most computerised tasks, our study is only a proxy for the real social world. To show that these psychological processes drive true social behaviour, our results must be replicated in tasks that better approximate live social interaction. For example, we may hope to obtain similar results in tasks whereby participants make consequential behavioural decisions based on information from these social stimuli. Similarly, we acknowledge that our face-situation compound stimuli are not fully realistic, as the faces do not perfectly merge into the background scenery. Therefore, future research may attempt to improve upon their realism by using more advanced image-manipulation techniques. Nonetheless, the use of dynamic faces and backgrounds is novel for assessing how context influences smile evaluations, and our stimuli do improve upon previous studies that have either used (a) static black-and-white photo contexts (Namba et al., 2020) or (b) non-visual verbal vignettes (e.g. Gagnon et al., 2022; Krumhuber et al., 2023; Mui et al., 2020).

Finally, our research only captures a small subset of the various factors that may influence facial expression processing during social interaction. Future research could attempt to assess how manipulating body language, eye gaze, gender, age, and perceiver-based variables, influences processing of different smile types.

To conclude, this research shows that the relationship between smiles and contexts is complex, with interactions between morphological features of the facial expression and the qualitative nature of the situation. Notably, different smile types lacked separable effects in negative scenarios. Such findings have both theoretical and practical implications. Theoretically, this study represents an early step towards understanding the huge complexity of cue integration during social interaction. Practically, greater knowledge of typical social psychological processes is required to help us to better understand how these processes deviate from normality in clinical conditions such as autism and facial palsy. Furthermore, our findings may help to specify constraints for computational models of social interaction.

Acknowledgements

We would like to thank Till Kastendieck for his advice and feedback throughout the entire research. Likewise, we thank Professor Brian Parkinson for his valuable input and comments on earlier drafts of this article.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This work was supported by the Economic and Social Research Council’s (ESRC) Grand Union Doctoral Training Partnership (GUDTP) under Grant ES/P000649/1, and by the Brasenose College Graduate Research Allowance.