Opposite effects of emotion and event segmentation on temporal order memory and object-context binding

ABSTRACT

Our daily lives unfold continuously, yet our memories are organised into distinct events, situated in a specific context of space and time, and chunked when this context changes (at event boundaries). Previous research showed that this process, termed event segmentation, enhances object-context binding but impairs temporal order memory. Physiologically, peaks in pupil dilation index event segmentation, similar to emotion-induced bursts of autonomic arousal. Emotional arousal also modulates object-context binding and temporal order memory. Yet, these two critical factors have not been systematically studied together. To address this gap, we ran a behavioural experiment using a paradigm validated to study event segmentation and extended it with emotion manipulation. During encoding, we sequentially presented greyscale objects embedded in coloured frames (colour changes defining events), with a neutral or aversive sound. During retrieval, we tested participants’ memory of temporal order memory and object-colour binding. We found opposite effects of emotion and event segmentation on episodic memory. While event segmentation enhanced object-context binding, emotion impaired it. On the contrary, event segmentation impaired temporal order memory, but emotion enhanced it. These findings increase our understanding of episodic memory organisation in laboratory settings, and potentially in real life with perceptual changes and emotion fluctuations constantly interacting.

KEYWORDS:

• Emotion
• event segmentation
• temporal order memory
• object-context binding
• aversive sounds

Our daily life is a continuous stream of information. Yet, when we think about the past, we tend to organise our memories into distinct and meaningful events, like chapters in a book. For example, one can remember from yesterday morning that they went running by the lake, where they passed a sailing boat and tried to outtake a cyclist. Then they came back home, took a shower and prepared their favourite coffee. Critically, one can remember each of these individual episodes in a specific context of time and space that defined a lake event and a home event in their memory book. When the spatiotemporal context changes, for example from the lake event to home event, we experience a process termed event segmentation (Zacks & Swallow, 2007). This process was demonstrated to affect how we experience and remember the world (Ezzyat & Davachi, 2014). Various types of context shifts can constitute event boundaries, including perceptual changes in space and time (like in the above example), as well as changes in tasks (DuBrow & Davachi, 2014; Heusser et al., 2018), goals (Magliano et al., 2014), task sets (Wang & Egner, 2022), and thoughts and motivations (Clewett et al., 2019).

Prior research consistently demonstrated that event segmentation drives our memory organisation at the behavioural level. First, it was shown that when the new context representation is unpredictable and surprising – at event boundaries – our attention is drawn to the novel perceptual features of the environment (Huff et al., 2012; Kosie & Baldwin, 2019; Pradhan & Kumar, 2022). As a result, memory for object-context binding (such as object-colour memory) is enhanced for objects presented at event boundaries, compared to the objects presented at other position in the events (Heusser et al., 2018; Siefke et al., 2019). This finding comes from a paradigm that presents sequences of grey objects surrounded by coloured frames, where all the objects presented with the same frame colour constitute one event and a change of frame colour constitutes an event boundary. Using this paradigm, researchers found that memory of the object-colour binding gets enhanced for the first object of a new colour event. On the contrary, it was demonstrated that event segmentation impairs the ongoing temporal integration, affecting memory of temporal order between different elements of events. For example, temporal memory was shown to be worse for items that span an event boundary, or change of context (e.g. objects coming from different frame colours) (Horner et al., 2016; Radvansky & Copeland, 2006), compared to items experienced in the same context (e.g. objects presented with the same frame colour) (DuBrow & Davachi, 2013, 2016; Ezzyat & Davachi, 2011; Heusser et al., 2018).

Recent research increased our understanding of event segmentation mechanisms at the neural and physiological level. fMRI studies showed that complex interactions between the hippocampus and prefrontal cortex drive the integration and separation of memorised information, leading to event segmentation (DuBrow & Davachi, 2014; Ezzyat & Davachi, 2014). Physiologically, it was shown that event boundaries trigger bursts of autonomic arousal, as indexed by peaks of pupil dilation (Clewett et al., 2020). Thus, the physiological mechanism of memory modulation by event segmentation is strikingly similar to the mechanism of memory modulation by emotionally arousing stimuli.

Indeed, emotion is another critical factor having an impact on our experience and memory. This complex construct was defined within several theoretical frameworks, one of the most prominent being affective dimensions (Osgood & Succi, 1957). According to this framework, each emotional experience can be described in terms of at least two orthogonal dimensions: valence (from negative to positive) and arousal (from calm to excited, or emotionally aroused). It has been widely shown that highly arousing and negative emotional events of multiple modalities (Apergis-Schoute et al., 2014; Baumann, 2018; Pavlov et al., 2022) are remembered better than neutral events (Hamann, 2001). However, the existing findings are less homogenous when it comes to the memory of their context, showing either enhancement or impairment of various aspects of contextual memory (Chiu et al., 2013). Only few studies investigated how emotion affects memory of time and these studies also brought mixed results. Some studies showed that participants’ memory of temporal order was impaired for sequentially presented, arousing images (Huntjens et al., 2015), words (Maddock & Frein, 2009) and movie clips (de Montpellier et al., 2021). However, other studies showed enhanced temporal order memory. In one of these studies (Schmidt et al., 2011), participants sequentially encoded positive, negative (both high- or low-arousal) or neutral objects overlaid on neutral scenes, with three unique objects per scene. Participants were instructed to form a story about the content of each scene and memorise it. During a cued recall test, a background scene was presented and participants recalled the names of associated objects, as well as the order of three objects presented with each scene. Their temporal order memory was better for high- vs. low-arousal objects independent of valence, suggesting that arousal enhanced temporal order memory. Similar conclusions come from another recent study using movie scenes and a chronological reconstruction task (Dev et al., 2022). In sum, evidence suggests that emotion can either impair or enhance temporal order memory, depending for example if a common narrative binds them together (Petrucci & Palombo, 2021).

Such “memory trade-offs” for emotional events and their context were first described by Loftus et al. (1987) as “weapon focus effect” (individuals remembered a weapon but not details of the perpetrator or surrounding context), “emotional memory narrowing” (Reisberg & Heuer, 2004) or “tunnel memory” (Safer et al., 1998). Adolphs et al. (2001) observed that emotion enhanced the gist of events but impaired memory for details, and Kensinger et al. (2006; Kensinger & Schacter, 2006) described how emotion can enhance memory for central details at the expense of their peripheral context (see: Williams et al., 2022 for a review). To explain these inconsistent findings, i.e. emotion enhancing or impairing memory, some theoretical models refer to the role of attention during the emotional experience. Since arousing stimuli are considered more perceptually salient, they attract more attention (Sutherland & Mather, 2012; Todd et al., 2019; Vuilleumier, 2005). One prominent theory – the Object-Based Framework (Mather, 2007) – posits that only intrinsic object features (or: within-object binding) should be enhanced by emotion due to a preferential allocation of attentional resources to the objects. On the contrary, extrinsic object features (such as background details or between-object binding) should be either impaired or unaffected by emotion due to reduced allocation of attention to these features. Importantly, this framework also implies that the temporal order memory for emotional items within a sequence would be enhanced if they are bound together over time (e.g. as a part of a broader emotional situation enhancing binding within emotional items), to greater attention and deeper processing of their connectedness (see: Petrucci & Palombo, 2021). Alternatively, temporal order memory for a sequence of emotional items would be impaired if they are perceived as distinct, separate stimuli thus disrupting between-item binding in memory. In addition, more recent theoretical accounts emphasise the role of “emotional context” (understood as subjective states, feelings and physiological reactions related to a stimulus). These models predict that “emotional context” competes for processing resources with the unfolding temporal context that normally gives our memories a sense of temporal cohesion (Talmi et al., 2019).

One account explaining the neural mechanisms of emotion and memory interactions posits that emotion upregulates activity of the amygdala to enhance item memory and downregulates activity of the hippocampus to impair contextual memory (Bisby et al., 2016). Finally, a recent review proposed that this relationship between emotion and temporal memory is bidirectional, such that temporal context representations in the hippocampus, entorhinal cortex, and prefrontal cortex interact with amygdala to shape emotional functioning and vice versa (Wang et al., 2022).

In sum, based on these different views, it remains unclear how emotion signals would affect the formation of event boundaries and the temporal binding of successive stimuli within or across boundaries. Even though event segmentation and emotion are key factors affecting our memory, and their physiological and neural mechanisms are strikingly overlapping (Clewett et al., 2017; Clewett & McClay, n.d.), they have not been systematically studied together. In other words, we do not know how one would remember the lake event or the house event from this morning if a scary thunderstorm happened during their morning run, or on their way home.

To address this gap, we manipulated both event segmentation and emotion, focusing on how these factors affect (i) object-context binding and (ii) temporal order memory. First, based on prior theoretical models and empirical research on event segmentation in memory carried in non-emotional contexts (Clewett & Davachi, 2017; Heusser et al., 2018; Kurby & Zacks, 2008; Zacks, 2020), we expected to replicate the effect of perceptual event boundaries (here: changes of a frame colour) as enhancing memory for the associative binding of objects and contextual elements (here: frame colours), leading to a better memory for the frame colour that surrounded the objects at event boundaries. Moreover, we expected to replicate a previously demonstrated effect of disrupted temporal order memory between pairs of items spanning perceptual event boundaries compared to item pairs not spanning boundaries (Clewett et al., 2019; Davachi & DuBrow, 2015; Ezzyat & Davachi, 2011; Horner et al., 2016). Second, based on previous theoretical accounts and empirical studies on memory modulation by emotion (Cooper & Ritchey, 2019; Mather, 2007; Mather et al., 2016; Palombo et al., 2021; Rimmele et al., 2012; Yonelinas & Ritchey, 2015), we hypothesised that overall, object-colour memory would be impaired for any objects paired with an aversive sound, regardless of their sequential position in the events. In terms of temporal memory, we hypothesised that presenting an object with an aversive (as opposed to neutral) sound provides a saliency signal (Ponzio & Mather, 2014; Sakaki et al., 2014; Sutherland & Mather, 2012) that drives enhanced processing and therefore acts as a boundary. Specifically, the emotional sound could have a similar effect as an event boundary, caused by emotional significance rather than perceptual change, and leading to lower temporal order memory. Finally, we hypothesised that the effects of perceptual changes and emotional saliency signals might have additive effects when they co-occur (Geerligs et al., 2022; Pradhan & Kumar, 2022) leading to an even stronger event segmentation and disruption of temporal order memory.

To verify these hypotheses, we ran two behavioural experiments (Exp.1, Exp.2). In each experiment, we used an experimental paradigm validated to study event segmentation (Heusser et al., 2018) and now extended with emotion manipulation (Exp.1) or oddballness manipulation (Exp. 2). In Exp. 1, participants encoded greyscale objects sequentially presented in blocks of a particular colour of the frame, together with a neutral or aversive sound. During retrieval, the participants completed several memory tests: temporal order, item recognition, and object-colour memory. In the control Exp. 2, the paradigm was identical except that we used another neutral, oddball sound that was presented at the same frequency as the aversive sound in Exp.1. This way, we controlled for possible oddballness effects underlying the observed emotion effects.

Exp.1. Emotion effects

In Exp. 1, we examined how two critical factors: emotion and event segmentation affect two aspects of episodic memory: (i) object-context binding and (ii) temporal order memory.

Methods

Participants

We recruited 13 female and 12 male participants; n = 25, aged 18–30 (M = 23.68, SD = 2.21). All participants were advanced English speakers with no history of neurological illness or treatment with psychoactive drugs, with normal hearing and normal or corrected-to-normal vision. Participants were students and young professionals living in Geneva, with at least secondary education. Two subjects were not included in the analyses of recognition and source retrieval performance due to a randomisation error and one subject failed to understand task instructions (resulting in a final sample of n = 22). The participants were recruited through social media and communication channels at the University of Geneva. All participants provided informed consent and were financially compensated with 30 CHF. The local ethics committee approved the experimental protocol of the study.

To determine the sample size, we first averaged the effect sizes reported in previous experiments showing event segmentation effects on temporal order memory (Clewett et al., 2020; Heusser et al., 2018). This average effect size (Cohen’s d) was .91. Based on that, we calculated that a minimum of 18 participants would be required in order to have a power of .8 (alpha = .05) to detect the effects of event segmentation on temporal memory of this magnitude (G*Power 3.1). Given that no previous study included both factors (event segmentation and emotion) in a within-subject ANOVA analysis, we decided to increase the sample by 25% and test approximately 25 subjects.

Stimuli

Objects. We used a stimulus subset consisting of 482 grey-scale pictures of everyday objects coming from a bigger set used in previous studies (Brady et al., 2008; Heusser et al., 2018). Each picture was resized to 350 × 350 pixels.

Frame colours. Twenty-four unique colours were selected from the whole colour continuum ranging from [0,0,0] to [255, 255, 255] RGB values, to avoid any confounds of affective value linked to particular colours (Kuhbandner & Pekrun, 2013). Each frame colour could only appear once per list and was re-used after seven (half of all) lists.

Sounds. The aversive sound was taken from the study by (Levita et al., 2009), who reported that this sound evokes negative valence ratings and increased skin conductance responses indicative of inducing negative emotion. The same aversive sound was also shown to efficiently serve as a US in a fear conditioning paradigm by (Johnson & Casey, 2015). The neutral sound was originally taken from the IADS dataset, cropped from 4s to 2s duration and validated by (Clewett & McClay, n.d.) on the scales of arousal (M = 2.08, SD = .90), valence (M = 4.00, SD = .60), and ambivalence (M = 3.25, SD = 2.38). The volume (digital intensity) of the aversive sound was higher than the volume of the neutral sounds, based on the previous literature investigating memory modulation by emotional sounds (Sutherland & Mather, 2012) and given the ecologically valid volume of sounds available from IADS dataset.

The pairing of objects, frame colours and sounds was randomised across subjects. The side of the presentation of images for the temporal order task was randomised across trials.

Study design and experimental procedure

The experiment consisted of 14 encoding lists comprising 24 objects each (Figure 1). Immediately after each encoding list, memory for temporal order was assessed for six object pairs. Following the encoding and temporal memory tests of all 14 lists, a surprise memory test was administered that assessed memory for the remaining 140 objects across all 14 lists for which memory had not been tested in the temporal memory tests. The first and the last objects from each list were not tested to avoid any primacy and recency effects.

Figure 1. Schematic of experimental procedure. Encoding task with sequentially presented grey-scale objects and experimental manipulation of 2 factors: event segmentation (changes of the frame colour) and emotion (neutral or aversive sound). Temporal order memory test administered immediately after each of 14 lists. Surprise object-colour test administered at the end of the experiment, after all 14 lists of encoding and temporal order memory tests.

Before starting the experiment, participants were presented with the instructions, signed the consent form, and filled in a socio-demographic form (age, gender, education), and performed a brief practice session to ensure that they understood the task. The last part of the experiment (a surprise memory test) was not included in the practice session. The experimental procedure was programmed using PsychoPy v2021.2.3 (Peirce et al., 2019).

Encoding. During each encoding list, 24 unique grey-scale objects were sequentially presented surrounded by a coloured frame and accompanied by a neutral or an aversive sound (here called: a neutral object and an emotional object, respectively). Each sound accompanied the presented object for a fixed duration of 2 s. Most objects (22 out of 24 objects per list) were paired with a neutral sound, and only some, infrequent objects (two out of 24 objects per list) were paired with an aversive sound.

The colour of the frame was identical for six consecutive objects (defining one event), then switched to a new colour for the next six objects (another event). There were four events per encoding list. On the trials starting a new event (here called boundary trials), the frame colour changed at the onset of the presentation of the boundary object. All other trials (event positions 2–6) are defined as nonboundary trials. Objects were presented for a fixed time of 3 s with a fixed 2 s intertrial interval (ITI) followed by a 500 ms fixation cross before the onset of the next trial. The coloured frame remained on the screen continuously during the presentation of each event and only changed concurrently with boundary objects.

The participants were instructed to imagine an object in the colour of its frame and holistically evaluate if the object-colour pair, presented with a particular sound, was pleasant to them. To indicate their decision, the participants were asked to press one of the two buttons on the keyboard (“j” pleasant, “k” unpleasant). This was possible only during the combined object and sound presentation (2 s). This simple task (as in the original paradigm from Heusser et al., 2018) allowed us to encourage forming episodic memories of objects within context, as well as to obtain subjective emotional evaluation of each trial by our participants. Additionally, the participants were asked to imagine the objects interacting with each other over time as an associative memory strategy to enhance their memory of temporal order of objects. Critically, participants were instructed to associate objects irrespective of the presence of different colour frames and different sounds. This instruction was added (as in the original paradigm from Heusser et al., 2018), because after the first temporal order memory test, participants may spontaneously adopt this kind of strategy to be successful on temporal order memory judgments in subsequent lists (hence we encouraged all participants to use the same strategy). Overall, these complex instructions were based on the original study (Heusser et al., 2018) to keep the studies as similar as possible. This would allow us to compare our results with the original findings and interpret any differences in terms of emotion effects.

Temporal memory tests. Following each encoding list, participants’ temporal order memory of the encoding list was assessed for six object pairs (Figure 1). During each trial of this session, two previously studied objects were presented on the screen. To assess memory of temporal order, participants were asked to indicate which of the two objects appeared first in the list by pressing one of two buttons on the keyboard (“1” object on the left, “0” object on the right). The tested objects had always been presented three objects apart during encoding, so the actual number of intermediate elements between experimental conditions was constant.

Hypotheses. Based on prior research, we expected that perceptual event boundaries, i.e. a change in frame colour, would disrupt temporal order memory between pairs of items spanning these perceptual boundaries compared to item pairs that did not span perceptual boundaries (Clewett et al., 2019; Davachi & DuBrow, 2015; Ezzyat & Davachi, 2011; Horner et al., 2016). To assess whether event segmentation indeed impaired temporal order memory, we compared temporal memory for object pairs paired with neutral sounds, coming from different events, i.e. had been presented with different frame colours (across-event neutral) vs. temporal order memory for pairs that had been presented with the same frame colour (within-event neutral).

Moreover, based on previous studies suggesting that emotion affects various aspects of temporal memory (for a review, see: Petrucci & Palombo, 2021), we predicted that presenting an object with an aversive (as opposed to neutral) sound provides a saliency signal that drives enhanced processing (Ponzio & Mather, 2014; Sakaki et al., 2014; Sutherland & Mather, 2012) and acts as another form of a boundary. Thus, the emotional sound could have a similar effect as perceptual boundaries and impair temporal order memory. To test this hypothesis, we compared temporal memory for pairs coming from the same frame colour but spanning an object paired with an emotional sound (within-event emotional) to temporal memory for neutral within-event pairs that spanned objects paired only with neutral sounds (within-event neutral), and to temporal memory for neutral across-event pairs that spanned objects paired only with neutral sounds (across-event neutral).

Additionally, we hypothesised that the effects of perceptual changes and emotional saliency signals could have additive effects leading to stronger event segmentation (Geerligs et al., 2022; Pradhan & Kumar, 2022). To test this hypothesis, we compared temporal memory for neutral object pairs spanning both a frame colour change and an object paired with an aversive sound (across-event emotional) to temporal memory for neutral pairs spanning a frame colour change but only over objects paired with neutral sounds (across-event neutral).

Surprise retrieval test. Following the final temporal memory test and a 5-minute break, a surprise memory test was administered. The test included an item recognition test and object-colour binding (i.e. source memory) test (Figure 1). Importantly, this test only included objects that were not used in temporal memory tests, i.e. 140 “old” objects, and 140 “new” objects. For each test trial, participants were first asked to indicate whether the presented object had been a previously studied object (“old”) or not (“new”). If they correctly replied “old”, the object remained on the screen, and two colours below the object, positioned on the left and right side of screen, appeared. One of these colours (target) had been originally paired with the object while the other colour (lure) was always one of the coloured frames that had been presented during the preceding or following event. The lure colour was counterbalanced in the object-colour recognition test, such that it was equally likely to precede or follow the target colour. Target and lure colours were also equally likely to appear in the left or right positions on screen during recognition test. If the participants incorrectly replied “old” to a new object, two lure colours were presented under the object. These test trials were self-paced and advanced after a response was given by a participant. The ITI between trials was fixed to 1 s. Since we did not have clear hypotheses about the results of recognition memory test, the results will be reported only in supplementary materials.

Hypotheses. Prior research showed that perceptual event boundaries – here, a change in frame colour – causes a shift in attention from a preceding to a novel colour of the frame. As a result, it enhances memory for the coloured frame that surrounded the object (i.e. associative colour-object binding) for objects learnt right after the boundaries (Clewett & Davachi, 2017; Heusser et al., 2018; Kurby & Zacks, 2008; Zacks, 2020). To replicate these findings, we compared the object-colour memory for the first objects presented with neutral sounds after a change in the colour (boundary neutral) with object-colour memory for objects that were studied in the middle of the events (nonboundary neutral).

At the same time, existing evidence suggests that emotion enhances memory for central items but impairs memory for contextual details (Cooper & Ritchey, 2019; Mather, 2007; Mather et al., 2016; Palombo et al., 2021; Rimmele et al., 2012; Yonelinas & Ritchey, 2015). Hence, we expected that overall, object-colour memory would be impaired for any objects paired with an aversive sound, regardless of their sequential position in the events. To verify this hypothesis, we compared object-colour memory for any objects paired with an aversive sound (emotional boundary and emotional nonboundary) and object-colour memory for objects paired with a neutral sound (boundary netural and nonboundary neutral).

Data analysis

The dependent variables analyzed in the present study included the proportion of pleasantness ratings and memory performance in all memory tests. We calculated the proportion of “pleasant” ratings among all given (“pleasant” or “unpleasant”) ratings. Temporal order memory was calculated as percentage of correct responses. Memory performance in the recognition test was calculated as the percentage of “old” objects correctly recognised as “old”, and object-colour associative memory was calculated as the percentage of correctly retrieved frame colours for the objects that were correctly recognised as “old”. Chance level in all memory tests was 50%, given two possible responses. We manipulated two factors: event segmentation (within-event vs. across-event and nonboundary vs. boundary) and emotion (two levels: neutral vs. emotional). Therefore, we analysed memory performance as a function of these two factors.

To analyze the data, we performed a repeated-measures analysis of variance (rm ANOVA) in a 2 × 2 design, with emotion and event segmentation as within-subject factors. Post hoc comparisons were performed using the Bonferroni adjustment for multiple comparisons. The effect sizes of the t-test results are reported as Hedges’ g and interpreted it as small effect if  = > .2, medium effect if  = > .5 and large effect if  = > .8. The results were visualised with the use of Python-based Seaborn package (Waskom, 2021) and Statannot package (https://github.com/webermarcolivier/statannot).

Results

Pleasantness ratings

First, we aimed to validate our experimental manipulation of the emotional vs. neutral sound affecting the pleasantness ratings of the objects differently. As presented in Figure 2a, we found that participants (n = 25) less frequently rated objects presented with an aversive sound as pleasant (M rate = .44, SD = .04) compared to objects presented with neutral sounds (M rate = 60, SD = .04), main effect of emotion [F(1,24) = 8.147, p = .009, η²= .253]. We did not find any effect of event segmentation on the pleasantness ratings [F(1,24) = .129, p = .723, η²= .005], and no interaction between these two factors [F(1,24) = .0, p = .983, η²= .0].

Figure 2. (a) Pleasantness rate (% of “pleasant” ratings) as a function of event segmentation (nonboundary vs. boundary) and emotion (neu – paired with a neutral sound, emo – paired with an aversive sound); (b) Temporal order memory as a function of event segmentation (within-event vs. across-event) and emotion (neu – spanning neutral sounds, emo – spanning an aversive sound); (c) Object-colour memory (for objects correctly recognized as “old” = cond) as a function of event segmentation (nonboundary vs. boundary) and emotion (neu – paired with a neutral sound, emo – paired with an aversive sound); error bars represent one SD, dots represent individual subjects’ scores; ∗ p < .05, ∗∗ p < .005.

Temporal order memory

Next, we analyzed temporal order memory as a function of emotion and event segmentation (Figure 2b). We found that both emotion [F(1,24) = 10.937, p = .003, η²= .313] and event segmentation [F(1,24) = 5.762, p = .024, η²= .194] influenced how well the participants (n = 25) remembered temporal order of object pairs, but there was no interaction between the two factors [F(1,24) = .506, p = .484, η²= .021]. Specifically, we replicated a finding from previous literature showing that overall, temporal order memory was impaired for across-event object pairs (M = .73, SD = .02) compared to within-event pairs (M = .77 SD = .02). As for the main effect of emotion, we found that participants better remembered temporal order of object pairs spanning an emotional object (M = .77, SD = .02) compared to object pairs spanning neutral objects (M = .73, SD = .02). Next, we ran a series of post-hoc tests to verify our a priori hypotheses. We found that the main effect of event segmentation (within-event > across-event) was driven by objects spanning neutral objects [t(24) = −2.871, p = .008, g = -.52], but was not significant for the objects spanning emotional objects [t(24) = −1.091, p = .286, g = -.286]. Temporal order memory did not differ between neutral and emotional within-event object pairs [t(24) = 1.503, p = .146, g = .268], but it was better for emotional than for neutral across-event pairs [t(24) = 2.291, p = .031, g = .453].

Figure 3. (a) Pleasantness rate (% of “pleasant” ratings) as a function of event segmentation (nonboundary vs. boundary) and oddballness (neu – paired with a neutral sound, neu2 – paired with an oddball neutral sound); (b) Temporal order memory as a function of event segmentation (within-event vs. across-event) and oddballness (neu – spanning neutral sounds, neu2 – spanning an oddball sound); (c) Object-colour memory (for objects correctly recognized as “old” = cond) as a function of event segmentation (nonboundary vs. boundary) and oddballness (neu – paired with a neutral sound, neu2 – paired with an oddball sound); error bars represent one SD, dots represent individual subjects’ scores; ∗∗ p < .005, ∗∗∗ p < .001.

Object-context binding

Both emotion and event segmentation were previously demonstrated to affect non-temporal aspects of episodic memory, such as the association of object with its context (Clewett et al., 2019). Here, we tested memory for object-colour binding for objects correctly recognised as “old” (Figure 2c). We found a main effect of event segmentation [F(1,21) = 11.25, p = .003, η2 = .349] indicating that participants (n = 22) remembered the colour of the surrounding frame better for boundary (M = .70, SD = .03) vs. nonboundary objects (M = .57, SD  = .03). In addition, we found a main effect of emotion [F(1,21) = 4.482, p = .046, η2 = .176] showing that participants’ memory of the colour was impaired for objects that had been paired with an aversive sound (M = .66, SD = .03) compared to neutral objects (M = .61, SD = .03). The interaction of event segmentation and emotion was not significant [F(1,21) = .25, p = .875, η2 = .001].

Notably, we did not observe any effect of emotion on recognition memory [F(1,21) = .669, p = .423, η2 = .031], and no interaction effect of emotion and event segmentation [F(1,21) = 1.177, p = .29, η2 = .053]. However, we found that recognition memory was modulated by event segmentation [F(1,21) = 5.579, p = .028, η2 = .21], such that overall nonboundary objects (M = .81, SD = .03) were remembered better than boundary objects (M = .77, SD = .04).

Exp. 2. Control for the oddballness effects

In Exp. 1, we found that event segmentation and emotion both affected object-colour binding and temporal order memory in a different manner. However, in our experimental paradigm, most objects were presented with a neutral sound and only some infrequent objects were presented with an aversive sound. This difference in frequency could lead to a confound such that emotional objects were the oddball ones in a sequence of encoding. This could result in the isolation or differentiation effects in memory (known as von Restorff effects). The goal of Exp. 2 was to control for this possible confound of oddballness.

Methods

Participants

We recruited 13 female and 13 male participants; n = 26, aged 19–29 (M = 22.36, SD = 2.67) for Exp. 2. All participants were advanced English speakers with no history of neurological illness or treatment with psychoactive drugs, with normal hearing and normal or corrected-to-normal vision. Participants were students and young professionals living in Geneva, with at least secondary education. The participants were recruited through social media and communication channels at the University of Geneva. All participants provided informed consent and were financially compensated with 30 CHF. The local ethics committee had approved the experimental protocol of the study.

Stimuli

We used the same objects and frame colours as in Exp. 1. As for the sounds, we used the same frequently played, first neutral sound as in Exp. 1. Instead of an aversive sound as in Exp.1, we used another self-generated neutral sound. This second neutral sound in Exp. 2 was matched with the aversive sound in all the spectro-temporal characteristics except for the ones determining auditory salience and evoked aversiveness, i.e. roughness, pitch and intensity (Arnal et al., 2015, 2019; Legendre et al., n.d.). Specifically, this second neutral sound was a pure tone of a lower pitch and intensity than the aversive sound. The volume (digital intensity) of the first and second neutral sound was identical (for more details see: supplementary materials, Figure S3).

Study design and experimental procedure

The experimental procedure was identical to Exp. 2, with the exception of using two neutral sounds (see: Stimuli) instead of one neutral and one aversive sound as in Exp. 1. As in Exp. 1, most objects (22 out of 24 objects per list) were paired with the first neutral sound, and only some, infrequent objects (two out of 24 objects per list) were paired with the second neutral sound. Thus, the structure was similar to typical oddball paradigms (over 80% of standard trials and less than 20% of deviant trials) (Schlüter et al., 2019) and we will refer to the second neutral sound as an oddball sound.

Data analysis

As in Exp. 1, we analyzed the effects of event segmentation (within-event vs. across-event and nonboundary vs. boundary) on memory. Instead of the second factor – emotion, here we analyzed the effects of oddballness across two levels: neutral (i.e. the frequent neutral sound) vs. oddball (the second neutral sound) on memory. Therefore, we analysed memory performance as a function of these two factors.

Similar to Exp. 1, we performed a repeated-measures analysis of variance (rm ANOVA) in a 2 × 2 design, with oddballness and event segmentation as within-subject factors. Post hoc comparisons were performed using the Bonferroni adjustment for multiple comparisons. The results were visualised with the use of Python-based Seaborn package (Waskom, 2021) and Statannot package (https://github.com/webermarcolivier/statannot).

Results

Pleasantness ratings

First, we compared the pleasantness ratings that the participants (n = 26) gave for objects paired with the frequent neutral sound (i.e. neutral objects) vs. the oddball sound (i.e. oddball objects). As shown in Figure 3a, we found no effect of oddballness [F(1,25) = 1.168, p = .29, η²= .005] and no effect of event segmentation [F(1,25) = 3.389, p = .078, η²= .016] on pleasantness ratings. No interaction effect between these two factors was observed [F(1,25) = .872, p = .359, η²= .003]. This means that pleasantness ratings did not differ between neutral object and neutral oddball objects, and that event segmentation had no influence on pleasantness ratings.

Temporal order memory

Next, we analyzed temporal order memory as a function of oddballness and event segmentation (Figure 3b). We found no effect of oddballness [F(1,25) = 3.037, p = .094, η²= .016]. On the contrary, we observed a very strong effect of event segmentation [F(1,25) = 19.3, p < .001, η²= .119], such that participants remembered temporal order better for within-event (M = .748, SE = .02) than across-event (M = .658, SE = .02) pairs. There was no interaction effect between these two factors [F(1,25) = .128, p = .723, η²= .001]. Next, we ran a series of post-hoc tests to verify our a priori hypotheses. We found that the main effect of event segmentation (within-event > across-event) was present both for objects spanning other neutral objects [t(25) = −3.83, p = .002, g = -.896], and for the objects spanning an oddball neutral object [t(25) = −3.109, p = .005, g = -.627]. This result is another replication of the effect of temporal memory impairment due to event segmentation, both for the neutral and the neutral oddball objects (unlike emotional objects in Exp. 1 where the effect was not significant anymore). This suggests the specificity of emotion effects in Exp.1 rather than their frequency.

Object-context binding

Finally, we also analyzed object-colour memory. Unlike other analyses included in the present manuscript, this analysis revealed outlier mean scores (that is, values above Q3 + 1.5 x IQR or below Q1 - 1.5 x IQR identified with box plots), which we subsequently excluded (n = 6, with the remaining sample n = 19). We did not observe any effect of oddballness on object-colour binding [F(1,19) = .864, p = .364, η²= .007], which speaks for the specificity of emotion effects observed in Exp.1, where we did find an effect of emotion on object-colour binding. At the same time, we did observe the expected effect of event segmentation [F(1,19) = 5.474, p = .03, η²= .088] and an interaction effect between these two factors [F(1,19) = 8.793, p = .008, η²= .040]. Post-hoc comparisons revealed that object-colour memory was higher for the oddball neutral objects at event boundaries (M = .693, SE = .03) than nonboundary oddball neutral objects (M = .541, SE = .04) [t(19) = −2.89, p = .019, g = .942]. There was no significant difference in object-colour memory between the neutral boundary (M = .657, SE = .03) and neutral nonboundary (M = .627, SE = .02) [t(19) = -.90, p = .379, g = .219] objects. These results replicate the previously shown enhancement of item-context binding at event boundaries for the neutral oddball objects. However, we did not find the same for the frequent neutral objects. This may be due to an additional allocation of attention to the oddball objects and a lower attention allocation to neutral objects, no longer boosted by the changes in frame colours occurring at event boundaries.

In terms of recognition memory, we did not observe any effect of oddballness [F(1,25) = .794, p = .381, η2 = .001], any effect of event segmentation [F(1,25) = 2.595, p = .12, η2 = .007] and any interaction effect between these two factors [F(1,25) = 1.198, p = .284, η2 = .002].

Discussion

Although our present experience unfolds continuously, our memories of the past are organised into coherent and meaningful events. Understanding which processes guide such memory organisation is critical for our everyday life functioning. Here, we found that perceptual context shifts (changing colour of the objects’ frame) and emotion (aversive sounds) influenced episodic memory in a different way. First, for object-context binding, we replicated prior studies, showing that participants better remembered the frame colour of the boundary objects (when the frame colour changed) compared to objects from any other event position. At the same time, object-colour binding was generally impaired for emotional objects. Second, we replicated a previously shown effect on temporal order memory, showing that it was impaired for object pairs coming from different colour events compared to object pairs coming from the same event. On the contrary, temporal order memory was generally enhanced if the object pairs spanned emotional objects. Finally, we ran an additional control experiment where instead of an aversive sound, we used another neutral sound that was presented at the same frequency as the aversive sound in the original experiment. The results of this control experiment strengthened our main findings by showing that they were specific to emotion rather than simply novelty or oddballness of infrequently presented sounds.

In the following subsections, we will first discuss the conditions limiting possible interpretation of the present findings. Second, we will discuss the main findings in light of previous event segmentation and emotion literature concerning object-context binding and temporal order memory. Finally, we will discuss general implications of our results. Three features of this work limit the conclusions we can draw about the influence of emotion and event segmentation on the episodic memory organisation. First, the present findings are limited to behavioural measures. As such, we cannot disentangle distinct contributions of factors such as autonomic arousal and attention allocation. Future studies could extend the present findings with evidence at the physiological and neural level to better understand underlying mechanisms and individual differences. Second, the complex instructions during our encoding task could have added an additional memory load or lead to a divided attention. However, prior studies showed that divided attention did not affect the emotional memory enhancement for arousing words or items (Kensinger & Corkin, 2004; Mickley Steinmetz et al., 2014) and poor attentional states did not affect event segmentation effects on temporal memory (Jayakumar et al., 2023). Third, although we used a well-established task to measure event segmentation and validated our experimental manipulation of emotion, this laboratory task has limited ecological validity. It would be interesting for the future studies to explore these issues in more naturalistic context of time and space, such as virtual reality.

Event segmentation and emotion affect object-context binding

Initial investigation of event segmentation in memory focused on memory enhancement for boundary items (Boltz, 1992; Newtson & Engquist, 1976; Schwan & Garsoffky, 2004). Subsequent studies extended this knowledge by showing that context shifts may also enhance object-context binding such that also contextual details are better remembered (Clewett et al., 2020; Heusser et al., 2018). Our results replicated this effect by showing that the participants better remembered the frame colours for objects presented at event boundaries (when the colour changed). However, contrary to some previous studies (Schwan & Garsoffky, 2004), we observed the opposite effect of event segmentation on recognition memory (see: supplementary materials). This may be specific to the recognition “old/new” test. It would be interesting for future studies to probe memory with the recall test, as it should engage both item memory and temporal memory, and was shown to reflect event segmentation in memory as well (Heusser et al., 2018).

Overall, emotion impaired object-colour memory. This results is partly in line with previous studies typically showing the emotional memory enhancement for intrinsic contextual details but emotional memory impairment for extrinsic contextual details (Chiu et al., 2013; Kensinger, 2009; Mather, 2007; Mihaylova et al., 2019). During encoding, the participants were instructed to imagine each object in the colour of its frame, strengthening the object-colour association, and potentially causing colour to become an intrinsic detail. However, the frame colour was not a constituent feature of the object in this particular task, which explicitly asked participants to only study the temporal order of objects (and not object-colour memory as well). Thus, the frame colour should be considered an extrinsic detail (Rimmele et al., 2012; Staresina & Davachi, 2008, 2010). As such, the overall impairment of object-colour binding due to emotion in our results was in line with previous literature (Antypa et al., 2019; Rimmele et al., 2011, 2012), including sequential paradigms with temporal context (Palombo et al., 2021).

Surprisingly, we did not observe an interaction of emotion with event segmentation, which was expected to be driven by the saliency of emotional signals regardless of their position in the events or ongoing perceptual changes. In other words, object-colour memory for emotional nonboundary objects was not equally high as for boundary objects, neither emotional nor neutral. This result suggests that emotion and event segmentation have distinct contributions shaping object-context binding in our episodic memory. While boundary objects benefit the in-the-moment binding of boundary objects with their new context in our memory, emotion generally enhances item memory at the expense of contextual memory. Of note, such item memory enhancement typically occurs or increases with consolidation (Cox et al., 2023; Sharot & Yonelinas, 2008; Yonelinas & Ritchey, 2015 for a review). Thus, it will be interesting for future studies to extend our paradigm by testing recognition memory after longer delays.

Event segmentation and emotion affect temporal order memory

Prior studies investigating the effects of emotion on temporal order memory brought mixed results (for a review, see: Petrucci & Palombo, 2021). Some of these findings concern temporal source memory, i.e. remembering in which block of the task or at what point during video a neutral or emotional object appeared. These studies showed higher temporal memory accuracy (Rimmele et al., 2012) and less bias (but no effect on precision) in temporal reporting (Palombo et al., 2021) for emotional events. In terms of temporal order memory, some studies demonstrated that emotion impaired temporal order memory for sequentially presented, arousing images (Huntjens et al., 2015), words (Maddock & Frein, 2009) and movie clips (de Montpellier et al., 2021). Other studies, however, showed enhanced temporal order memory for emotional stimuli, both encoded incidentally (Dev et al., 2022) and intentionally (Schmidt et al., 2011). Such a discrepancy in previous results may result from different types of binding recruited in experimental tasks (Palombo & Cocquyt, 2020), and different types of emotion manipulation. Emotion seems to enhance temporal order memory if the stimuli are encoded with an overarching storyline that binds them together (Petrucci & Palombo, 2021). Even though we did not expect similar effects from the instruction that we introduced in our study (i.e. “Imagine the objects interacting with each other over time”), it possibly enhanced building a strong, common narrative. While typically the experiments used visual emotional stimuli by showing emotional pictures or movie clips, we used an auditory manipulation with aversive sounds. We validated this experimental manipulation by showing that participants rated objects paired with aversive sounds as less pleasant than objects paired with neutral sound. Yet, using auditory stimuli could possibly disrupt temporal memory less than visual emotional stimuli of a single modality.

Crucially, we did not observe a significant interaction effect of emotion and event segmentation on temporal order memory. Existing evidence suggests that these two factors could have an additive effect, resulting in a stronger temporal memory disruption by emotional boundaries compared to neutral boundaries (Clewett & McClay, n.d.). In their study, participants also encoded sequentially presented objects. Instead of frame colour changes, pure tones were played at their right or left ear to define a stable auditory context, whereas other emotionally negative, positive or neutral naturalistic sounds defined event boundaries. Using generalised mixed effects models (GLMMs), the authors found that event segmentation impaired temporal order memory, which aligns with our finding that event segmentation impaired temporal order memory for across-event compared to within-event object pairs. In addition, they found that the more arousing a sound defining the event boundary was, the more temporal order memory was impaired for objects spanning this boundary. Although this latter finding seems contrary to our findings, these results cannot be directly compared due to above mentioned differences in the experimental paradigms. In addition, our study differs from their study in its analytical approach (we focused on comparing the mean temporal memory performance whereas Clewett & McClay focused on single-trial effects) and research questions asked. We both investigated whether emotional event boundaries could be stronger than neutral boundaries, yet Clewett & McClay also asked how these emotional boundaries affect memory for subsequent event (emotional carry-over effect). Our post-hoc comparisons showed that emotional boundaries disrupted temporal order memory less than neutral boundaries (in line with an impaired object-colour memory for emotional compared to neutral objects). This reduced “cost of boundary” suggests that emotion provided a strong binding context despite perceptual context shifts. Moreover, we expected that similar to reward prediction errors, objects paired with aversive sounds could act as event boundaries that interrupt the sequential integration of events (Rouhani et al., 2019). This should result in lower temporal order memory for object pairs spanning emotional sounds within the same event, comparable to temporal order memory for object pairs spanning neutral boundaries and possibly emotional boundaries. However, we did not observe such a pattern of results.

Event segmentation but not emotion affect item recognition

In our study, we used a paradigm that was originally designed (Heusser et al., 2018) to investigate how event segmentation affects item-context binding. Extending the original paradigm, we included an item recognition memory test. Interestingly, we observed an enhancement of item recognition for nonboundary compared to boundary objects. This enhancement of item recognition for nonboundary compared to boundary objects is consistent with previous findings (Morse et al., n.d.). Other studies using similar sequential paradigms reported no effects of event segmentation on item recognition (DuBrow & Davachi, 2013). Yet other studies using movie-based paradigm, found the opposite effect, i.e. enhanced item recognition for items that had appeared at event boundaries compared to items that appeared in the middle of events (Gold et al., 2017; Schwan & Garsoffky, 2004).

A potential explanation of these discrepant findings may be due to attentional processes occurring at event boundaries. Indeed, most influential theories of event cognition suggest that event boundaries trigger a prioritisation of the new sensory inputs (Zacks & Swallow, 2007). This increase in attention might lead to better encoding of information at event boundaries. Accordingly, eye-tracking evidence suggests that memory enhancements at event boundaries depend on whether those boundary items are explicitly attended at that moment, as indexed by fixations (Swallow et al., 2009). Similarly, attentional processes likely enhanced item recognition in the movie paradigm mentioned above, where participants’ attention was focused on boundary items cued with visual or auditory stimuli (Gold et al., 2017). However, in our study the encoding task may have guided attention away from the boundary items to the non-boundary items. In particular, we had asked participants to imagine the object in the colour of the frame, holistically evaluate the object-colour pair and to imagine the objects interact with each other over time. Such prioritised integration of coloured frames with the objects and the binding of objects across time may thus have guided attention to the non-boundary items resulting in better item recognition memory for non-boundary vs. boundary items.

It should also be mentioned that we observed no effect of emotion on item recognition, a result that differs from prior theoretical models and empirical research suggesting that emotion typically enhances item recognition (Bisby et al., 2016; Dolcos et al., 2005). Several reasons may explain the absence of an emotion-induced recognition memory enhancement in our study. First, our recognition memory test did not distinguish between recollection and familiarity (as in “remember/ know” test, for instance). Given that emotion effects on memory typically affects recollection but not familiarity (Yonelinas & Ritchey, 2015), we possibly did not capture these recollection effects. Second, memory performance in the recognition test was very high among our participants (over 80%), and possibly there was not enough variability to reveal any significant differences due to emotion. Third, we may not have captured emotion effects on the recognition memory test because our memory test immediately followed encoding. The emotional memory enhancement has been shown to be time-dependent (Yonelinas & Ritchey, 2015). Specifically, it was shown that the item memory enhancement and the context memory impairment might be reliably observed only after one or two weeks delay (Cox et al., 2023). Forth, prior studies typically tested recognition memory for emotional vs. neutral items per se. On the contrary, we tested memory for objects paired with an aversive or neutral sounds, not memory for the sounds per se. Finally, our results may also be influenced by the encoding task demands with instructions prioritising both the integration of objects with their coloured frames, and the temporal integration between them. In line with the Object-Based Framework (Mather, 2007), arousal should in fact amplify a prioritisation in attention and memory for the information that is goal relevant (here the object-colour binding as prioritised due to task instructions).

Effects of emotion on object-context binding and temporal order memory are not due to oddballness

In Exp.1, the loud and unpleasant aversive sound was presented less frequently than the neutral sound, and thus could have been more distracting. In other words, due to this difference in frequency and auditory properties, emotional objects could be considered oddballs in the encoding sequence. Further, one could argue that the effects that we observed were due to the sound being distracting rather than being emotional. To control for this potential confound, we ran a control Exp. 2. in which we replaced the aversive sound with another neutral sound. As in Exp. 1, this sound was less frequent than the first neutral sound. In addition, we carefully controlled for the auditory properties of this oddball neutral sound in order to isolate purely emotional properties of the aversive sound used in Exp. 1. First, we found no differences in pleasantness ratings between the objects paired with frequent neutral sound and the neutral oddball sound, validating our experimental manipulation. Second, consistent with Exp. 1 and prior literature, we replicated the effect of temporal order memory impairment for across-event vs. within-event object pairs. At the same time, we found no effects of oddballness on temporal order memory, illustrating the specificity of the pattern of results in Exp 1 to emotional stimuli. Third, as Exp.1 and prior literature, we replicated the object-colour memory enhancement for boundary vs. nonboundary objects, now driven by boundary vs. nonboundary oddball objects. Crucially, we observed no effect of oddballness on object-colour memory, again illustrating the importance of emotional stimuli in Exp. 1. The object-colour memory enhancement at event boundaries was not replicated for the frequent neutral objects (as in Exp. 1). Possibly, it was due to an additional attentional allocation to the oddball objects and a lower attentional allocation to neutral objects, no longer boosted by the changes in frame colours occurring at event boundaries (Bailey et al., 2017; Kosie & Baldwin, 2019; Swallow et al., 2009). Overall, we conclude that the effects of emotion that we observed in Exp. 1 were not driven by the fact that aversive sounds were less frequent (i.e. oddball) and thus more distracting (Exp. 2).

General implications

Our findings are partly in line with Event Segmentation Theory (Zacks & Swallow, 2007). Namely, we replicated an enhanced object-context binding and an impaired temporal order memory due to event boundaries for objects paired with neutral sounds. However, the original theory did not consider the role of arousal (due to novelty or emotion) and the results of our emotion manipulation were not predicted by this theory. Only recently, it was noticed that there are some commonalities in how event segmentation and emotional arousal affect human memory (Clewett & McClay, n.d.) such as enhanced item-context binding (Clewett et al., 2020; Heusser et al., 2018; Kensinger et al., 2006; Rimmele et al., 2012) or distorted perception and memory for temporal duration of events (Brunec et al., 2017; Droit-Volet & Gil, 2009; Ezzyat & Davachi, 2014; Lake et al., 2016; Sherman et al., 2023). It was also demonstrated that event boundaries elicit bursts of autonomic arousal (indexed by peaks in pupil dilation) (Clewett et al., 2020). By combining these two elements within the same paradigm, our study therefore contributes to disentangling their respective role by showing opposite effects of emotion and event segmentation on object-context binding and temporal order memory.

It is important to address the conceptual problem of whether emotion effects can be differentiated from novelty-driven event segmentation effects that may evoke surprise. Indeed, surprise is considered one of basic emotion categories (Ekman, 1992). On the other hand, several studies and reviews proposed a distinction between emotion and surprise (Barto et al., 2013; Xu et al., 2021). Moreover, we should emphasise that in our experimental design we did not distinguish between basic emotion categories. Instead, we operationalised emotion more broadly, in line with a theoretical framework of affective dimensions (Osgood & Succi, 1957) – in terms of arousal evoked by an aversive sound (Levita et al., 2009). Surprise might have been primarily elicited by changes in colour but the latter were not particularly salient or unexpected given the task structure, hence they may conceptually be akin to perceptual novelty rather than affective relevance (Sander et al., 2003).

In sum, in the present study, we showed that emotion and event segmentation affect our episodic memory in a different way. While object-colour binding was enhanced by event boundaries but impaired by emotion, temporal order memory was impaired by intervening event boundaries and enhanced by emotion. Thus, according to our results, a scary thunderstorm during a morning run would impair one’s memory of the contextual detail (e.g. the colour of the sailing boat that was passed), but overall enhance their memory of temporal order (e.g. whether the sailing boat or the cyclist was passed first). We did not find evidence for this emotional episode driving event segmentation in memory without any other context shifts. However, we observed that emotional event provided a strong binding context and reduced a cost of boundary in temporal order memory – one would remember temporal order of the cyclist, the shower, and the coffee equally well - thanks to a thunderstorm on their way back home.

Author contributions

Monika Riegel – conceptualization, methodology, investigation, data curation, resources, software, formal analysis, visualization, validation, writing – original draft, writing – review & editing, project administration, funding acquisition.

Daniel Granja – methodology, recruitment of participants, data collection, investigation.

Tarek Amer – conceptualization, methodology, validation, writing – review & editing.

Patrik Vuilleumier – conceptualization, methodology, editing.

Ulrike Rimmele – conceptualization, methodology, Investigation, writing – review & editing, project administration, funding acquisition.

Ethics statement

In the present study, all participants provided written informed consent and were financially compensated. The local Research Ethics approved the experimental protocol of the study.

Acknowledgements

We would like to thank: Anne Cristine Rodrigues Ferreira, Benoit Kevin Favre and Tiffany Alicia Amor for their help with data collection, David Clewett for his methodological advice on the experimental design, Luc Arnal and Guillaume Legendre for their methodological advice on the auditory stimuli, Ilaria Sani for her conceptual advice, Ben Meuleman for his statistical support.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Data availability statement

All data necessary to verify, interpret and extend published research will be freely available through a dedicated repository: https://osf.io/g2xer/?view_only=7417395df7ae421e88f30798e13afc8d.

Additional information

Funding

This work was supported by Narodowa Agencja Wymiany Akademickiej: [Grant Number Bekker grant PPN/BEK/2020/1/00459]; European Union’s Framework Programme for Research and Innovation Horizon Europe (2021–2027) under the Marie Skłodowska-Curie: [Grant Number Grant Agreement No. 101064781]; Schweizerischer Nationalfonds zur Förderung der Wissenschaftlichen Forschung: [Grant Number Excellenza PCEFP1_186911].