“It took me back 25 years in one bound”: self-generated flavor-based cues for self-defining memories in later life

KEYWORDS:

• 3D printed food
• self-defining memories
• emotions
• memory cues
• flavor
• multisensory use experiences
• sensory data
• perceptual data
• older adults

1. Introduction

“… those short, plump little cakes called petites madeleines […] I raised to my lips a spoonful of the tea in which I had soaked a morsel of the cake. No sooner had the warm liquid, and the crumbs with it, touched my palate, a shudder ran through my whole body […] an exquisite pleasure had invaded my senses […] and suddenly the memory returns. The taste was that of the little crumb of madeleine which on Sunday mornings at Combray […] when I went to say good day to her in her bedroom, my aunt Léonie used to give me, dipping it first in her own cup of real or of lime-flower tea […] when from a long-distant past nothing subsists, after the people are dead, after the things are broken and scattered, still, alone, more fragile, but with more vitality, more unsubstantial, more persistent, more faithful, the smell and taste of things remain poised a long time, like souls, ready to remind us […] the vast structure of recollection (Proust, 2006, pp. 61–63).

The above quote captures the evocative power of chemical senses for triggering memory recall with a feeling of traveling back in time, or the so-called Proust phenomenon. While the phenomenon has been explored mostly in relation to the sense of smell, Proust’s account involves also the sense of taste (Gibson, 2016) as shown in our introductory quote and this additional sentence: “the sight of the little madeleine had recalled nothing to my mind before I tasted it” (Proust, 2006, p. 63).

Scholarly attempts explored it, mostly beyond the HCI community, by contrasting odor cues with other modalities such as auditory, visual, or verbal ones with consisting findings indicating the superiority of odor cues for prompting vivid recall of emotional autobiographical memories (Arshamian et al., 2013; Chu & Downes, 2004; Chu et al., 2000; Greenberg et al., 2011; Hackländer et al., 2019; Mojet & Köster, 2016; Van Campen, 2014; Willander & Larsson, 2006) of adults or elderly people (De Bruijn & Bender, 2018; Chu & Downes, 2002; Herz, 2004; Herz & Engen, 1996; Larsson et al., 2014; Zucco et al., 2012). Such psychological research has focused mostly on unimodal cues, in particular aromas or odors experienced by being sniffed through the nose, i.e., orthonasal smells (Shepherd, 2013). Proust phenomenon however relates to the second sense of smell namely retronasal, stimulated by the food or drink molecules in the nasal passages, which in turn, allows for the perception of flavor. In other words, Proust phenomenon relies on the less explored multisensory cues involving taste, smell, as well as touch which Jellinek called olfactory-gustatory-textural-temperature stimuli (Jellinek, 2004), rather than mere orthonasal smell.

Over the last decade, there has been a growing HCI interest in food and its experiential qualities for commensality, playfulness, expressive, or emotional communication. As a resource for design, we argue that food is excellently positioned within the 3^rd wave HCI that emphasizes experience, embodiment, and meaning (Bødker, 2006). First, food experiences are rich and multisensory, so exploring them can inform the design of multisensory user experience more broadly (Comber et al., 2014; Obrist et al., 2016). For instance, Obrist and colleagues (Obrist, Comber et al., 2014) developed a framework for designing gustatory experiences based on characteristics such as temporality, affect, and embodiment. Second, another body of research has looked at design opportunities around food (Gayler et al., 2020b), which brought to light its broader social and cultural meanings (Altarriba Bertran et al., 2020; Dolejšová & Lišková, 2017; Gayler et al., 2021). Third, research on designing with food has emphasized the human body as a site of pleasurable interactive experiences especially for sweet (Gayler et al., 2021), umami and bitter tastes, and their embodied qualities (Obrist, Comber et al., 2014). Food experiences are not just hedonically but also perceptually rich, constructed from flavors that integrate tastes, retronasal olfaction, and somato-sensations such as texture and temperature (Breer, 2008; Gayler et al., 2021; Miranda, 2012; Prescott, 2012). Food, however, has been less explored in HCI research in relation to memory recall, and we also have limited understanding of how chemical senses can be leveraged in design.

In contrast to visual and aural modalities that have been extensively explored in interaction design (Gayler et al., 2022a), the taste (Crisinel & Spence, 2009; Mesz et al., 2017) and smell modalities (Braun et al., 2016; Brewster et al., 2006; Kaye, 2001; Maggioni et al., 2020; Obrist, Tuch et al., 2014) have been relatively less investigated, although work in this space has started to grow. This is due to the challenges of their chemical substrate, not trivial to deliver computationally, and designers’ limited understanding of these senses’ perceptual and experiential qualities that are key for inspiring novel design opportunities (Maggioni et al., 2020). Given people’s limited familiarity with chemical senses as interaction modalities, a promising starting point is the exploration of flavor, which also has the added benefit of multimodality. Moreover, the highly subjective qualities of flavor experiences (Obrist, Tuch et al., 2014), and their associated individual differences (Running & Hayes, 2016) can be best addressed by personalizing them (Maggioni et al., 2020) through tailored design approaches.

With the development of 3D food printing technologies, new design opportunities are opening up. Given the emotionality and vividness of autobiographical memories’ recall that can be cued by chemical stimuli, we argue that 3D printed flavors have the potential to benefit a specific type of episodic memories, namely self-defining ones. Self-defining memories are emotionally intense and vivid, related to enduring concerns in people’s lives (Blagov & Singer, 2004; El Haj & Gallouj, 2019; EL Haj et al., 2018), which aim to support the sense of self (Singer et al., 2007) as illustrated by the self-memory system (Conway & Pleydell-Pearce, 2000). In old age in particular, these memories are vivid, important and emotionally positive, albeit sadly, diminished in specificity even in the early stages of Alzheimer’s disease that has been also linked to impoverished sense of self (El Haj & Gallouj, 2019; EI Haj et al., 2018; Sas, 2018). HCI research on both episodic and self-defining memories in old age has highlighted the value of crafting personalized memory cues (Sas et al., 2015, 2017). Such self-generated cues (Wheeler & Gabbert, 2017), however, have been mostly investigated in traditional visual, aural or haptic modalities, and less so for flavor.

Our work explores the feasibility of 3D printed flavor-based cues for the recall of self-defining memories in old age. In a two-month project with 12 older adults, we employed a 3 stage mixed methodology through which we elicited 72 self-defining memories, codesigned bespoke flavor-based cues for each one, and explored in participants’ homes the impact of these 3D printed flavor-based cues on the recall of their self-defining memories. This paper focuses on the following research questions:

• What is the role of food in self-defining memories and how can it be leveraged for cuing them?

• How can we support older adults to co-design personalized 3D printed flavor-based cues?

• What is the value of such personalized 3D printed flavors for prompting older adults’ self-defining memories, and more broadly for multisensory food interaction and memory technologies?

The main contributions of this paper include (i) sense data and perceptual data as concepts reflecting flexible resource for design of memory technologies in the form of perceived sensory qualities of stimuli impacting each of our senses, and as subjective experience of perceived sensory integration, respectively, the latter illustrated through flavor-based memory cues whose experiential qualities we have also unpacked; (ii) tailored approach, engaging bodily senses, for designing 3D printed personalized flavors as memory cues that capture and integrate unimodal sensory fragments; (iii) implications for designing novel interactions leveraging flavor as resource for design integrating sense data, in particular for novel recreational and therapeutic multisensory reminiscing, and for body-centric multisensory design methods.

2. Related work

Our work draws from food interaction research in HCI and beyond, and two additional research areas focusing on memory technologies and sense of self in later life that have been limitedly explored in relation to food. Beyond HCI, a rich body of work on the link between memory and food has explored the Proust phenomenon (Leonor et al., 2018; Reid et al., 2015), with consistent findings indicating the richer and more vivid recall of emotional autobiographical memories (Chu et al., 2000; Hackländer et al., 2019; Larsson et al., 2014; Van Campen, 2014). For instance, naturalistic exploration of autobiographical memories indicated their more emotional recall when cued by odors rather than visual or verbal cues (Herz, 2004), with such cues being often generic and identical for all participants. Other findings suggested that generic odors prompted richer, and more emotional recall of negative events compared to auditory but not visual cues (Toffolo et al., 2012), or richer recall of autobiographical memories compared to visual cues or odors not matching those of the retrieved event (Chu & Downes, 2002). This superiority has been attributed to the role of affect in odor cuing, and to the encoding specificity principle (Chu et al., 2000) emphasizing the match of the encoding and recalling contexts, i.e., the closer the context in which the event occurred and the one in which its memory is recalled, the better the recall (Tulving & Thomson, 1973). Despite such rich work in memory research, three key limitations are the predominant focus on odors involving orthonasal rather than retronasal experience, unimodal rather than multimodal cues, and generic rather than personalized ones.

2.1. HCI research on multisensory food interaction

HCI scholars have shown a growing interest in Human-Food Interaction (HFI) over the last 10 years (Altarriba Bertran et al., 2018, 2019; Choi et al., 2014; Gayler et al., 2022a; Obrist, Tuch et al., 2014) from how technologies can support social dining experiences (Jiménez Villarreal & Ljungblad, 2011; Korsgaard et al., 2019; Y. Y. Chen et al., 2019), communicate data in edible 2D or 3D printed forms (Khot et al., 2017; Wang et al., 2016; Wei et al., 2011), explore food qualities for novel user experiences (Bruijnes et al., 2016; Dolejšová & Lišková, 2017; Gayler, 2017; Obrist, Tuch et al., 2014), or emphasize the sensory aspects of food experiences (Koizumi et al., 2011; Lin et al., 2018; Mesz et al., 2017). The HFI space has been described as design around food where the focus tends to be on the larger, social and cultural context (Altarriba Bertran et al., 2020; Dolejšová & Lišková, 2017; Gayler et al., 2021), and design with food, where the focus tends to be on the body, and pleasurable eating experiences (Altarriba Bertran et al., 2020; Arza et al., 2018; Wang et al., 2020) such as positive emotional communication and intimacy (Gayler et al., 2019, 2020b). Efforts to integrate these approaches have started to emerge, for instance, through the proposed framework for designing gustatory experiences based on characteristics such as temporality, affect, and embodiment (Obrist, Comber et al., 2014).

HFI research has also focused on the multisensory quality of food experiences emerging from integrated flavor perception of taste, smell, temperature, and texture, as well as sound and vision (Velasco et al., 2018). While the latter two senses are extrinsic to the perception of flavor, the former are intrinsic (Covaci et al., 2018) and strongly coupled (Auvray & Spence, 2008). Some HCI work in this space has looked how flavor can be intensified (Koizumi et al., 2011; Y. Lee et al., 2019; Vi et al., 2017) for richer dining experiences, how to support cross-modal sensory augmentation (Aisala et al., 2020) with smart cutlery (Lin et al., 2018), sonic seasoning (Q. J. Wang et al., 2019; Spence et al., 2019) or how sound can strengthen the perception of texture in crunchy foods (Koizumi et al., 2011).

To summarize, the HFI research has focused mostly on positive emotional multisensory experiences, and how they can be technologically augmented for pleasure, intimacy, communication and sharing. Despite its potential, there has been however limited HCI work exploring the value of food interaction for memories and sense of self.

2.2. Memory technologies in HCI and beyond: cue modality, generation and effectiveness

A parallel strand of work is the rich HCI research on memory technologies focusing on digitally augmented cues for supporting recall of episodic memories. While emerging work has shown the challenges of lifelogging technologies based-cues for other forms of memory impairments such as those associated with depression (Qu et al., 2019), the main focus of technologically mediated cues has been on memory impairments due to aging. Predominantly in visual and sound modalities, cues are often photos (Dib et al., 2010), recorded sounds (Frohlich & Murphy, 2000; Isaacs et al., 2013), or videos (Le et al., 2016) capturing the situated context in which the memory event has occurred. Cues have also been captured in text format such as brief self-reports tagging emotions or thoughts by self-tracking applications (Isaacs et al., 2013; Qu et al., 2020), and as visual-biodata showing the value of arousal for recognition and recall (Sas et al., 2013; Umair et al., 2018, 2020, 2019).

While HCI literature has looked mostly at visual and auditive cues (Le et al., 2016; Sas & Coman, 2016; Sas, Davies et al., 2020; Sas & Whittaker, 2013), psychology research has explored a broader range of cues’ modalities including chemosensory ones, such as gustatory and olfactive cues (De Bruijn & Bender, 2018; Chu & Downes, 2002; Herz, 1998, 2004). In this respect, a wealth of lab-based findings have shown that olfactive cues evoke more emotional (Herz, 1998, 2004; Herz & Engen, 1996) and vivid memories (De Bruijn & Bender, 2018; Chu & Downes, 2002; Herz, 2004), and stronger recollective experience of travel back in time (Larsson et al., 2014) compared to verbal or visual cues. In both HCI and psychology research, memory cue modalities have been explored mostly in isolation, with a few exceptions pointing to the value of multimodality, integrating for instance, visual and auditory cues (Le et al., 2016), or text, photo, and music cues (Peesapati et al., 2010).

Cues can also be distinguished by how they are captured or generated. While in psychology research, most studies relied on cues prepared by researchers, HCI work tends to distinguish cues by how they are captured: automatically (Eldridge et al., 1993) or manually (Carter, 2005). HCI work comparing different forms of capture across different modalities is limited. A landmark example explores manually and automatically captured photos using SenseCam, where better recall was cued by manually captured photos, which authors attributed to their saliency (Sellen et al., 2007). Manual capture of cues does require more user involvement compared to passive capture, but the additional effort of making the cue remains limited. Work has also emerged looking at how cues can be actively created or crafted through users’ effortful input that goes beyond the mere recording of data. Such self-generated cues have been shown in psychological studies to be particularly effective (Hunt & Smith, 1996) but there has been limited HCI exploration of them. Exceptions include self-generated cues in doodle modality creatively communicating emotional meaning (Sas et al., 2015), or users’ crafted video summaries from photos that led to increased ability to recall memories (Le et al., 2016). Psychological memory research has also emphasized the generation effect (Slamecka & Graf, 1978) of cues being created by participants, with the increased mental effort required for cue generation leading to cues’ stronger connection with the initial event (Bertsch et al., 2007), and personal relevance (Slamecka & Graf, 1978).

Regarding cue effectiveness, HCI work has identified the importance of being recognizable, in terms of belonging to the original experience, personal relevancy, and distinctiveness so that only one memory is prompted by a given cue (Lee & Dey, 2007; Mazzoni et al., 2014). In addition, consistent findings on episodic retrieval have shown reliance on salient features from the content of the episodic memory that are shared with its cue (Schlagman et al., 2009). Such features reflect the sensory perceptual content of the memory event, such as the smell of the sea or sound of the waves (Ball & Little, 2006). The effectiveness of these features has been explained by the principle of encoding specificity (Conway, 2005; Tulving & Thomson, 1973). This is commonly reflected in the complexity of the cue, derived from its content as amount of distinct information such as colors, patterns, or textures for visual cues; and modality, i.e., one or more sensory modalities. The latter is particularly important as multimodal sensory cues and vivid episodic retrieval appear to be underpinned by the same neural substrates, i.e., angular gyrus (Tibon et al., 2019).

To conclude, most of HCI research on memory cues has focused on traditional visual and sound modalities of automatically or manually captured cues, but less so on users’ self-generated chemosensory cues and how these can be codesigned and leveraged in memory technologies. Chemosensory cues, and in particular olfactory ones are more specific, evoking more emotional (Herz, 2004; Herz & Engen, 1996) and vivid (De Bruijn & Bender, 2018; Chu & Downes, 2002; Herz, 2004) episodic memories, and their recollective retrieval (Larsson et al., 2014). In contrast, flavor based cues have been much less explored, especially outside the lab, although their multimodality can provide even stronger encoding specificity and hence ability to cue recall, especially if they are self-generated and personalized through codesign.

2.3. Research on aging and sense of self in HCI and beyond

HCI research on aging has focused on key aspects such as memory, personhood, and particularly the sense of self. Since episodic memories or memories with sensory content of personally experienced events situated in specific time and space are the most impaired type of autobiographical knowledge in both healthy aging and in dementia (Hamel et al., 2016; Lee & Dey, 2007; Lindenberger & Mayr, 2014; Piolino et al., 2006), it is not surprising that most HCI work supporting memory decline in aging has targeted them. Older people’s increased reliance on external rather than internal information (Lindenberger & Mayr, 2014), coupled with the transient, sensory content of episodic memories, makes these suitable candidates to be captured through personal and ubiquitous technologies that record the here and now situated content of personal events, usually in the form of visual or auditory cues. For instance, research on wearable cameras such as SenseCam has shown their value for supporting episodic memory recall (Harper et al., 2007), while additional findings indicated also that best cues are memorable, distinctive and self-relevant (Lee & Dey, 2007).

While chemosensory memory cues (gustatory and olfactory) have been less explored in HCI, aging research has shown their benefits for older adults helping them recall more autobiographical memories compared to younger people (Zucco et al., 2012). Flavor memory has been also shown to evoke autobiographical memories (Mojet & Köster, 2016), and that for people living with Alzheimer’s disease, odor-based cues led to increased number of emotional and specific autobiographical memories, from both childhood and adulthood, when compared to non-odor cues (Glachet et al., 2019).

Scholars also explored older adults’ engagement in codesigning personalized cues to support reminiscing (Wheeler & Gabbert, 2017). For instance, in HCI research, visual or audio content has been used to create digital or hybrid scrapbooks such as Memento (West et al., 2007) or multimedia biographies (Frohlich & Murphy, 2000), integrated with physical possessions such as Memory Box (Frohlich & Murphy, 2000) to support reminiscing in old age. Crafting has been also beneficial due to increased need for sensory stimulation in old age, and particularly in dementia as shown by multi-sensory interventions (Livingston et al., 2014; Reisberg et al., 2002; Sas, Davies et al., 2020). Similar research highlighted older adults’ preference for physical cues (Thiry & Rosson, 2012) that leverage haptic experiences (Huber et al., 2019) and active engagement in craft-based activities (Sas et al., 2015, 2017) or codesigning (Wallace et al., 2012, 2013). This is not surprising, as episodic memories are intrinsically related to the sense of self: upon integration in autobiographical memory system they become stable, durable, and available for recollective experience: “the sense or experience of the self in the past” (Conway et al., 2001b, p. 1375) as illustrated in the self-memory system (Conway & Pleydell-Pearce, 2000). This system emphasizes another type of autobiographical memories, namely self-defining memories that are emotionally intense and vivid because they recruit those episodic memories linked to enduring concerns with the aim to support individual’s self-coherence, i.e., “retrieval from the long-term self of episodic and conceptual knowledge structures that help to give meaning to experience” (Conway et al., 2004, p. 511). Because of their link to the sense of self, sadly, the negative impact of aging on the recall of episodic memories also extends to the diminished sense of self (EL Haj et al., 2018; Sas, 2018). Therefore, efforts to support self-defining memories can be beneficial to the sense of self, by tapping into the sensorial richness of the episodic memories underpinning them.

In contrast to episodic memory cues, the exploration of cues for self-defining memories has been limited. A few exceptions include findings that music-based cues lead to better recall of self-defining memories of people living with Alzheimer’s disease, when listening to their own chosen music rather than music provided by researchers (Haj et al., 2015). Beside interest in episodic memories, HCI work on self-defining memories has also started to emerge showing the benefit of craft-based projects to support older people to elicit such memories around key events (Sas & Whittaker, 2013; Sas et al., 2016) or to employ craft, in order to design no longer accessible cues (Sas et al., 2017). Another study has shown the positive affect of self-defining memories, their link with identities, predominantly achievement self and relational self, and how these may be evocatively cued by crafted objects (Sas, 2018).

To conclude, aging is associated with impoverished retrieval of contextual details so that episodic memories become increasingly generic or semanticized, negatively impacting on the recollective experience of the sense of self in the past, that is essential for both episodic and self-defining memories (Piolino et al., 2006). This emphasizes the value of supporting the sensorial and emotional phenomenological aspects of recollective experience (Piolino et al., 2006), for which a promising avenue is self-generated multimodal chemosensory cues.

3. Method

The aim of the study is to explore the feasibility and value of codesigning self-generated 3D printed flavor-based cues for self-defining memories. We employed a mixed study design (Figure 1) with qualitative data being supported by quantitative data (Vasileiou et al., 2018). As an exploratory study, our work follows an idiographic, interpretative approach focusing on in depth analysis of data from a small sample of participants (Castro et al., 2010; Whitley, 2007).

Figure 1. Method diagram showing; Stage 1 visual probe kit, Stage 2 free recall, flavor cue design and flavor cue pilot, Stage 3 flavor cue and word cue recall and interview.

Findings from social sciences have shown that small sample sizes are sufficient for qualitative analysis (Ando et al., 2014; Fugard & Potts, 2015; Guest et al., 2006; Schweitzer et al., 2015), especially when the research questions have clear focus and scope, samples are homogenous, methods for data collection are diverse, and data has richness and experiential qualities (Braun & Clarke, 2021). We aimed to address these criteria by gathering not just one set of interviews, but diverse data throughout the three stages of our study (2-week engagement with sensory probes followed by 45 min interviews for memories elicitation, 75 min codesign workshops followed by 15 min interviews, 75 min cued recall followed by 30 min interviews). Such triangulation ensured both “thick data” in terms of amount (over 49,000 words) and “rich data” in terms of level of details and nuances (Creswell & Clark, 2006; Fusch & Ness, 2015). Memory research on autobiographical memories also tends to employ small sample sizes (Schlagman et al., 2006). Regarding sample size, there are also expectation related to “local context of the discipline” (Braun & Clarke, 2021), which given the additional demands of technology design, tends to be rather low in HCI studies. For instance, a review of papers published in 1 year at the ACM CHI conference has shown lower sample sizes for qualitative compared to quantitative studies, and that the most common sample size is 12 (Caine, 2016). Moreover, in two landmark HCI works on memory technologies, authors reported interviews with 12 older participants (Crete-Nishihata et al., 2012), or a single participant involved in a multistage design-led inquiry (Wallace et al., 2013).

Through adverts on social media, we recruited 12 participants (Mean age 65.83, range 62–78), (8 females, 3 males, 1 participant preferred to not identify their gender). They all reported having no taste, smell, or memory impairments.

All participants, were middle class, university educated, living independently in their homes in the UK, most with their partners. We selected this group for three reasons. First, the cognitive decline due to normal aging negatively impacts episodic memories and sense of self (Singer et al., 2007). Second, chemosensory cues such as tastes or odors are particularly beneficial in supporting self-defining memories in old age (Zucco et al., 2012), and third, older people’s benefit from rich multi-sensory stimulation and engagement in crafts (Sas, 2018).

Our three-stage methodology (Figure 1) followed a participatory approach that goes beyond mere extraction of knowledge (Van Mechelen et al., 2017) by co-constructing knowledge about the role of food in people’s self-defining memories, and how 3D printed flavor-based cues may prompt their recall. Informed of its purpose, at each stage of the study, participants were working toward the purposeful design of such flavor-based cues. Our approach focused on participants’ lived experiences (Stage 1) around their personally relevant self-defining memories, and fostered provocation (Rizzo, 2011) in order to creatively co-generate the best flavor-based cues (Stage 2) rather than being imposed by us (Wright & McCarthy, 2015). Similar co-design approaches have been employed both in HCI to generate so-called “material food probes” consisting of personalized 3D printed flavors to support emotional communication in intimate relationships (Gayler et al., 2020b), and in design studies to generate for instance, “Smell Memory Kit” integrating smells from participants’ autobiographical memories (Leret & Visch, 2017) whose evaluation indicated the value of smells as design material supporting playful memory recall.

We also note that in the context of participatory research methods, as opposed to traditional positivist approach, the Hawthorne effect needs a more nuanced interpretation (Graham et al., 2007; Smith & Coombs, 2003). Indeed, cultural probes used in Stage 1 required active engagement whose outcomes inspired and provoked participants to become deliberately engaged in the making of novel personalized cues, and thus, cues’ value for recall is interlinked with their making. This strong engagement with users in the two first stages of ours study has most likely been a contributor to the cues’ values for memory recall. All 12 participants took part in Stages 1 and 2, with 10 completing Stage 3 (due to attrition) and the three stages outline a 10 step process as shown in Table 1.

Table 1. The 10 step process for the codesign and evaluation of flavor-based memory cues within the 3 stages of sensitizing toward food experiences and memories, co-design and evaluation of flavor-based cues.

Stage 1: Sensitizing toward food experiences and memories
Introduction session in participants’ homes	Step 1	Each participant uses the probe kit over 2 weeks
	Step 2	Probes were collected and reviewed by researchers to inform Stage 2
Stage 2: Codesigning multimodal flavor-based cues
Participants’ homes	Step 3	3 days before co-design meeting, participants are instructed to identify self-defining memories for codesigning cues for
Codesign workshops	Step 4	Memory elicitation: recall of self-defining memories following the process of (Piolino et al., 2006)
	Step 5	Participants eat the taste calibration foods
	Step 6	Codesign of memory cues for selected memories: association of non food memories (NFM) with foods and flavors, resulting in participants selecting appropriate flavors as cues
	Step 7	Codesign of memory cues for selected memories: description of flavor qualities for cuing food memories (FM) and non food memories (NFM) with details of their most salient features.
Research lab	Step 8	Preparation of flavor-based cues: sourcing, mixing, cooking and piloting/iteration of flavor cues
Stage 3: Evaluating the impact of flavor-based cues on recall
Evaluation session	Step 9	Cued recall of the 6 self-defining memories of each participant, 4 cued with personalized 3D flavor-based cues and word cues, and 2 cued with word cues only
	Step 10	Participants are interviewed about their experience

3.1. Stage 1: sensitizing toward food experiences and memories

Stage 1 consisted of a two-week diary study during which participants used a package of cultural probes in their homes to sensitize them toward food experiences and self-defining memories.

Step 1: Building on previous use of probes in HFI (Gayler et al., 2020b, 2021), our probes focused on food-based experiences through sensory deprivation and augmentation such as eating with blindfold, nose clip, earplugs and gloves, body mapping of food experiences over time, and flavor descriptions through gameplay. In this paper, we extend our work on sensory probes (Gayler et al., 2021) by using their outcomes to further inform the codesign of flavors, as shown in Stage 2. Each activity prompted by these probes was intended to explore multisensory, embodied, emotional and cultural aspects of food experiences and to familiarize participants with connections between memories and food. The outcomes of these activities were captured through written letters describing recipes for food memories, booklets for drawing and writing about bodily experiences, and photos documenting gameplay, which were used as input into the next stage. The probes were handed in the introduction session which took place in participants’ homes.

Step 2: Researchers collected the probes and reviewed them to identify potential relationships between flavors and memories, as well as to provide context for participants’ perception of multisensory food, and flavor experiences more widely. Participants’ increased awareness of their multisensory flavor experiences and of relationship between food and memory that Stage 1 aimed to support, was important for engaging them in Stage 2.

3.2. Stage 2: codesigning multimodal flavor-based cues

In Stage 2 we introduced participants to the concept of self-defining memories, gave time to identify them, then facilitated the elicitation of these memories, followed by co-design workshops for designing flavor-based cues to prompt their recall. The codesign of flavor-based cues is likely to depend on how food features in these memories, so an important distinction is that between self-defining memories including food that we call food memories (FM) such as a wedding breakfast, and those without, such as starting at university, which we call non-food memories (NFM). Stage 2 includes six steps, i.e., from 3 to 8.

Step 3: Three days prior to the co-design workshops, participants were prompted to identify the 6 required memories. Informed by Singer and Salovey’s (2010) procedure we introduced participants the definition of self-defining memories as those that can be remembered clearly, are important, lead to strong feelings, and help people understand who they are and how they come to be the person they currently are. Participants were also given examples of self-defining memories involving food such as: “meals at a wedding, christening, anniversaries, romantic dinner dates, food eaten on holiday, or made by a specific person in a specific place.”

Step 4: At the beginning of the co-design workshops, we asked participants to recall as many details as possible for each of the six identified self-defining memories, following Piolino et al. (2006) process consisting of prompts for location, time, people, feelings, and sensory experiences. These sessions lasted ~45 minutes and were audio recorded. While a preliminary analysis of 4 interviews completed during the Stage 2 was reported in Gayler et al. (2020a), this paper significantly extends this prior work to include the preparation of the 3D printed flavors (Step 8) and their evaluation (Stage 3) with input from the final sample of 12 participants.

3.2.1. Co-design of flavor cues for selected memories

Step 5: We started the individual codesign workshops with a calibration process. Similar to prior HFI research (Gayler et al., 2019), we provided samples of 3D printed food of the five basic tastes (sweet, bitter, salty, sour, and umami) to calibrate participants’ perception.

Step 6: The next step was to identify associations between NFM and foods or flavors. Drawing on the structure of episodic memories (Lee & Dey, 2007; Sas et al., 2013; Sellen et al., 2007), participants identified places, events, objects, people and feelings associated with each memory. For each detail, they were asked if any food or flavors could cue them, e.g., food eaten in specific places or at specific events, food preferred by people associated with that memory, food matching the feeling of the memory, or food they can creatively associate. Once each detail had been explored for associations, the researcher prompted the participant to review and chose one, or a combination of foods/flavors as the most suitable to act as a flavor-based memory cue. For this, they were instructed to consider which ones would have the most recognizable and strong relationship with a specific memory.

Step 7: This step was to provide a full description of the flavor cues, both for NFM and FM. The flavor cues for FM were defined by the food that was present in the memory recalled. In this process the researcher asked participants to identify ingredients and cooking processes pertaining to the identified food. Participants also rated the recalled flavor cues on 6-point Likert scales for the intensity of each of the five basic tastes, and two semantic differential 6-point scales for texture (from liquid to solid), and for flavor duration (from momentary to lingering), based on sensory profiling techniques from sensory science (Ozcelik & Karaali, 2002). Example of facilitative questions include those about recipes (“was lemon added?”) and cooking (“cooked with skin on?”) which were asked until participants were unable to provide further details. This was followed by asking participants to provide sensory description (Williams & Arnold, 1985) of selected food in terms of key flavors, smells and colors, with the aim to bring the sensory food experience associated with the FM or NFM into focus, in order to better inform the design of their flavor-based cues. Participants were also asked to describe sensory aspects of the environment in which the cue was experienced (for FM) or where the memory was encoded (NFM); this included sounds, smells, sights or haptic experiences that previous work has shown as impacting flavor perception (Auvray & Spence, 2008). Finally, to support the creation of the flavor-based cues, participants were asked to identify their most salient aspects. These qualities were used to provide the brief against which the flavor cues were created. This stage concluded with individual semi-structured interviews to explore participants’ perception of their codesign of flavor-based cues. The codesign workshops lasted around 75 minutes, with 15 minutes for the interviews.

3.2.2. Preparation of flavor cues

Step 8: Between Stage 2 and Stage 3, the first author used the insights from Stage 2 to create in the lab, the flavor-based cues for each self-defining memory. For example, P2 described “barbequed mackerel” as part of the memory of Golden wedding anniversary. For this, mackerel was sourced after considering different available sources such as smoked vacuum-packed mackerel and unsmoked tinned mackerel. To better deliver the barbequed flavor, the smoked mackerel was chosen, which was then placed under a grill, to create charring on the skin side. As the charring was particularly salient in the flavor description, all the charred skin was kept with about half of the remaining flesh, in order to prioritize the charring character in the flavor cue. The solids were then mixed with water, blitzed in a food processor and strained to produce a liquid with the barbequed mackerel flavor.

This is an illustrative examples as how flavor cues were made, with some of the key ingredients being sourced to match those described by participants, prioritizing participant’s preference, flavor saliency, and memorability. Where it was difficult to source the exact ingredient (e.g., lobster thermidor), a proxy was found that matched key aspects of the flavor and experience, i.e., lobster bisque was used, as it contained both lobster and creamy rich flavors in line with the described thermidor. Once ingredients were sourced, they were cooked or prepared according to the details provided by participants. Once prepared, any solid ingredients were put into a food processor to purée, adding just enough water as was necessary to achieve a smooth mixture but avoid dilution of flavor. This was then filtered through a cheesecloth to remove any remaining large particles. The puréed and filtered foods were then mixed with the gelling agents used in the 3D food printing process, to achieve a batter-like consistency. These were refrigerated, until they ready to print, into the 10 ml jar like samples in line with prior work using the nūfood printer (Gayler et al., 2019, 2020a, 2020b).

The prepared 3D printed flavor-based cues were piloted with 6 participants (P1-6), who ate each sample, and commented on the match between the flavor cue and its related foodstuff. From the 36 piloted flavors, 13 required adjustments, most of them minor, to increase the flavor intensity or better balance the ingredients. Of the 12 participants, 10 took part in Stage 3 (P1-4, P7-12).

3.2.3. 3D printed food and the nūfood printer

The 3D printed flavors consisted of 20 mm cube-like structures made of ~125 gel balls (each 2 mm diameter) with liquid centers, joined together. When eaten, these gel structures burst, releasing the liquid inside, which in contrast to solid printed foods, more readily stimulate via volatile compounds the retronasal olfaction – key for flavor experience (Breer, 2008; Miranda, 2012; Prescott, 2012). Gel like texture also better carries flavors’ intensity (Moskowitz & Arabie, 1970) ensuring also easier swallowing (Steele et al., 2015).

3.3. Stage 3: evaluating the impact of 3D printed flavor-based cues on recall

Stage 3 consisted of an experimental study to explore the impact of 3D printed flavor-based cues on the recall of self-defining memories in participants’ homes. We employed a within-subject design where each participant was given 4 of their codesigned, bespoke 3D printed flavor-based cues, alongside the name of the memory for 4 of their self-defining memories, 2 FM and 2 NFM (described hereafter as flavor cued), while the recall of the remaining 2 self-defining memories, 1 FM and 1 NFM, was cued only with the name of the memory. This meant that flavor-based cues for the latter 2 memories were unused, but codesigning them was important, in order to account for the impact of the design process on memory recall. The study involved two independent variables namely the type of self-defining memory: FM and NFM, and the cue type: free recall, flavor cued (cued with flavor and memory name) and word cued (cued only with memory name), and four dependent variables: emotional content and sensory details in the recall (both across free recall, word and flavor cued), participants’ ratings of the experience of time travel, and of emotional intensity (across the word and flavor cued conditions). To account for the order effect, we also randomized the order of cues for FM and NFM memory recall.

Step 9: All flavor cues were presented to participants on identical stainless steel teaspoons. When prompted, participant ate the cue for each memory and began the recall, whose procedure followed the one from Stage 2.

Step 10: The study concluded with individual semi-structured interviews where we asked participants to reflect on the experience of codesigning the flavor-based cues, the perceived impact of flavor-based cues on the recall of their self-defining memories, and potential future uses of such cues. The experimental study lasted around 75 minutes for each participant, with about 30 minutes being used for the interview.

3.4. Data analysis

The approach to analysis involved triangulating qualitative and quantitative methods to ensure complementary findings, convergence of agreement between methods, and integration of different measures of recall (Jick, 1979). Such triangulation of interviews and scales is not uncommon in HCI, with much cited papers on emotional experiences and memories using it (Isaacs et al., 2013; McDuff et al., 2012). In particular, the quantitative methods provided descriptive statistics to identify themes, and allowed us to test hypotheses on key memory recall concepts and self-report measurements through validated scales. Indeed, memory research area has differentiated between study participants and autobiographical memories as different units of analysis, with the former being used for research questions on different subgroups generating the memories, and the latter for research questions on memories qualities (Zimprich & Wolf, 2018). Thus, we aimed for sufficiently large sample of self-defining memories, so that we could run such statistical tests, in particular three MANOVA and two Chi Square tests, all ensuring power of 0.80, alpha of 0.05 and a medium effect size, where power analysis was computed using G Power 3.1.9.7 version (Faul et al., 2009). In particular, the three MANOVA tests were used to further explore the impact of different memory types on sensory modalities of the free recall content, the impact of memory type and time of recall such as before or after the codesign workshops on sensory modalities of the recalled material, and on its emotional content, respectively.

We also computed two chi-square to explore the relationship between high/low time travel and word/flavor cues, and between high/low time travel and intense positive/negative emotional content reflected in the recalled memories.

To assess the emotional content, we ran Linguistic Inquiry and Word Count (LIWC), a linguistic analysis calculating the frequency of words for positive and negative emotions (Pennebaker et al., 2015) on each memory recall. Sensory details were generated from linguistic analysis informed by the Lancaster Sensorimotor Norms (Lynott et al., 2020) derived from over 39 K English words (Lynott et al., 2020) computing the dominance of six perceptual modalities (touch, hearing, smell, taste, vision, and interoception) for each word.

After recall, participants immediately rated the travel in time on a 5-point Likert scale (0 = not at all, 4 = extremely) for the statement, “I feel that I have travelled back to the time it happened.” They also rated the emotional experience on a 5-point Likert scale from −2, very negative to +2, very positive, both scales being based on work on multimodal autobiographical memory (Herz, 2004; Willander et al., 2015).

The interviews including memory recall from Stage 2 and 3, and the codesign workshops were also audio recorded and fully transcribed. For qualitative data analysis we employed a hybrid coding (Fereday & Muir-Cochrane, 2006) integrating theoretically-informed deductive codes such as themes of self-defining memories (Blagov & Singer, 2004), life periods (Crete-Nishihata et al., 2012), emotional and sensory content (Blagov & Singer, 2004), as well as self-identity levels (Sas, 2018). The inductive codes included the role of food in self-defining memories, components of flavor-based cues, qualities of flavor experience, and of flavor-based cues, with the code list being iteratively refined between the 1st and 2nd author over several months.

4. Findings

We now report our findings; starting with the role of food in self-defining memories, flavor-based cues and their design process, followed by the analysis of the value of 3D printed flavor-based cues for the recall of self-defining memories, and more broadly for memory technologies in old age.

4.1. The role of food in self-defining memories

In this section, we explore the role of food for self-defining memories as they were elicited in the free recall in Stage 2. In particular, we contrasted FM and NFM on how they relate to the sense of self, emotions, sensory perception, and lifetime periods when the memories were encoded.

4.1.1. Food-based self-defining memories more strongly reflect relational self

An important outcome is the large presence of relationship theme among the 72 self-defining memories, both overall (60%), and particularly for FM (75%) compared to NFM (40%). We identified this theme using Singer’s taxonomy (Singer et al., 2007) differentiating achievement, relational, and negative themes (Table 2, col 1) with 8 memories belonging to more than one group. Singer’s findings show that achievement memories were most prevalent in old age, which is reflected also in our NFM. FM, however, emphasizes relationship memories including group celebrations such as anniversaries (P2), weddings (P2, P3, P6, and P12), and significant small group or dyadic experiences such as honeymoon meals (P1, P12), overseas trips with loved ones (P1, P4, P5, P9, P11, and P12), or the birth of a child (P1, P3, P5, P7, and P9). These outcomes are important since self-defining memories with achievement themes are key for the personal self, while those with relationship theme support the relational self (Sas, 2018) which has been shown as the most important aspect of self in old age (Singer et al., 2007), and particularly in dementia (Addis & Tippett, 2004). To better illustrate our rich qualitative findings, we provide brief descriptions of 5 self-defining memories from the prevalent relational theme, 2 for FM and 3 for NFM (Figure 2).

Table 2. Percentages of self-defining memories grouped according to relationship, achievement and negative themes for both food memories (FM) and non food memories (NFM) (column 1); percentages for both FM and NFM from childhood, youth, adulthood, or old age as life periods (column 2–5), and percentages of emotional terms measured using Linguistic Inquiry and Word Count (LIWC) analysis for both FM and NFM when freely recalled (column 6–7).

	Life periods				Emotional terms measured through LIWC analysis
Self-defining memories themes	Childhood	Youth	Adulthood	Old Age	Positive	Negative
Relationship (60)
FM (75)	22	15	49	14	2.77	0.4
NFM (45)	16	5	57	22	2.34	0.76
Achievement (26)
FM (18)	14	14	57	14	2.84	0.22
NFM (34)	0	8	62	31	2.68	0.95
Negative (14)
FM (7)	67	33	0	0	1.80	0.14
NFM (21)	63	0	25	13	2.20	0.45

Figure 2. Five self-defining memories; The best strawberries ever, a food memory from adulthood (P9); Golden wedding barbeque (BBQ) mackerel, a food memory from old age (P2); A beautiful wedding, a non-food memory from adulthood (P3); Saying goodbye to Mum, a non-food memory from adulthood (P7) and An exhilarating dip in the sea, a non-food memory from old age (P12).

4.1.2. Food-based self-defining memories are mostly from adulthood

Findings indicate that the self-defining memories reported by our participants relate to different time periods in their lives when they were first encoded namely childhood, youth, adulthood and old age. Self-defining memories from childhood tend to involve good family times such as holidays, those from youth usually relate to first time experiences focused on identity, while those from adulthood usually include the birth of one’s child or interview for career promotion. In addition, self-defining memories from old age include experiences related to grandparenting or retirement. These four periods also reflect the themes from Erikson’s life cycle stages of psychosocial development focused on themes of trust, autonomy, and identity in childhood and adolescence, intimacy in youth, generativity in adulthood, and integrity or acceptance in old age (Erikson, 1994), with the latter stage usually starting after 60 years of age (Conway & Holmes, 2004).

Our findings confirm previous ones indicated that self-defining memories are likely to be encoded along the entire life span (Conway & Holmes, 2004). However, while previous work provides limited insights into the distribution of self-defining memories along the lifespan (Table 2, col 2–5), ours indicate that for older adults, they are predominantly encoded during adulthood (over 45%). This is an important outcome in the light of the consistent findings in autobiographical memory research (albeit not self-defining memories) on the reminiscing bump: the preference of older adults to recall such memories from their early life or youth (Rathbone et al., 2008; Singer et al., 2007). Even in the space of autobiographical memories research, there has been less work exploring their distribution on the life span with the main findings showing that the larger percentage of autobiographical memories are encoded during youth, about 35% (Conway & Holmes, 2004).

Our findings indicate that self-defining memories reported by our participants were prevalently encoded during adulthood (45%), with achievement and relational themes being particularly strong. This emphasis could reflect in part the increased agency of adults to eat novel foods and food they like. Later reminiscing bump was also found in older adults who collectively experienced later in life significant social events like those leading to national independence (Conway & Haque, 1999). Our findings indicate a similar delayed reminiscence bump, and therefore the value of food to prompt particularly self-defining memories from adulthood, which can be valuable to strengthen the sense of self in older age.

Related to life periods, previous work has also classified self-defining memories as generic (repeated general events over a life period), episodic (general events of lengthy duration), or specific (unique events, less than a day) which tend to occur with the frequency of 4%, 24%, and 72%, respectively (Blagov & Singer, 2004). Our findings confirm the prevalence of specific self-defining memories, albeit more so for NFM (36 out of 36) compared to FM (28 out of 36). The remaining 8 FM were generic, often from childhood such as “Grandma’s Yorkshire puddings for Sunday lunch” (P6), or from holiday locations where a particular food was repeatedly eaten “moussaka at Dimitris’ restaurant” (P10). Such prevalence of over 22% of generic FM is important as unlike specific ones, they contain more abstract, less specific sensorial content (Meléndez et al., 2018), and are more prevalent in old age (Levine et al., 2002).

4.1.3. Food-based self-defining memories: Sensorially richer, more positive emotions

Findings indicate a high prevalence of positive emotions in participants’ descriptions of self-defining memories, most commonly happy and delight, while negative ones were usually disappointment or poignant. To explore such emotional content of self-defining memories, we employed Linguistic Inquiry and Word Count (LIWC) analysis (Pennebaker et al., 2015). While inspired by Pennebaker’s research on expressive writing of emotionally negative experiences (Pennebaker & Chung, 2011), the software has been further developed to capture broader linguistic features and psychological process such as positive and negative emotions, cognitive or social processes. Its psychometric properties and cross cultural coverage of natural language led to its extensive use for text analysis in social sciences (Tausczik & Pennebaker, 2010, Pennebaker et al., 2001), including autobiographical memory research (Zator & Katz, 2017) as well as HCI work from designing mobile apps for reflection on autobiographical memories (Isaacs et al., 2013) to voice interfaces and spoken dialogue systems (Rothwell et al., 2021).

Based on LIWC analysis, findings indicate that FM were described by 2.77% positive emotions and 0.4% negative ones, while NFM by 2.65% positive emotions and 0.96% negative ones (Table 2 col 11–14). Compared to 2.7% for positive, and 2.6% for negative affect from Pennebaker’s personal writing corpus (Pennebaker & Chung, 2011), our findings show similar frequencies for positive emotions, albeit higher for FM, and lower frequency of negative emotions, particularly for FM (over 6 times fewer). Since Pennebaker’s personal writing corpus (Pennebaker & Chung, 2011) focuses on upsetting experiences, we have also compared our findings with those on a corpus for casual conversations (Pennebaker at el., 2011) which showed similar frequencies for positive emotions of 2.7%, and as expected, lower frequency of negative emotions of 1.3%. Interestingly, when compared to our findings, there are still fewer such negative emotions used to describe self-defining memories, especially for FM (over 3 times fewer).

Participants’ free recalls of their self-defining memories were also rich in sensory details, which has been suggested in previous work (Conway et al., 2004), albeit limited empirical exploration supported it. To address this, we performed a linguistic analysis informed by the Lancaster Sensorimotor Norms (Table 3) which previously indicated that vision is the most common dominant modality in English language (74%), followed by auditory (11%), interoceptive (9%), haptic (2%), gustatory (2%), and olfactory modality (1%; Lynott et al., 2020). Chi-square tests revealed that compared to dominant modality in Lynott et al.’s corpus (2022), our self-defining memories have significantly more audio, visual, and interoceptive terms (describing stats such as feeling full) when they do not relate to food, i.e. (NFM (X²(1,2) = 27.1, p < .001), and significantly more audio, visual, interoceptive, and gustatory terms (describing eating and food terms such as eating, lunch, meat) for memories involving food, i.e. (FM (X²(1,3) = 25.17, p < .001). These findings are interesting, indicating that self-defining memories are freely recalled with richer contextual audio-visual details, and in terms of interoceptive or bodily internal states. Important however is the finding showing that when comparing the free recall of food and non-food self-defining memories, there are significantly more visual, gustatory, and olfactory terms describing FM rather than NFM (X²(1,2) = 18.14, p < .001). Figure 3 shows these percentages for each modality, for both FM and NFM.

Table 3. Percentages of self-defining memories grouped according to relationship, achievement and negative themes for both food memories (FM) and non food memories (NFM) (column 1); and percentages for both FM and NFM of the dominant perceptual classifications of terms by Lancaster Sensorimotor Norms (Lynott et al., 2020): visual, interoceptive, auditory, gustatory, haptic and olfactory, used in food and non-food related self-defining memories when freely recalled (remaining columns).

Self-defining memories themes	Dominant perceptual classifications using Lancaster Sensorimotor Norms (Lynott et al., 2020)
	Visual	Interoceptive	Auditory	Gustatory	Haptic	Olfactory
Relationship (60)
FM (75)	66.5	13.4	15.2	4.1	1.3	0.5
NFM (45)	65.2	14.5	17.4	0.8	1.9	0.3
Achievement (26)
FM (18)	65.4	13.8.	15.8	2.7	2.1	0.2
NFM (34)	65.8	15.1	16.9	0.4	1.6	0.1
Negative (14)
FM (7)	59.4	16.8	16.6	4.3	2.4	0.5
NFM (21)	65.8	13.8	17.8	0.8	1.8	0

Figure 3. Percentages for Food Memories (FM) and Non Food memories (NFM), and corpus scores for dominant perceptual terms: visual, audio, interoceptive, gustatory, haptic, according to Lancaster Sensorimotor Norms (Lynott et al., 2020).

Similar to other studies using memories as unit of analysis (Zimprich & Wolf, 2018) we elicited 72 memories, i.e., 6 from each participant, 3 involving food and 3 not involving food, each recalled twice: before and after the codesign workshops. This ensured a minimum size for the sample of memories to run a between-subject MANOVA test in order to further explore the impact of Memory Type: FM or NFM on these sensory modalities on the free recall. Findings show significant main effect of Memory type (F(6, 53) = 34.06, p < .001, ηp² = 0.33), with significantly more gustatory content within the free recall (F(1,58) = 79.75, p < .001, ηp² = 0.58) of FM (Mean = 4.65, S.D. = 0.95) than NFM (Mean = 0.83 S.D. = 0.63) as shown in Table 4 (Gustatory row, FM – Free Recall, NF – Free Recall columns), and similarly more olfactory content (F(1, 58) = 3.94, p < .05, ηp² = 0.64) of FM (Mean = 0.53 S.D. = 0.61) than NFM (Mean = Mean = 0.20, S.D. = 0.23; Table 4 Olfactory row, FM – Free Recall, NFM – Free Recall columns).

Table 4. Mean scores (SD in brackets) for positive and negative emotional content measured through Linguistic Inquiry and Word Count (LIWC) analysis (row1–2), and Dominant Perceptual terms (row 3–8) from Lancaster Sensorimotor Norms (Lynott et al., 2020) between Memory Types (Food Memories and Non Food Memories) and Cue Types (word and food), with significant differences marked with asterisk: *: p value <.05, **: p value <.01, ***: p value <.001.

	Food Memories (FM)				Non-Food Memories (NFM)
	Free Recall	Cued Recall			Free Recall	Cued Recall
		Word	Food	All		Word	Food	All
LIWC_Pos	2.73 (1.01)	3.83 (2.21)	3.41 (1.32)	3.16 *** (0.22)	2.57 (1.13)	2.33 (1.10)	2.67 (1.01)	2.54 *** (0.22)
LIWC_Neg	0.40 (0.44)	0.39 (0.35)	0.39 (0.35)	0.39 ** (0.65)	0.81 (0.51)	0.60 (0.58)	0.90 (0.65)	0.84 ** (0.87)
Visual	87.53 (30.19)	81.7 (27.59)	86.10 (34.63)	85.11 (26.27)	86.3 (35.85)	99.2 (31.40)	83.95 (34.53)	89.82 (30.85)
Auditory	21.00 (6.6)	20.3 (5.42)	20.00 (6.76)	20.43 (1.71)	22.57 (7)	22.7 (6.77)	22.6 (4.43)	22.62 (1.46)
Gustatory	4.65 (0.95)***	3.6 (2.17)***	4.70 (2.05)***	4.48 (1.11)	0.83 (1.23)***	3.6 (3.53)***	3.3 (2.96)***	2.58 (1.95)
Olfactory	0.53 (0.61)*	0.3 (0.48)	0.80 (0.67)	0.54 (0.41)	0.2 (0.23)*	0.1 (0.32)	0.2 (0.26)	0.17 (0.17)
Haptic	2.17 (1.42)	3.1 (2.69)	3.1 (1.86)	2.81 (1.42)	2.7 (1.83)	4.1 (2.88)	2.9 (1.47)	3.23 (1.89)
Interoceptive	18.53 (6.26)	16.5 (6.15)	17.1 (5.85)	17.38 (5.03)	19.77 (6.35)	20.1 (4.79)	17.6 (7.21)	19.16 (5.22)

4.2. The value of 3D printed flavor-based cues for self-defining memories’ recall

We now report on the cued recall, particularly the impact of flavor-based cues on the sensorial and emotional aspects of recall, on the feeling of being brought back in time, the qualities that make the best flavor-based cues, and participants’ perception of these cues and their value.

4.2.1. Flavor-based cues: rich, visceral and emotionally positive recall

Another key finding is that all participants provided rich sensory accounts when prompted by flavor-based cues, most of such details not being present in the free recall in Stage 2. An illustration of the increased sensory richness is provided in Table 5 showing both the free recall of a specific self-defining memory, and its flavor-based cued recall.

Table 5. Green Thai curry dinner in Cambodia, a food-based self-defining memory from adulthood from P10: freely recalled (left), cued by the co-designed flavor (right).

“I liked [the green Thai curry] very much was when I was in Cambodia and I was teaching at the school for English. They would prepare the food for us, the teachers at lunchtime. But we went to help prepare the food ourselves. We went into the kitchen area, which was very basic and preparing all sorts of types of green vegetables, which I have no idea what they were, sitting on the floor. And then we would help cook them, stir fry them, and then we would help dish them up. And then we all sat at the very long table and all these vegetables. I’ve got an amazing flavor and they would always serve it with a poached egg on top. And it’s just had a flavor, all of its own. I could have just eaten it every day, so nice. [it was myself, the other teachers teaching with me and usually two other Cambodian ladies […] Yeah, the smell of the cooking was amazing”

“I remember preparing the vegetables, sitting on the floor in the school kitchen. The heat, the chatting, the floor being hard […] it was just a piece of old cardboard to sit on the floor and the faces of the ladies from Cambodia who were there. The chopping noises of cutting up the vegetables, me sitting on the floor cross legged with my friend […] chatting together. And then when we went out, put stuff on the tables, the rest of the group coming out and we sit on long tables outside, the front of the school, so it’s outside in the open air to eat. There was about twelve of us sitting at the table, including the staff and ones who were doing the voluntary work. White table cloth on it. And then we put out some stuff but then they’d bring out all the stuff to go with it the drinks, and then they’d bring that the poached eggs to put on the top of the food. So, you helped yourself to what you wanted, and there was noodles as well. And we ate […] it was very, very tasty. You just wanted to eat more of it. It was really nice. [I was] happy. It was a very nice time. I really enjoy being with all these people and it was just a rewarding week to be there. And, the smiles of the children that were, they’re such lovely people. It just made you feel good about everything. They’re so respectful, so pleasant. The children wanted to hug you all the time. And just a lot of respect and kindness to each other there. It was very nice”

Interestingly, the mere act of eating the cue was seen as a bodily reenactment of the original event: “[When I had the food] I tended to talk a little bit faster and a little bit louder [like I did in the speech at the Christmas party … .] I think memory and physical reactions are quite closely linked […] you’re using an extra sense […] because it’s very real because it’s in your mouth and then, that generates other feelings” (P3). As indicated by most participants, flavor-based cues supported a strong visceral experience: “the taste encapsulates, in a tiny thing [the memory; it is] visceral; you’ve got to kind of feel it with your body more” (P3). “It just kind of triggers a few more sensations. Perhaps when you’re tasting it, you imagine yourself there” (P2).

To further explore the impact of cues on sensorial recall, we ran a mixed MANOVA with Time (before and after the co-design workshops) as within-subject independent variable, Memory Type (FM and NFM) as between-subject independent variable, and Sensory scoring (visual, interoceptive, audio, gustatory, olfactory and haptic) as dependent variables. Findings indicate a main effect of Time (F(6,53) = 4.43, p < .005, ηp² = 0.33) with significantly more gustatory content F(1, 58) = 79.75, p < .001, ηp² = 0.58) in the cued recall after the codesign workshops (Mean = 4.05, S.D. = 2.80) than in the initial free recall (Mean = 2.72, S.D. = 2.5). This indicates that following the co-design, the cued recall supports sensorially richer recall, in terms of gustatory modality, for both FM and NFM as shown in Table 4 Gustatory row, FM/NFM – Free Recall, FM/NFM – Cued Recall columns. Findings also indicate an interaction effect between Time and Memory Type for gustatory content (F(1, 58) = 10.01, p < .005, ηp² = 0.15) with significantly more such content in the cued recall of FM (Mean = 4.7, S.D. = 2.05) than of free recall of NFM (Mean = 0.83, S.D. = 1.23), suggesting that cued recall supports sensorially richer recall, in terms of gustatory modality for FM compared to free recall of NFM. Another important outcome is that self-defining FM recall is rich in positive emotions as reflected in most participants’ answers and shown in this illustrative quote: “[emotions] were part of the experience. Like in Greece, we anticipate that going to the restaurant […] because it was so nice and I couldn’t get it anywhere else […] Mum’s roast potatoes […] without the potatoes, it wouldn’t have been the same dinner […] Sundays were special dinners” (P10).

To further explore the impact of cues on the emotional content of participants’ recall we also ran a mixed MANOVA with Time (before and after the co-design workshops) as within-subject independent variable, Memory Type (FM and NFM) as between-subject independent variable, and LIWC emotion scores on recalled content (positive, and negative emotions) as dependent variables. As shown on Table 4, findings indicate a main effect of Memory Type (F(3,56) = 56.0, p < .001, ηp² = 0.32), with significantly more positive emotions (F(1,58) = 3.82, p = .05, ηp² = 0.06) present in FM (Mean = 3.16, S.D. = 0.22) compared to NFM (Mean = 2.54, S.D. = 0.22; Table 4 LIWC Pos row, FM – All and NFM – All columns), as well as significantly less negative emotions (F(1, 58) = 14.71 p < .01, ηp² = 0.20) present in FM (Mean = 0.39, S.D. = 0.65) than in NFM (Mean = 0.84, S.D. = 0.87) as shown in Table 4 LIWC Neg row, FM – All and NFM – All columns. This indicates that, irrespectively of how they are cued, FM are more likely to be recalled with more positive emotions than NFM, highlighting the value of food for imbuing positive emotional content in the original event when the memory was encoded. Without reaching significance, findings indicate that this benefit may extend to flavor-based cues prompting more positive emotional content of FM (Mean = 3.41, SD = 1.32) than Free Recall (Mean = 2.73, SD = 1.01).

Flavor-based Cues: Intense and Emotionally Positive Recollective Retrieval or Feeling of Traveling Back in Time

A striking outcome is the large number of self-defining memories cued by flavors that were recalled with strong feelings of being brought back in time: “[The roast beef and horseradish cue] took me back 25 years in one bound […] I was bit skeptical until it suddenly happened […] I could place myself [at the table in the room …] I ate that, and that actually provoked out of all the memories quite strong reaction actually. Just suddenly I was back” (P3, A Beautiful Wedding, Figure 2). This illustrative quote reflects not just the recall of the knowledge about the original event and its context but more importantly, its recollective retrieval or the associated feeling of actually being brought back there, at the original event. Another similar example: “[The BBQ] mackerel […] was the most evocative of all of them and was […] a trigger [that] brings you back” (P2, Golden wedding BBQ mackerel, Figure 2).

These qualitative findings are supported by quantitative ones that show that 70% of the 40 self-defining memories, that were cued by flavors, were recalled with strong feeling of being brought back in time. Figure 4 shows such data with 36 memories being recalled with high feeling of travel in time (scored with the two top scores of 3 or 4 on the 0–4 scale), and 24 with low such feeling (scores of 0, 1 or 2).

Figure 4. Bar charts showing the number of all memories recalled with high and low time travel by Cue Type (40 flavor cues, 20 word cues).

Flavor cues seem to be also particularly useful in supporting high time travel. To further explore the relationship between high /low time travel and flavor/word cues, we run a chi-square test. Findings indicate that significantly more memories recalled with high time travel feeling were in fact cued by participants’ bespoke flavor-based cues (78%) rather than word-based ones alone (22%) (X²(1,60) = 5, p < .05; Figure 4).

To better understand recollective retrieval we also explored how the type of self-defining memory impacted on it. Figure 5 is an extension of Figure 4, showing the split for FM and NFM for high/low time travel, as well as flavor/word cues. Findings indicate that both NFM and FM were mostly recalled with high feeling of time travel (70% and 50%) albeit the two types of cues impacted differently. Thus, flavor-based cues were particularly more effective in supporting recollective retrieval with intense feeling of travel in time for both NFM and FM (50% and 43% respectively) rather than with low feeling of time travel (17% and 23%). These outcomes are important, as they suggest that NFM can also benefit from strong recollective retrieval, particularly when cued with flavor-based cues. In other words, the intense feeling of being brought back in time when eating a personalized flavor-based cue is not restricted to food memories only, but extends to non-food memories too, massively increasing the potential value of such cues for memories more broadly.

Figure 5. Bar charts showing the number of Food Memories (FM) and Non Food memories (NFM) recalled with high and low time travel by Cue Type (40 flavor cues, 20 word cues).

Findings also indicate that recollective retrieval is emotionally charged, being experienced with intense emotions, particularly positive ones: “it’s very real, because it’s in your mouth and then that generates other feelings [of Christmas celebration]” (P3). As shown in Figure 6, high time travel feeling was associated with intense positive emotions (X²(4,40) = 12.15, p < .05), with over 3 times more self-defining memories characterized by “very positive” emotions being recalled with high, rather than low feeling of travel in time (37.5% vs 12.5%). We can also see how less intense emotional memories cued by flavors are associated with low feeling of time travel. Figure 6 also shows that this pattern does not hold true for the recollective retrieval cued by words, where strong positive emotions supported both high, and low feeling of travel in time, i.e., 12.5 and 7.5%, respectively.

Figure 6. Bar charts showing the number of all memories by their emotion rating (very negative, slightly negative, neutral, slightly positive, very positive) and by high and low time travel and Cue Type (flavor and word cues).

4.2.2. Value of flavor-based cues beyond self-defining memories

An important outcome is that most participants also saw value in the flavor-based cues, beyond the recall of self-defining memories, and particularly for multisensory reminiscing. Here, participants draw from their experience of reminiscing with photos, highlighting the value of flavor-based cues to heighten both the emotional quality: “I think the trouble is some of my memories are quite strong but not as emotional as some; so that [flavor based cue] might actually be more of a sort of trigger if you like to combine food and memory” (P4), and its sensorial richness: “if I’m going to look through my holidays in Italy, photographs, then it would be worth perhaps recreating the taste or enjoying the taste while you looked at them” (P1).

Participants made further suggestions of integrating sound cues, since different modalities were identified as able to provide complementary support for recall: “photographs would go quite well with [flavor-based cues]. Possibly sound recordings would be good, the sound is always another good trigger” (P2).

Another important envisaged value of flavor-based cues concerned dementia and was mentioned by 4 participants based on their own experiences of caring for the loved ones. “My mother has Alzheimer’s […] a couple of times we bought her food [she grew up with] in theory, she would reject it because she said she doesn’t want to eat very much. As soon as she smelled and tasted the food, she would say something like, ‘Oh, this is like old fashioned food. This takes me back’ […] the taste definitely triggered [memories] she felt that it was something that she had had a long time ago.” (P4). This quote suggests ways in which food can not only trigger memory recall but could support a shared experience between the adult daughter and her elderly mother. Another participant envisaged multisensory scrapbooks where photos and flavors may be stored together: “My dad has dementia, [so with flavor cues] I could put ideas in the scrapbook […] and foods connected to [his memories]”(P10). This echoes previous suggestion of leveraging smell for reminiscing technologies (Obrist, Tuch et al., 2014).

4.3. Overall process engaged the senses: before and during the co-design of flavors

Our research and design approach emphasized participants’ senses, both before, and during the codesign of the personalized 3D printed flavor cues.

4.3.1. Before co-designing flavor-based cues

Prior to design, participants engaged in a two-week cultural probes study intended to provoke exploration of food through sensory deprivation and augmentation. Insights from this include participants’ increased familiarization with flavor terms: “[the gameplay facilitated] better range [of flavors] and then, sort of sitting and actually tasting something, and thinking about what you’re tasting rather than just eating it” (P7). The following quote further illustrates participants’ focus, stimulated by the probes, to differentiate between the bitter and sour tastes that untrained observers usually confuse (McAuliffe & Meiselman, 1974): “question about sour and bitter [is] each time in my head […] what’s sour and what’s bitter? I know a lemon is sour, what’s bitter?” (P8).

Another important outcome was participants’ increased sensitivity toward their body experiences and insights into their food cultures: “we couldn’t eat very much interesting foods […] if I am [eating] a Christmas dinner or something, two hours later I probably would have still been feeling it and would have been able to comment on it. But [after] just a light lunch, even probably 30 minutes later there was not much to report” (P9). The body mapping exercise, pointing participants to their bodily responses during and after eating, offered both novel and challenging experiences: “I did find it difficult to say how I felt in the different areas of my body after the foods. Like the initial thing, where in the mouth and the tongue that was not too difficult. But then feeling anything like in my stomach and then my guts later, or like nervous parts of your body, your nerves. I wasn’t quite sure what to put. I couldn’t feel anything or didn’t know what I was supposed to be feeling” (P10). Also related to bodily experiences, the sensory deprivation/augmentation probes facilitated novel perspectives of eating experiences especially with limited aural stimulation: “Yeah, it was the earplug one that was the strongest experience, I found the most fascinating, just really, just focusing on the experience of eating. So that’s what, I got out on that one strongly” (P8), or further supported by another quote: “the sensation that gave me the greatest experience of taste was wearing earplugs and not being able to hear anything. That was the one that really heightens my sense of taste” (P9).

4.3.2. During co-designing flavor-based cues

During the codesign workshops, the conversations were tailored to support participants’ identification of the foodstuff associated with their self-defining memories, and to engage them in the remembered multisensory experience of that foodstuff. This allowed for rich descriptions of flavors and tastes, cooking processes, and assessment of texture and flavor duration, with the aim to create flavor-based cues that would reproduce the target flavor experiences. In turn, relevant unimodal fragments of experience were recalled, selected, and integrated in order to generate the flavor cues. This worked well, with all participants enjoying it, and some even suggested the use of a spiciness scale to better describe the flavors. In contrast to FM, for NFM, half of participants (6) mentioned the additional work needed to identify relevant foodstuff: “it is easy when food is at the center of it [but] it’s harder when it’s a tangential [and] not a direct experience” (P1), which “perhaps didn’t have that sort of realism [while for FM it] reinforces the memory […] made it richer […] triggered some more details” (P2). For NFM, most participants searched for other associated episodic memories featuring food, albeit not self-defining ones. Findings show that most of these associated memories are temporally proximal, such as meals occurring closely: before, during, or after the original event (11/36). Other associations (25/36) were made with temporally distant episodic memories featuring food, selected because of some shared content with the original event such as places, involving foods eaten at a specific location (8); people, involving foods liked or cooked by a relevant person (7); contexts involving foods usually eaten in that context (5); or feelings, related to how people felt during original events (5).

Each identified foodstuff was intended to be reproduced as a flavor cue, with one exception, where it was creatively made from scratch to capture the feeling of the original event where the self-defining memory was encoded:“[To match the feeling] I would say something lemon-y because it was so vivid and then also like sugary. Something very lemony and sugary […] because the other two [choices for cues] are quite comforting. And it was lemon or sugary because it was quite daunting […] I want it to be crunchy but not too chewy [and the color should be] acid yellow” (P12) (An exhilarating dip in the sea memory, Figure 2). This is a key finding, indicating that a memory experience and its emotional and sensorial, yet not gustatory, details can be explored and harnessed to identify emotionally evocative ways of associating them with flavors, such as the association of vividness quality to lemon flavor.

Once the foodstuff was identified, participants rated the intensity of each of the basic tastes present in that foodstuff, with findings indicating the prevalence of umami (33/72) for both FM and NFM, often co-present alongside salty or sweet taste. This was followed in frequency by salty taste (27/72) mostly co-present with umami, and sweet taste (25/72) often co-present with umami and sour. Sour (5/72) and bitter tastes (5/72) were less frequent. Participants also rated foodstuff’s texture with an average score of 3.17 out of 5, and the lingering quality with average score 3.71 out of 5. This suggests that longer lasting taste experiences may be needed to ensure that flavor experiences are intense enough for memory recall. These outcomes extend previous ones on temporal and affective characteristics of taste showing the strongest intensity and lingering quality of umami (Obrist, Comber et al., 2014), toward flavors involving combinations of tastes where umami features high albeit possibly alongside salty and sweet tastes, or just salty taste, or together with sweet and sour.

Apart from tastes, participants also identified additional sensory details, most of them not present when the self-defining memories were initially recalled in Stage 2. Such details included smells (41 identified smells for all the 72 cue designs), both from the foodstuff (22 of 41 identified smells) and the external environment. We classified the latter according to (Almagor, 1990) as related mostly to social and natural processes (30/41), e,g., “sea water drying” (P9), with fewer culturally specific odors (9/41), e.g., “school dinner smell” (P8), and even fewer that held individual significance (2/41), e.g., “my dad’s car, the smell of the leather seat” (P12).

Participants also recalled additional auditory details (17 sounds from 72 cue designs): “sound [of] the crashing waves and my granddaughter’s shrieking” (P12), or visual (14/72) from both food itself: “very much a red meal” (P1, pasta and tomato sauce), or from environment: “a quality of light that was very clear and bright. And I was wearing a hospital gown” (P9, Best strawberries ever memory, Figure 2). These are important findings, illustrating the value of our design activities and the purposeful engagement of senses. Interestingly, such additional sensory details were seldom included in the foodstuff, with the noticeable exception of the creative flavor cue (lemon and sugar) designed for the memory An exhilarating dip in the sea (Figure 2).

4.4. Best flavor-based cues

After the codesign activities, the cues were crafted by the first author who prepared the food material and 3D printed it. Attention was paid to recreate as many specific details of the cues as possible, for example, ensuring that the cue for Golden wedding BBQ mackerel (Figure 2), was made from charred mackerel to develop a smokey flavor. Oil-based recipes were more challenging to produce through 3D food printing, including P12ʹs truffle butter pasta. Most easily reproduced were single ingredients cues that were moist or fluid such as marmite (P7, Saying Goodbye to Mum, Figure 2). Given the diversity of the identified foodstuff, in both solid and liquid form, and the fact that liquids better support taste sensations (Moskowitz & Arabie, 1970), as mentioned earlier, we chose nūfood, a liquid 3D food printer (Nufood, n.d.). It produces small balls of liquid-filled gel that burst in the mouth when bitten, which need to be stored in refrigerator, so are consumed chilled. When piloted, about half of the tested cues (17/36) provided good matching quality as illustrated in this quote: “mackerel absolutely, wow, that’s really good. That’s amazing” (P2), while for 13 flavors participants made minor suggestions to improve them, mostly by increasing the flavor intensity (6 cues), or by adjusting ingredient balance: “herbs [are] overpowering” (P4).

As shown above, the 3D printed flavors cued recollective retrieval, eliciting sensorially rich and strong positive emotional experiences that participants deeply enjoyed. Among the many flavor cues that facilitated such strong feelings of time travel, 14 cues stood out in our participants’ spontaneous accounts. We now reflect on these 14 best flavor-based cues, and their shared aspects that set them apart.

4.4.1. Strong match: Flavor cue experience – original food experience

Matching flavor’s details was the most common aspect of the best cues, referring to both the presence of relevant flavor details, and absence of less relevant ones. For instance, when such matching occurred, the recognition and time travel was strongly supported: “the truffle one was exactly like I [remembered] it to be” (P12). In contrast, 10 of the least successful cues were mentioned as not being well matched or with irrelevant details, confirming their negative impact on recall (Mazzoni et al., 2014): “[The cue for The best strawberries ever (Figure 2)] doesn’t really work […] because the point about that memory of the strawberries is that it was absolutely intense, [the flavor cue] is only very slightly reminiscent of strawberries and fromage frais” (P9). Interestingly, for the only one flavor-based cue which was made to recreate the memory experience rather than to reproduce foodstuff flavor, namely An exhilarating dip in the sea (Figure 2), the matching was also successful, despite not being mediated by a remembered food experience. P12 illustrates this enjoyment: “I actually really enjoyed the ones we had to create new foods for the memories […] the lemon and sugar one, I found that was actually really interesting.”

4.4.2. Distinctiveness: intense or new flavors

Unlike the matching aspect which all cues attempted, distinctiveness was not easy to achieve, and most participants did not explicitly consider it in their design. The best cues however ensured distinctiveness in two ways. First, it is the intense flavors often experienced with heighted delight like in this quote: “chickens spit roast. I’ll think: Oh that time, it was the best I ever had” (P1). In contrast, the perceived lack of flavor intensity was a common aspect of less successful cues: “[They did work] the roast potatoes [as they] just tasted like potatoes, not quite the richness of the roast potatoes that my mom did, not that slightly salty, Oxo type taste” (P10). Similar views of “watered down” (P10) flavors were shared by 6 participants: “[some cues] were disappointing; I don’t quite know why, I would expect them to carry the weight of the intensity of the flavors, I wanted them to have” (P9). This is an important outcome, particularly in the context of aging’s impact on taste sensitivity and people’s increase need for sensorial stimulation (Suto et al., 2014). Second, apart from being intense, distinctive cues could also reflect novel flavors: “when I taste, lobster thermidor, I’m always going to think of lobster thermidor. It’s a standalone experience” (P1). These were usually linked to first-time experiences such as “papaya” (P9), “mango cordial” (P10), or less common combinations such as “fruitcake and champagne” (P3). Distinctiveness of chemosensory stimuli as memory cues has been previously suggested for smells, albeit with limited exploration of how it can be achieved (Obrist, Tuch et al., 2014).

4.4.3. Strong match: flavor-based recall and original experience

The best cues also ensure strong positive emotional recall, cued by intense flavor-based cues, albeit of self-defining memories that are exclusively emotionally positive. In contrast, less effective cues evoked emotional ambivalence, as a result of some of the strong positive self-defining memories being transformed through the passage of time into less positive ones such as a wedding followed by negative events such as divorce (P3), or bereavement (P3). Findings indicate that self-defining memories of positive events are better recalled at times when participants already experience positive emotions, and less so when they experience negative emotions, i.e., wedding memories when one is going through divorce or bereavement.

Interestingly, unlike audio-visual cues that tend to capture an event as it happened, flavor-based cues are more flexible, offering an additional surprising benefit compared to photos, as indicated in this quote: “what you remember [with food] there is no challenge … nice things in your memory tend to happen on sunny days really, if you had a photo to prove actually it wasn’t that nice a day. So that wasn’t quite how I remember this. So you could get potentially conflicting signals” (P3).

5. Discussion

We now revisit the Research Questions to theoretically contextualize our key findings. We also discuss the novel theoretical implications that they entail, in terms of their value for multisensory food interaction, and particularly memory technologies. We introduce 3D printed flavor-based memory cues as flexible resource for design of memory technologies integrating sense data, with specific value for the phenomenological experience of memory recall, and reflect on their qualities.

5.1. 3D printed flavor-based cues integrating sense data and their qualities

Our outcomes indicated that personalized 3D printed flavor-based cues have rich sensorial and emotional qualities supporting strong recollective retrieval, especially when they distinctively match the food in the original experience and prompt emotionally positive self-defining memories.

We advance theory by framing such 3D printed flavor-based cues as perceptual data and contrast it with sense data. We refer to sense data as perceived data about the sensory qualities of stimuli impacting different sense organs and eliciting specific sensations (Auvray & Spence, 2008). Building on this, we framed flavor experience as perceptual data (Hatfield, 2021) and positioned these two concepts in somatic phenomenology, somaesthetics and sensory science. Informed by Gibson’s ecological approach to perception (Gibson & Carmichael, 1966), flavor has been suggested as a perceptual system (Auvray & Spence, 2008; O’Callaghan, 2015; Gibson & Carmichael, 1966; Prescott, 2012). Generated while eating, such system integrates the different qualities of foodstuff and therefore of taste, smell, trigeminal, tactile as well as visual and auditory sensations leading to novel transmodal experiences (Auvray & Spence, 2008). The two concepts of sense data and perceptual data also reflect the long acknowledged distinction between primary and secondary qualities. Sense data relates to properties of perceivable foodstuff things in the world that can stimulate our senses. In contrast, perceptual data relates to secondary qualities, or how such primary qualities are perceived and subjectively experienced by the lived body in the world through tastes, smells, textures, colors or sounds (Martens, 1999).

The lived body in the world as site for the making of subjective meaning reflecting perceptual data and particularly flavor, is a central tenet of Merleau-Ponty’s somatic phenomenology (2013). Phenomenology has long argued for the value of embodiment. This concept captures the way in which the human body engages with, and acts on the physical world, and how by doing so, we both construct and revise the meaning of our experiences, i.e., embodied interaction (Dourish, 2001). Merleau-Ponty’s somatic phenomenology (2013) also distinguishes between specific aspects or phenomenal properties such as sensory aspects, and the phenomenal character of an experience or how this is subjectively experienced (Millar, 2011). Related to Merleau-Ponty somatic phenomenology, the somaesthetics perspective also integrates material aspects of physical world with bodily and subjective aspects of lived experiences, but argues for the value of not merely the implicit somatic awareness but also the reflective one particularly involved in learning or reflecting on one’s experience (Shusterman, 2011). In his insightful work, Shusterman subsequently applied somaesthetics approach to “the fine art of eating” where he unpacked the sensory qualities of foodstuff and the processes involved in ingesting the food in order to support increased sensory discrimination and transmodal appreciation of food (Shusterman, 2016). The latter involves perception of both external stimuli related to food itself and eating environment, as well as those experienced within inner bodily spaces such as mouth, i.e., chewing, sucking, to viscera i.e., fullness; or the spaces internal to the body including proprioception. Shusterman also noted that through retrospective analysis we may distinguish different modalities contributing to the integrated, transmodal perception of flavor. Research on multisensory experiences for the sense of self, have also shown how both external stimuli as well as those experienced within the body, i.e., gastrointestinal system, contribute to embodiment (Tsakiris, 2017), and that people may benefit from developing sensitivity for the latter, i.e., viscerosensory (Daudén Roquet & Sas, 2021b; Häfner, 2013; Jones, 1994; Umair et al., 2021). Neuroscience findings have also indicated the value of such sensitivity for emotional processes (Herbert & Pollatos, 2012).

Merleau-Ponty's phenomenology (2013) has contributed to the philosophical foundation of sensory science exploring and measuring human perception of food (for a detailed account see Martens, 1999). Sensory sciences developed approaches for both quantitatively describing the food sensory qualities, and qualitatively describing the emotional and subjective sensory experience of food, with most of these techniques focusing on individual sense modalities (Martens, 1999).

We frame flavor as perceptual data which integrates multisensory aspects of taste, retronasal olfaction and somato-sensations which are perceived together as a transmodal holistic experience (Small & Green, 2012; Stevenson, 2016), and that, as argued by somaesthetics approach (Shusterman, 2016), by retrospective analysis, we can distinguish flavor’s sensory aspects impacting on our senses, including internal ones. We also leveraged sensory science techniques to support this process of identification of sensory aspects: intensity of the five basic tastes, texture, flavor duration (Ozcelik & Karaali, 2002), sensory description of food’s smells and colors (Williams & Arnold, 1985), and those of the environment where the food was eaten (Auvray & Spence, 2008).

HCI work has also focused on materializing bodily aspects. For instance, Nissen and Bowers translated bodily data of movement onto “data-things” through laser cutting and 3D printing, leading to objects embodying emotional meaning (Nissen & Bowers, 2015). However, in contrast to other modalities, the HCI exploration of experiential qualities of chemosensory stimuli has been limited, with noticeable exception being the exploration of memorable smell experiences showing those associating past personal events with a smell, or those cuing previous memories through a present smell (Obrist, Tuch et al., 2014). We agree with the argument (Maggioni et al., 2020; Obrist, Tuch et al., 2014) that such exploration of lived experiences involving chemosensory modalities is key in order to leverage them in interaction design. This in turn could lead to users’ engagement with novel chemosensory interfaces as emerging new practices (Kuutti & Bannon, 2014; Sanches et al., 2019).

While flavors are particularly prone to being memorialized (Stevenson, 2016), both smell and taste can separately prompt the recall of eating experiences, with each one enhancing the experience of the other (Stevenson, 2016). Moreover, their neural pathways link to similar cortical areas related to emotions, memory, and sensory recall (Miranda, 2012; Stevenson, 2016). The difference is that smell data alone can prompt such recall of sensory and emotional aspects of eating experience without actually tasting the food. Perceptual data within flavors also cues such experience, but unlike smell, flavor-based cues are likely to be experienced in a more embodied way, both in the mouth, nose but also within the viscera (Obrist, Comber et al., 2014), which can lead to more intense phenomenological record (Conway, 2001a) or sensory stimulation during recall, that resembles more closely the original eating experience (Utermohlen, 2002) . Future research can further explore flavor-, taste- and odor-based personalized cues to compare their value in prompting recall of episodic memories. While previous work has focused on such cues, the focus has been mostly on generic rather than personalized taste- and odor-based cues (Ernst et al., 2021). Flavor experiences require ingestion and inhalation, being thus more embodied and visceral than the more distal, HCI mainstream visual and aural modalities. Here, we build on the concepts of embodied interaction (Dourish, 2001) and lived body (Svanæs, 2013) emphasizing the social aspects of interacting with technology and in particular the role of the physical body for embodied perception.

The 3D printed flavor-based cues are small, gel-like, edible balls, modeling the original food, while providing also easier to swallow, more intense flavors, without requiring all ingredients and preparation processes. The perceptual data related to these personalized 3D printed flavor-based cues differ from the original food in three key aspects. First, identifying the original foods that may cue self-defining memories is not trivial especially when such memories do not directly involve food. Second, preparing the flavor-based cues requires only some, rather than all of the original ingredients and cooking processes which may be challenging or possibly detrimental to replicate. The detrimental impact of using original food consisting often of solids, relates to swallowing impairments estimated to affect between 15% and 35% of elderly adults (P. H. Chen et al., 2009) for whom clinical interventions have been designed to involve texture-modified foods in the form of gel like, so that softened solids can be easier swallowed, and thickened liquids can be swallowed without risks (Steele et al., 2015). Moreover, co-designed flavors may differ from the original food in in terms of flavor intensity, addressing thus the impact of aging on decreased perceived intensity of taste from bitter, sour, salty to sweet tastes (Nordin et al., 2003) . Third, once designed, the flavor-based cues become flexible resource material to be used as needed for the recall of relevant memories across a range of interactions.

While the personalized 3D printed co-designed flavors are sensory rich, especially in terms of various flavors, tastes, and color (as they share the color of their main ingredients), their shape and gel like texture presented less variation. Arguably, there is value in the small size shape of these hedonic flavors which may make them be perceived as treats rather than deceptively attempting to replace the shape of the original food. Future work is needed to confirm these insights, by focusing on different shapes and textures. We now reflect on the qualities of 3D printed flavor-based cues, highlighting their value for strong recollective retrieval that is emotionally positive and sensory rich, the quality of emotional catalyst for prompting intense emotional recall, the value for supporting experiential recall of a dynamic past also able to address the limitations of semanticized memories in older age, as well as their distinctiveness.

5.1.1. Flavor-based cues support recollective retrieval: sensory rich, intense positive emotions

Food-related self-defining memories are sensorially richer in gustatory and olfactory modalities, have higher positive emotional and less negative content than non-food memories. Most importantly however, when cued by bespoke flavor-based cues, the recall of self-defining memories is also viscerally rich in gustatory and olfactory modalities, marked by strong positive emotions and experienced to a high degree through an intense feeling of being brought back in time. This extends the value of chemosensory cues and particularly olfactory ones to prompt more emotional recall of episodic memories (Herz, 2004; Herz & Engen, 1996), vivid (De Bruijn & Bender, 2018; Chu & Downes, 2002; Herz, 2004), and their strong recollective retrieval (Larsson et al., 2014) to flavor-based cues. Also key, is that such qualities of flavor-based cues not only hold true for both food and non-food self-defining memories, but that the recollective retrieval is even stronger for non-food ones, thus extending the value of flavor-based cues beyond the niche space of food-related memories, and indicating their potential for memory technologies more broadly.

Another key finding is the richness of both contextual audio-visual details, and interoceptive ones in the recall of self-defining memories. The presence of audio-visual details is probably less surprising given that eating experiences usually involve rich audio-visual stimuli from the food itself, and from the context in which it is eaten, both social and ambiental (Koizumi et al., 2011). Indeed, a wealth of findings has also shown the value of audio-visual stimuli impacting flavor experiences from the color (Spence & Piqueras-Fiszman, 2016) or presentation of the food itself (Michel et al., 2014) or the sounds of eating it, to the ambient light and sound (Koizumi et al., 2011). The prevalence of interoceptive details however is particularly interesting, highlighting the value of body in the recall of these important memories for the sense of self. The role of bodily cues for self-defining memories in older age has been previously suggested in HCI albeit with reference to less able body and redemption narrative (Sas, 2018), rather than the visceral experiences that may be cued.

5.1.2. Rich, flavor cued experiential recall could address older people’s Semanticized memories

Findings indicate that flavor-based cues are particularly strong in supporting highly embodied (Petit et al., 2016) experiential recall which is rich in emotional and sensorial aspects. Our findings indicate that this experiential richness characterizing flavor as perceptual data is grounded in the bodily aspects including interoceptive ones. It is this integrative transmodal flavor experience, that builds on different unimodal sense data, that leads to a richer embodiment quality than for instance, that of individual tastants.

Our findings on the interoceptive, visceral flavor-related experiences also highlight the value of the inside of the body as an emerging new space for interaction design especially for HFI technologies (Gayler et al., 2021), which recent HCI work has started to explore through concepts such as inbodied (Andres et al., 2020) or interoceptive interactions (Daudén Roquet & Sas, 2021a). Our findings also indicate that this experiential richness also relates to an intense feeling of time travel while recalling self-defining memories cued by flavors. This appears to support the idea about the role of encoded sensory information and its activation for episodic memory retrieval (Waldhauser et al., 2016), since mental time travel has been long assumed to rely on the perceived information at encoding stage (Tulving 1993). While arguing for the high embodiment of flavor-cued recollective retrieval, we also draw from the growing evidence, including those from neuroscience (Damasio, 1989) for the somatic or bodily aspect of memory, particularly the sensorimotor model of memory. According to this model, bodily senses engaged during the encoding of episodic memories lead to sensorimotor aspects or so-called embodied memories, and these traces become activated when associated episodic memories are recalled: “Mnemonic traces are not fully amodal mental representations, independent of the body. Rather, they are at least partly reenactments of the original bodily and somatic states, which are simulated through the same sensorimotor pathways involved when the event was encoded.” (Ianì, 2019, p. 1758). Such embodied memories are particularly relevant with respect to eating experiences, given their rich bodily states and embodied mental simulation (Petit et al., 2016).

Our findings also emphasize the value of the increased embodiment characterizing the flavor-cued recollective retrieval. For this, we draw from previous findings showing that the frequency of mental time travel and richness of sensory details of recollective retrieval, particular visual ones about spatio-temporal context, both decrease with the cognitive decline of healthy aging (Costello & Bloesch, 2017; Viard et al., 2011). We have seen how self-defining memories were recalled with rich details that characterize episodic retrieval. This is particularly relevant for older people, given the impact of aging on the specificity of episodic memories, namely fewer episodic details, and more generic or semanticized memories (Piolino et al., 2006). As further argued by Piolino (Piolino et al., 2006), episodic memory theory has started to emphasize more the phenomenal experience: the emotional, sensorial, and spatiotemporal details of the episodic memory, which in turn facilitates recollective retrieval. In other words, the high occurrence of strong feeling of traveling back in time, that was cued by participants’ flavor-based cues, is a reflection of such phenomenal experience, and of cues’ potential to both support and guard against the growing number of semanticized memories characterizing old age. This is an important future direction for integrating HFI and HCI research on aging and in particular on memory technologies for dementia.

5.1.3. Flavor-based cues are emotional catalysts

Findings indicate that while flavor-based cues predominantly prompt intense positive emotional recall, on fewer cases, they can also prompt strong negative affect. This was due either to the negative emotions of the original memory, or to the emotional ambivalence experienced toward a positive emotional memory altered over time through loss. Such outcomes suggest that the rich experiential qualities of the flavor-based cues help them catalyze the emotional content of the recalled memory, irrespectively of its emotional valence, extending findings on the impact of taste on emotions (Gayler et al., 2019, 2020b). These outcomes suggest much-needed sensitivity when designing flavor-based cues, to ensure exclusive matching of these emotional catalysts to positive memories. Future work may also explore their value for originally negative memories that people transformed into redemption narratives, by drawing from the value of crafts for sensory stimulation and self-expression in old age (Livingston et al., 2014; Reisberg et al., 2002; Sas et al., 2015; Sas, Davies et al., 2020; Sas et al., 2017; Wallace et al., 2012, 2013).

5.1.4. Flavor-based cues support emotion-congruent and experiential recall of a dynamic past

Our findings further extend HCI work on memory cues, from the emphasis on visual and aural modalities (Dib et al., 2010; Frohlich & Murphy, 2000; Isaacs et al., 2013; Le et al., 2016; Sas & Coman, 2016; Sas, Davies et al., 2020; Sas & Whittaker, 2013), toward the less-explored flavor one. Photos and videos are ideal for capturing episodic content in the here and now, supporting thus more accurate recall of the original event (Herz, 1998, 2004). Participants also appreciated flavor-based cues for helping them reexperience the original event, albeit not through the factual details that photos depict. Thus, flavor-based cues may be better suited to support experiential recall, i.e., the bodily feeling of the memory and less the accuracy of recall in terms of specific visual details. In other words, unlike the representation aspect of photos that freeze the past, flavor-based cues allow for dynamic nature of memory processes by supporting more integrated emotional recall and meaning, more like flavors to feel the past. Indeed, in contrast to photos or videos, the nonrepresentational quality of flavor-based cues does not necessarily demand verbatim recall. This is important suggesting the additional unexpected benefit of the nonrepresentational quality of flavor-based cues which unlike photos (Herz, 1998, 2004), offer increased flexibility to support the recall of emotional experience albeit not as it has been originally encoded, but how it has evolved through time, after being processed and integrated within the self-memory system (Conway & Pleydell-Pearce, 2000; Conway et al., 2004; Singer et al., 2007). Thus, flavor-based cues may be particularly suited to accommodate the dynamic nature of our memories and support recall in later life of events integrated in self-memory system, for which the accuracy of visual details is less relevant compared to experiential recall, i.e., the bodily feeling of the memory, and as shown earlier, the feeling of travel back in time. Our findings also extend the mood congruity theory according to which people recall better those memories whose emotional content matches their emotional state at the time of recall (Rusting & DeHart, 2000), to self-defining memories and more importantly, to the value of flavor-based cues for the exclusive recall of positive self-defining memories and not negative or ambivalent ones.

5.1.5. Distinctiveness of flavor-based cues: crafting “the flavor of memory”

Our study shows an approach to cue design that has focused on reproducing key aspects of the food that either formed part of FM or was associated with NFM. Therefore, our outcomes confirm the encoding-specificity principle (Tulving & Thomson, 1973). However, an additional, cue distinctiveness principle argues that matching is necessary but not sufficient for accurate recall, and that the cue should also be uniquely associated with one memory only (Schmidt, 1991; Wheeler & Gabbert, 2017). The distinctiveness of chemical stimuli for memory recall has been also previously suggested for smell-based cues, albeit with limited exploration of how such uniqueness can be achieved (Obrist, Tuch et al., 2014). Our findings indicate that most successful cues ensured distinctiveness through intense or novel flavors, which were however difficult to identify among the foodstuffs people usually gravitate toward due to their preference for familiar over novel food, i.e., neophobia (Gayler et al., 2018; Pliner & Salvy, 2006). One way to ensure cue distinctiveness (Lee & Dey, 2007) is through the crafting of new flavors, an approach taken surprisingly by only one participant who explored the emotional and sensorial aspects of their memory experience (which did not involve food) to identify cross-modal associations to flavor domain, such as vividness to lemon flavor. In this way, they created a unique “lemony and sugary” flavor, or in other words, The flavor of that memory. These outcomes suggest the value of supporting the crafting of such cues, possibly by integrating new sensory design methods within crafts approaches that older adults particularly enjoy (Lazar et al., 2017b, 2017a; Sas et al., 2015, 2017; Wallace et al., 2012).

5.2. Body-centric approach engaging the senses for designing personalized flavors as self-generated memory cues

The novel outcome here is the multi-step process for co-designing the flavor-based cues including the two preparation steps of identifying self-defining memories for later recall and their elicitation, the taste calibration step, the actual co-design of the flavor cues based on description of flavor qualities, preceded if needed by association between NFM and flavors, and the final step for the preparation of the flavor-based cues. The two co-design steps are particularly important, and their design has been informed by previous work on the structure of episodic memories (Lee & Dey, 2007; Sas et al., 2013; Sellen et al., 2007) supporting the identification of places, events, objects, people and feelings associated with each memory, and of potential flavors that may cue the recall of these individual aspects, followed by the review and selection of the most recognizable combination of flavors to cue the memory as a whole.

For the next co-design step we elicited sensory descriptions of these flavors in terms of intensity of basic tastes, texture, temporal aspects (Ozcelik & Karaali, 2002), as well as key flavors, smells and colors (Williams & Arnold, 1985) inspired from sensory science, including also relevant ambient aspects impacting flavor perception (Auvray & Spence, 2008).

With respect to cultural probes, their content is similar to the one presented in previous HFI research (Gayler et al., 2020b, 2021). Our work however innovatively extends the exploration of these probes and their value to inform the co-design of flavor-based cues.

Our proposed approach for generating the 3D printed flavor-based cues has a strong body lens (Dourish, 2001; Svanæs, 2013) leveraging the human body throughout the entire process, from the initial stage of sensitizing participants toward their food experiences, to the iterative co-design and evaluation of flavor-based cues. The three-stage method that we advanced and its steps proved useful in supporting participants throughout, including cultural probes to help them reflect and identify sense data as key sensory fragments of self-defining memories, further refined through the co-design process where these become integrated in perceptual data as 3D printed flavors. Finally, we evaluated the latter, by having participants consume them in order to explore their value for recall. Throughout our body-centric approach, the sense data has evolved and become integrated in perceptual, flavor-based data. Within this approach, our bespoke cultural probes share qualities with sensory probes previously described as exploratory design research methods (Gayler et al., 2021b) well suited to capture fragments of user experience. We extended this by using the sense data captured through the probes not only to understand eating experiences but to also support the co-design of personalized flavors. Through the co-design process, the identified sense data has been reflected upon, selected, consumed, refined, and integrated in perceptual data in the form of 3D printed flavors highly infused with personal meaning. This meaning then become both emotionally and viscerally reexperienced through the eating of flavor-based prompts in order to cue the recall of their associated self-defining memories.

The 3D printed personalized flavors offer an interesting illustration of hybrid crafting, previously used in HCI to integrate computational and non-computational elements in one artifact (Golsteijn et al., 2014; Umair et al., 2020). However, most of previous work involved non-computational elements such as colors or textile which can be manipulated but not ingested or inhaled. In contrast, our 3D personalized flavors consist of edible organic materials that have been less explored beyond HFI research (Obrist, Tuch et al., 2014), and even there with less emphasis on how they can be co-designed to cue memories.

Because of their ability to integrate personally relevant salient features, self-generated cues hold strong potential to support recall (Hunt & Smith, 1996). However, most HCI research has focused on automatically or manually captured cues, with a few exceptions that looked at visual cues self-generated by young people (Le et al., 2016; Sas et al., 2015). Our successful flavor-based cues indicate the feasibility of their self-generation through codesign. This is an important outcome, as previous work on older people’s self-generated memory cues has shown that these are less distinct (Mäntylä & Bäckman, 1990) or contain more generic and semantic details rather than episodic ones, similar to the cues generated by others rather than the self (Wheeler & Gabbert, 2017). Such work however has looked at word-based cues, arguably less suitable to address the more semanticized memories in the old age (Piolino et al., 2006), and more importantly, less engaging. In contrast, our codesign approach and its sensorial and emotional richness is more likely to boost the generation effect (Glenber et al., Glenberg et al., 2013; Slamecka & Graf, 1978) and participants’ investment in their cues, with the additional benefit of ensuring recall marked by rich phenomenal experience instead of generic recall.

5.3. From multisensory food experiences to emotionally meaningful food memories

Our outcomes make key contributions to the understanding of chemosensory modalities more broadly, and in particular flavors, and implicitly smells experienced through orthonasal stimulation, showing that the well-documented value of olfactive cues for emotional (Herz, 1998, 2004; Herz & Engen, 1996), vivid recall (De Bruijn & Bender, 2018; Chu & Downes, 2002; Herz, 2004), and strong recollective retrieval (Larsson et al., 2014), also applies to the less-explored flavor cues. Our findings build on those from HCI research on commensality showing that food is quintessentially social within the cultural practices of food sharing (Andrea et al., 2011) where significant events for the relational self tend to take place. In addition, our findings extend previous ones by indicating that food sociality is also reflected in older people’s self-defining memories, both food and non-food related, as these are interlinked with enduring concerns regarding loved ones (enduring concerns being key in self-defining memories).

This new perspective on food as resource for the design of memory technologies has been less explored in multisensory food interaction research (Altarriba Bertran et al., 2018; Choi et al., 2014; Gayler et al., 2022a). Our findings however suggest that not just multisensory food experiences and their hedonic qualities are worth designing for (Altarriba Bertran et al., 2020; Arza et al., 2018; Dolejšová & Lišková, 2017; Wang et al., 2020), but also for emotionally meaningful food memories such as self-defining ones, whose recollective retrieval has rich multisensory experiential qualities.

6. Implications for design

Personalized 3D printed flavors can open up the space of HFI technologies and when integrated with memory technologies they can support novel interactive systems leveraging food for a range of purposes. We now discuss our findings’ implications for design, highlighting the value of recreational reminiscing, of therapeutic multisensory reminiscing through emotionally meaningful flavor stimulation in dementia, and of body-centric multisensory design methods.

6.1. Recreational multisensory reminiscing

Findings suggest the value of flavor-based cues for multisensory reminiscing. This is a new space for interaction design that can be extended from augmenting the capture of memory content with additional flavor qualities, to consuming flavor-based cues for multisensory reminiscing in familial domestic settings. For the former we can imagine rich vocabularies and icon libraries that can be used to capture episodic content of flavor experiences for instance, through smartphones. Such vocabularies would include expressive terms for tastes, smells, and flavors, while libraries would consist of expressive icons for tastes, aromas, textures, or lingering feelings visualized with affective qualities.

The use of chemosensory modality for reminiscing has been previously suggested, for example, as digitized incense stick for multimodal storytelling that can be used both for nonclinical populations and those with memory impairments (Obrist, Tuch et al., 2014). We further extend such implications for multisensory reminiscing technologies centered on flavors. For instance, one can think of novel interactions where captured flavor experiences can be browsed by emotional, perceptual or spatio-temporal metadata, and selected for 3D printing, either at home with small printer like nūfood (Nufood, n.d.), or by new 3D printing services providing on demand personalized flavors in the form of precious pods, or as one of participant referred to as “a tiny thing” encapsulating the memory in a taste. For instance, for an evening in, a family decides to reminisce over the photo album of their last holiday in Morocco. They had some delightful experiences from a small restaurant with open fire that they would love to relieve. The flavor printed pods are small, so they need to be slowly savored for extending the delicious flavor sensation, and one by one, family members share their feelings of being brought back in time.

6.2. Therapeutic multisensory reminiscing through flavor stimulation in dementia

Participants also saw value of flavor-based cues for their loved ones living with dementia. This is an interesting design opportunity, given older adults’ need for increased sensorial stimulation (Sas, Davies et al., 2020), especially for flavors (Møller et al., 2007), and their increasing eating difficulties (Watson, 2002). In contrast, the small 3D printed flavor encapsulating the memory of their wedding, or the birth of their child could support intense feeling of time travel, strong positive emotions, and sensorial richness, much needed in dementia. The value of chemosensory stimuli such as tastes for mood regulation (Gayler et al., 2020b), or smells (Obrist, Tuch et al., 2014) has been previously suggested, and our findings indicate the potential of extending it to flavor-based cues. Such flavors will need to be carefully designed, given the reduced taste/flavor recognition and sensitivity in dementia (Suto et al., 2014). Future work would explore the feasibility of a sensory codesign approach with people living with dementia and their loved ones, to understand how such cues can be crafted with vulnerable users in sensitive settings and how they can be leveraged as a site for reminiscing experiences shared by those with dementia and their loved ones.

6.3. Toward body-centric multisensory design methods

We now reflect on our overall design approach that purposefully engaged both participants’ and researchers’ senses through the sensory deprivation and augmentation probes in the sensitizing stage, to the codesign, making, piloting, and consuming the flavor-based cues in order to viscerally experience their impact. We learned four key insights. First, it is the acknowledged challenge of participants’ accessing the felt-life quality (McCarthy & Wright, 2005) of their rich multisensory experiences. Despite the growing body of HCI work focusing on accessing the felt-life experience from explicitation interviews (Obrist, Comber et al., 2014) or posture cards (Sas, Davies et al., 2020), capturing the depth and somatic richness of flavor experiences is not trivial, yet key in HCI (Maggioni et al., 2020; Obrist, Tuch et al., 2014)l, given also the multiple factors impacting on them (Spence & Deroy, 2013).

We advanced a multi-step approach to the design of flavor-based cues, where the human body features strongly in each step. By reflecting back on our methods, we acknowledge the value of our range of objects in the cultural probe kit for evoking distinct sensory experiences by turning off some senses. So did the process of sensory deconstruction in the memory elicitation stage, where we focused on each modality separately and their sensory fragments, or in the codesign stage, where we brought an even stronger focus on the flavor experience.

One key aspect of the description and exploration of the sensory experience was to capture as complete a picture as possible but then refine it through participants’ comments on the most salient aspects. This helped the production of the cues as it orientated the researchers toward the most important qualities to reproduce. Nevertheless, this challenge required an introspective, bodily and inwards looking stance, less familiar to our participants, who could benefit from a more structured facilitation inspired for instance by the emerging microphenomenology approach in HCI (Prpa et al., 2020).

Second, the challenge of communicating about rich and nuanced experiences with our participants. While all participants shared them, the depth of their verbal descriptions varied largely, and we could benefit from sensory vocabularies to better support sharing of expressive multimodal experiences that could leverage for instance, soma design (Hook, 2018) and somatic approaches (Schiphorst et al., 2020).

Third, the challenge of writing about our work. As researchers we felt the need to be more creative in the writing of the paper, as traditional descriptive accounts would have left out much of the richness of our data. Hence, we reached deep into our data to craft concise descriptions of 5 self-defining memories (Figure 2). We used these experiential vignettes to better illustrate key findings throughout the paper, alongside evocative quotes but could benefit from even more tailored approach inspired for instance, by sensory ethnography (Pink et al., 2013).

Finally, the employed three-stage process has been effective but comes with significant researchers’ involvement. To streamline this process and support users to create flavor-based cues with less support from researchers or designers, we can think of novel mobile apps. This could capture food experiences and their rich multisensory qualities, support users make creative associations between qualities of flavors, tastes, smells, colors, sounds, or textures and the key aspects of the episodic memories of these experiences such as place, time, people. This in turn can support them to select and combine such multisensory qualities in order to generate distinctive cues. For instance, in a recent work, we described the initial design of a mobile app interface that can support users to capture their multisensory flavor experiences and co-design flavor-based cues (Gayler et al., 2022b). This app explores how episodic memories and their sensory aspects can be captured by users themselves, without the support of researcher/facilitator, through different evocative icons and visual patterns for smells, and how sensory evaluation scales for texture, temperature, and color can be visualized in engaging ways. Future work could focus on evaluating such apps in the wild.

To conclude, there is a strong emerging foundational HCI research for how we can better work with the body, both our own (Alfaras et al., 2020, Karpashevich et al., 2022) and participants’ (Daudén Roquet & Sas, 2020), but more research is needed to inject an explicit multisensory lens into such approaches and to better inform new sensory research and design methods. These would firmly support the sensory turn in HCI (Brulé & Bailly, 2018), started two decades ago in social sciences (Harris & Guillemin, 2012) and humanities (Lauwrens, 2012).

7. Limitations

Given the qualitative approach to research, our findings should be cautiously generalized to similar homogenous population of university educated, older adults in Western context. Our sample also reflects a skewed gender balance. However consistent findings have shown no impact of gender on taste perception especially for older people (Sanders et al., 2007). The impact of gender on autobiographical memories recall and especially self-defining memories has been less explored, with initial findings suggesting no impact on vividness of recall, albeit potential impact on emotional content which although richer in women’s descriptions (Niedźwieńska, 2003) does not appear to lead to increased emotional ratings of such memories (Wood & Conway, 2006). Future work is needed to further explore such potential gender impact.

8. Conclusion

We explored the value of codesigning flavor-based memory cues for self-defining memories in old age. We advance theory on flavor-based cues by finding strong evidence for their value in recollective retrieval. We also articulate the experiential qualities of flavor-based cues and provide rich new data about the sensory approach to their codesign, arguing for the value of crafting the flavor of memory. Our findings led to three implications for the design of recreational, and therapeutic multisensory reminiscing, and for novel multisensory design methods.

Acknowledgments

This work was supported by the Designing Multisensory Interactions: Emotional User Experience of 3D Printed-Food Consumption project (Grant Number 1962364) funded by EPSRC and Dovetailed Ltd. under a CASE PhD award and AffecTech: Personal Technologies for Affective Health, Innovative Training Network funded by the H2020 People Programme under Marie Skłodowska-Curie GA No 722022.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This work was supported by the Engineering and Physical Sciences Research Council [1962364]; European commission Horizon 2020 Framework Programme [722022].

Notes on contributors

Tom Gayler

Tom Gayler is a Service designer, he holds a PhD in Human-Computer Interaction from Lancaster University.

Corina Sas

Corina Sas is Professor in Human-Computer Interaction and Digital Health with the School of Computing and Communications at Lancaster University, UK.

Vaiva Kalnikaite

Vaiva Kalnikaite is the founder of Dovetailed and holds a PhD in Human-Computer Interaction from the University of Sheffield.