Introduction to this Special Issue on Designing for Personal Memories: Past, Present, and Future

Introduction

This special issue focuses on new uses of digital media to help people remember in everyday situations. We begin this introduction by describing the field's origins (personal memories past), using this to contextualise the papers presented here (personal memories present). We conclude by identifying a number of important research challenges that we feel must be addressed by future work in this area (personal memories future).

PERSONAL MEMORIES PAST: ORIGINS OF THE FIELD

Memory is central to our sense of self and critical to our everyday functioning. We use our memories to maintain our personal identity, to start and mediate relationships, to shape our likes and dislikes, to regulate our moods and solve problems. Recording memories and passing them on to others has been important for as long as we know, through storytelling and writing. Recently however digital technology has started playing a role, giving rise to questions about how we can use technology to support everyday remembering and leading to this special issue. The growing importance of the field is indicated by “Memories for Life”, a ‘grand challenge’ identified by the UK Computing Research Committee, and by ambitious research programs in industry, such as Microsoft Research's MyLifeBits and SenseCam.

People are all too aware of the fallibility of their own memories. More and more memory-related media are now digitally capturable, including videos, audio snippets, and even olfactory or haptic cues. One common response to the emergence of these personal digital media has been for people to hoard as much personal data as possible, assuming this will shore up fallible memory and promoting ‘total recall’ (Bell & Gemmell, 2009; Hoven & Eggen, 2008; Sellen & Whittaker, 2010). People increasingly have vast collections of digital media about their pasts, including photos, videos, text and music files. They collect because they can (Van House & Churchill, 2008); data is cheap to store and keeping everything is less effort than deleting or organising it (Mayer-Schönberger, 2009; Whittaker, 2010).

One additional trend that will further contribute to these huge collections of personal media is the emergence of new ‘lifelogging’ tools that make everyday recording effortless (Gemmell et al., 2002, 2004; Mann, 2004). These tools originated in research labs (Hodges et al., 2006; Vermuri et al., 2004), but have recently become commercially available. One example is the Vicon Revue based on Sensecam (Hodges et al., 2006). This is a wearable camera where recording is triggered by sensors, such as changes in light, heat or movement generating 5–7000 images/day. Another contributor to the emerging personal data deluge is data generated as a side effect of other online activities, including facebook posts, locational data or blogs.

This abundance of digital media means that people are now required to organize their personal data, since there is a lack of useful commercial products doing it for them. This curation problem is exacerbated both because people have a resistance to deletion (Bergman et al., 2009; Whittaker, 2010) but also because personal data are not as well organised as their owners think (Whittaker, Bergman & Clough, 2010). Another critical observation is that digital media are invisible: comparisons of physical and digital media show that digital memorabilia are less salient and judged to be generally less important than their physical counterparts (e.g. Kirk & Sellen, 2010; Petrelli, Hoven & Whittaker, 2009; Petrelli & Whittaker, 2010).

But collecting and curating serve little purpose without retrieval and much new work centres on retrieval of media and how this supports remembering. Photos are the most used media type for remembering, and this has led to a number of field studies exploring the use of photos in collaborative remembering (Frohlich et al., 2002; Kirk et al., 2006; Sarvas & Frohlich, 2011) and interactive applications and new devices that support interactive photo retrieval (e.g. Balabanovic′, Chu & Wolff, 2000; Hoven & Eggen, 2003; Kirk et al., 2010; Nunes, Greenberg & Neustaedter, 2009; Shen, Lesh & Vernier, 2003; West, Quigley & Kay, 2007). New technologies to assist organization and retrieval include techniques for tagging photos, automatic methods for indexing using face or object recognition and different interactive retrieval interfaces. However the majority of these tools are not yet widely deployed.

PERSONAL MEMORIES PRESENT: WHERE THE FIELD IS NOW

The seven papers in this special issue provide a representative snapshot of the field, revealing its current scope and diversity. Since these papers are multidimensional, there are different ways to organise them.

One classification is by application area, including lifelogging (Doherty et al., this issue; Whittaker et al., this issue), addressing the needs of people with memory challenges (Crete-Nishihata et al., this issue; Lindley, this issue), designing technology heirlooms (Banks, Kirk & Sellen, this issue), facilitating everyday communication by involving our past (O'Hara et al., this issue) and everyday reminding (Cosley et al., this issue).

The papers could also be classified according to the remembering process they support. Memory research has typically distinguished three stages of remembering: creation, storage and retrieval, with each stage being potentially performed passively or actively. For example, passive creation encompasses the lifelogging work of Doherty et al., Whittaker et al. and Crete-Nishihata et al. Active creation is studied by Crete-Nishihata et al. and Lindley. Both passive and active storage are covered by the heirloom study of Banks et al., while Cosley et al. and Lindley discuss active storage. Passive retrieval, or unrequested triggering of memories, is the main focus of Cosley et al.'s Pensieve system, whereas active retrieval involving deliberate recollection or reminiscence, is described by Crete-Nishihata et al., Doherty et al., Whittaker et al. and O'Hara et al.

A third way to classify the papers in this special issue is by method. Each paper in this special issue also employs a multi-faceted method. Most papers include design interventions to study how memory can be supported using technology and include fieldwork and/or evaluations in the wild to get to grips with people's memory needs. We use this classification to introduce the papers. Three important strands of the current personal memory research are represented in this special issue and discussed in more detail below: fieldwork showing current practices and providing implications for design; evaluations of novel designs; and lifelogging perspectives and memory tools.

Strand 1: Fieldwork Showing Current Practices and Providing Implications for Design

The fieldwork strand explores current practices of capturing, storing, retrieving, and sharing personal memories, such as Dib et al. (2010), Durrant et al. (2009), Golsteijn & Hoven (in press), Odom et al. (2009), Petrelli, Hoven & Whittaker (2009), Petrelli, Whittaker, & Brockmeier (2008), Petrelli & Whittaker (2010). In addition to characterising practice, this strand contributes design guidelines for memory technology. The fieldwork strand is represented in this issue by Lindley and Whittaker et al.'s papers.

Siân Lindley presents fieldwork on older people looking at how they prepare mementos to be passed on to family or close friends. She identifies 3 main activities: active curation, preparing mementos, and creating personal reminiscences. Curation is the active process of taking family records and annotating them so that someone else can make sense of them. Preparing mementos involves taking curated materials such as family photos and creating an artefact such as a scrapbook or family album that represents one interpretation of the past. Active reminiscence involves reconstructing the past without reference to past archives relying solely on one's own memories. The paper reflects across these activities noting that in general people tend to focus on remembering the unique rather than the mundane, and presents design implications concerning new tools for annotation, abstraction and reminiscence.

Steve Whittaker, Vaiva Kalnikaitė, Daniela Petrelli, Abigail Sellen, Nicolas Villar, Ofer Bergman, Paul Clough and Jens Brockmeier provide an overview of (mostly ethnographic) studies into lifelogging. The authors argue from their results that prior perspectives on lifelogging have been too technology-oriented, instead proposing a more user-centered perspective based on four new design principles: Selection, Embodiment, Reminiscence and Reflection, Synergy not Substitution. Specifically, lifelogging applications need to be more selective in what they present, to facilitate access. Personal digital media should be designed to be more salient, through augmentation or embodiment, so that they more closely resemble physical media. We should move away from an exclusive focus on factual recall, to think more about reminiscing and reflecting. Life-logs should not be designed as replacements for unaided memory, but instead deployed where they provide demonstrable advantages over unaided memory. Each principle is used as motivation for implementing and evaluating successful real-life working demonstrators.

Strand 2: Evaluations of Novel Designs

The second strand of current research is concerned with the development and evaluation of innovative designs for memory technology. Examples of such technologies are the Memory Box (Frohlich & Murphy, 2000), Living Memory Box (Stevens et al., 2003), MemoryLane (Kalnikaité & Whittaker, 2011), Sonic Souvenirs (Petrelli et al., 2010), Personal Audio Loop (Hayes et al., 2004) or augmented mementos with digital information (Hoven & Eggen, 2008). The special issue also presents work on this strand, particularly in the papers by Banks et al. (this issue), Crete-Nishihata et al. (this issue), O'Hara et al. (this issue) and also parts of Whittaker et al. (this issue).

Richard Banks, David Kirk and Abigail Sellen explore technology heirlooms: artifacts with physical and digital properties that are meant to become memorials after their owners have passed away. The authors use a design research approach to characterise this space in terms of four emerging themes, “relations to people”, “connections to memory”, “object qualities” and “types of record”. These themes inspired multiple design concepts of which three were implemented in deployable or conceptual designs. Timecard is a wooden photo frame that can be used as a personal timeline of the deceased. The Backup Box automatically backs up twitter feeds in ways that might later acquire the same meaning as diary entries. Digital Slide Viewer is a physical device that could potentially contain online photo collections of the deceased, to make them accessible to relatives. This paper shows how design methods inspire innovation in a difficult and delicate matter such as remembering the deceased.

Masashi Crete-Nishihata, Ronald Baecker, Michael Massimi, Deborah Ptak, Rachelle Campigotto, Liam Kaufman, Adam Brickman, Gary Turner, Joshua Steinerman and Sandra Black focus on people who have challenges in remembering, e.g. those with Alzheimer's disease or mild cognitive impairments. They describe three studies that were originally aimed at supporting memory, but evolved to include different design considerations as participant identity became an emerging issue. The authors built multiple life review applications, including multimedia biographies, narrated slideshows and a SenseCam-fed photo display drawing on both recent and long term past. The applications were used over extended periods by 18 patients, family members and caretakers. The results show that biographies and significant life events can be represented using multiple media types and follow many different storylines, depending on who creates them and their intended audience. The authors conclude by proposing new design implications.

Kenton O'Hara, John Helmes, Abigail Sellen, Richard Harper, Martijn ten Bhömer and Elise van den Hoven describe a novel device that presents personal photos in a public social setting to facilitate social reminiscence. A key aspect of the device is that control is shared across multiple participants. Use of the device was explored in a meal setting. Participants had previously uploaded personal photos, and the device cycles through these images, allowing participants to share and discuss those images with others. The authors explore the effects of the device on social conversation between relative strangers, old friends, and parents with children. The device is shown to enable strangers getting to know each other, while allowing also for reminiscing, social reflection and strengthening of family ties. In addition, it supports children to participate in family conversations while being lightweight enough to be integrated into conversation without disrupting the social activity of eating.

Strand 3: Lifelogging Perspectives and Memory Tools

The third strand of research on personal memory concerns lifelogging practices and technologies and includes prior work on MyLifeBits project (Gemmell et al., 2006; Bell & Gemmell, 2009) or the development and evaluation of SenseCam technology (Lindley et al., 2009; Kalnikaité et al., 2010; Sellen et al., 2007). This strand also concerns the developing and evaluation of reminding tools and memory aids, such as the MemoClip (Beigl, 2000), the CybreMinder (Dey & Abowd, 2000) and Memory Glasses (DeVaul, Pentland & Corey, 2003). Some of this work has focused on supporting people with memory impairment such as Alzheimer's patients or elderly users (Wu et al., 2008). In this special issue this topic is represented by Doherty et al. and Cosley et al.

Aiden Doherty, Katalin Pauly-Takacs, Niamh Caprani, Cathal Gurrin, Chris Moulin, Noel O'Connor and Alan Smeaton work with a unique set of images collected using Sensecam over a period of several years. They review cognitive psychology research concerning the nature and functions of autobiographical memory, describing how new tools such as Sensecam support the recall of everyday events, by passively recording images relating to everyday activity. The authors then describe and evaluate four different software tools that are intended to help access Sensecam images. These tools segment events—allowing participants to browse organised images; they allow event retrieval by detecting similarities between different events; and they rank events in terms of their importance, as signaled by their similarity to other habitual events. Finally the authors describe a faceted browser that allows people to access Sensecam images in terms of people, places, time or events.

Dan Cosley, Victoria Schwanda Sosik, Johnathon Schultz, S. Tejaswi Peesapati and Soyoung Lee describe a long-term deployment and evaluation of Pensieve, a tool to support reminiscing through memory triggers. Triggers include emailed reminders of previously posted social media content, and generic questions that encourage reflection. The originality of this work is its focus on reminiscing with social media, allowing everyday practices to be made explicit. The system was evaluated with 20 participants using the system for six months. Findings indicate the value of providing reminders about past social media posts for supporting reminiscence and provide insights into people's current practices in reminiscing using social media. The paper concludes with issues to consider when designing systems to support reminiscence and reflection about personal experience.

PERSONAL MEMORIES FUTURE: ISSUES FOR FUTURE RESEARCH

Having charted past and present personal memory research, in this section we identify gaps in the current state-of-the-art, future challenges and new directions for advancing the field in meaningful ways.

Human memory types

Many current memory technologies claim to support general aspects of remembering without making their exact function explicit. Future systems need to better target specific types of human memory; this is necessary as 256 types of human memory have been described in the literature (Draaisma, 2010). Recently simpler taxonomies for memory technology have been proposed. Sellen & Whittaker (2010) suggest five different types of memory we need to support: Recall, Reminiscing, Retrieving, Remembering Intentions and Reflecting. Other taxonomies derived from autobiographical memory research suggest systems should focus on personal identity, relationships, regulating moods, shaping likes or dislikes or problem solving (Cohen, 1996; Pillemer, 1998). Being clear about system function is especially critical for lifelogging applications which currently focus on exhaustive data capture without analysis of what functions such data might serve.

Forgetting forgetting

The focus on exhaustive capture brings us to an often overlooked aspect of memory: forgetting. Intuitively people construe forgetting as a weakness of human memory (Bannon, 2006), however it can also be seen as the by-product of successful remembering (Schacter, 1999; Harris, Sutton & Barnier, 2010). But forgetting (or not having access to information) is usually seen as an undesirable state, because information loss seems irreversible as you never know what you will need in the future (Churchill & Ubois, 2008; Whittaker, 2010). However, simply storing everything ‘just in case’ clearly leads to problems of severe information overload at retrieval (Whittaker et al., this issue). One important question is how systems might be designed to ‘forget’, although this will require subtle new designs to overcome people's strong biases against information loss. One approach might be to reduce the salience of certain information.

Supporting commemoration

When thinking about forgetting, it is a small step to people who do not want to be forgotten: the deceased. Remembering the deceased and supporting mourners is a new and upcoming topic. It is currently acquiring momentum in Asia, where the perspective on the afterlife is different from the West. But also in the West there is a need for innovation since younger generations are increasingly using technology to commemorate their loved ones, and older people are leaving digital traces behind when they die. This opens up a new design space for commemoration (e.g. Hoven et al., 2008; Massimi et al., 2011; Banks, et al., this issue).

Media effect on memory

Many personal digital memory media, such as photos, videos and music, have now been established for a number of years, making it possible to study the effects of these media types on memory and vice versa (Hoven & Eggen, 2009). One critical question is whether the existence of these new media are fundamentally transforming the ways that we remember (Dijck, 2007; Middleton & Brown, 2005). This is an age-old debate going back to Plato who argued against writing because it would compromise our ability to remember through spoken narrative. We already know that people's memory for factual content weakens when they have access to information online, known as the ‘Google effect’ on memory (Sparrow, Liu & Wegner, 2011), leading to speculations that technology might cause people to lose certain remembering skills (Bos, 1995). However other work suggests that people are adaptive in their use of technology being highly strategic about how and when they exploit external mnemonic resources (Kalnikaité & Whittaker, 2007).

Cherishing the digital

We also need to overcome barriers to creating compelling digital memory applications. One clear issue with digital memorabilia is that they seem to lack salience compared with their physical counterparts leading them to be overlooked (Kirk & Sellen, 2010; Petrelli & Whittaker, 2010). Here we might look to theories of external and distributed cognition (Hutchins, 1995) for design inspiration about compelling ways to embody digital media or to augment our everyday physical environment. Future memory media may include both physical and digital aspects, the physical being visible, tangible, unique and an implicit memory cue, while the digital supports copying, sharing, editing and dynamic display.

New technologies and tools

Other important trends concern technical developments in retrieval and management tools, including face recognition, locational tagging and other automatic processes for organizing content (Doherty et al., this issue). New algorithms and techniques that automatically tag digital media with relevant people, places and even content type promise to address some of users' management and retrieval problems with their ever growing archives.

Longing for longevity

Another new challenge is longevity. How can we guarantee the long term viability of digital media, when people already cannot retrieve their personal media from just a few years ago (Van House & Churchill, 2008; Whittaker, Bergman & Clough, 2010)? People currently do not trust media to be available in the future and many have already experienced loss of important personal information (Petrelli, Hoven & Whittaker, 2009; Petrelli & Whittaker, 2010). Future studies will therefore have to pay attention to longevity and of actively preserving digital data (Marshall, Van House & Churchill, 2008). New services are emerging that allow users to delegate archiving to third parties, but there are important technical requirements beyond simple storage that require backward compatibility of data as applications change or become obsolete.

Reflective remembering

The relations between emotion, memory and reflection constitute another future research direction. A less explored reason for retrieving personal memories is to reflect and learn from them (Boud et al., 1985). Reflection on experience has been long advocated by Schön (1983), with the aim of improving action and professional practice. However, memory technology and in particular lifelogging tools have the potential to define new research directions into exploring reflection about personal experience (Sas & Dix, 2011). One way to support this is through the narratives that people build around significant events. From this perspective, memory research may focus on how individual memories can be integrated in coherent narratives from which sensemaking can emerge. Efforts to support people developing narratives have led to novel narrative-based interfaces which use stories as an organizing principle for tagging media (Tomas et al., 2010).

Emotional memory

A related research direction is exploring emotion regulation and memory technologies. Emotions can have a powerful impact on memory (Neisser, 1982). While the role of emotions in autobiographical memory has been extensively explored in the cognitive sciences, HCI research has benefited less from such findings. However, the advent of affective wearable sensors that can be integrated with lifelogging technologies or social media use, creating the possibility for exciting new applications.

We see a huge opportunity for Designing for Personal Memories, where current work has barely scratched the surface of what is possible.

Notes

Background. This special issue is the culmination of multiple community activities. It began with three successful workshops: the CHI 2006 workshop “Designing for collective remembering” organized by Sas and Dix, the HCI 2007 workshop “Supporting Human Memory with Interactive Systems” by Lalanne and Hoven, and the CHI 2009 workshop “Designing for reflection on experience” organized by Sas and Dix. These workshops indicated the field was mature enough to warrant a special issue. This special issue could not have succeeded without the authors' contributions and extended efforts to improve them.

Acknowledgments. We thank Thomas Moran and Patricia Sheehan for giving us the opportunity and for being involved in both content and process in the making of this HCI special issue.