Lifelogging as a rehabilitation tool in patients with amnesia: A narrative literature review on the effect of lifelogging on memory loss

ABSTRACT

Visual lifelogging is the procedure that enables individuals to visually record daily life activities by means of small wearable cameras, which can be worn around the neck or on the clothing. Lifelogging devices automatically take pictures or videos after pre-set time intervals or after dynamic input changes. Although some studies have shown effectivity of reviewing lifelogging images in brain-damaged individuals with forms of amnesia as a rehabilitation tool, systematic endeavours to overview this literature is yet missing. The aim of this narrative literature review was to review all the available case-studies, experimental studies and group studies in brain-damaged individuals applying lifelogging devices in a clinical context. The included studies showed efficacy for both subjective and objective measures of memory. In mild to severe amnesia, reviewing images recorded by the lifelogging device was beneficial to subjective and objective measures of memory. Lifelogging is demonstrated to have a great potential in helping people who are suffering from memory loss. It can offer an excellent alternative to currently more frequently used memory rehabilitation techniques and can be applied more in clinical settings.

KEYWORDS:

• Lifelogging
• amnesia
• Alzheimer’s disease
• Sensecam
• memory recollection
• neuropsychological rehabilitation

Introduction

With a total of over 50 million patients, and about 10 million new cases every year, memory loss is one of the leading causes of chronic disability worldwide (WHO, 2019). Many brain diseases, including Alzheimer’s disease, and other forms of dementia can cause severe memory loss. Apart from this, many forms of Traumatic Brain Injury can result in amnesia. Furthermore, also healthy aging can lead to mild memory problems. In particular episodic memory appears at high risk to be affected by many sorts of brain deterioration. Episodic memories include memories of one’s personal past (Roediger & Marsh, 2003) and can include spatial information or locations, associated emotions, and further contextual information about specific events. In combination with semantic memories, episodic memories, form our explicit memory (Hampton & Schwartz, 2004). According to Tulving (1993), episodic memory recollection is based on three main components, namely: mental time travelling – which can be explained as a subjective sense of time -, autonoetic awareness, allowing individuals to be aware of the “self” in a memory; and a connection of memories to an individual’s “self.” Recollection of memory, however, doesn’t always work. Amnesic patients often have a hard time remembering events in the (recent) past, but healthy people also experience “normal” forgetting. The ability to recall a certain memory can be increased when repetition of this memory takes place (Hintzman, 1976; Xue et al., 2010). The repetition increases the strength of a memory in the brain by increasing the neural activity in certain brain regions (Xue et al., 2010). Typical brain areas associated with episodic memory include the medial temporal lobe, the hippocampal formation and the prefrontal lobes (Yonelinas et al., 2005).

Given the importance of memory functions for both daily life activities and sense of identity and personal self, the attention for rehabilitation techniques to deal with memory loss has substantially increased in the last decades. Memory rehabilitation techniques can be divided into internal and external memory aid techniques and aim to rehabilitate memory. Moderately successful examples of memory rehabilitation techniques have mainly focused on these internal rehabilitation strategies, such as cue-based learning and errorless learning (Kessels & Haan, 2003; Middleton & Schwartz, 2012). Such methods, although proven to be effective, are mainly focused on procedural memory, i.e., skill learning. For the domain of episodic memory, effective rehabilitation techniques are sparse and limited successful. A promising exemption might be offered by so called “lifelogging devices.” Lifelogging can be described as a process by which individuals are able to create and elaborate, an external record of daily life activities (Dodge & Kitchin, 2007), that they as well as others can review at a later moment.

Typically, this record is visual. It should be noted though that lifelogging is not strictly limited to visual recordings of one’s life. The definition of the procedure of lifelogging can also include “recording” one’s life by verbal or auditory means (Gurrin et al., 2014). Commonly used examples of such applications of lifelogging are a diary and an audio recorder, but also sharing parts of one’s life via social media could count as a tool for logging one’s life. In this paper, however, we focus on visual lifelogging. Visual lifelogging can be seen as the most encompassing form of lifelogging, as it objectively captures activities in people’s lives in the most complete way. Any form of a camera that captures “lifelogging footage” can be seen as a lifelogging device. In practice however, we see that lifelogging devices are, almost without exceptions, first-person point-of-view cameras (Gurrin et al., 2014). The literature search for this review paper only found studies using first-person point-of-view cameras as no other studies met the inclusion and exclusion criteria. The use of first-person point-of-view cameras is further elaborated upon in the discussion section of this paper.

Lifelogging devices may be successful for memory rehabilitation because the recollection of certain events may be facilitated by the fact that reviewing the lifelogging images offers effective retrieval cues for the target memories. Even when these target memories need to be reactivated some time after the reviewing, individuals still can profit from these preceding cues-based retrievals. Furthermore, when using external memory aid, such as a camera, a patient is notified of his or her memory deficiencies and therefore tries harder to remember things. This also is beneficial for the effectiveness of the device (Block & Morwitz, 1999).

In the current paper, a number of visual lifelogging devices are reviewed. Wearable cameras often function as lifelogging devices to help memory loss patients (Doherty et al., 2012; Woodberry et al., 2015). Especially to create lifelogging footage, several devices have been developed. One of the most frequently applied devices, is “Sensecam,” which is one of the first lifelogging devices which has been used in scientific research (Dubourg et al., 2016; Doherty et al., 2011; Hodges et al., 2011). Two other wearable cameras that are mentioned in this review, are the “Autographer camera” (Selwood et al., 2020) and the “Narrative clip” (Dassing et al., 2020). These devices are small wearable cameras, which can be worn around the neck or on the clothing of a participant. The devices automatically take pictures after pre-set time intervals. The recording of these images is something that happens effortlessly for patients and they do not notice it. In some cases, a wearable camera can be programmed to take sensor-based images, rather than time-based images (Finley et al., 2011). The images, after data recollection, can easily be downloaded onto a computer to be reviewed by patients (Hodges et al., 2011). For experiments, researchers can select whether patients review pictures in random temporal order, or in chronological temporal order, and often a selection of images is made before the review (Mair et al., 2017). The selection of images, usually relies on quality and content of the images.

There seems to be a general consensus in the literature about the effectivity of lifelogging devices. However, to our knowledge there is no complete overview yet of studies that address this claim. In this light the aim of the current paper is to give a review of all relevant studies on lifelogging and memory rehabilitation and to discuss its success. Furthermore, we offer a further analysis of the factors that determine the effectiveness of lifelogging.

Methods

For the following part of this paper, a systematic literature search was performed according to the PRISMA guidelines, using Scopus, Web of Science, Pubmed and Medline. The following MeSH term was used: ((Memory OR (Memory loss) OR (Memory rehabilitation) OR Amnesia OR Recollection) AND (Lifelogging OR (wearable camera))). For selecting relevant articles for this review, a number of inclusion criteria were applied:

Inclusion criteria:

1. The article describes a scientific study using an experimental setup

2. The study uses a wearable camera for the purpose of lifelogging.

3. For the experiment wearable camera images are reviewed for the purpose of memory rehabilitation

4. The language used is English

Exclusion criteria:

1. Article is a review article

The initial search on Scopus, Web of Science, Pubmed and Medline, yielded about 507 results (October/ 2020). Of the articles found with this search, about 420 were excluded based on the inclusion criteria, after a reading the titles and removing duplicates. An additional 48 papers were excluded after reading through the abstracts, as they did not comply with the predetermined inclusion criteria. A list of 36 studies was compiled. After reading through the methods and results of these studies, an additional number of 18 studies were excluded, as they did not meet all inclusion criteria. The total number of articles included in this review is 18 (see Figure 1). The main reason for the exclusion of the 18 articles that did make it to the last 36, is that the experiments as described in the articles, did not review the Sensecam images before the testing, to later conduct a memory recall or recognition test.

Figure 1. Flowchart literature search.

After collecting the studies, an assessment was made of the number of moments the lifelogging devices have been deployed per participant in the studies. A greater number of recorded and reviewed moments provides more opportunities to examine the impact of use of the device. For each study therefore the number of moments where the device was used to make recordings were categorized into Low (1-2 times); Medium (3-5 times); High (more than 5 times).

Results

Case studies

Seven of the included studies in this systematic review (study 1-7) were case studies and SCEDs conducted in brain-damaged patients, causing them to experience mild to more severe amnesic symptoms. Berry included a single amnesic patient, a 63-year-old librarian suffering from limbic encephalitis causing amnesia in two studies (Berry et al., 2007; 2009). In both studies the patient did benefit from the use of a Sensecam device for both recollection and recognition of events that happened during the application of the device. One patient with Korsakoff’s syndrome and one patient with amnesia as the consequence of brain trauma, were discussed in two other SCEDs (Brindley et al., 2011; Svanberg & Evans, 2014). Specifically, the Korsakoff’s syndrome patient decided to stop wearing the device after more than a week of training, despite subjective memory benefits (Svanberg & Evans, 2014). In Brindley’s case, a 21-year-old patient with multiple neurological conditions did benefit regarding memory recall based on the use of the Sensecam. Also, the studies on a patient with pre-existing right hemisphere brain damage (Loveday & Conway, 2011) and one with mild cognitive impairment (Browne et al., 2011), led to better recall of events with the sense cam. Finally, there was one child with neurological impairments, leading to amnesia (Pauly-Takacs et al., 2011), that did benefit from the use of the Sensecam.

Most of the case studies and SCEDs are performed over a longer time and have relatively similar procedures. Berry et al. (2007) and Browne et al. (2011) performed some of the first studies using a wearable camera for lifelogging. For both of their studies, the researchers asked a patient to wear a Sensecam during memorable events and keep a diary of these events. Both studies asked a patient to perform a memory test and review either Sensecam images of an event or review a diary. After the memory test, the answers were converted into a score. These scores were based on the key points of the event as determined by the patient’s spouse (eg. 6 out of 10 points remembered correctly results in 60% score). Berry et al. (2007) and Browne et al. (2011), both studies found an improvement of memory scores in the Sensecam condition, which scored better than the diary condition. During a follow up after three months, one of the two studies found that even after three months the recollection score was 76% in the Sensecam condition, when after only seven days, in the control condition the recollection score was as low as 2% (Berry et al., 2007). The second study also found long-term benefits of lifelogging. The positive effect of reviewing Sensecam image was also established after six months (Browne et al., 2011).

Berry et al. found similarly positive results in a second study, where they looked at recognition, rather than recollection (Berry et al., 2009). In this study, a patient performed a memory test in an fMRI scanner. During the test diary texts, Sensecam images recorded by the patient, Sensecam images recorded by someone else, during an event the patient attended to and novel Sensecam images of similar events were reviewed. The researchers found that reviewing Sensecam images improved recognition scores, which it was more successful in than diary reviews.

Studies by Loveday and Conway (2011), and Brindley et al. (2011) support the previous findings. Their studies showed improvement in recollection when a patient reviewed previously recorded lifelogging images compared to when images were not reviewed. Loveday and Conway (2011), found that in both a diary and Sensecam condition, a 47-year-old patient with anterograde amnesia, was able to recall more episodic details and better qualify his memory. When Brindley et al. (2011) compared the effect of reviewing Sensecam images to the effect of Automatic Thought Record sheets (which can be compared to a diary), they found the reviewing of the lifelogging data to be more successful for memory rehabilitation. Furthermore, Loveday and Conway (2011) also noticed an increase in self-references and Proustian moments in the lifelogging condition, compared to the diary condition (137 self-references in lifelogging-condition, 33 in diary-condition).

In another study Svanberg et al., observed a subjective improvement of memory (Svanberg & Evans, 2014). In their study, a 51-year-old patient with memory loss recorded one activity per day for five weeks and re-watched the Sensecam images on a daily basis. Although no objective memory scores were obtained in this study, the patient indicated that with the help of reviewing the images, her memory of what she did during the events that were recorded, had improved. Furthermore, the researchers showed that quality of life improved as well. Measures for depression and anxiety rates decreased as a result of the Sensecam intervention. Furthermore, the data showed an improvement in measures such as self-efficacy, mood, confidence and usefulness.

The last case study looked at a 13-year-old patient and took a slightly different approach (Pauly-Takacs et al., 2011). The patient was taken on a walk on a campus with his parents and researchers, while both the patient and one of the researchers wore a Sensecam. After the walk, during which the researchers told the patient facts about the campus and the things they walked past, the lifelogging images were reviewed, and a free recall took place. After two weeks, recognition was tested. The researchers observed that recognition scores were higher in the reviewed condition, compared to the non-reviewed condition. This was also the case when memory was tested after ten and fifteen weeks.

Clinical group studies

Three studies applied the lifelogging technology in patients diagnosed with Alzheimer’s dementia (study 8-10). The study by Crete-Nishihata et al. (2012) enrolled five patients. The researchers showed that the use of Sensecam improved the recollection of episodic memories. The second study, by Woodberry et al. (2015), enrolled six patients, and also showed that Sensecam was successful in improving Alzheimer’s patients’ memories. Silva, Pinho, et al. (2017), used a bigger group of patients (51) and concluded that using Sensecam is more effective for memory rehabilitation compared to a number of other techniques. The clinical study by Dassing et al. (2020) enrolled 17 patients diagnosed with chronic schizophrenia and concomitant memory disorders as well 15 healthy controls. The researchers employed a Narrative Clip for their study and concluded that using the device can improve memory of amnesic patients.

Crete-Nishihata et al. (2012) asked participants to go on a two to four-hour (cultural) event while wearing a Sensecam. After the event, there were five review sessions spread out over a number of weeks. The review session consisted of multiple stages of questioning regarding the event, followed by the reviewing of the Sensecam images. This setup was repeated several times during a three to four-month period, with at least one week of “break” between the end of the reviewing sessions of one event and the next event, during which no testing, recording or reviewing of images took place. The results showed that the review of lifelogging footage recorded by Sensecam improved recollection in patients.

The two other studies compared the effect of a diary and lifelogging as a memory aid (Woodberry et al., 2015; Silva, Pinho, et al., 2017). For the study by Woodberry et al. (2015), a small group of patients were asked to use a Sensecam and a diary for three-and-a-half months and record events that were noteworthy to them. Within two weeks after the event, a free recall session took place after which a review session took place based on the condition (SC review, diary review, no review). Recollection scores were awarded based on key points, as established by the patients’ spouse. This procedure of scoring memory is similar to most of the aforementioned case studies. Recollection tests took place again after one and three months respectively. The results showed that in the Sensecam review condition, more details were recalled compared to the diary condition, which did score better than the baseline.

The second study comparing a Sensecam and diary condition, was performed by Silva, Pinho, et al. (2017). However, in this experiment, a third cognitive memory rehabilitation technique was used. For the study, all participants used one of the memory aid techniques for a total of six weeks. Twice a week, review sessions of the events during the previous week and free recall sessions took place. The researchers found that in the Sensecam condition, patients scored highest on the recollection tests and recalled most details. These memory improvements were still visible after a period of six months, when another memory test had taken place. Silva et al. attributed the positive effect of Sensecam to the activation of brain regions which are responsible for autobiographical memory.

Dassing et al. (2020), for their study, enrolled 17 Schizophrenic patients and 15 healthy controls. All participants were asked to wear a Narrative clip for 4 days, at least 7 h per day. For their study, the participants distinguished a different testing condition for each day: absence of retrospective (AbsR), verbal retrospective (VerR), visual retrospective (VisR), visual retrospective+ event cueing (VisR + EC). One day without any retrospective aid, one day with verbal retrospective, a day with simple visual retrospective and a day with a visual retrospective in combination with event cueing. After the events, participants were asked to review their days, based on what condition they were in. Two weeks post-events, recollection and recognition tests were performed. The researchers found that the use of the Narrative Clip improved memory in patients and healthy controls, which did, as expected, score better overall compared to the patients. More internal detail was recollected and recognition scores were higher in VisR and VisR + EC condition than in the VerR condition and the AbsR condition.

Experimental studies in healthy individuals

The last category of studies in Table 1, included healthy participants (study 11–18). In these studies, healthy participants were selected to determine the effect of lifelogging on memory rehabilitation, which was objectively measured. Two of the experimental group studies focus on students as a target group for the experiments (Finley et al., 2011; Seamon et al., 2014). Finley et al. (2011) looked at 12 undergraduate students, who were asked to wear a Sensecam. The researchers found Sensecam to be a good tool for memory rehabilitation. Seamon et al. asked a bigger group, of 144 students to use the same device. Although in some cases Sensecam was found to be effective, overall, the researchers found Sensecam not to improve the memory of participants. The studies conducted by Mair et al. and Silva et al., compare younger adults to older adults (Mair et al., 2017; Mair et al., 2019; Silva et al., 2013). Mair et al., in their first study, compared the effects of reviewing Sensecam images in random temporal order vs forward temporal order. The latter was found to be more successful. In their second study, Mair et al. came to the same conclusion. Selwood et al. (2020), looked at a group of older adults, while comparing a normal digital camera to an Autographer camera. The researchers found that the use of the Autographer camera was most successful for memory rehabilitation.

Table 1. Summary of results of lifelogging studies.

Study number	Authors (year)	Study type	Sample	Study design	Results	Number of measure points	Conclusion + / 0*
1	Berry et al. (2007)	Case study	63-year-old, librarian with limbic encephalitis and memory loss	– SC vs diary condition during memorable events – Memory test after 1 day (scores are noted), after which SC or diary is reviewed – Subsequent memory test every 2 days, until 7 trials completed – After 3 months memory recall test	– In SC condition significant memory improvement compared to baseline (p < 0.001) – SC condition significantly higher memory scored than diary condition (p < 0.001) – In diary condition no significant improvement – After 3 months SC recollection was 76%	High	+
2	Berry et al. (2009)	Case study	66-year-old, librarian with limbic encephalitis and memory loss	– Use SC/ diary “memorable” events – Review SC images, review diary, or no review – Review every 2 days, for 2 weeks, – fMRI scan during testing – 4 conditions of images shown during testing: 1: reviewed SC 2: not reviewed SC 3: SC images of diary reviewed event 4: another person’s SC	– No behavioural and neural difference between condition 2 and 4 ◊ confirm severity memory loss – Images condition 1 were significantly more familiar than condition 2 (p < .001) – SC provides subjective and objective improvements of autobiographical memory – Associated with activation frontal and posterior cortical regions.	High	+
3	Brindley et al. (2011)	SCED	21-year-old male, who suffered a number of pre-existing neurological problems, and memory loss	– Automatic though record sheet (ATR) vs SC – use both techniques for 3 events – recollection test 7 days post event, 2 times per week, for 3 weeks – for each event, one review strategy: SC/ ATR/ no review – after review, recollection test	– Recall increased to 94% in SC condition – ATR recall decreased from 56% to 22% – Baseline decreased from 78% to 39%	Medium	+
4	Browne et al. (2011)	Case study	55-year-old, retired social worker, MCI	– SC vs written diary condition compared as memory rehabilitation techniques – Use SC/ diary during event – Recall sessions: 1 cueing question, 1. & 2 days, & 2–week post event – After recall review of SC images or diary and second recall test – After 1-, 3- & 6-months long term recollection tests	– 2-week recall was significantly higher in SC condition (p < 0.001) – recall in SC condition: 64%–68%, vs 30%–50% for diary condition and 38% for baseline condition – 6-month recall was also best with SC: 41% recollection vs 20% in diary condition	High	+
5	Loveday and Conway (2011)	Case study	47-year-old patient with damage to right brain hemisphere and suffers from amnesia	– Diary vs SC condition – Use diary and SC for 1 event per day for 4 weeks – Free recall session every weekend (pre-cue), followed by review of SC or diary – Patient was asked to rate SC and diary on vividness during review – After review, same recall procedure (post-cue)	– Vividness ratings went down over time – No significant difference vividness rating between SC and diary – SC condition (pre-cue) improved significantly most on recall of episodic detail and qualified memory (p < 0.05) – SC condition (post-cue) improved significantly most on recall episodic detail & qualified memory (p < 0.001) – Most self-references in SC condition	High	+
6	Pauly-Takacs et al. (2011)	Case study	13-year-old boy with anterograde amnesia due to multiple brain disorders	– Walk at 4 locations, accompanied by experimenter, both wearing SC – At 2 locations patient’s SC was off – Post event, SC images reviewed 3 times a week for 2 weeks – 2, 10 and 15 weeks after the walk, recognition of SC images was tested – recognition tested for own SC images, reviewed, own SC images not reviewed and experimenters SC images	– Recognition scores were significantly above change at short term test for own reviewed SC as well as for own not reviewed SC (both p = 0.0009) – Recognition scores were significantly above change after 10 weeks, for own reviewed SC (p = 0.009) but not for own not reviewed SC (p = 0.205), as well as after 15 weeks: own reviewed SC (p = 0.009) own not reviewed SC (p = 0.117) – Patient understood difference between own SC and experimenters SC – Recognition often based on familiarity	Medium	+
7	Svanberg and Evans (2014)	SCED	51-year-old woman with memory loss as a result Korsakoff	– Patient recorded 1 activity per week, total of 5 weeks – Review SC images on daily basis until new event – No objective memory test, but questionnaire about subjective memory improvement.	– Patient quit study before finished – Patient indicated subjective improvement of memory when SC images were reviewed	High	+
8	Crete-Nishihata et al. (2012)	Clinical group study	5 patients with AD or MCI	– Use SC at 2–4 hr events at novel location – 5 review sessions in 2.5 weeks – Review sessions consist of free recall, general probed recall, general probed questioning and SC review or no SC review –(repeated for 3–4 months, with 1–2-week break after each event+ review)	– Recollection improved when patients had reviewed SC images (p < 0.05)	High	+
9	Woodberry et al. (2015)	Clinical group study	6 AD patients	– SC vs diary – Patients used SC for 3.5 months, record noteworthy events – After any recorded event, 6 review sessions, every 2 days – Free recall, followed by review SC or diary – Another recollection test after review and after 1 and 3 months	– Recollection episodic details highest in SC condition (p = 0.032) – After 3 months SC recollection also highest	High	+
10	Silva, Pinho, et al. (2017) and Silva, Salome Pinho, et al. (2017)	Clinical group study	51 AD patients	– SC vs diary vs alternative cognitive memory rehabilitation technique – Use one technique for a number of weeks – Recollection test performed after review per condition	– Sensecam was proved to be the most effective strategy for memory rehabilitation (p < 0.001) – Effect on autobiographical, semantic and episodic memory established	High	+
11	Dassing et al. (2020)	Clinical group study	17 (female) AD patients, 15 (female) control participants	– Participants used NC for 4 days, 7 hrs per day – 4 conditions (absent of retrospective (AbsR), verbal retrospective (VerR), visual retrospective (VisR), visual retrospective+ event cueing(VisR + EC)) – Memory test 2 weeks after event (cued recall and recognition)	– More internal detail were recollected in VisR + EC than in VerR (Pr (VirSR + EC > VerR) = 0.99) – In cued recall test higher memory details in VisR than VerR (Pr (VisR > VerR) = 0.95) – Recognition scored higher in VisR + EC than in AbsR (Pr (VisR + EC > AbsR) = 0.99) and higher in VisR + EC than in VerR (PR (VisR + EC > VerR) = 0.95)	High	+
12	Sellen et al. (2007)	Experimental study	19 undergraduate students	– 2 days, participants wore SC for 8 hrs – Participants manually had to take photos, or automatic (active vs passive) – Memory test 3 and 10 days after recording – Memory test: free recall, review images, final recall, followed by recognition test, to distinguish “own” images from “other’s” images	– Higher event recollection in SC condition compared to control after image review (p < 0.03) – In the passive condition recollection scores were higher than active condition (p < 0.03)	High	+
13	Finley et al. (2011)	Experimental study	12 undergraduate students	– Time-triggered SC vs Sensor-triggered SC – Use SC for 4 days – Recognition test: participants asked about their day while review SC – 2 out of 4 days SC images reviewed, the other 2 days no review – Recognition& picture cued recall test 1, 3 & 8 weeks after SC recordings	– No significant difference between time-triggered and sensor-triggered SC – Significant effect of condition (p < 0.05) – Recognition scores higher on review days (M = 3.55, SD = 0.69) compared to no-review days (M = 3.25, SD = 0.80) – Picture-cued recall scores higher and review days (M = 50.55, SD = 20.85) compared to no-review days (M = 45.51, SD = 18.49)	High	+
14	Seamon et al. (2014)	Experimental study	144 university Students	– SC vs diary vs review without external aid – Recordings during walk on campus – 6 different groups, divided over SC-, diary- and control condition (2 per condition) – Per condition 1 group of participants went on walk alone, the other group accompanied by experimenter – After walk, review of event – 1-week post event another review session and memory test	– Participants accompanied by experimenter scored significantly highest in free recall (p < 0.001) – No significant difference between memory aid conditions found	Low	0
15	Silva et al. (2013)	Experimental study	29 participants, 15 young adults, 14 old adults	SC vs diary – Use SC for 3 days straight, followed by use of diary for 3 days – Review sessions followed by neuropsychological assessment of memory	– Higher performance on recollection test in SC condition (p < 0.05) – Similar results for both groups (young and old participants)	High	+
16	Mair et al. (2017)	Experimental study	21 young participants, 21 old participants	Diary vs SC – Record 15 typical events in 5 days – 2-weeks post event free recall test – Review after test, either SC in forward temporal order, or random order, or review diary or no review – After and during review, second recollection test	– Review SC images significantly improve episodic memory, compared to diary and baseline (p < 0.0005) – Review SC in forward temporal order was more successful than random temporal order (p = 0.03) – No significant age effect	High	+
17	Mair et al. (2019)	Experimental study	Experiment I: 18 young adults and 25 older adults Experiment II: 17 young adults and 19 older adults	Experiment I: – SC during staged events in room, multiple participants present (3 rooms) – Perform difficult cognitive task – 14 days post event, recall session – Asked about experience in rooms, 3 conditions: no SC shown, SC shown in forward temporal order, SC shown in random order – 2 questionnaires, 1 pre and 1 post review SC Experiment II: – encoding sessions same as 1st experiments, but 2 rooms – 14 days post event, recall session – Participants asked to remember as many details about the event as possible – Review SC: SC shown for 1st room (forward temporal order), not for 2nd room	Experiment I: – Younger participants scored higher than older participants (p = 0.006) – Memory scores in SC condition in forward temporal order higher than control (p < 0.005) – Memory scores in SC condition in random temporal order higher than in control (p = 0.002) – No significant difference between random temporal order and forward temporal order Experiment II: – Younger participants score higher than older participants (p = 0.006) – Memory scores in SC condition in higher than in control (p = .001)	Low	+
18	Selwood et al. (2020)	Experimental study	51 older adults	Use AC vs DC* (with audio recorder) during visit to predetermined site (45–60 min) – Three photo viewing conditions (immersive AC, desktop AC, DC) – 4 stages (baseline testing, initial site visit, review 1, review 2) – Review 1 (1 day after visit): free recall, followed by photo review and recall –Review 2 (14 days after review 1): identical to review 1	– In recall, in AC conditions more details were recalled than in DC condition (p = 0.002), no significant difference between AC conditions – This was the case in photo review as well (p < 0.001) – In free recall in AC conditions also more details relating to activity, place, cognition and sensory perceptive information was recalled (p < 0.024), no significant difference between AC conditions – This was the case in photo review as well (p < 0.007)	Low	+

*Abbreviations: SCED = Single case experimental design; AD = Alzheimer’s dementia; MCI = Mild Cognitive Impairment; + = research concludes that lifelogging has a positive effect on memory rehabilitation; 0 = research does not come to the conclusion that lifelogging has a positive effect on memory rehabilitation; SC = Sensecam; AC = Autographer camera; DC = Digital camera; NC = Narrative clip.

Finley et al. (2011), asked 12 undergraduate students to wear a Sensecam, which was programmed to take either time-triggered or sensor-based pictures, for four full days. After the recordings, a review session took place at respectively one, three- and eight-weeks post-events, during which recognition was tested. For two of the testing days, participants reviewed their Sensecam images before the test, whereas for the other two days, no review of images took place. No difference between sensor-triggered and time-triggered image-taking was found. However, the results did show general improvement of recognition scores when Sensecam images had been reviewed, compared to when those had not been reviewed.

Seamon et al. (2014) studied a large group of students. While using Sensecam for memory support, the focus of their experiment was on social aspects, a relatively new element. Participants were divided into two groups; one group, where students went on a walk on their own, and one group where one of the experimenters joined the participants for the walk. Another division of three subgroups was made. One group kept a diary on the event, one group wore a Sensecam, and one group did not have any memory aid. Although the researchers found that the participants that went on a walk with an experimenter, scored better in a memory test, no significant differences were found between any of the sub-conditions.

One of the first studies comparing the effect of lifelogging between younger and older adults, was performed by Silva et al. (2013). In a group of 25 participants, there were 15 younger adults and 14 older adults. All participants were asked to wear a Sensecam for three days and were also asked to keep a diary. Subsequently, review sessions took place during which participants either reviewed Sensecam images or their diary notes and recollection was tested. The results indicated that reviewing Sensecam images was more successful for the support of memory than reviewing diary notes. Besides, in all conditions, younger participants were able to recall more detail than older participants.

Mair et al. (2017) also looked at a group of younger and older adults. For the study, recordings of Sensecam images on a total of 15 occasions took place, followed by review sessions of those images, based on three conditions. The images were either not reviewed, reviewed in forward temporal order or reviewed in random temporal order. After the review sessions, memory tests took place. The results showed that reviewing Sensecam images, in general, was beneficial for participants’ recollection of events, even more so, if the images were reviewed in forward temporal order. For the age-wise comparison, no significant effect on age was found.

In a follow up study, Mair et al. (2019) conducted two different experiments. In the first experiment, participants performed a difficult cognitive task in three different rooms. While performing the tasks, the participants were wearing a Sensecam. Fourteen days after the task, recollection tests, during which the participants were asked about their experiences, took place. After the recollection test, participants were either shown no Sensecam images, Sensecam images in random temporal order, or in forward temporal order. The results showed that younger participants recalled more episodic details than older participants. Furthermore, when participants had reviewed Sensecam image, they scored better than when this was not the case. No difference between reviewing the images in forward and random temporal order was found. The second experiment (Mair et al., 2019) was very similar to the first experiment. However, now there were only two conditions: a Sensecam review condition and a non-review condition. The results showed that in the review condition, recollection scores were higher than in the non-review condition. Besides, in this experiment, younger participants scored higher than older participants.

Selwood et al. (2020), looked at a group of 51 older adults, with an average age of 78 years old. For the study, a normal digital camera and an Autographer camera were used. The full study consisted of a total of four sessions: a baseline recollection test session, a visit to a pre-determined site, a review session one day after the event, and a review session two weeks after the event. The review sessions consisted of free recall and a review of the event based on one of three conditions, during which participants were also asked to recall additional information about the event. One “immersive condition,” in which participants could see their location at the time of the visit, while viewing the Autographer images in landscape position, and a simple review condition for both the Autographer camera and digital camera, during which participants saw their images on a normal desktop screen. The results showed no significant difference between the two Autographer review conditions. However, during free recall participants had higher recollection scores in both Autographer conditions than in the Digital camera condition. This was the case during the photo viewing recall as well.

Discussion

Lifelogging involves capturing visual information from one’s environment, typically from a first-person point of view. Reviewing the obtained recordings, can help rehabilitate memory. The literature thus far has lacked an encompassing overview of the relevant studies using (different kinds of) lifelogging devices. The aim of this review was to give an overview of all studies using lifelogging devices and potential for memory rehabilitation. Moreover, we aimed to further reflect on how lifelogging causes its memory effects and which factors can modulate these effects.

In all clinical studies enrolled in this review, lifelogging was found to have beneficial effects on the subjective and objective measurements of memory. All case studies on diverse brain-damaged samples (Berry et al., 2007, 2009; Brindley et al., 2011; Browne et al., 2011; Loveday & Conway, 2011; Pauly-Takacs et al., 2011; Svanberg & Evans, 2014), and also all group studies on patients with Alzheimer’s dementia (Crete-Nishihata et al., 2012; Dassing et al., 2020; Silva, Pinho, et al., 2017; Woodberry et al., 2015) concluded that lifelogging is beneficial in rehabilitation of memory. In healthy samples, this finding was not always supported. The study by Seamon et al. (2014), found no significant difference between a Sensecam-, diary- and control condition. Also compared to a baseline, the Sensecam was not found to be more effective for memory rehabilitation. Notably these findings contrast with those in the other studies. An explanation for the different findings, as mentioned before, could be the aberrant aim of the study.

Overall, considering the relevant literature discussed here, we can conclude that lifelogging and reviewing the footage of a lifelogging device, is beneficial for memory rehabilitation. Objective improvements of memory scores, which can be observed throughout the discussed literature, support this claim. Apart from these objective measures, patients also gave the feedback that they enjoyed using Sensecam (Berry et al., 2007; Loveday & Conway, 2011; Svanberg & Evans, 2014; Woodberry et al., 2015). Patients indicated that this was not only the case because of memory improvements, but also because it was nice to look through the pictures and recall what happened during the events.

In line with the foregoing, the case studies included in this review article strongly suggests that lifelogging can successfully aid memory rehabilitation in patients with amnesia of various neurological aetiologies. In all case studies, the results consistently show that patients perform better on memory tests when they use a lifelogging device. Moreover, there is a large variety of patients with differing brain-impairments. This makes it hard to generalize the results. Nonetheless, the conclusions of these studies are very relevant because they show that lifelogging devices do not solely benefit only one group of specific patients, but a multitude of different patients can benefit from it. This is important for the applicability of lifelogging devices. It shows that the devices can be used for a big group of patients with neurological disorders. Besides, the statistical significance of the individual studies is sufficiently great. Furthermore, the case studies give an important insight in how patients experience the use of a wearable camera. In multiple case studies, the patients indicated subjective benefits of the use of a wearable camera (Berry et al., 2007; Loveday & Conway, 2011; Svanberg & Evans, 2014). Besides, the case studies show that lifelogging devices are easy in use, as they do not require professional supervision.

Two of the seven case reports applied the Single Case Experimental Design (SCED) (Brindley et al., 2011; Svanberg & Evans, 2014). SCEDs provide researchers with a flexible alternative to group designs with large sample sizes and have more stringent criteria than a case study. We would therefore recommend more SCED publications on the use of wearable technology in specific populations, such as TBI, specific dementia’s and Korsakoff’s syndrome.

In the clinical group studies, a focus on Alzheimer’s disease patients becomes evident (Crete-Nishihata et al., 2012; Silva, Pinho, et al., 2017; Woodberry et al., 2015). This group of patients is not represented in the case studies. The inclusion of Alzheimer’s disease patients is relevant, because this group of patients often suffers from severe cognitive problems and serious amnesia; and Alzheimer’s disease is a main cause of dependence on intensive care. The use of assistive technology is limited in AD, making it particularly hard to assist AD patients in daily life activities and improving their quality of life. Incorporating lifelogging devices in the treatment of AD patients can therefore have great implications. The clinical group studies as described in the literature review, do have higher statistical power, and can, therefore, be seen as a more reliable source of information on the efficiency of lifelogging. They clearly subscribe the potential of life logging.

Reviewing of the experimental studies in healthy subjects we can conclude that lifelogging is successful in its goal in supporting memory (Finley et al., 2011; Mair et al., 2017, 2019; Sellen et al., 2007; Selwood et al., 2020; Silva et al., 2013). We may note here that the studies on healthy individuals have attractive characteristics: One is statistical power, which is obtained by the large groups used for the studies. The other one is that we can further determine that lifelogging also works in case of no or just subtle memory failures, which can also be experienced in healthy participants.

Explaining the success of lifelogging in general

In this section of the discussion, we consider a number of reasons for the success of lifelogging. One of the main reasons of its success lies within the fact that memories of the event where the lifelogging device is used, is “repeated” during the review sessions and therefore becomes more readily available. When a memory is “discussed” more often, or simply thought about more often, this memory becomes more available and is easier to access in general (English & Visser, 2014; Wang et al., 2012). Consistently reviewing lifelogging images, therefore, makes the memory of specific events more stable as a whole. Using a lifelogging device, but also for example a diary, for that reason, can help rehabilitating memories. The reviewed studies in which a lifelogging condition is compared to a diary condition, however, show a higher success rate in a lifelogging condition (Berry et al., 2007, 2009; Browne et al., 2011; Loveday & Conway, 2011; Mair et al., 2017; Seamon et al., 2014; Silva et al., 2013; Silva, Pinho, et al., 2017; Woodberry et al., 2015).

For the success of lifelogging, reviewing first-person perspective images, according to Conway (2005) , is essential. This can also be seen as an explanation as to why all of the studies described in this review use wearable cameras. Furthermore, according to Klein et al. (2004), reviewing images from a first-person perspective, gives people the sensation that indeed, the images that are reviewed, belong to one’s personal past. This could enhance the strength of a memory (Prebble et al., 2013). In comparison to reviewing images from a third-person perspective, the ego-centric pictures could possibly increase the vividness of a memory, which is essential for remembering details of events (Robinson & Swanson, 1993). Marcotti and St. Jacques (2018) recently confirmed the idea that memory retrieval is more successful when using a first-person perspective. The researchers showed that the accuracy and vividness of memories is higher when they are reviewed from a first-person perspective compared to a third-person perspective. Furthermore, according to Conway, the networks in the brain that are involved in the retrieval of autobiographical memories, are more active, when first-person perspective images are being reviewed (Conway, 2005). Besides, specific brain regions that are active when reviewing lifelogging images also play an essential role in the storing of autobiographical memories (Mickes et al., 2009; St. Jacques et al., 2011). Moreover, a process involving the lateralization of episodic memory might be involved and attribute to the improved recollection when someone reviews lifelogging images (Habib et al., 2003; McDermott et al., 1999). Future work might further investigate which aspects of episodic memory can be enhanced by third-person review perspective. One of these could involve allocentric transformations and recoding.

Additionally, in line with the findings related to the neural correlates of lifelogging, there can be another reason for its success. When someone reviews a diary, for example, thoughts can easily drift off to other memories, meaning that there is a possibility that there is only limited focus on the target event (O’Callaghan et al., 2019). When reviewing lifelogging images, there is a large number of images of a single target event. As such it is way harder to drift off, because there are many more details in a picture of a target event that one can focus on as Browne et al. (2011) concluded.

A further reason for the success of wearable cameras is the fact that the devices are very user friendly. The only thing that patients have to do is put the camera around the neck and let it take the pictures automatically. Next, the images are very easily downloaded to a computer, where a patient can easily look again at the pictures. The fact that a patient is using a wearable camera might also have another benefit. The use of a lifelogging device raises the awareness of a patient that the event which is happening at that moment is important and memorable. As mentioned in the introduction, this can also be beneficial for the usefulness of the device. The awareness and accessibility of using a wearable camera is also a reason for the improvement of the quality of life for patients using it. Browne et al. (2011), in their study, focussed on whether lifelogging can improve patients’ lives and found lifelogging to have a significant effect on this. They mention that reviewing the images was not only helpful because it helped their patient remember more of her personal past, but also because reviewing images of events that a patient wants to remember can be enjoyable (Browne et al., 2011).

Outcome measures and methodologies

Although most studies presented in this narrative review used highly similar methods, certain differences in methodology and outcome measures can be noticed. A first noticeable point of difference concerns temporal dimensions such as how long and frequently the lifelogging devices were used, the length of time after which reviews took place and factors such as reviewing images in forward versus random temporal order.

The studies included in this review employed lifelogging devices for single events, multiple unrelated events as well as for a variable a number of hours for a variable number of days. Despite this variation in the length of the usage of the devices and consequently in the number of review moments, there is a great consistency in other procedural dimensions. In all but one (Finley et al. (2011)) of the studies, the lifelogging camera automatically took pictures after a pre-set time interval. In the exceptional study by Finley et al. (2011), a Sensecam was programmed to also take sensor-based pictures, which was based on the amount of light coming into the camera. Furthermore, the studies as described, consistently based the images that were selected for participant review on quality of the images.

Often, the lifelogging devices were used during multiple (predetermined) occasions, such as a memorable event as determined by participants or the researchers (Berry et al., 2007, 2009; Brindley et al., 2011; Browne et al., 2011; Loveday & Conway, 2011; Mair et al., 2017; Silva, Pinho, et al., 2017; Svanberg & Evans, 2014; Woodberry et al., 2015). These studies show that especially during “memorable” events, the use of a lifelogging intervention can be successful in helping participants remember what happened during those events. The studies by Dassing et al. (2020); Sellen et al. (2007); Finley et al. (2011); Silva et al. (2013), employed a slightly different approach. These (group) studies asked participants to operate a lifelogging device for a certain time (a number of hours) for a couple of days. The positive results from these studies again show that lifelogging can also have a more general effect on memory rehabilitation and can support memory of day-to-day events. Overall, these results clearly underscore the effectiveness of the memory rehabilitation tool.

A second domain of difference is the type of memory measure obtained to assess lifelogging effects. All but one of the studies described in this review article use objective memory measures. As an exception the study by Svanberg and Evans (2014) did only draw conclusions on basis of subjective measures. The researchers asked the patient whether she noticed memory improvements. Furthermore, they also focussed on other “quality of life” measures. Although these subjective results obtained should be more carefully considered, they are still very relevant as they give an insight on how the practical use of a lifelogging device in general, is perceived and evaluated. Furthermore, the Svanberg and Evans conclusions are very much in line with the other studies included in this review article.

While the other studies do employ more objective memory testing measures, we can also observe several differences here. For example, Berry et al. (2007, 2009) used a procedure, where memory scores of participants were based on key points as determined by the patient’s spouse. When the patient recalled for example 5 out of 10 key points, the patient was awarded a score of 50%. As such, this study gives an idea of how well memory work for the core contents of events. Silva et al. (2013), Silva, Pinho, et al. (2017) included a more rigorous memory testing method, entailing more detailed measurements. Despite these differences we wish to emphasize that reviewed studies all clearly point towards improved memory in their participants as a result of the interventions.

When integrating lifelogging devices in non-clinical settings, it is also beneficial to look at how and when a device takes pictures. In almost all studies using lifelogging devices, the device takes pictures on a time basis. The research by Mair et al. (2017), however, also focusses on a Sensecam which is triggered to take so-called sensor-based images instead of only temporal-based images. Their research shows that a sensor-based version of lifelogging is actually more effective than the traditional temporal-based version. Integrating a function that determines when it is the best moment to take pictures, therefore has much potential as well.

One limitation that needs to be taken into account, however, is that the memory testing throughout the described studies is not standardized. Moreover, assessment of cognitive functioning by means of neuropsychological assessment is not the standardized outcome measure in many studies. Rather, the researchers look at data of specific memory scores based on memory tests about the events which are also depicted in the reviewed images.

Number of measurements

The number of moments the lifelogging devices were employed per study has been reviewed. This is a key feature of studies of lifelogging as a greater number of uses of the devices provides more opportunities to evaluate the impact of the device on memory recollection. In Table 1 we give an assessment of the number of moments the devices have been employed. Most studies use the device at least 5 times. Some of the studies ask participants to use the device throughout the day, such as the study by Sellen et al. (2007). Other studies employ the devices for a number of specific events such as study by Svanberg and Evans (2014). Because of the greater number of recording and reviewing moments, these studies are able to employ the lifelogging devices in more and more different settings, showing better what the effect of these devices is in practise.

Familiarity vs recollection

The literature search shows that most of the discussed papers are based on the active recollection of episodic details during the events that took place for the experiment. For a few studies, however, memory tests were based on recognition (Berry et al., 2009; Finley et al., 2011; Pauly-Takacs et al., 2011). In these studies, no clear attempt was made to separate recollection based recognition form mere recognition on basis of familiarity. Possibly, what could happen after reviewing images, is that a participant simply recognizes a scene because he or she has seen this particular scene so many times as an image during review sessions. The actual event itself and what happened before and after the event might have been forgotten. In this case, a participant would no longer be able to actively recollect what happened during the event, but only what he or she has seen multiple times on a picture. Some researchers have tried to control for this by asking specific and more “active” questions (Crete-Nishihata et al., 2012; Silva et al., 2013, Silva, Pinho, et al., 2017). Examples of such questions can be: “what do you remember about how you felt when this picture was taken?,” or “what did the experimenter talk about when this picture was taken?.” This offers a qualitative analysis of how and what people remember.

The future of lifelogging

Lifelogging can be seen as a successful memory rehabilitation tool. It has a significant effect on memory rehabilitation, but also it can improve a patient’s quality of life for several reasons. Integrating lifelogging devices of patients suffering from memory loss can, therefore, be very beneficial and has vast potential. In our digitalizing world, using cameras is becoming more and more normalized, and perhaps integrating a lifelogging function in, for example, mobile phones can be helpful to make lifelogging more readily available. One can already see that the concept of lifelogging is developing. Although the Google Clip and Narrative Clip have seldomly been used in research project, both devices could be relevant for future investigations into lifelogging as a rehabilitation tool for memory disorders. One way to implement lifelogging in the daily lives of patients could be using mobile phones, or simply continuing the development of the wearable cameras into even smaller and easier to use devices. One major concern that needs addressing when integrating lifelogging devices in people’s daily lives is privacy. A lifelogging device takes pictures and does not take into account the privacy of bystanders that might not want to be recorded. For those reasons, a framework to incorporate people’s privacy into a future form of lifelogging is essential.

In light of the overall positive pattern of results obtained by the visual life-logging devices it is quite surprising that most lifelogging projects have been discontinued and there seems to be no standardized clinical routines developed. Privacy concerns may have played a role (not every external person wants to be part of the movie made for the participant). Another reason for the failure of lifelogging to make it into the “real world,” might lie in the practical usability of the device. In none of the studies included in this article participants were allowed to operate the tool autonomously. A researcher was responsible for selecting the images for review, and spouses or other family members of patients were often involved in the procedure to allow proper use of the lifelogging devices. This raises the question whether the lifelogging cameras are too hard to use. However, many new cameras that can easily record images from a first-person perspective and are very easy to use are currently on the market (e.g., GoPro, DJI). Furthermore, the selection of images to be reviewed is based largely on the quality of the recordings, which could easily be determined by computer programming.

It seems as if there is little holding lifelogging back from being used in advanced settings. The reviewed studies clearly show its success and the issues which were a problem with older versions seem to be surmountable. The main issue probably will prove to be a lack of commercial funding for getting lifelogging interventions on the market.

In conclusion, in this paper for the first time an encompassing, systematic review of the existing studies on the effects of lifelogging to counter memory loss is provided. We show that lifelogging is successful in helping people who are suffering from both severe and mild memory loss. The evidence follows from case studies, clinical trials and trials with healthy participants. People with a large variety of brain-damage benefit from the use of the memory rehabilitation technique. Given the efficiency of the method, lifelogging has a great potential.

Disclosure statement

No potential conflict of interest was reported by the author(s).