Dealing with daily challenges in dementia (deal-id study): process evaluation of the experience sampling method intervention ‘Partner in Sight’ for spousal caregivers of people with dementia

ABSTRACT

Objectives: This study describes the process evaluation of the experience sampling method (ESM) intervention ‘Partner in Sight’ for spousal caregivers of people with dementia. The aim was to determine internal and external validity of the intervention and provide information for future implementation in clinical practice.

Method: Qualitative and quantitative data on sampling quality (recruitment and randomization, reach) and intervention quality (relevance and feasibility, adherence to protocol) were evaluated using descriptive statistics and conventional content analysis.

Results: The participation rate included 31.4%. Due to recruitment difficulties and time constraints the original goal to include 90 caregivers was not met. The intervention was largely performed according to protocol and well received by the participants. Overall, the ESM-derived feedback was considered supportive and increased participants’ awareness of their feelings and behavior. A large variance was found in the extent to which caregivers applied the feedback into their daily lives. The importance of the personal coach to provide face-to-face feedback and stimulate caregivers to implement new insights into their daily lives was emphasized. Suggestions for improvement were to reduce the time intensity of the program, to better tailor the program content to one's personal situation, and to improve the ESM device.

Conclusion: Although recruitment barriers were encountered, results indicate that future implementation of the ESM intervention ‘Partner in Sight’ is likely to be feasible in regular health care. If the intervention turns out to be (cost-) effective, a fine-tuned version of the program could be a valuable addition to the current health care system.

KEYWORDS:

• Process evaluation
• experience sampling method
• intervention
• dementia
• caregiver

View correction statement:

Correction

Introduction

The prevalence of people with dementia is increasing rapidly as a consequence of the global ageing of the population. Dementia is associated with intense need for care and has a huge economic and societal impact (Prince et al., 2015). The current labor force will not be capable of dealing with such an increased future demand. Therefore, the care for persons with dementia (PwD) will depend increasingly on informal caregivers in the upcoming years. Caring for a PwD puts one at risk of becoming overburdened and of developing psychological and physical symptoms (Brodaty & Donkin, 2009). Yet, caregivers differ in their emotional response when dealing with daily challenges of dementia. Caregiver resources, such as sense of competence and mastery, play an important role in reducing caregivers’ emotional reactivity to daily life stressors (van Knippenberg, 2017). Moreover, positive emotions have been found to positively impact feelings of competence to care for the PwD (van Knippenberg, de Vugt, Ponds, Myin-Germeys, & Verhey, 2016a). Existing psychosocial interventions mostly focus on the negative consequences of the caregiving process and do not account for individual differences among caregivers (Pinquart & Sörensen, 2006). Therefore, the experience sampling method (ESM) intervention ‘Partner in Sight’ for spousal caregivers of people with dementia was developed, aimed at empowerment of positive caregiver experiences, and tailored to the individual caregiver. The ESM is a structured diary method that can be used to self-monitor subjective experiences in daily life. The main advantages of the ESM are that it assesses experiences in-the-moment, resulting in less memory biases compared to traditional retrospective measures and it allows for exploration of temporal relationships between variables and revelation of detailed information on daily fluctuations in subjective experiences. This is of particular importance in caregivers of PwD, as caregiver experiences are likely to fluctuate over time in response to constantly changing care demands (van Knippenberg, 2017).

To evaluate the effectiveness of the intervention with respect to caregiver sense of competence, mastery, momentary positive and negative affect, and psychological complaints, a randomized controlled trial (RCT) was performed.

While RCTs are considered as the ‘gold standard’ to study the effectiveness of interventions on prespecified outcomes, in-depth information on the internal and external validity of the intervention is essential for the interpretation and generalizability of the results (Leontjevas, Gerritsen, Koopmans, Smalbrugge, & Vernooij-Dassen, 2012). A process evaluation can be used to explore the context, implementation, and receipt of an intervention. Key elements of a process evaluation are the evaluation of sampling quality (recruitment of participants and reach) and intervention quality (the extent to which the intervention was performed and perceived) (Leontjevas et al., 2012). Process data not only help to understand possible intervention effects, but also provide necessary information for replication studies and implementation of the intervention in health practice (Oakley, Strange, Bonell, Allen, & Stephenson, 2006).

Several frameworks for conducting a process evaluation have been proposed, covering numerous elements that can be used for different aims (Leontjevas et al., 2012). In this study, we apply a previously used model (Leontjevas et al., 2012) to evaluate process data about sampling and intervention quality of the intervention ‘Partner in Sight’ prior to the effect analyses.

Methods

Study design

The process evaluation is a descriptive mixed methods study in which both quantitative and qualitative data were gathered alongside an RCT with three treatment arms (van Knippenberg, de Vugt, Ponds, Myin-Germeys, & Verhey, 2016b). As part of the RCT, participants were randomly assigned to the intervention group (‘Partner in Sight’: ESM data collection with feedback); pseudo-intervention group (ESM data collection without feedback); or control group (care as usual). The pseudo-intervention group was included to examine whether the feedback added any extra value to the collection of ESM data on its own. A post-intervention assessment was conducted after the six-week intervention period and at two-month follow up. In the process evaluation we only focused on the participants who were allocated to the intervention group and followed the program ‘Partner in Sight’. Detailed information on the study design and the intervention are described elsewhere (van Knippenberg et al., 2016b). A brief description is presented below.

The Medical Ethics Committee of the Maastricht University Medical Center Plus approved this study (#143 040).

Intervention

The intervention program ‘Partner in Sight’ consists of ESM data collection for six consecutive weeks and three face-to-face sessions in which ESM-derived feedback is provided by a personal coach. ESM is a repeated self-assessment approach to assess subjective experiences and context in the flow of daily life (Delespaul, 1995; Hektner, Schmidt, & Csikszentmihalyi, 2007; Myin-Germeys et al., 2009). The ‘PsyMate’, a palmtop, was used to digitally collect momentary assessments and to provide visualized feedback on daily life situations that elicit positive emotions (Myin-Germeys, Birchwood, & Kwapil, 2011). The rationale of this positive focus in the intervention was that it facilitates a more positive interaction between the caregiver and the PwD and it increases positive emotions and well-being in both parties. Additionally, positive emotions enhance one's ability to cope with stressful situations and help regulate negative emotions (Boots, Wolfs, Verhey, Kempen, & de Vugt, 2015; Fredrickson, 1998; van Knippenberg, 2017).

The feasibility of the ‘PsyMate’ in caregivers of PwD has recently been demonstrated (van Knippenberg et al., 2016c). During the six-week intervention period, the ‘PsyMate’ was programmed to generate 10 alerts (sound and vibration) per day for three consecutive days per week (6 × 10 × 3 = 180 beeps in total). Alerts were emitted at random intervals between 7:30 AM and 10:30 PM. After each alert, participants were asked to digitally complete a brief questionnaire on the screen of the ‘PsyMate’, including current mood (e.g. positive and negative affect) as well as current context (e.g. social company, activities, location, and important events). Standardized sets of ESM items are not yet available (Ebner-Priemer UW & Trull, 2009). Therefore, the choice of items and development of the ESM questionnaire was made on the basis of information available from previous ESM studies (Oorschot, Lataster, & Thewissen, 2012), guidelines from ESM experts for designing an ESM study (Palmier‐Claus et al., 2011), and knowledge about the range of experiences that spousal caregivers of people with dementia could be expected to encounter. An initial version of the questionnaire was piloted with three health care professionals and three caregivers of people with dementia to ensure that no relevant domains have been missed. Modifications were made based on their comments (van Knippenberg et al., 2016c).

Every two weeks participants received ESM-derived feedback from a personal coach (psychologist) according to a standardized protocol. The standardized protocol was based on a previous study that used ESM-derived feedback on positive affect in persons with depression and it was adapted to the context of caregivers of PwD (Kramer et al., 2014).

Feedback was provided both graphically and verbally and contained information on experienced levels of positive affect and their relationship with daily activities and social interactions in daily life. Participants were encouraged to implement new insights into their daily lives. Before the start of the study, all coaches received training with detailed instructions on how to provide the feedback.

Process data

Process data were evaluated prior to the effect analyses. Data on sampling and intervention quality should be evaluated at an early stage to further fine-tune the effect analyses and to provide essential information about credibility and generalizability of the results (Leontjevas et al., 2012). Table 1 shows how the components sampling quality (recruitment and randomization, barriers and facilitators for recruitment, and reach) and intervention quality (relevance and feasibility, and adherence to protocol) were operationalized and measured in the current process evaluation.

Table 1. Components and subcomponents of process evaluation and ways of measurement.

Components and subcomponents	Operationalization	Measurement Da | Qi | Qli | Qp | Ili
Sampling quality
– Recruitment and randomization	Number of caregivers approached	X
	Number of caregivers randomized	X
	Reasons for refusal	X
– Barriers and facilitators for recruitment	Information from clinicians/dementia case managers regarding recruitment procedure	X
– Reach	Number of caregivers from different institutions	X
Intervention quality
– Relevance and feasibility	Experiences with intervention	X X
	(program content, program structure, advantages for participants)
	Experiences with ESM in general
	(ESM device, ESM content, ESM structure)	X X
– Performance according to protocol	Total duration of intervention	X
	Reasons for variance in intervention period	X
	Duration and structure of feedback sessions	X
	Compliance	X

Da = data recorded by researchers during intervention period, Qi = questionnaire completed by participant per intervention contact, Qli = questionnaire completed by participant during last intervention contact, Qp = questionnaire completed by participant during post-intervention assessment, Ili = semi-structured interview conducted by coach during last intervention contact.

The sampling quality was based on information recorded digitally by the researchers throughout the intervention period (Da). Information on barriers and facilitators for recruitment was provided by clinicians and dementia case managers during the recruitment process and stored digitally by the researchers (Da).

The intervention quality was based on data collected from the participants in the intervention group who completed the intervention (N = 20). Information on the relevance and feasibility of the intervention was collected during the last intervention contact through a questionnaire (Qli) and a semi-structured interview, conducted by the personal coach (Ili). General experiences with the ESM were gathered from a questionnaire completed by the participants and discussed with their coach, during each intervention contact (Qi). In addition, user experience with the ‘PsyMate’ was evaluated by means of a questionnaire during the post-intervention assessment (Qp). Multiple-choice items in the self-administered questionnaires were rated on 7-point Likert scales (1 = not at all to 7 = very much). Answers to open-ended questions and the semi-structured interview were categorized in order to identify relevant themes. Compliance with the ‘PsyMate’ was recorded electronically during the intervention period. Finally, protocol deviations with respect to total duration of the intervention, frequency and duration of the feedback sessions, and structure of the feedback sessions, were registered to evaluate adherence to protocol.

Data analysis

Quantitative data were analyzed by means of descriptive statistics (STATA version 12.1). Qualitative data were analyzed with a conventional content analysis, in which an open coding approach was applied to attain a codebook with categories derived from the data at hand without preconceived categories (Hsieh & Shannon, 2005). The emerging categories were merged into relevant themes by the authors RvK and CS. Three themes were identified with respect to the intervention ‘Partner in Sight’ (program content, program structure, advantages for participants) and three themes were identified with respect to the ESM procedure in general (ESM device, ESM content, ESM structure).

Results

Sampling quality and descriptives

Recruitment and randomization

Informal caregivers (N = 295) were recruited from memory clinics (Maastricht University Medical Center +, Zuyderland Medical Center), ambulatory mental health care institutions (Virenze-RIAGG Maastricht, Lionarons GGZ), dementia day care centers (Sevagram, NOVIzorg, Orbis Glana, Proteion, care farm Ransdalerveld), caregiver support services in the southern Netherlands, and via dementia case managers (‘Hulp bij Dementie’) and the Dutch Alzheimer Association. Of the 295 caregivers, 242 met the in- and exclusion criteria and were eligible to participate. If interested, caregivers (N = 172) received a detailed information letter. Informed consent (IC) was signed by 44.1% (76/172). Reasons for declining to receive the information letter or to sign the IC after more detailed information were: no need for support (N = 57), intervention considered as too burdensome (N = 53), too time-consuming (N = 24) or too confronting (N = 5), worries that the intervention causes agitation or suspicion in the care recipient (N = 10), feeling unfamiliar with technological devices (N = 9), care recipient almost institutionalized (N = 6), or restrictions to participate due to hearing loss (N = 2). The overall participation rate of eligible caregivers was 31.4% (76/242). The original aim to enroll a total of 90 participants was not accomplished due to recruitment difficulties and time constraints.

After the baseline assessment, 76 caregivers were randomly assigned to the intervention (‘Partner in Sight’: ESM including feedback) (N = 26), pseudo-intervention (ESM without feedback) (N = 24), or control group (N = 26). Allocation to the three groups was conducted by a researcher (RvK) who was not involved in the assessments. Randomization was performed using a computerized sequence generator for block randomization with variable sizes of three, six, and nine. An independent research assistant who was blinded to treatment allocation conducted the assessments at baseline, post-intervention, and at two-month follow-up. Research assistants blinded to allocation performed the assessments and recorded success of blinding. In total, 84.2% of the caregivers (N = 64) completed the post-intervention assessment. The number of dropouts included 5 in the intervention group, 3 in the pseudo-intervention group, and 2 in the control group. Blinding of the researchers for group allocation was intact for 37.5% (24/64), unsuccessful for 14.1% (9/64), and for 48.4% (31/64) a conjecture of allocation was reported. Figure 1 shows the participant flow throughout the study.

Figure 1. Study flowchart.

Barriers and facilitators for recruitment

Clinicians and dementia case managers involved in the recruitment of participants frequently mentioned that their caseload comprised lots of people with dementia that were living alone without a registered partner. Other recruitment barriers included concerns that the intervention would be too time-consuming and burdensome for caregivers in the moderate to severe stages of dementia, worries that the technical ESM device would be too difficult for older caregivers, and involvement in other caregiver support interventions. Clinicians and dementia case managers reported that a mild stage of dementia severity in the care recipient and familiarity with technological devices were facilitators for program recruitment.

Reach

Caregivers were approached to participate by the clinician who treated their partner with dementia (N = 120), their dementia case manager (N = 104), their care counselor (N = 8), or by the day care center of their partner (N = 41). Others were informed about the intervention by the Dutch Alzheimer Association (N = 8); requested information based on editorials in local newspapers, information brochures, or information stands in the southern Netherlands (N = 10); or knew caregivers who already participated in the intervention (N = 4). The Dutch Alzheimer Association promoted the intervention program via (1) Alzheimer Cafés for people with dementia and their caregivers, (2) a digital newsletter, and (3) their website. Almost all participating caregivers were recruited by health care professionals; only seven caregivers were informed about the intervention by the Dutch Alzheimer Association or in another way.

Sample characteristics

Mean age in the sample was 72.1 years (SD = 8.4) with 25 males (32.9%) and 51 females (67.1%). Educational level was low (n = 39, SD = 51.3), middle (n = 15, SD = 19.7) or high (n = 22, SD = 28.9). Severity of dementia in the care recipient was very mild to mild (n = 41, 54.7%), moderate (n = 25, 33.3%) or severe (n = 9, 12%) as measured with the Clinical Dementia Rating scale.

Intervention quality

Relevance and feasibility

ESM intervention

Program content. Overall, participants were satisfied with the content of the feedback sessions and indicated on a 7-point Likert scale that the feedback was easy to understand (M = 6.5, SD = 0.8). They appreciated getting the feedback in the form of pie charts and graphs (M = 6.6, SD = 1.0) and found it easy to understand the information in the graphs (M = 6.1, SD = 1.4). Most participants indicated that the amount of feedback was sufficient (M = 6.1, SD = 1.070, but some suggested to add information on problem behavior of the care recipient, and to elaborate more on time spend on different care tasks:

"Caregiving entails so many different aspects – next to assistance in activities of daily living, I spend a lot of my time on supervision and providing emotional support to my partner [person with dementia]. It could be valuable to differentiate between these different levels of care during the feedback sessions." [P174]

Participants did not feel like they needed more specific advice following the feedback that was given (M = 2.8, SD = 2.4). One participant, however, stressed that he would have preferred to receive more practical advice to help him applying the suggestions from the feedback in daily life.

Program structure. The program structure of blending ESM data collection with face-to-face contacts with a personal coach was experienced positively. Participants appreciated that the personal coach was available to explain and discuss the feedback and indicated that the coach was able to answer their questions well (M = 6.6, SD = 0.8).

"She [coach] made me feel at ease and encouraged me to talk about my feelings. A digital coach would have depersonalized the program." [P113]

Moreover, the written summary of the feedback provided by the coach at the end of each session, was seen as a useful addition (M = 6.5, SD = 0.8) and was considered to be easy to understand (M = 6.8, SD = 0.6).

Participants reported that the duration of contact for the feedback sessions was suitable (M = 6.5, SD = 0.5). Overall, the frequency of the sessions was considered to be sufficient. Two participants suggested including weekly sessions in the program instead of one session every two weeks, and two participants would have preferred fewer sessions. The overall duration of the intervention (6 weeks) was considered to be rather long. When being asked if participants would have preferred to prolong the program, only two out of twenty confirmed. Proper information about the total time investment before the start of the program was emphasized to be essential. Lastly, participants appreciated the possibility to situate the sessions in their own homes.

Advantages for participants. The feedback was considered supportive and increased awareness of both positive and negative feelings and behavior. Participants reported to gain more insight into their own situation and felt stimulated by the coach to talk about it.

"The situation with my partner [person with dementia] always felt ‘normal’ to me. I never really took the time for self-reflection. The feedback was like a mirror and wake-up call to me." [P138]

Some participants mentioned that the feedback was not new to them, but a confirmation and acknowledgement of their feelings.

"I have always tried to continue spending time on relaxation next to my care task. The feedback gave me more insight into my daily activity pattern and confirmed that my life is perfectly balanced at the moment." [P129]

Participants differed with respect to the extent to which they tried to apply the suggestions from the feedback in their daily lives (M = 4.6, SD = 2.3). Some felt that the feedback was not applicable to their personal situation, as they already believed to be sufficiently aware of their daily functioning. Others indicated that the intervention would be more relevant in the future, since they did not experience any difficulties in the caretaking process yet.

ESM in general

ESM device. Participants indicated the ‘PsyMate’ to be user-friendly and easy to operate (M = 6.9, SD = 0.2). They were satisfied with the instructions provided verbally by the research assistant (M = 6.2, SD = 1.7) and in written by means of a leaflet (M = 6.7, SD = 0.6). In general, participants reported that they were able to hear the alerts (M = 5.9, SD = 1.5) and read the text on the screen clearly (M = 6.1, SD = 1.2). However, participants with hearing or vision loss experienced difficulties due to a too weak alert sound and vibration, or too small letters on the screen of the device. Problems particularly arose when being in a noisy environment or outside in bright sunlight. In total, 9 participants experienced technical issues with the ‘PsyMate’ due to software problems (N = 7), an empty battery (N = 1), or a defective touchscreen (N = 1).

ESM content. A few items in the ESM questionnaire were mentioned to be difficult or unclear with respect to content or phrasing. In addition, some participants were struggling with the classification of their answers into one of the response categories. However, most indicated to experience few difficulties during completion of the items (M = 2.4, SD = 1.6). Generally, participants felt able to accurately describe their feelings and experiences in the ESM questionnaire (M = 5.4, SD = 1.1). Yet, reflecting on their emotions and expressing them quantitatively was considered to be challenging. Participants often mentioned that they would have preferred to explain their answers in more detail.

"I find it rather difficult to rate on a scale whether I feel down or not. The situation with my husband [person with dementia] makes me feel sad. However, it is like I got used to these negative feelings." [P127]

Participants reported having made several mistakes while completing the ESM questionnaire (M = 3.2, SD = 1.2), which was primarily due to the change in positively and negatively formulated items, answering items too rapidly, or being distracted by other activities (e.g. care task, shopping, driving, social interactions).

"I regularly gave the wrong answer when I got distracted by my partner [person with dementia] who kept asking questions. I missed an option in the PsyMate to correct my answers afterwards." [P132]

The content of the ESM questionnaire was considered to be rather negative. Some participants could not identify themselves with items such as ‘I feel desperate’ and ‘I am ashamed of myself’. They also mentioned that certain items (e.g. ‘I am in pain’ and ‘I have problems in walking’) were not applicable to them. It was suggested to adjust the ESM questionnaire conform the positive focus of the feedback sessions and to tailor it to one's personal situation.

ESM structure. In general, participants did not experience the ESM procedure as too aggravating or stressful with respect to the number of alerts per day (M = 1.9, SD = 1.3), the time it took to answer the questions for a single alert (M = 2.1, SD = 1.8), and the sound volume of the alert (M = 1.8, SD = 1.5), nor did they feel that the ESM interfered with their daily lives. Participants reported the ESM to affect their mood (M = 2.2, SD = 1.6), activities (M = 2.1 SD = 1.5), and contact with other people (M = 1.8, SD = 1.5) to a minimum extent. Both positive and negative influences were mentioned.

"Carrying the PsyMate with me all the time makes me feel better. It feels like a buddy that is always there for me." [P130]

"In the beginning I found it rather confronting to answer all these questions repeatedly. It forced me to dig into my own feelings." [P140]

Moreover, it was mentioned that the repeated assessments increased self-awareness in some participants.

"The diary made me realize that I spend most of my time at home. Now I see that I became more isolated due to the situation with my wife [person with dementia]." [P152]

"At times of the alert I noticed that I was often in company of friends or family. I became more aware of the good social network we have and of the importance to maintain it." [P154]

However, some participants indicated to get annoyed by the repeated nature of the ESM in which the same questions were asked persistently. The time between two succeeding alerts was perceived as too short and it was suggested to decrease the number of alerts per day.

"Answering the same questions over and over again made me feel bored and sometimes even irritated. Especially when two alerts followed each other so quickly, I had the feeling that I was giving the same answers twice." [P168]

Adherence to protocol

Intervention adherence to protocol contained: ESM data collection for six weeks, three face-to-face feedback sessions (session 1: 30 min; session 2 & 3: 45 min), and a written summary of the feedback after each session.

In total, 76.9% of the participants allocated to the intervention group completed the intervention (20/26). Five dropped out after completing the baseline assessment and before the actual start of the intervention. Reasons for withdrawal were health problems (N = 1), institutionalization of the care recipient (N = 1), difficulties with the ESM device due to vision loss (N = 1), and considering the intervention as too time-intensive (N = 2). One participant dropped out during the third feedback session due to institutionalization of the care recipient.

Total intervention time ranged from 6 to 8 weeks (M = 6.5, SD = 0.7). Deviations in intervention time were reported for eight participants: one week longer for six participants and twee weeks longer for two participants. Reasons for intervention period variance were: busy schedules, holidays, and technical problems with the ‘PsyMate’. The duration of each feedback session varied from 45 to 150 min (M = 89.6, SD = 23.1). All feedback sessions, except for one, lasted longer than initially planned in the study protocol. The average duration of session 1, 2, and 3 were respectively 96 min (SD = 21.9, range: 60–135), 81 min (SD = 22.4, range: 45–120, and 92 min (SD = 23.2, range: 60–150). Coaches reported adherence to protocol during all sessions with respect to the structure of the feedback sessions and the delivery of a written summary of the feedback after each session.

Compliance during the ESM data collection was high, with an average response rate of 74.7% (2690/3600) to the alerts. During the course of the intervention no fatigue effect was present according to response rates examined after 2 weeks (75.0%; 900/1200), 4 weeks (72.3%; 867/1200), and 6 weeks (76.9%; 923/1200). Reasons for missing alerts were: not having heard the alert, not having been able to read the text when being outside, having forgotten the ‘PsyMate’ at home, and inconvenient circumstances (e.g. car driving, church visits, funerals, swimming, sleeping, sports, care task).

Discussion

This study reveals process data about sampling and intervention quality of the ESM intervention ‘Partner in Sight’ for spousal caregivers of people with dementia.

Sampling quality

Data on sampling quality showed an overall participation rate of 31.4% (76/242). Since many participants refused to participate, the inclusion rate was lower than expected and the original goal to include 90 participants was not met. In general, response rates in caregiver studies vary widely as selection criteria, recruitment methods, and the content of research projects differ between studies (Tarlow & Mahoney, 2000). To the best of our knowledge, this is the first ESM intervention for caregivers of people with dementia and cross study comparisons are, therefore, not possible. A recent study in which a comparable ESM intervention was provided to persons with depression also reported recruitment difficulties (only 102 instead of 120 participants were included) (Kramer et al., 2014). As a consequence of the smaller sample size, statistical power might be too low to establish significant effect size differences between groups in the effect analyses.

Main recruitment barriers in our study included no need for additional support and considering the intervention as too time-consuming or burdensome. Previous research has indicated that informal caregivers may struggle with a stigma associated with the term dementia, which could make them refuse to participate in research that places emphasis on the care recipient's diagnosis (Garand, Lingler, Conner, & Dew, 2009). Furthermore, advanced age, research skepticism, and the perception that the intervention will lack direct personal benefit may have influenced decisions about study participation (Connell, Shaw, Holmes, & Foster, 2001). Finally, the intrusive nature of ESM could have withheld caregivers from participating in the study. Especially caregivers who spent a lot of time on caregiving, or who experience high levels of burden, might have been more inclined to reject study participation (R. Schulz, Visintainer, & Williamson, 1990). Also, a bias in recruitment is highly likely, because clinicians and case managers had a substantial impact on who was recruited. This might challenge the external validity of the study results, since it negatively impacts the generalizability of the results (Groves, 2006; MacDonald, Newburn-Cook, Schopflocher, & Richter, 2009). Future analyses on the sample characteristics in the effect study are important to reveal whether our sample was representative of the general caregiver population.

At post-intervention assessment (N = 64), blinding of the researchers for group allocation was intact for only 37.5% (24/64) of the participants. Blinding is an important safeguard against bias, particularly when assessing subjective outcomes (K. F. Schulz, Altman, & Moher, 2010). However, blinding in psychosocial research is challenging and maintenance of blinding has seldom been described (Mayo-Wilson et al., 2013). The complexity of our study design, including alternate visits from either the coach or researcher, hindered complete masking from group allocation.

Intervention quality

Overall, participants considered the intervention program ‘Partner in Sight’ acceptable and feasible. Participants were satisfied with the respect to the content and structure of the program. The ESM-derived feedback was considered supportive and increased participants’ awareness of their feelings and behavior. However, there was a large variance in the extent to which participants tried to apply the feedback into their daily lives. The feedback was not always considered to be applicable to one's personal situation, as participants were already sufficiently aware of their daily functioning or did not experience any difficulties in the caregiving process yet. Participants emphasized the importance of the personal coach in providing face-to-face feedback and encouraging them to implement new insights into their daily lives. This finding is supported by a systematic review on internet-based interventions, in which guidance by a personal coach has proven to be a noteworthy extension to online interventions for informal dementia caregivers (Boots, de Vugt, van Knippenberg, Kempen, & Verhey, 2014). Exchanges with a coach might increase commitment to the intervention and boost confidence to implement the provided feedback into one's daily life (Ducharme, Dubé, Lévesque, Saulnier, & Giroux, 2011). Participant compliance to the intervention was high (76.9%), which could be explained by the motivational aspect of having a coach (Wilhelmsen et al., 2013). However, participants indicated the overall duration of the program (6 weeks) to be rather long. Although the ESM data collection was not considered to be overly burdensome and the average response rate to the alerts was high (74.4%), it was suggested to decrease the number of alerts per day. A study by Stone et al. (2003) demonstrated that perceived burden reduced by choosing a less intensive sampling density. Lowering the time-investment might facilitate recruitment and future implementation of the intervention in clinical practice. Other suggestions for improvement of the program were to tailor the ESM questionnaire more to one's personal situation and to formulate items in a more positive rather than negative way to create a better link with the positive focus of the feedback sessions, as negative formulated questions raised distress in some caregivers. Moreover, it was recommended to improve readability and audibility of the ESM device in order to be easier to use for elderly persons.

Considering the intervention adherence to protocol, results showed that the protocol was largely followed. Variances in total intervention time were reported and may influence the effectiveness of the intervention (Moncher & Prinz, 1991). However, reasons for protocol deviations were not uncommon for informal caregivers (e.g. time constraints) and for ESM studies in general (e.g. technical problems). Furthermore, feedback sessions lasted considerably longer than initially planned in the study protocol. Given that more face-to-face contact with a personal coach might benefit caregivers, the longer duration of the sessions may influence the results and should be taken into consideration in the effect analyses.

Strengths and weaknesses

To our knowledge, no other study has reported process evaluation outcomes of an ESM intervention for caregivers of people with dementia. The application of ESM in the field of dementia is still in its infancy (van Knippenberg et al., 2016c). Our study, therefore, provides valuable information for future implementation of ESM as a supportive tool in clinical practice. However, several limitations of this study need to be considered. First, protocol deviations were mainly measured based on data directly collected from the coaches who were responsible for delivering the intervention. No measurements were available to examine adherence to protocol more objectively, for example through independent observations. Previous research has shown large discrepancies between self-report measures and ratings based on tape-recordings, indicating that professionals might not always be aware of their treatment fidelity (Dorresteijn, Rixt Zijlstra, Van Haastregt, Vlaeyen, & Kempen, 2013). Second, participants’ satisfaction with the program was measured by the coach during the last intervention visit. Therefore, participants may have given socially desirable answers and might not have had enough time to reflect on the relevance and feasibility of the intervention. A follow-up measurement on participants’ satisfaction with the program would have been useful to examine whether they actually implemented the feedback into their daily lives. At last, intervention quality was only determined from the perspective of participants in the intervention group. Process data on relevance of the pseudo-intervention (i.e. ESM data collection and three face-to-face sessions with a coach without receiving ESM-derived feedback) would have been useful to evaluate the extent to which the collection of ESM data in itself increased self-awareness and elicited behavioral changes. In the intervention group some participants also mentioned that the repeated assessments led them to pay more attention to their internal states and behavior. The effect analyses may provide more insight into the potential benefits of ESM data collection in itself and the added value of the ESM-derived feedback.

Conclusion

The ESM intervention ‘Partner in Sight’ was generally well performed and received by spousal caregivers of people with dementia. Overall, participants were satisfied with the program and considered it to be supportive. Suggestions for improvement were provided to reduce the time intensity of the program, to align the ESM questions with the positive focus of the feedback, and to better tailor the program content to one's personal situation. Although recruitment barriers were encountered, results indicate that future implementation of ESM interventions is likely to be feasible in regular health care for caregivers of people with dementia. If the ESM intervention ‘Partner in Sight’ proves to be (cost-) effective, it could be a valuable addition to the current health care system.

Disclosure statement

No potential conflict of interest was reported by the authors.

Additional information

Funding

This work was supported by the Dutch Alzheimer Society [grant number WE03-2011-06].