Abstract
Objective
To investigate the predictors of early and late dropout among community users of the MoodGYM website, a five module online intervention for reducing the symptoms of depression.
Method
Approximately 82,000 users accessed the site in 2006, of which 27% completed one module and 10% completed two or more modules. Adherence was modeled as a trichotomous variable representing non-starters (0 modules), early dropouts (1 module) and late dropouts (2–5 modules). Predictor variables included age, gender, education, location, referral source, depression severity, anxiety severity, dysfunctional thinking, and change in symptom count.
Results
Better adherence was predicted by higher depression severity, higher anxiety severity, a greater level of dysfunctional thinking, younger age, higher education, being female, and being referred to the site by a mental health professional. In addition, users whose depression severity had improved or remained stable after the first intervention module had higher odds of completing subsequent modules.
Conclusions
While the effect of age and the null effect of location were in accordance with prior adherence research, the significant effects of gender, education and depression severity were not, and may reflect user characteristics, the content of the intervention and unique aspects of online interventions. Further research directions are suggested to investigate the elements of open access online interventions that facilitate adherence.
Introduction
Open access websites are characterized by dropout of substantial numbers of users before the completion of the programs (CitationEysenbach 2005; CitationGlasgow et al 2007). As greater exposure to website content is associated with increased benefit (CitationChristensen et al 2004), poor adherence may represent a limitation of Internet interventions that are designed to reduce the prevalence of common mental health disorders. To improve adherence, more information is needed about the particular predictors of website usage on these sites.
Previous research of adherence rates for face to face treatment of depression has found that older age, medication side effects and more severe depression tend to be associated with faster trial dropout and/or decreased adherence to treatment (CitationPampallona et al 2002; CitationYildiz et al 2004; CitationAkincigil et al 2007). However a recent study observed that younger age was associated with greater dropout (CitationArnow et al 2007) and some studies have found that initial depression severity does not predict adherence to treatment (CitationAkincigil et al 2007; CitationLoh et al 2007). Adherence research from studies examining a range of health conditions (Citationvan Dulmen et al 2007) has indicated that treatment factors such as the presence of reminders, rewards and monitoring are associated with higher retention, but that demographic factors such as education, socioeconomic status, and marital status do not predict adherence. A recent review undertaken on the dropout associated with randomized controlled trials (RCTs) of websites for depression and anxiety broadly supported these findings (CitationChristensen and Griffiths 2008). For depression websites, predictors found to be associated with increased adherence were lower baseline rates of depression, younger age, and less knowledge of psychological treatments. Gender, education or quality of life, when measured, did not predict adherence.
Within the research trial context, dropouts have been sub-classified as no shows, (those who do not proceed to the consent or treatment stage and do not complete assessments), early dropouts (those that drop out relatively early from a program and complete a small number of assessments), or late dropouts (those that drop out after additional sessions or modules) (CitationDavis and Addis 1999). Taking into account this categorization, the present study investigates factors that predict no engagement, early and late dropout from a high volume, free access website for depression. In the case of open access web-based interventions, dropout is effectively synonymous with nonadherence, as trial participation is measured concurrently with the administration of the treatment. That is, a participant can only drop out of the trial by discontinuing the treatment, as progress in the trial is measured by the completion of mandatory assessments throughout the intervention.
Demographic characteristics examined in the present study were age, gender, education, country of origin, and remote or rural location. Level of severity of depression was measured by the self-report of a history of marked depression, referral to the site by a mental health professional, and current anxiety and depression symptom levels. Dysfunctional thinking was measured by a new scale (CitationParslow et al 2006), to examine whether dysfunctional thinking styles were associated with greater dropout. Using a subsample of the website users, we also examined the hypothesis that an initial improvement in depression symptoms will predict greater retention.
Data were collected from the website MoodGYM (http://moodgym.anu.edu.au), which has been shown in a number of RCTs to reduce anxiety and depression symptoms at post test and at twelve months follow-up (CitationChristensen et al 2004; CitationGriffiths and Christensen 2007; CitationMackinnon et al 2008).
Methods
Sample
The sample included community users who registered on the MoodGYM website between January 2006 and April 2007. Registrants are individuals who enroll on the site (by providing a name and password), automatically creating a record in the database. Of 194,840 unique visitors to the site in the period, 94,270 (48%) registered for the intervention. From the 94,270 users who registered, exclusions were made on the following criteria: test users (36 users); self-identified psychiatrists, psychologists or therapists (7,147 users); researchers reviewing depression or anxiety sites (2,967 users); or students studying cognitive behavior therapy (CBT) as part of a college or university psychology course (2,887 users). The sample used for the initial analysis included 82,159 users, as some users fell into multiple exclusion categories. A subsample of users who finished all items in the introductory survey and the mental health questionnaires (n = 59,453 eligible users) was used to examine the effect of various demographic and mental health variables on module completion. Finally, a subsample of these users who completed at least one module of the program (n = 19,304) formed a final analysis group, examining the effect of any changes in symptoms on the number of modules completed. A flowchart of the sampling process is shown in . Ethical approval for the study was provided by the Human Ethics Committee of the Australian National University.
Figure 1 Sample selection process.
Site description
The site consists of a set of five cognitive behavioral training modules, a personal workbook (containing 29 exercises and assessments) that records and updates each user’s responses, an interactive game, and a feedback evaluation form. Module 1 introduces the site “characters” (who model patterns of dysfunctional thinking) and demonstrates the way mood is influenced by thinking, using animated diagrams and interactive exercises. Module 2 describes types of dysfunctional thinking and the methods to overcome them, and provides scales for self-assessment of “warpy” (dysfunctional) thoughts. Module 3 provides behavioral methods to overcome dysfunctional thinking, and includes sections on assertiveness and self-esteem training. Module 4 assesses life-event stress, pleasant events, and activities, and provides three downloadable relaxation tapes. Module 5 covers simple problem-solving and typical responses to relationship break-up. Workbook exercises are integrated into each of the modules.
Each module was designed to take between 30 and 45 minutes to complete, but users can opt to skip certain sections. Module 1 has approximately 30 pages but many of these contain browser-supported interactive features (creating additional pages) and supplementary pop-up windows. Module 3 has more than 60 pages, but users are directed to specific sections depending on their scores on earlier tests and thus may not access all pages (CitationChristensen et al 2002). Users were required to complete the modules in order and to complete core assessments at the end of each module. Past RCTs have shown the MoodGYM site to be effective in reducing depression symptoms at post-test and 12 month follow-up (CitationChristensen et al 2004; CitationGriffiths and Christensen 2007; CitationMackinnon et al 2008), and that receiving the first two modules may be sufficient for a significant reduction in depression symptoms (CitationChristensen et al 2006).
Measures
Users were asked to complete a brief survey prior to starting the first module of the MoodGYM program. The survey covered demographics, including age (seven categories), gender, highest level of education (seven categories), country of origin (recoded into continent for analysis), whether the user lived in an urban or rural/remote area, history of marked depression (yes/no) and whether the user was referred to MoodGYM by a mental health professional (yes/no). Screening items were also included to identify psychiatrists, psychologists, therapists, depression, and anxiety researchers and students studying CBT.
Following the introductory survey, three brief psychometric scales were administered measuring depression, anxiety and dysfunctional thinking. The Goldberg Depression and Anxiety Scales (CitationGoldberg et al 1988) each comprise nine items that measure current symptoms of affective disorders. The number of “yes” responses were summed for these scales to derive two scores (range 0–9), with higher scores reflecting a greater level of depression/anxiety. The Warpy Thoughts Scale, a 42-item scale that measures dysfunctional thinking (CitationParslow et al 2006) was also administered. The scale is comprised of seven 6-item subscales that measure the need for approval, the need to be loved, the need to succeed, perfectionism, influence on others, external requirements for happiness and expectations of rights. Responses were given on a 5-point scale ranging from “strongly agree” to “strongly disagree”. The total scale score is calculated by obtaining the mean of the 42 items, resulting in a continuous score ranging from 1–5 points, with higher scores reflecting higher levels of dysfunctional thinking.
In addition to the survey data, time and date data were collected to measure completion rates and times. The start date was used as a predictor variable to investigate whether adherence rates changed over time. This variable was defined as the number of months after January 2006 that the user signed up to the site. Month units were used for simplicity of interpretation, although the measure was kept continuous (to the nearest second).
Module completion was investigated as the primary outcome and was defined as a trichotomous variable, comparing those who finished no modules, one module and two or more modules. This categorization was made because a large proportion of users finished no modules and the majority of the CBT-related material is introduced in the second module and beyond. The categorization also reflects the “no-show”, “early dropout” and “late dropout” classifications suggested by CitationDavis and Addis (1999).
Analysis
Descriptive analyses examined the univariate effects of the various independent variables on module completion. For the descriptive analysis, chi-square and F statistics were calculated for the categorical and continuous effects respectively. Predictors of module completion for those who completed both the introductory survey and the mental health measures were investigated with a nominal logistic regression that compared users who completed zero modules with those who completed one module and those who completed two or more modules. A subsequent logistic regression analysis of the final subsample investigated the effect of depression severity changes on module completion, comparing users who completed one module with those who completed two or more modules. The analyses were performed using SPSS v15 (SPSS Inc., Chicago, IL, USA).
Results
The initial analysis sample comprised 82,159 users, of whom 63% completed no modules, 27% completed one module and 10% completed two or more modules. Overall, 81% of users had completed high school and 40% of users had a bachelors degree or higher. 54% of users were aged less than 35 years and 66% were female. 43% of registrants were from Oceania (41% from Australia), 35% were from Europe (31% from the United Kingdom), 16% were from North America (12% from the United States), 3% were from other continents, and 4% did not respond. 22% of users reported being from a rural or remote area. 85% of registrants reported having a history of marked depression and 18% were referred to MoodGYM by a mental health professional. Descriptive statistics for each variable are broken down by module completion in .
Table 1 Descriptive statistics by module completion
The number of respondents for the Goldberg subscales (Cronbach’s α = 0.69 for depression; .68 for anxiety) and the Warpy Thoughts Scale (Cronbach’s α = 0.95) was smaller as a result of early dropout and non-response. Based on the univariate comparisons (), higher educational attainment, younger age, living in Oceania or Europe, being female, reporting a history of marked depression, being referred to the site by a mental health professional, and having a higher initial level of depression, anxiety or dysfunctional thinking were all significantly associated with greater completion of modules. There was no significant effect of location (city vs. rural) on module completion. Referral to the site by a mental health professional was associated with a higher severity of depressive symptoms (mean Goldberg score for referred = 6.62; mean for nonreferred = 5.93; F = 1152, p < 0.001; not shown in table).
The results of the nominal logistic regression are displayed in . The regression included 59,453 users, as many of the users dropped out before reaching the latter stages of the introductory survey. Among these 59,453 users, 52% completed no modules, 35% completed one module and 13% completed two or more modules. A one-unit increase on the Goldberg depression subscale was associated with a 3% increase in the odds of completing one module and a 5% increase in completing two or more modules. A one-unit increase in the anxiety subscale was associated with a 3% increase in the odds of completing one module but was not significantly associated with the completion of additional modules. A one-point increase in the dysfunctional thinking score was associated with 4% greater odds of completing one module and 13% greater odds of completing two or more modules. There was also a significant effect of time, with a 2% decrease per month in the odds of completing two or more modules.
Table 2 Nominal regression of module completion (N = 59,453)
Reporting a history of depression was not significantly associated with module completion. However, reporting past depression was strongly associated with current depression and anxiety (F = 9761, p < 0.001 for current depression score; F = 4564, p < 0.001 for current anxiety score). To examine the effect of past depression severity in the absence of the strong effect of current severity, the regression was rerun omitting the Goldberg depression and anxiety scores from the model (not displayed). Without the current severity scores in the model, past history of depression significantly predicted lower odds of dropout (χ2 = 14.9, p < 0.001 for one module; χ2 = 13.1, p < 0.001 for two or more modules).
Age was strongly associated with module completion, with younger users significantly more likely to complete modules. The trend across age groups appeared to be a linear effect. Higher education was significantly associated with higher odds of completing two modules. However the effect was not as clear for one module, with some lesser-educated groups having higher odds of completion than higher-educated groups. There was a significant effect of location, as North American users had significantly lower odds of module completion while European users had somewhat higher odds of finishing two or more modules than those in Oceania. However, users in a rural or remote location had no difference in completion than users in urban areas. There was a small but significant effect of gender, with male users 6% less likely to complete two or more modules than female users. Those users that were referred to the site by a mental health professional had 19% greater odds for completing one module and 32% greater odds for completing two or more modules than those who reached the site in some other way.
An additional analysis examined whether dropout was associated with changes in depression status, specifically whether users who had improved in the number of symptoms of depression were more likely to continue with the intervention. This analysis was restricted to the subsample that finished one or more modules, using a logistic regression to compare those who completed one module with those who completed two or more. The change in depression score was measured as the initial Goldberg depression score subtracted from the second Goldberg depression score, which was administered immediately after the first module. An increased depression score at follow-up indicated deterioration (that is, an increase in the number of symptoms experienced), while a decreased score indicated improvement. 19,304 users completed the second depression assessment and were included in the analysis. shows the outcome of the logistic regression. Demographic variables (age, education, gender) and initial Goldberg depression score were retained in the regression. The effect of the depression score change was significant, with users who had a decreased score having 16% higher odds of continuing the intervention than users who had an increased score. Users who had no change in depression score also had 20% greater odds of completing two or more modules than those who had an increased score. These differences account for the initial level of depression, which was not significantly associated with completing two or more modules.
Table 3 Logistic regression of completing one vs. two or more modules (N = 19,304)
In addition to modeling the three categories of dropout, we separately examined other measures of adherence, including the number of modules completed (0–5), the number of exercises completed (out of 29 possible exercises), the amount of time spent on the first module, and the amount of time spent on all modules. Regression analyses revealed fairly similar predictors across all of the models, with younger age, higher level of education, greater depression severity and a greater level of dysfunctional thinking consistently predicting improved adherence.
Discussion
The present study found that younger rather than older participants, those with higher education, and those from European countries were more likely to adhere to the website. Users from North America were less likely to adhere to the intervention. Although females had significantly greater odds of completing two or more modules, the overall effect of gender was small. Rural or remote location did not predict adherence. These findings confirm previous work in that they support the observation that younger people are more likely to adhere, while other demographic variables tend not to predict dropout. In contrast to previous research where negative findings have been reported, we did find that higher education predicted adherence to two or more sessions. This significant finding may reflect the fact that successful CBT training requires a reasonably high level of ‘psychological mindedness’, which is likely to be associated with more education. The effects of age and education may also be due to increased Internet savvy among younger and better-educated users. In addition, the intervention was originally developed for a youth audience, so some of the examples and exercises may have been less relevant to mature users.
The level of depression severity also predicted performance, as judged by two criteria: referral by a health professional and level of anxiety and depression as measured by levels of symptoms at pre-assessment. The size of this effect was small. Nevertheless, this finding was in contrast to previous research from both standard face to face trials and Internet-based randomized controlled depression and anxiety trials which have reported that those with lower levels of symptoms may be more likely to adhere to treatment protocols (CitationYildiz et al 2004; Christensen and Griffiths pers comm). There are a number of potential reasons to explain why an open access website might differ from the results of other research investigations. Patients accessing help through open access websites may have fewer options than those who seek help in face to face interventions or those who choose to enroll in research trials, and thus be more likely to remain on the site if the site is useful, relative to comparative samples selected into other types of interventions. This effect needs further investigation. A recent study (CitationMojtabai et al 2002) reported that 49% of individuals with a mood disorder perceived the need for care, but only 13% sought help from a mental health professional. This finding is consistent with the idea that many individuals with high levels of depressive symptoms do not seek face to face help and may not seek help through a research trial. Another explanation is that dropout is greater in more severe individuals in standard antidepressant drug trials, because these individuals may suffer side effects, and side effects are a known cause of nonadherence to treatment. The latter explanation does not explain, however, the discrepancy between previous research trials and the findings from open access web sites.
We also expected that the variable “previous history of depression”, which was indicative of current depression or anxiety severity, would also predict greater adherence. We did not find this association. Given that depression is a chronic disease, and chronic disease is associated with greater dropout over time, we might have expected that those with a previous history to experience higher dropout, even with more severe symptoms. The lack of a significant finding for depression history may be due to the strong correlation between past and current depression. This explanation is consistent with our finding that the univariate effect of depression history on adherence was strongly significant, yet the effect disappeared in the multivariate model.
Higher levels of dysfunctional thinking were associated with a greater level of adherence. Although the Warpy Thoughts Scale was designed to measure seven domains of dysfunctional thinking, further validation studies are required before employing these subscale measures. We did, however, examine each individual item of the scale and found consistent patterns of association with the adherence outcome, with higher scores predicting greater adherence. We had predicted that higher dysfunctional thinking would lead to poorer adherence, but found the opposite result. This effect may be related to the content of the intervention, which is aimed at decreasing dysfunctional thinking. Users with lower levels of dysfunctional thinking may have experienced less benefit from the intervention and consequently dropped out.
We examined the issue of whether improvement in symptoms was associated with continuing in the intervention by comparing users who completed one module with users who completed two or more modules. Users whose depression improved or remained constant had significantly greater odds of continuing past the first module than users whose depression scores deteriorated. Although these effects were small, this finding suggests that users whose depression is worsening through the course of the first module may find it more difficult to continue the intervention or choose to discontinue the program because it is having no effect on their symptoms. That is, if the program does not appear to be relevant or help to improve symptoms, users drop out.
Limitations
The limitations of the present research need to be outlined. First, these data apply to an open website where there was no monitoring or feedback, apart from automated interactivity. Thus the findings may not generalize to other online delivery contexts, or more particularly to research involving randomized controlled trials where contact with participants is encouraged and monitoring of progress is standard. However, the divergence in findings also highlights the importance of this research, as investigating the determinants of adherence in a variety of contexts will lead to better targeting and tailoring of different intervention modalities. Secondly, the size of the effects described here are small, with some factors having only a small impact on adherence rates. For example, the effect of depression severity on completion of none compared to two modules was relatively small (only a 5% increase in odds of completing two or more modules). Demographic factors such as education had greater impact, with the effect for young participants much higher than that for older factors. Because of the large sample size, the size of these effects could be estimated with precision. Variables that failed to predict significantly can be dismissed relatively robustly, given the opportunity for such effects to be observed, if they existed. The positive variables failed to predict very much of the variance. Overall, these findings indicate that we know very little about unmeasured variables which may predict adherence.
Thirdly, the methodology is correlational, and many variables are highly correlated with each other (eg, current severity of depression, previous history of depression and referral by a mental health professional). Thus it is not clear which of these effects has primacy. It is also unclear why the date of participation would predict (negatively) adherence. This effect was relatively small and may be associated with an increase in North American users (who tend to adhere less) or an increase in the number of users who were referred to the site from external online articles later in the study period. Finally, as the website is open to the general public and users are anonymous, it is difficult to verify the accuracy of all responses. While some cleaning was performed to exclude test users, there remains the possibility that some users had multiple registrations or fabricated responses. Given the large sample size, the effect of a moderate number of fabricated responses would be negligible.
Future directions
Overall, the findings from this study do indicate that adherence rates are relatively low on this open access website, although approximately 10% of the sample does achieve a minimum dose of treatment (CitationChristensen et al 2006). The rates are much lower than those achieved when research on the same website was undertaken within the structure and framework of an RCT (CitationChristensen et al 2004). Nevertheless, the findings of the study do confirm that age and education are reliable predictors of adherence for CBT depression treatment sites, although the amount of variance explained is relatively small. The study also found that depression severity was associated with higher levels of adherence, a result that contrasts with findings from face to face trials (often anti-depressant medication trials). This difference may reflect the low symptom level of the website sample relative to face to face trials. Alternatively, the difference in findings could be related to the greater motivation of spontaneous web users who either have no other resources available or prefer the online format for reasons of privacy or convenience.
The present website remains a vehicle for further research on open access website adherence. Such research can be achieved by broadening the range of pretest variables that we examine, for example, introducing questions related to user expectations, such as whether they have knowledge of the content of the site before they commence, whether they believe the website will be effective in producing positive outcomes, if they believe therapy can be delivered online, whether they prefer the medium of the internet, and information about levels of use and satisfaction with standard treatments obtained face to face.
Perhaps more significantly, given the existing capacity to randomize online, we could endeavor to investigate experimentally whether relatively small additional components such as email reminders will substantially improve adherence and compliance. Evidence from previous research (CitationClarke et al 2005; CitationRitterband et al 2005; CitationRobertson et al 2006) suggests such components may well pay off.
Acknowledgments
Helen Christensen is supported by NHMRC Research Fellowship No. 366781. We wish to thank Anthony Bennett and Lou Farrer for their contributions to the research. There are no conflicts of interest to report.
References
- AkincigilABowblisJRLevinC2007Adherence to antidepressant treatment among privately insured patients diagnosed with depressionMed Care45363917496721
- ArnowBABlaseyCManberR2007Dropouts versus completers among chronically depressed outpatientsJ Affect Disord9719720216857266
- ChristensenHGriffithsK2008Adherence in internet interventions for anxiety and depressionAnn Behav Medin press
- ChristensenHGriffithsKGrovesC2006Free range users and one hit wonders: Community users of an Internet-based cognitive behaviour therapy programAust N Z J Psychiatry40596216403040
- ChristensenHGriffithsKMJormAF2004Delivering interventions for depression by using the internet: randomised controlled trialBMJ32826514742346
- ChristensenHGriffithsKMKortenA2002Web-based cognitive behavior therapy: analysis of site usage and changes in depression and anxiety scoresJ Med Internet Res4e311956035
- ChristensenHGriffithsKMKortenAE2004A comparison of changes in anxiety and depression symptoms of spontaneous users and trial participants of a cognitive behavior therapy websiteJ Med Internet Res6e4615631970
- ClarkeGEubanksDReidE2005Overcoming Depression on the Internet (ODIN) (2): a randomized trial of a self-help depression skills program with remindersJ Med Internet Res7e1615998607
- DavisMJAddisME1999Predictors of attrition from behavioral medicine treatmentsAnn Behav Med213394910721442
- EysenbachG2005The law of attritionJ Med Internet Res7e1115829473
- GlasgowRENelsonCCKearneyKA2007Reach, engagement, and retention in an Internet-based weight loss program in a multi-site randomized controlled trialJ Med Internet Res9e1117513282
- GoldbergDBridgesKDuncan-JonesP1988Detecting anxiety and depression in general medical settingsBMJ29789793140969
- GriffithsKMChristensenH2007Internet-based mental health programs: a powerful tool in the rural medical kitAust J Rural Health1581717441815
- LohALeonhartRWillsCE2007The impact of patient participation on adherence and clinical outcome in primary care of depressionPatient Educ Couns65697817141112
- MackinnonAJGriffithsKMChristensenH2008Comparative randomised trial of online cognitive – behavioural therapy and an information website for depression: 12-month outcomesBr J Psychiatry192130418245031
- MojtabaiROlfsonMMechanicD2002Perceived need and help-seeking in adults with mood, anxiety, or substance use disordersArch Gen Psychiatry59778411779286
- PampallonaSBolliniPTibaldiG2002Patient adherence in the treatment of depressionBr J Psychiatry180104911823317
- ParslowRAChristensenHGriffithsKM2006The Warpy Thoughts Scale: A new 20-Item instrument to measure dysfunctional attitudesCogn Behav Ther351061616754265
- RitterbandLMBorowitzSCoxDJ2005Using the internet to provide information prescriptionsPediatrics116e643716263978
- RobertsonLSmithMCastleD2006Using the internet to enhance the treatment of depressionAust Psychiatry1441317
- van DulmenSSluijsEvan DijkL2007Patient adherence to medical treatment: a review of reviewsBMC Health Serv Res75517439645
- YildizAPaulerDKSachsGS2004Rates of study completion with single versus split daily dosing of antidepressants: a meta-analysisJ Affect Disord781576214706727