Predicting dropout from psychological treatment using different machine learning algorithms, resampling methods, and sample sizes

Abstract

Objective: The occurrence of dropout from psychological interventions is associated with poor treatment outcome and high health, societal and economic costs. Recently, machine learning (ML) algorithms have been tested in psychotherapy outcome research. Dropout predictions are usually limited by imbalanced datasets and the size of the sample. This paper aims to improve dropout prediction by comparing ML algorithms, sample sizes and resampling methods. Method: Twenty ML algorithms were examined in twelve subsamples (drawn from a sample of N = 49,602) using four resampling methods in comparison to the absence of resampling and to each other. Prediction accuracy was evaluated in an independent holdout dataset using the F₁-Measure. Results: Resampling methods improved the performance of ML algorithms and down-sampling can be recommended, as it was the fastest method and as accurate as the other methods. For the highest mean F₁-Score of .51 a minimum sample size of N = 300 was necessary. No specific algorithm or algorithm group can be recommended. Conclusion: Resampling methods could improve the accuracy of predicting dropout in psychological interventions. Down-sampling is recommended as it is the least computationally taxing method. The training sample should contain at least 300 cases.

Keywords:

• dropout prediction
• machine learning
• supervised learning
• sample size
• resampling methods
• data imbalance

Clinical or methodological significance of this article: This paper aims at improving the prediction of the occurrence of dropout in psychological interventions in a naturalistic setting and providing evidence-based guidelines to set a standard for, support researchers in modelling dropout, and increase comparability of findings. To achieve this the performance of 20 machine learning (ML) algorithms is compared in twelve differently sized subsamples under the influence of four different resampling methods as well as without resampling. The results of this paper can support future research on dropout prediction by providing empirically supported parameters for sample size recommendation and the selection of suitable algorithms. Improved dropout predictions at the beginning of treatment could be integrated into feedback systems for psychotherapists and could result in a more accurate clinical prognosis, enhanced case conceptualization and clinical decision-making. This could enable therapists to assess the risk of dropout for individual patients and apply clinical techniques to improve motivation, alliance, or treatment expectations, thus potentially reducing dropout.

Introduction

Dropout or the premature discontinuation of psychological therapies is an important topic in mental health practice and research. Fortunately, dropout of psychotherapy is a relatively rare event. The meta-analysis of Swift et al. (2017) reported an average of 21.9% of clients ending therapy prematurely. It is, however, a very costly event, as it is associated with poor treatment outcomes, higher hospitalization rates for individual patients, and high health, societal and economic costs (e.g., Barrett et al., 2008; Delgadillo et al., 2014; Druss & Rosenheck, 1999; Karterud et al., 2003; Sainsbury Centre for Mental Health, 2003). However, the definition of dropout is heterogeneous. Previously applied definitions included for example patients ending treatment against the advice of the therapist, a predefined treatment length, the completion of a treatment protocol or missing two consecutive sessions (e.g., Beckham, 1992; Hatchett et al., 2002; Horner & Diamond, 1996; Kolb et al., 1985; Maher et al., 2010).

From a methodological standpoint, dropout prediction is comparable to cancer screenings. Cancer is also a rare occurrence that is associated with high costs (Sung et al., 2021). The goal is to identify as many of the rare events as possible. Not identifying a patient with cancer (false negative) is associated with very high costs and should be avoided. Identifying a healthy patient as having cancer (false positive) is associated with decisively lower costs. Which ratios are deemed important depends on the content of the classification problem. In an interview process to select a candidate for example a lower false positive rate is desired.

Regarding precision mental health care, the same is true for the prediction of the occurrence of dropout in a routine care setting. Not identifying a possible dropout case (false negative) is associated with higher costs than falsely identifying a non-risk case as a potential dropout case (false positive; e.g., Lutz et al., 2019). Alerting a therapist at the beginning of the therapy of a patient’s increased risk of dropping out of therapy could enable the therapist to intervene and, for example, increase the focus on the therapeutic alliance or therapy motivation. Although, a good therapeutic alliance and therapy motivation is important for all patients, therapists are limited in their resources and need to allocate those as best as possible. Patients at risk of dropping out of therapy might need a more motivation-oriented strategy. Some patients, however, might profit more from early problem-solving (Lutz, Deisenhofer, et al., 2021).

Patient-specific predictors of dropout have been identified in several studies. Male sex, young age, low education level, personality disorders, low initial global functioning, and high initial distress have been found to be associated with dropout (Bower et al., 2013; Karterud et al., 2003; McMurran et al., 2010; Swift & Greenberg, 2012; Zimmermann et al., 2017). However, findings on predictors of dropout are heterogeneous, most studies used small samples, applied heterogeneous methods, and their statistical precision needs further improvement.

Recently, machine learning (ML) algorithms have been introduced in psychotherapy outcome research. Larger datasets and a steady increase in computational power have led to ML techniques being discussed as an option to improve prediction models in psychotherapy (Aafjes-van Doorn et al., 2020; Delgadillo & Lutz, 2020). ML refers to approaches combining statistics and computer science, and these algorithms can be grouped by the similarity of their function (Figure S1; Brownlee, 2019; Gillan & Whelan, 2017; Iniesta et al., 2016). Different ML approaches have been used to predict treatment outcomes, such as elastic net, random forest, k nearest neighbours, ensembles, etc. (Dinga et al., 2018; Lutz et al., 2005; Pearson et al., 2019; Webb et al., 2020). Treatment selection models for individual patients have been developed by applying the Personalized Advantage Index and different ML approaches (e.g., Cohen et al., 2020; Cohen & DeRubeis, 2018; Deisenhofer et al., 2018; Delgadillo & Gonzalez Salas Duhne, 2020; DeRubeis et al., 2014; Huibers et al., 2015; Kessler et al., 2019; Lorenzo-Luaces & DeRubeis, 2018; Schwartz et al., 2020). Furthermore, ML has been used to predict the risk of dropout for individual patients based on their intake variables (Lutz et al., 2019) and ambulatory assessments before initiating treatment (Lutz et al., 2018). Machine learning is a potentially useful tool to improve prediction on an individual patient-level, as it can model linear as well as non-linear and complex relationships (i.e., interactions) between variables (Makridakis et al., 2018).

In contrast to the prediction of continuous outcomes, predictions of treatment dropout are based on imbalanced datasets (i.e., a disproportionate ratio of observations), with the rare class being the one of interest. Ignoring class imbalance may lead to adverse consequences regarding model estimation and the evaluation of the model’s accuracy, as classification methods tend to favour the majority class and neglect the minority class (Hand & Vinciotti, 2003; Japkowicz & Stephen, 2002; Menardi & Torelli, 2014). This holds true for different kinds of classifiers, such as logistic regression, linear discriminant analysis, classification trees, k nearest neighbours, neural networks or support vector machines (Akbani et al., 2004; Chawla, 2003; Cieslak & Chawla, 2008; Cramer, 1999; Hand & Vinciotti, 2003; King & Ryan, 2002; King & Zeng, 2001; Kubat & Matwin, 1997; Lawrence et al., 1998; Owen, 2007). Imbalanced data can be addressed either by data pre-processing (e.g., resampling methods, feature selection and extraction, cost-sensitive weighting) or algorithm centred approaches (e.g., cost-sensitive learning or one-class learning) as well as hybrid methods combining data pre-processing and algorithm centred approaches (Kaur et al., 2020). This paper addresses solutions on the data level referred to as resampling methods, which aim to generate two classes of equal size, as well as feature selection via least absolute shrinkage and selection operator (LASSO) regression (Guo et al., 2015).

Apart from class imbalance, the predictive power of a classifier depends largely on the size and quality of the training sample (Cohen et al., 2020; Dobbin et al., 2008; Kalayeh & Landgrebe, 1983; Kim, 2009; Mukherjee et al., 2003; Nigam et al., 2000; Tam et al., 2006). However, how large a training sample has to be to achieve good predictions is still a matter of methodological debate (Balki et al., 2019). In psychotherapy research, most studies on dropout prediction are limited by relatively small samples. Therefore, evaluating the performance of ML algorithms in differently sized samples is an important task to advance dropout prediction in psychotherapy research.

This methodological paper aims at improving the prediction of dropout from psychological interventions by comparing the performance of 20 different ML algorithms in twelve differently sized subsamples under the influence of four different resampling methods as well as without resampling. The results can act as an evidence-based guideline to set a standard for, support researchers in modelling dropout predictions, and increase comparability of findings.

Method

Setting and Interventions

This study was conducted using multi-service archival data from sixteen psychological therapy teams covering a geographically and socioeconomically diverse clinical sample across England (London, Cambridge, Cheshire & Wirral, Bury, Heywood, Middleton, Rochdale, Oldham, Stockport, Tameside & Glossop, Trafford, Barnsley, and East Riding). The London - City & East NHS Research Ethics Committee (06/01/2016, Ref: 15/LO/2200) approved the assembly and analysis of a fully anonymised dataset. Verbal consent was obtained from patients and documented in their clinical records. The data was collected for all patients who accessed the participating services within a three-year data collection period (2014–2017).

The participating psychological services were part of the Improving Access to Psychological Therapies (IAPT) programme, a national treatment system offering time-limited and evidence-based psychological interventions for depression and anxiety that are delivered in a stepped care model (Clark, 2011; NICE, 2011). Initially, most patients access low intensity cognitive behavioural therapy (LiCBT), which consists of a short (up to eight sessions) guided self-help intervention based on CBT principles. LiCBT is highly structured and follows treatment protocols based on a national training curriculum (National IAPT Team, 2015). LiCBT is delivered by qualified practitioners under the weekly supervision from senior practitioners and psychotherapists. In the present sample, 88% of LiCBT interventions involved individual support, ∼10% were delivered in groups, and ∼2% were delivered by blended care (online CBT and telephone support).

More severe and complex cases, or patients who do not respond to LiCBT, are offered more intensive and costly psychotherapies that can last up to 20 sessions, such as formal CBT, person-centred experiential counselling, and interpersonal psychotherapy, and is based on evidence-based protocols recommended by national clinical guidelines (National Institute for Health and Care Excellence, 2011).

In this study, only cases receiving LiCBT were examined as it was the larger sample (80.5% of patients accessed LiCBT) and to exclude treatment intensity as a possible confounder. To obtain a sample that most clearly represents LiCBT, only patients who started and ended with LiCBT and had at least one session of LiCBT were included.

Measures

IAPT services collect three validated patient-reported measures on a session-to-session basis as part of routine outcome monitoring (Clark, 2011). The Patient Health Questionnaire-9 (PHQ-9; Kroenke et al., 2001) measures depression symptoms, with nine items rated from 0 to 3, resulting in an overall severity score ranging between 0 and 27. The GAD-7 (Kroenke et al., 2007) is a measure of anxiety disorder, with seven items rated from 0 to 3, resulting in an overall severity score ranging between 0 and 21. The Work and Social Adjustment Scale (WSAS; Mundt et al., 2002) rates functional impairment across five domains on a scale from 0 to 8: work, home management, social life, private leisure activities, and family relationships. Additional demographic data included age, gender, employment status, diagnosis, and the Index of Multiple Deprivation (IMD; Table S1). The IMD is a measure of socioeconomic deprivation for neighbourhoods in England (Department for Communities and Local Government, 2015), aggregated into decile groups where 1 = most deprived and 10 = least deprived areas.

Dropout Definition

Dropout (vs. treatment completion) was determined by the therapist and documented in routine clinical records for all patients. It was defined as the one-sided decision of the patient to end therapy before the end of the scheduled treatment. However, patients that dropped out of therapy in the IAPT services might have sought help in other services, though this is beyond the information available from clinical records. Patients who fell in other categories denoting reasons for the end of treatment which did not fit with either “completion” or “dropout” were excluded from the sample. This refers to patients that were for example unsuitable for treatment in IAPT services (i.e., acute suicidality, psychosis, or organizational reasons), patients that moved away or died.

Statistical Analyses

Sample Selection. The complete procedure of sample selection is illustrated in Figure S2 in the supplementary material. From the 157,946 cases in the full dataset, 25,369 had to be removed either because of missing data in the dependent variable, no clear way of categorizing cases to dropout or completer cases, or more than 50% missing values. To maintain comparability, high intensity psychotherapy cases were excluded, and all analyses were run exclusively on the LiCBT sample (n = 49,602).

These cases were split into a holdout sample (30%, n = 14,881) and a training sample (70%; n = 34,721). Twelve subsamples were selected randomly from the training sample, ranging from 50 to 10,000 cases, with a finer-grained spacing between smaller samples (i.e., 50; 75; and 100) than larger samples (i.e., 2,500; 5,000; and 10,000). All reported results are based on the complete holdout sample, to compare the performance of different ML algorithms in new data and to improve generalizability.

Missing Data. Missing data points in the potential predictor variables were imputed separately for the training (n = 34,721) and holdout sample (n = 14,881) via the R package missForest version 1.4 (Stekhoven & Bühlmann, 2012) to analyse the whole sample of patients seeking treatment in LiCBT instead of only the highly committed subsample of patients who completed all questionnaires. MissForest uses random forest to impute missing values and can handle continuous and categorical data in addition to nonlinear associations and interactions. As simulation studies have shown that missForest performs well for up to 30% of missing values, variables with more than 30% missing values and cases with more than 50% of missing values were excluded from the analysis (Stekhoven & Bühlmann, 2012).

Predictor Transformation and Selection. All continuous predictors were standardized, and categorical predictors were dichotomized. All 14 available predictors (Table S1) were entered into a LASSO regression (Guo et al., 2015) to select relevant features. This was done using the R-Package glmnet (Friedman et al., 2010) with a ten-fold cross-validation. Glmnet chooses the hyperparameter lambda empirically. Predictor selection was performed on the complete training sample (n = 34,721) and not for each subsample separately as this paper aimed at examining the performance of the prediction models in the different sample sizes and not the influence of varying predictors. The LASSO approach is considered to be a parsimonious and conservative method for variable selection, which prioritizes out-of-sample generalizability and is robust to multicollinearity (Tibshirani, 1996).

Resampling Methods. The analyses were run with five different approaches: [1] no resampling, [2] up-sampling, [3] down-sampling, [4] synthetic minority oversampling technique (SMOTE), and [5] random oversampling examples (ROSE). Up-sampling of the minority class seeks to equalise class imbalance by randomly duplicating cases in the minority class (e.g., patients that dropped out of therapy). It may lead to overfitting by copying the same information and greatly increases the computational effort. Down-sampling consists of randomly excluding cases from the majority class (e.g., patients that completed treatment). It reduces the sample size and may lead to a loss of information, which raises the potential for underfitting and poor generalization to the holdout sample (Estabrooks et al., 2004; Japkowicz & Stephen, 2002; McCarthy et al., 2005). SMOTE and ROSE are hybrid methods combining up and down sampling. SMOTE generates new examples for each rare training observation using a nearest neighbours modelling approach (Chawla et al., 2002). ROSE creates artificially balanced samples generated with a smoothed bootstrap approach (Lunardon et al., 2014).

ML Algorithms. Guided by the SMART Mental Health Prediction Tournament (Cohen et al., 2018) we selected a total of 20 machine learning algorithms (Figure S1). All algorithms applied repeated cross-validation with ten splits and five repeats. All models were run using the R package caret, which tunes parameters and selects the optimal model autonomously (Kuhn, 2008). ML algorithms were trained in the training samples and validated in the holdout sample. For the sake of brevity, we provide only a short overview of each of the applied ML algorithm groups (regression, Bayesian, decision tree, instance-based, regularization, artificial neural networks, dimensionality reduction, and ensemble), grouped by their function.

Regression algorithms are modelling the relationship between variables and are iteratively refined by using a measure of error in the models’ predictions. The algorithms Generalized linear model (glm) and support vector machine (svmLinear2) were applied from this group. Bayesian algorithms apply Bayes theorem of probability for classification or regression problems. From this group Naïve Bayes (naivebayes) and Bayesian Generalized Linear Model (bayesglm) were applied. Decision Tree algorithms construct models of decisions based on the values of features in the data. Decisions branch out in tree-like structures until a prediction is made. They can be used for classification and regression problems and are often fast and fairly accurate. The algorithm Conditional Inference Trees (ctree2) was used in the analysis. Instance-based algorithms base predictions on the best match to a given case by using similarity measures. From this group the algorithm k-nearest neighbours (knn) was used. Regularization algorithms penalize models based on their complexity and favour simpler, more generalizable models. Elastic net (glmnet) was applied from this group. Artificial Neural networks are influenced by the function and/or structure of biological neural networks. Three neural networks were used: feed-forward neural network (Nnet), model averaged neural network (avNNet), and monotone multi-layer perceptron (monmlp). Dimensionality reduction algorithms explore and exploit the structure of the data. From this group linear discriminant analysis (LDA) was applied. Ensemble algorithms are comprised of multiple, independently trained models and make overall predictions based on the combination of these models. This is the largest category of algorithms with nine different algorithms: Bayesian additive regression trees (BartMachine), boosted generalized linear model (glmboost), boosted classification tress (ada), bagged CART (treebag), bagged MARS (bagEarth), random forest (rf), extreme gradient boosting (xgbTree), boosted logistic regression (LogitBoost), and stochastic gradient boosting (gbm; Brownlee, 2019).

Evaluation Measure. The selection of the appropriate evaluation metric depends on the classification problem and no metric can generally be considered the best. However, when evaluating imbalanced data Brier score, accuracy and area under the curve (AUC) have recently been criticised as values for selecting the best performing algorithm (He & Garcia, 2009; Menardi & Torelli, 2014; Weiss, 2004; Weiss & Provost, 2001). Overall the Brier score may show a good calibration, but might perform poorly in the classification of the rare class (Wallace & Dahabreh, 2014). Algorithm performance may gain the best overall accuracy by classifying each case in the majority group. If the minority class represents only 1% of the data and the algorithm classifies every case in the majority class, the overall accuracy would be .99. In a simulation study Jeni et al. (2013) suggest that the receiver operating characteristics (ROC) curves may mask poor algorithm performance and ROC curves do not take class distribution and different misclassification costs into account (Menardi & Torelli, 2014).

Overcoming the problems of these evaluation measures, the F₁-Score measures the accuracy of a binary classification by weighting precision and recall equally: $F_1=2\times {\displaystyle \frac{\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}\times \mathrm{R}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l}}{\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}+\mathrm{R}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l}}}$with precision describing the sum of true positives divided by the sum of predicted positives and recall describing the sum of true positives divided by the sum of actual positives. The F₁-Score ranges from 0 to 1, with 1 indicating a perfect precision and a perfect recall. It is commonly used when learning from imbalanced datasets (Weiss, 2013) and in fields with similar objectives such as cancer research (e.g., Jiao et al., 2020; Li et al., 2018; Mohammed et al., 2021).

With a base rate of dropout in the holdout sample of .33, and assuming a chance level of .50, the threshold for a model performing better than chance was an F₁-Score of $2\ast {\displaystyle \frac{0.33\ast 0.5}{0.33+0.5}=.40}$. Additionally, other common indices of performance were also computed, including Brier scores, AUC, sensitivity, and specificity.

Results

For simplicity, the focus of the results is placed on the F₁-Scores as the primary performance index. Further information regarding other performance measures (AUC, accuracy, balanced accuracy, sensitivity, specificity, true positive, true negative, false positive, and false negative values) are only reported in brevity and more detail is contained in the supplementary materials. Seven predictors of dropout were selected by LASSO regression: Age (younger patients), ethnicity (ethnic minority), unemployment, IMD (more socioeconomically deprived neighbourhoods), and higher baseline severity on the PHQ-9, the GAD-7, and WSAS (Table I).

Table I. Sample characteristics of the predictors selected by LASSO regression for the complete sample, the training sample, and the holdout sample.

	Complete sample LiCBT (N = 49,602)	Training sample LiCBT (n = 34,721)	Holdout sample LiCBT (n = 14,881)
Demographics
Age (years)	40.67 (15.36)	40.66 (15.33)	40.70 (15.42)
Unemployed	10,310 (20.79)	7,320 (21.08)	2,990 (20.01)
Ethnicity
White British	42,450 (85.58)	29,757 (85.70)	12,693 (85.30)
Other	7,152 (14.42)	4,964 (14.30)	2,188 (14.70)
IMD	4.94 (2.75)	4.93 (2.75)	4.95 (2.76)
Baseline
PHQ-9	14.22 (6.18)	14.22 (6.19)	14.24 (6.15)
GAD-7	13.04 (5.11)	13.02 (5.12)	13.09 (5.07)
WSAS	18.56 (9.67)	18.58 (9.68)	18.52 (9.67)
Dropout (%)	16,632 (33.53)	11,730 (33.78)	4,902 (32.94)

Note: Data are M (SD) or n (%). PHQ-9 = Patient Health Questionnaire-9; GAD-7 = Generalised Anxiety Disorder-7; WSAS = Work and Social Adjustment Scale; IMD = Index of Multiple Deprivation.

The average F₁-Scores across all algorithms and subsamples for each resampling method ranged from .32 to .46. The highest mean F₁-Score of .46 was achieved with the down-sampling method, followed by the up-sampling method with a mean F₁-Score of .45. The smote resampling method reached a mean F₁-Score of .44 and the ROSE method reached a mean F₁-Score of .43. Using no resampling methods resulted in a mean F₁-Score of .32 (Table S2; Figure 1).

Figure 1. Mean F₁-scores across all algorithms and sample size for the different resampling methods.

The average true positive values (i.e., correctly identified dropout cases) across all algorithms and subsamples for each method ranged from 1377 to 2773 cases. The highest mean true positive value of 2773 cases was achieved by the down-sampling method, followed by the up-sampling method with a mean true positive value of 2642 cases. The smote resampling method reached a mean true positive value of 2494 cases, and the ROSE method reached a mean true positive value of 2585 cases. The lowest mean true positive value of 1377 was reached when no resampling method was applied (Tables S9–S12; Figures S9–S12).

Within the resampling methods, the threshold of a mean F₁-Score of .40 was exceeded in all subsamples. The highest F₁-Score of .47 was reached in the subsample of 750 cases using the down-sampling method. All subsequent results presented are restricted to those observed in the algorithms using the down-sampling method (Table II).

Table II. F₁-scores of all algorithms in each sample size using the down sampling method.

	Sample Size
Algorithm^1	50	75	100	200	300	400	500	750	1000	2500	5000	10000	mean
glm	.47	.46	.46	.47	.48	.49	.49	.49	.49	.50	.49	.50	.48
bayesglm	.46	.48	.50	.47	.49	.49	.49	.49	.49	.50	.50	.50	.49
naivebayes	.46	.47	.45	.47	.45	.46	.45	.47	.46	.48	.45	.46	.46
svmLinear2	.49	.48	.46	.50	.50	.49	.50	.50	.50	.50	.50	.50	.49
knn	.44	.47	.44	.43	.45	.45	.44	.46	.46	.46	.45	.45	.45
ctree2	.34	.43	.48	.48	.49	.49	.49	.48	.47	.46	.43	.49	.46
nnet	.44	.46	.49	.42	.46	.46	.51	.49	.50	.49	.49	.49	.48
monmlp	.39	.39	.39	.50	.51	.51	.51	.50	.50	.49	.49	.49	.47
avNNet	.47	.43	.48	.43	.47	.45	.47	.47	.49	.49	.49	.49	.47
glmnet	.46	.48	.48	.48	.49	.48	.49	.49	.50	.50	.50	.50	.49
lda	.48	.47	.47	.46	.49	.49	.49	.49	.50	.50	.50	.50	.49
gbm	NA	.44	.47	.47	.49	.47	.48	.49	.48	.48	.50	.49	.48
rf	.48	.45	.48	.46	.47	.45	.46	.47	.47	.46	.47	.46	.47
glmboost	.46	.48	.49	.48	.48	.49	.49	.50	.49	.50	.50	.50	.49
ada	.33	.33	.34	.32	.32	.32	.32	.31	.31	.31	.31	.30	.32
xgbTree	.46	.44	.45	.47	.47	.48	.47	.49	.49	.49	.49	.50	.48
LogitBoost	.44	.45	.45	.47	.49	.48	.40	.47	.45	.33	.48	.43	.45
treebag	.49	.46	.42	.44	.45	.46	.46	.45	.46	.45	.44	.45	.45
bagEarth	.34	.38	.31	.34	.34	.32	.34	.31	.29	.31	.31	.30	.32
bartMachine	.47	.47	.48	.47	.49	.48	.49	.49	.49	.49	.48	.49	.48
mean	.44	.45	.45	.45	.46	.46	.46	.47	.46	.46	.46	.46	.46
SD	.05	.04	.05	.05	.05	.05	.05	.05	.06	.06	.06	.06	.05
CI lower	.42	.43	.43	.43	.44	.44	.44	.44	.44	.43	.44	.44	.43
CI upper	.46	.46	.47	.47	.49	.48	.49	.49	.49	.49	.49	.49	.48

Note: Best performing algorithms per sample size highlighted in bold and worst performing algorithms per sample size highlighted in italic. ¹ Algorithm names of the r packages are displayed. glm = Generalized Linear Model; bayesglm = Bayesian Generalized Linear Model; naivebayes = Naïve Bayes; svmLinear2 = Support Vector Machine; knn = k-Nearest Neighbours; ctree2 = Conditional Inference Trees; nnet = Feed-Forward Neural Network; monmlp = Monotone Multi-layer Perceptron; avNNet = Model Averaged Neural Network; glmnet = Elastic Net; lda = Linear Discriminant Analysis; gbm = Stochastic Gradient Boosting; rf = Random Forest; glmboost = Boosted Generalized Linear Model; ada = Boosted Classification Tress; xgbTree = Extreme Gradient Boosting; LogitBoost = Boosted Logistic Regression; Treebag = Bagged CART; bagEarth = Bagged MARS; bartMachine = Bayesian Additive Regression Trees.

The three best and worst performing algorithms in each subsample using the down-sampling method are depicted in Figure 2 and highlighted in bold and italic in Table II. The best performing algorithm groups were linear algorithms, artificial neural networks, ensemble algorithms, decision trees and dimensionality reduction. Among the three best performing algorithms per subsample, linear algorithms (generalized linear model; mean F₁-Score = .48, Bayesian generalized linear model, support vector machine, elastic net; mean F₁-Score = .49) were selected 28 times, artificial neural networks (feed-forward neural network; mean F₁-Score = .48, model averaged neural network, monotone multi-layer perceptron; mean F₁-Score = .47) 9 times, and ensembles algorithms (stochastic gradient boosting, extreme gradient boosting, Bayesian additive regression trees; mean F₁-Score = .48, boosted generalized linear model; mean F₁-Score = .49; boosted logistic regression, bagged CART; mean F₁-Score = .45) 13 times. Additionally, there were three standalone algorithms that were not grouped with other algorithms. The decision tree algorithm conditional inference trees (mean F₁-Score = .46) was selected three times, the dimensionality reduction algorithm linear discriminant analysis (mean F₁-Score = .49) was selected seven times and random forest (mean F₁-Score = .47) was selected once.

Figure 2. Best and worst performing algorithms in each sample size using the down-sampling method.

Note: The F₁-Scores are centred at the threshold of .40 and the deviation from the threshold is shown in the bars. If multiple algorithms reached the same F₁-Score, they were all selected and grouped together.

The algorithms with the highest mean F₁-Scores across all subsamples were naïve Bayes, support vector machine, elastic net, linear discriminant analysis and boosted generalized linear model with a mean F₁-Score of .49 each. The algorithm naïve Bayes was selected in seven samples (75; 100; 300; 400; 2,500; 5,000; and 10,000), the support vector machine was selected in eleven samples (all samples except for 100), and the algorithm elastic net was selected in seven samples (75; 200; 300; 1,000; 2,500; 5,000; and 10,000) as one of the best algorithms. The dimensionality reduction algorithm linear discriminant analysis was selected in seven samples (50; 300; 400; 1,000; 2,500; 5,000; and 10,000) and the ensemble algorithm boosted generalized linear model was selected in eight samples (75; 100; 200; 400; 750; 2,500; 5,000; and 10,000) as one of the best algorithms (Table II).

Secondary outcomes are depicted in the supplementary material, containing the full results for the Brier Score (Table S3; Figure S3), the accuracy (Table S4; Figure S4), the AUC (Table S5; Figure S5), the sensitivity (Table S6; Figure S6), the specificity (Table S7; Figure S7), and the balanced accuracy (Table S8; Figure S8). Additionally, a confusion matrix (Figure S9) illustrates the calculation methods of the different indices.

In the following results of some secondary outcome measures are reported. The mean Brier scores (Table S3; Figure S3) for predictions made without using resampling methods was slightly lower (mean = .24) than the mean Brier score across all resampling methods (mean = .26). The mean accuracy (Table S4; Figure S4) of the predictions made without resampling methods was higher (mean = .62) than the mean accuracy across all resampling methods (mean = .57). The balanced accuracy (Table S8; Figure S8) for predictions made without resampling methods was lower (mean = .53) than the mean accuracy across all resampling methods (mean = .56). The mean AUC scores (Table S5; Figure S5) for predictions made with or without resampling were identical (mean = .60). The mean sensitivity (Table S6; Figure S6) for predictions made without resampling was lower (mean = .28) than for predictions made with resampling (mean = .54). The mean specificity (Table S7; Figure S7) for predictions made without resampling was higher (mean = .79) than for predictions made with resampling methods (mean = .58).

Discussion

This paper examined the influence of resampling methods and sample size on the prediction of the occurrence of dropout in psychological treatment using several state-of-the-art ML algorithms. To our knowledge, no previous study has compared algorithm performance in dropout prediction using different resampling methods and varying sample sizes.

Main Findings

Resampling methods improved the averaged F₁-Scores across all algorithms and samples. Therefore, we recommend employing resampling when developing ML models for dropout prediction. Between the four resampling methods, however, no method can be identified as being the best method, as the confidence intervals are overlapping. Nevertheless, the resampling methods differ in their requirement of computational power and time consumption, with the down-sampling method being the least computationally taxing and the fastest method among the examined methods. When dealing with large data sets or when computational power is limited, we recommend applying the down-sampling method.

Increasing the sample size slightly improved the predictive power when using resampling methods and remained stable in samples containing 300 cases or more. Accurate predictions, therefore, can be achieved in samples as small as N = 300. This finding represents an empirically-based threshold for sample sizes when predicting treatment dropout using ML algorithms, and it adds to the ongoing debate on sample size recommendations for classifiers (Balki et al., 2019). It is in line with recommendations by Luedtke et al. (2019) who recommend a minimum sample size of n = 300 patients per treatment arm for treatment selection models.

To identify the models that are best suited for dropout prediction, 20 different ML approaches were compared in the present study. Although linear methods and artificial neural networks performed slightly better than other methods, the differences between the algorithms are not substantial enough to justify a recommendation for specific algorithm groups or individual algorithms.

The Brier score (Table S3; Figure S3) was slightly lower when not using resampling methods, which implies a better prediction. This however may be due to a systematic underestimation of the minority class, i.e., the dropout cases (Wallace & Dahabreh, 2014). The accuracy (Table S4; Figure S4) is higher when not using resampling methods. This is most likely due to the data imbalance and the correct classification of completer cases (He & Garcia, 2009; Menardi & Torelli, 2014; Weiss, 2004; Weiss & Provost, 2001). The balanced accuracy (Table S10, Figure S10) seems to confirm this, as the predictions made with resampling methods performed better than prediction without resampling. The AUC is not influenced by the use or the absence of resampling methods. This is likely due to the fact that the AUC does not take different misclassification costs or class distribution into account (Menardi & Torelli, 2014). The sensitivity (Table S6, Figure S6) is higher when using resampling methods, indicating that more dropout cases were identified correctly when using resampling methods. Conversely the specificity (Table S7, Figure S7) is higher when not using resampling methods, indicating that more completer cases were identified correctly when not using resampling methods. This is mostly due to the fact that sensitivity and specificity are one-dimensional performance metrics and are therefore not subjected to the bias of data imbalance (Luque et al., 2019). The mean true positive values (i.e., actual dropout = predicted dropout) are higher when using resampling methods, while the mean true negative values (i.e., actual completer = predicted completer) are higher when not using resampling methods (Tables S9 – S10; Figures S9 – S10).

Limitations and Future Directions

Methodological limitations of this study are related to the selection of the predictors, the sample selection, the selection of the algorithms, and the structure of the data. First, predictor selection for dropout via LASSO regression was done on the complete training sample to avoid different predictors in different subsamples impairing the comparability of performance across different algorithms (Guo et al., 2015). Nevertheless, we acknowledge that a different subset of predictors may have been derived by different approaches to variable selection, though a comparison of variable selection techniques was outside of the scope of the present study and keeping predictors stable across all calculations was paramount to make a comparison. Preselecting relevant features leads to more parsimonious models and reduces the requirements for computational power.

Second, each sample size was selected only once. A repeated selecting of the subsamples would improve the stability of the F₁-Scores, especially in the smaller samples. This was done for computational reasons, as this study already required a large computational power, even without repeated sample selection.

Third, we selected 20 algorithms that represent different families of ML approaches, but there are many more algorithms available. Therefore, the possibility that there might be other algorithms better suited to predict dropout cannot be excluded. However, the results suggest that there is not one best algorithm to predict dropout from this IAPT dataset. Furthermore, the default calibration settings for algorithms provided by the R package caret (Kuhn, 2008) were applied. Caret tunes the algorithm parameters automatically, but manual fine-tuning might have improved the predictions.

Finally, the study is limited by the structure of the data, as potentially relevant predictors of dropout, such as personality styles or disorder as well as education level, were not available (Karterud et al., 2003; McMurran et al., 2010; Swift & Greenberg, 2012), dropout was determined exclusively by the treating therapist and there were cases that could not definitively be categorized into dropout, such as patients deemed not suitable for the IAPT system. Dropout from psychological interventions is a complex subject and identified predictors of dropout are heterogeneous, therefore, dropout prediction still proves to be difficult.

Further research is needed to improve dropout predictions via ML. The high prevalence of dropout in psychological interventions and its high costs for the patients’ health, the society at large, and the economy make the improvement of dropout prediction a highly important research topic (e.g., Barrett et al., 2008; Delgadillo et al., 2014; Druss & Rosenheck, 1999; Karterud et al., 2003; Sainsbury Centre for Mental Health, 2003). A reliable dropout prediction for individual patients could have important practical implications for clinicians and improve the personalization of psychological interventions. Including a dropout prediction for individual patients based on their intake variables in a feedback system can alert therapists to a risk of premature termination of treatment (Lutz et al., 2019). However, such prediction tools will only be useful or acceptable to clinical services if they are accurate. Therefore, this paper makes an important contribution to guide the field on how to train accurate models. Similar to cancer screenings, a false positive alert of a potential dropout can be seen as less detrimental as missing a dropout case. An alert can, however, lead to further clinical exploration through the therapist. Additionally to integrating dropout prediction in a feedback system for therapists, prospective studies are important to determine the reliability of the predictions in a clinical setting (Lutz, Deisenhofer, et al., 2021).

Conclusions

Despite the above-mentioned limitations, the present findings can support future research on dropout prediction by providing empirically supported parameters for sample size recommendation and for the selection of ML algorithms. Results suggest that a minimum of 300 patients might be necessary to build accurate prediction models and that the predictive accuracy of a model is improved when using resampling methods to handle imbalanced data. The down-sampling method can be recommended as it is the least computationally taxing and the fastest method. When integrated into a feedback system for therapists improved dropout predictions could result in a more accurate clinical prognosis, enhance case conceptualization and clinical decision-making (Cohen et al., 2018; Delgadillo & Lutz, 2020; Lutz et al., 2015, 2019; Lutz, Schwartz, et al., 2021). Identifying patients at risk for dropout reliably and early in treatment, or even before the start of treatment, might enable therapists to consider this risk and to apply clinical techniques to improve motivation, alliance or treatment expectations, thus potentially reducing dropout and thereby enhancing treatment outcomes and limiting negative consequences of premature treatment termination. However, we acknowledge that further research into the accuracy of dropout prediction and their potential real-life benefits are required. Prospective studies where dropout predictions are integrated into a feedback system are necessary to evaluate the usefulness in reducing dropout from psychological interventions.

Data Sharing and Data Accessibility

The data that support the findings of this study are available on request from the corresponding author. The data are not publicly available due to privacy or ethical restrictions.

Disclosure Statement

No potential conflict of interest was reported by the author(s).

Supplemental data

Supplemental data for this article can be accessed https://dx.doi.org/10.1080/10503307.2022.2161432.