Component studies of psychological treatments of adult depression: A systematic review and meta-analysis

Abstract

Objectives: A recent report from the US Institute of Medicine indicated that identifying core elements of psychosocial interventions is a key step in successfully bringing evidence-based psychosocial interventions into clinical practice. Component studies have the best design to examine these core elements. Earlier reviews resulted in heterogeneous sets of studies and probably missed many studies. Methods: We conducted a comprehensive search of component studies on psychotherapies for adult depression and included 16 studies with 22 comparisons. Results: Fifteen components were examined of which four were examined in more than one comparison. The pooled difference between the full treatments and treatments with one component removed was g = 0.21 (95% CI: 0.03∼0.39). One study had sufficient statistical power to detect a small effect size and found that adding emotion regulation skills increased the effects of CBT. None of the other studies had enough power to detect an effect size smaller than g = 0.55. Only one study had low risk of bias. Conclusions: The currently available component studies do not have the statistical power nor the quality to draw any meaningful conclusion about key ingredients of psychotherapies for adult depression.

Clinical or methodological significance of this article

A recent report from the Institute of Medicine pointed once again at the need for identifying the key elements of psychosocial interventions that drive an intervention’s effect. Component studies are an important tool for examining how therapies work and they provide an elegant way to identify the active ingredients of psychotherapies. These studies decompose multicomponent therapies and compare the full therapy with a therapy in which one component is left out (dismantling studies) or in which a component is added to an existing therapy. We conducted a systematic review of component studies by searching an existing database of randomized trials examining the effects of psychological treatments of adult depression. This had the advantage that we probably missed few component, in contrast to earlier reviews, because component studies often do not use this design as index term.

Obiettivi: Un recente report dall'Istituto di Medicina USA ha indicato che identificare gli elementi centrali degli interventi psicosociali rappresenta un passaggio chiave al fine di portare con successo nella pratica clinica interventi basati sull'evidenza. I “component studies” hanno come obiettivo principale esaminare questi elementi centrali. Le prime reviews erano derivate da un set eterogeneo di studi e probabilmente mancavano molti lavori. Metodo: Abbiamo condotto una ricerca completa dei “component studies” su psicoterapie per la depressione adulta e incluso 16 studi con 22 confronti. Risultati: Quindici componenti erano esaminati di cui quattro in più di un confronto. La differenza cumulata tra i trattamenti completi e i trattamenti con una componente rimossa era g=0.21 (95% CI: 0.03∼0.39). Uno studio aveva una potenza statistica sufficiente da rilevare un piccolo effect size e trovato che aggiungendo le capacità di regolazione delle emozioni aumentavano gli effetti della CBT. Nessuno degli altri studi aveva abbastanza potere da rilevare un effect size minore di g=.55. Un solo studio aveva un basso rischio di bias. Conclusioni: I “component studies” attualmente disponibili non hanno potenza statistica nè la qualità per delineare alcuna conclusione significativa sugli elementi chiave delle psicoterapie per la depressione adulta.

Significato clinico o metodologico di questo articolo: Un recente report dall'Istituto di Medicina USA ha sottolineato ancora una volta la necessità di identificare gli elementi chiave degli interventi psicosociali che conducono all'effetto di un intervento. I “component studies” sono un importante strumento per esaminare come le terapie funzionano e forniscono un modo elegante per identificare gli elementi attivi delle psicoterapie. Questi studi decostruiscono le terapie multicomponenti e mettono a confronto la terapia globale con una terapia in cui una componente è lasciata fuori (studi di smantellamento) o in cui una componente è aggiunta a una terapia esistente. Abbiamo condotto una rassegna sistematica dei “component studies” attraverso una ricerca sui database esistenti di studi randomizzati esaminando gli effetti dei trattamenti psicologici della depressione adulta. Ciò ha avuto il vantaggio che probabilmente sono mancati pochi componenti, al contrario delle precedenti rassegne, perché i “component studies” spesso non usano questo disegno come termine indicizzato.

Zusammenfassung Fragestellung: Ein aktueller Bericht des US Institute of Medicine ergab, dass die Identifizierung von Kernelementen psychosozialer Interventionen ein wichtiger Schritt ist, um evidenzbasierte psychosoziale Interventionen erfolgreich in die klinische Praxis einzubringen. Komponentenstudien haben das beste Design, um diese Kernelemente zu untersuchen. Frühere Reviews führten zu heterogenen Studiensets und haben wahrscheinlich viele Studien verpasst. Methode: Wir führten eine umfassende Suche nach Komponentenstudien zu Psychotherapien bei Erwachsenen für Depression durch und schlossen 16 Studien mit 22 Vergleichen ein. Ergebnisse: Es wurden fünfzehn Komponenten untersucht, von denen vier in mehr als einem Vergleich untersucht wurden. Die gepoolte Differenz zwischen den Vollbehandlungen und den Behandlungen mit einer entfernten Komponente betrug g=0,21 (95% CI: 0,03∼0,39). Eine Studie hatte eine ausreichende statistische Power, um eine kleine Effektgröße zu erkennen, und fand heraus, dass das Hinzufügen von Fähigkeiten zur Emotionsregulation die Wirkung von CBT erhöht. Keine der anderen Studien hatte genug Power, um eine Effektgröße kleiner als g=0,55 zu erkennen. Nur eine Studie hatte ein geringes Bias-Risiko. Schlussfolgerung: Die derzeit verfügbaren Komponentenstudien haben weder die statistische Power noch die Qualität, um aussagekräftige Schlussfolgerungen über die wichtigsten Inhalte von Psychotherapien für Depressionen bei Erwachsenen zu treffen.

Objetivos: Um relatório recente do Instituto de Medicina dos EUA indicou que a identificação de elementos centrais das intervenções psicossociais é um passo fundamental para levar com sucesso as intervenções psicossociais baseadas em evidências para a prática clínica. Os estudos de componentes têm o melhor design para examinar esses elementos principais. Revisões anteriores resultaram em conjuntos heterogêneos de estudos e provavelmente perderam muitos estudos. Métodos: Realizamos uma pesquisa abrangente de estudos de componentes em psicoterapias para depressão em adultos e incluímos 16 estudos com 22 comparações. Resultados: Quinze componentes foram examinados, dos quais quatro foram examinados em mais de uma comparação. A diferença combinada entre os tratamentos completos e tratamentos com um componente removido foi g = 0,21 (IC 95%: 0,03∼0,39). Um estudo teve suficiente poder estatístico para detectar um pequeno tamanho de efeito e descobriu que adicionar habilidades de regulação de emoções aumentava os efeitos da TCC. Nenhum dos outros estudos tinham poder suficiente para detectar um tamanho de efeito menor que g = 0,55. Apenas um estudo apresentou baixo risco de viés. Conclusões: Os estudos de componentes disponíveis atualmente não têm o poder estatístico nem a qualidade para obter conclusão significativa sobre os principais ingredientes das psicoterapias para a depressão em adultos.

Significância clínica ou metodológica deste artigo: Um relatório recente do Instituto de Medicina apontou mais uma vez a necessidade de identificar os elementos-chave das intervenções psicossociais que impulsionam o efeito de uma intervenção. Estudos de componentes são uma ferramenta importante para examinar como as terapias funcionam e fornecem uma maneira elegante de identificar os ingredientes das psicoterapias. Estes estudos decompõem terapias multicomponentes e comparam a terapia completa com uma terapia em que um componente é deixado de fora (estudos de desmantelamento) ou em que um componente é adicionado a uma terapia existente. Realizamos uma revisão sistemática de estudos de componentes, pesquisando um banco de dados existente de ensaios randomizados examinando os efeitos dos tratamentos psicológicos da depressão em adultos. Isso teve a vantagem de provavelmente perdermos poucos componentes, em contraste com as revisões anteriores, porque os estudos de componentes geralmente não usam esse design como termo de índice.

目的：美國國家醫學研究院最近的一份報告指出， 辨識出心理社會介入的核心要素是成功地將實證為基礎的心理社會介入引進臨床實務的關鍵步驟。成份研究具有檢視這些核心要素的最佳設計。早期的回顧研究產生參差不齊的研究並可能遺漏許多研究。方法：我們針對為成人憂鬱進行的心理治療進行詳盡的成份蒐尋研究，包括16項研究，涵蓋22組比較。結果：共檢視15個成份，其中四個成份有進行多於一組的比較。比較完整治療與移除一項成份的治療，其合併差異為 g=0.21（95% CI: 0.03∼0.39）。其中一項研究具備足夠的統計考驗力，能偵測到小的效果值，並且發現添加情緒調節技巧可增加認知行為治療的效果。其他的研究則皆無足夠的統計考驗力來偵測 g 小於 0.55之效果值。只有一項研究具有低度的偏差危險。結論：現有的成份研究並無足夠的統計考驗力或是品質，以形成任何關於成人憂鬱心理治療之核心成份的有意義結論。

本文在臨床或方法學上的重要性：美國國家醫學研究院最近的一份報告再次指出，有必要辨識出導致心理社會介入有效的核心要素。成份研究在檢視治療是如何產生功效上，是個重要的工具；他們同時能以簡鍊的方式辨識出心理治療的活躍要素。這些研究分解多重成份的治療，並將完整治療與未包括其中一個成份的治療（即，拆解研究），或是與一個現存的治療再添加某一成分的治療做比較。我們透過蒐尋現有針對成人憂鬱的心理治療成效進行隨機詴驗之資料庫，進行系統性的成份研究回顧。與早期的回顧研究相較，這樣做有些優點，可能可以錯失較少的成份，因為成份研究通常不會以這個研究設計為其關鍵字。

Keywords:

• dismantling study
• meta-analysis
• component study
• additive study
• depression
• specific and nonspecific mechanisms

Keywords:

• studi di smantellamento
• meta-analisi
• component study
• depressione
• meccanismi specifici e non specifici

Schlüsselwörter:

• Dismantling-Studie
• Meta-Analyse
• Komponentenstudie
• additive Studie
• Depression
• spezifische und unspezifische Mechanismen

Palavras-chave:

• estudo de desmantelamento
• meta-análise
• estudo de componentes
• estudo aditivo
• depressão
• mecanismos específicos e não específicos

關鍵字:

• 拆解研究
• 後設分析
• 成份研究
• 添加研究
• 憂鬱
• 特定和非特定機制

Introduction

It is well established that several psychological interventions are effective in the treatment of depression, including cognitive behavior therapy (Churchill et al., 2001; Cuijpers et al., 2013), interpersonal psychotherapy (Cuijpers et al., 2011; Cuijpers, Donker, Weissman, Ravitz, & Cristea, 2016), problem-solving therapy (Malouff, Thorsteinsson, & Schutte, 2007), behavioral activation (Cuijpers, van Straten, & Warmerdam, 2007; Ekers, Richards, & Gilbody, 2008), and possibly psychodynamic therapies (Driessen et al., 2015; Leichsenring & Rabung, 2008). It is less well established, however, how these therapies work and which mechanisms are responsible for their effects (Kazdin, 2007, 2009).

Understanding how psychotherapies work is important, because this may result in more knowledge about how to optimize treatments for depression. Knowledge about how therapies work may also result in a better understanding of the causal mechanisms that lead to depression (Whisman, 1993). Optimizing treatments for depression is important because modeling studies estimate that current treatments can reduce the disease burden associated with depression by only about one third, in optimal conditions (Andrews, Issakidis, Sanderson, Corry, & Lapsley, 2004).

Component studies are an important tool for examining how therapies work and they provide an elegant way to identify the active ingredients of psychotherapies (Borkovec & Costonguay, 1998). These studies decompose multicomponent therapies and compare the full therapy with a therapy in which one component is left out (dismantling studies) or in which a component is added to an existing therapy (additive studies; Bell, Marcus, & Goodlad, 2013). Because of their experimental design, the strength of evidence resulting from component studies is high. If a component study finds a difference between a therapy with a component and a therapy without that component, this indicates that this component is indeed responsible for (part of) the effects of the intervention. This is in contrast with research on mediators, which is always indirect and correlational, and when a significant mediator is found it is always possible that a third unknown variable is responsible for the change in outcome as well as the change in the mediator (Kazdin, 2007).

A recent report from the US Institute of Medicine pointed once again at the need for identifying the key elements of psychosocial interventions that drive an intervention’s effect (Institute of Medicine, 2015). This report considered the identification of these elements to be one of the key steps entailed in successfully bringing an evidence-based psychosocial intervention into clinical practice.

The first major review and meta-analysis of component studies did not find specific components that are responsible for part of the effects of therapies (Ahn & Wampold, 2001). This finding was one of the reasons why some researchers assume that the effects of therapies are not caused by the specific techniques used in these therapies (Luborsky et al., 2002; Luborsky, Singer, & Luborsky, 1975; Wampold, 2001), but by universal, non-specific factors that are common to all therapies, such as the therapeutic alliance and expectancies of the patient. If dismantling studies don’t find effective components, then that could be seen as an indication that these components don’t exist and as support for the assumption that it is not these components that cause change, but the universal mechanisms. In an important early review of 27 dismantling studies suggestively entitled “Where oh where are the specific ingredients?”, the authors did not find many indications for significant components and concluded that therapies are probably not working through specific techniques, but through non-specific, universal mechanisms (Ahn & Wampold, 2001).

A second major review examined a much larger set of component studies and found that dismantling studies did not result in significant differences between the full and the partial treatment (in which one component was left out; Bell et al., 2013). They did find, however, a small but significant effect when a component was added to an existing therapy and this effect was larger at follow-up. The authors concluded that specific components may add to the effects of existing treatments. Their results did not support the conclusion of the earlier meta-analysis that it is universal factors, and not specific components, that are responsible for change. This meta-analysis has been criticized (Flückiger, Del Re, & Wampold, 2015; Wampold & Imel, 2015), and a re-analysis with longitudinal meta-analytical methods could not replicate the finding that effects of components were found at the longer term (Flückiger et al., 2015). Despite this criticism, however, the meta-analysis of Bell and colleagues is the most comprehensive and most recent meta-analysis of component studies.

The review by Bell and colleagues also has a number of limitations. One important limitation was that the quality of the included studies was not assessed. It is well known that the quality of randomized trials can have a large impact on the effect sizes found in these trials (Cuijpers, van Straten, Bohlmeijer, Hollon, & Andersson, 2010; Higgins et al., 2011). A meta-analysis can never be better than the studies it summarizes, no matter how sophisticated the meta-analysis is done.

Another problem of Bell et al. (2013) is that the statistical power of most published component studies is low (Kazdin & Whitley, 2003). While the authors discussed this problem extensively, they did not quantify the statistical power of the included studies nor the corresponding effect sizes these studies could in theory evidence, considering their sample sizes. Most published dismantling studies are markedly underpowered and consequently the absence of significant effects in these studies cannot reliably lead to the conclusion that the components examined are indeed not effective. Non-significant results from underpowered studies can in no way be considered as evidence of absence of effects.

One more problem of Bell et al. (2013) is that it was focused on a broad range of health problems, ranging from depression, anxiety, and posttraumatic stress disorder to tension headaches, rheumatoid arthritis, and marital discord. Due to the large differences in the examined components, the clinical heterogeneity of the included studies was very high. Although the authors found that statistical heterogeneity was low, this has to be considered with caution because the uncertainty around statistical heterogeneity can be considerable and makes it impossible to assess the true statistical heterogeneity (Ioannidis, Patsopoulos, & Evangelou, 2007).

A final problem of Bell et al. (2013) was that it was mostly based on systematic searches in bibliographical databases using search terms to identify component studies (although studies were also identified through handsearching key journals). But because many component studies do not use this design as index term, it is unlikely that all component studies from the past 30 years are identified with such search strings (Bell et al., 2013).

In the current meta-analysis, we partially solved this problem by using an existing database of studies on psychological treatments of adult depression, which uses broad search strings for identifying all studies in this field and is updated every year (Cuijpers, van Straten, Warmerdam, & Andersson, 2008). We examined all included studies and selected all component studies. Hence it is improbable that we missed many studies, and we also reduced the clinical heterogeneity of the target problems by focusing specifically on depression. This emphasis on one disorder also makes it also possible to examine whether there are groups of component studies focusing on the same components, and whether pooled effects of these studies result in significant outcomes. We conducted a comprehensive systematic review and meta-analysis of dismantling studies of psychotherapies for adult depression. In it, we not only focus on the effects of (groups of) components, but also on the quality of the included studies and the statistical power of these studies.

Methods

Identification and Selection of Studies

We used an existing database of studies on the psychological treatment of depression. This database has been described in detail elsewhere (Cuijpers et al., 2008), and has been used in a series of earlier published meta-analyses (www.evidencebasedpsychotherapies.org). For this database we searched four major bibliographical databases (PubMed, PsycInfo, Embase and the Cochrane database of randomized trials) by combining terms (both MeSH terms and text words) indicative of depression and psychotherapies, with filters for randomized controlled trials. We also checked the references of earlier meta-analyses on psychological treatments for the included disorders. The database is continuously updated and was developed through a comprehensive literature search (from 1966 to January 2016).

We included component studies that were: (a) a randomized trial (b) in which a psychological treatment (c) for adult depression was (d) compared with the same psychological treatment but with one component added or removed. We defined a component as a specific ingredient that is hypothesized to be critical to the success of the treatment (Ahn & Wampold, 2001). We only included trials in which the full therapy was compared with the adjusted therapy in which one component was removed or added, when the content of this component was associated with the therapy. Although we tried to be as inclusive as possible, we excluded studies in which it was not clear whether an actual ingredient was examined, such as religious versus non-religious therapy (for religious people) and culturally adapted versus non-adapted therapy. We also excluded other types of components, such as the treatment format or characteristics of the therapist were not included to keep clinical heterogeneity as small as possible. For example, face-to-face therapies compared with guided self-help (in which part of the contact time with the therapist was reduced) were not included in the review. We also did not include trials in which professional and paraprofessional therapists were compared with each other. Trials in which a therapy was compared with non-directive supportive counseling were also excluded, assuming that the non-directive approach contains the elements that every therapy offers. We excluded these trials because they provide the effects of all components of a therapy taken together and compared with the basic empathy and support in the non-directive approach, instead of specific components (Cuijpers et al., 2012). We made an exception for studies comparing experiential therapy with non-directive counseling, because experiential therapy is built on the principles of non-directive counseling and can be seen as an extension or additional component to it (Greenberg & Watson, 1998).

Depression could be established with a diagnostic interview or with a score above a cut-off on a self-report measure. The therapies could be delivered individually, in groups, as guided self-help, or by telephone. Co-morbid mental or somatic disorders were not used as an exclusion criterion. Studies on children and adolescents (below 18 years of age) were excluded. We also excluded maintenance studies, aimed at people who had already recovered or partly recovered after an earlier treatment, and studies that did not report sufficient data to calculate standardized effect sizes. Studies in English, German, and Dutch were considered for inclusion.

Quality Assessment and Data Extraction

We assessed the validity of included studies using four criteria of the “Risk of bias” assessment tool, developed by the Cochrane Collaboration (Higgins et al., 2011). This tool assesses possible sources of bias in randomized trials, including the adequate generation of allocation sequence; the concealment of allocation to conditions; the prevention of knowledge of the allocated intervention (masking of assessors; when only self-report measures were used for the assessment of depression we rated this as positive as well); and dealing with incomplete outcome data (this was assessed as positive when intention-to-treat analyses were conducted, meaning that all randomized patients were included in the analyses). We did not rate selective outcome reporting, because there are important flaws in its assessment (Page & Higgins, 2016), such as the conflation of nonreporting with selective reporting of outcomes, and because most psychotherapy trials are still not prospectively registered (Bradley, Rucklidge, & Mulder, 2017).

Two independent researchers assessed the validity of the included studies and disagreements were solved through discussion.

We also examined whether the interventions met the criteria for a “bona fide” therapy as defined by Wampold et al. (1997). This meant that a treatment had to involve a therapist who had at least a master’s degree, who met face to face with the client, developed a relationship with the client. Furthermore, the treatment had to contain at least two of the following four elements: (a) The treatment was based on an established treatment that was cited; (b) a description of the treatment was contained in the article; (c) a manual was used to guide administration of the treatment; and (d) active ingredients of the treatment were identified and cited. The study’s research design also had to involve a comparison of one group with another group, and one of the following two conditions had to be satisfied: (a) One, two, or three ingredients of the treatment were removed, leaving a treatment that would be considered logically viable (i.e., coherent and credible), or (b) one, two, or three ingredients that were compatible with the whole treatment and were theoretically or empirically hypothesized to be active were added to the treatment, providing a “super treatment” (Ahn & Wampold, 2001).

We did not assess the quality of the interventions in other ways, because we examined several criteria for intervention quality in a large, previous meta-analysis, but found no indication that these criteria were associated with the effects of the interventions (Cuijpers et al., 2010).

Meta-analyses

The effect size indicating the difference between the two groups at post-test was calculated (Hedges’ g) for each comparison between the full psychotherapy and the partial therapy (in which one component was removed). The full psychotherapy could consist of an existing therapy to which a component was added (additive studies), or from which a component was removed (dismantling studies). This allowed us to examine the pooled effect of all component studies in one analysis.

Effect sizes of 0.8 can be assumed to be large, while effect sizes of 0.5 are moderate, and effect sizes of 0.2 are small (Cohen, 1988). We calculated effect sizes by subtracting (at post-test) the average score of the psychotherapy group from the average score of the control group, and dividing the result by the pooled standard deviation. Because some studies had relatively small sample sizes, we corrected the effect size for small sample bias (Hedges & Olkin, 1985). If means and standard deviations were not reported, we used the procedures of the Comprehensive Meta-Analysis software (see below) to calculate the effect size using dichotomous outcomes; and if these were not available either, we used other statistics (such as t-value or p-value) to calculate the effect size. If a study reported more than one depression outcomes we pooled the effect sizes within each study, before pooling across studies, so that each study only had one effect size.

In order to calculate effect sizes we used all measures examining depressive symptoms (such as the Beck Depression Inventory/BDI (Beck, Ward, Mendelson, Mock, & Erbaugh, 1961), the BDI-II (Beck, Steer, & Brown, 1996) or the Hamilton Rating Scale for Depression/HAMD-17 (Hamilton, 1960)). To calculate pooled mean effect sizes, we used the computer program Comprehensive Meta-Analysis (version 3.3070; CMA). Because we expected considerable heterogeneity among the studies, we employed a random effects pooling model in all analyses. We also explored whether it was possible to pool secondary outcomes from the trials, but found that these were too heterogeneous to be pooled.

As a test of homogeneity of effect sizes, we calculated the I² -statistic, which is an indicator of heterogeneity in percentages. A value of 0% indicates no observed heterogeneity, and larger values indicate increasing heterogeneity, with 25% as low, 50% as moderate, and 75% as high heterogeneity (Higgins, Thompson, Deeks, & Altman, 2003). We calculated 95% confidence intervals around I² (Ioannidis et al., 2007), using the non-central chi-squared-based approach within the heterogi module for Stata (Orsini, Bottai, Higgins, & Buchan, 2006).

We conducted subgroup analyses according to the mixed effects model, in which studies within subgroups are pooled with the random effects model, while tests for significant differences between subgroups are conducted with the fixed effects model.

We tested for publication bias by inspecting the funnel plot on primary outcome measures and by Duval and Tweedie’s trim and fill procedure (Duval & Tweedie, 2000), which yields an estimate of the effect size after the publication bias has been taken into account (as implemented in CMA). We also conducted Egger’s test of the intercept to quantify the bias captured by the funnel plot and to test whether it was significant.

Power Calculations

We expected to find only a limited number of component studies. Furthermore, we expected that the difference between a full and a partial therapy would be limited (resulting in small effect sizes) because this only measures the effects of one component. Power and the sample size of included studies is therefore a crucial element in a systematic review of these studies (Kazdin & Whitley, 2003). Because previous meta-analyses did describe the problem of low power of most component studies extensively, but did not examine this problem quantitatively, we decided to examine the power of included studies in more detail.

We first calculated the effect size for each study that could be detected with the sample size in that individual study. We conducted these analyses with G*Power software (Faul, Erdfelder, Lang, & Buchner, 2007), assuming a power of 0.80 and an alpha of 0.05.

The threshold for a clinically relevant treatment effect in depression has been estimated to be g = 0.24 (Cuijpers, Turner, Koole, van Dijke, & Smit, 2014). Because dismantling studies should be expected to result in small effect sizes, we calculated in G*Power how many participants should be included in a trial to detect an effect size of at least g = 0.24. We found that the required number of participants in such a trial was 548. In order to get an idea of the statistical power of the included dismantling studies we calculated for each study separately the percentage of the 548 participants that was included in the trial.

Then, for meta-analyses, in which more than one study was pooled, and different power calculations (based on number of studies and number of participants per study) are needed, we conducted power calculations according to the procedures described by Borenstein, Hedges, Higgins, and Rothstein (2009).

Results

Selection and Inclusion of Studies

After examining a total of 16,407 abstracts (13,384 after removal of duplicates), we retrieved 1,885 full-text papers for further consideration. We excluded 1,868 of the retrieved papers. The PRISMA flowchart describing the inclusion process, including the reasons for exclusion, is presented in Figure 1. A total of 16 studies met inclusion criteria for this meta-analysis. Selected characteristics of the included studies are reported in Table I, and references of the included studies are listed in Appendix 1.

Figure 1. Flowchart of inclusion of studies.

Table I. Component studies of psychotherapy for adult depression.

1st author	Population	Definition of depression	Full psychotherapy	For mat	Nsess	N	Therapy with component removed	N	Same Nsess	Added / removed	Bona fide therapy	RoB^a
Alladin and Alibhai (2007)	Out patients	Chronic MDD (DSM-IV)	− Hypnosis + CBT	Ind	16	42	CBT only	42	Same	Added	−	− − sr −
Berking, Ebert, Cuijpers, and Hofmann (2013)	Inpatients	MDD (DSM-IV; SCID); BDI > 11;	− CBT + intense emotion regulation skills training	Ind / Grp	84	195	CBT only	237	Same	Added	+	+ + sr +
Hamamci (2006)	University students	BDI ≥ 19	− Psychodrama integrated in CBT	Grp	11	8	CBT only	8	Same	Added	+	− − sr −
Jacobson et al. (1996)	Adults in general	MDD (SCID / DSM-III-R) + BDI > 20 + HRSD > 14	− Full CBT (including behavioral activation and cognitive restructuring)	Ind	20	50	Behavioral activation only	56	Same	Removed	+	− − − −
			− Full CBT (including behavioral activation and cognitive restructuring)	Ind	20	50	Cognitive restructuring only	43	Same	Removed	+
Kontunen, Timonen, Muotka, and Liukkonen (2016)	Primary care patients	MDD (DSM-IV; MINI)	− Interpersonal psychotherapy	Ind	7	20	Interpersonal counseling	20	Same	Removed	+	− − sr −
Manicavasgar, Parker, and Perich (2011)	Adults in general	MDD (DSM-IV; CIDI) + BDI ≥ 20	− Mindfulness-based CBT	Grp	8	19	CBT	26	Same	Added	+	− − sr −
McNamara and Horan (1986)	University counsel. Center	BDI ≥ 18 + modified HAMD ≥ 20	− CBT	Ind	10	10	CT (without BT)	10	More	Removed	+	− − sr −
			− CBT	Ind	10	10	BT (without CT)	10	More	Removed	+
Nezu (1986)	Adults in general	Depressive disorder (RDC) + BDI ≥ 16 + MMPI-D > 70	− Full problem-solving therapy	Grp	8	11	Problem focused therapy (partial PST)	9	Same	Removed	+	− − sr −
Nezu and Perri (1989)	Adults in general	MDD (RDC) + BDI > 20 + HRSD > 18	− Full problem-solving therapy	Grp	10	14	Abbreviated problem-solving therapy	14	Same	Removed	+	− − sr −
Omidi, Mohammadkhani, Mohammadi, and Zargar (2013)	University counsel. center	MDD (DSM-IV)	− Mindfulness-based CBT	Grp	8	30	CBT	30	Same	Added	−	− − sr −
Rehm (1981)	Adult women	MMPI criteria	− Full self-control therapy	Grp	7	9	self-monitoring only	9	Same	Removed	+	− − sr −
			− Full self-control therapy	Grp	7	9	self-monitoring + self-evaluation	11	Same	Removed	+
			− Full self-control therapy	Grp	7	9	self-monitoring + self-reinforcement	12	Same	Removed	+
Sudweeks (1996)	Adults	BDI > 13	− Hypnotherapy + CBT	Grp	6	15	CBT	15	Same	Added	+	− − sr −
			− Hypnotherapy + CBT	Grp	6	15	Hypnotherapy	15	Same	Added	+
Taylor and Marshall (1977)	Young adults	BDI > 13 + D-scale > 30	− Full CBT (including behavioral activation and cognitive restructuring)	Ind	6	7	Behavioral activation only	7	Same	Removed	+	− − sr −
			− Full CBT (including behavioral activation and cognitive restructuring)	Ind	6	7	Cognitive restructuring only	7	Same	Removed	+
Teismann, Dymel, Schulte, and Willutzki (2011)	Out patients	MDD or dysthymia (DSM-IV; SCID) + BDI ≥ 20	− CBT + resource activation	Ind	24	28	CBT	24	Same	Added	+	− − − −
Tovote et al. (2014)	Patients with diabetes	BDI-II ≥ 14	− Mindfulness-based CBT	Ind	8	31	CBT	32	Same	Added	+	+ − − +
Zettle and Rains (1989)	Adult women	BDI ≥ 20 + MMPI criteria + HDRS ≥ 14	− Full CBT	Grp	12	10	CBT without distancing procedures	10	Same	Removed	+	− − + −

Note. BDI: Beck Depression Inventory; BT: Behavior Therapy; CBT: Cognitive Behavioral Therapy; CIDI: Composite International Diagnostic Interview; Counsel.: Counseling; CT: Cognitive Therapy; D-scale: Depression-scale; DSM-III-R: Diagnostic and Statistical Manual Version III-Revised; DSM-IV: Diagnostic and Statistical Manual Version IV; Grp: group format; HAMD: Hamilton Rating Scale for Depression; HRSD: Hamilton Rating Scale for Depression; Ind: individual format; MDD: Major Depressive Disorder; MINI: Mini International Neuropsychiatric Interview; MMPI-D: Minnesota Multiphasic Personality Inventory Depression scale; PST: Problem Solving Therapy; RDC: Research Diagnostic Criteria; SCID: Structured Clinical Interview for DSM-IV criteria.

^aIn this column a positive (+) or negative (−) sign is given for four quality criteria of the study, respectively: allocation sequence; concealment of allocation to conditions; blinding of assessors; and intention-to-treat analyses. Sr in the third criterion indicates that only self-report measures were used (and no assessor was used).

Characteristics of Included Studies

The 16 studies included a total of 22 comparisons between a full and partial therapy. A total of 1146 patients participated in the trials, 499 in the full therapies and 647 in the partial therapies (the mean number of patients per comparison was 52 and respectively 26 per arm). In the 22 included comparisons, 15 separate components were examined of which 4 were examined in more than one comparison (three components were examined in three comparisons, and one in two comparisons). Nine of the 22 comparisons had an additive design and the other 13 had a dismantling design. In two of the 22 comparisons, the full therapy had more sessions than the partial therapy.

In 10 studies patients had to meet diagnostic criteria for a depressive disorder, while in the other six studies patients had to score above the cut-off on a self-report measure. Ten of the 16 studies were aimed at adults in general, while the other 6 were aimed at more specific target groups, like students and patients with co-morbid general medical disorders. In 16 comparisons CBT was examined, the other 6 examined another type of therapy. In 10 comparisons an individual treatment format was used, 12 used a group or mixed format.

Quality Assessment

The quality of the 16 included studies was not optimal. Only two studies reported an adequate sequence generation. One of the studies reported that allocation to conditions was properly concealed. One study reported blinding of outcome assessors, while another 12 used only self-report outcomes. Two studies conducted intention-to-treat analyses. Only one study met four criteria, one met two criteria and the remaining 14 studies met one or none of the criteria.

Overall Effects of Components in Dismantling Studies

The effect sizes found for each study (full vs partial therapy) are presented in the forest plot in Figure 2. The effect sizes indicating the difference between the full and partial therapies range from g=−0.66 to g = 1.76 (median g = 0.08). The pooled effect size for all studies together was g = 0.21 (95% CI: 0.03∼0.39; p = .02), with low to moderate heterogeneity (I² =39). Exclusion of the two comparisons of interventions that did not meet criteria for a bona fide therapy, resulted in a comparable effect size (g = 0.20), which was still significant, and comparable levels of heterogeneity.

Figure 2. Dismantling studies in psychotherapies for adult depression.

In six studies, more than one effect size indicating the difference between a full and partial therapy was calculated (4 studies had two effect sizes, and one study had three effect sizes). Because these effect sizes were not independent of each other, they may have artificially reduced heterogeneity and influenced the effect sizes. We conducted two separate analyses to examine this. In the first analysis we only included the largest effect size from each study, and in the second only the smallest effect size. As can be seen in Table II, the effect sizes and levels of heterogeneity were comparable to those in the main analyses.

Table II. Dismantling studies of psychotherapies for adult depression: pooled effect sizes.

		Nst	g	95% CI	I^2	95% CI	Q (df)	p	Nneeded
All studies		22	0.21	0.03∼0.39	39	0∼62	34.29 (21)*
Non bona fide studies excluded	20	0.20	0.01∼0.40	41	0∼64	32.14 (19)*
One effect size per study (highest)	16	0.23	0.04∼0.43	40	0∼66	25.15 (15)*
One effect size per study (lowest)	16	0.25	0.04∼0.46	48	0∼69	28.70 (15)*
Subgroup analyses
Number of sessions	The same	20	0.22	0.03∼0.41	43	0∼65	33.16 (19)*	.74
	More in full therapy	2	0.11	−0.49∼0.71	0	–^a	1.00 (1)
Component	Added	9	0.22	0.08∼0.36	0	0∼54	5.73 (8)	.82
	Removed	13	0.26	−0.10∼0.62	58	5∼76	28.55 (12)***
Type of therapy	CBT	16	0.21	0.08∼0.33	0	0∼45	14.16 (15)	.74
	Other	6	0.33	−0.40∼1.06	74	21∼87	19.58 (5)**
Format	Individual	10	0.23	0.03∼0.43	0	0∼53	8.76 (9)	.82
	Other	12	0.19	−0.10∼0.48	57	0∼76	25.52 (11)**
Target group	Adult in general	16	0.28	0.01∼0.54	51	0∼71	30.73 (15)*	.54
	More specific	6	0.18	0.02∼0.34	0	0∼61	2.99 (5)
Risk of bias	3 or 4 criteria	1	0.25	0.06∼0.45	0	–^a	0.00 (0)	.75
	lower	21	0.21	0.00∼0.41	41	0∼64	34.10 (20)
Subsets of studies
CBT vs BA only	3	0.06	−0.55∼0.68	43	0∼83	3.53 (2)			956
CBT vs CT only	3	0.46	0.01∼0.91	0	0∼73	0.76 (2)			∼
Hypnotherapy + CBT vs CBT only	2	0.23	−0.37∼0.83	55	–^a	2.22 (1)			21
Mindfulness CBT vs. CBT	3	0.04	−0.26∼0.34	0	0∼73	0.19 (2)			692
Long term outcomes
5–8 weeks	5	0.53	−0.16∼1.22	63	0∼84	10.69 (4)
6 months	8	0.28	−0.08∼0.65	66	3∼82	20.58 (7)
12 months	3	0.15	−0.19∼0.50	0	0∼73	0.22 (2)

Note. BA: Behavior Activation; CBT: Cognitive Behavior Therapy; CT: Cognitive Therapy; N_needed: number of studies needed to show that the effect size found is significantly different from zero; N_st: number of studies used for the calculation of the effect size; p: this p-value indicates whether the effect sizes between the subgroups differ significantly from each other.

^aThe 95% CI cannot be calculated when the number of studies is lower than 2.

*p < .05.

**p < .01.

***p < .001.

In four of the 22 comparisons a significant difference between the full and the partial therapy was found, one in which hypnosis was added to CBT (Alladin & Alibhai, 2007), another one in which emotion regulation skills were added to CBT (Berking et al., 2013), and two in which problem-solving therapy was compared with partial or abbreviated problem-solving therapy (Nezu, 1986; Nezu & Perri, 1989).

Eleven studies reported longer term outcomes. In Table II we have reported the outcomes for 5–8 weeks, 6 and 12 months follow-up. None of the pooled effect sizes was significant, possibly because of low statistical power.

We also conducted a series of subgroup analyses (Table I), but did not find any indication that the effect size differed significantly between subgroups (the number of sessions was the same in the full and partial therapies or not the same; the component was added to an existing therapy or removed; CBT vs other types; individual vs other treatment format; adults in general vs more specific subgroups; three or four criteria for low risk of bias vs less criteria).

We did find that the pooled effect size of the additive studies (in which a component was added to an existing therapy) was significant (g = 0.22). But this was no longer significant when the large study from Berking et al. (2013) was removed (g = 0.18; 95% CI: −0.02∼0.38; p = .008). The pooled effect size of the dismantling studies was not significant (p > .1). However, the effect size was comparable to that of the additive studies (g = 0.26). This may have been not significant because of low power.

We found no indication for publication bias (Egger’s test of the intercept: p > .1; Duvall & Tweedie’s trim and fill procedure: adjusted and unadjusted effect sizes are identical, zero imputed studies).

Power Calculations

We calculated for each study how large the effect size was that could be found given the number of participants in the trial (Figure 2). As can be seen, there was one study that had sufficient power to detect an effect size of g = 0.27 (Berking et al., 2013). The effect sizes that could be detected by the other studies ranged from g = 0.55 to g = 1.63. (as indicated earlier: the threshold for clinical relevance for treatments of depression has been estimated to be g = 0.24 (Cuijpers et al., 2014)). Thirteen comparisons (59%) had only sufficient power to detect effect sizes of g > 1.00.

As indicated earlier, we also calculated for each included study the percentage of the 548 patients that is needed to detect an effect size of g = 0.24 (the threshold for a clinically relevant effect). The results are given in Figure 2 and show that one large trial (Berking et al., 2013) included 79% of these 548 patients, while none of the other studies included more than 19% of the 548 patients needed to detect a clinically relevant effect size. Moreover, 16 of the 22 comparisons had less than 10% of the needed 548 patients.

Effects for Specific Components

There were several comparisons that examined the same components (Table II, subsets of studies). As can be seen, the three studies in which cognitive behavior therapy was compared with cognitive therapy only (the behavioral activation component was removed) resulted in a significant effect size of g = 0.46 (95% CI: 0.01∼0.91) with zero heterogeneity. The other subsets of comparisons did not result in significant effect sizes, but the number of studies in each of these subsets was very small (three with three comparisons and one with two comparisons).

We conducted power calculations based on the number of studies and number of participants per study, to examine how many studies would have been needed to find a significant effect. In these calculations we assumed that the effect size found was the true effect size, and assuming that the number of included patients was the same as in the included trials. As shown in Table II, the number of studies needed to show that a subset of studies resulted in significant outcomes was high for the subsets resulting in small effect sizes. For instance, the subset of studies comparing full CBT with behavioral activation only resulted in an effect size of g = 0.06, and a total of 956 studies with the same sample size would be needed to find this effect size to be significantly different from zero; 692 trials would be needed to show that mindfulness-based CBT is more effective than CBT.

Discussion

We conducted a systematic review of component studies by searching an existing database of randomized trials examining the effects of psychological treatments of adult depression. This had the advantage that we probably missed few component, in contrast to earlier reviews, because component studies often do not use this design as index term. The largest and most recent review of component studies (Bell et al., 2013) identified only four studies on depression, while in the current meta-analysis we identified 16 component studies on therapies for depression with 22 comparisons between a full and partial therapy. In these comparisons, 15 separate components were examined in which 4 were examined in more than one comparison (three components were examined in three comparisons, and one in two comparisons).

In our review, we explicitly focused on the power of component studies, and found that only one study came close to having sufficient statistical power to detect a clinically significant effect (Berking et al., 2013). This study also found a significant effect of the component (emotional-regulation skills in addition to CBT compared to CBT alone), and because risk of bias was low, the evidence that this component was effective can be considered quite strong. Evidently, this finding has to be replicated in further trials. Furthermore, there is a considerable risk of researcher allegiance in this study because the first author was also the author of the manual of the intervention. This study does show, however, that it is not impossible that significant components can be identified in well-powered dismantling studies.

One other significant component was identified after pooling the results of three studies comparing full CBT (consisting of CBT plus behavioral activation) with cognitive restructuring alone (without the behavioral activation). This finding suggests that behavioral activation is one of the effective components of CBT, and leaving it out reduces the effects of CBT considerably. However, there was considerable risk of bias in these studies and the results should therefore be considered with caution. Nonetheless, this set of studies also shows that (series of) dismantling studies can indeed identify effective components of therapies.

The other 13 components that were examined in the dismantling studies we identified in our search did not in any way have sufficient power to detect an effective component. Except for the earlier mentioned study by Berking et al. (2013), none of the studies had more than 19% of the participants that are needed to detect a significant effect of a component, and 16 of the 22 comparisons had less than 10% of the participants that are needed. Moreover, none of the studies (except Berking et al., 2013) had enough power to detect an effect size smaller than g = 0.55, and 13 of the 22 comparisons only had enough power to detect an effect size larger than g = 1.0. So none of these studies came even close to an effect size like for instance the one that would separate the effects of antidepressants from placebo (g = 0.31; Turner, Matthews, Linardatos, Tell, & Rosenthal, 2008).

We also pooled the results of all component studies and found that these pooled effects were significantly different from zero, suggesting that there may be some components that are actually effective ingredients of therapies. The additive studies showed a significant effect size, and although the effect size of the dismantling studies was not significant, it was about as large as that of the additive studies. Consequently, the lack of significance could very well be caused by low statistical power. Nonetheless, the pooling of component studies is generally not very useful, especially when the statistical power is low. When only a small number of components are indeed effective ingredients, and the statistical power of the included studies is small, it is very well possible that the pooled effects of all studies are not significant, while in fact several components do in fact have significant effects.

Apart from the statistical power, we also focused on the quality of the included component studies and found that only one of the 16 included studies had low risk of bias. This body of research can, therefore, in no way be considered as evidence that specific components of psychotherapies are not effective and that all effects can be explained by non-specific or universal effects (Ahn & Wampold, 2001; Wampold, 2000). Almost none of the studies has sufficient power to detect effective components and the risk of bias is so high in these studies that it is difficult to draw any conclusions from these studies at all. The one exception of a trial with more or less sufficient power and low(er) risk of bias, did indicate the presence of a significant effect for a component. On the other hand, this body of research can also not be considered as evidence that specific components of psychotherapies do exist. The evidence is simply too weak to draw any conclusions. The pooled significant effect of all studies together could very well be the result of bias in the included studies and there were simply too few studies with low risk of bias to draw any conclusion.

We did not assess researcher allegiance in the included studies. Most instruments of researcher allegiance focus on trials where two therapies are directly compared with each other. That is not the case for the studies in our review. Because the trials in the current study examined only one therapy (with or without one component) it is difficult to assess whether authors had allegiance toward this treatment. However, it is very well possible that allegiance played a role in these studies, and that the outcomes were affected, making them even less credible.

Apart from these methodological issues and the uncertainties they bring about, there are two important caveats to this research. The first is that whether or not distinct specific components of psychotherapies do exist and are effective is a separate issue from whether or not universal factors are responsible for therapeutic change. Lack of evidence for the former is not sufficient evidence for the latter. Universal factors must come with evidence of their own and cannot be postulated on the basis of insufficient evidence (or indeed lack thereof) for specific factors. The second is that it is very likely that not all purported components of therapeutic packages are equal, not should they be expected to be equally relevant. The theory behind most psychotherapy packages usually carries a degree of subjectivity and is rooted in the developers’ clinical experience and its subsequent interpretation. Consequently, some of the components included in the package might have simply been based on erroneous intuitions and looking for evidence to their existence or effectiveness doomed to fail.

Limitations

This study has several important limitations. First, the number of included studies was relatively low, and both the quality and the statistical power in these studies to detect effects of the examined components is very low. Second, the components that were examined in the included studies differed widely, with 15 different components examined in 16 studies. Only three components were examined in three studies and 11 were examined in only one study. Considering that effective components had small effect sizes, many more trials examining the same components would be needed to detect whether these are effective or not if we were to use meta-analytic techniques. Third, we did not assess the quality of the interventions, because previous criteria of intervention quality did not indicate that these were associated with the effects of the interventions. However, this may be related to problems with measuring intervention quality. It is possible therefore that this may have influenced outcomes.

Conclusion

Based on this set of studies, the only conclusion that can be drawn is that we simply don’t know neither if specific components of specific therapies are effective ingredients of these therapies, nor that all effects are caused by universal, nonspecific factors that are common to all therapies. If we really want to examine effective components as recommended in the IOM report on psychosocial interventions, we need studies with lower risk of bias and considerably larger than the ones that have been conducted so far.

ORCID

Pim Cuijpers http://orcid.org/0000-0001-5497-2743

Mirjam Reijnders http://orcid.org/0000-0002-4272-2576

Additional information

Funding

This work was supported by Romanian National Authority for Scientific Research and Innovation [grant number PN-II-RU-TE-2014-4-1316].

Appendix 1

References That Were Used in the Final Analyses

Alladin, A., & Alibhai, A. (2007). Cognitive hypnotherapy for depression: An empirical investigation. International Journal of Clinical and Experimental Hypnosis, 55(2), 147–166. doi:10.1080/00207140601177897

Berking, M., Ebert, D., Cuijpers, P., & Hofmann, S. G. (2013). Emotion regulation skills training enhances the efficacy of inpatient cognitive behavioral therapy for major depressive disorder: A randomized controlled trial. Psychotherapy and Psychosomatics, 82(4), 234–245. doi:10.1159/000348448

Hamamci, Z. (2006). Integrating psychodrama and cognitive behavioral therapy to treat moderate depression. Arts in Psychotherapy, 33(3), 199–207. doi:10.1016/j.aip.2006.02.001

Jacobson, N. S., Dobson, K. S., Truax, P. A., Addis, M. E., Koerner, K., Gollan, J. K., … Prince, S. E. (1996). A component analysis of cognitive-behavioral treatment for depression. Journal of Consulting and Clinical Psychology, 64(2), 295–304. doi:10.1037/0022-006X.64.2.295

Kontunen, J., Timonen, M., Muotka, J., & Liukkonen, T. (2016). Is interpersonal counseling (IPC) sufficient treatment for depression in primary care patients? A pilot study comparing IPC and interpersonal psychotherapy (IPT). Journal of Affective Disorders, 189(1), 89–93. doi:10.1016/j.jad.2015.09.032

Manicavasgar, V., Parker, G., & Perich, T. (2011). Mindfulness-based cognitive therapy vs cognitive behaviour therapy as a treatment for non-melancholic depression. Journal of Affective Disorders, 130(1), 138–144. doi:10.1016/j.jad.2010.09.027

McNamara, K., & Horan, J. J. (1986). Experimental construct validity in the evaluation of cognitive and behavioral treatments for depression. Journal of Counseling Psychology, 33(1), 23-30. doi:10.1037/0022-0167.33.1.23

Nezu, A. M. (1986). Efficacy of a social problem-solving therapy approach for unipolar depression. Journal of Consulting and Clinical Psychology, 54(2), 196–202. doi:10.1037/0022-006X.54.2.196

Nezu, A. M., & Perri, M. G. (1989). Social problem-solving therapy for unipolar depression: An initial dismantling investigation. Journal of Consulting and Clinical Psychology, 57(3), 408–413. doi:10.1037/0022-006X.57.3.408

Omidi, A., Mohammadkhani, P., Mohammadi, A., & Zargar, F. (2013). Comparing mindfulness based cognitive therapy and traditional cognitive behavior therapy with treatments as usual on reduction of major depressive disorder symptoms. Iranian Red Crescent Medical Journal, 15(2), 142–146. doi:10.5812/ircmj.8018

Rehm, L. P., Kornblith, S. J., O’Hara, M. W., Lamparski, D. M., Romano, J. M., & Volkin, J. I. (1981). An evaluation of major components in a self-control therapy program for depression. Behavior Modification, 5(4), 459–489. doi:10.1177/014544558154002

Sudweeks, C. (1996). Effects of cognitive group hypnotherapy in the alteration of depressogenic schemas. Washington, DC: Washington State University.

Taylor, F. G., & Marshall, W. L. (1977). Experimental analysis of a cognitive-behavioral therapy for depression. Cognitive Therapy and Research, 1(1), 59–72. doi:10.1007/BF01173505

Teismann, T., Dymel, W., Schulte, D., & Willutzki, U. (2011). Resource-focused treatment for unipolar depression: A randomized controlled psychotherapy study. Psychotherapie Psychosomatik Medizinische Psychologie, 61(7), 295–302. doi:10.1055/s-0030-1270453

Tovote, K. A., Fleer, J., Snippe, E., Peeters, A. C., Emmelkamp, P. M., Sanderman, R., … Schroevers, M. J. (2014). Individual mindfulness-based cognitive therapy and cognitive behavior therapy for treating depressive symptoms in patients with diabetes: Results of a randomized controlled trial. Diabetes Care, 37(9), 2427–2434. doi:10.2337/dc13-2918

Zettle, R. D., & Rains, J. C. (1989). Group cognitive and contextual therapies in treatment of depression. Journal of Clinical Psychology, 45(3), 436–445. doi:10.1002/1097-4679(198905)45:3<436::AID-JCLP2270450314>3.0.CO;2-L