Allegiance effects in cognitive processing therapy (CPT) for posttraumatic stress disorder (PTSD): a systematic review and meta-analysis

ABSTRACT

Objective

We sought to determine whether there is evidence of researcher allegiance bias in the reporting of cognitive processing therapy (CPT) for posttraumatic stress disorder (PTSD).

Method

We used a reprint analysis approach – whereby papers were coded for indications of potential bias – to determine the presence and magnitude of researcher allegiance in published randomized controlled trials (RCTs) of CPT.

Results

Twenty trials met inclusion criteria. Evidence of allegiance to CPT rather than the respective comparison conditions was typically small to negligible. A meta-regression analysis of the 17 studies which included an active comparison group did not find an association between allegiance scores and study effect size for the reduction of PTSD symptoms (95% CI: −0.05, 0.19).

Conclusion

There is no evidence at present that the CPT literature has been unduly influenced by allegiance held to CPT or the comparator conditions.

Key Points

What is already known about this topic:

1. Cognitive processing therapy (CPT) is an empirically supported treatment for posttraumatic stress disorder.

2. Researcher “allegiance effects” can include allegiance to a given therapeutic modality.

What this topic adds:

1. Reprint analysis indicated that researcher allegiance to CPT was small to negligible.

2. A meta-regression of 17 included studies did not find evidence of an association between allegiance scores and study-level effect size.

KEYWORDS:

• Researcher allegiance
• allegiance bias
• cognitive processing therapy
• posttraumatic stress disorder

Introduction

While randomized controlled trials (RCTs) have come to be considered the “gold standard” of measuring efficacy of a novel therapy, many biases can compromise their integrity. These biases include, at the study level, selection bias and response bias, and, in the broader literature, publication bias. Such biasing factors can distort the evidence presented either within a given paper or via the decision to publish or not publish results and thus diminish the academic value of a study’s outcomes.

A researcher’s own belief in the superiority of a given therapy included in a study, dubbed “researcher allegiance” (RA), is a key bias that can compromise clinical trials. The therapeutic allegiance of the researcher was first referred to in the psychotherapy literature by Luborsky (1975), who posited that a researcher could inadvertently distort the true effects of psychotherapies depending on their pre-existing belief in the superiority of one of the treatments. In this way, the researcher’s allegiance may lead to an “allegiance bias”, whereby allegiance partially or wholly accounts for treatment effects (Leykin & DeRubeis, 2009).

Definitions of allegiance have varied since this initial hypothesis was proposed. Hollon (1999) described allegiance as an expression of both interest and expertise in one treatment, and neither in the alternative treatment. More recently, Yoder et al. (2019) defined it as a form of conflict of interest involving higher level of skill or training leading to reduced objectivity. As these definitions imply, allegiance can stem from vested interests, such as financial or reputation gain if a study’s results were to support superiority of a given treatment. If the developer of a given treatment conducted a clinical trial evaluating this treatment, they may favour results that show support for their therapy. However, allegiance can also stem from more innocuous sources such as convenience or availability, and thereby have unintended consequences. A researcher can bias a study by selecting a less effective competing treatment, by providing more extensive training to therapists in one condition compared to the other, selecting therapists who have had more exposure or have higher skill in one therapy, or instructing therapists to implement the therapies with different levels of fidelity to original protocols (Dragioti et al., 2015).

Mechanisms and measurement

With reference to the mechanisms underlying allegiance’s effect on study outcomes, McLeod (2009) suggested that the therapist’s or investigator’s allegiance leads to better treatment protocol adherence and competence through more effective training or more motivation to learn the treatment. They claim that investigators may intentionally select highly motivated therapists. Additionally, Yoder et al. (2019) hypothesised that researcher allegiance can also affect outcomes via publication bias, positing that an allegiant author is more likely to elect to not publish results of a “negative” trial, in which no significant difference was found between the study intervention and control. Yoder et al. state in their published protocol (2019) that they intend to examine this hypothesis by analysing grant applications for indicators of allegiance and compare rates of non-publication of negative results, however their findings are yet to be published.

At present, allegiance is very rarely explicitly reported in published research; Dragioti et al. (2015) investigated disclosure of allegiance in published meta-analyses and RCTs and reported that 793 out of 1198 RCTs were “allegiant”. However, only 25 disclosed allegiance. Given this low rate of explicit self-reporting, various methods of coding and measuring allegiance have been devised. Leykin and DeRubeis (2009) used four distinct methods to measure allegiance: “reprint analysis”, which examines attributes in the paper’s introduction and methodology that indicate likely allegiance; examination of previous publications from the author that may indicate a pre-existing bias; conducting interviews with colleagues of the authors; and interviews with authors themselves. Among these methods, reprint analysis would seem to be the most valid: Compared to the examination of previous papers by authors in a given area, reprint analysis allows an explicit weighting of each of a range of indicators of potential allegiance (not just past publications), as opposed to a somewhat subjective interpretation as to whether previous papers indicate an allegiance. The possibility of reliably identifying allegiance simply through interviewing an author or colleague of an author who may be unconscious of their own bias or reluctant to disclose it seems questionable. Finally, reprint analysis is also the most feasible and valid approach given that it relies on a composite of indicators.

With respect to reprint analysis, inferences about author allegiance are necessarily indirect. Authors and “experts” who have previously published papers outlining the development of a therapeutic intervention could be assumed to have a greater than chance likelihood of endorsing or showing allegiance to an approach in which they have invested substantial time and stand to gain career-related benefits. Likewise, if research team members receive more training in one approach than another, it may reflect an implicit favouritism towards delivering one approach correctly, with a potential assumption that quality control is less critical for a comparator arm.

Evidence for researcher allegiance

Mixed conclusions have been drawn in meta-analyses and reviews examining whether allegiance is associated with treatment outcomes (Munder et al., 2013). One early study conducted by Luborsky et al. (1999) reported that 69% of variance in outcomes could be attributed to a researcher’s allegiance. However, more recent meta-analyses have yielded inconsistent findings. S. Miller et al. (2008) found significant RA effects in studies of treatments for mental disorders in youth populations such that entering RA as a covariate reduced heterogeneity in outcomes. However, Tolin (2010), using a similar methodology to investigate studies of cognitive behavioural therapy (CBT) used to treat a range of mental disorders, including anxiety and depression, found that entering RA as a covariate had no significant effect on outcome. Notably, several studies have investigated RA effects across a variety of interventions, potentially obfuscating existing biases by inappropriately combining highly heterogeneous data (Leykin & DeRubeis, 2009).

Several reviews have been more focused with respect to therapy modality. Gaffan et al. (1995) conducted a re-analysis of a meta-analysis of CBT interventions for depression. They concluded that while CBT appeared to be consistently superior in efficacy, half the difference in effect size between CBT and other interventions could be predicted by RA. Using a similar methodology, Goldberg and Tucker (2020) conducted a meta-re-analysis of mindfulness-based interventions (MBIs). While the original meta-analysis (Goldberg et al., 2018) had reported superiority of MBIs compared to other evidence-based treatments, Goldberg and Tucker found that RA was associated with reported superiority of MBIs, and additionally, that MBIs did not show superior efficacy where RA was absent or balanced. Additionally, in the field of PTSD research, Munder et al. (2012) reported a significant RA-outcome association in a meta-analysis of studies of trauma-focused therapy which explained 12% of the variance in outcomes.

However, even in studies of narrower scope, some findings have been directly contradictory; several aforementioned studies detected an association between RA and outcomes in depression-focused therapies, while other meta-analyses failed to establish a similarly robust association. These studies include a meta-re-analysis of cognitive therapy for depression (Gaffan et al., 1995) which found that in older studies (pre-1987) half of outcome variation was predictable by RA, while more recent studies did not demonstrate a strong allegiance effect, and Tolin’s (2010) meta-analysis of CBT for depression which found that even when controlling for allegiance, CBT continued to be associated with significantly stronger treatment effects. This suggests that certain psychotherapy literatures may be more prone to RA than others, with mixed findings regarding treatments for depression, and consistent but limited results in other fields, such as PTSD.

In addition to meta-analyses and reviews, a small number of experimental designs have examined allegiance, such as Wilson et al. (2011) clinical trial conducted across two sites with self-identified differing allegiances. They found no difference between treatment outcomes on each site, indicating no apparent effect of allegiance on outcome.

Given the varied findings across the allegiance literature, it continues to be important to investigate this potentially significant bias in both new and established therapies. “Gold standard” treatments derived from RCTs in many respects have a rigorous evidence base. However, in the clinical psychology literature, there are numerous established therapies from which a non-trivial proportion of the evidence base is comprised of studies from one particular author or author-group. Without investigation of allegiance and other biases, it remains possible that treatment effects may be over-estimated.

Researcher allegiance in the posttraumatic stress disorder literature

The findings regarding researcher allegiance may thus vary according to the measurement approach and the domain of the literature assessed, with relatively less consistent evidence for researcher allegiance effects in studies of the depression literature (e.g., Gaffan et al., 1995; Tolin, 2010). With respect to posttraumatic stress disorder (PTSD), Munder et al. (2012) reported a robust RA-outcome association in a meta-analysis of studies of trauma-focused therapy for PTSD, suggesting that the literature in this area may be prone to bias. Interestingly, Munder et al. (2013) review of meta-analyses found that the strength of the association between RA and outcome did not vary depending on how RA was measured, suggesting that various methods of coding may be equally capable of effectively detecting allegiance. As such, there may be value in further investigating whether the PTSD literature is characterised by high rates of RA, including in more recently developed interventions.

Cognitive processing therapy (CPT) was developed in the late 1980s by Resick and has in recent decades come to be considered an efficacious treatment for PTSD on the basis of its successful symptom reduction in controlled trials (Asmundson et al., 2019). Not only is it a “gold-standard” intervention, it is also increasingly recommended in treatment guidelines (e.g., National Institute for Health and Care Excellence [NICE], 2018; Phelps et al., 2022). Given that CPT has gained such widespread acceptance as an effective treatment approach, it is therefore important to establish that the evidence base has not been undermined by research biases. While there have been different iterations of CPT developed, it remains a relatively homogenous and well-defined therapeutic approach when compared to the wide range of variations to other empirically supported approaches, such as prolonged exposure therapy. For this reason, CPT serves as a good candidate intervention approach for the examination of allegiance effects in the PTSD literature as the lead proponents and developers can be relatively reliably identified when compared with other, more diffuse, therapeutic approaches which have emerged from multiple research traditions.

The present study examined the association between researcher allegiance and outcomes of RCTs assessing the efficacy of CPT for PTSD. As the CPT literature has not previously been examined in this way, this paper aimed to provide insight into the role of RA within this specific field of research, as well as contribute to a broader understanding of RA bias in psychotherapy research. CPT was considered an appropriate candidate and selected as the therapy of focus for the present study on the basis that CPT emerged in recent decades and has come to be considered an efficacious intervention for PTSD (Asmundson et al., 2019; Jericho et al., 2021). To date, no other study has examined the association between RA and outcome measures in the CPT literature.

The existing allegiance literature has provided mixed results regarding allegiance-outcome associations. Without assuming directional hypotheses, the present study intended to explore whether researcher allegiance was associated with direction and magnitude of CPT treatment outcomes (reduction in PTSD symptoms).

Method

This study was pre-registered with PROSPERO, registration ID: CRD42021226145. The review was conducted in line with the PRISMA recommendations, and the PRISMA reporting checklist is included in Supplementary Material.

Inclusion and exclusion criteria

The present study aimed to capture all English-language RCTs that included CPT as a treatment group. Inclusion criteria were: publication in a peer-reviewed journal, study reported in English, inclusion of at least one non-CPT control or comparison group, with participants randomly allocated to condition, use of a structured or semi-structured diagnostic interview, all participants meeting full criteria for PTSD (however, given the time-frame of the studies included, participants met DSM-III-TR, DSM-IV, or DSM-5 criteria), and use of a validated measure for the reporting of PTSD symptom outcomes. We required all participants in each included study to meet criteria for PTSD on the grounds that CPT was developed for treatment of PTSD and given that studies of a clinical intervention – such as CPT – using non-clinical samples would likely be a minority in our overall pool of included studies and introduce unnecessary methodological heterogeneity into the review. We also wanted to ensure that our clinical outcome variable would not be affected by floor effects or a restriction of range if derived from non-clinical samples. Studies which reported secondary data were excluded.

Search strategy

For the present review and meta-analysis, a literature search was conducted on 24 February 2021 and repeated on 9 September 2023 using Ovid, including PsycInfo, Medline, and Embase databases (see full details of search in Supplementary Material). A research librarian was consulted, and search terms were generated using the PICO framework (S. A. Miller & Forrest, 2001). Our search strategy used search terms: “cognit* process* therapy” AND “post traumatic stress” OR “PTSD” OR “post-traumatic stress” OR “posttraumatic stress”.

Screening of records and data extraction

See Figure 1 for an overview of the study inclusion process.

Figure 1. PRISMA flow diagram of the study selection process.

The original 2021 systematic search returned a total of 1233 results, 577 of which were identified as duplicates (by Covidence software) and removed, leaving 656 studies in total. The remaining records were progressed to title-abstract review, which was completed by the author CM and an additional reviewer. Of the 656 studies, 533 (81.25%) were excluded by both reviewers, 65 (9.91%) were included by both reviewers, and 58 (8.84%) were in conflict, yielding a 91% agreement rate and Cohen’s Kappa of 0.64 (which is considered a “substantial” level of agreement by Landis & Koch, 1977). Conflicts were resolved through discussion between reviewers, and a total of 90 papers were included for full-text review.

Full-text review was conducted by author CM. A total of 14 papers were included for data extraction and analysis. For a summary of study characteristics, see Table 1.

Table 1. Study characteristics.

	Study ID	Number of Authors	Population description	Total participants	Trauma type	Non-CPT Condition	CPT Condition	Additional Condition	Primary Outcome Measure
1	Sloan et al. (2018)	4	veteran and nonveteran adults with primary PTSD and stable medication (outpatient)	126	mixed: (highest: adult nonsexual assault) combat, sudden death/violence to friend, adult/child sexual/nonsexual assault, motor vehicle crash, other accident w injury	Written exposure therapy	CPT	0	CAPS-5
2	Resick et al. (2002)	5	female rape victims	171 (ITT)	adult sexual assault (at least 3 months prior, at least one incident)	Prolonged exposure	CPT	minimal attention wait list (MA)	CAPS
3	Forbes et al. (2012)	8	community military veterans	59 (ITT)	military and other comorbid trauma	Treatment as usual (varied; 14% supportive counselling, 52% non trauma-focused symptom management, 34% CBT including exposure)	CPT	0	CAPS
4	Bohus et al. (2020)	12	female survivors of child sexual abuse who also met at least 3 DSM5 criteria for BPD	200	child sexual abuse	DBT-PTSD	CPT	0	CAPS-5
5	Litz et al. (2021)	7	(active duty) deployed marines and sailors with current PTSD diagnosis (DSM-IV)	122 (ITT)	military	Adaptive disclosure	CPT-C (CPT minus written trauma account)	0	CAPS-IV
6	Monson et al. (2006)	6	veteran, chronic military-related PTSD	60 (ITT)	military	Waitlist control	CPT	0	CAPS
7	Surís et al. (2013)	5	veterans with military sexual trauma (MST)	86 analysed (129 enrolled including those withdrawn/not completed)	military sexual trauama	Present-centred therapy	CPT	0	CAPS
8	Maxwell et al. (2016)	6	adults with PTSD	16 (no dropouts or missing data)	mixed (child/adult physical/sexual abuse, witnessing violent death of loved one, and others)	Memory specificity training	Group CPT	0	MPSS-SR
9	Galovski et al. (2012)	5	adults with PTSD	100 (ITT)	physical or sexual child or adult abuse	Symptom monitoring delay treatment	MCPT	0	CAPS
10	Dedert et al. (2019)	6	adults with PTSD who smoke 10 or more cigarettes per day	40	veteran, trauma unspecified	Present-centred smoking cessation therapy (PS), based on ICSC	Combined CPT and smoking cessation (CPTS)	0	CAPS-5
11	Butollo et al. (2016)	4	adults with PTSD related to trauma that occurred in adulthood	141	mixed (accident, interpersonal, loss, medical, other)	Dialogical exposure therapy	CPT	0	IES-R
12	Resick et al. (2015)	11	active duty military personnel	108	deployment-related trauma (except one who cited childhood sexual abuse)	Group present-centred therapy	CPT-C	0	PCL-S
13	Resick et al. (2008)	6	adult women with PTSD	150 (ITT)	mixed (child/adult physical/sexual abuse, mostly multiple traumas)	Written account	CPT	CPT-C	CAPS
14	Chard (2005)	1	women with at least one incident of child sexual abuse	71	child sexual abuse	Minimal attention waitlist	CPT-SA (sexual assault specific)	0	CAPS-SX
15	Sloan et al. (2022)	9	active duty military personnel	169	military	Written exposure therapy	CPT-C	0	CAPS-5
16	Watkins et al. (2023)	10	middle-aged adults (40–65) with PTSD	112	mixed (childhood physical or sexual trauma, adult physical or sexual trauma, military)	Waitlist control	CPT	0	PCL-5
17	Kelly et al. (2021)*	4	women veterans	104	military sexual trauma	Trauma-sensitive yoga	CPT	0	PCL-5
18	Schnurr et al. (2022)	35	veterans	916	military	Prolonged exposure	CPT	0	CAPS-5
19	Kearney et al. (2021)	4	veterans	184	military (combat, sexual, assault, accident, sudden death)	Loving-kindness meditation	CPT-C	0	CAPS-5
20	Simpson et al. (2022)	10	comorbid AUD and PTSD, mixed veteran and civilian	101	mixed	Relapse prevention	CPT	Assessment only	CAPS-5

This search was repeated in September 2023, returning a total of 1880 results, 1485 of which were identified as duplicates and removed, leaving 395 studies. These records were progressed to title-abstract review which was completed by CM. Thirty-four papers were included for full-text review. Six new papers were included for data extraction and analysis, in addition to the original 14 that were included in 2021. A total of 20 papers were included. Data were extracted (means and SDs for post-treatment PTSD scores) by CM and checked by DB.

Coding procedures

As described earlier, many methods for coding allegiance have been devised, each of which may be able to adequately capture an allegiance-outcome association (Munder et al., 2013), however requiring different time and resources. Given the feasibility and time-efficiency of “reprint analysis” compared to contacting authors or their colleagues, much of the recent allegiance bias research has utilised this method (e.g., Goldberg & Tucker, 2020), whereby the content of the publication is analysed and features indicative of allegiance are coded, providing an allegiance “score”. Gaffan et al. (1995) reported that researcher allegiance has frequently been coded on a two- or three-point scale, including positive and negative or positive, negative, and neutral. Highly simplified approaches to coding allegiance that are based on single-item assessment of authorship have also been used (e.g., Manea et al., 2017).

Allowing for specific details that indicate both positive and negative allegiance to be recorded and synthesised as a continuous score, Yulish et al. (2017) devised a 7-item rating scale which includes authorship, as well as supervision of therapists, and any alterations to treatment manuals. These items attempt to target mechanisms of allegiance bias, rather than possible consequences thereof as used by Gaffan et al. (1995). For the present study, Yulish et al’s (2017) rating scale was selected as it has been used multiple times within the allegiance literature (Goldberg & Tucker, 2020). This rating scale (see Table 2) is comprised of seven items that code indicators of both “positive” and “negative” allegiance to each therapy included in a study. The authors carefully reviewed each included study and discussed examples of potential allegiance. For example, the following was considered to be an instance of “advocating” for a particular treatment (adaptive disclosure [AD] in this instance): “An open trial showed that AD was well-received, well-tolerated and led to large effect size reductions in PTSD” (page 3 of Litz et al., 2021).

Table 2. Allegiance scoring tool.

Item	Value	Description
1	+1	If author advocates for treatment or developed treatment
2	+1	If #1 is true, and authors supervised the therapists, were the therapists in their own condition, or the therapists were extensively trained in the treatment
3	+1	If therapists received more supervision/training than other treatment
4	+1	If greater face-to-face dosage compared with other treatment
5	−1	If supervisor is not a recognised expert in treatment
6	−1	If treatment protocol manual was altered by removing ingredient(s) or changing order in a theoretically deleterious manner
7	−1	If therapists were proscribed from responding in ways a reasonable therapist would routinely do AND proscription was egregious (i.e., proscription was judged to be deleterious to treatment)

Researcher allegiance coding was adapted from Goldberg and Tucker (2020), who adapted the tool from Yulish et al. (2017). Only the order of items was edited, to allow for ease of data extraction and scoring. If an item was unclear, it was coded as zero.

Unlike other rating scales, it includes items that may indicate a non-preference for a given therapy, such as therapists being proscribed from conducting activities that would be considered a routine part of the therapy. It also allows for rating of both included therapies, as recommended by Yoder (2019). To assist with the interpretation of results, we considered the difference score (i.e., allegiance score for CPT minus allegiance score for each respective comparator) to be “small” if it ranged from 0 to 2, medium sized if it ranged from 3 to 4, and large if it was >4, on the basis that rarely would all 7-items of allegiance be endorsed in any one treatment arm resulting in a score of 7 and given that differences between treatment arms would rarely be as great as the maximum in any given arm. These categorisations were also derived from careful consideration of the number of allegiance items that would need to be endorsed for any given category and the total range of scores possible for the allegiance difference score. While ultimately arbitrary, this categorisation is slightly more conservative than that used by Munder et al. (2011), in that “large” levels of researcher allegiance required a great degree of discrepancy between allegiance scores for treatment vs comparator arms in the present study.

Data was extracted by the first author. Intent-to-treat (ITT) analyses were favoured whenever reported (n = 10 estimates were ITT). Outcome data extracted was based on availability and measure validity. The primary outcome data extracted were CAPS or CAPS-5 for most studies, however not all studies reported this measure at pre- and post-intervention. For these studies, PTSD Checklist-Specific (PCL-S), Modified PTSD Symptom Scale (MPSS-SR), or Impact of Events Scale-Revised (IES-R) were extracted, based on which primary outcome measure was reported in the study.

Coding for the researcher allegiance scores was completed by the first author and a colleague with 100% agreement. Where an item was not able to be answered based on the information provided in the published text, it was scored 0.

Meta-analysis

A random effects meta-analysis was conducted whereby the posttreatment means and pooled standard deviations were used to generate treatment outcome effect sizes between the CPT and comparator conditions using Comprehensive Meta-Analysis version 3 (CMA; Borenstein et al., 2013). In line with current recommended practice, effect sizes were calculated based upon posttreatment means and standard deviations, rather than pre-post intervention differences (Cuijpers et al., 2017). In instances where standard error (SE) was reported instead of standard deviation (SD), the SE was converted to SD, and when confidence intervals were reported, but not SD, the confidence intervals were converted to SD prior to synthesis of the results (see Higgins et al., 2023). Standardised mean differences were converted to Hedges g as a bias-corrected estimate of effect size that could then be associated with the calculated researcher allegiance scores (summary of extracted data available at: https://osf.io/ntsmy/). A meta-regression analysis was then conducted to determine whether calculated researcher allegiance scores explained heterogeneity in treatment outcome among studies. Only allegiance and no other study characteristics were included as predictor variables in these analyses given the relatively low statistical power for the regression (20 studies in total).

Results

Following the exclusion of unsuitable studies, 20 studies were included in the review (Figure 1). A study by El Barazi et al. (2022) was excluded on the basis that it compared CPT against a pharmacological intervention, precluding a valid assessment of allegiance for the pharmacological condition. The 20 included studies included a total of 3036 participants. Seven of the included studies comprised mixed trauma samples, nine studies only included veteran participants, two studies only included victims of child sexual abuse, one study included only victims of adult sexual assault, and one study included only victims of military sexual assault. Seventeen studies included an active intervention comparison condition, with some form of exposure therapy being the most common (four studies) and three studies included a waitlist control or otherwise non-active comparison condition.

Scores of allegiance to CPT, without consideration of the respective comparator conditions, varied between 0 and 4 (mean = 1.60). Allegiance to non-CPT conditions ranged between −1 and 3 (mean = 0.55). See Supplementary Material for an item-by-item summary of the scores for each study. This slightly lower level of allegiance is consistent with the nature of the included studies, the majority of which included CPT as the experimental condition and therefore may have had a “stake” in its success.

When difference scores were calculated between CPT allegiance and allegiance scores for each respective comparison condition for the 17 studies which compared CPT to an active control condition (as opposed to wait-list), the mean difference score was 0.88 (SD = 1.65), corresponding to a small sized overall allegiance effect per paper.

Overall, the items that accrued the most positive scores were items 1, 2, and 3, relating to authorship and supervision/training. It therefore appears that the quality and the quantity of supervision and training, as well as authorship and authors supervising therapists in their own studies, showed the most evidence of difference between studies, such that training was sometimes more extensive or closely supervised in the CPT condition than the comparator condition.

Meta-analysis

The random effects meta-analysis of the 20 included studies indicated a medium-to-large effect size for reductions in PTSD symptoms following CPT (Hedges g = 0.40, SE = 0.12). A forest plot of study effect sizes is provided in Figure 2. Given that the determination of publication bias was not a priority for the present analyses, a funnel plot is not presented here, but is available for the interested reader in Supplementary Material. When we repeated the meta-analysis based only on the 17 studies which included an active intervention (including “treatment as usual”) the effect size (Hedges g) favouring CPT was 0.25 (SE = 0.11). When we repeated the analyses based only on the 17 of the 20 included studies which used structured or semi-structured interviews to assess PTSD symptoms, Hedges g was 0.42 (SE = 0.14).

Figure 2. Forest plot of effect sizes for cognitive processing therapy (N = 20 studies).

The posttreatment scores for (Sloan et al., 2018) were at 12 weeks for cognitive processing therapy (CPT) and 6-weeks for written exposure therapy (WET), given that the WET intervention was only a 6-week intervention.

The above forest plot includes 17 studies with an active comparison group (including treatment as usual; TAU) and three studies with a waiting list comparison group. The summary effect size should thus be interpreted with caution.

AD = adaptive disclosure; CPT = cognitive processing therapy; DBT = dialectical behaviour therapy; PCT = present-centered therapy; PE = prolonged exposure; TAU = treatment as usual; WA = written accounts; WET = written exposure therapy; WL = waiting list.

The next step was to determine whether ratings of allegiance (on a scale −4 to 3) were associated with treatment outcomes (Hedges g for each study). A meta-regression was conducted whereby the study effect size (Hedges g) was regressed upon the researcher's allegiance difference score (i.e., allegiance score for CPT minus allegiance score for the control/comparison condition). No other predictor variables were included to preserve statistical power. Results indicated that the co-efficient for study allegiance difference score in predicting effect size was not significantly different to zero (95% CI: −0.02, 0.26), suggesting that allegiance was not associated with study effect size. See Figure 3 for a regression plot of effect size on allegiance difference scores. Likewise, when the CPT allegiance score alone (rather than the difference score) was considered, the results were also not significant (95% CI: −0.02, 0.26). When we repeated the regression analysis without the three studies which included non-active control comparisons, the results for the remaining 17 included studies also indicated that allegiance was not associated with study effect size (95% CI: −0.05, 0.19). The regression co-efficient of 0.07 for these 17 studies indicates that study effect size changes by only 0.07 for every 1-unit increase in researcher allegiance difference score.

Figure 3. Regression plot of effect size on allegiance difference scores.

We conducted three sensitivity analyses to determine whether variations in calculation of the total score for the allegiance measure were associated with any changes in the association between allegiance and treatment outcome. First, we repeated the analyses when the allegiance total score was derived only from the positive allegiance items. Next, we repeated the analyses by focusing only allegiance scores for the CPT (not comparator/control). Finally, we repeated the analyses after excluding items 1 (if the author advocates for treatment or developed treatment) and 5 (if supervisor is a recognised expert in treatment) in case our coding of expert status lacked reliability. In each case, there was no significant association between allegiance score and study effect size. A detailed summary of the results of the sensitivity analyses is reported in Supplementary Material for the interested reader.

Discussion

While allegiance bias has now been investigated across a variety of treatment literatures, this is the first investigation of allegiance bias in the CPT literature. The current study focused on RCTs of CPT given the methodological advantages of random allocation of participants to comparator conditions of CPT for PTSD.

A reprint analysis of the included papers found that allegiance scores for CPT were typically greater than those for the respective comparison conditions, but only an overall small-sized allegiance effect when considered as a difference score. Overall allegiance effect scores were mostly contributed to by papers where authors were developers of CPT, but not the comparator therapy. The reported supervision and training characteristics of each study also contributed to the scores. With this in mind, efforts to counter allegiance effects might aim to ensure that treatment approaches are rapidly disseminated to non-developer researchers for further efficacy testing or that comparisons with existing interventions involve developers or proponents of each approach. The non-inferiority trial of Sloan et al. (2018) – whereby established experts and proponents of each intervention served as supervisors or each arm – serves as an example of how this might be done.

Our meta-analysis identified no significant relationship between scores of researchers’ allegiance to CPT and its performance in reducing PTSD symptoms in RCTs. Additionally, researchers’ allegiance to the various non-CPT comparator therapies was also not associated with therapeutic outcomes. The study therefore failed to detect an “allegiance effect” either of positive bias towards CPT or of a negative bias against comparators. Given these findings, there is no evidence that the existing body of clinical research investigating the effects of CPT on PTSD is biased by researcher allegiance. This provides confidence that the benefit derived from CPT in trials is indeed an effect of the therapy itself, rather than an artefact of researcher allegiance.

The present study’s failure to find an association between scores of researcher allegiance and study outcomes is not an isolated finding. While numerous studies have found a robust association between allegiance and outcome, others, such as Tolin (2010) who investigated allegiance in studies of CBT for depression, did not establish that allegiance contributed strongly to treatment effects. At the experimental level, Wilson et al. (2011) clinical trial conducted across two sites with differing allegiances found no difference between treatment outcomes on each site. As with all null findings, it is important to note that this absence of evidence is not evidence of absence, particularly in allegiance bias research where measurement is inherently difficult. It does not therefore prove that allegiance is not present in the study’s investigated literature base. Rather, it provides no evidence that there is reason for concern about the impact of researcher allegiance bias in this literature to date. We acknowledge that our review may have overestimated effect sizes, as only English language articles were included (Egger et al., 1997), and there is evidence to suggest that systematic reviews that exclude non-English articles may be of lower quality (see Moher et al., 2003). Future reviews of allegiance should endeavour to include non-English articles.

Regarding the existing research on allegiance bias in the PTSD literature, the present study contrasts with Munder et al. (2012) meta-analysis of trauma-focused therapies for PTSD, which found that allegiance was a significant moderator of treatment outcomes, accounting for 12% of variance. However, Munder’s review included only two out of 20 studies which included CPT. It is therefore possible that CPT’s structured, manualised approach may be a protective factor against allegiance bias relative to other trauma-focused therapies.

Notably, the inclusion of positive and negative allegiance scoring items applied to both the CPT condition and the comparator therapy allowed for directional allegiance to be assessed in both therapies. This practice of rating the researcher’s allegiance to both interventions to allow for detection of directional or balanced allegiance in a study was encouraged by Yoder et al. (2019) and therefore implemented in the present study, increasing the breadth of findings as allegiance to both interventions was estimated. The null finding in both CPT and comparator therapies indicates that not only is there no evidence that researchers biased conditions in favour of their preferred therapy, but there is also no evidence that they designed studies to undermine the comparator condition by implementing the therapy less rigorously.

While “reprint analysis” to estimate allegiance has become standard in the researcher allegiance literature and was used in the present study, Leykin and DeRubeis (2009) cautioned against this method. They state that authors frequently write a study’s manuscript after seeing results and therefore the reprint method may detect the author’s post-result allegiance rather than a true pre-existing allegiance to the intervention. Berman et al. (1985) tested this assertion by examining allegiance in researchers’ current and previous publications; if allegiance was influenced by findings in the present paper, a difference would be expected between markers of allegiance in the present paper compared to the author’s previous publications. No significant difference was found, suggesting that RA remained stable, supporting the reprint method as an effective measure. However, it remains possible that if a paper’s introduction and method were written after results were known, the results may have influenced the content of these sections. It is therefore recommended that this post-hoc writing practice is avoided and that best practice guidelines are adhered to, to avoid the undue influence of results on a paper’s reporting.

One inherent challenge of the allegiance bias literature is the difficulty of defining and measuring the concept of allegiance. Any measurement of allegiance based on scoring possible indicators within a publication is an estimate, rather than a factual statement of an allegiance. As no tool for measuring allegiance has yet been empirically validated, it could be questioned whether the method used for scoring allegiance was adequately sensitive to detect an allegiance effect in the present review. This limitation is compounded by an absence of detailed coding instructions and descriptors for some items which are difficult to quantify (e.g., how much does an author need to be “advocating” an approach for the advocating item to be endorsed). However, this same tool was used by Yulish et al. (2017) and Goldberg and Tucker (2020), who reported that outcomes of problem-focused therapies for anxiety and mindfulness-based therapies, respectively, could be predicted by allegiance, suggesting that the scoring tool is sufficiently sensitive to allegiance to have captured this construct in previous studies. Therefore, it is unlikely that the present study’s failure to find an effect is due to the selection of this scoring tool. Additionally, the use of reprint analysis is consistent with current best practice in researcher allegiance studies, as alternative methods rely on self-report or colleague-report, which fail to capture the potentially implicit nature of allegiance and may result in under-estimation of allegiance due to reluctance to self-disclose bias. It is therefore recommended that reprint coding systems continue to be refined and validated, as Yoder et al. (2019) reported they would undertake in their published protocol. It will also be important that thresholds of the extent of allegiance are validated for any improved tool, rather than remaining arbitrary as they were in the present study. The validation of an allegiance scoring tool would provide significant safeguards for future allegiance research, ensuring that an association is able to be detected if present, increasing confidence in the accuracy of such studies.

While the selected tool for scoring allegiance had been used previously, it presented challenges in accurately answering all questions, primarily where the required data was not provided in the respective publications, resulting in several items providing too little information. For example, only one of the included studies explicitly described their supervisors as “non-expert” (Resick et al., 2002) in any given therapy (item 5 in the scoring tool). Neither Yulish et al. (2017) nor Goldberg and Tucker (2020) reported whether they operationalised this item in some way other than self-reported expert status in-text, so for the purposes of the present study and consistency with previous use of the tool, this item was scored zero for all other studies as it was not possible to establish with confidence that the supervisor was not of expert skill. Additionally, few studies provided complete descriptions of the type and amount of supervision provided to therapists in each condition. Items 2 and 3 relating to supervision therefore were not always able to be accurately assessed, and variation may have been underestimated.

As a result of these reporting practices that made it difficult to quantify researcher allegiance, it is likely that a greater number of items were scored zero than if all data had been provided. While an absence of information about allegiance is not strictly equivalent to a lack of allegiance, the allocation of a zero score for items where there was insufficient information is conservative in the sense that it does not actively contribute to a total score which would indicate the presence of possible allegiance. The zero scores for some items appear to have contributed to the limited range of total scores, which ranged from −1 to 4, indicative of a trend towards conservative ratings, which may have failed to capture the full variance in allegiances held by the respective study authors. This limitation reflects the insufficient information provided in many publications about therapy supervisors’ level of expertise and experience and details regarding manual use and therapy instruction. It is therefore recommended that future comparative studies report these details in aid of transparency regarding therapeutic practices.

There have been a range of responses to previous findings of allegiance effects. Some authors, including Berman and Reich (2010), have suggested that if an allegiance effect is present, it should be statistically corrected for in RCTs, while others (e.g., Wilson et al., 2011) have emphasised the importance of reducing the potential for allegiance bias in the first instance. It has been noted that if an association is not present, and an attempt is made to statistically correct for it, further bias may be introduced rather than removed (Gaffan et al., 1995). The present study’s failure to find an association between allegiance and outcome supports the view that if an allegiance bias is only sometimes present, generalised recommendations of statistical correction may introduce further bias. Rather, it is recommended that controlled trials be held to high reporting standards, ensuring integrity and transparency in reporting practices, using guides such as the Journal Article Reporting Standards (JARS; Appelbaum et al., 2018). This would also allow for more accurate estimates of allegiance to be derived from reprint analysis, which remains the most feasible and efficient method of coding allegiance. Additionally, Munder et al. (2011) found researcher allegiance-outcome associations to be significantly lower in comparisons with high internal validity, noting that where there were experimental control deficits, allegiance was more predictive of outcomes. As such, allegiance is not only more easily measured when rigorous reporting practices are followed, these practices may also themselves be a defence against the biasing effects of allegiance.

Summary and recommendations

CPT is an increasingly recommended treatment for PTSD. It appears to lead to significant therapeutic outcomes and symptom reduction. However, as with all emerging therapies, it is important that biases such as researcher allegiance be considered with respect to the evidence base. The current study did not find an association between researcher allegiance and therapy outcomes, either in the direction of creating favourable conditions for CPT or unfavourable conditions for the comparator condition, increasing confidence that the outcomes achieved by CPT in clinical trials are in fact a result of the therapeutic intervention and not an artefact of author allegiance. Nevertheless, there is a need to better operationalise and measure author allegiance and future trials, both in the CPT literature and in psychotherapy research broadly, need to implement transparent and thorough reporting practices, not only to allow for more accurate estimates of allegiance but also to control for the potentially biasing effects of researcher allegiance.

Author contributions

CM devised search terms, conducted the literature search, selected included articles, summarised and synthesised the data and drafted the paper. DB conducted the meta-analysis, assisted with the drafting of the paper and approved the final version of the paper for submission.

Acknowledgments

The authors would like to thank Brittany Smith who served as a co-rater for the inclusion of studies and the calculation of allegiance scores.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Data availability statement

The data used in the present study are available from the authors upon request.

Supplemental data

Supplemental data for this article can be accessed at https://doi.org/10.1080/13284207.2024.2347643.