Selective outcome reporting in trials of behavioural health interventions in health psychology and behavioural medicine journals: a review

ABSTRACT

Selective outcome reporting can result in overestimation of treatment effects, research waste, and reduced openness and transparency. This review aimed to examine selective outcome reporting in trials of behavioural health interventions and determine potential outcome reporting bias. A review of nine health psychology and behavioural medicine journals was conducted to identify randomised controlled trials of behavioural health interventions published since 2019. Discrepancies in outcome reporting were observed in 90% of the 29 trials with corresponding registrations/protocols. Discrepancies included 72% of trials omitting prespecified outcomes; 55% of trials introduced new outcomes. Thirty-eight percent of trials omitted prespecified and introduced new outcomes. Three trials (10%) downgraded primary outcomes in registrations/protocols to secondary outcomes in final reports; downgraded outcomes were not statistically significant in two trials. Five trials (17%) upgraded secondary outcomes to primary outcomes; upgraded outcomes were statistically significant in all trials. In final reports, three trials (7%) omitted outcomes from the methods section; three trials (7%) introduced new outcomes in results that were not in the methods. These findings indicate that selective outcome reporting is a problem in behavioural health intervention trials. Journal- and trialist-level approaches are needed to minimise selective outcome reporting in health psychology and behavioural medicine.

KEYWORDS:

• Selective outcome reporting; outcome reporting bias; behavioural health interventions; health psychology; behavioural medicine; open research

Selective outcome reporting in clinical trials involves reporting only some of the prespecified outcomes examined, including outcomes that were not prespecified, and/or changing the importance of outcomes in terms of primary and secondary importance (Kirkham et al., 2010; Thomas & Heneghan, 2022). Outcome reporting bias (ORB) arises from selective outcome reporting and involves reporting a subset of prespecified outcomes in the final publication based on knowledge of the results (Hutton & Williamson, 2000). This can occur due to awareness that significant results are more likely to be published than negative or null findings (DeVito & Goldacre, 2019; Hopewell et al., 2009) or that there may be a ‘time lag’ in publishing studies with null and/or negative findings (Dwan et al., 2008).

Selective outcome reporting is problematic because it can lead to overestimation of treatment effects and inflated effect sizes (Ioannidis, 2008; Macura et al., 2010; Shah et al., 2020; Thomas & Heneghan, 2022), increased frequency of false positive results (Olsson-Collentine et al., 2023), and misinterpretation of trial findings (Cristea & Naudet, 2019; Thomas & Heneghan, 2022). Selective outcome reporting can also contribute to research waste (Thomas & Heneghan, 2022; Yordanov et al., 2018) and removes transparency and openness in research reporting (Hagger, 2019; Ioannidis et al., 2017). Despite the adverse consequences of selective outcome reporting, high levels have been observed in trials within and across health areas. For example, in a systematic review of studies examining ORB in reports of randomised controlled trials (RCTs), 40–62% of studies reported at least one outcome that was changed, omitted, or introduced (Dwan et al., 2013). In a review of trials submitted to the BMJ, 14% of reported outcomes were not specified a priori, and 10% of pre-specified outcomes were not reported (Weston et al., 2016). Trials across a range of health areas have identified selective outcome reporting. For instance, in a review of surgical RCTs, 30% of outcomes were found to be incompletely reported (Wang et al., 2023); 14% of pharmacotherapy trials examined in another review demonstrated irregularities between published primary outcomes and those included in the trial registration (Lancee et al., 2022). A review of rehabilitation trials identified discrepancy rates of 61% and 27% between trial abstracts and main texts respectively, and study registrations (Komukai et al., 2024). A review of psychotherapy randomised controlled trials found that only 4.5% of reviewed trials had no discrepancies between the article and the trial registry (Bradley et al., 2017).

Evidence of selective outcome reporting in health psychology and behavioural medicine is scant. However, the prevalence of selective outcome reporting has been shown to be similar across psychological disciplines, with over 40% of American (John et al., 2012) and Italian (Agnoli et al., 2017) and over 50% of Brazilian (Rabelo et al., 2020) psychology researchers self-reporting some form of selective outcome reporting in their research. Examinations of differences between doctoral dissertations in psychology and subsequent publications have also found that 18% omitted outcomes and 9% added new outcomes in publications (Cairo et al., 2020). Further, a review of concordance between behavioural health trial registrations and primary outcome reporting specifically in the journal BMC Public Health identified that 70% of papers had selectively reported outcomes (Taylor & Gorman, 2022).

Behavioural interventions, which aim to change health and/or health behaviour outcomes (Michaelsen & Esch, 2022), focus on a range of disease areas and health outcomes. Findings from trials of these interventions inform health research, practice, and policy (Heneghan et al., 2017; Matvienko-Sikar et al., 2020). Thus, selective outcome reporting in such trials represents a ‘threat to evidence-based healthcare’ (Weston et al., 2016). Moreover, research in health psychology and behavioural medicine has the potential to profoundly influence individual, community, and population health and wellbeing (Segerstrom et al., 2023).

There is an increasing focus on improving the conduct and reporting of research in health psychology and behavioural medicine, particularly in terms of openness and transparency (Hagger, 2019; Kwasnicka et al., 2021; Segerstrom et al., 2023), which is impeded by selective reporting practices (Hagger, 2019; Ioannidis et al., 2017). For instance, a recent survey of international experts in open science and health psychology identified ‘examination of the extent that open science behaviours are currently practiced in health psychology’ as the top research priority in this area (Norris, Prescott, et al., 2022). Examining selective outcome reporting in trials of behavioural health interventions will likely enable identification of whether, and to what extent, selective reporting occurs, which is important to guide future research to improve the conduct, reporting, and transparency of trials of behavioural health interventions. To date, we are not aware of such a review across the health psychology and behavioural medicine literature. The aim of this review was therefore to examine selective outcome reporting in a broad sample of trials of behavioural health interventions, and to determine the potential for ORB.

Materials and methods

This review is reported in line with the Preferred Reporting for Items in Systematic Reviews and Meta Analyses (PRISMA) statement (Page et al., 2021). The review was pre-registered on PROSPERO (registration number: CRD42022345015).

Eligibility criteria

Inclusion criteria

Studies were eligible for inclusion if they reported on a RCT of a behavioural health intervention. The definition of a behavioural health intervention was operationalised as any intervention involving the active participation of any target group, with the goal of changing health outcomes and/or health behaviour outcomes (Michaelsen & Esch, 2022). Eligible studies had to report at least one health or health behaviour outcome. To gain insight into selective outcome reporting across health psychology and behavioural medicine trials, there were no restrictions on specific types of health or health behaviour outcomes. Studies were required to be published in or after January 2019. This range was chosen to allow sufficient time following the publication of the Consolidated Standards of Reporting Trials Statement for Reporting Social and Psychological Interventions (CONSORT-SPI 2018) in July 2018 (Montgomery et al., 2018) for researchers to follow guidelines regarding outcome reporting in this area. As such, the date range was chosen to capture those papers for which we would expect best practice in outcome reporting due to area-specific guidance on this presented in CONSORT-SPI 2018. Studies were required to be published in English, but there were no restrictions on trial location, population, or intervention or behaviour type.

Exclusion criteria

Studies that did not report on a RCT were excluded. We thus excluded quasi-randomised studies, feasibility/pilot trials, interim analyses, and secondary analyses of trials.

Search strategy

To enable the inclusion of a broadly representative sample of trials of behavioural health interventions, nine health psychology and behavioural science journals were searched: Health Psychology, Journal of Health Psychology, British Journal of Health Psychology, Journal of Behavioural Medicine, Annals of Behavioural Medicine, Psychology & Health, Applied Psychology: Health and Well-being, Translational Behavioural Medicine, and International Journal of Behavioral Medicine. These journals were chosen as they are all Q1 or Q2 journals, with good impact factors (currently ranging from 3.1 to 8.1), which are recognised by the research community in these areas as publishing robust and high-quality behavioural health research. Journals were searched for articles published from 1 January 2019 to 14 June 2023. Journals were individually searched using the online journal/publisher search function using the following broad search strategy: Random* Contro* Trial OR RCT. Broad search terms were used to identify all potentially eligible RCTs of behavioural health interventions because this review was not focused on specific intervention types or behaviours.

Identification of protocols and trial registrations for included studies was conducted via hand-searching papers for references to a trial registration and/or a published protocol. Where no trial registration or protocol was reported in-text, manual search was conducted up to 14 June 2023, on Clinicaltrials.gov and the World Health Organization's (WHO's) International Clinical Trial Registry Platform (ICTRP) search portals to identify corresponding registrations. Where protocols and/or trial registrations could not be identified through journal and registry searches, corresponding trial authors were emailed.

Screening

Titles and abstracts of identified papers were independently screened in duplicate by two reviewers (JOS, KMS, SK, or ST), followed by full text screening, which was also conducted independently in duplicate by two reviewers (JOS, KMS, SK, or ST). Inter-rater reliability was κ > 0.8 at all stages, indicating very high levels of agreement (McHugh, 2012). Any disagreements during screening were resolved by consensus discussion and/or recourse to a third reviewer where needed.

Data extraction

Data were extracted from the methods and results sections of each trial report and from the corresponding protocol/trial registration, where available, under the following headings: general information (author name, title, date of publication, country, protocol/trial registration, year of protocol/registration); study information (health area, intervention type, setting, population, outcome type); participant characteristics (sample size); intervention characteristics (intervention target, duration, mode of delivery, number of assessments); and outcome reporting (number and details of pre-specified outcomes in protocol/registration and methods, number and details of outcomes reported in the results section, number and details of statistically significant and null outcomes), and selective outcome reporting (number and details of prespecified outcomes not reported, number of new outcomes introduced, changes in outcome importance level). Statistically significant outcomes were considered as those reported as significant in trial reports, typically if p<0.05 or, in the absence of a p-value, an effect estimate with a CI that was indicative of p<0.05. All data were extracted using a pre-specified data extraction form (Supplementary File 1) by at least one reviewer (JOS, SK) and checked by a second reviewer (KMS, SK). For trial registrations and protocols, data were extracted from the most recent version.

Identification of outcome reporting discrepancies

At least one reviewer (JOS or SK) compared outcomes reported in the results sections of included studies with outcomes reported in the methods sections and the study protocols/registrations where available. All comparisons were checked by a second reviewer (KMS). Classification of consistencies and discrepancies between prespecified outcomes and those reported in results sections was informed by previously described classification approaches (Chan et al., 2004; Howard et al., 2017; Mathieu, 2009). Studies were considered to have no discrepancies from prespecified outcomes if all outcomes were reported in the methods and results section as they were in the study protocol/registration (where available). Similarly, if no study protocol/registration was available, studies were considered to have no discrepancies in outcome reporting between methods and results, if outcomes were reported in the study results section as they were in the methods section.

Decisions on discrepancies were informed by previously used methods (Chan et al., 2004; Mathieu, 2009) and were noted if: (1) a pre-specified outcome in the study protocol/registration was omitted from the results paper; (2) a new outcome that was not prespecified in the protocol/registration was introduced in the results paper; (3) an outcome included in the methods was omitted from the results; (4) a new outcome that was not included in methods was introduced in the results; (5) a primary outcome in either the methods and/or study protocol/registration was downgraded to a secondary outcome in the results; (6) a secondary outcome in the methods and/or study protocol/registration was upgraded to a primary outcome in the results. In addition, discrepancies were considered to potentially indicate bias if an outcome introduced in the results section, that was not included in the protocol/registration or the methods section, was reported as statistically significant. Discrepancies were also considered to potentially indicate bias if an outcome upgraded from a secondary outcome to a primary outcome was subsequently reported as statistically significant, or an outcome downgraded from a primary outcome to a secondary outcome was subsequently reported as non-significant.

Data synthesis

Data were assessed using narrative synthesis and in tabular format. Descriptive statistics were calculated for study characteristics. Outcome reporting was presented narratively and in tabular format to include the number and percentage of the types of outcome discrepancies as well as the number and percentage of trials for which there are no discrepancies. Results are presented first in terms of discrepancies between prespecified outcomes in the study protocol/registration and the results. As discrepancies between methods and results can be considered to occur at a different level to discrepancies from pre-specified outcomes, such discrepancies are then presented separately. Descriptive subgroup analysis was conducted for trials with a corresponding protocol/registration based on whether the protocol/registration was before publication of CONSORT-SPI 2018 (i.e., before or after 2019). This represents a change from the review protocol which was conducted to examine if availability of CONSORT-SPI 2018 had an impact on selective outcome reporting. Relationships between ORB and statistical significance were also examined descriptively due to the small sample size that would limit the meaningfulness of inferential statistics.

Results

The search of the nine health psychology and behavioural medicine journals identified 1134 potentially relevant records. Of these, 43 trials were eligible for inclusion in this review. See Figure 1.

Figure 1. PRISMA flow chart of included studies from original search to final stage. Note. All trials with protocols also had corresponding registrations.

The majority of the 43 trials were conducted in North America (n = 16, 37%) or Europe (n = 14, 33%). Health areas targeted included physical activity and/or sedentary behaviour (n = 10, 24%), obesity and/or weight (n = 8, 19%), and diet (n = 5, 12%). Behavioural health interventions examined included physical activity interventions (n = 8, 19%), implementation intention interventions (n = 4, 9%), and informational/educational interventions (n = 4, 9%). The majority of populations included were non-clinical (n = 29, 67%) and the majority of trials were conducted with adults (n = 41, 95%). Twenty-nine trials (67%) were registered on a trial registry; 24 (56%) also had a published protocol. The majority of trials were registered and/or had published a protocol before 2019. The mean sample size was 239 participants. See Table 1 for full study characteristics.

Table 1. Study characteristics.

Characteristics	N (%) of trials
Trial Country  Australasia  Asia  Europe  North America  South America  Multiple countries*	2 (4.65%) 8 (18.60%) 14 (32.56%) 16 (37.21) 2 (4.65%) 1 (2.33%)
Broad Health Area  Cancer  Cardiovascular disease  COVID-19  Diabetes  Diet  Health (general)  Multiple  Obesity/Weight related  Oral health  Physical activity/sedentary behaviour  Sexual health  Substance use (e.g., alcohol, cigarettes)	3 (6.98%) 1 (2.33%) 2 (4.65%) 3 (6.98%) 5 (11.63%) 2 (4.65%) 3 (6.98%) 8 (18.60%) 1 (2.33%) 10 (23.81%) 1 (2.33%) 4 (8.89%)
Intervention Type  Cognitive behavioural therapy (CBT)  Implementation intentions intervention  Informational/educational based intervention  Mindfulness  Motivational intervention  Other**  Physical activity intervention  Psychological inoculation  Self-regulation intervention  Support-based intervention  Therapeutic/counselling (other than CBT)	3 (6.98%) 4 (9.30%) 4 (9.30%) 2 (4.65%) 3 (6.98%) 7 (16.28%) 8 (18.60%) 2 (4.65%) 3 (6.98%) 4 (9.30%) 3 (6.98%)
Population  Clinical/patient population (child/adolescent)  Clinical/patient population (adult)  General population (child/adolescent)  General population (adult)	1 (2.33%) 13 (30.23%) 1 (2.33%) 28 (65.12%)
Trial protocol/registration  Prospective  Trial registration  Protocol  Retrospective  Trial registration  Protocol  No protocol/registration	24 (55.81%) 24 (100%) 19 (79.17%) 5 (17.24%) 5 (100%) 5 (100%) 14 (32.56%)
Year of protocol/registration
2016 and earlier	9 (31.03%)
2017	4 (13.79%)
2018	6 (20.69%)
2019	7 (24.14%)
2020	3 (10.34%)
	M(SD)
Trial sample size	238.63 (301.13)

*Multiple countries included: United Kingdom, United States of America, New Zealand, Serbia, Bangladesh.

**’Other’ interventions included, for example, Health Action Process Approach-based interventions, just-on-time interventions.

Discrepancies in outcome reporting were observed for trials published in all reviewed journals. Twenty-eight of the 43 reviewed trials (65%) demonstrated at least one discrepancy in findings either between the trial protocol/registration and the results paper or between the methods section and the results section of the final report. Only three of the 29 trials with discrepancies (10%) reported reasons for discrepancies. Reasons for discrepancies included omission due to missing outcome data and modification to data collection due to COVID-19; one trial stated some outcomes were ‘not reported here’ but did not provide additional information. See Table 2 (see Supplementary File 2 for study level discrepancies). In a subgroup analysis based on whether trials had protocols/registrations prior to 2019, the majority of both trials with protocols/registrations prior to 2019 (86%) and after 2019 (90%) demonstrated some discrepancies between the protocol/registration and the results. More trials with protocols/registrations after 2019 (76%) than those with protocols/registrations before 2019 had outcomes omitted. (See Supplementary File 3).

Table 2. Outcome reporting in included trials (n=43).

Trial Level Discrepancies
	N applicable trials	N trials with a discrepancy	(%) trials
Any discrepancies in outcome reporting
Outcomes in results different from either protocol/registration or methods	43	29	67.44%
Outcomes in results paper different from protocol/registration	29	27	93.10%
Outcomes in results different from methods	43	6	13.95%
Outcomes omitted in results
Pre-specified outcome from protocol/registration	29	21	72.41%
Outcome from methods	43	2	4.65%
New outcomes introduced in results
New outcome not prespecified in protocol/registration	29	16	55.17%
New outcome not included in methods	43	4	9.30%
Both omitted and introduced prespecified outcomes in results*	29	11	37.93%
Change in level of outcome importance
Primary outcome in protocol/registration downgraded to secondary outcome	29	3	10.34%
Primary outcome in methods reported downgraded to secondary outcome	43	0	0%
A secondary outcome in the study protocol/registration upgraded to primary outcome	29	5	17.24%
A secondary outcome in the methods upgraded to primary outcome in the	43	0	0%
Potential outcome reporting bias
Newly added outcomes reported as significant	18	8	44.44%
Downgraded outcomes reported as significant	3	1	33.33%
Upgraded outcomes reported as significant	5	5	100%
Reasons for discrepancies reported	29	3	10.34%
Outcome Level Discrepancies across Trials		Range (Median)	Range (%)
No. outcomes included in results that were prespecified in protocol/registration	29	1–17 (7)	20–100%
Pre-specified outcomes from protocol/registration omitted in the results	29	1–20 (5)	14–83%
New outcomes, not prespecified in protocol/registration, introduced in results	29	1–9 (2)	13–83%
Outcomes from methods omitted in the results	43	2–5 (2)	12–45%
New outcomes, not in methods, introduced in results	43	1–2 (1.5)	10–67%

Note: percentages for discrepancies are based on number of applicable trials relevant for that discrepancy (e.g., 29 trials with registrations/protocols).

* no trials had outcomes both omitted and introduced between the methods and results section; this information is presented for discrepancies with protocols/registrations only.

Discrepancies or changes to levels of outcome importance between trial registration/protocol and results

Corresponding registrations/protocols were available for 29 (67%) of the 43 included trials (Figure 2(a)). Twenty-six (90%) of these 29 trials had at least one discrepancy from the prespecified outcomes. Twenty-one (72%) of the 29 trials omitted at least one prespecified outcome from the trial registration/protocol in the results paper. Sixteen (55%) of the 29 trials introduced new outcomes in the results paper that were not included in the trial registration and/or protocol. Eleven (38%) of the 29 trials both omitted prespecified outcomes and introduced new outcomes in results paper.

Figure 2. Discrepancies in outcome reporting in reviewed trials.

The number of outcomes reported in the results paper that were prespecified in trial registrations/protocols ranged from 1-17 outcomes (median = 7), with 20%-100% of outcomes in reported papers being prespecified in trial registrations/protocols. The number of outcomes that were prespecified in trial registrations/protocols but were omitted in the results paper ranged from 1-20 outcomes (median =  5); 14%-83% of prespecified outcomes were omitted. The number of new outcomes that were introduced in the results paper (i.e., not prespecified in trial registrations/protocols) ranged from 1-9 outcomes (median =  2); 13%-83% of reported outcomes of outcomes reported in the results were newly introduced and not prespecified in trial registrations/protocols.

In terms of changes to levels of outcome importance, in three (10%) of the 29 trials with corresponding registrations/protocols, outcomes were downgraded from a primary outcome in the trial registration and/or protocol to a secondary outcome in the published results. In five (17%) of the 29 trials, outcomes were upgraded from a secondary outcome to a primary outcome.

Discrepancies between trial methods and results sections

Six (14%) of the 43 reviewed trials had discrepancies between methods and results sections of the published report. Three trials (7%) omitted outcomes from the published methods section in the results (Figure 2(b)). Of these, one trial omitted two (13%) of 17 outcomes, one omitted one (10%) of 10 outcomes, and the other omitted five (45%) of 11 outcomes. Three trials (7%) introduced new outcomes in the published results that were not included in the methods section. The number of outcomes introduced ranged from one to two; the percentage of outcomes reported in the results that were newly introduced ranged from 10%-67% of outcomes. There were no observed instances of the level of outcome importance being changed between the methods and the results sections in any reviewed trials. In a subgroup analysis based on whether trials had protocols/registrations prior to 2019, no trials with protocols/registrations prior to 2019 demonstrated any some discrepancies between the methods and the results. Nineteen percent of trials with protocols/registrations after 2019 demonstrated discrepancies. (See Supplementary File 3).

Outcome reporting bias

In 19 trials, outcomes were added into the results section from the protocol/registration (n =  16; Figure 2(a)) and/or methods section (n = 3; Figure 2(b)). See Table 3. In these trials, a newly added outcome was reported as statistically significant in 16 trials (84%). Of these 16 trials, 26 newly added outcomes were reported as statistically significant across 13 trials (81%). Of the three trials in which outcomes were introduced in the results that were not in the methods, three newly added outcomes were reported as statistically significant across the 3 trials (100%). Of the three trials in which outcomes were downgraded from primary outcomes in the protocol/registration to secondary outcomes in the results (Figure 2(a), Table 3), one outcome was reported as statistically significant in only one trial (33%); seven downgraded outcomes were reported as null across two trials. Of the five trials in which outcomes were upgraded from a secondary outcome in the protocol/registration to a primary outcome in the results (Figure 2(a), Table 3), all upgraded outcomes were reported as statistically significant.

Table 3. Potential outcome reporting bias.

	Outcome reported as significant		Outcome reported as null
	N trials (%)	N outcomes (range)	N trials	N outcomes (range)
Outcome added (total n = 19 trials)	16 (84%)	29 (1–7)	12 (63%)	14 (1–4)
From protocol/registration (n = 16 trials)	13 (81%)	26 (1–7)	10 (59%)	12 (1–4)
From methods (n = 3 trials)	3 (100%)	3 (1)	2 (67%)	2 (1)
Outcome downgraded from primary to secondary (n = 3 trials)	1 (33%)	1 (1)	2 (67%)	7 (1–6)
Outcome upgraded from secondary to primary (n = 5 trials)	5 (100%)	10 (1–5)	0	0

Note. Omitted outcomes are not presented here as information on significance of these outcomes was not included in reviewed studies. Trials could present both statistically significant and null newly added, downgraded, and upgraded outcomes.

Discussion

To our knowledge, this review is the first examination of selective outcome reporting in a broad cohort of trials of behavioural health interventions in health psychology and behavioural medicine. Our findings demonstrate that some form of selective outcome reporting occurred in 67% of the trials included in this review. This is higher than previously observed levels of selective outcome reporting in broad cohorts of RCTs in health research, where selective outcome reporting ranged from 40-62% (Dwan et al., 2013; Weston et al., 2016) and in specific health areas, such as 15% in obesity clinical trials (Rankin et al., 2017). This finding is comparable to findings from a review of outcome reporting in behavioural health trials that were specifically published in the journal BMC Public Health (Taylor & Gorman, 2022). Outcome reporting discrepancies identified in our review included omission of pre-specified outcomes, introduction of new outcomes, and/or changes to prespecified levels of outcome importance in final published results. Identified discrepancies may be due to ‘spin’ or bias to present significant and/or novel/’exciting’ findings, poor reporting behaviours, and/or peer-review and editorial practices that fall short of ensuring transparency and confidence in research results.

Our findings have important and worrying implications for health psychology and behavioural medicine because selective outcome reporting compromises transparency (Hagger, 2019; Ioannidis et al., 2017), contributes to research waste (Thomas & Heneghan, 2022; Yordanov et al., 2018), and impacts evidence syntheses (Ioannidis et al., 2017). Further, selective outcome reporting distorts pooled treatment effects (Ioannidis, 2008; Macura et al., 2010; Shah et al., 2020; Thomas & Heneghan, 2022), which can lead to mis-informed decision making, with important implications for healthcare professionals and for patient, community, and population health and wellbeing (Segerstrom et al., 2023).

Making pre-specified research plans, such as protocols and trial registrations, publicly available enhances transparency and enables evaluations of outcome reporting (Calméjane et al., 2018; Chen et al., 2019; Thomas & Heneghan, 2022; Weston et al., 2016). That over half of the reviewed trials had a corresponding published protocol and/or trial registration is encouraging, and is comparable to findings in specific health areas, such as physical activity behaviour change interventions (Norris et al., 2022). However, that journals had published RCTs without corresponding trial registrations and/or protocols is surprising and worrying. It is particularly worrying given availability of guidance such as CONSORT-SPI 2018 (Grant et al., 2018), and that most of the journals searched in this review either require or encourage pre-registration, with some journals recommending specific registries, such as clinicatrials.gov.

Our findings indicated that more trials omitted than introduced outcomes. However it was not possible to examine whether this was due to knowledge of results, which is a common reason for such omissions (Smyth et al., 2011), because we did not have data on omitted outcomes. There are other potential reasons why trial authors may change outcomes; for instance, as noted in our findings, in one instance outcome discrepancies were due to steps taken to modify data collection during COVID-19. Regardless of the reasons for outcome omission, if outcomes were measured and subsequently not reported, this amounts to a waste of research time and resources. For instance, evidence from a recent review of 329 therapeutic, public health, and rheumatology-specific interventions found that the median time spent to collect data was 56.1 hours for primary outcomes and 190.2 hours for secondary outcomes (Gardner et al., 2022). That review also found a considerable range in costs for data collection, from approximately £53 in a small drug trial to almost £31,900 in a large phase III prostate cancer screening trial; the median overall data cost was found to be £8016 (Gardner et al., 2022). Given that our review identified a large number of omitted outcomes, this implies a large amount of resource waste in these trials.

Introduction of new outcomes, and changing levels of importance assigned to prespecified outcomes, as were observed in this review, are also problematic and reduce transparency, particularly where discrepancies are not explained. Trial samples sizes are derived based on the effect sizes researchers expect to observe in their primary outcome if the intervention is effective (Schulz & Grimes, 2005). Thus, changing a secondary outcome to a primary outcome impacts whether the trial is sufficiently powered to detect an effect in that outcome. Further, changing levels of importance is often tied to knowledge of outcome results and so introduces ORB (Hutton & Williamson, 2000). While we did not identify evidence of ORB for newly introduced outcomes in this review, in the majority of trials in which changes to the level of outcome importance was identified, the change was found to favour a statistically significant result, indicating ORB.

While discrepancies between methods and results sections were less frequent than discrepancies between protocols and the final report in the reviewed studies, identification of such discrepancies indicates issues with reporting behaviour and peer-review/editorial practices. Previous research has found that peer-reviewers often fail to identify discrepancies and deficiencies in reported methods and results (Chauvin et al., 2019; Hopewell et al., 2014) and there have been calls for greater robustness in peer-review and editorial processes (Ioannidis et al., 2017; Weston et al., 2016). Health psychology and behavioural medicine journals can help to minimise selective outcome reporting at the author, reviewer, and editorial levels. For instance, by requiring authors to account for any discrepancies in outcome reporting as a submission requirement, in addition to requiring pre-registration. While all reviewed journals instruct authors to follow relevant reporting guidelines, with most specifying CONSORT (Moher et al., 2010), only one journal (International Journal of Behavioral Medicine) provides explicit instruction on reporting and explaining differences between prespecified and reported outcomes in the final manuscript. Such explicit instruction is needed in all journals whereby authors must declare any outcome discrepancies and, where discrepancies occur, to confirm that they are explained in the text and/or in a supplementary file. Authors should be made aware that disclosing changes from protocols does not represent misconduct but instead better aligns with accurate, transparent, and complete reporting (Ioannidis et al., 2017).

It is the responsibility of peer-reviewers and editors to ensure that such transparent outcome reporting behaviour occurs, and that reporting guidance is followed. Recommendations for peer-reviewers and editors to more actively check trial protocols and registrations against trial publications to ensure transparent reporting, should consider burden-related challenges faced by peer-reviewers and editors, however. This is because, for the most part, peer-reviewers and editors act in a voluntary capacity (Severin & Chataway, 2021). As such, more pragmatic approaches are also needed, such as health psychology and behavioural medicine journals explicitly requiring (not just encouraging) pre-registration, structured reporting, and inclusion of clear statements and/or supplementary files explaining changes from protocols/registrations or, in the absence of changes, statements that changes have not occurred. For instance, that only 10% of trials with discrepancies reported reasons for discrepancies in this review, highlights a clear gap in reporting and peer-review/editorial checking. Requiring trialists to include such information provides an ‘easy win’ for journals and editors to maximise transparency moving forward. Future research into development and implementation of improved peer-review and editorial processes is needed to support improvements in transparent outcome reporting at this level.

Health psychology and behavioural medicine journals are making efforts to enhance transparency and openness. This includes the simultaneous publication of a statement from the Behavioral Medicine Research Council entitled ‘Open Science in Health Psychology and Behavioral Medicine’ in three leading journals in the field (i.e., Health Psychology, Psychosomatic Medicine & Annals of Behavioral Medicine, see Segerstrom et al., 2023) encouraging the increased use of open research practices and providing resources for researchers. The introduction of submissions of ‘registered reports’ (where peer review happens before data collection commences) and calls for pre-registration for all types of behaviour change interventions and research studies generally will help to improve transparency in outcome reporting and minimise selective outcome reporting (O’Connor, 2021; Thomas & Heneghan, 2022). Such efforts are applicable across study designs, which is important given that selective outcome reporting can also be problematic in non-randomised design studies and systematic reviews. For instance, 43% of Cochrane systematic reviews included in a recent examination demonstrated discrepancies from the review protocols (Shah et al., 2020). The role of selective outcome reporting in non-RCT studies in health psychology and behavioural medicine warrants further examination to identify and minimise selective outcome reporting across research conducted in this area.

Further efforts to improve outcome reporting in behavioural health trials in health psychology and behavioural medicine should also focus on trialist motivations and behaviours in outcome choice, measurement, and reporting. This includes the need for research to better understand the causes of selective outcome reporting, and barriers and facilitators to full and transparent outcome reporting, as has been done in other outcome methods areas, such as core outcome set use (Bellucci et al., 2021; Hughes et al., 2022; Matvienko-Sikar et al., 2022). Behavioural approaches to maximise open and transparent research practices (Norris & O’Connor, 2019) can also potentially help to improve outcome reporting. For instance, two recent studies have utilised the behaviour change wheel framework (Matvienko-Sikar et al., 2022; Osborne & Norris, 2022) to better understand barriers and facilitators and develop behaviour change theory-based approaches to increase uptake of core outcome sets (Matvienko-Sikar et al., 2022) and pre-registration (Osborne & Norris, 2022).

This study has some limitations. First, including RCTs published in nine health psychology and behavioural medicine journals limits generalisability of the findings beyond these journals. However, the chosen journals are recognised as publishing high quality research and cover a large proportion of published health psychology and behavioural medicine research. By limiting the review to papers published in these journals, we also limited the review to studies published in the English language, further limiting the generalisability. Second, inclusion of papers published from 2019, while intended to capture best practice in outcome reporting due to publication of the CONSORT-SPI-2018 guidance (Montgomery et al., 2018), limited ability to examine potential changes over time; future research could examine this, particularly in relation to specific health outcomes and/or interventions. Third, the majority of trials commenced before publication of the CONSORT-SPI-2018 checklist (Grant et al., 2018). However, the CONSORT statement (Moher et al., 2010) was available to all trialists during trial conduct, and all trials were published after the CONSORT-SPI-2018 checklist (Grant et al., 2018). As such, all trials should include reasons for changes as outlined as per CONSORT guidelines (Grant et al., 2018; Moher et al., 2010) and interestingly, only studies with protocols/registrations prior to CONSORT-SPI 2018 included explanations of discrepancies. Fourth, we did not contact the authors of the papers to identify reasons for discrepancies and so have limited knowledge of why discrepancies arose because only three papers provided reasons. Further research into this area would facilitate better understanding of why discrepancies occur to help minimise their occurrence. Finally, not all studies had corresponding protocols or trial registrations and so, in these instances, examinations were limited to discrepancies between methods and results sections only. However, we identified that discrepancies still emerged between these sections, highlighting an area for improvement in trial reporting. In addition, if no study protocol/registration was available we classified studies as having no discrepancies where changes were not evident from the methods section. It is plausible that studies without protocols/registrations are more likely to have selective outcome reporting than studies without protocols/registrations. Therefore, we are likely underestimating selective outcome reporting by only comparing methods and results sections for papers without protocols/registrations.

As has been noted by Moher (Moher, 2007), researchers have a moral obligation to transparently and completely report trial outcomes. However, our findings indicate high levels of selective outcome reporting and some evidence of ORB. Such findings clearly indicate an important area for improvement in health psychology and behavioural medicine. To reduce selective outcome reporting in this area, top-down and bottom-up efforts and approaches are needed. This includes improved peer-review and editorial processes, as well as improved trialist conduct and reporting behaviours. Such processes and behaviours should include, at a minimum, pre-registering trials completely and accurately, adhering to pre-registered plans, transparently explaining any changes from the protocol, and adhering to reporting guidelines; peer-review and editorial checks that these actions have occurred is also essential. Reducing selective outcome reporting in this area will improve the quality and robustness of reporting of behavioural health interventions, with important implications for patient and public health.

Authors contributions

KMS was involved in conceptualisation, methodology, investigation, data curation, analysis, supervision, writing the manuscript draft, and reviewing and editing it. JOS was involved in methodology, investigation, data curation, analysis, writing- original draft and review and editing. SK was involved in investigation, data curation, analysis, and review and editing of the manuscript draft. ST was involved in investigation, and review and editing of draft. KA, MB, SMcH, DOC, VS, ET, KD, IJS, and JK were involved in conceptualisation, methodology, and review and editing of the manuscript draft.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

KMS is supported by a Health Research Emerging Investigator Award (HRB-EIA-2022-005).