Network meta-analysis in health psychology and behavioural medicine: a primer

ABSTRACT

Progress in the science and practice of health psychology depends on the systematic synthesis of quantitative psychological evidence. Meta-analyses of experimental studies have led to important advances in understanding health-related behaviour change interventions. Fundamental questions regarding such interventions have been systematically investigated through synthesising relevant experimental evidence using standard pairwise meta-analytic procedures that provide reliable estimates of the magnitude, homogeneity and potential biases in effects observed. However, these syntheses only provide information about whether particular types of interventions work better than a control condition or specific alternative approaches. To increase the impact of health psychology on health-related policy-making, evidence regarding the comparative efficacy of all relevant intervention approaches – which may include biomedical approaches – is necessary. With the development of network meta-analysis (NMA), such evidence can be synthesised, even when direct head-to-head trials do not exist. However, care must be taken in its application to ensure reliable estimates of the effect sizes between interventions are revealed. This review paper describes the potential importance of NMA to health psychology, how the technique works and important considerations for its appropriate application within health psychology.

KEYWORDS:

• Evidence synthesis
• policy-making
• meta-analysis
• health behaviour change

Introduction

Progressing the science and practice of health psychology and the related field of behavioural medicine depends on the systematic synthesis of evidence from health behaviour change interventions. In particular, meta-analyses of randomised controlled trials (RCTs) have led to important advances in our understanding of the health impact of health behaviour change interventions. The vast majority of these meta-analyses have involved pairwise comparisons, i.e., the comparison of one intervention against another, or against a control condition. However, both national and global health policy organisations are increasingly relying on evidence synthesis involving the comparison of multiple interventions (Kanters et al., 2016).

Indirect comparisons can be made if interventions that have not been directly compared with each other have been compared to a common alternative intervention (Bucher, Guyatt, Griffith, & Walter, 1997). More generally, network meta-analysis (NMA) is a tool which enables the synthesis of evidence from both direct (i.e., within-trial comparisons of randomised groups) and indirect (i.e., between trial) comparisons of multiple interventions that may not have been compared within the same trial (Dias, Ades, Welton, Jansen, & Sutton, 2018; Higgins & Whitehead, 1996; Lu & Ades, 2004). All that is required is that all the trial evidence being quantitatively synthesised has at least one intervention in common with another, as this allows a network of trial comparisons to be constructed. This maximises the use of available evidence, allows comparisons between any pair of interventions in the evidence network and can increase the precision of the effect size for an intervention, compared with direct evidence alone (Caldwell, Ades, & Higgins, 2005; Ioannidis, 2006; Jansen et al., 2014). It is due to these advantages that NMA has become a key component of the development of clinical guidelines and reimbursement recommendations by national health technology assessment agencies and the World Health Organisation (Kanters et al., 2016). The utility of NMA in clinical medicine has resulted in some scholars, suggesting that it could constitute a higher level in the hierarchy of evidence than traditional systematic reviews and pairwise meta-analyses (Leucht et al., 2016; Roever & Biondi-Zoccai, 2016). However, while there has been a significant and rapid increase in the use of the method in health research more broadly over the last 10 years (Lee, 2014), uptake in the field of Health Psychology has been more limited. For example, a search of the present journal which is one of the internationally leading review journals in the discipline, as indicated by impact factor (7.24 in 2016), identified no instances use of NMA over the last 10 years. The application of NMA in health psychology has the potential to strengthen the link between evidence from behavioural trials in health and healthcare decision-making. This paper describes the potential importance of this method of evidence synthesis to health psychology, how the technique works and important considerations for its appropriate application within health psychology.

Why NMA is useful?

In health psychology, a considerable evidence base has been established on the effects of a wide variety of interventions for behaviour change on health. For a given patient population, there are typically several interventions available, and practitioners need to make evidence-based decisions between them. Ideally, this evidence would take the form of a well-powered RCT with as many intervention arms as there are decision options. However, it is clearly not feasible to conduct such a study, as the complexity of the study design and the resources required would be too great (Catalá-López, Tobías, Cameron, Moher, & Hutton, 2014). For example, whereas several types of behaviour change interventions are known to be effective in reducing blood pressure, including increased physical activity, smoking cessation and dietary modifications (Mancia et al., 2013), it would be impractical to attempt implementing even one multi-arm RCT that compared the effects of changes to one of these behaviours on blood pressure, let alone an RCT that compared the different techniques used to change each of these behaviours (Grant & Calderbank-Batista, 2013). Furthermore, even if such complex studies could be conducted, the pairwise evidence synthesis methods normally employed in health psychology could not coherently synthesise their results.

The current evidence base for the efficacy of behavioural interventions is mostly formed from studies comparing specific types of behavioural interventions with a control condition, such as waitlist or treatment-as-usual, and occasional examples of trials evaluating competing or alternative behavioural interventions, tested against each other (Michie, Abraham, Whittington, McAteer, & Gupta, 2009). There are no examples of trials comparing every possible type of behavioural intervention for a given population, illness and outcome being simultaneously evaluated against one another. Additionally, ‘treatment as usual’ can be very different across studies, as can the behavioural interventions themselves (Oberjé, Dima, Pijnappel, Prins, & de Bruin, 2015). If ignored, this intervention-level variation can lead to high levels of heterogeneity when pooled in a meta-analysis (de Bruin, Viechtbauer, Hospers, Schaalma, & Kok, 2009). The result of working with this kind of evidence base is a tendency to rely on expert opinion in deciding what interventions to implement (Kanters et al., 2016). NMA can treat each type of control condition as a distinct intervention, and similarly for behavioural interventions with different characteristics or components, hence minimising heterogeneity.

Additionally, many health outcomes targeted by health behaviour change interventions (e.g., blood pressure reduction) are often managed, first, through medical treatment (e.g., anti-hypertensive medication). Typically, behavioural interventions are not included as comparators in clinical trials of medical interventions, as regulatory bodies only require that they are compared with placebo conditions or treatment-as-usual/standard care (Falissard et al., 2009; Song, Altman, Glenny, & Deeks, 2003; Sutton & Higgins, 2008). For example, there is very limited evidence comparing physical activity interventions to drug interventions in those with illnesses related to cardiovascular disease, as this is often not required for licensing (Naci & Ioannidis, 2013). Thus, to make better-informed healthcare decisions, evidence regarding the comparative efficacy of all available interventions, whether behavioural or medical, is required. The absence of such comparison is critical. If behavioural interventions are as effective and cost-effective as medical treatments for a given illness or if they provide clinically important amplifications to medical treatment, then the likelihood for policy change that promotes the practice of health psychology and behavioural medicine will be enhanced (Jansen et al., 2011). This can highlight future directions for confirmatory research and provide greater scientific justification for the design and implementation of RCTs (Meulemeester et al., 2018).

In summary, current decision-making regarding interventions in health psychology is limited, because only evidence-based claims about what works can be made, rather than what works best (Salanti, 2012). The emergence of better comparative evidence on what interventions work best is critical for the further development of health psychology in healthcare. NMA provides a methodology to achieve this and therefore has the potential to elevate both the science and practice of health psychology and behavioural medicine from its current status as a relatively minor component in the delivery of healthcare globally (Cheung & Hong, 2017). Despite its potential to transform the field, NMA has yet to be fully embraced by health psychology and behavioural science more broadly. As a relatively new evidence synthesis method, NMA is rarely a standard part of postgraduate training in health psychology; therefore, the requisite knowledge and skills do not typically exist within this discipline.

Next, we provide a brief primer on the essential concepts which must be understood in order to conduct an NMA. See Table 1 for a description of some key terms related to NMA.

Table 1. Key terms related to NMA.

Term	Definition
Pairwise meta-analysis	A statistical analysis method for synthesising evidence from a set of individual trials which involved similar populations and which all compare the same (or very similar) two intervention conditions with a focus on the same (or a very similar) outcome
Network meta-analysis Also known as ‘mixed treatment comparison meta-analysis’ and ‘multiple treatment meta-analysis’	A statistical method for synthesising both direct and indirect evidence from a set of individual trials which involved similar populations, and which may include multiple different intervention conditions with a focus on the same (or a very similar) outcome
Indirect treatment comparison Also known as ‘adjusted indirect comparison’ and ‘simple indirect comparison’	A statistical analysis method for synthesising evidence from individual trials of two interventions which have not been directly compared in head-to-head trials, but which have been compared to a common intervention in head-to-head trials
Evidence network	A body of evidence from trials which compared multiple interventions in a homogenous population with a focus on the same (or a very similar) outcome
Network diagram	A graphical representation of an evidence network which usually uses nodes, the size of which represent the number of participants which took part in a specific intervention across multiple trials, and edges – lines connecting the nodes which indicate what interventions have been compared. The thickness of the edges represents the number of trials which have compared the two interventions represented by the connected nodes.
Closed loop	A closed loop can be seen in a network diagram whenever there are both direct and indirect evidence connecting a set of three or more interventions
Disconnected network	Disconnection occurs when there is neither direct nor indirect comparisons between certain interventions in the network
Effect modifiers	Clinical or methodological characteristics of studies which affect the relative effect between interventions
Transitivity	Transitivity implies that interventions, methods and populations in an evidence network are comparable in terms of the distribution of effect modifiers
Consistency	Consistency is the statistical demonstration of agreement between the direct evidence and the indirect evidence for all pairwise comparisons in a network for which both direct and indirect evidence are present

Please see Dias et al. (2018) for further information and a comprehensive definitive resource on NMA.

How NMA works?

The simplest application of NMA is the comparison of two interventions which are both viable intervention options for a given population, illness and outcome and which have been compared to similar alternative interventions (e.g., treatment-as-usual); but which have not been directly compared. Returning to the example of blood pressure reduction for people with hypertension, consider two broad types of behaviour change interventions which have been found to be effective but which, to our knowledge, have not been compared: increasing physical activity and salt-intake reduction. Interventions within these two categories are typically compared to treatment-as-usual control groups. An indirect comparison between physical activity interventions and salt reduction interventions (see Figure 1 for a network diagram) can then be made using the following formula (Bucher et al., 1997):$\begin{array}{rl}& {\text{Indirect Comparison}}_{\mathrm{Physical}\mathrm{Activity}\mathrm{vs}.\mathrm{Salt}\mathrm{Reduction}}={\text{Direct Comparison}}_{\mathrm{Physical}\mathrm{Activity}\mathrm{vs}.\mathrm{Control}\mathrm{Group}}\\ & -{\text{Direct Comparison}}_{\mathrm{Salt}\mathrm{Reduction}\mathrm{vs}.\mathrm{Control}\mathrm{Group}}.\end{array}$

Figure 1. An example of a network diagram.

Note that this assumes that the control group is similar in the Physical Activity studies to the control group employed in the Salt Reduction studies.

More generally for interventions A, B, and C, the indirect comparison can be presented as:${\hat{\mu }}_{AB}^{Ind}={\hat{\mu }}_{AC}^{Dir}-{\hat{\mu }}_{BC}^{Dir},$where ${\hat{\mu }}_{AB}^{Ind}$ is the indirect estimate of B vs. A, ${\hat{\mu }}_{AC}^{Dir}$ is the direct estimate of C vs. A and ${\hat{\mu }}_{BC}^{Dir}$is the direct estimate of C vs. B.

The variance of this estimate is equal to the sum of the variances of each of the direct estimates, meaning the indirect comparison alone is less precise than either of the direct estimates.

The network represented in Figure 1 is usually referred to as a simple indirect comparison. A simple indirect comparison can be extended to include any number of interventions which have been previously tested against a single common comparator. Panel B of Figure 2 provides an example of a network with four competing interventions, each of which has been compared to the common comparator intervention ‘A’. This ‘star’ network of evidence is likely to be common in health psychology, where behavioural interventions are most often compared to treatment-as-usual (de Bruin et al., 2009; Mohr et al., 2009). Of course, care should be taken to ensure that each treatment-as-usual intervention is similar enough across the studies to be combined into a single comparator ‘node’.

Figure 2. Some possible configurations of networks of evidence.

A ‘star’ network can readily be extended to include further comparisons. These can be interventions which have been compared to specific interventions present in the network, i.e., they do not need to be connected via a single common comparator. There will many such situations in health psychology where more than one common comparator exists; for example, whereas many studies employ a waitlist control, some studies employ an active control group. The hypothetical evidence network depicted in panel C of Figure 2 represents this situation, where A could be a waitlist control group, B to E could be competing interventions and F could be an active control group which has been included in trials of B and E. This network also demonstrates a closed loop, where there are both direct and indirect evidence available to inform the comparison conditions A and B and conditions A and E.

Panel D of Figure 2 depicts another hypothetical evidence network that may arise in health psychology, where both behavioural and medical interventions are compared. How these two sources of evidence are connected will depend on the population, illness and outcome that are being investigated. In this example, we imagine a treatment-as-usual comparator, common to both behavioural and medical intervention studies, as represented by condition A. Again, the behavioural interventions are represented by conditions B to E, with condition F representing an active behavioural control group. In this example, conditions G to I represent medical interventions that have been compared to both treatment-as-usual (A) and a placebo condition (J). Still, the evidence networks which are most likely to be well connected are those where several behavioural interventions which target the same outcome are being compared. A hypothetical example can be seen in panel E of Figure 2.

It is also possible that there might be no single common comparator connecting all available interventions, for a given health outcome (Goring et al., 2016). For example, behavioural interventions can be compared to waitlist control groups, behavioural active control groups or treatment-as-usual, while medical interventions might only be compared to placebo control groups. If there is direct evidence comparing behavioural interventions directly with medical interventions, then the network ‘connects’ and NMA can be performed. If not, then the network is disconnected (Goring et al., 2016). Standard NMA techniques cannot be applied to disconnected networks unless the different types of control can be considered similar enough to ‘lump’ together and connect the network. A recent example of this is a health technology assessment of smoking cessation interventions (Health Information and Quality Authority, 2017). Behavioural interventions and pharmacological interventions were analysed separately because there were systematic differences in the nature and effects of the control groups used in trials of these two types of intervention. Some extensions of NMA have been proposed which can analyse disconnected networks but these rely on extra assumptions (Goring et al., 2016).

To estimate the indirect comparisons in the more complex networks that may emerge in the synthesis of evidence from behavioural interventions that are typically studied in the health psychology literature, additional modern statistical models such as NMA are required (Dias et al., 2018). Such methods produce more precise effect sizes than using direct evidence alone (Caldwell et al., 2005; Ioannidis, 2006; Jansen et al., 2014). However, for all NMA models, there are some key assumptions that must be met to ensure the resulting effect size estimates are meaningful.

Assumptions of NMA

In NMA, as in pairwise meta-analysis, care must be taken to estimate and account for heterogeneity. Heterogeneity across a set of studies implies the presence of effect modifiers, examples of which may include: participant characteristics at baseline; intervention dosages; intervention setting; type and timing of measurements, among others. However, these effect modifiers may or may not be measured or even measurable. If measurable and measured, a trial-level variable is shown to be an effect modifier when it interacts significantly with the intervention effect (Dias, Welton, Sutton, & Ades, 2013). Critically, estimates of the effect sizes from NMA can be confounded by the uneven distribution of effect modifiers across the network of evidence (Kovic et al., 2017). This is an example of the violation of the key assumption underpinning NMA, which can be considered in two parts (i) transitivity and (ii) consistency.

According to Salanti (2012, p. 83), transitivity refers to the assumption that the ‘indirect comparison validly estimates the unobserved head-to-head comparison’. It should be possible, in principle, that participants could be randomised to any of the interventions included in the evidence network in a hypothetical RCT (Salanti, 2012). For example, receiving one kind of intervention technique should not mean that another one is contraindicated. Consistency is the term used for the statistical manifestation of transitivity, and can only be assessed when both direct and indirect evidence are available. Estimates in an NMA are said to be consistent when the indirect evidence and the direct evidence agree. Checking that the conditions for both transitivity and statistical consistency are met is an essential step in running an NMA, where evidence is available from both direct and indirect sources (see the following for a detailed description of strategies for checking consistency; Dias et al., 2013; Higgins et al., 2012; White, Barrett, Jackson, & Higgins, 2012). However, when direct evidence is absent, and a statistical check of consistency is therefore not possible, transitivity must still be assessed. It is always possible to check for transitivity, regardless of whether direct evidence is available or not. This can be achieved by qualitatively examining relevant clinical and methodological aspects of the relevant intervention comparators to ascertain whether there is an even distribution of clinical and methodological effect modifiers across the intervention comparators (Dias et al., 2018).

The assumption of transitivity is crucial to the validity of the results of any NMA as the violation of this assumption leads to biased indirect comparison estimates, which leads to biased NMA estimates (i.e., the estimates which integrate both direct and indirect evidence; Jansen & Naci, 2013). The next section discusses specific challenges which may arise in applying NMA in health psychology. These challenges may affect the validity with which health behaviour change intervention studies can be synthesised by NMA.

Challenges in applying NMA in health psychology

Although there are many potential benefits of using NMA in health psychology, particular care must be taken in comparing multiple behavioural interventions, as there may be important differences in the reasons why a particular behaviour is being targeted or why a particular set of behaviour change techniques (BCTs) is being used, or additionally why a specific comparator is chosen.

Choosing to change a specific health behaviour and applying specific BCTs to achieve this involves careful development work that considers patient characteristics, available resources and contextual factors (Bartholomew, Parcel, & Kok, 1998; Michie, van Stralen, & West, 2011). Each decision in the intervention development process has the potential to modify the intervention effect. Therefore, in applying NMA in health psychology, researchers must examine how each intervention in the evidence network was developed, in order to ensure transitivity and consistency. Combining behavioural interventions that apply multiple interacting BCTs in different ways, across different settings, and with different patient groups, has the potential to violate transitivity if there is an uneven distribution of clinical and methodological characteristics across the set of interventions being analysed. Therefore, we strongly recommend that this methodology is only used when there is appropriate statistical and clinical expertise within the review team, as this is essential to apply this method appropriately.

Researchers should also consider the type of control groups used in testing different interventions, which may include the application of some BCTs, and which may be unevenly distributed across control conditions (de Bruin et al., 2009). This is a considerable threat to the assumption of transitivity and one that is difficult to identify due to the poor reporting of the contents of control conditions (Oberjé et al., 2015). However, if the contents of control conditions are coded carefully rather than lumped together, NMA can be usefully applied to identify how intervention effects differed according to the type of control group employed. Notably, the use of NMA identified different intervention effects for cognitive–behavioural therapy in depression depending on the nature of the control group employed and revealed a possible nocebo effect attributable to waiting-list control groups (Furukawa et al., 2014). Note, however, that by creating distinct control group effects, the precision in the summary intervention effect estimates will be reduced.

It is likely that NMAs in health psychology will rely on indirect evidence. This is due to the common practice of comparing interventions to treatment-as-usual rather than suitable alternative, competing interventions (Ayling et al., 2015; Bruin & Viechtbauer, 2014; Freedland, Mohr, Davidson, & Schwartz, 2011; Oberjé et al., 2015). As discussed above, this precludes statistical assessment of consistency. Care must be taken in the design of any NMA in health psychology as a clear definition of the population, interventions, comparators and outcomes (PICO) will enhance the validity of the analysis.

Another characteristic of NMA that may limit its usefulness in health psychology, as in other areas of psychology, is the predominance of small studies (Crutzen & Peters, 2017). These may suffer from methodological limitations usually associated with small sample sizes which can lead to biased estimates (Roever & Biondi-Zoccai, 2016). This issue applies equally to pairwise meta-analysis, but bias can propagate through a network and affect different parts of the network in different ways (Li, Puhan, Vedula, Singh, & Dickersin, 2011). NMA would not be recommended in cases where evidence is only available from very small, underpowered trials.

Finally, the suitability of NMA for synthesising evidence in health psychology is expected to improve as existing calls for increased rigour and reproducibility are heeded. Health psychologists should continue to respond to calls for: better measurement (Beauchamp & McEwan, 2017); increased use of standard outcome sets (Williamson et al., 2012); more transparent reporting of intervention methodology and results (Boutron, Moher, Altman, Schulz, & Ravaud, 2008; Hoffmann et al., 2014); and the compulsory sharing of individual-level data (Peters, Abraham, & Crutzen, 2015).

Opportunities for NMA in health psychology

There are many opportunities to apply NMA and synthesise evidence regarding behavioural intervention for some of the most pressing health problems. Foremost among these include the main behavioural contributors to mortality such as smoking, sedentary behaviour, dietary behaviour, sleep and alcohol consumption. Indeed, there are several recent and ongoing NMAs that aim to elucidate the comparative efficacy of behavioural and medical interventions for addressing health outcomes and related behaviours (Cheng et al., 2017; Iftikhar et al., 2017; Schwingshackl, Chaimani, Hoffmann, Schwedhelm, & Boeing, 2017; Suissa et al., 2017). The increased application of NMA in addressing health-relevant behaviours, in recent times, demonstrates that researchers, in a variety of fields, have identified NMA as a potential means of providing both richer syntheses of existing evidence and new insights into whether and which behavioural interventions should be prioritised in healthcare.

Another important area for future development involves linking NMA to other recent developments in meta-analysis, such as spatiotemporal, multivariate and automated meta-analyses (Card, 2017). The integration of these methods would increase the amount of valuable information contributing to decision-making regarding the comparative effectiveness of health interventions. Specifically, spatiotemporal meta-analysis is a technique designed to account for heterogeneity in research findings due to variability in study environments (Johnson, Cromley, & Marrouch, 2017). This approach expands the traditional process of conducting meta-analysis to include methods for the coding and modelling of geographical and temporal information. Factors related to the timing and location of interventions can be significant effect modifiers. Integrating the spatiotemporal meta-analysis and NMA will therefore allow for more accurate and systematic examination of the assumption of transitivity.

Multivariate meta-analysis is an extension of meta-analysis which allows for the examination of intervention effects for multiple outcomes (Jackson, Riley, & White, 2011). In addition to the primary outcome, studies in health research usually involve several secondary outcomes, which are correlated to some extent, e.g., healthy eating and participation in the regular physical activity. Like multivariate meta-analysis, methods have been developed for including multiple outcomes in NMA (Jackson, Bujkiewicz, Law, Riley, & White, 2017). For example, Taieb, Belhadi, Gauthier, and Pacou (2015) analysed the effects of two classes of anti-diabetic drugs (i.e., dipeptidyl peptidase-4 inhibitors and sulphonylureas) and placebo pills on three outcomes related to glycaemic control in Type-2 diabetes patients, including change in HbA1c from baseline, the change in fasting plasma glucose from baseline and the proportion of patients reaching HbA1c <7%. The advantage of multivariate NMA is that it allows for the estimation of intervention effects across all comparators for all outcomes of interest – even those for which there is currently no direct evidence available. In this case, no evidence was available regarding the proportion of patients reaching HbA1c <7% for the comparison of sulfonylureas and placebo pills. Multivariate NMA not only revealed that these drugs had a significant benefit, but also produced more precise estimates of the intervention effects of the other drugs included in the analysis (Taieb et al., 2015). Examining multiple outcomes is vital to ensuring that all relevant outcomes, including benefits and harms, contribute to the estimation of the intervention effect and also avoids problems related to overestimation of the variance of effects sizes, biased effect sizes and type-2 error due to multiple comparisons (Mavridis & Salanti, 2013).

With respect to automated meta-analyses, one particularly ambitious project focuses on developing advanced techniques for synthesising health research is the Human Behaviour Change Project (Michie et al., 2017). This project aims to identify the extent to which health behaviour change interventions work and the contribution of effect modifiers, such as participant characteristics, setting and target behaviour. This project will apply artificial intelligence and machine learning technology to code studies based on an ontology of behaviour change and then extract data in order to perform automated meta-analyses (Larsen et al., 2016). While the prospect of evidence synthesis being facilitated in this way is exciting, the decision-making value of the outputs of this project will be limited if a purely pairwise approach to meta-analysis is taken. For the Human Behaviour Change Project to fulfil its aims, it must integrate network and multivariate analytic approaches into its design. Such an approach, known as live cumulative NMA, has already been developed in clinical medicine, though further development of the methodology and of the reporting of systematic reviews in health research is needed before it is commonly applied (Créquit, Trinquart, & Ravaud, 2016; Vandvik, Brignardello-Petersen, & Guyatt, 2016).

Not only are there interesting opportunities for application of NMA in health psychology, there are also exciting opportunities for health psychology to contribute to the development of NMA, particularly in the area of evidence synthesis for complex interventions. It has been proposed that NMA would provide a useful framework for analysing the contribution of specific components (i.e., elements of an intervention which actively influence the intervention effect; Kühne, Ehmcke, Härter, & Kriston, 2015) within complex interventions (Caldwell & Welton, 2016; Madan et al., 2014; Welton, Caldwell, Adamopoulos, & Vedhara, 2009). A high degree of heterogeneity is introduced by attempting to synthesise evidence from complex interventions in pairwise meta-analysis (Kühne et al., 2015). This is because complex interventions, by definition, involve multiple components which may interact and these components can vary between studies (Craig et al., 2008). Applying NMA allows for components, e.g., which are common across interventions in an evidence network to be represented as nodes in the network (Caldwell & Welton, 2016). Welton and colleagues (2009) have demonstrated three analytic models which make different assumptions regarding the relationships between intervention components. The additive main effects model assumes that the effects of each intervention component sum together. In this model, the components are assumed not to interact or cancel each other out in any way. The two-way interaction model allows pairs of components to have a larger or smaller effect when found together in an intervention than that would be expected of an intervention involving one of those components alone. The full-interaction model treats each specific combination of intervention components as a unique intervention with an associated intervention effect (Caldwell & Welton, 2016).

However, there is debate regarding the best way to identify and model the components within complex interventions. Many methods of coding intervention components can be employed. These have been described as falling into two categories: clinically meaningful unit methods and component dismantling methods. Focusing on the clinically meaningful unit means addressing which broad approach to intervention is most effective. Dismantling methods involve the examination of how specific components (or their combinations) affect intervention efficacy (Melendez-Torres, Bonell, & Thomas, 2015). This debate represents an opportunity for health psychology to contribute a considerable amount of accumulated knowledge regarding the coding of intervention components in terms of modes of delivery, settings, BCTs, theoretical constructs and mechanisms of action (Kok et al., 2016; Michie et al., 2013; van Genugten, Dusseldorp, Webb, & van Empelen, 2016).

Conducting an NMA

Once the assumptions of NMA are met, there are models available for conducting an NMA on many different types of effect size estimates including those most commonly used in health psychology, mean differences and odds ratios. NMA can be carried out within a frequentist or Bayesian framework. Comparisons of the two approaches appear to show similar outcomes (Hong et al., 2013). However, Bayesian methods for conducting NMA are more flexible, as they can make use of prior information regarding model estimates; account for uncertainty and inconsistency; and yield easily interpretable results (Hong et al., 2013; Neupane et al., 2014).

Bayesian NMA is most commonly conducted using Bayesian inference Using Gibbs Sampling (BUGS) software, including WinBUGS and OpenBUGS (Lunn, Thomas, Best, & Spiegelhalter, 2000). These programmes were developed to allow for the use of Markov Chain Monte Carlo methods for analysing Bayesian statistical models. Dias, Welton, Sutton, and Ades (2011) provide WINBUGS/OpenBUGS code for a wide range of commonly encountered evidence/outcome types. Similar programmes include JAGS and Stan (Stephenson, Fleetwood, & Yellowlees, 2015). While the BUGS environment may be difficult to adapt to, Brown et al. (2014) have developed an accessible tool called NetMetaXL, which runs within Microsoft Excel and interfaces with WinBUGS to better facilitate Bayesian NMA. The gemtc (van Valkenhoef & Kuiper, 2016), LaplacesDemon (Hall et al., 2016) and pcnetmeta (Lin, Zhang, & Chu, 2017) packages for the R environment can also be used for the same purpose. There are packages available in Stata for conducting NMA within the frequentist framework, including mvmeta, network (White, 2009) and network graphs (Chaimani et al., 2013). The ‘netmeta’ package for the R environment is also based in a frequentist framework (Rucker, SchwarzerKrahn, & Konig, 2017). Most of these software packages are available free and many come with accessible guides on how to use them. See Table 1 for a comparison of some of the most popular packages available. Next, we present a step-by-step example of the application of NMA to a set of trials of behavioural interventions (Table 2).

Table 2. Comparison of a sample of popular software packages capable of NMA.

Statistical Framework	Features	gemtc - R	pcnetmeta - R	netmeta - R	laplacesDemon - R	WinBUGS/ OpenBUGS/ JAGS/ Stan/ NetMetaXL	mvmeta/ network graphs - Stata
Bayesian
Frequentist
Tasks
Forms of Input Data	Arm-level data
	Contrast-level data
	Accepts multi-arm trials
Types of Outcome Data that Can be Analysed	Binary
	Count
	Continuous
	Survival
Extracts descriptive measures	Total number of studies
	Total number of multi-arm studies
	Total number of participants
	Total number of treatments
Network plot and options	Network plot
	Add node labels
	Node size reflects network characteristics
	Edge thickness reflects network characteristic

Note: Adapted from Neupane et al. (2014). White blocks indicates the presence of the feature; black blocks indicates that the feature is not present in the software package.

A step-by-step example of the development and conduct of an NMA

Background: Kanters and colleagues (2017) provide a useful illustration of how NMA has been applied in synthesising the evidence on these behaviour change interventions which are not often compared directly to each other. The main steps involved in conducting this NMA are described below.

Step 1: The research question for this study was generated in the context of a need to update the WHO global consolidated guidelines on HIV. This required the examination of the comparative effectiveness of medication adherence interventions on adherence to ART and HIV viral load.

Step 2: A detailed protocol was developed using the PRISMA extension to NMA (Hutton et al., 2015) to guide the study design, analyses and reporting. This set out a clear focus on the population (people living with HIV), interventions (those targeting enhanced adherence to ART), comparators (standard care) and outcomes (treatment adherence and viral suppression; PICO) and described the key search terms.

Step 3: The database search was conducted and supplemented by additional standardised strategies to identify grey literature.

Step 4: Two investigators independently reviewed any identified abstracts and subsequently relevant full-text articles to identify the relevant RCTs. The quality of the included studies was assessed using the Cochrane tool for assessing the risk of bias (Higgins et al., 2016) and the GRADE criteria for assessing the strength of evidence in NMAs (Caldwell et al., 2016).

Step 5: Two investigators independently extracted the pre-specified data.

Step 6: They categorised intervention and control arms in the identified RCTs using the following categories: standard of care, enhanced standard of care, telephone, SMS, behavioural skills training or medication adherence training, multimedia, cognitive behavioural therapy, supporter, incentives and device reminder interventions. Due to the considerable heterogeneity across the term standard of care, they defined the enhanced standard of care as interventions that provided more support than the usual standard of care. Standard of care was defined as instructions by the healthcare provider at treatment initiation regarding how to take ART medication and the importance of adhering to it. Included studies were also classified according to whether they were based in high-income and low-income and middle-income (LMIC) settings.

Step 7: NMAs were conducted to compare the effect of intervention categories on adherence and viral suppression for all study settings (i.e., the global network) and for studies in the LMIC network only. These NMAs were conducted using logistic regression models which included dichotomised variables indicating medication adherence success and viral load suppression as outcome variables. Both fixed-effects and random-effects models were considered – the model with the lowest deviance information criterion was selected. Potential effect modifiers were identified (e.g., sample characteristics and time of measurement), and meta-regression was used to evaluate their influence. Sensitivity analyses were conducted to assess the influence of different follow-up periods and the use of either the intention-to-treat or per-protocol results. All analyses were carried out with R (version 3.1.2) and OpenBugs (version 3.23). The authors do not report any analysis of the consistency between direct evidence and indirect evidence for the comparisons in the evidence network. Ideally, models for checking for the presence of consistency are applied – for example, the design-by-treatment interaction model (Higgins et al., 2012). This is an informative approach as it provides information on the appropriateness of the categorisation of the nodes and the reliability of the effect size estimates. Tabular ranking strategies and visual depictions of intervention rank are also sometimes employed to identify the best intervention approaches (Salanti, Ades, & Ioannidis, 2011). In Kanters et al. (2017), forest plots were employed to compare effect sizes for intervention approaches on ART adherence and HIV viral load.

Step 8: The results of these NMAs demonstrated, using the direct and indirect evidence available, an estimate for the effect size between each pair of interventions for both ART adherence and viral suppression. These are presented as a table of odds ratios with each effect size representing the comparisons between the interventions.

Considering these estimates, the authors concluded that supportive strategies and behavioural strategies are more effective than standard adherence support. Medication adherence interventions which involved both in-person and telephone support were more effective than most other interventions.

For a summary of the steps usually taken in conducting a study involving NMA, see Table 3.

Table 3. Generic steps in the planning and execution of an NMA.

Step	Aim	Considerations
1	Generate Research Question	The research question should be constructed with consideration of both the clinical and methodological characteristics of the studies of interest
2	Plan Systematic Review	This should be guided by PRISMA extension for NMA A clear definition of the PICO must be presented and the associated inclusion and exclusion criteria should allow the inclusion of as many relevant interventions and comparators as possible Potential effect modifiers should be identified The plan for the systematic review should be registered in PROSPERO and detailed in a study protocol
3	Conduct Search	In situations where a large body of literature exists and high-quality systematic reviews have been carried out, the search may focus on identifying these, as identifying individual studies through a primary search may not be feasible.
4	Select Studies	Studies should be selected according to the inclusion and exclusion criteria ideally by two independent reviewers Studies which involve interventions which are not central to the research question may be included if they are compared to interventions which are central to the research question and this provides more useful evidence to the network
5	Extract Data	This stage will generally focus on extracting the relevant data regarding outcomes and potential effect modifiers Risk of bias and evidence quality should be assessed using the tools provided by Cochrane and GRADE as these characteristics also affect transitivity
6	Build Network	Decisions regarding splitting and lumping are made at this stage and planned approaches may have to be modified according to the nature of the collected data (e.g., if there is a lack of data, some lumping may have to be done) A network diagram should be constructed and its geometry should be evaluated, e.g., Figure 2
7	Analyse Data	For all comparisons for which there are both direct and indirect evidence, consistency checks should be carried out to ensure that the direct and indirect evidence agree. Generally, pairwise analyses are conducted first and then NMA models are conducted The data should be analysed as set out in the study protocol
8	Interpret and Report Results	The PRISMA extension for NMA provides guidance on reporting the results in a clear and comprehensive manner. Data from individual studies should be summarised in tables The estimates of comparative effectiveness are usually presented in tables and sometimes in a rankogram

Exemplar applications of NMA of relevance to health psychology

NMAs that have a particular resonance for health psychology and behavioural medicine are increasingly being reported over the last 5 years. We briefly illustrate three such studies here. One such example examined the comparative efficacy of exercise and drug interventions on mortality outcomes (Naci & Ioannidis, 2013). This analysis incorporated data from 305 RCTs and found that exercise and many drug interventions are often similarly effective with respect to their impact on survival, in the context of secondary prevention of coronary heart disease, rehabilitation after stroke, treatment of heart failure and prevention of diabetes. The study also found that diuretics were more effective than exercise in reducing mortality in those with heart failure. The findings from this analysis highlighted the need to perform RCTs on the comparative effectiveness of exercise and drug interventions. These findings are important for health psychology as they demonstrate that behavioural intervention, in the form of physical activity promotion, may be as effective as medical intervention (i.e., secondary prevention medications) in some contexts.

Mayo-Wilson et al. (2014) examined the comparative efficacy of psychological and pharmacological interventions for social anxiety disorder in adults. They used a ‘class-effect’ model, where each type of intervention is considered to be distinct, but that effects are similar within classes. This provides a balance between avoiding heterogeneity due to ‘lumping’, and avoiding imprecision due to ‘splitting’. The analysis used data from 101 RCTs and found that the efficacy of some psychological interventions for social anxiety disorder (e.g., individual cognitive behavioural therapy) was comparable to some classes of pharmacological interventions (e.g., selective serotonin-reuptake inhibitors and serotonin–norepinephrine reuptake inhibitors). As cognitive behavioural therapy has been shown to have a lower risk of side effects than some pharmacological interventions, this review recommended that it should be regarded as the best intervention for the initial treatment of social anxiety disorder. Once again, such findings are important for health psychology, as they demonstrate that psychological intervention may be as effective as medical treatment, but with the added benefit of a reduced risk of adverse side effects. This evidence, an integration of direct and indirect comparisons derived through NMA, supports the prioritisation of psychological intervention for this significant health problem.

A final example of NMA that may potentially change health psychology intervention for Type-2 diabetes treatment was reported by Pillay et al. (2015). Pillay and colleagues’ review aimed to identify factors moderating the effectiveness of behavioural programmes for adults with Type-2 diabetes. This synthesis included 132 RCTs and found that several aspects of the content and delivery of these programmes were associated with outcomes. For example, self-management education, offering 10 or fewer hours of contact with delivery personnel, provided little benefit and that these programmes seem to benefit persons with suboptimal or poor glycaemic control more than those with good control.

These findings have resonance for health psychology as they provide indirect comparative effectiveness data that can be used to optimise the delivery of health psychology intervention in the context of a specific chronic illness. When considering these and any other applications of NMA, it is vital to scrutinise how the evidence network was determined, whether transitivity and consistency were established. Useful tools for evaluating the quality of studies which have applied NMA can be found in the work of Salanti et al. (2014), Chaimani, Salanti, Leucht, Geddes, and Cipriani (2017) and Jansen et al. (2014).

Conclusion

The primacy of direct evidence will, and should, continue to determine the most effective and cost-effective means of health psychology intervention to improve health outcomes. However, the appropriate and judicious use of indirect comparisons can provide insights that can shed light on the potential value of health psychology interventions that may influence the role of the discipline in the delivery of healthcare. NMA and its variants provide a useful evidence synthesis methodology that is currently underused in health psychology. This methodology is expected to make a significant contribution to the evolution of both the science and practice of health psychology in the years to come.

Acknowledgement

Thanks to Dr Chris Dwyer for proofing this manuscript and providing advice on style.

Disclosure statement

No potential conflict of interest was reported by the authors.

Additional information

Funding

This work was funded by the Irish Research Council under the New Horizons grant scheme [REPRO/2016/31].