Meta-analysis: A tool for clinical and experimental research in psychiatry

Abstract

The meta-analysis study is a type of systematic review with strong scientific rigor; it has a number of characteristics that makes it a very useful tool. However, performing and reading meta-analysis could be a challenge—the meta-analysis overcomes the limitation of small sample sizes or rare outcomes by pooling results from individual studies in order to generate a single and better estimate. It also increases statistical power and allows the evaluation of discrepancies among the results of different studies. In this paper, we will present examples to illustrate how psychiatrists can utilize a meta-analysis in clinical and experimental research.

• Meta-analysis
• Psychiatry
• Tools

Scientific publications have shown a remarkable growth in the last years. However, among these publications it is common to find studies with the same aims but with results that can be almost homogeneous or contradictory (1). These discrepancies may be due to various limitations in the studies such as the possibility of biases derived from original studies, the establishment of inclusion criteria, lack of quality of the studies or incorrect interpretation of the outcomes (2). As a consequence, it is necessary to perform analyses that allow the integration of evidences to clarify the results. Nowadays, meta-analysis has become a useful tool for medicine, epidemiology, social studies, ecology, medical engineering and other areas in which a synthesis of the scientific evidence is required (3).

Objectives

In the present study, we present a typical example of an association between genotype frequency and suicidal behavior to illustrate how psychiatrists may utilize a meta-analysis in clinical and experimental research. Second, we describe some generalities on the process to be followed in a meta-analysis and the main points to be considered when performing one.

Definition

The term meta-analysis was first introduced in 1976 and designated “any statistical analysis of a large collection of literature in order to integrate results” (1).

Meta-analysis is a tool that has as objective the synthesis of published scientific evidence to obtain a better integration of the results. This analysis has been very useful in clinical and medical areas, since it provides several benefits (4). First, meta-analysis allows a further generalization of the results than those obtained from individual studies; this more general validation is a consequence of the fact that the sample does not come from a single population. In addition, this methodology increases the statistical power, and in turn increases the ability to find significant differences, as well as allowing more precision in the effect estimation. The meta-analysis study enables the measurement of discrepancies among the results of different studies and also provides possible explanations for this heterogeneity (5).

The updated revisions of Cochrane Database of Systematic Reviews facilitate the performance of meta- analysis in experimental studies that combine results or randomized clinical traits to ensure medical treatments (6). There are other statements for conducting revisions of this kind of studies such as the Preferred Reporting Items for Systematic Reviews and Meta-analyses, better known as PRISMA. This statement was developed using a consensus process oriented and updated by evidence; PRISMA consists of a 27-item checklist and a four-phase flow diagram. Its original version was proposed by QUOROM (Quality Of Reporting Of Meta-analysis). PRISMA is an essential tool for summarizing accurate and reliable evidence (7).

Aims

The first objective of meta-analysis is to obtain clear and reliable results useful in the management of patients and possibly as a basis for clinical guidelines (1). Meta- analysis, when is used correctly, must comply with the following objectives: 1) to test the hypothesis related to the effect of the intervention under study; 2) to increase the accuracy of the estimators of the effect of the intervention under study; 3) to assess the consistency between clinical trials of similar interventions associated with the topic and generate a more efficient estimator of the effect; 4) to assess the consistency between trials of different interventions performed for the same purpose and generate an estimate of the effect of such care; 5) to identify with accuracy subgroups of patients who would most likely be affected by the intervention, either in a favorable or unfavorable manner, and 6) to calculate the requirements, in terms of sample size, of future clinical trials to be performed in the same field (8).

The meta-analysis study contributes not only its versatility but also its pertinence in several aspects of clinical research. This type of research study increases the statistical power of comparison and also improves the estimation effects (9). Several issues can be evaluated with a meta-analysis; it is particularly useful in studies with contrasting results when one wants to combine these results, or when it is necessary to analyze sub-groups of subjects selected from different studies, or in the searching for answers to new questions (5).

Design of a meta-analysis study

Stating the problem

The first phase in the meta-analysis is stating the problem. The researcher has to formulate the problem of interest, otherwise the lack of clear and precise answers and the possible factors can become a matter of confusion and lead to biased results (6). In addition, the extraction of specific data from the studies is a relevant process, as well as the selection of the most appropriate statistical techniques for the analysis. Also, if there are previous meta-analyses of the relationship studied, the researcher must clarify the possible differences between them. A correct definition and delimitation of these aspects facilitates the next stages of the process of meta-analysis (4).

Example 1. The following text comes from a previous meta-analysis where we evaluated the association of suicidal behavior and 5-HTR2A gene (8).

Suicidal behavior is a major health problem worldwide. Several recent studies have been carried out that support a possible relationship between genetic factors and suicidal behavior (10). Historically, evidence for the involvement of serotonin (5-HT) in suicide originated from findings of low 5-hydroxyindoleacetic acid concentration (5-HIIA) in cerebrospinal fluid (CSF) of depressed suicide attempters and in brain stems of completed suicides (11). These studies provided evidence for altered serotonergic neural transmission in the pathogenesis of suicidal behavior. In consequence, genes pertaining to the serotonergic system have been proposed as candidates to establish biological correlates between suicidal behavior and the serotonergic system. One candidate gene in the study of suicidal behavior is the gene encoding for the serotonin 2A receptor (12). In this example, we can appreciate the delimitation of a topic for study.

Effect size

The term effect size is an index used to quantify the relationship between two variables or a difference between two groups. This measure is based on means, binary data or correlations (5).

Effect size based on means

This type of index is employed when the studies report means and standard deviations. The preferred effect size is usually the raw mean difference, the standardized mean difference, or the response ratio (6). Also, Cohen's is an example of standardized mean difference in the effects sizes based on means (13).

Effect size based on binary data

This measure includes data from prospective studies, such as a randomized trial, which is originally reported as the number of events and non-events in two groups. Researchers typically compute a risk ratio, and/or risk difference (14, 15).

Effect size based on correlations

This index is used in studies that report a correlation between two continuous variables; the correlation coefficient itself could serve as the effect size index. The correlation is an intuitive measure that has been standardized to take into account different metrics in the original scales (5).

Example 2. The following example was extracted from a previous meta-analysis where we evaluated the association of suicidal behavior and the COMT gene (14).

Table 1 presents the results of a meta-analysis of case-control studies on the role of the COMT (catechol- O-methyltrasferase) val158/108Met polymorphism in suicidal behavior.

Table 1. Meta-analysis of case–control studies on the role of the COMT (catechol-O-methyltrasferase) val158/108Met polymorphism in suicidal behavior.

	Number of COMTval alleles		Number of COMTmet alleles
References	Cases	Controls	Cases	Controls	Odds ratio (95% CI)
Tovilla-Zárate (2011)	126	272	84	200	0.90 (0.65–1.26)
Lee (2011)	223	273	117	121	1.18 (0.86–1.61)
Perround (2010)	848	255	784	221	1.06 (0.86–1.30)
Zalsman (2008)	182	121	220	117	1.25 (0.90–1.72)
Rujescu (2003)	139	323	159	333	1.10 (0.84–1.95)
Liou (2001)	95	275	29	101	0.83 (0.51–1.33)
Russ (2000)	52	58	46	40	1.28 (0.72–2.25)
Ohara (1998)	11	175	13	95	2.17 (0.93–5.04)
Random effects	1676	1752	1452	1228	1.09 (0.97–1.23)

Searching of information

In general, the information of interest in the studies included in a meta-analysis must be extracted taking into consideration the following aspects (7, 16).

Characteristics of the study

These include type of design, description of the sample (age, gender, diagnosis, among others), type of intervention (dose, active ingredient, just to mention a few), follow-up time of an evaluation and other features that may help in assessing the homogeneity or heterogeneity in a pool of results of the studies included.

Quality of the methodological study

To evaluate this aspect it is important to use instruments that can detect the possibility of bias.

Results

At this point, measures of the observed effect (odds ratio, relative risk, significant difference, among others) are provided with indicators of variability (confidence intervals) and statistical significance.

Localization of the research studies

The quality of a meta-analysis depends on the type of search performed to identify and locate the original papers (16).

A literature search is made of informal, primary and secondary sources. Informal sources comprise books, personal files, review articles, among others (6). Primary sources are known journals related to the topic, as well as complementary revisions consisting of further selection of articles cited in primary sources. Also, secondary sources are automatized databases; among the most important are MEDLINE, EMBASE, Web of Science and Cochrane databases (17). In addition, is important to note that the selection of the databases depends on the topic, because there are many specific databases.

Example 3. The following example was taken from a previous meta-analysis where we evaluated the association between suicidal behavior and the 5-HTR1A gene (18).

Identification and selection of publications

A literature search was performed covering from April to June 2010. Relevant publications were identified using the following search terms in Medline, PubMed and Web of Science databases: “HTR1A and suicidal behavior”, “HTR1A and suicide”, “rs6295 and suicidal behavior”, “rs6295 and suicide”, and “HTR1A C-1019G and suicide”. These words were combined to retrieve the summaries. The search also implicated the review of the bibliography cited at the end of various research articles (18). For more examples, see (19).

Inclusion and exclusion criteria of the studies

Not all the papers localized in the search may be included in the meta-analysis. The researchers should comply with the requirements to consider their inclusion in the meta-analysis. At this step of the meta-analysis one may incur in a “selection bias”. To reduce the risk of committing bias, the review of the studies should be conducted by different researchers following a list of inclusion and exclusion criteria. In this manner, the reliability and accuracy of the meta-analysis is maximized (5).

Evaluation of the quality of the studies included

In every meta-analysis it is important to assess the quality of the studies included and this implies to perform a control study. There are some basic aspects that must be controlled in the information that is being acquired such as study design, the possibility of combining different studies, bias controls and statistical analyses for each study included in the meta-analysis. The Newcastle– Ottawa Scale (NOS) was developed to assess the quality of non-randomized studies. The value of this instrument lies in its design, content and ease of use directed to the task of incorporating quality assessments in the interpretation of the results of a meta-analysis (20).

Example 4. The following example was drawn from a previous meta-analysis where we evaluated the association of suicidal behavior and the 5-HTR2A gene (8).

Statistical analysis

Studies deemed for inclusion in the systematic review were scored for methodological quality using the Newcastle–Ottawa Assessment Scale. A score of six was taken as the cut-off point to distinguish higher from lower quality studies. Quality assessment was done by the same two authors (TBG-C, CAT-Z) based on the NOS instrument.

Analysis of heterogeneity

There are several statistical and graphical methods to evaluate the heterogeneity of the meta-analysis, but in general all statistical tests designed to verify the existence of heterogeneity are based on the assumption of zero variability among studies. One of the most appropriate tests to evaluate heterogeneity is the Q test proposed by Dersimonian and Laird, preferred for issues of validity and computational simplicity. The Q test can be supplemented with some graphical representation to allow a visual analysis of the variability among studies. The most used representations are the Galbraith graph—recommended in observational and experimental studies—and the L’Abbé plot, which is more restrictive and is only applicable in meta-analysis of clinical trials (5).

The Galbraith graph exhibits the precision of each study and is calculated as the inverse of the standard error versus the standardized effect; it also represents the fitted regression line to these points and a confidence band. Studies that fall outside this band are the main contributors of heterogeneity. In addition, the position of the studies on the abscissa (x-axis) allows a visual identification of those studies with more weight in the meta-analysis. The Galbraith graph can also be used to detect additional sources of heterogeneity in labeling studies as well as different variables, such as the year of publication (3). On the other hand, the L’Abbé plot is more restrictive than the Galbraith graph. It is only applicable in meta-analysis of clinical trials. Moreover, this type of graph can only be built when the effect size is based on binary data (7).

Many statistical tests are available for evaluating heterogeneity between studies. Higgins and colleagues, in two highly cited papers (21, 22), proposed the routine use of the I2 statistic. This statistic can be used to compare the amount of inconsistency across different meta-analyses even with different numbers of studies. I2 is routinely implemented in all Cochrane reviews and is increasingly used in meta-analyses published in medical journals (23).

To explain heterogeneity requires experience. When heterogeneity is encountered in a meta-analysis, there are some options available. The researcher may provide a summary measure, even when heterogeneity is present or may not proceed with a summary of the primary studies (6). If the researcher decides to analyze the results despite heterogeneity, it is necessary to measure the variability “among studies”, “intra-studies” and the variation coefficient among studies. If the researcher suspects that there are reasons to explain the heterogeneity of results among studies, the most recommended option is to perform a subgroup analysis combining only those studies that meet a certain condition or feature, to get a more homogeneous sample (5).

Example 5. The following example comes from a previous meta-analysis in which we evaluated the association of suicidal behavior and the COMT gene (14).

The pooled OR derived from all studies indicated a non-significant association of the met allele in the COMTval/met polymorphism with suicidal behavior (Random effects model: OR = 1.07; 95% CI 0.85–1.33; P(Z) = 0.19). We observed heterogeneity in all studies (Q = 57.08, df = 1; P = 0.0005). Subsequently, we carried out a second analysis, which only included studies inside the heterogeneity curve. As a result, we could not find an association either (OR = 1.09, 95% CI 0.97–1.23; Z = 1.11, P(Z) = 0.26).

Example 6. The following example was taken from a previous meta-analysis where we evaluated the association between suicidal behavior and the 5-HTR1A gene (18).

The pooled OR derived from all studies indicated a non-significant association of the G allele of rs6295 and suicidal behavior (Random effects model: OR = 1.08; 95% CI = 0.80–1.45; P(Z) = 0.80). Heterogeneity was observed in all studies (Q = 17.84; df = 4; P = 0.0013). Subsequently, we performed a second analysis, which only included studies inside the heterogeneity curve (Italian, German, Ukrainian and Korean samples) (18).

Combination of results

There are several statistical techniques for combining and presenting the results in a meta-analysis study. The choice of the method depends on the type of the outcome/effect used and on assessing the degree of heterogeneity of the study results. Most meta-analyses are based on one of two statistical models: the fixed effects model and the random effects model (7).

Fixed effects model

Under the fixed effects model, it is assumed that all studies in the meta-analysis share a common effect size, that is, all factors that may influence the effect size are the same in all the studies. It makes sense to use the fixed effects model if two conditions are met. First, the idea that all studies included in the meta-analysis are functionally identical. Second, when the goal is to compute the common effect size for the identified population, and not to try and make a generalization to other populations (2).

Random effects model

This model assumes that the studies have enough in common that it makes sense to synthesize their information, but there is in general no reason to believe that they are all identical in the sense that the common effect size is exactly the same in all the studies. When the researcher is accumulating data from a series of studies performed by researchers operating independently, it is unlikely that all the studies will be functionally equivalent (6). Typically, the subjects or interventions in these studies may differ in ways that can have an impact in the results. Therefore, we must not assume a common effect size. In these cases, the random effects model is more appropriate than the fixed effects model (2, 3).

Interpretation of results

The evaluation of the size of the pooled effect, the possible causes of heterogeneity and the evaluation of the stability of the meta-analysis are involved in the interpretation of the results (5, 18).

Graphical representation of results

The usual way for displaying data from a meta-analysis is a pictorial representation. The forest plot provides context for the analysis. The plot highlights the effect sizes, the sum of effects, confidence interval, and odds ratio or risk ratio. To this end, when building a graph, the abscissa axis (x-axis) depicts the viewed effect (odds ratio, relative risk, among others) and on the coordinate axis (y-axis) lie the different studies, usually ordered by year of publication or some other arrangement (3, 7).

Example 7. The following example was extracted from a meta-analysis where we evaluated the association of suicidal behavior and the 5-HTR2A gene (8).

Figure 1 shows only the evaluation of C allele vs T allele. Odds ratios and forest plots of all models in overall studies.

Fig. 1. Odds ratios and forest plots of all models in overall studies. C allele vs T allele.

Limitations

The meta-analysis technique presents certain limitations in methodology. These limitations must be recognized and taken into account when interpreting the results. First, the meta-analysis can lead to distorted results due to possible bias in the selection and publication of studies. In addition, the validity of the results and conclusions of the meta-analysis depend on the quality of the individual studies, so the combination of biased studies can further enhance the bias. Finally, the interpretation of meta-analysis for heterogeneity or variability between studies is difficult and controversial. The compromise of those who use the meta-analysis technique is to understand these limitations and discuss them explicitly and in each case. Below we describe some of the main methodological problems of this approach (7, 15).

Heterogeneity among studies

Some variation in the results of the studies is expected due to chance alone, but an excess of variability reflects true differences in the results of the trials; this situation is called “heterogeneity”. A first methodological criticism of meta-analysis is to attempt a statistical combination of results from studies that show great variability among them. This difficulty is not unique to meta-analysis, since it is shared by all clinical research. The wide variety of features inherent in the subjects of the study makes it necessary to design a uniform protocol, conduct a rigorous process of selection of the participants in the study and a subsequent careful analysis of the influence of the extreme cases or outliers in the results (6, 18).

Publication bias

While a meta-analysis yields a mathematically accurate synthesis of the studies included in the analysis, if these studies are a biased sample of all relevant studies, then the mean effect computed by the meta-analysis will reflect this bias. Several lines of evidence show that studies that report relatively large effect sizes are more likely to be published than studies that report small effect sizes. Since published studies are more likely to find their way into a meta-analysis, any bias in the literature is likely to be reflected in the meta-analysis (18).

Example 8. The following example was taken from a previous meta-analysis in which we evaluated the association between suicidal behavior and the COMT gene (14).

Our study presents some limitations. With regard to the meta-analysis, publication bias has to be considered, since negative studies are less likely to be published. Also, an overrepresentation of the results showing an association between the polymorphism and the investigated disorder is also possible (24). Although the contribution covering from genetic factors to personality traits may differ between male and female subjects, we did not analyze for gender. Other limitations are inherent in many meta-analysis of association (including this one) such as their retrospective nature and the inclusion of study-level data.

Meta-analysis of GWAS

In other hand, recently, in order to know the Genes of Major Effects in complex diseases as suicide behavior, bipolar disorder, schizophrenia and others psychiatric disorders, the meta-analysis of genome-wide association studies (GWASs) has become a popular method. Some researchers sometimes run a GWAS followed by a replication study and then meta-analyses the replication data with the discovery data in order to capture the totality of the evidence. In this regard, we can observe two types of meta-analysis of GWAS: for discovery and for replication. Within the meta-analysis for discovery, two designs are commonly found in the literature. In the first case, the members of a consortium have the opportunity to work together to ensure the comparability of their quality control and primary analyses, and to collaborate on more detailed follow-up analyses should interesting effects be observed. Probably, at the second design the researcher find difficult to obtain genome-wide significance, so go to the web in search of publicly available GWAS data that can be combined with their primary studies in order to obtain more precise results. For a review, see (25, 26).

Conclusions

Finally, as occurs with all research studies, a meta-analysis must draw conclusions; this part of the meta-analysis must be clear, not extensive, and must summarize the principal results of the meta-analysis.

Conclusions of this article

The meta-analysis study has become a helpful tool in psychiatry since it systematically summarizes available evidence. The meta-analysis allows more generalization in the results of individual studies, possesses higher validity because the samples come from different studies and are not restricted to the same population. Even though the meta-analysis study provides several benefits in clinical and research areas, it is rather important to understand each of its component parts.

Authors’ contributions

TZC and GCTB conceived the study, participated in its design and drafted the manuscript.

Acknowledgements

This work was supported by UJAT-DAMC-2012-02 and CONACYT CB2012–177459 grants.

Declaration of interest: The authors report no conflicts of interest. The authors alone are responsible for the content and writing of the paper.