Meta-Study

ABSTRACT

Meta-study is a method for analysing the content and the process of knowledge production in a body of qualitative research. Conducting a meta-study involves four steps: (1) meta-data-analysis which involves the study of empirical findings; (2) meta-method which examines the epistemological soundness and rigour of methods; (3) meta-theory which examines the structures, assumptions, and principles underpinning the primary research studies; and (4) meta-synthesis which brings the three steps together and considers the plausibility of existing accounts, what has been neglected, and what new avenues have been opened for advancing knowledge. Qualitative researchers in sport and exercise psychology (SEP) have recently started using meta-study to examine bodies of qualitative research in various areas including positive youth development, junior-to-senior transition, athletic identity and mental toughness development. Our review shows that meta-study has been a useful method for demonstrating how methodological developments have influenced how qualitative researchers apply methods and conceptualise the phenomena of interest. However, there have been diverse applications of meta-study and, in the absence of recent updates on the method, meta-study is in danger of remaining underdeveloped or becoming outdated. Based on the review, we outline guidelines for SEP scholars to employ meta-study rigorously.

KEYWORDS:

• Epistemology
• methodology
• validity
• synthesis
• qualitative research

Following the growth in the use of qualitative research methods in sport and exercise psychology (SEP) over the last 30 years (Poucher et al., 2020), the field is increasingly finding the need to take stock of knowledge on particular topics by bringing together the findings of multiple studies through some form of synthesis. Methods of synthesising the findings of separate studies are well developed for quantitative meta-analyses (Curran & Williams, 2020) and are used widely for qualitative research outside of sport and exercise (Williams & Shaw, 2016). All methods of synthesis to some degree are based on the idea that having an overview of many studies is better than just knowing about one study – an assumption that not all qualitative researchers would agree on in the first place (Walsh & Downe, 2005). These tensions notwithstanding, qualitative researchers in SEP are often turning to research synthesis and are faced with a menu of available options that are grounded in various traditions. The purpose of this article is to outline and critically discuss one such approach; meta-study.

This review seeks to provide an updated presentation of meta-study, to critically evaluate how the approach has been used in sport and exercise psychology (SEP), and to identify best practices in using it in our field. We begin by outlining the origin, history and scope of meta-study before providing a succinct characterisation of its essential components. We then offer insights into the current state of meta-study within the SEP field by giving an overview of the published meta-studies addressing SEP topics. To critically appraise the approach, we then suggest what some of the strengths and weaknesses might be and then note how the approach could be developed in future. Finally, the article concludes with guidelines for SEP researchers interested in conducting a meta-study.

Origins of meta-Study

The majority of contemporary meta-studies rely heavily on Paterson et al.’s (2001) key text, Meta-Study of Qualitative Health Research: A Practical Guide to Meta-Analysis and Meta-Synthesis. Within this text, meta-study is introduced as being an ‘advanced method’ for ‘seasoned researchers’ (p. ix) and an approach that involves ‘the analysis of theory, methods, and findings of qualitative research and the synthesis of those insights into new ways of thinking about phenomena’ (p. 1). As a resource, the book provides researchers with the necessary conceptual underpinnings of the method as well as offering effective practical guidance through drawing heavily on the authors’ previous attempts to synthesise qualitative research in chronic illness (e.g. Paterson et al., 1998; Thorne & Paterson, 1998). While one of the authors in particular – Sally Thorne – has continued to contribute to debates about methods of qualitative research synthesis (see Thorne, 2017; Thorne et al., 2004), there has been no second edition of the book and little development of the meta-study method (at least under this label) since its publication two decades ago.

While Paterson et al. (2001) remains the ‘go-to’ text for researchers wishing to undertake a meta-study of qualitative research, their work largely operationalises ideas that have a deeper history which may be of interest. For example, much is drawn from Noblit and Hare’s (1988) development of meta-ethnography as well as Ritzer’s (1991) work on meta-theorising in sociology. Indeed, another essential reading for researchers wishing to fully appreciate the origins of meta-studies in qualitative research is Zhao’s (1991) clear articulation of the meta-study ‘boom’ which identifies the underlying structures common to meta-studies and a description of how it was being practised at the time. Importantly, there is a case to argue that all of these influences converge and capitalise on the ongoing appreciation in the second half of the twentieth century that the knowledge produced by science is to an extent socially constructed. Kuhn’s (1970) seminal work in the philosophy of science along with scholars contributing to the sociology of knowledge – such as Robert Merton (1973) and Ian Hacking (1999) – served to highlight that the process and historical situatedness of doing research cannot be ignored.

Such ideas are incorporated into the assumptions of meta-studies. Put simply, if the findings of primary research are seen as constructions, then attempts to synthesise findings should pay greater attention to the socio-historical context of the research that produced those findings. For meta-study advocates, this includes accounting for the methods, concepts and theories used to construct research findings. Indeed, this is where meta-study can be seen as unique compared to other types of qualitative synthesis (e.g. realist review or meta-ethnography) in that a central interest is placed on analysing the processes of knowledge construction and how theories and methodologies have shaped the generated findings. In these terms, meta-study is based on a ‘constructivist’ underpinning and aims to provide a framework for analysing the problems of knowledge construction in social and health sciences.

From these origins, the meta-study approach has grown in popularity and has been used across a range of disciplines where qualitative research is valued. According to Google Scholar, Paterson et al.’s (2001) text has over 1000 citations, suggesting that the method has gathered a substantial following. Indeed, meta-study is often recognised among one of the established methods of qualitative synthesis (Barnett-Page & Thomas, 2009; Campbell et al., 2011) and hence remains on the list of viable options for researchers to choose from. That said, meta-study could be seen by some as an increasingly marginal and specialist approach to synthesising qualitative research within contemporary trends – only one meta-study can be found in the Cochrane library of systemic reviews (Roe et al., 2007), for example.

Sally Thorne’s (2017) recent critical commentary suggests that current trends in research synthesis are moving away from the characteristics of meta-study by moving towards a focus on rather ‘formulaic’ and ‘standardised’ procedural precision and a focus on reporting ‘hollow’ and ‘superficial’ findings without the contextual criticality that the original research deserves. Although some may wish to object to these critical characterisations of contemporary trends, it is certainly worth noting how key authors view the contrasts and distinctions between different approaches to research synthesis. Indeed, calling for something of a resistance to contemporary trends, what Thorne strongly advocates for is synthesis studies that,

include the full spectrum of methodological orientations within the perspectival kaleidoscope, making informed interpretations about the disciplinary and theoretical traditions within which each primary researcher or research team was operating, and the influence those conditions may have had upon what they chose to articulate as their findings (Thorne, 2017, p. 10).

This being said, a different view is put forward by Greenhalgh et al. (2018) which emphasises that it is more appropriate to see different approaches to synthesis as serving different and complementary purposes. Rather than existing in a hierarchy, some approaches serve the scientific community well by summarising data or findings, while others serve us well by deepening our understanding through providing interpretation and critique. Meta-study emphasises the latter of these two categories.

Notwithstanding these debates about the position of meta-studies within the contemporary landscape of qualitative research synthesis, meta-study has demonstrated utility across a wide scope of research topics and disciplines. To offer a brief but helpful flavour of this work, a recent study used meta-study to synthesise research about women's stories of living with breast cancer (Smit et al., 2019). The study produced core themes from the literature including ‘the burden of breast cancer’, ‘existential ordeal’ and ‘changes in self-image’ and then put forward an explanatory framework for future research to use. In education, Thurlings and den Brok (2017) conducted a meta-study to investigate the professional development activities of teachers and showed that there is a good evidence base for claiming that teacher knowledge, teacher skills, and student learning are achieved through professional development activities. As another example, Graves (2019) used the approach to explore literature about people’s experiences of neighbourhood violence. The study revealed the multitude of factors that were significant in the decision-making process about residential re-location because of neighbourhood violence.

While the scope of meta-studies appears to be broad in terms of the disciplines within which it is used and the topics that it is used to investigate, the approach does encourage researchers to bring their attention to particular areas of focus. This focus is the result of the three explicit phases of conducting a meta-study (as will be outlined later in greater detail) which begins with – perhaps most unsurprisingly – an attempt to combine findings from multiple studies in the hope of revealing a ‘birds-eye view’ by pooling knowledge from otherwise isolated investigations. However, as Morgan (2003) pointed out, the next two phases are what distinguish meta-study from more traditional methods of qualitative synthesis. These phases of ‘meta-method’ and ‘meta-theory’ broaden the scope of the approach by asking the research team to say something about, respectively, how the methods used and the theoretical constructs deployed in the primary investigations actively shaped the reported findings. In this way, the scope of meta-studies extends beyond the synthesis of findings from separate studies and requires that methods and theories are brought into view.

Components of meta-Study

According to Paterson et al. (2001), meta-study involves analysis followed by synthesis. The analysis part consists of three components: meta-data-analysis, meta-method, and meta-theory. The three analysis components do not necessarily unfold sequentially; it is common for them to be conducted concurrently. The meta-synthesis stage must follow the analytic components, however, because it is derived from the results generated from them. Although it is beyond the scope of this article to give a detailed, step-by-step guide to conducting a meta-study the remainder of this section give an outline of its four components.

Meta-data-analysis is the study of the findings of research in a particular area of inquiry. It involves, therefore, processing data which has already been processed by other researchers. In other words, it is an analysis of analyses reported in the publication of primary research. Meta-data-analysis should go beyond simply aggregating the findings of these studies, though. It requires critical interpretation of multiple accounts of a given phenomenon ‘in order to reveal the similarities and discrepancies among accounts’ (Noblit & Hare, 1988; cited in Paterson et al., 2001, p. 10). In this way, meta-data-analysis can be used to identify themes and divergences apparent in the literature on the topic of interest. Paterson et al. (2001) advocate an approach that includes both hermeneutic (involving representing the interpretations contained in the primary research) and dialectic (entailing a comparison of the interpretations of the primary researcher with other possible interpretations) processes. Nonetheless, any systematic interpretive approach that is used in the primary analysis of data could be appropriate for use in a meta-data-analysis. However, meta-study researchers should adopt a data analysis method or strategy that fits with their research question/aims and design, whilst also keeping their paradigmatic assumptions in mind during meta-data-analysis.

Meta-method involves studying the rigour and epistemological soundness of the method used in primary research studies. A key aspect of this includes determining the appropriateness of the specific methods for investigating the phenomenon of interest. Paterson et al. (2001) consider meta-method to involve noting both the details of the procedures (participant characteristics, sampling, data collection techniques) as well as broader considerations of how disciplinary assumptions, methodological traditions and researcher positionality shapes the research that is being produced. Since methodological decisions are an important factor in directing a particular field of study, meta-method can contribute to the development of theory and the creation of an explicit strategy for future research because it allows for a reinterpretation of how fundamental disciplinary assumptions shape the research we do and the conclusions we arrive at as a consequence of that research (Paterson et al., 2001). Meta-method techniques can also be applied to compare and contrast the findings of studies employing different methodological frameworks to explore the distinct implications of a range of approaches and epistemologies. Paterson et al. (2001) had themselves adapted Burns’s (1989) criteria for appraising quality of primary studies, but also suggested that there can be some criteria specific to the method being used. However, since qualitative research literature in this respect has significantly developed since the publication of these works, meta-study researchers are advised to consult more recent literature and be transparent in the choice of criteria used in evaluating methodological rigour.

The third component – meta-theory – is the analysis of the structures, assumptions, and principles underpinning the primary research studies. This involves the scrutiny of features including ‘the philosophical, cognitive, and theoretical perspectives underlying research design strategies; the sources and assumptions inherent in emerging theory; and the consideration of the relationships between emerging theory and the larger contexts in which it has been generated’ (Paterson et al., 2001, p. 13). Meta-theorizing can contribute towards novel or enhanced understandings about the application of theory in a substantive area, including creating strategies for evaluating and critiquing existing theories and providing stimulus for the generation of new theories.

Finally, meta-synthesis draws together the ideas which have been deconstructed during engagement with the three analytic meta-study components. The aim at this stage should be to produce a new interpretation of a phenomenon, which is derived from analysing the data, method, and theory by which the phenomenon has been studied by others. Thus, the meta-study furnishes the opportunity to articulate theories that account for empirical complexities and contradictions within a field of study. It also provides the basis for insights into the implications of a variety of theoretical, methodological, and structural decisions of past, present, and future research on the phenomenon.

Current state of knowledge in Sport and Exercise Psychology (SEP)

To explore how SEP researchers have used meta-study, we conducted a review of meta-studies in the field. Since our main aim was to appraise the ways meta-studies are used in SEP, explore current debates, and identify ways to improve practice, we followed a narrative review approach (Ferrari, 2015) and not a systematic review approach. We located the relevant studies using database searches (Google Scholar, ScienceDirect, PsycInfo) using keywords ‘meta-study’, ‘sport psychology’ and ‘exercise psychology’ and reviewed citations and reference lists of already identified studies. Our inclusion criteria were: peer-reviewed review articles published in English that use the meta-study method and address a topic in SEP.

While the meta-study was first introduced at the turn of the millennium, in SEP the method has only more recently started to attract attention. The first meta-study in SEP was conducted by Tamminen and Holt (2010) who analysed qualitative studies to explore adolescent sport participants’ stressor appraisals and coping in sport. After that, more meta-studies appeared in 2016 and focused on mental toughness development (Anthony et al., 2016) and athletic identity (Ronkainen et al., 2016a, 2016b), to be then followed by more studies in the next few years (Drew et al., 2019; Holt et al., 2017; McHugh et al., 2019; Ryba et al., 2020). The emergence of meta-studies in SEP is likely linked to the expansion and diversification of the qualitative research landscape in the discipline and debates around paradigms, methodologies, and quality of qualitative research (e.g. Culver et al., 2012; Smith & McGannon, 2018; Sparkes & Smith, 2009). The meta-study, which involves the analysis of assumptions underlying theories and methods, has been ideally suited to scrutinise these features of qualitative research that have been a focus of debate. In Table 1, we have provided four examples of meta-studies and their key findings in meta-theory, meta-method and meta-data-analysis. In the next sections, we analyse how researchers in SEP have approached the three analysis components of the meta-study: meta-data-analysis, meta-method, and meta-theory.

Table 1. Examples of Meta-Studies in SEP.

Year	Authors	Topic	Aims of the study	Key findings: Meta-theory	Key findings: Meta-method	Key findings: Meta-data-analysis
2020	Massey & Williams	Childhood trauma survivors’ sporting experiences	Examining the philosophical, theoretical, historical, and methodological trends and their impact on knowledge production and how this knowledge can inform future research.	Most studies did not explicitly connect their theoretical frameworks to philosophical assumptions	7/16 studies provided enough detail to assess methodological coherence. Concerns for the lack of rationale for chosen procedures and researcher reflexivity	Four themes: a sense of belonging, environmental considerations, psychological escape, embodied experience. Sport is not a panacea for those who have experienced trauma.
2019	Drew et al.	Junior-to-senior transition	Critiquing methodological and theoretical issues in the literature; synthesising key factors influencing athletes’ junior-to-senior transition	Two main theoretical models have been used. Cultural considerations have been often missing	Participants were mainly from Euro-American contexts. Most studies relied on interviews. Lack of data from female participants.	Identification of (1) individual (2) external and (3) cultural factors affecting transition, and intervention strategies to support athletes.
2017	Holt et al.	Sport-based positive youth development (PYD)	Creating a model of PYD based on (1) reviewing and evaluating qualitative studies and (2) analysing and synthesising the findings from identified studies	26/45 studies did not specify a conceptual framework. Only 14 studies explicated the philosophical perspective	Studies used of multiple data collection methods. Samples were often not adequately described. ‘Named’ methodologies were rarely used. Validity issues not addressed only in eight studies	Three categories were inductively developed: PYD climate, life skills programme focus, and PYD outcomes
2016a	Ronkainen et al.	Athletic identity	Examining the processes and findings of studies on athletic identity	37/45 studies identified a theoretical framework. The theorising of identity has shifted from role-based conceptions to cultural and contextualist approaches.	Most common data collection method was interviewing. A broad range of data analysis methods was used. There was a lack of transparency in how methodological decisions were made	Findings were analysed in relation to each theoretical perspective. Athletic identity could be a source of meaning and self-esteem, but problematic in moments of disruption and retirement.

Meta-data-analysis considerations

Researchers in SEP have typically prioritised either the theoretical and methodological, or the empirical findings in their meta-studies. Studies with the primary research objective on theorising based on the meta-data-analysis (e.g. Anthony et al., 2016; McHugh et al., 2019; Tamminen & Holt, 2010) have developed new integrated models of the studied phenomena. For example, the main aim of Tamminen and Holt’s (2010) research was to develop an integrated theoretical perspective on adolescent athletes’ sport stressor appraisals and coping. In the meta-data-analysis, they inductively coded the findings of the primary studies and organised them into themes (e.g. stressor appraisals are contextual and dynamic; coping is contextual and dynamic) and sub-categories (e.g. coping resources, reflection and learning) to synthesise them into a new model.

As another example, Anthony et al. (2016) focused on developing a framework of mental toughness development from extant qualitative literature. In the meta-data-analysis, they used an adapted version of Noblit and Hare’s (1988) meta-ethnography, which involves determining the relationships between the reviewed studies and translating them into one another. Based on reinterpretation of primary studies, they identified four themes (personal characteristics, progressive development, interactions with environment, and breadth of experience). Based on the meta-data-analysis, they proposed a bioecological model of mental toughness development which was theorised based on the bioecological model of human development (Bronfenbrenner & Morris, 2006).

On the other hand, some studies in SEP have had a more limited focus on reinterpreting findings of the primary studies or building a new framework (e.g. Massey & Williams, 2020; Ronkainen et al. 2016a), or no focus on this analytic stage at all (Ryba et al., 2020). These studies have centred the attention on how paradigmatic debates, theorising and the use of qualitative methodologies have evolved. As an example of meta-study that did not involve meta-data-analysis at all, Ryba et al. (2020) examined how mixed methods research had been conducted in SEP. Their aim was to understand how researchers had positioned their mixed-methods investigations (for example, in pragmatism, interpretivism or realism), the degree of integration of quantitative and qualitative elements of the study, and how validity had been addressed. Meta-study has been also used outside of SEP in a similar way to explore how qualitative researchers use analytic pluralism, where the findings of the studies was not researchers’ central interest (Clarke et al., 2015).

Meta-method considerations

Reflecting the ‘state of the art’ in qualitative research, meta-study researchers have drawn on diverse ways to appraise the primary studies. Anthony et al. (2016) and Drew et al. (2019) used the Critical Appraisal Skills Programme checklist (CASP, 2018), Holt et al. (2017) formulated a list of criteria based on previously published meta-studies, and Massey and Williams (2020) highlighted the lack of consensus in qualitative research community on how to assess research quality and evaluated the primary studies based on questions asked at each level of analysis (meta-theory, meta-method, and meta-data-analysis). Massey and Williams (2020) and Ronkainen et al. (2016a) also highlighted the criteria of methodological coherence and transparency when assessing the quality of primary studies.

The accumulated meta-studies in sport and exercise contexts have been shaped by different understandings of the further scope of meta-method analysis. In short, some studies have focused the meta-method analysis on details of the methods (i.e. tools of research such as interview or thematic analysis), whereas others were more focused on evaluating broader methodologies (i.e. justifications of the use of particular methods and overall methodological coherence). For example, Drew et al. (2019) focused more on methods and reported participant characteristics (e.g. country of origin, role in sport, gender, type of sport, etc.) and methods of data collection (e.g. semi-structured interviews). While their study provided important insights on limitations of the participant samples that had been studied (e.g. lack of ethnic diversity and studies on female athletes) and noted the potential cultural variations in athletes’ junior-to-senior transition that had been underexplored, it did not explore how research questions were derived or address researcher positionality the methodological coherence of examined studies.

On the other hand, Massey and Williams (2020) focused on methodology and issues of coherence and transparency (e.g. whether philosophical assumptions were aligned with research questions, the researcher’s role in knowledge production was discussed, and a rationale for the procedures was provided). In their study, the analytic attention was placed on whether a coherent and transparent justification for the use of methods was provided, and whether the type of analysis was aligned with the stated method and purpose of the study. On the other hand, they did not discuss the specifics of sample characteristics (e.g. geographic location) and how these choices might have impacted the knowledge that had been generated.

Meta-theory considerations

Meta-study researchers have reported the types of theoretical and conceptual frameworks used in primary studies – or, frequently, the lack of them (Holt et al., 2017). Studies have reported the types of theory used and sometimes noted the potential consequences that the dominant theories could have had on research that had been carried out, such as the focus on studying ‘normative’ athlete pathways and the focus on individual experiences (Drew et al., 2019). Also, some studies have traced the shifts in theorising over time, such as a ‘cultural turn’ in how athletic identity has been conceptualised (Ronkainen et al., 2016a). However, meta-theory considerations in meta-studies in SEP have often remained fairly descriptive and received less attention than meta-method and meta-data-analysis. Furthermore, most studies have not sought to compare, contrast and evaluate the theories used, or how new theories might have advanced understandings of the phenomenon compared to previous ones. This presents an important avenue for strengthening meta-studies in the future.

As a part of the meta-theory analysis, Holt et al. (2017) Massey and Williams (2020) and Ronkainen et al. (2016a, 2016b) also analysed the paradigmatic positioning of the primary studies. Holt et al. (2017) reported that 49/63 articles did not explicate the paradigmatic stance, which demonstrated that the consideration of philosophical underpinnings of the research was not common practice in studies exploring sport-based youth development to date. However, in their meta-study on athletic identity research, Ronkainen et al. (2016a) reported that more recent studies more often explicitly stated the ontological and epistemological assumptions guiding the study, which is likely to reflect the growth of methodological literature and recommendations to do so (e.g. Culver et al., 2012).

Strengths and weaknesses

As with any method of synthesis, we see both strengths and weaknesses in the meta-study. We identify three key strengths of Paterson et al.’s (2001) meta-study approach. First, when the body of literature to be synthesised is heterogeneous (e.g. not on the same topic of research; see Clarke et al., 2015, for example), meta-study may be more appropriate than other forms of meta-synthesis that prioritize synthesising findings because meta-study is about scrutinising the research process in its entirety (Massey & Williams, 2020). Secondly, because of this focus on the research process, meta-study supports the examination of methodological strengths and weaknesses (McHugh et al., 2019), which cannot be said of other methods of synthesis. Thirdly, as Reyes et al. (2020) demonstrate, meta-study can aid the development of theory from multiple empirical studies.

Despite these strengths, we caution researchers to consider three significant limitations of meta-study before adopting it. The first of which is the flip side of the first strength of the meta-study approach outlined above. The more diverse a sample of literature becomes, the more diluted the findings of a meta-study may become (Kelly et al., 2018). For the most part, this issue will depend upon the aims of the meta-study. If the aims are to synthesise findings from multiple empirical studies to develop theory, then a more homogenous sample is likely to be more appropriate. However, consistent with meta-study, researchers need to be cognisant that aggregation does not always make the evidence more compelling. That is, the findings always need to be considered in light of potential weaknesses and threats to validity identified in the meta-theory and meta-method analyses. On the other hand, if the primary aims are to evaluate different methodological or theoretical/philosophical approaches within a particular field, then a more varied sample (in terms of research topic) may be less important.

Secondly, meta-study appraisal relies on the ability of the primary researchers to communicate their decision-making process regarding study design and research findings (Massey & Williams, 2020; Paterson et al., 2001). It is possible that a rigorous study is communicated poorly, which would make it difficult for someone conducting a meta-study to adequately evaluate it. This may be less significant for those wanting to synthesise research findings, but it is very important for a thorough meta-study that also focuses on meta-methods and meta-theory. There is no simple solution to this problem from a meta-study perspective. To better enable the possibility of conducting meta-studies in future, we would support the view that researchers ought to take care in communicating their research processes clearly.

Thirdly, as with other methods of synthesising qualitative research findings, meta-study can risk decontextualising data from the original studies, for example, by not describing the details of specific communities when providing quotes from participants (McHugh et al., 2019). This type of decontextualization is problematic in at least two ways. Firstly, it assumes a degree of generalisation or universality of findings that may not be intended by the primary research and is not likely to be supported by the data. Secondly, not acknowledging the particulars of context could be considered as unethical because it does a disservice to the participants of those studies – removing their humanity and reducing them to mere data.

Meta-study is attracting more interest in SEP, but it is important to keep in mind that it remains a fairly underdeveloped approach. Paterson et al.’s Meta-Study of Qualitative Health Research. A Practical Guide to Meta-Analysis and Meta-Synthesis (2001) remains the ‘go-to’ text for meta-study, but the authors themselves concluded that the method, as they presented it, is ‘in its infancy’ (p. 132) and ‘has not yet been sufficiently tested or evolved’ (p. 132). Twenty years later, however, we do not have a new edition of their book and there have been few systematic efforts to develop and refine the method. At the same time, debates surrounding qualitative research have been vibrant and there are important shifts in how qualitative researchers conceptualise and study phenomena, and how they evaluate the quality of qualitative inquiry. This raises the question of whether the meta-study is becoming outdated. In the absence of updated literature on the method, researchers need to be paying careful attention to other developments in the qualitative research landscape and their implications for conducting meta-studies. Recent reviews of qualitative research (McGannon et al., 2019; Poucher et al., 2020) and debates on rigour or validity (Ronkainen & Wiltshire, 2021; Smith & McGannon, 2018) provide good overviews of discussions in SEP that are likely to be relevant to qualitative researchers interested in using the meta-study in their work.

Future perspectives and methodological advances in the meta-Study

As this review has indicated, the meta-study has recently started to appear in SEP publications as one viable approach to synthesising qualitative studies. The meta-study seems well suited for responding to increasing calls for qualitative researchers to increase awareness of epistemological questions and focus attention on methodological coherence (McGannon et al., 2019; Poucher et al., 2020). Meta-studies help charter the methodological landscape, the dominant theoretical assumptions that shape the research that is being conducted, and how researchers are addressing these issues when working with qualitative research designs. At the same time, meta-studies also typically provide reinterpretations of the primary data to develop new knowledge of the studied phenomenon. While these strengths make meta-study a potentially appealing approach to qualitative researchers, several tensions appear in the application of meta-study that are important to consider when moving the methodology forward.

As our review has indicated, some meta-studies in SEP heavily accentuate meta-data-analysis, whereas other studies have focused more on meta-theory and meta-method. While the ratio of focus on these different elements might not be clear cut, Paterson et al. (2001) emphasised that meta-study is an approach that moves beyond a synthesis of research findings, indicating that the processes of knowledge production (i.e. meta-theory and meta-method) should be an important interest for researchers employing this method. The particular strength of the meta-study is that it provides answers to not only what was found, but also how it was found and why. In future work, it is important to continue emphasising the unique contribution of the meta-study compared to other types of qualitative synthesis and exploit the strengths of the method when using it to make sense of a body of qualitative work.

In future work, it is also important that meta-study researchers in SEP move beyond describing theories and methods (and methodological weaknesses) and use the meta-study to evaluate competing theories and the impact of methodologies on the knowledge base that has been generated. Some of the meta-studies in SEP have been fairly descriptive and reported the more technical details of methods and theories used without critically evaluating their strengths and weaknesses. While reporting the more technical details of primary studies is an important step in the meta-study and sample characteristics, for example, certainly have important implications for the knowledge that is generated, it seems vital that meta-studies move beyond this step. As Paterson et al. (2001) noted, insightful meta-studies help understand how researchers’ disciplinary, personal and socio-cultural positionality has shaped research questions, how methodological perspectives have guided the use of methods, and what ways the chosen theories and methods have shaped (and perhaps limited) understandings of the phenomenon of interest.

Paterson et al. (2001) also advocated that the key focus of meta-studies should not be in critiquing the methodological quality of individual studies, but on examining broader issues including historical developments and how basic disciplinary assumptions have shaped how researchers have chosen to investigate the phenomenon under scrutiny. A further danger evident in the reviewed SEP studies is that of placing too much focus on identifying weaknesses (e.g. not providing enough detail of the sample or procedures, not addressing researcher positionality, lack of transparency) rather than analysing how the methodological choices shaped the accounts produced. Indeed, Paterson et al. (2001) cautioned that ‘meta-study research has been challenged as being nothing more than a critique of work by other researchers’ (p. 132). They remind readers that high-quality meta-studies should be able to identify and discuss both strengths and weaknesses in the reviewed body of literature. Furthermore, they reflected on their own work and noted that ‘we presented the future imperatives for research in the area so passionately that we unintentionally conveyed that much of what researchers had done to date had been misguided’ (p. 130). Especially since the methodological landscape and best practice guidelines of qualitative research have developed significantly in the past decades, it is important to be mindful of the context in which the primary studies were produced.

Surveying the landscape of meta-studies in SEP reveals that meta-studies in the field often undertake their investigations in a critical spirit and, in line with recommendations of Paterson et al. (2001), investigate issues such as epistemological transparency and methodological coherence. It is our contention, however, that meta-studies in SEP have typically also noted strengths in the body of literature and sought to advance knowledge in the topic area by developing new frameworks and models from the meta-data-analysis and pointing towards ways to build upon the existing literature. We are hoping to see SEP scholars continuing to produce critical but constructive meta-studies that can help guide future studies and applied practice in the area of interest.

Guidelines to ensure best practice in Sport and Exercise Psychology

1. Determine whether meta-study is the most suitable method for addressing your research objectives. Conducting a meta-study involves an analysis of not only empirical findings but also of the choice of theories and methods and how they shape how we have come to know the studied phenomenon.

2. Consult the original text by Paterson et al. (2001) as well as studies that have applied the method. Relatedly, stay up to date on debates on qualitative research methodology in SEP and elsewhere.

3. Working with meta-data-analysis: Decide the type of analysis you employ to reinterpret the findings of the primary studies. For example, you could use thematic analysis, grounded theory or meta-ethnography (Paterson et al., 2001). Keep track of your coding decisions and analytic moves and, if possible, involve more than one researcher in the data analysis process.

4. Working with meta-method: Be transparent and justify the way you appraise the primary studies. Construct the questions you are investigating in your meta-method-analysis. It is useful to develop tables to document the methodological characteristics of primary studies. Also, remember to scrutinise elements that might not be explicitly addressed in the studies’ methodology section, such as: how has disciplinary positionality shaped the conceptualisation of the ‘problem’ and research questions? What about the political, institutional or geographic context? How has the culture of conducting and disseminating research impacted the way the research method is employed and reported?

5. Working with meta-theory: It is useful to document the theories used in primary studies in a table. Remember to examine the assumptions underpinning the theories, how theories have evolved, as well as how broader disciplinary and socio-cultural developments have shaped theorising over time.

6. Working with meta-synthesis: Consider the competing accounts and explanations of the studied phenomenon you have identified in the previous steps as well as threats to their validity. Which accounts are most plausible and most coherent? What has been omitted or missed possibly due to theoretical or methodological commitments? What avenues have been opened up for developing new theorising and a more complete understanding?

7. While meta-study is intended to be a critical methodology, make sure to offer your reflections on the state of the art in a constructive spirit and avoid centring your message on methodological weaknesses in the body of literature.

Conclusion

Meta-study offers a method for analysing a body of qualitative research for both the content and the process of knowledge construction. Through the analytic steps of meta-data-analysis, meta-method and meta-theory, researchers are able to synthesise current trends, identify the ways that the use of methods and theories have shaped the findings, and propose new avenues for knowledge advancement. While the interest in meta-study is increasing in SEP, the literature on the method is in danger of becoming outdated. Therefore, researchers using meta-study need to be consulting other qualitative research literature especially in relation to rigour and research quality when conducting their studies.

Disclosure statement

No potential conflict of interest was reported by the author(s).