Is Multi-Method Research More Convincing Than Single-Method Research? An Analysis of International Relations Journal Articles, 1980–2018

Abstract

While some social scientists see multi-method research (MMR) as a promising strategy for strong causal inference, others argue that it does little to strengthen the validity of research. This paper offers a systematic review of how MMR has been used in mainstream International Relations (IR) and specifically in security studies. Using the TRIP Journal Article Database and Web of Science citation data, I examine whether MMR has reached its full potential. MMR has grown in prominence since the 2000s. Scholars use it most often to examine domestic rather than interstate issues. They cite MMR articles less than they cite quantitative single-method articles and about as often as they cite qualitative single-method research. This suggests that MMR is not more influential, nor perceived as more persuasive. However, this gap has decreased in recent years. The study provides insights into IR at the research design and disciplinary levels, the utility of MMR, and knowledge accumulation in social science.

Keywords:

• methodology
• multi-method research
• research design
• International Relations
• security studies

Acknowledgements

I thank Irene Weipert-Fenner, Jens Stappenbeck, Barış Kesgin, Julian Junk, Caroline Fehl, Ben Christian, Felix Bethke, two anonymous reviewers, and the Security Studies editors, Ronald R. Krebs and Ron Hassner, for their very helpful and constructive comments. I am grateful to Irene Entringer at TRIP and Catherine Weaver and Jason Sharman for sharing their data.

Disclosure Statement

No potential conflict of interest was reported by the author(s).

Data Availability Statement

The data and materials that support the findings of this study are available in the Harvard Dataverse at https://doi.org/10.7910/DVN/7OWHLM.

Correction Statement

This article has been corrected with minor changes. These changes do not impact the academic content of the article.

Notes

1 I use “multi-method research” rather than “mixed-method research.”

2 Jason Seawright, Multi-Method Social Science: Combining Qualitative and Quantitative Tools (Cambridge, UK: Cambridge University Press, 2016); Alejandro Avenburg, John Gerring, and Jason Seawright, “How Do Social Scientists Reach Causal Inferences? A Study of Reception,” Quality & Quantity, 2022, 2, https://doi.org/10.1007/s11135-022-01353-5.

3 On the increased usage of MMR, see Mario Luis Small, “How to Conduct a Mixed Methods Study: Recent Trends in a Rapidly Growing Literature,” Annual Review of Sociology 37, no. 1 (2011): 60, https://doi.org/10.1146/annurev.soc.012809.102657; Seawright, Multi-Method Social Science, 1–4. On MMR in IR, see Gary Goertz, “Multimethod Research,” Security Studies 25, no. 1 (2016): 3–24, https://doi.org/10.1080/09636412.2016.1134016; Jason Seawright, “Better Multimethod Design: The Promise of Integrative Multimethod Research,” Security Studies 25, no. 1 (2016): 42–49, https://doi.org/10.1080/09636412.2016.1134187. On MMR in CP, see Dan Slater and Daniel Ziblatt, “The Enduring Indispensability of the Controlled Comparison,” Comparative Political Studies 46, no. 10 (2013): 1321–22, https://doi.org/10.1177/0010414012472469.

4 On MMR and causal inference, see Evan S. Lieberman, “Nested Analysis as a Mixed-Method Strategy for Comparative Research,” American Political Science Review 99, no. 3 (2005): 435–52, https://doi.org/10.1017/S0003055405051762; Macartan Humphreys and Alan M. Jacobs, “Mixing Methods: A Bayesian Approach,” American Political Science Review 109, no. 4 (2015): 653–73, https://doi.org/10.1017/S0003055415000453; Seawright, Multi-Method Social Science; Seawright, “Better Multimethod Design”; Avenburg, Gerring, and Seawright, “How Do Social Scientists Reach Causal Inferences?,” 2. On MMR and external and internal validity, see Kathleen M.T. Collins, Validity in Multimethod and Mixed Research, ed. Sharlene Nagy Hesse-Biber and R. Burke Johnson (Oxford University Press, 2015), https://doi.org/10.1093/oxfordhb/9780199933624.013.17; See also Slater and Ziblatt, “Controlled Comparison.” On MMR and policy recommendations, see Tanisha M. Fazal, “An Occult of Irrelevance? Multimethod Research and Engagement with the Policy World,” Security Studies 25, no. 1 (2016): 34–41, https://doi.org/10.1080/09636412.2016.1134186. On the comparison to SMR, see R. Burke Johnson and Anthony J. Onwuegbuzie, “Mixed Methods Research: A Research Paradigm Whose Time Has Come,” Educational Researcher 33, no. 7 (2004): 18, https://doi.org/10.3102/0013189X033007014.

5 John Gerring, “Comprehensive Appraisal,” in The Production of Knowledge, ed. Colin Elman, John Gerring, and James Mahoney (Cambridge University Press, 2020), 356, https://doi.org/10.1017/9781108762519.014.

6 Amel Ahmed and Rudra Sil, “Is Multi-Method Research Really ‘Better’?,” Qualitative & Multi-Method Research 7, no. 2 (2009): 2–6, https://doi.org/10.5281/ZENODO.938958; Scott Gehlbach, “The Fallacy of Multiple Methods,” Comparative Politics Newsletter, 2015, https://scottgehlbach.net/publications/the-fallacy-of-multiple-methods/; Derek Beach and Jonas Gejl Kaas, “The Great Divides: Incommensurability, the Impossibility of Mixed-Methodology, and What to Do about It,” International Studies Review 22, no. 2 (2020): 214–35, https://doi.org/10.1093/isr/viaa016.

7 Avenburg, Gerring, and Seawright, “How Do Social Scientists Reach Causal Inferences?,” 2.

8 Avenburg, Gerring, and Seawright, “How Do Social Scientists Reach Causal Inferences?”

9 On social science reception, see Ibid.

10 The full TRIP JAD features 9.118 articles, some of which are individual pieces of correspondence combined under one DOI. The databases are merged by DOI, so items such as this are dropped.

11 2014–2018, the five most recent years.

12 Avenburg, Gerring, and Seawright, “How Do Social Scientists Reach Causal Inferences?”

13 Cyrus Samii, “Causal Empiricism in Quantitative Research,” The Journal of Politics 78, no. 3 (2016): 950–51, https://doi.org/10.1086/686690; See also John Gerring, Sebastian Karcher, and Brendan Apfeld, “Impact Metrics,” in The Production of Knowledge, ed. Colin Elman, John Gerring, and James Mahoney (Cambridge University Press, 2020), 389, https://doi.org/10.1017/9781108762519.015.

14 Colin Elman, John Gerring, and James Mahoney, eds., The Production of Knowledge: Enhancing Progress in Social Science, Strategies for Social Inquiry (Cambridge, UK: Cambridge University Press, 2020); Avenburg, Gerring, and Seawright, “How Do Social Scientists Reach Causal Inferences?”

15 On such best practices, see Ingo Rohlfing and Peter Starke, “Building on Solid Ground: Robust Case Selection in Multi-Method Research,” Swiss Political Science Review 19, no. 4 (2013): 492–512, https://doi.org/10.1111/spsr.12052; Seawright, Multi-Method Social Science. On combining methods, see Julia Brannen, “Mixing Methods: The Entry of Qualitative and Quantitative Approaches into the Research Process,” International Journal of Social Research Methodology 8, no. 3 (2005): 173–84, https://doi.org/10.1080/13645570500154642; Julian Junk and Valentin Rauer, “Combining Methods: Connections and Zooms in Analysing Hybrids,” in Transformations of Security Studies: Dialogues, Diversity and Discipline, ed. Gabi Schlag, Julian Junk, and Christopher Daase (Routledge, 2015), 216–32, https://doi.org/10.4324/9781315707839. For a formal approach to combining methods, see Humphreys and Jacobs, “Mixing Methods.”

16 Slater and Ziblatt, “Controlled Comparison,” 1322.

17 Ibid., 1321.

18 Lieberman, “Nested Analysis.”

19 Sharlene Nagy Hesse-Biber, Deborah Rodriguez, and Nollaig A. Frost, A Qualitatively Driven Approach to Multimethod and Mixed Methods Research, ed. Sharlene Nagy Hesse-Biber and R. Burke Johnson (Oxford University Press, 2015), https://doi.org/10.1093/oxfordhb/9780199933624.013.3.

20 Lieberman, “Nested Analysis”; Hesse-Biber, Rodriguez, and Frost, A Qualitatively Driven Approach to Multimethod and Mixed Methods Research, 5; Seawright, Multi-Method Social Science.

21 Sharlene Nagy Hesse-Biber and Amy J. Griffin, Feminist Approaches to Multimethod and Mixed Methods Research, ed. Sharlene Nagy Hesse-Biber and R. Burke Johnson (Oxford University Press, 2015), 87, https://doi.org/10.1093/oxfordhb/9780199933624.013.6.

22 Ahmed and Sil, “Is Multi-Method Research Really ‘Better’?”

23 Beach and Kaas, “The Great Divides.”

24 Gehlbach, “The Fallacy of Multiple Methods,” 12; See also Samii, “Causal Empiricism,” 950–51.

25 Avenburg, Gerring, and Seawright, “How Do Social Scientists Reach Causal Inferences?”

26 Avenburg, Gerring, and Seawright, “How Do Social Scientists Reach Causal Inferences?,” 5, 11.

27 Peter Marcus Kristensen, “International Relations at the End: A Sociological Autopsy,” International Studies Quarterly, 2018, 248–50, https://doi.org/10.1093/isq/sqy002; Martin Caon, Jamie Trapp, and Clive Baldock, “Topical Debate: Citations Are a Good Way to Determine the Quality of Research,” Physical and Engineering Sciences in Medicine 43, no. 4 (2020): 1146, https://doi.org/10.1007/s13246-020-00941-9.

28 Daniel Maliniak, Ryan Powers, and Barbara F. Walter, “The Gender Citation Gap in International Relations,” International Organization 67, no. 4 (2013): 895–96, https://doi.org/10.1017/S0020818313000209.

29 Daniel Maliniak, Susan Peterson, and Michael J. Tierney, “Journal Article Database Codebook. Version 2.1. Revised: 6/11/2018.,” 2018, https://trip.wm.edu/data/replication-and-other-data/TRIP_Journal%20Article%20Database_Codebook2.1.pdf; Daniel Maliniak et al., “International Relations in the US Academy,” International Studies Quarterly 55, no. 2 (2011): 437–64, https://doi.org/10.1111/j.1468-2478.2011.00653.x; Maliniak, Powers, and Walter, “The Gender Citation Gap in International Relations”; Daniel Maliniak et al., “Is International Relations a Global Discipline? Hegemony, Insularity, and Diversity in the Field,” Security Studies 27, no. 3 (2018): 448–84, https://doi.org/10.1080/09636412.2017.1416824; Jack Hoagland et al., “The Blind Men and the Elephant: Comparing the Study of International Security Across Journals,” Security Studies, 2020, 1–41, https://doi.org/10.1080/09636412.2020.1761439.

30 Gerring, Karcher, and Apfeld, “Impact Metrics,” 375–76.

31 It is important to note the changes in the annual composition of this sample, most notably that Security Studies (SS) is added in 1991 and the European Journal of International Relations (EJIR) in 1995 (their first volumes). International Studies Quarterly (ISQ) has grown from about 25 articles annually up until the early 2000s to about 60 in recent years. For a discussion of the TRIP JAD, see also Anton Peez, “Contributions and Blind Spots of Constructivist Norms Research in International Relations, 1980–2018: A Systematic Evidence and Gap Analysis,” International Studies Review 24, no. 1 (2022): viab055, https://doi.org/10.1093/isr/viab055.

32 In general, these substantive traditions include security studies in IS and SS, international organization in International Organization (IO), peace and conflict studies in the Journal of Conflict Resolution (JCR) and the Journal of Peace Research (JPR), IR theory in EJIR, as well as ISQ and World Politics (WP) as more generalist journals. Methodological traditions include qualitative approaches in International Security (IS), SS, and EJIR, quantitative work in ISQ and JPR, and pluralist approaches in IO and WP. See also Table 3.

33 Stanley Hoffmann, “An American Social Science: International Relations,” Daedalus 106, no. 3 (1977): 41–60.

34 E.g., EJIR. Mathis Lohaus and Wiebke Wemheuer-Vogelaar, “Who Publishes Where? Exploring the Geographic Diversity of Global IR Journals,” International Studies Review, 2020, viaa062, https://doi.org/10.1093/isr/viaa062.

35 E.g., Millennium and Review of International Studies, respectively.

36 Hoffmann, “An American Social Science: International Relations”; Maliniak et al., “Is International Relations a Global Discipline?”; Kristensen, “International Relations at the End,” 247.

37 Daniel Maliniak et al., “TRIP 2017 Faculty Survey (International),” 2017, https://trip.wm.edu/data/our-surveys/faculty-survey/TRIP_2017_International_Faculty_Survey__Topline_Results.pdf.

38 Beach and Kaas, “The Great Divides.”

39 The other key practice is publishing in university presses, Jason C. Sharman and Catherine E. Weaver, “Between the Covers: International Relations in Books,” PS: Political Science & Politics 46, no. 1 (2013): 125, https://doi.org/10.1017/S1049096512001291. Felix S. Bethke and Christian Bueger, “Bursts! Theoretical Fashions in the Study of International Organizations. A Bibliometric Analysis.,” 2014, 10–11, www.dropbox.com/s/2w5j69cjjnjp1ya/Bursts_Theoretical_Fashions_in_the_Study.pdf.

40 Kristensen, “International Relations at the End,” 246.

41 See also ibid., 247 on similar limitations in bibliometric work on IR.

42 Beach and Kaas, “The Great Divides,” 2; emphasis in original.

43 Maliniak, Peterson, and Tierney, “TRIP JAD Codebook V2.1.”

44 Ibid., 17–19.

45 The codebook references two works on “small-N-oriented political science research”; Ibid., 18. In other words, this does not cover ‘counterfactual’ in the sense of the potential outcomes framework for causal inference.

46 Although the codebook states that the “analytic/non-formal conceptual” label is not combined with any of the empirical methods labels, it is on several occasions, see Figure 2; Ibid. It is therefore here used for the “simple” operationalization.

47 Although the codebook states that the “descriptive” label is not combined with any of the empirical methods labels, it is on several occasions, see Figure 2; Ibid.

48 Goertz, “Multimethod Research,” 5.

49 David Collier, Henry E Brady, and Jason Seawright, “Sources of Leverage in Causal Inference: Toward an Alternative View of Methodology,” in Rethinking Social Inquiry: Diverse Tools, Shared Standards, ed. Henry E Brady and David Collier (Rowman and Littlefield, 2004), 229–66; Humphreys and Jacobs, “Mixing Methods,” 653; Very similarly, see Goertz, “Multimethod Research,” 6.

50 Gary Goertz and James Mahoney, A Tale of Two Cultures: Qualitative and Quantitative Research in the Social Sciences, 2013, https://doi.org/10.23943/princeton/9780691149707.001.0001.

51 David Kuehn and Ingo Rohlfing, “Are There Really Two Cultures? A Pilot Study on the Application of Qualitative and Quantitative Methods in Political Science,” European Journal of Political Research 55, no. 4 (2016): 885–905, https://doi.org/10.1111/1475-6765.12159; David Kuehn and Ingo Rohlfing, “Does the Application of Qualitative and Quantitative Methods Reflect Two Distinct Cultures? An Empirical Analysis of 180 Articles Suggests ‘No,’” 2020, 24–28, 31, https://osf.io/6uhpd/.

52 Kuehn and Rohlfing, “Are There Really Two Cultures?”

53 Goertz, “Multimethod Research,” 5–6.

54 Rohlfing and Starke, “Building on Solid Ground.”

55 Avenburg, Gerring, and Seawright, “How Do Social Scientists Reach Causal Inferences?,” 3, 6.

56 Lisa D. Pearce, Thinking Outside the Q Boxes, ed. Sharlene Nagy Hesse-Biber and R. Burke Johnson (Oxford University Press, 2015), 42, https://doi.org/10.1093/oxfordhb/9780199933624.013.4.

57 Seawright illustrates his point with the “suggestive data” of Google Scholar hits relative to OLS, while mentioning other academic trends relating to digitization that might affect the results; 2016, 2–4.

58 I.e., all work that is not any of the other four categories, based on the eight TRIP categories and narrow MMR definition. The high shares of this category for JCR and IS are largely driven by unusually large shares of formal modelling (28.0%) and descriptive work (28.2%), respectively.

59 Articles may be assigned multiple substantive foci. The average number per article is 2.9.

60 This aligns with the expectations in Rohlfing and Starke, “Building on Solid Ground.”

61 Maliniak, Powers, and Walter, “The Gender Citation Gap in International Relations.”

62 Gerring, Karcher, and Apfeld, “Impact Metrics,” 375–76.

63 On “epistemological validity” see Ahmed and Sil, “Is Multi-Method Research Really ‘Better’?”

64 Gerring, Karcher, and Apfeld, “Impact Metrics,” 380.

65 Kristensen, “International Relations at the End,” 248–51; See also Gerring, Karcher, and Apfeld, “Impact Metrics” for a comprehensive discussion and critique of citation counts as an assessment metric.

66 Kristensen, “International Relations at the End,” 249.

67 Ibid., 248, 256.

68 Ibid., 250.

69 Ibid.

70 D. Lindsey, “Using Citation Counts as a Measure of Quality in Science Measuring What’s Measurable Rather than What’s Valid,” Scientometrics 15, no. 3–4 (1989): 195–96, https://doi.org/10.1007/BF02017198.

71 Examples of these three points, in order, are Benedict Anderson’s conceptualization of nations as “imagined communities”; the way Francis Fukuyama’s argument regarding the “end of history” is often cited; and Samuel Huntington’s “Clash of Civilizations.”

72 Gerring, Karcher, and Apfeld, “Impact Metrics,” 396–97.

73 Caon, Trapp, and Baldock, “Citations Are a Good Way to Determine the Quality of Research,” 1146.

74 I.e., a “‘bounded’ entity—a formalized separate research practice,” Sharlene Nagy Hesse-Biber, Introduction: Navigating a Turbulent Research Landscape, ed. Sharlene Nagy Hesse-Biber and R. Burke Johnson (Oxford University Press, 2015), xxxviii, https://doi.org/10.1093/oxfordhb/9780199933624.013.1.

75 Small, “How to Conduct a Mixed Methods Study,” 77–79.

76 Ibid., 79.

77 Maliniak, Powers, and Walter, “The Gender Citation Gap in International Relations”; Iman Tahamtan, Askar Safipour Afshar, and Khadijeh Ahamdzadeh, “Factors Affecting Number of Citations: A Comprehensive Review of the Literature,” Scientometrics 107, no. 3 (2016): 1195–1225, https://doi.org/10.1007/s11192-016-1889-2.

78 Maliniak, Powers, and Walter, “The Gender Citation Gap in International Relations.”

79 Ibid.

80 This is based on the ‘narrow’ methods definition, see above.

81 Bethke and Bueger, “Bursts!”

82 On self-citations and citation groups, see Maliniak, Powers, and Walter, “The Gender Citation Gap in International Relations,” 915–17. On coerced citations, see Caon, Trapp, and Baldock, “Citations Are a Good Way to Determine the Quality of Research,” 1147.

83 Andrew Gelman, Jennifer Hill, and Aki Vehtari, Regression and Other Stories (Cambridge, UK: Cambridge University Press, 2021), 266–68, https://doi.org/10.1017/9781139161879. For IR, see Maliniak, Powers, and Walter, “The Gender Citation Gap in International Relations,” 899; in general, see Mike Thelwall and Paul Wilson, “Regression for Citation Data: An Evaluation of Different Methods,” Journal of Informetrics 8, no. 4 (2014): 963–71, https://doi.org/10.1016/j.joi.2014.09.011.

84 George A. Barnett and Edward L. Fink, “Impact of the Internet and Scholar Age Distribution on Academic Citation Age,” Journal of the American Society for Information Science and Technology 59, no. 4 (2008): 530, https://doi.org/10.1002/asi.20706.

85 Maliniak, Powers, and Walter, “The Gender Citation Gap in International Relations,” 897, 899.

86 Bethke and Bueger, “Bursts!”

87 Maliniak, Powers, and Walter, “The Gender Citation Gap in International Relations,” 905. This amount to 485+ citations for the full TRIP sample.

88 See also Avenburg, Gerring, and Seawright, “How Do Social Scientists Reach Causal Inferences?,” 5. This means removing all articles for which the only coded method is “analytic/non-formal conceptual”, i.e., those articles “without reference to significant empirical evidence or a formal model”, as well as those with the ‘issue area’ ‘Methodology’; Maliniak, Peterson, and Tierney, “TRIP JAD Codebook V2.1.”

89 Gerring, Karcher, and Apfeld, “Impact Metrics,” 384.

90 See also Kristensen, “International Relations at the End,” 257.

91 Maliniak, Powers, and Walter, “The Gender Citation Gap in International Relations.”

92 IO is chosen as the reference category due to its prominence.

93 Note that the effect of an all-female author team is slightly positive, in contrast to Maliniak, Powers, and Walter, “The Gender Citation Gap in International Relations.” Their study’s full model examines 2,541 articles from 1980–2006 with citation data as of 2011 or 2012. The present study examines 9,033 articles from 1980–2018 with citation data as of January 2023. This is potentially in line with the original study’s suggestive finding that the gender citation may have been declining over time at the time of publication. Ibid., 912–14.

94 Models 1–4 in the main analysis displayed below are Models 4a–4d in the Online Appendix.

95 Maliniak, Powers, and Walter, “The Gender Citation Gap in International Relations,” 899.

96 Gelman, Hill, and Vehtari, Regression and Other Stories, 267.

97 Avenburg, Gerring, and Seawright, “How Do Social Scientists Reach Causal Inferences?,” 9–12.

98 See also Maliniak, Powers, and Walter, “The Gender Citation Gap in International Relations,” 904 for a similar figure regarding author gender and tenure status.

99 Paradigm set to ‘realist,’ epistemology to ‘positivist,’ issue area to ‘international security.’ All other variables held at means.

100 See also Maliniak, Powers, and Walter, “The Gender Citation Gap in International Relations,” 904.

101 This covers articles that are coded as addressing international security with a substantive focus or terrorism and/or intrastate conflict; Hoagland et al., “The Blind Men and the Elephant,” 400.

102 Fazal, “An Occult of Irrelevance?”

103 Thelwall and Wilson, “Regression for Citation Data.”

104 Sharman and Weaver, “Between the Covers.”

105 The monograph dataset does not cover author characteristics, so they are not included in the models. Citation data as of 27 February 2023.

106 Goertz, “Multimethod Research.”

107 Avenburg, Gerring, and Seawright, “How Do Social Scientists Reach Causal Inferences?”

108 Lohaus and Wemheuer-Vogelaar, “Who Publishes Where?”

109 Margaret E. Roberts, Brandon M. Stewart, and Richard A. Nielsen, “Adjusting for Confounding with Text Matching,” American Journal of Political Science 64, no. 4 (October 2020): 887–903, https://doi.org/10.1111/ajps.12526.

110 This borrows Seawright’s metaphor; Seawright, Multi-Method Social Science.

111 See Patricia Bazeley, Writing Up Multimethod and Mixed Methods Research for Diverse Audiences, ed. Sharlene Nagy Hesse-Biber and R. Burke Johnson (Oxford University Press, 2015), https://doi.org/10.1093/oxfordhb/9780199933624.013.20.

112 I thank Reviewer 1 for pointing this out.

113 I thank Caroline Fehl for stressing this point.

114 Samii, “Causal Empiricism,” 950–51; See also Gerring, Karcher, and Apfeld, “Impact Metrics,” 389.

115 Elman, Gerring, and Mahoney, The Production of Knowledge; Avenburg, Gerring, and Seawright, “How Do Social Scientists Reach Causal Inferences?”

116 Thad Dunning et al., eds., Information, Accountability, and Cumulative Learning: Lessons from Metaketa I (Cambridge University Press, 2019), https://doi.org/10.1017/9781108381390.

Additional information

Notes on contributors

Anton Peez

Anton Peez is a Postdoctoral Researcher at the University of Frankfurt and an Associate Fellow at Peace Research Institute Frankfurt (PRIF).