Method Bias in Comparative Research: Problems of Construct Validity as Exemplified by the Measurement of Ethnic Diversity

Abstract

This study investigates indices of ethnic diversity for method effects due to differences in operationalization. It adapts the methodology of multitrait–multimethod analysis to the field of socioeconomic macro research. While approaches for checking construct validity or method bias are common in psychology and educational research, they are rarely applied in other social science disciplines such as economics and political science. We find that measures of polarization show considerable method effects which call their empirical utilization for multivariate modeling into question. We conclude that improved overall measures, which are based on a theoretical augmentation of a construct's measurement instrument, do not necessarily lead to improved results, neither on the measurement level nor for hypothesis testing.

Keywords:

• ethnic diversity
• index operationalization
• multitrait–multimethod
• multivariate modeling
• validity

Notes

¹To be clear about the objectives of the article, we emphasize that the article does not focus on the relationship of ethnic diversity and conflict. Conflict research represents the origin of two of the instruments we will disentangle, and there is a lively discussion about the validity of different measurement instruments of ethnic diversity (see Esteban & Schneider, 2008, and the subsequent articles in this issue, Journal of Peace Research 45[2]). Still, in our article the examples from conflict research serve mainly illustrative purposes.

²Observable aspects may be the spoken language or origin of birth on the micro level or represent subsequent facets of the social structure like number of groups and relative size of ethnic groups on the macro level.

³If such indices are used as independent variables in multivariate designs, this could come at the expense that it is no longer clear which content is actually represented by the indicators. Under such conditions, results derived by the indices are more blurred the larger the method variance turns out to be.

⁴Podsakoff et al. (2003, p. 882) list four groups of potential sources of method bias for micro-level studies that include rather common effects, item characteristics, and item context issues, as well as measurement context effects. Examples are for instance framing effects of questions (e.g., Gamliel & Peer, 2006) or the potential of possible social desirability for certain answers (e.g., Ganster, Hennessey, & Luthans, 1983).

⁵Potential method bias in macro research might occur particularly on two stages of index building: when data are being operationalized and aggregated. The first step usually constitutes a precondition for the second step (Adcock & Collier, 2001) whereas both aspects may or may not rely on assembled micro-level data which are prone to the common method biases listed by Podsakoff et al. (2003).

⁶As summarized by Wimmer, Cedermann, and Min (2009), the “greed-and-opportunity perspective,” “the minority-mobilization school,” “the diversity breeds conflict perspective,” and Wimmer's (2008) own “institutionalist configurational perspective” all compete for the explanatory recipe of how ethnic diversity and “ethnic boundary making” may translate into conflict or war. For useful overviews on the topic of war and civil conflict see Collier and Hoeffler (2007) and Blattmann and Miguel (2010).

⁷Over the years, some of these critical points have been acknowledged but the same mathematical formula for calculating the ELF still applies (Alesina et al., 2003; Fearon & Laitin, 2003).

⁸The present study will apply the same threshold values of k = 5 and r = 0.5 in accordance with Cederman and Girardin (2007, p. 176): k is a slope parameter, and r represents a threshold value.

⁹We restrict our analyses to three measures of ethnic diversity. Numerous other instruments exist that may grasp the meaning of ethnic diversity. For our analysis, some indices like the index of Ethnic Dominance (Collier & Hoeffler, 2004) appear ill-suited due to its dichotomous scale, and others simply lack the appropriate amount of country specific data.

¹⁰They classify ethnic groups according to their access to state power into Included Groups (Monopoly/Dominant), Senior Partner, Junior Partner, Excluded Groups, Regional Autnonomy, Powerless, Discriminated, and Separatist Autonomy (Cederman, Wimmer, & Min, 2010, p. 104). The groups enter their calculations as dummy variables.

¹¹All symbols are denoted in accordance to the program package LISREL 8.51 (Jöreskog & Soerbom, 1993).

¹²In the vast literature on MTMM analyses it is common strategy to allow noncorrelation among latent factors as well as correlated error terms based on theoretical and modeling purposes (see Widaman, 1985).

¹³As Neumann and Graeff (2010) have pointed out, there might be a trade-off between both types of validities due to the dimensionality of the instruments when aggregate measures are compiled. Discriminant validity will rather be sacrificed to achieve higher level of convergence toward latent phenomena such as corruption. In the present case, all three indices differ only in their mathematical operationalization as they are based on the same data set and, therefore, are not aggregate measures from different data sources.

¹⁴Descriptive statistics are shown in the Appendix.

***denotes significance at the 1% level (2-tailed).

¹⁵This represents of course a quite indirect consequence of “ethnic practice.” Especially for the area of Geography we even would expect that the latent trait factors will not reflect the variation between these rather exogenous variables and the ethnicity measures. Note that we do not claim that geography and ethnicity are not interrelated (see, e.g., Weidmann, 2009), the argument is restricted to the variables we use in our study.

¹⁶Due to the a priori restriction of τ equivalence among toward the trait factor of ethnicity, this theoretically valid restriction implies an invariance condition that in turn constrains the values of the chi-square statistic to remain constant across models as well. Even if small Chi-square values become difficult to judge, inflated chi-square values would have indicated the equivalence restriction as unjustified (see Bollen, 1989, pp. 263–267, for details).

¹⁷All estimation procedures converge to a solution. When we do not assume τ equivalence, we do gain nonconverging estimations and nonidentified solutions.

***denotes significance at the 1% level (2-tailed).

Note. MTMM = multitrait–multimethod; NFI = normed fit index; CFI = comparative fit index.

^a Coefficients represent the average magnitude of the standardized factor loadings. root mean square error of approximation = 0, χ² = 28.928, df = 38 for all models, see fn 5. Full results displayed in the Appendix.

¹⁸For example, this strategy would imply that one decides to analyze not all violent conflicts of the past 50 years, but, for example, only secessionist conflicts. This strategy is reflected in the development of the aforementioned schools of thought on causes of war and conflict (e.g., Sambanis, 2004).

Note. MTMM = multitrait–multimethod.

Note. MTMM = multitrait–multimethod.

*^,**^, and *** denote significance at the 10%, 5% and 1% level (2-tailed).

Note. MTMM = multitrait–multimethod.