Where to draw the line? From degree to dichotomy in measures of democracy

Abstract

While there is much debate about the merits of dichotomous versus continuous measures of democracy, surprisingly little attention is paid to the question as to how to go from degree to dichotomy. This study identifies no less than 38 different ways in which Freedom House and Polity scores have been used to distinguish between democracies and non-democracies. The analysis shows that it is difficult to draw the line in measures of democracy, even for Freedom House and Polity themselves. These problems are illustrated with the help of a recent study on democratization in Africa. The conclusion formulates some guidelines for good practice and points at the potential of disaggregated scores to distinguish between democracy and dictatorship.

Keywords:

• democracy
• measurement
• Freedom House
• Polity
• dictatorship

Acknowledgements

The author would like to express his thanks to Carsten Schneider, and especially Franziska Deutsch for their helpful comments.

Notes

See, for example, Sartori, Theory of Democracy Revisited and Przeworski et al., Democracy and Development, versus Elkins, ‘Gradations of Democracy?’ and Inglehart and Welzel, Modernization.

Collier and Adcock, ‘Democracy and Dichotomies’.

Bennett, ‘Toward a Continuous Specification’, 319.

See Bogaards, ‘Measures of Democratization’.

Dahl, Polyarchy.

See, for example, McHenry and Mady, ‘Critique of Quantitative Measures’; Harik, ‘Democracy’; Treier and Jackman, ‘Democracy as a Latent Variable’; Giannone, ‘Political and Ideological Aspects’.

Berg-Schlosser, ‘Determinants of Democratic Successes’.

See http://www.freedomhouse.org/template.cfm?page=15. The first year covered is 1972.

See Giannone, ‘Political and Ideological Aspects’.

Munck and Verkuilen, ‘Conceptualization and Measuring Democracy’, 9.

See http://www.freedomhouse.org/template.cfm?page=351&ana_page=363&year=2010.

The first scores for subcategories are thus available for the year 2006.

Goertz, Social Science Concepts.

Hiroi and Omori, ‘Perils of Parliamentarism?’; Elgie, ‘Semi-Presidentialism’.

Doorenspleet, ‘Electoral Systems and Good Governance’; Kuenzi and Lambright, ‘Voter Turnout’.

See, for example, Kegley and Hermann, ‘Putting Military Intervention’; Anckar, ‘Democratic Standard and Performance’; Andersson, ‘United Nations Intervention’; Karvonen, ‘Preferential Voting’; Tavits, ‘Size of Government’.

In addition, there are studies for which it is not clear how exactly Freedom House scores were used to derive categories, such as Wibbels, ‘Federalism’; Drury, Krieckhaus, and Lusztig, ‘Corruption’; and Tsai, ‘Macro-Structural Determinants’.

The apparent discrepancy between Fredericksson and Wollscheid and their source of inspiration, Persson and Tabellini in Table 1, is caused by a change in the Freedom House boundary between not free and partly free states (from 5 to 5.5). Both studies operationalize democracy as all free and partly free countries.

Persson and Tabellini, Economic Effects of Constitutions, 75.

Hadenius and Teorell, ‘Pathways from Authoritarianism’ and Teorell, Determinants of Democratization, use the mean of a country's Freedom House and Polity scores, converted to a scale of 0 (least democratic) to 10 (most democratic), and classify as authoritarian all countries with a score lower than or equal to 7.5. This threshold is chosen by ‘estimating the mean cutoff point separating democracy from autocracy in five well-known categorical measures of democracy’ (Hadenius and Teorell, ‘Pathways from Authoritarianism’, 146), including Freedom House and Polity categories. It is not clear how this inductive procedure will result in a more valid and reliable determination of regime type.

Howard and Roessler, ‘Liberalizing Electoral Outcomes’.

Ibid., 368.

The same goes for scholars who use the Freedom House categories of free, partly free and not free, but draw the boundaries differently from Freedom House itself, as do for example Rakner and van de Walle, ‘Opposition Parties’, 208.

See http://www.cidcm.umd.edu/polity.

See http://www.systemicpeace.org/inscr/p4manualv2009.pdf, 15.

Jaggers and Gurr, ‘Tracking Democracy's Third Wave’, 474.

Kurzman and Leahey, ‘Intellectuals and Democratization’, 945.

Mainwaring and Zoco, ‘Political Sequences’, 158.

Bogaards, ‘Electoral Volatility’.

Reiter, ‘Does Peace Nurture Democracy’, 940.

Peceny, ‘Forcing Them to be Free’, 558.

Gleditsch and Ward, ‘Double Take’.

Gleditsch and Hegre, ‘Peace and Democracy’.

Peceny, Beer, and Sanchez-Terry, ‘Dictatorial Peace?’, 20; Jaggers and Gurr, ‘Tracking Democracy's Third Wave’.

Li, ‘Does Democracy Promote or Reduce’, 285; Dixon, ‘Democracy and the Peaceful Settlement’.

Lai and Melkonian-Hoover, ‘Democratic Progress’, 554; Pevehouse, ‘With a Little Help’.

Berg-Schlosser, ‘Determinants of Democratic Successes’.

The author thanks Berg-Schlosser for sharing his data.

Huntington, Political Order in Changing Societies.

Berg-Schlosser, ‘Determinants of Democratic Successes’, 271.

Ibid., 272. The reference is to the now defunct PIOOM project, which however took the data from the Political Terror Scale. See http://www.politicalterrorscale.org/.

Berg-Schlosser, ‘Determinants of Democratic Successes’, 272. Neither does he do so in a previous publication, where he presents the same overview of measures of democracy and the same classification for Africa. Berg-Schlosser, ‘Indicators of Democracy’.

Thus, Mozambique, with a Freedom House combined average of 3.5 and a Gross Human Rights Violations score of 3 is classified as authoritarian. Berg-Schlosser, ‘Determinants of Democratic Successes’, 274.

Equatorial Guinea and Sudan are both given the lowest possible score by Freedom House. The first is classified as authoritarian by Berg-Schlosser and the second as praetorian, based on their differential scores on Gross Human Rights Violations (3 versus 5). Berg-Schlosser, ‘Determinants of Democratic Successes’, 273/275.

In a previous publication, Cameroon is classified as praetorian and therefore not an outlier. Cameroon is the only country for which the classification differs between the two publications and it is not clear why, as the values of all variables are identical. Berg-Schlosser, ‘Indicators of Democracy’.

In general, Berg-Schlosser, in ‘Determinants of Democratic Successes’, closely follows his PhD student Temelli in the conceptualization and measurement of the dependent variable, the selection and coding of the independent variables, and the combination of statistical analysis with qualitative comparative analysis. Temelli, Demokratisierung im Subsaharischen Afrika. For a discussion of Temelli's methodology and classification of African regimes, see Tetzlaff and Jakobeit, Das Nachkoloniale Afrika, 136–8.

The strongest correlations are between polyarchy and the five World Bank good governance indicators. However, there are at least two reasons that caution against their use as independent variables. First, governance data are from 2002, whereas democracy is measured for 2000, which logically excludes them as a cause. Second, governance indicators are highly correlated among each other. In Berg-Schlosser’s data set, regulatory quality, treated as an independent variable, has a Pearson correlation coefficient of 0.749 (significant at the highest level) with voice and accountability, listed as part of the dependent variable. The variable ‘losers accept’ has the third highest correlation coefficient with polyarchy, after the governance indicators and press freedom and has the second highest correlation coefficient in the ‘best model’, after regulatory quality. Berg-Schlosser, ‘Determinants of Democratic Successes’, 278–81/290.

These variables are borrowed from Bratton and van de Walle, Democratic Experiments in Africa. Losers are considered to have accepted the election results when they take up their seats in parliament.

cf. Maeda and Nishikawa, ‘Duration of Party Control’.

The set of cases coded by Polity is smaller because it excludes the smallest countries and does not provide a democracy score for countries in transition or turmoil.

Bratton and van de Walle, Democratic Experiments in Africa.

Ibid., 194.

Dahl, Polyarchy.

Berg-Schlosser, ‘Determinants of Democratic Successes’, ‘Indicators of Democracy’. In an earlier analysis of African regimes, Berg-Schlosser did not use any publicly available measure of democracy but employed two coders to score regimes on selected variables. Berg-Schlosser, ‘African Political Systems’.

For example, Brown, ‘Democracy and Social Spending’, uses Polity's democracy score to distinguish between democracies and non-democracies. He concludes in a robustness check that ‘varying the dividing point between 1 and 9 [on an eleven-point scale] does not significantly alter the results’ (691).

Goertz, Social Science Concepts.

Ibid., 114–5. Vanhanen, Democratization.

Przeworski et al., Democracy and Development.

Bogaards, ‘Elections, Election Outcomes’, ‘Measuring Democracy’.

Bueno de Mesquita, Downs, and Smith, ‘Thinking Inside the Box’, create dummy variables for four Polity component variables coding the highest value as 1 and all others as zero. This is done to facilitate a disaggregated analysis of democracy's relationship with human rights and is not intended as an alternative operationalization of democracy.

Doorenspleet, ‘Reassessing the Three Waves’, Exploring the Structural Sources.

With the exception of competitiveness of participation, where the absence of regulation is deemed sufficient.

Goertz, Social Science Concepts, 115.

Doorenspleet, Exploring the Structural Sources, 30. cf. Paxton et al., ‘Half-Century of Suffrage’; Moon et al., ‘Voting Counts’.

Ulfelder, ‘Natural Resource Wealth’.

If XRCOMP = 3 and XROPEN = 3, Doorenspleet classifies the regime as democratic, whereas Ulfelder classifies the regime as non-democratic. This is because Ulfelder does not use the individual Polity criteria directly, but uses the Polity concept variable EXREC (Executive Recruitment), whose values reflect particular combinations of values of the three contributing component variables (XRREG, XRCOMP, and XROPEN). For ease of presentation and comparison, Table 3 only presents information on the underlying variables and their minimal values, which is why this discrepancy cannot be observed there.

Mansfield and Snyder, Electing to Fight; Vreeland, ‘Effect of Political Regime’.

Andrews and Montinola, ‘Emerging Democracies’, do the same. They include in their analysis of emerging democracies only countries which select their leaders (that is, have a XRCOMP score of 3).

For an overview and analysis of the various ways in which scholars have used Polity and Freedom House scores to derive regime typologies, see Bogaards, ‘Measures of Democratization’.

See http://www.systemicpeace.org/inscr/p4manualv2009.pdf.