Complex Politics in Single Numbers? The Problem of Defining and Measuring Democracy

Abstract

Whether the Arab world has become more democratic since the Arab Spring is a heated debate. References in these discussions often are made to comparative democracy indices that are considered either with appreciation or skepticism. In this article, the regional coordinator for Middle East and North Africa of the Bertelsmann Transformation Index (BTI) discusses some insights from the assessment process of the most recent data for the BTI's 2014 edition. It shows that the revolutionary changes since 2011 have brought major challenges to those who want to measure democracy. Case studies from certain Arab countries illustrate practical problems and methodological challenges that resulted from the dramatic changes for the BTI 2014 assessment. However, it concludes that the BTI weathered the storms convincingly well, and that the 2014 findings are a useful tool for a deeper understanding of the region's current political dynamics.

Keywords::

• Arab Spring
• Bertelsmann Transformation Index (BTI)
• democratization
• indices
• MENA countries
• methodology
• ranking
• transformation

Acknowledgements

The author would like to thank Chantal Lavallée, Hauke Hartmann and two anonymous reviewers for valuable inputs and comments.

Notes

¹ M. Foucault (1977) Discipline and Punish (New York: Pantheon); originally published in French as Surveiller et punir (Paris: Gallimard, 1975).

² C. H. Slager (2012) SRI Indices and Responsible Corporate Behaviour: A Study of the FTSE4GOOD Index, p. 129 (Nottingham, UK: University of Nottingham).

³ H. Tzameret-Kertcher (2013) The Gender Index: Gender Inequality in Israel 2013, p. 9 (Jerusalem: Van Leer Jerusalem Institute). Some scholars even argue that quantitative indices through their ‘numeric difference,’ meaning ‘rationalization,’ would help to de-dramatize massive rights abuses by regimes using political violence; see D. Sack and J.-N. Wirsbinna (2014) Menschenrechtsverletzungen im Vergleich. Zur Kommunikation über politische Gewalt durch numerische Differenz [Human rights violations in comparison: on the communication of political violence by numeric difference], in: F. Martinsen & O. Flügel-Martinsen (eds) Gewaltbefragungen. Beiträge zur Theorie von Politik und Gewalt [Inquiring violence. Contributions to the theory of politics and violence], pp. 117–142 (Bielefeld: Transcript, 2014).

⁴ See http://www.bti-project.org.

⁵ M. Coppedge & J. Gerring (2011) Conceptualizing and Measuring Democracy: A New Approach, in Perspectives on Politics, 9, 2, p. 253; see also J. C. Isaac (2011) Analyzing Democracy, in Perspectives on Politics, 9, 2, p. 242; and E. McMahon & E. Kornheiser (2010) Assessing the Assessors: Correlating Democracy Methodologies, in Social Indicators Research, 97, 2, p. 269.

⁶ The opinions and views expressed in this article are the opinions of the author and do not necessarily reflect the opinions or views of the Bertelsmann Foundation.

⁷ J. Møller & S.-E. Skaaning (2010) Beyond the Radial Delusion: Conceptualizing and Measuring Democracy and Non-democracy, International Political Science Review, 31, 3, p. 264; see S. Ziaja (2012) What do fragility indices measure? Assessing measurement procedures and statistical proximity, in Zeitschrift für Vergleichende Politikwissenschaft [Journal of Comparative Politics], 6, 1, p. 59.

⁸ For the vivid debate about democracy indices and their differing methodologies, see, for example, C. R. Apaza (2009) Measuring Governance and Corruption through the Worldwide Governance Indicators: Critiques, Responses, and Ongoing Scholarly Discussion, in Political Science and Politics, 42(1), pp. 139–143; R. Bandura (2008) A Survey of Composite Indices Measuring Country Performance: 2008 Update (New York: UNDP Office for Development Studies); M. Brusis (2005) Governance Ratings as useful tools to measure democratic performance in Central and Eastern Europe? Paper presented at the ECPR Joint Sessions, Granada, Spain, available at http://epub.ub.uni-muenchen.de/13107/1/Brusis_13107.pdf, accessed June 6, 2014; C. Bui (2010) Mass Attitude Toward Democracy in Different Types of Regime: Testing an Assumption of Democratic Qualities, Paper presented at the APSA Annual Meeting, Washington D.C., available at http://papers.ssrn.com/sol3/papers.cfm?abstract_id = 1663881, accessed June 6, 2014; H. Giebler (2012) Bringing methodology (back) in: Some remarks on contemporary democracy measurements, European Political Science, 11(4), pp. 509–518; J. Grävingholt, S. Ziaja & M. Kreibaum (2012) State Fragility: Towards a Multi-Dimensional Empirical Typology, DIE discussion paper 3 (Bonn: German Development Institute); F. Gutiérrez (2011) Measuring poor state performance: Problems, perspectives and paths ahead (London School of Economics and Political Science: Crises States Research Centre); M. J. Kurtz & A. Schrank (2007) Growth and Governance: Models, Measures, and Mechanisms, The Journal of Politics, 69(2), pp. 538–554; D. Kaufmann, A. Kraay & M. Mastruzzi (2007) Growth and Governance: A Reply, The Journal of Politics, 69(2), pp. 555–562; J. Møller & S.-E. Skaaning (2012) Concept-measure inconsistency in contemporary studies of democracy, Zeitschrift für Vergleichende Politikwissenschaft, 6(1), pp. 233–251; W. Muno (2012) Die Vermessung der Welt: Eine Analyse der Worldwide Governance Indicators der Weltbank [Measuring the world: An analysis of the World Bank's worldwide governance indicators], Zeitschrift für Vergleichende Politikwissenschaft, 6(1), pp. 87–113; S.-E. Skaaning (2010) Measuring the Rule of Law, Political Research Quarterly, 63(2), pp. 449–460; and S. Werthes, C. Heaven & S. Vollnhals (2011) Assessing Human Insecurity Worldwide: The Way to A Human (In)Security Index, INEF Report 102 (Duisburg: Institute for Development and Peace). Specifically with respect to the BTI, see T. Müller & S. Pickel (2007) Wie lässt sich Demokratie am besten messen? Zur Konzeptqualität von Demokratie-Indices [How can democracy best be measured? On the concept quality of democracy indices], Politische Vierteljahresschrift [Political Quarterly], 48(3), pp. 511–539; and A. Rudnik (2009) Privatisierung des Policy-Making? Die Bertelsmann-Stiftung als dominanter deutscher Think Tank [Privatization of policy-making? The Bertelsmann Foundation as dominant German think tank], in M. Heindl (ed.) Privatisierung und öffentliche Verwaltung [Privatization and Public Administration] (Bamberg: Otto Friedrich University), pp. 218–237.

⁹ See the detailed description of the BTI methodology at http://www.bti-project.org/index/methodology/, accessed December 3, 2014.

¹⁰ ‘Democracy’, or better, ‘Democratization,’ and ‘Political Transformation’ are not full synonyms, as the latter also includes questions concerning the ‘stateness’ of a country, namely the question of how elaborated and well-functioning the state is (monopoly on the use of power, outreach of basic administration, etc.). The BTI includes these indicators in its Political Transformation Status Index and hence goes beyond the simple question: How ‘democratic’ is a country?

¹¹ A. Garcia-Schmidt & H. Hartmann (2012) Taking stock of the Arab Spring, Europe in Dialogue, 2, p. 9.

¹² The BTI regions are: Asia and Oceania, East-Central and Southeast Europe, Latin America and the Caribbean, Middle East and North Africa, Post-Soviet Eurasia, South and East Africa, and West and Central Africa.

¹³ Bertelsmann Stiftung (2012a) Transformation Index BTI 2012: Political Management in International Comparison, p. 133 (Gütersloh: Bertelsmann Foundation).

¹⁴ Algeria, Bahrain, Egypt, Iran, Iraq, Jordan, Kuwait, Lebanon, Libya, Morocco, Oman, Qatar, Saudi Arabia, Sudan, Syria, Tunisia, Turkey, UAE, Yemen. Excluded are Israel, which is part of the Bertelsmann Foundation's Sustainable Governance Indicators (see footnote 18), and the Palestinian Territories, which are perceived as not a sovereign state. South Sudan has been added to the BTI South and East Africa region.

¹⁵ Muno, Die Vermessung, p. 93.

¹⁶ M. Bühlmann, W. Merkel, L. Müller & B. Weßels (2008) Wie lässt sich Demokratie am besten messen? Zum Forumsbeitrag von Thomas Müller und Susanne Pickel [How can democracy best be measured? A reply to the forum contribution of Thomas Müller and Susanne Pickel], Politische Vierteljahresschrift, 49(1), p. 117.

¹⁷ See Møller & Skaaning, Concept-measure inconsistency, p. 244.

¹⁸ To cover the political and economic state in consolidated democracies, the Bertelsmann Foundation set-up in 2009 the Sustainable Governance Indicator (SGI), comprising all 31 OECD member states. Here, the questions do not predominantly ask whether democracy is realized and to what extent, but rather how ‘robust’ democracies are; see the SGI Website at http://www.sgi-network.org, accessed December 3, 2014.

¹⁹ R. A. Dahl (1971) Polyarchy. Participation and Opposition (New Haven & London: Yale University Press).

²⁰ W. Merkel (2004) Embedded and Defective Democracies, Democratization, 11(5), pp. 38–42. Merkel identifies five partial regimes that mutually enforce themselves to ‘full democracy’: (a) elections, (b) political liberties, (c) civil rights, (d) horizontal accountability and (e) effective power to govern of the elected rulers; see W. Merkel, A. Croissant & H.-J. Puhle (2003) Defekte Demokratien [Defective democracies] (Baden-Baden: Nomos).

²¹ The mathematic formula is 1/5(s1+s2+s3+s4+s5), with s being the average score of each criterion. This means that the two indicators of criterion 4 (4.1 and 4.2) have double weight compared with the four indicators within the other criteria. Critics hint at an ‘aggregation problem’ here, as it is a problematic decision regarding ‘which indicators to combine into a single index, whether to add or multiply them, and how much to weigh them. It goes without saying that different solutions to the aggregation problem lead to quite different results’ (Coppedge & Gerring, Conceptualizing, p. 250). To minimize this problem, the BTI treats each indicator strictly equal, and aggregates its scores arithmetically, not geometrically; see Bertelsmann Stiftung (ed.) (2008) Bertelsmann Transformation Index 2008: Political Management in International Comparison, p. 82 (Gütersloh: Bertelsmann Foundation).

²² As Coppedge & Gerring, Conceptualizing, p. 249 phrase it, ‘the precision or reliability of all indices is too low to justify confidence that a country with a score a few points higher is actually more democratic.’ Accordingly, score changes of less than, say, ± 0.10 points must be seen as irrelevant.

²³ J. C. Völkel (2014) Regional Survey Middle East and North Africa, in: Bertelsmann Stiftung (ed.), Bertelsmann Transformation Index 2014 (Gütersloh: Bertelsmann Foundation).

²⁴ Garcia-Schmidt & Hartmann, Taking stock of the Arab Spring, p. 8.

²⁵ J. C. Völkel (2012) The BTI 2012: Looking Back on the Arab Spring: An Interpretation of Recent Political Developments, in Europe in Dialogue, 2, p. 16.

²⁶ Skaaning, Measuring the Rule of Law, p. 454; see Coppedge & Gerring, Conceptualizing, p. 250.

²⁷ The complete revision of scores also can be an advantage in cases where previous grades reflected a too positive or too negative assessment. Then a trade-off between keeping scores consistent over time and assigning the more appropriate score must be made; in most of these few cases, the BTI prefers the second option.

²⁸ Figure 1 is reprinted with the kind permission of the Bertelsmann Foundation.

²⁹ Bertelsmann Stiftung, Transformation Index BTI 2012, p. 136.

³⁰ The differentiation between ‘national’ and ‘international’ expert is sometimes misleading, as often ‘national’ experts also work in an international environment outside their home country. Expatriates working in their country of evaluation, meanwhile, are counted as ‘internationals.’

³² See criticism of the BTI calibration process in Giebler, Bringing methodology, p. 511.

³³ The board members are listed at http://www.bti-project.org/about/board/index.nc, accessed December 3, 2014.

³⁴ The lowest possible score for a democracy would be 1.8, if a country fulfilled in each core criteria the minimum requirement and otherwise received always a score 1. In reality, however, this possibility can be excluded.

³⁵ Numbers in brackets indicate the criteria where the country does not qualify for democracy.

³⁶  ↑ ↓  indicate changes by one category, and ⇑ ⇓  by two categories, compared with the BTI 2012.

³⁷ J. Motion, S. Davenport, S. Leitch & E. Merlot (2011) Translating productivity: The role of concepts. Presentation to EGOS colloquium, Gothenburg, 7–9 July; Available at https://www.victoria.ac.nz/som/researchprojects/research-projects/buildingourproductivity/files/EGOS_2011_Motion_et_al_Translating_productivity_June_1_2011.pdf, accessed June 6, 2014; see also M. Foucault (1991) Politics and the Study of Discourse, in G. Burchel, C. Gordon and P. Miller (eds) The Foucault Effect: Studies in Governmentality (Chicago: University of Chicago Press), p. 56.

³⁸ It will be interesting to see how the country author and reviewer will evaluate Algeria's April 2014 presidential elections during the upcoming BTI 2016 assessment.

³⁹ Bertelsmann Stiftung (2012b) BTI 2014: Codebook for Country Assessments, p. 18 (Gütersloh: Bertelsmann Foundation).

⁴⁰ Excerpts taken from the BTI 2016 manual for authors and reviewers.

⁴¹ Bühlmann et al., Wie lässt sich, p. 115f.

⁴² Bertelsmann Stiftung, Bertelsmann Transformation Index 2008, p. 69.

⁴³ See the remarks on the ‘aggregation problem’ in footnote 21.

⁴⁴ Indicator 14.2 and 16.4 are part of the BTI Management Index.

⁴⁵ Völkel, Regional Survey Middle East and North Africa.