Comparing Freedom House Democracy Scores to Alternative Indices and Testing for Political Bias: Are US Allies Rated as More Democratic by Freedom House?

Abstract

Several scholars have criticized the Freedom House democracy ratings as being politically biased. Do countries indeed incorrectly receive better ratings that have stronger political ties with the United States? This article tests whether differences between a number of alternative indices of democracy and the FH ratings can be explained in a systematic manner by variables that record relationships between the US and the countries under investigation. Differentiating between the periods before 1988 and after 1989, strong and consistent evidence of a substantial bias in the FH ratings is obtained for the former period. For the latter period, the estimates are less consistent, but still hint at the presence of a political bias in the FH scores.

Keywords:

• democracy index
• Freedom House
• measurement
• political bias
• bilateral relations
• validity
• democracy

Acknowledgments

I thank Simon Bauer, Felicitas Belok, Sara Ceyhan, Lara Redmer, Benjamin Sack and Thomas Speth for excellent research assistance. Julian Bergmann, Tero Erkkilä, Christian Martin, Marlene Mauk, Steffen Mohrenberg, Thomas Pfister, Ossi Piironen, David Redlawsk, Edeltraud Roller, Iñaki Sagarzazu, Detlef Sack and David Siroky, as well as three anonymous reviewers provided helpful comments on earlier drafts. Previous versions of this article were presented at the EPSA conference in Dublin, 2011, at the ECPR General Conference in Reykjavik, 2011, and in seminars at the University of Kiel and the Johannes Gutenberg-University of Mainz. On these occasions, I received constructive feedback from further audience members. Ken Bollen and Daniel Pemstein kindly responded to requests and sent replication material.

Notes

^1. The online appendix with additional information as well as replication files are available from the author’s webpage at URL: http://comparative.politics.uni-mainz.de/staff/nils-steiner.

^2. Despite the conceptual difference, there is a very high correlation between the two scales, as noted by other authors (e.g. Banks 1986: 658–659) and it is precisely the Civil Liberties index of FH that loads most heavily (and even slightly higher than the Political Rights scale) on the contestation factor extracted in the principal components analyses of Michael Coppedge et al. (2008). Coppedge et al. (2008: 645) conclude that both scales are “indicators of contestation” based on a “conceptual distinction without an empirical difference” and, therefore, should not be used as “indicators of distinct aspects of democracy”. In light of the conceptual differences between these two scales, these results are somewhat surprising. In consequence, one should not expect that it makes a huge difference whether the combined score or the Political Rights score only are used in the analysis of bias.

^3. As this article aims to analyze whether FH favors countries that are affiliated to the US relative to other indices that basically share this understanding of democracy (more on that below in the discussion of the benchmark scores), it sidesteps the question whether some sort of bias is inherent in these shared understandings of democracy as procedural, representative and liberal democracy. The question whether some common “Western” bias is inherent in these concepts of democracy is beyond the scope of this contribution.

^4. The first survey coordinator after Gastil R. Bruce McColm (Freedom in the World 1989/90–1992/93) also served as executive director of Freedom House. Adrian Karatnycky, listed as survey coordinator in the reports for 1993/94–1999/2000 and as editor until Freedom in the World 2003, first served as executive director and then from 1996 to 2003 as president of Freedom House.

^5. This assumption might be questioned on the grounds of the suggestion that the FH scores are a more valid measure of democracy than alternative indices, and that the US tends to hold more favorable relations with more democratic countries. Any remaining correlation between the residuals and the proxies for US affinity could then reflect this pattern. While this possibility cannot be dismissed out of hand, it seems a very implausible explanation for a systematic pattern of biased deviations of the FH scores from a number of other democracy scores or latent measures of democracy as detected with the help of a number of different proxy variables (some of which can hardly be argued to be strongly endogenous to the “true” level of democracy).

^6. The next three benchmark measures also have the advantage of more fine-grained measurement scales than the BLM and MBP measures which allow for only three types of regimes. The possibility that the results presented so far are a consequence of the FH measure being more fine-grained than the benchmark data cannot be entirely dismissed. The residuals might then still reflect some real differences in democracy which might be correlated with the bilateral relations measures.

^7. This plot uses only observations for which the Polity rating is below 9 for reasons explained below.

^8. I excluded the observation for Panama in 1989 which is a huge outlier from models 4 to 6. The substantial discrepancies seem to be driven by the assessments focusing on slightly different time points within the year during a time of rapid and massive democratization.

^9. It might be the case that alliance similarity is, at least in a global sample, less successful in validly discriminating between countries in terms of affinity of countries to the US in this period as compared to the Cold War era. Note that (for the year 2000) the most dissimilar countries in terms of alliance structure include Kuwait, Saudia Arabia, Bahrain, Quatar and the United Arab Emirates. Some of these countries could be considered strategic allies of the US notwithstanding this similarity measure indicating the opposite. Nonetheless, I refrained from an ex post exclusion of this predictor.

^10. Note that due to availability restrictions on other variables the data for the later period are mostly limited to the period 1989 to 2000, as mentioned above. Extrapolations of the findings to the most recent Freedom in the World data should, if at all, only be made with all the appropriate caution.

Additional information

Notes on contributors

Nils D. Steiner

Nils D. Steiner has been a PhD student at the Department of Political Science, Johannes Gutenberg University of Mainz, Germany, since 2009. His PhD project analyzes how economic interconnectedness affects the democratic process in developed democracies. Methodologically, his research applies quantitative techniques to substantial questions in Comparative Politics. Steiner’s work has been published in Electoral Studies, The Journal of European Social Policy and West European Politics.