Measuring Populism: Comparing Two Methods of Content Analysis

Abstract

The measurement of populism – particularly over time and space – has received only scarce attention. In this research note two different ways to measure populism are compared: a classical content analysis and a computer-based content analysis. An analysis of political parties in the United Kingdom, the Netherlands, Germany and Italy demonstrates that both methods can be used to measure populism across countries and over time. Recommendations are presented on how to combine these methods in future comparative research on populism.

Acknowledgements

Earlier versions of this research note have been presented at various locations, including the ECPR Joint Sessions in Münster and the Comparative Politics PhD Club at the University of Amsterdam. We want to thank all participants for their valuable feedback. In particular we thank Wouter van der Brug, Kris Deschouwer, Kirk Hawkins, Liesbet Hooghe, Sarah de Lange and Bojan Todosijevic. Finally we want to thank the reviewers of West European Politics for their useful comments.

Notes

1. The unit of analysis should not be confused with the unit of measurement. The classical and the computerised content analysis approach have the same unit of analysis (election manifestos), but different units of measurements (paragraphs and words respectively). This is due to the different points of departure of the two methods. More on this in the next sections.

2. For Italy we simply included the most important leftist and rightist parties considering the absence of liberal parties (see Appendix A). We also included the 1992 manifesto of the social democrats (DS) because the number of available Italian election manifestos was rather low.

3. We only included the 2002 and 2005 party manifestos for Germany because manifestos prior to these dates were often not available in a legible digital format, which is needed for the computerised analysis.

4. These words are: people, citizen(s), community, society, public, population, nation(al), all of us, each of us, everyone, our, we, voter(s), electorate, referenda, direct democracy, public opinion, country. And words such as: United Kingdom, Britons, Netherlands, Dutch, Italians, Gemany, etc. (depending of course on the country under analysis.)

5. For the codebook the first author can be contacted.

6. While there are different approaches available in computerised textual analyses – such as Wordscores or Wordfish – we draw on a dictionary approach (Laver and Garry 2000). A drawback of Wordscores is that it requires scores to be computed by other methods such as expert surveys (Laver et al. 2003). Wordfish works well for extracting single left–right dimensions (Slapin and Proksch 2008), while it is less suited to explore a specific ideological aspect such as populism.

7. The word ‘taxes’, for instance, might be associated with cutting taxes but it can equally be used to indicate that a party wants to increase taxes. In practice, however, this latter meaning will hardly be found in party manifestos, and the word taxes is hence a good indicator for the category ‘reduce state involvement in the economy’, identifying socio-economic rightist parties.

8. Indeed, the classical content analysis empirically confirmed this: there is only a weak correlation between people-centrism and anti-elitism (r = 0.04, not significant at p < 0.05), whereas – the other way around – almost every anti-elitist paragraph also contains a reference to the people.

9. For instance, populists in the Netherlands sometimes talk about ‘regenten’ to express anti-elitism. This word refers to the Dutch political rulers in the sixteenth, seventeenth and eighteenth centuries. Although the ‘regenten’ did not form a hereditary class, they did form a closed group that reserved government offices for themselves. This specific word is not used by populists in countries other than the Netherlands.

10. This becomes even more apparent when we regress the results of the two methods on each other and look at the standardised residuals: the LPF and the SP (in 1994) are more than three standard deviations removed from the mean residual of 0.

11. The sample of paragraphs in the reliability tests contained about 5 per cent of the total amount of paragraphs. The results for people-centrism are: α = 0.78 (NL), α = 0.73 (UK), α = 0.74 (GE) and α = 0.81 (IT). The results for anti-elitism are: α = 0.84 (NL), α = 0.66 (UK), α = 0.81 (GE) and 0.81 (IT).

12. Reliability =  .