Three sides of the same coin? comparing party positions in VAAs, expert surveys and manifesto data

ABSTRACT

Existing research on political parties’ policy positions has traditionally relied on expert surveys and/or party manifesto data. More recently, Voting Advice Applications (VAAs) have been increasingly used as an additional method for locating parties in the policy space, with a closer focus on concrete policy issues. In this manuscript, we examine the reliability of party positions originated from a VAA, utilising the euandi longitudinal dataset, which provides data on positions of over 400 unique political parties across 28 EU member states from the European Parliament elections of 2009, 2014 and 2019. We cross-validate euandi data with the Comparative Manifesto Project (CMP) and the Chapel Hill Expert Survey (CHES). Our results attest the reliability of the euandi trend file vis-à-vis remaining data sources, demonstrating the validity of VAA-based methods to estimate the policy positions of European political parties. Convergence is especially high with CHES party placements. We also explore the sources of divergence in the estimation of policy positions across the three methods, finding little evidence of a systematic source of bias in the estimates between datasets. We conclude with an inventory of arguments in favour of party position measurements used by VAAs for the study of policy-making in European democracies.

KEYWORDS:

• Expert surveys
• manifesto analysis
• party placement methods
• policy positions
• Voting Advice Applications (VAAs)

Acknowledgements

The authors wish to thank all the institutional and technological partners participating to the project: the EUI, Kieskompas and the National Centre of Competence in Research of the Swiss National Foundation (2009); the EUI, the Berkman centre for Internet & Society at Harvard University, LUISS University in Rome and RnD Lab (2014); the Robert Schuman Centre for Advanced Studies at the EUI, the University of Luzern, Statistikalabor OÜ and Mobi Lab (2019). The authors also would like to acknowledge the work and commitment of the country experts and members of the Steering Committee involved in implementing the three waves of the VAA, as well as the political party representatives who took part in the self-placement procedure.

Finally, the authors would like to thank the participants of the 2020 ECPR General Conference, 2021 RSCAS EGPP Conference, and the 2021 ECPR SGEU Conference for their valuable feedback – in particular to Liesbet Hooghe, Simon Hix, Martin Rosema, Eric Linhart, and Mihail Chiru. We are also grateful to the anonymous reviewers and editors of the Journal of European Public Policy for their comments and suggestions, which have contributed to significantly improve the original version of the manuscript.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Notes

1 For simplicity we use the generic naming euandi also when referring to the 2009 edition of the VAA, at the time called ‘EU Profiler’.

2 For example, in CHES, we used lrecon instead of lrgen, as the euandi statements only capture Left-Right economic positioning. Therefore, for CHES, we relied on the original variables lrecon, position, and galtan, respectively. Despite these efforts, some non-negligible differences subsist between the statements used to build the dimensions in the euandi dataset, and in the CHES and CMP. In fact, one of the upsides of the euandi dataset lies in its ability to longitudinally tap into party positions not only across dimensions but, especially, on concrete policy positions. This is also the reason why we preferred using individual ‘content analytical data’ items from the CMP instead of solely resorting to ‘programmatic dimensions’ variables such as rile, planeco or markeco, as the latter may not encompass all the policy items comprised in the euandi dimensions, or may include other, absent items.

3 per605 was used instead of per605_1–per605_2 because the latter option significantly depressed the number of observations, due to missing values on the original CMP dataset. Given that it refers to law and order issues, which approach to valence issues, not having polar positive and negative measures is arguably not as problematic as in other policy issues. In fact, the value of the difference obtained when subtracting per605_1–per605_2 is quite similar to the value of per605 (3.85 and 4.34, respectively). In any case, the analyses were replicated using per605_1–per605_2 and the results do not substantively deviate (detailed analyses in Appendix B)

4 For example, in the euandi dataset, if Party X was coded 4 in the statement ‘Social programmes should be maintained even at the cost of higher taxes’ and 5 in the statement ‘Government spending should be reduced in order to lower taxes’, it would score 4.5 on the Left-Right dimension.

5 Note that the number of valid observations varies substantially across datasets: 768 for the euandi, 560 for CHES, and 386 for CMP. In all three-way comparisons (Figure 1) we have kept only the parties for which we have data on the three dimensions simultaneously available across all three data sources (N=348). Conversely, in pairwise comparisons (Table 3 and Figure 2) we have tried to maximise the number of cases, thus keeping all the parties for which we have data simultaneously available across the two data sources being compared. In these instances, the N varies, depending on the datasets being compared.

6 Due to very low number of parties that belong to these party families, we have not included the families of regionalist, confessional and agrarian parties. These and any other parties that do not fall under any of the seven distinguished families, are compiled into the ‘Others’ category.

7 Following the recommendations from Gemenis (2012, p. 601; 2013a, p. 289), we compare the measures using not only the Pearson product-moment correlation coefficient but also the concordance correlation coefficient (ρ_c), accompanied by the bias correction factor (C_b). The results reveal high accuracy and do not show meaningful differences from the Pearson correlation coefficient (see Table A4 in Appendix).

8 The N for the last time-period is substantially smaller for CMP, since the data collection is still ongoing. This likely explains the clear differences compared to the previous two data points and, for this reason, the coefficients for 2019 should be interpreted with caution.

9 If there are no differences between the euandi and CHES in the estimates for a given political party in any of the three dimensions the dependent variable will score 0. If, for example, in the euandi, Podemos scores 1 standard deviation above the mean on the GALTAN dimension whereas in CHES that party scores 0.5 standard deviation above the mean on the same dimension, the absolute difference for this data entry will be 0.5 on the that dimension. If the absolute difference is also of 0.5 in the remaining two dimensions, Podemos will score a total value of 1.5 in the dependent variable. There is no theoretical upper-bound for this variable since it is measured in standard deviations, and is thus dependent on the distribution of the data. The minimum value for the dependent variable in the sample is 0.13 and the maximum 7.6. However, values above 5 are only present for the euandi vs. CMP. The mean disagreement score for the euandi vs. CHES is 1.67 (σ = 0.77), and for the euandi vs. CMP is 2.23 (σ = 1.1). Full details on the distribution of these variables are available in Appendix G.

10 For the three parties in the sample that ran as part of a pre-electoral coalition, we used the vote share of the whole coalition.

Additional information

Funding

The author(s) reported there is no funding associated with the work featured in this article.

Notes on contributors

Frederico Ferreira da Silva

Frederico Ferreira da Silva (PhD European University Institute, 2019) is a Senior SNSF Researcher at the University of Lausanne, Switzerland. His research focuses on elections, voting behaviour, and voting advice applications.

Andres Reiljan

Andres Reiljan (PhD European University Institute, 2021) is a post-doctoral researcher at the University of Tartu. He is currently working on a two-year project studying affective polarisation, funded by the Estonian Research Council.

Lorenzo Cicchi

Lorenzo Cicchi (PhD IMT Institute for Advanced Studies, 2013) is research associate at the Robert Schuman Centre for Advanced Studies of the European University Institute, where he coordinates the European Governance and Politics Programme. His research interests include European Union politics and institutions, political parties, elections, voting behaviour and voting advice applications.

Alexander H. Trechsel

Alexander H. Trechsel is professor of political science at the University of Lucerne (Switzerland), where he also serves as Vice-Rector for Research.

Diego Garzia

Diego Garzia is an SNSF Professor of Political Science at the University of Lausanne. He currently serves as founding convenor of the ECPR Research Network on Voting Advice Applications and as a member of the Scientific Committee of the Italian National Election Study (ITANES)