Pre-election polling and the democratic veneer in a hybrid regime

ABSTRACT

Hybrid regimes have consolidated on the back of techniques that balance strong regime structures with tokenistic pluralism. This democratic veneer is performed through pseudo markers of democracy such as weak political parties and semi-competitive elections, which aim to ratify regime legitimacy. How public opinion polling fits into authoritarian landscapes, however, is an aspect of hybrid regimes that remains less understood. Scholars of public opinion research in democracies believe that polling can contribute to constructing the world around it, prompting this paper to examine whether public opinion research – and pre-election polling in particular – contributes to the democratic veneer in hybrid regimes by constructing a perception of participatory democracy. It examines the nature and quality of pre-election polling undertaken in authoritarian Iran in the lead-up to the 2017 presidential election in order to make preliminary observations about the potential impact of polling on a regime’s pluralist credentials. It finds that while most polls were poor quality, no polling in an authoritarian environment is benign because the very process of asking citizens their opinions and publicizing responses creates an impression that individual opinions count, in an environment where the opposite is often true.

KEYWORDS:

• Authoritarianism
• democratic veneer
• Iran
• contentious politics
• competitive authoritarianism
• public opinion research

Events such as the Colour Revolutions and the Arab Uprisings appeared to sound the death knell for autocrats, but the 2019 Freedom in the World report noted a “13th consecutive year of decline in global freedom”, as the regimes that survived the upheavals displayed increasing resilience. Scholars have defined many such states as “electoral authoritarian” regimes,¹ “defective democracies”² or “liberalized autocracies”,³ although all descriptors are united in recognition of the presence of “hybrid” governance structures. Tilly and Tarrow described hybrid regimes as displaying “systematic segments of democratic and undemocratic rule operating side by side”.⁴ Hybrid regimes balance strong regime structures with a thin democratic veneer that is performed through pseudo-democratic markers such as semi-competitive elections.

How public opinion polling contributes to authoritarianism is an aspect of hybrid regimes that remains less understood. Scholars of polling in democratic environments believe the process enhances democracy: Verba explained that good polling transmits voices to governments more equally than voting or letter writing,⁵ while Pearson argued that “opinion polling, by its very nature, lends itself to constructing the edifice of a liberal regime that aims to be both liberal and democratic”.⁶ Indeed, polling is not divorced from its environment; it helps construct the political reality that surrounds it. This raises questions about how polling might also contribute to the democratic veneer in hybrid regimes by constructing a perception of participatory democracy.

Research on pre-election polling in authoritarian regimes has focussed mostly on its functional aspects, especially related to the consequences of inaccurate polling. Although good polling could well hold regimes to account, Kalinin found in Russia that inaccurate polling was used to endorse falsified election results⁷ while Willnat et al. concluded that erroneous polling in Taiwan drew legitimate election results into question.⁸ Song et al.’s research went further, finding that the People’s Daily’s failure to disclose methodology prevented the Chinese public from verifying results, enabling state media to use “polls not so much to take a measure of public opinion as to create it; not so much to collect information as to present ‘correct’ attitudes for emulation and to mobilize mass acquiescence or mass anger”.⁹ Although inaccurate polling’s vulnerability to authoritarian manipulation is well established, scholars are yet to undertake empirical research to understand whether public opinion research links to the democratic veneer in authoritarian states. This requires detailed analysis beyond simple accurate/inaccurate binaries to understand if polling processes have broader pluralism-signalling effects that contribute to the construction and reproduction of the democratic veneer.

This paper examines pre-election polling in hybrid Iran, building a theory of pre-election polling in authoritarian regimes that it then uses to analyse 41 polls conducted during the 2017 presidential election. By examining polls for both methodological transparency and quality, it takes stock of the landscape to understand what kind of polling was undertaken, who conducted it, if agendas were evident, whether there was a concerted effort to undertake good polling, and ultimately what effects such polling may have had. The answers to these questions provide a picture of Iranian polling that acts as a starting point to understand how such research might contribute to the country’s pluralist veneer. Contributing to the literature on authoritarian elections, the paper argues that although the bulk of the polls were low quality, no polling in an authoritarian environment is benign because the very process of asking citizens their opinions and publicizing responses creates an impression that individual opinions count, in an environment where the opposite is often true.

A theory of pre-election polling in hybrid regimes

This paper defines the “democratic veneer” as the democratic-looking institutions and processes that exist within authoritarian regimes to create an impression of pluralism, but are too weak to threaten regime hegemony. This veneer has long been considered a pillar of regime resilience by acting as a “safety valve” for opponents to express frustration and compete for some spoils of political inclusion.¹⁰ Autocrats often go to great lengths to perform this veneer: during the 2014 election, the Syrian regime recruited “independent” election monitors from Iran, Russia and North Korea. Similarly, once citizens started using non-voting as protest in the Soviet Union, turnout rates became a politically sensitive bellwether of regime legitimacy.¹¹ In this regard, the democratic veneer is both overt and performative: it encompasses formal structures, but is also the end product of processes designed to transmit pluralist signals.

Authoritarian elections are one of the best understood aspects of the democratic veneer. Elections can shore up autocrats by offering spoils to dissuade ruling party defections¹² or by channelling dissidents’ energies into a proscribed but limited political space.¹³ But Simpser argued that elections serve a broader purpose by also transmitting hegemonic messages from regimes to their opponents. Asking why autocrats often overtly defraud contests they were already likely to win, he found that:

Electoral manipulation has informational properties. By shaping information and expectations about the power and the prospects of political parties, electoral manipulation can decisively influence the behaviour of a wide range of social and political actors.¹⁴

Wedeen identified a similar process in Yemen, noting that electoral fraud “did more than exemplify political power; it was also doing the work of creating power by demonstrating to regime officials and citizens alike that the regime could get away with the charade”.¹⁵ Indeed, electoral processes in authoritarian regimes – be they opaque nominations procedures or pseudo-open televised debates – create a spectacle through which a regime as the arbiter of pluralism reminds its subjects of its dominance. Elections are therefore also simultaneously formal and performative, raising the prospect that pre-election polling also contributes to constructing and reproducing the democratic veneer.

Scholars of pre-election polling in democracies have noted that polls are not neutral. Some research suggests that polling impacts turnout, which as noted above is a sensitive electoral statistic for autocrats. Kraut and McConahay postulated that this effect took place because polling decreased isolation between surveyed individuals and electoral processes,¹⁶ although Yalch concluded that such effect declined after one month.¹⁷ More recently, Burszytn et al. linked the perceived closeness of an election (as conveyed by polls) to turnout.¹⁸ This raises the possibility that if polling increases the sense of electoral competitiveness, its very presence may complement authoritarian agendas.

Scholars have also examined how polls interact with their surroundings, with Krippendorf asking: “Are polls descriptive or do they create what they claim to describe?”¹⁹ Building on this, Jtichenor mused that conducting polling on a certain issue may increase that issue’s salience in the minds of citizens:

Consider the case of a citizen group conducting a telephone poll asking local people whether they favor or oppose building a new prison on the edge of town. The newspaper report of that poll the next day is like any other agenda-setting media content, in that it reinforces the perception that the prison is a high priority local issue.²⁰

Indeed, the very act of conducting a poll on the electoral spectacle may increase the salience of pseudo-democratic electoral processes as a whole. Lipari added that

Polling may be more than a mirror reflecting the will of the people or a tool to manipulate the public. Seen from a ritual perspective, polling is born in part from the needs of a vast democratic nation to create a city or state feel.²¹

Polls too are performative, helping to construct the very democratic processes that they claim to benignly reflect.

While studies of pre-election polling in democracies have therefore found that polling can contribute to stimulating, constructing and reproducing democratic political life, it is not clear whether this pattern holds in authoritarian regimes. Although Kalinin and Willnat et al. observed that inaccurate polling and poor methodological disclosure was vulnerable to misuse by entrepreneurial autocrats, Simpser’s findings on the political value of overt manipulation suggest that accurate polling is no singular inoculant. The democratic veneer is more sophisticated than accurate/inaccurate poll binaries. Understanding any relationship between the veneer and pre-election polling, therefore, requires careful qualitative examination of the biases, agendas, transparency and nature of polling research in authoritarian regimes. This paper examines individual polls to answer questions about the broader polling landscape such as: What kind of polling is taking place? Who is conducting polls? What agendas might be at play? Is there evidence of an effort to conduct good polling? And what effects might these polls have? Using the findings of scholars of polling in democracies, it provides conceptual observations on how Iranian pre-election polling complements the regime’s democratic veneer.

The 2017 presidential election in Iran

The democratic veneer is an important feature of Iran’s hybrid political system in which elections for president, parliament and city councils are conducted at four-year intervals. Unelected officials such as Supreme Leader Ali Khamenei hold the ultimate power, but factions compete for executive and legislative influence. Victories usually correlate with small policy changes, giving a sense of democratic alternance and creating public expectation. Results (including President Rouhani’s 2017 re-election) often prompt spontaneous street celebrations.

The 2017 presidential election was conducted alongside city council elections, although this paper examines only presidential election polling. The electoral process violates many democratic norms: candidates must submit their credentials to the Guardian Council, which opaquely vets candidate qualifications, including perceived regime allegiance. In 2017, 1636 Iranians, including 137 women, nominated for the presidency. Six candidates (and no women) were approved: incumbent President Rouhani, Mostafa Aqa-Mirsalim, Mostafa Hashemi-Taba, Eshaq Jahangiri, Mohammad-Baqer Qalibaf and Seyyed Ebrahim Raeisi. Jahangiri and Qalibaf withdrew days before the poll. As of the 2016 census, Iran’s population was just under 80 million people, spread across 30 provinces, and many ethnic and religious groups.²² 55 million were eligible to vote in 2017: Iran has no formal voter registration process and eligibility extends to all citizens over the age of 18 with a government ID card. Just over 40 million voted (73.33%), with President Rouhani re-elected with nearly 24 million votes (57.14%).

Electoral integrity became central to Iran’s democratic veneer after the 2009 election, which was expected to be a close race between President Mahmoud Ahmadinejad and Mir Hossein Mousavi. After Ahmadinejad was declared victor in a landslide on election night, hundreds of thousands took to the streets. The regime suffered immense reputational damage in the six-month crackdown that followed and has since gone to significant lengths to avoid accusations of ballot rigging, prioritizing the perception of competitiveness, even though elections are not free and fair. Similar to the Soviet Union, turnout is prized: Supreme Leader Khamenei hailed turnout in 2017, declaring that “the winners of the election are the people of Iran and the Islamic Republic”. It is in this context that pre-election polling aligns particularly well with Iran’s carefully managed pluralism.

The government of Iran tightly controls the electoral process. Many activists were arrested in 2017, while the regime enacted heavy online and offline surveillance. Polling too is sensitive: after publishing a poll that showed that Iranians favoured dialogue with the US in 2002, Ayandeh Polling Institute personnel were convicted of conspiring with Gallup. During the January 2018 protests, Iranian Students Polling Agency (ISPA) personnel were arrested. In such a repressive context, it is therefore notable that polling takes place at all. Nonetheless, even Supreme Leader Khamenei conducted election-related research in 2017, polling his Telegram followers about the country’s greatest economic problems, and receiving 77,390 responses. Although the self-selected sample could not bear meaningful results, the very fact of being a powerful person asking questions – and the nature of the question – suggests an effort to signal responsiveness, and recognition of the utility of polling.

Methods

Opinion polling under repressive regimes is fraught with challenges. Scholars have expressed concerns related to participant reactivity: Tannenberg found that surveys produced different results depending on who was said to have commissioned the research.²³ Polling censorship and self-censorship is also a challenge, with Otava and Wilson noting that pollsters may face pressure if results are unfavourable to governments.²⁴ Kalinin also observed a social desirability bias in 2012 Russian pre-election polls, with results skewed towards Vladimir Putin because respondents did not feel comfortable indicating support for another candidate.²⁵

Nonetheless, reliable polling has taken place in authoritarian regimes. Geddes and Zaller conducted a survey in Brazil in 1972–3, finding that US polling models were applicable in authoritarian environments.²⁶ This was echoed by Beltrán and Valdivia, who criticized observers for blaming polling errors in authoritarian states on political context, while attributing inaccuracies in democratic states to methodology.²⁷ Some scholars used measures such as the endorsement method²⁸ or list experiments²⁹ to mitigate social desirability bias, although Frye et al. argued that subterfuge may not be necessary.³⁰ Even so, Horne urged vigilance, finding that while, “it is possible to conduct meaningful surveys of public opinion in an authoritarian setting … [every survey] must be treated with extreme caution, even skepticism”.³¹ Elsewhere, he recommended analysing individual polls for: (1) Proper and Transparent Survey Design, (2) Survey Reliability and (3) Survey validity.³²

In order to understand how polling might interact with the democratic veneer in Iran, this paper first uses the AAPOR Code of Professional Ethics and Practice’s Minimum Disclosure Requirements (MDR). AAPOR describes the MDR as setting “the standard for the ethical conduct of public opinion and survey research”.³³ It is primarily designed to assess quality, but for the purposes of this study, it is also a valuable tool for identifying the types of polls that were conducted, who undertook polling, and the biases, interests and nature of the research. Such requirements were designed for the US and are not implemented by pollsters worldwide, but Song et al.’s research in China noted above highlighted the particular importance of methodological transparency in authoritarian climates given the lack of independent oversight. The use of the AAPOR model is consistent with Geddes and Zaller’s findings that polling techniques used in democratic contexts were applicable in authoritarian states, and Horne and Beltrán and Valdivia’s findings that it is possible to undertake good polling in authoritarian regimes.³⁴ At least one of the pollsters included in this research (Iranpoll) was an AAPOR member. The AAPOR MDR are:

Table

MDR 1	Name of the survey sponsor
MDR 2	Name of the organization that conducted the survey
MDR 3	The exact wording of the questions being released
MDR 4	A definition of the population under study
MDR 5	A description of the sampling frame used to represent this population
MDR 6	An explanation of how the respondents to the survey were selected
MDR 7	The total sample size
MDR 8	The method or mode of data collection
MDR 9	The dates and location of data collection
MDR 10	Estimates of sampling error, if appropriate
MDR 11	A description of how the data were weighted (or a statement that they were not weighted)
MDR 12	If the survey reports findings based on parts of the sample rather than the total sample, then the size of the subgroups reported should be disclosed^35

Although Frye et al. concluded that additional instruments were not required in authoritarian environments, the authors added two Additional Criteria (AC) to more fully understand poll content:

Table

AC 1	Details of surveyor provided
AC 2	Include survey instruments to minimize impact of authoritarian climate

The authors assigned one point for each MDR (maximum score of 12 as the two AC were optional) and then ranked polls by overall score (Appendix 1). Polls which partially met an MDR criterion received a half point.

After completing the preliminary analysis, the paper applies Horne’s model to undertake a deeper analysis of survey reliability and validity for the two highest ranked pollsters to understand whether any polls results produced in 2017 reflect actual public opinion.³⁶ The survey reliability analysis assesses intra-survey consistency, such as whether similar questions within the one survey produce consistent responses. Survey validity is then examined to understand whether each poll’s demographics accurately reflect the population that it claims to represent, and whether responses reflect actual public opinion.

As noted above, existing research on polling in authoritarian regimes has focussed on inaccuracy, but the polling community continues to debate whether predicting the correct result is a prerequisite for good polling because opinion climates are fluid. A poll taken 10 days before an election might be accurate that day, but voter intentions may change by election day. It is, therefore, difficult to independently validate accuracy. Where relevant, this paper does comment on accuracy, but given these issues and the history of electoral irregularities in Iran, high quality poll results may not mimic official election results.

The paper evaluates 41 polls conducted between 21 April and 19 May 2017. This marks the formal campaign period from the date when approved candidates were announced to election day. This included 16 polls conducted by polling organizations and 25 polls undertaken by media organizations. The authors recorded every Persian and English poll published online that they found each day during the period, although do not claim to have captured every poll. The authors found polls through Persian and English Google searches with the keywords: “Iran election”, “Iran election poll”, “Iran election survey”, “election 1395”, “Rouhani”, “Mirsalim”, “Hashemi-Taba”, “Jahangiri”, “Qalibaf” and “Raeisi”. Each poll was downloaded and analysed in a table (Appendix 1) against the 12 criteria. Polls were excluded if (1) the original poll could not be found or (2) too little information was available. It is worth noting only one state-linked poll was included in this research, even though regime organs such as the Research Center of Islamic Republic of Iran Broadcasting (IRIB) conducted at least pre-election 11 polls. Artefacts of the IRIB research were available online, but the research team was unable to access sufficient details for inclusion in the study.

The MDR analysis below, therefore, outlines the parameters of the 2017 Iranian polling landscape, describing the nature of the polling and highlighting methodological shortcomings which indicate the quality and agendas of pollsters, and the potential impact of polls on Iran’s democratic veneer.

Results

MDR 1: name of the survey sponsor

No polls clearly acknowledged an external sponsor. This could be because most polls were self-funded, or may have reflected security considerations given the Ayandeh Polling Institute’s experience. Of note however was the Toronto-based Iranpoll, whose methodology hinted at sponsorship, stating: “Like most polling agencies, majority [sic] of polls conducted by IranPoll is [sic] owned by clients”. It did not categorically state that the 16 May poll was sponsored, although later confirmed via email that the poll was funded by a private client, whose name they could not release. The vagueness of this public disclosure is problematic as it may have important implications for polling bias, particularly given the polarization of the international debate on Iran and the myriad of actors who might see value in polling.

MDR 2: name of the organization that conducted the survey

All polls met the MDR 2 requirement of disclosing the name of the organization conducting the survey.

MDR 3: the exact wording of the questions being released

The questions asked by the polls fell into two main categories. The first were “horse-race” style questions typical of pre-election polls elsewhere, including: Who will you vote for? Do you prefer Rouhani or Raisi (the two frontrunners)? and If you change your mind before the election, who will you vote for instead? The results for many such polls were depicted in professionally designed charts that purported to convey electoral competitiveness. It is worth considering whether enhanced competitiveness in authoritarian regimes has the turnout-increasing effect observed by scholars of polling in democracies. If so, polling may complement the Iranian regime’s agenda of attracting high turnout to offset the damage that its democratic veneer endured in 2009. The democratic veneer is, after all, both performative and tangible.

The second category of questions, predominantly used by media organizations, addressed elements of the electoral spectacle, including: Who won the candidate debate? To what extent are Iranians interested in the presidential election? When will voters decide on their preferred candidate? Do you intend to vote? and Which political faction do you most closely identify with? Such polling may seem benign, but the very practice of asking tens of thousands of participants (Table 1) non-critical questions about the process may heighten perceptions of electoral integrity and potentially increase the salience of the election in the minds of everyday Iranians, as Jtichenor observed in democratic systems. The reality that most polls were conducted by non-government actors, and therefore lacked associations with a regime agenda, could have strengthened any such effect.

Table 1. Sample size.

Poll	Sample size	Poll	Sample size
IPPO Group 1	947	Bazar News Agency 1	7534
IPPO Group 2	1000	Bazar News Agency 2	431
IPPO Group 3	1076	Bazar News Agency 3	Not Specified
IPPO Group 4	1189	Bazar News Agency 4	Not Specified
IPPO Group 5	1189	Bazar News Agency 5	1321
IPPO Group 6	1212	Bazar News Agency 6	7398
IPPO Group 7	1199	Entekhab News Agency 1	25,712
IPPO Group 8	1012	Entekhab News Agency 2	9608
IPPO Group 9	1122	Entekhab News Agency 3	12,672
IPPO Group 10	1376	Fararu News Agency	26,699
IPPO Group 11	1194	Iranian Labour News Agency	1500
IPPO Group 12	1203	Islamic Republic News Agency	6047
Iranpoll 1	1007	Khabar Online News Agency	“Approximately 15,000”
ISPA 1	1000	Net RAY 1	1798
ISPA 2	4500	Net RAY 2	1827
ISPA 3	Not specified	Net RAY 3	2086
Aban Press	2336	Peace News Agency	17,239
Asr Iran News Agency 1	47,377	Rooz Plus	Not specified
Asr Iran News Agency 2	22,797	Tabnak News Agency 1	61,872
Asr Iran News Agency 3	31,143	Tabnak News Agency 2	14,778
Asr Iran News Agency 4	18,918

MDR 4: a definition of the population under study

No media organization polls upheld MDR 4, although the research institutes performed better. IPPO Group declared that it had conducted country-wide surveys which: “can [be] generalize [sic] to the whole of the 18 years and older Iranian” population. Iranpoll explained that it had polled a “representative sample”. The Iranian Students Polling Agency’s (ISPA) Poll 1 purported to represent Iranian public opinion as a whole, but polled just the 22 districts of the capital Tehran. ISPA polls 2 and 3 undertook nationwide surveys, paying attention to both the gender and urban/rural make-up of participants.

MDR 5: a description of the sampling frame used to represent this population

MDR 5 was poorly upheld. Only IranPoll declared using all landlines as their frame, although it was not clear where they sourced the list.

MDR 6: an explanation of how the respondents to the survey were selected

IranPoll was again notable for its transparency, explaining that it used Random Digit Dialling,

stratified first by Iranian provinces and then in accordance to settlement size and type … When a residence was reached, an adult was randomly selected from within that household using the random table technique. An initial attempt and three callbacks were made in an effort to complete an interview with the randomly selected respondents.

This follow-up technique may explain Iranpoll’s reported 75% response rate. IPPO Group used what it described as “two-stage proportional sampling … on the basis of each of the service operators’ market share and then simple random sampling” for most of its polls. Participants in most media organization polls were self-selected website users.

MDR 7: the total sample size

MDR 7 was upheld by almost every poll (see Table 1). Media organizations reported significantly larger sample sizes, with an average of 15,277 participants.

MDR 8: the method or mode of data collection

Almost all polls declared their data collection method (see Appendix 2 for detailed MDR 8–10 disclosures). Most polling conducted by research institutes was undertaken by telephone, the challenges of which are discussed in greater detail below in relation to IPPO Group and Iranpoll.

The state-run Islamic Republic News Agency (IRNA) was the only poll to undertake in-person data collection, which can be useful as participants may be more willing to discuss their views frankly with an interviewer to whom they have developed a rapport. IRNA predicted an expected 63.9% turnout rate (almost 10 points lower than actual turnout) and declared that 46% of voters were yet to choose their candidate. The results are unlikely to be credible owing to the poll’s broader methodological oversights, the fact that it was conducted more than two weeks before the election, and IRNA’s regime links that likely predisposed participants to a reactivity effect similar to that found by Tannenberg. Nonetheless, IRNA declared that “given the increasing popularity of presidential candidates and the holding of televised debates, the turnout is expected to increase in the coming days”, in what appeared to be a regime-linked effort to convey electoral competitiveness.

By contrast, most media organizations polled their website visitors, making no effort to build valid population samples. This raises significant concerns about sampling bias because decisions to participate may be guided by observable characteristics such as gender, age, religion, political views or hours worked, undermining any claim to the representativeness of a sample. An indicative poll was that undertaken by the pro-Reformist website Entekhab, which asked: “Do you think the election debates should be broadcast live?” The poll followed President Rouhani’s (who was supported by the Reformists) criticism of the Interior Ministry’s decision to not live-broadcast candidate debates. Reflecting Entekhab’s audience, 97.21% of respondents agreed with Rouhani’s position, with a high-quality graphic of the poll published alongside an article on the Entekhab website describing the results. Although the poll is not statistically valid, it nonetheless gave the sense that a dynamic debate was underway. Most of the media polls had similar methodological shortcomings, although results were frequently reported by other outlets. This ad hoc nature suggests that non-government media pollsters were not pursuing an organized malevolent agenda, but nonetheless only one of the polls collected was censored by the authorities: The very fact that the remaining polling persisted suggests that the regime viewed independent polling as at best benign, or at worst a useful tool.

MDR 9: the dates and location of data collection

With the exception of IRNA, ILNA and Peace News Agency, news organizations also performed poorly on MDR 9. Although location is difficult to track in web polls, the absence of dates in most cases meant that results were estranged from their political context.

MDR 10: estimates of sampling error, if appropriate

IPPO Group and IranPoll were the only pollsters that disclosed sampling error.

MDR 11: a description of how the data were weighted (or a statement that they were not weighted)

A statement on data weighting is required for every poll. IPPO Group was the only organization to acknowledge weighting “based on the last national Iranian census (2012) with gender, age group and place of residence (urban/rural) as weighting variables”.

MDR 12: if the survey reports findings based on parts of the sample rather than the total sample, then the size of the subgroups reported should be disclosed

No polls declared excluding population subgroups from their final results.

AC 1: details of surveyor provided

Media organizations were readily identifiable, while Iranpoll publicly declared itself the Toronto-based division of People Analytics Inc. ISPA’s website explained that ISPA was established in 2001 and is hosted at the University of Tehran.

IPPO Group’s background was opaquer. Its website appeared to be a simple Wordpress site with a vague organizational description. There were no details about company officers on the website (or elsewhere). Its listed address was a virtual office in K-Street, Washington, DC. Its Facebook page was created on 30 April 2017, and last posted on 24 July 2017, suggesting a singular organizational focus on the election. IPPO Group did not respond to repeated email requests for information, making it difficult to evaluate the organization’s agenda.

AC 2: include survey instruments to minimise impact of authoritarian climate

None of the polls overtly declared using techniques to mitigate authoritarian impact. However, IPPO Group explained:

Each interview was assessed twice by the interviewer and the supervising team – in terms of respondent’s trust in the interviewer and the interviewer’s assessment of the respondents’ honesty. Those respondents who had received very low scores for trust and honesty have been removed.

There is precedent for excluding untruthful respondents, but it must follow transparent protocols. IPPO Group’s methodology however appears based on the judgement of the interviewer and supervisor, providing no criteria for scrutiny. There was no indication of how many respondents were excluded. This may represent a technique for managing social desirability bias, but its poor execution undermined the results.

Iranpoll also used techniques that might act as a protective mechanism against authoritarianism by building a strong rapport with respondents. Iranpoll’s first question asked the “degree to which you have a favorable or an unfavourable view” of personalities including non-candidates such as the then-IRGC chief Qassem Soleimani. Iranpoll also made efforts to manage Iran’s population heterogeneity by allocating interviewers to each province who were native speakers of that province’s predominant languages. This may be another key trust-building mechanism in an authoritarian context, or at least an effort to collect higher quality data.

Analysis

The MDR and AC analysis was designed to extract a granular view of the Iranian polling landscape to describe the nature of the polls, and understand if they may have had the competitiveness raising, turnout stimulating or issue salience effect noted of polling in democratic environments. The results revealed that while polling was prolific, most surveys – with the exception of Iranpoll and IPPO Group – failed to meet the minimum requirements for quality research and cannot be viewed as a meaningful gauge of Iranian political sentiment. The MDR and AC transparency tests also highlighted how these processes interact with Iran’s democratic veneer, albeit likely unintentionally in the case of the majority of pollsters. This included the prevalence of questions related to the web of processes and symbols surrounding the election (MDR 3) including the debates, participation rates and voter decision-making timeframes. Although such questions seemed politically neutral, many engaged tens of thousands of participants (MDR 7) whose involvement gave the impression of competitiveness and may have raised the salience of the electoral process. Many media organizations’ results were subsequently reported by other outlets, amplifying any such effect. At the same time, the regime-linked IRNA engaged directly with questions related to turnout, while questions surrounding the agendas of top performing pollsters Iranpoll and IPPO Group raised further questions. In all, this provides a picture of murky polling environment that shares many features with democratic jurisdictions, and may therefore contribute to the spectacle surrounding authoritarian elections.

It is important, however, not to dismiss the significance of Iranpoll and IPPO Group’s polls given that good polling can hold autocrats to account. Although Simpser identified the utility of overt electoral manipulation, the presence of polls that reflect actual public opinion nonetheless increases the cost and risk of implementing such a strategy. But Horne argued that survey transparency alone is not a sufficient guarantee of statistically valid results: further checks of reliability and validity are required.³⁷ The section below analyses these two pollsters for measures of survey reliability and validity to understand the representativeness of the datasets and whether they might accurately reflect public opinion. It is worth noting that intra-survey reliability analysis from individual-level responses was not possible as Iranpoll could not release its client-owned dataset, while IPPO Group did not respond to repeated email requests.

Iranpoll conducted the study’s most transparent poll on 16 May. It was published by The Economist on election day. Iranpoll surveyed 1007 Iranians on questions including: “Please say the degree to which you have a favorable or an unfavorable view” of x and “If only Rouhani and Raisi remain in the race, who would you vote for?” Iranpoll upheld almost all of the MDR, adding additional measures relevant to the Iranian context as noted above.

Iranpoll only conducted one poll during the campaign, meaning that demographic or inter-survey reliability could not be assessed. Nonetheless, intra-poll reliability tests found that respondents had given relatively consistent answers across questions, with Don’t Know/No Answer results ranging between 9% and 13% for all three questions relating to voter intent.

The authors identified several demographic validity issues. First, it was not clear whether the survey was representative on gender or age, nor where Iranpoll sourced its population data. This could be a disclosure oversight in an otherwise appropriate methodology. Iranpoll’s reliance on landlines raised further concerns as landline penetration rates were only 38% (versus 92% cell phone penetration) in 2016, drawing questions about the poll’s population sample. Landlines may skew the sample towards older Iranians, and those who spend more time at home. Telephone lines are also not secure in Iran, which may create a social desirability bias.

Iranpoll collected data three days before the election, which meant that its polling did not exactly mimic election conditions, including Jahangiri’s eleventh-hour withdrawal. Nonetheless, the official election result was within Iranpoll’s ±3.09 margin of error. This, along with the poll’s overall methodological integrity suggests that high quality polling does take place on Iran. The opaque nature of the survey’s sponsorship however highlights that polling in authoritarian contexts can serve the agendas of multiple actors: it would be a mistake to consider authoritarian regimes the only interested parties.

IPPO Group administered 12 daily polls from 4 May 2017. It published results in Persian and English on a “rolling average basis” from 7 May until 17 May, plotting new data points on colourful charts that claimed to depict the crystallization of public opinion as the campaign wore on. Some questions were repeated daily, while new questions were posed as the campaign proceeded, including: “if the election were held today, who would you vote for?” and “How likely are you to participate in the upcoming presidential election?”

IPPO Group was one of the most MDR compliant pollsters, but in addition to concerns noted above on data exclusion and opaque organizational identity, IPPO Group’s methodology raised questions about the representativeness of its population samples. First, the interviews were conducted during “daylight hours”, which may skew the sample towards those who are home during the day. On the first day of data collection (3 May), sunrise took place at 6.11am and the sun set at 7.50pm in Tehran. Although this impact may have been partially mitigated by its gender, age group and urban/rural weighting, the weighting did not include employment status.

IPPO Group’s sequential release of daily polls provided an opportunity to observe demographic and intra-survey reliability in a way that was not possible with other polls. In order to be comparable, such surveys should have consistent demographics and samples. IPPO Group claimed that its results provided a sense of changing political opinion as the election approached, but inconsistencies across data collection method, sample size and collection timeframes prompted doubt over whether the claimed variations reflected changes in the political climate or variance between population samples. For example, polls 2, 3 and 5–12 collected data via “phone”, while polls 1 and 4 explained that respondents had been polled by landline and cellular phone. Although phone polling raises the same security questions outlined above, it was also not clear whether the polls conducted via “phone” encompassed dual-frame collection via both landline and cell phones (as in polls 1 and 4), or whether it was just landlines. The latter scenario could have significant sampling implications by polling different populations. Sample sizes also varied significantly: Poll 1 had the smallest sample of 947 respondents; Poll 10 was 45% larger with 1376 respondents. Data collection timeframes also varied without explanation. Polls 1–7 and 10 collected data over four days, polls 8 and 9 collected data for three days, poll 11 was conducted over two days, while poll 12 collected responses for a single day. These inconsistencies compromised IPPO Group’s trend graphs and underlined the validity of Horne’s model: While IPPO Group performed relatively well in terms of transparency, detailed analysis revealed significant shortcomings.

Sample selection methods were also inconsistent. All polls with the exception of poll 4 used “two-stage proportional sampling … on the basis of each of the service operators’ market share and then simple random sampling”. As noted above, this disclosure lacked important sampling frame details, but is also noteworthy because poll 4 used an entirely different method:

In the first stage the provinces were divided in three groups based on Human Development Index (HDI). Within each group with similar HDI, simple random sampling was used to choose respondents.

It is not clear how such an Index was applied, but it would no doubt have had significant sampling implications, further eroding the integrity of IPPO Group’s daily graphs.

Finally, it is worth commenting on IPPO Group’s 17 May results. The final election result fell outside IPPO Group’s margin of error, however of more significance was the difference between IPPO and Iranpoll’s reported number of respondents who did not provide answers. Iranpoll reported that 12% of participants had given don’t know/no answer responses, while IPPO Group indicated a significantly larger 16% of undecided and 20% who did not wish to disclose their vote (36% combined). This in conjunction with a plethora of other methodological shortcomings and the questions surrounding IPPO Group’s agenda, serves as a warning against taking polls at face value.

Pre-election polling and the hybrid state

Iran experienced a vibrant election-related polling environment in 2017. While this paper noted that good polls are no panacea for authoritarian manipulation, it closely observed Iran’s polling landscape to understand what polling was being conducted, by whom, and to consider how it might interact with the democratic veneer. A full spectrum of pre-election polling existed in Iran’s 2017 campaign, although most failed to meet the minimum requirements for quality research, highlighting that questions related to relationship between pre-election polling and the democratic veneer in Iran are based on perceptions of the will of Iran’s body politic much more than actual public opinion. Although Iranpoll and IPPO Group performed significantly better than their competitors, IPPO Group’s polls were vulnerable to misuse because its professional delivery concealed significant methodological shortcomings. Whether polls accurately mirror public opinion is nonetheless not the only important area of enquiry: scholars such as Jtichenor and Lipari noted the potent role that polling plays in democracies by raising issue salience and reproducing the complex web of symbolism that surrounds elections.

Further research is required to understand whether the existence of polling in Iran reflects an active regime effort to augment the pluralist spectacle surrounding elections, but there is significant theoretical reason to believe that polling complements the regime’s post-2009 pseudo-electoral integrity agenda. The paper noted that the regime conducted election-related public opinion research, including at the 11 IRIB polls and Supreme Leader Khamenei’s Telegram survey that were excluded from the MDR analysis, as well as the in-person IRNA poll that was analysed. Although many other polls at the time focussed on predicting the final result, IRNA undertook questioning that directly engaged with the electoral spectacle. In a political system in which the democratic veneer is tightly controlled and all pluralist concessions are carefully managed, any such questioning has informational purposes. Indeed, if polling helps construct perceptions of an idealized democratic process, then autocrat-led polling could too contribute to the spectacle of authoritarian elections.

The campaign period was also notable for a flurry of independent polls, many of which seemed unproblematic because of their superficial nature and lack of government connection. Undeniably, the presence of good independent polling may represent an important check on authoritarian power, although few polls meet such a benchmark. Nonetheless, many engaged tens of thousands of participants whose involvement gave the impression of mass politics and competition. This process was augmented by the striking graphs distributed alongside results and their reporting by other outlets. The polls did not ask whether voters trusted the system or believed that their vote mattered; questions implied that a genuine election process was underway and that independent parties endorsed the process. Voters might wonder why else the outlets would bother asking the question. The Iranian government could have blocked such polling given its strict policing of the electoral process, but it turned a blind eye to most surveys in 2017. This may be because of polling’s potential to contribute to the sense of pluralism in Iran. If polling in 2017 created such a sense, the strategy would align with the Iranian regime’s desire to create a perception of electoral competitiveness and integrity. Indeed, this paper noted that the democratic veneer is not just a set of formal structures, but is also a performative process that constructs the spectre of pluralism. Both poll results and processes may be used to reproduce a pseudo-democratic reality.

These findings are relevant to the study of authoritarian elections in other contexts in that they highlight that pre-election polling may not be a benign side effect of semi-competitive electoral processes, but a tool that bolsters the pluralist credentials of regimes. It also highlights that polling results in authoritarian contexts should never be used in isolation from other good research. Although Iranpoll (and IPPO Group to a far lesser extent) performed well in this analysis, questions about polling agendas underlines the importance of approaching all polling with caution. The fact that the two highest performing pollsters were not fully transparent on their funding – and that one such poll was published on election day itself – suggests that a multitude of actors recognize the value of polling. This paper therefore creates a starting point for further research on the role of polling in authoritarian contexts and its possible impact on the pluralist veneer that many autocrats today are so keen to cultivate, and others seek to undermine.

Acknowledgements

The authors wish to thank Azadeh Davachi and Dr Stephanie Carver for their invaluable research assistance, and Professor Carolien van Hamm and Dr Matteo Vergani for their feedback on earlier versions of this paper. The paper also benefited immeasurably from a rigorous peer review process, and the authors are grateful to the two anonymous peer reviewers for their deep and constructive engagement.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Notes on contributors

Dara Conduit

Dr Dara Conduit is a Research Fellow at the Alfred Deakin Institute for Citizenship and Globalization at Deakin University, where she works on authoritarianism and oppositions, mostly in the Middle East. Her work has been published in journals including Studies in Conflict and Terrorism, the Middle East Journal and Journal of Contemporary China, and her book The Muslim Brotherhood in Syria was published by Cambridge University Press in 2019.

Shahram Akbarzadeh

Professor Shahram Akbarzadeh is Research Professor in Middle East and Central Asian Politics at Deakin University (Australia) and Deputy Director (International) of the Alfred Deakin Institute for Citizenship and Globalization. He held a prestigious ARC Future Fellowship (2012–2016) on the Role of Islam in Iran’s Foreign Policy-making and recently completed a Qatar Foundation project on Sectarianism in the Middle East.

Notes

1 Schedler, Electoral Authoritarianism, 3.

2 Bogaards, “How to Classify Hybrid Regimes?” 399.

3 Brumberg, “Transforming the Arab World,” 90–91.

4 Tilly and Tarrow, Contentious Politics, 75.

5 Verba, “The Citizen as Respondent,” 4.

6 Pearson, “Public Opinion and the Pulse,” 59.

7 Kalinin, “The Social Desirability Bias.”

8 Willnat, Lo and Aw, “The Good, the Bad and the Ugly.”

9 Song et al., “Polls in an Authoritarian Space,” 352.

10 Buehler, “Safety-valve Elections and the Arab Spring.”

11 Karklins, “Soviet Elections Revisited.”

12 Magaloni, “Voting for Autocracy.”

13 Wiktorowicz, “The Limits of Democracy.”

14 Simpser, “Why Governments Manipulate Elections,” 239.

15 Wedeen, “Peripheral Visions,” 77.

16 Kraut and McConahay, “How Being Interviewed Affects Voting,” 405.

17 Yalch, “Pre-election Interview Effects.”

18 Bursztyn et al., Polls, the Press, and Participation, 21.

19 Krippendorf, “The Social Construction of Opinion,” 143.

20 Jtichenor, “Public Opinion and Social Reality,” 552.

21 Lipari, “Polling as Ritual,” 84.

22 Barry, Armenian Christians in Iran.

23 Tannenberg, The Autocratic Trust Bias.

24 Otava and Wilson, “Public Opinion Research in Czechoslovakia,” 249.

25 Kalinin, “The Social Desirability Bias,” 192.

26 Geddes and Zaller, “Sources of Popular Support,” 326.

27 Beltrán and Valdivia, “Accuracy and Error in Electoral Forecast,” 115.

28 Bullock, Imai and Shapiro, “Statistical Analysis of Endorsement Experiments.”

29 Corstange, “Sensitive Questions, Truthful Answers?” 61.

30 Frye et al., “Is Putin’s Popularity Real?” 2.

31 Horne, “The Structure of Public Opinion,” p 81.

32 Horne, “Measuring Public Opinion.”

33 AAPOR, The Code of Ethics. Paragraph 4.

34 Beltrán and Valdivia, “Accuracy and Error in Electoral Forecast.”

35 Survey Disclosure Checklist.

36 Horne, “Measuring Public Opinion.”

37 Ibid.

Appendices

Appendix 1. Note that all polls listed here remained available online at their original location as of 1 February 2019, or via web.archive.org

Table

Poll	Link	MDR 1	MDR 2	MDR 3	MDR 4	MDR 5	MDR 6	MDR 7	MDR 8	MDR 9	MDR 10	MDR 11	MDR 12	AC 1	AC 2	Total
Professional polling organizations
Iranpoll	https://bit.ly/2TBLNy4	0.5	1	1	1	0.5	1	1	1	1	1	0	0	Y	?	9
IPPO Group 1	https://bit.ly/2Jnr10u	0	1	1	1	0	1	1	1	1	1	1	0	N	?*	9
IPPO Group 2	https://bit.ly/2TBn7FZ	0	1	1	1	0	1	1	1	1	1	1	0	N	?*	9
IPPO Group 3	https://bit.ly/2HmsIsU	0	1	1	1	0	1	1	1	1	1	1	0	N	?*	9
IPPO Group 4	https://bit.ly/2UDA3ay	0	1	1	1	0	1	1	1	1	1	1	0	N	?*	9
IPPO Group 5	https://bit.ly/2F14fGa	0	1	1	1	0	1	1	1	1	1	1	0	N	?*	9
IPPO Group 6	https://bit.ly/2TO6wxO	0	1	1	1	0	1	1	1	1	1	1	0	N	?*	9
IPPO Group 7	https://bit.ly/2F2nFKL	0	1	1	1	0	1	1	1	1	1	1	0	N	?*	9
IPPO Group 8	https://bit.ly/2JaujUn	0	1	1	1	0	1	1	1	1	1	1	0	N	?*	9
IPPO Group 9	https://bit.ly/2F2nXBl	0	1	1	1	0	1	1	1	1	1	1	0	N	?*	9
IPPO Group 10	https://bit.ly/2VYWNSH	0	1	1	1	0	1	1	1	1	1	1	0	N	?*	9
IPPO Group 11	https://bit.ly/2F0HAcT	0	1	1	1	0	1	1	1	1	1	1	0	N	?*	9
IPPO Group 12	https://bit.ly/2FcpJ4c	0	1	1	1	0	1	1	1	1	1	1	0	N	?*	9
ISPA 1	https://bit.ly/2oLqTZD	0	1	1	1	0	0	1	0	1	0	0	0	Y	N	5
ISPA 2	https://bit.ly/2TzqL2S	0	1	1	1	0	0	1	0	1	0	0	0	Y	N	5
ISPA 3	https://bit.ly/2Hm3Odg	0	1	1	1	0	0	0	1	1	0	0	0	Y	N	5
Media organization polls
Islamic Republic News Agency	https://bit.ly/2HwwxLv	0	1	1	1	0	0.5	1	1	1	0	0	0	Y	N	6.5
Net RAY 3	https://bit.ly/2TBLaoc	0	1	1	0	0	0	1	1	1	0	0	0	Y	N	5
Peace News Agency	https://bit.ly/2W738f9	0	1	1	0	0	0	1	1	1	0	0	0	Y	N	5
Aban Press	https://bit.ly/2O7goNH	0	1	1	0	0	0	1	1	0	0	0	0	Y	N	4
Asr Iran News Agency 1	https://bit.ly/2JrGDQD	0	1	1	0	0	0	1	1	0	0	0	0	Y	N	4
Asr Iran News Agency 2	https://bit.ly/2HzIiB4	0	1	1	0	0	0	1	1	0	0	0	0	Y	N	4
Asr Iran News Agency 3	https://bit.ly/2T3r8h3	0	1	1	0	0	0	1	1	0	0	0	0	Y	N	4
Asr Iran News Agency 4	https://bit.ly/2T6GMbI	0	1	1	0	0	0	1	1	0	0	0	0	Y	N	4
Bazar News Agency 1	https://bit.ly/2FcwV0d	0	1	1	0	0	0	1	1	0	0	0	0	Y	N	4
Bazar News Agency 2	https://bit.ly/2T3rgx3	0	1	1	0	0	0	1	1	0	0	0	0	Y	N	4
Bazar News Agency 5	https://bit.ly/2TbG6Ss	0	1	1	0	0	0	1	1	0	0	0	0	Y	N	4
Bazar News Agency 6	https://bit.ly/2Y0DDgY	0	1	1	0	0	0	1	1	0	0	0	0	Y	N	4
Entekhab News Agency 1	https://bit.ly/2O7gO6J	0	1	1	0	0	0	1	1	0	0	0	0	Y	N	4
Entekhab News Agency 2	https://bit.ly/2Jevd2f	0	1	1	0	0	0	1	1	0	0	0	0	Y	N	4
Entekhab News Agency 3	https://bit.ly/2TAWqRE	0	1	1	0	0	0	1	1	0	0	0	0	Y	N	4
Fararu News Agency	https://bit.ly/2UBHU8B	0	1	1	0	0	0	1	1	0	0	0	0	Y	N	4
Iranian Labour News Agency	https://bit.ly/2u4JJ2m	0	1	0	0	0	0	1	1	1	0	0	0	Y	N	4
Net RAY 1	https://bit.ly/2T8X83s	0	1	1	0	0	0	1	1	0	0	0	0	Y	N	4
Net RAY 2	https://bit.ly/2Y0DBWo	0	1	1	0	0	0	1	1	0	0	0	0	Y	N	4
Tabnak News Agency 1	https://bit.ly/2T6GNMO	0	1	1	0	0	0	1	1	0	0	0	0	Y	N	4
Tabnak News Agency 2	https://bit.ly/2HzuI0c	0	1	1	0	0	0	1	1	0	0	0	0	Y	N	4
Khabar Online News Agency	https://bit.ly/2O6QGt0	0	1	1	0	0	0	0.5	1	0	0	0	0	Y	N	3.5
Bazar News Agency 3	https://bit.ly/2TNiOX7	0	1	1	0	0	0	0	1	0	0	0	0	Y	N	3
Bazar News Agency 4	https://bit.ly/2F0gym2	0	1	1	0	0	0	0	1	0	0	0	0	Y	N	3
Rooz Plus	https://bit.ly/2HymsOi	0	1	1	0	0	0	0	1	0	0	0	0	Y	N	3

Appendix 2. Further details for MDR 8-10

Table

Poll	MDR 8	MDR 9	MDR 10
Professional polling organizations
IPPO Group 1	Cellphone and landline	3–6 May, countrywide	3.18 to 3.90
IPPO Group 2	Phone	4–7 May, countrywide	3.10 to 3.90
IPPO Group 3	Phone	5–8 May, countrywide	±2.99 to ±3.91
IPPO Group 4	Cellphone and landline	6–9 May, countrywide	±2.83 to ±3.82
IPPO Group 5	Phone	7–10 May, countrywide	±2.82 to ±3.75
IPPO Group 6	Phone	8–11 May, countrywide	±2.81to ±3.74
IPPO Group 7	Phone	9–12 May, countrywide	±2.83 to ±3.75
IPPO Group 8	Phone	11–13 May, countrywide	±3.08 to ±4.73
IPPO Group 9	Phone	12–14 May, countrywide	±2.93 to ±4.30
IPPO Group 10	Phone	11–14 May, countrywide	2.64 to ±3.92
IPPO Group 11	Phone	15–16 May, countrywide	±2.83 to ±3.89 for the 95% confidence interval
IPPO Group 12	Phone	17 May, countrywide	±2.83 to ±4.06 for the 95% confidence interval (based on responses to questions)
Iranpoll 1	Landline	16-May, countrywide	±3.09%
ISPA 1	Unclear	17–18 April, 22 districts of Tehran	N
ISPA 2	Unclear	21–22 April, countrywide	N
ISPA 3	Telephone poll	7–8 May, countrywide	N
Media organization polls
Aban Press	Online	N	N
Asr Iran News Agency 1	Online	N	N
Asr Iran News Agency 2	Online	N	N
Asr Iran News Agency 3	Online	N	N
Asr Iran News Agency 4	Online	N	N
Bazar News Agency 1	Online	N	N
Bazar News Agency 2	Online	N	N
Bazar News Agency 3	Online	N	N
Bazar News agency 4	Online	N	N
Bazar News agency 5	Online	N	N
Bazar News Agency 6	Online	N	N
Entekhab News Agency 1	Online	N	N
Entekhab News Agency 2	Online	N	N
Entekhab News Agency 3	Online	N	N
Fararu News Agency	Online	N	N
Iranian Labour News Agency	Telephone	16–17 May	N
Islamic Republic News Agency	In-person	1–2 May, capitals of 31 provinces	N
Khabar Online News Agency	Online	N	N
Net Ray 1	Online	N	N
Net Ray 2	Online	N	N
Net Ray 3	Online	1 April–28 April 2017	N
Peace News Agency	Online	21 April–17 May	N
Rooz Plus	Online	N	N
Tabnak News Agency 1	Online	N	N
Tabnak News Agency 2	Online	N	N