Are citizens responsive to interest groups? A field experiment on lobbying and intended citizen behaviour

Abstract

The ability to mobilise public opinion is central to interest group politics. Yet, whether and how groups succeed in swaying the public remains inconclusive. The article assesses this by conducting a field experiment in which a consumer group sent different versions of campaign material to a representative sample of over 5000 citizens. Relying on a two-wave panel survey, it shows that while the campaign affected intended consumer behaviour, it did not influence attitudes. Surprisingly, material by the organisation alone was more effective than material sent with a partner. Moreover, campaign references to personal experiences and facts were not more effective than material referring to public opinion. The findings challenge existing evidence on how sender and message characteristics affect the likelihood of influencing citizens. At the same time, they underline that public opinion is hard to change and have important implications for understanding political representation and interest groups in democratic politics.

Keywords:

• Interest groups
• public opinion
• campaigns
• political representation
• political communication

The public image of lobbying is often one of organised interests meeting directly with politicians brokering deals shielded from the public eye. According to several accounts, ‘lobbying’ even has its roots in actual gatherings between representatives and constituents, such as the lobby of Westminster in London or the former Willard hotel on Pennsylvania Avenue in Washington DC. While this image is still widespread, a substantial share of modern lobbying goes beyond direct encounters between representatives and organised interests. Political competition and lobbying are to an increasing degree taking place in the ‘public sphere’ rather than in ‘the smoke-filled backrooms of parliamentary committees and the central offices of parties and associations’ (Kriesi 2009: 154).

Existing interest group scholarship has not been blind to this development. It typically distinguishes between inside and outside strategies or access. The former here refers to direct contacts of groups to various decision-makers, and the latter to their indirect attempts to influence policy by mobilising the public in various ways (e.g. Kollman 1998; Maloney et al. 1994; Thrall 2006; De Bruycker 2019, Grose et al. 2022; Eady and Rasmussen 2022; Crepaz et al. 2022). When it comes to outside strategies, an underlying assumption is thus that the public is an important vehicle for interest groups to persuade policymakers to adopt their preferred line of action. We know that such outside strategies are widely used and that they can contribute to lobbying success (e.g. De Bruycker and Beyers 2019; Junk and Rasmussen 2019). At the same time, there is evidence that higher media attention can actually both be a curse and a blessing for political advocates who seek to attain specific policy goals (De Bruycker 2019). Given that research on lobbying and political advocacy often focuses on effects on policy, it remains relatively underexplored whether and how public advocacy campaigns actually achieve their intended effect on citizens.

Studies on policy representation, on the other hand, have long had a keen interest in focussing on whether policies actually correspond with what citizens want. However, they can be criticised for assuming that public opinion is exogeneous (e.g. Gilens 2012; Monroe 1998; Rasmussen et al. 2019; Rasmussen et al. 2021), ignoring that the public’s view on specific issues may be influenced by a number of factors, including outside lobbying by organised interests. An important question therefore remains whether and how organised interests can successfully mobilise citizens to care about new issues. Put differently: (how) can interest groups affect citizens through their campaigns? This is a crucial relationship in the political process, given that the receptiveness of citizens to non-state actors like non-governmental organisations (NGOs) and businesses can impact their direct demand for new legislation, as well as incentives for self-regulation in the market (cf. Malhotra et al. 2019).

A small set of studies has now begun to examine this relationship (notably: Broockman and Kalla 2016; Dür 2019; Jungherr et al. 2021; Leeper 2013; McEntire et al. 2015; Weber et al. 2012; Kalla and Broockman 2022). Success in affecting public opinion and behaviour varies considerably depending on the characteristics of the message (cf. McEntire et al. 2015; Dür 2019) and the source or sender of the message (cf. Druckman 2001; Hartman and Weber 2009; Weber, Dunaway, and Johnson 2012). Yet, based on existing evidence, it is still unclear which specific types of messages help interest groups affect the public, and whether it helps them to team up with other organisations to sway the public. We conducted a field experiment, which tests both elements in a setting where a credible, well-known interest group tried to affect citizen attitudes and behaviour on a relatively new issue.¹

Specifically, the field experiment was conducted in cooperation with the largest Danish consumer organisation ‘Forbrugerrådet Tænk’ (FT) on a campaign related to Internet of Things-products (IoT-products), e.g. smart TV’s, smart watches and other internet-connected devices. The broad aims of the campaign were to enhance consumers’ understanding of the privacy risks of using IoT-products and advise them how to protect their data when using these products. These products are regulated by policymakers in various ways (e.g. regulation of wireless communications standards) and raise ongoing concerns regarding the design of legislation to protect consumer privacy. By trying to affect citizen attitudes on this issue, the interest group campaign therefore has potential political effects, because heightened concerns among consumers might impact policymakers’ incentives to regulate further. At the same time, the campaign can have direct effects on consumer behaviour, which might exert market-based effects, including incentives for self-regulation by firms. Our experiment therefore tests both attitudinal effects on the political issue and effects on consumers’ behavioural intent.

Our field experiment was run before the campaign on this new issue was made public in order to avoid potential pre-treatment effects (cf. Slothuus 2016). We randomly assigned different versions of the campaign material, as well as placebo material on an unrelated issue, to a sample of over 5000 Danish citizens, and were able to measure the attitudes and intended behaviour of approximately 3700 respondents in a two-wave panel survey before and after this treatment. Using this causal design, we can estimate the general effectiveness of reaching citizens with information material on this issue. Moreover, we carefully assess variation in these effects depending on (1) the characteristics of the sender, i.e. whether the message is sent alone or as a diverse coalition of credible organisations, and (2) the characteristics of the message, i.e. types of the supplied information in the campaign.

Our findings underline some of the difficulties faced by interest groups in swaying public opinion in practice. Citizens targeted by the campaign do not seem to have become more worried about the campaign issue, or to find the issue more salient compared to other aspects. However, we find evidence that that the campaign affected intended consumer behaviour on the concrete issue. Citizens treated with the campaign report that, the next time they buy an IoT product, they are more likely to check different kinds of information on data protection, such as the privacy policy. At the same time, the findings challenge conventional expectations on how sender and message characteristics affect the likelihood of influencing citizens. Notably, material sent together with a partner organisation is not more effective at changing public opinion than the same material sent by the consumer organisation alone. Furthermore, we do not see that campaigns conveying personal stories are more potent in achieving the desired effect compared to references to facts and public opinion (see in contrast: McEntire et al. 2015). Instead, factual information is the most effective message type to reach the public, according to our findings.

Overall, our findings are important for understanding whether and how organised interests can mobilise and activate the public. They illustrate the ability of interest group campaigns to nudge the public into more cautious behaviour. However, they also show that interest groups face considerable obstacles in affecting public opinion in practice. It seems that concrete, but admittedly small, changes in behavioural intent are easier to achieve than general changes in attitudes, beliefs or issue salience. This arguably makes it less likely that public opinion is strongly endogenous to the activities of interest groups. At the same time, it might mean for practitioners that campaign with clear calls to action can be more potent than general attempts to raise issue salience.

Theory: sender and message characteristics in interest group campaigns

Existing research contains mixed evidence concerning how successful organised interests are in their endeavours to affect the public. On the one hand, Dür’s (2019) survey experiments show that interest groups can affect public opinion through their (strong) arguments. Moreover, McEntire et al. (2015) demonstrate that campaign messages used by human rights organisations can be effective in ‘micromobilization’, e.g. by having an impact on the attitudes of individual citizens (also see: McKnight and Hobbs 2013). On the other hand, other studies present a more modest view regarding the ability of groups to influence public opinion. Leeper’s (2013) survey experiments in the U.S. context show that interest groups have a hard time to shape public opinion. Similarly, a recent field experiment finds only a modest, short-term effect of an intervention by a German business group on public opinion (Jungherr et al. 2021). An earlier, observational study by Page et al. (1987) even found evidence that the overall influence of interest groups on public opinion is negative (also see Smith 2000). In sum, while the ability of interest groups to affect the public is a crucial question for scholars and practitioners of lobbying and public policy, answers are still inconclusive as to whether and how interest group campaigns are effective.

Our study contributes to a better understanding of how the information provided by interest groups potentially persuades the public. Specifically, we focus on diffuse interest groups representing consumer interests, and zoom in on their ability to affect opinion formation and behavioural change. Our main expectation is that new information can affect attitudes and intended consumer behaviour, but that the effects depend on the characteristics of the message and the sender of this information. To formulate hypotheses on the effects of information material distributed by a diffuse interest group, we pair research on outside lobbying (e.g. Kollman 1998) with insights from studies on the effects of information-provision and persuasion (e.g. Leeper and Slothuus 2020; Slothuus 2008; Weber et al. 2012; Zaller 1992).

Our first and basic hypothesis is that receiving new information on an issue from a credible, well-known interest group affects citizen attitudes and behavioural intent. This expectation builds on key insights from studies of opinion-formation: first, studies on information-provision stress that previous knowledge affects the impact of information (Lupia 1994; Seeberg et al. 2017; Zaller 1992). We, therefore, expect especially relatively new information to have the potential to change attitudes and behaviour. Second, there is evidence that information sources matter (cf. Druckman 2001; Hartman and Weber 2009; Weber et al. 2012, but also see: Dür 2019). Accordingly, citizens should be responsive to campaigns by interest groups that are seen as credible sources.

The case of cooperation on the Internet-of-Things (IoT) products campaign with the consumer organisation FT was here chosen and designed to provide a likely scenario for these conditions. The campaign on the issue of privacy and data protection in IoT-products such as smart TV’s, smart watches and other internet connected devices (other than smart phones, computers, tablets), is assumed to be relatively new to citizens, which is why our partner organisation decided to run this campaign. Given our field experiment ran before any campaign material was public, we minimise previous knowledge of campaign arguments (i.e. pre-treatment effects). Secondly, the organisation we worked with is Denmark’s biggest independent consumer organisation, which makes it a suitable case for a well-known, credible source.

Notably, this case of diffuse consumer interests also avoids a strong partisan dimension (cf. Slothuus and de Vreese 2010). We assume data protection and product security to be seen as generally desirable – rather than a question with a left–right positional alignment. Our assumption is that the more citizens learn about this issue (through the information provided by the consumer group), the more they update their attitudes, for instance about the salience of this issue. Put differently, one might see this case as an attempt by the consumer group to provide information on a public good (consumer safety) through outside lobbying (Kollman 1998). Directly addressing the public about this can be seen as an attempt to increase the salience of this issue and motivate a change in attitudes and behaviour.

Based on these assumptions, we formulate our first hypothesis. We expect information provided by the interest group to alert citizens to be more concerned about the issue, attach more salience towards it, and adjust their consumption behaviour by checking and requesting different types of information on the IoT-products when purchasing them.

H1: Citizen attitudes and intended behaviour are affected by (new) information supplied by a (credible) interest group.

In addition to this main effect, we also expect characteristics of the sender to affect the effectiveness of the supplied information. In particular, a key question in interest group studies is whether to act alone or work in a coalition (Beyers and De Bruycker 2018; Hanegraaff and Pritoni 2019; Heaney and Leifeld 2018; Hojnacki 1997; Holyoke 2014; Hula 1999; Junk 2020). Source credibility is likely to be affected by sending information as a coalition of diverse organisations (cf. Leeper 2013). A diverse coalition of two credible sources might here be able to pool and signal diverse expertise to citizens. Recent research on interest groups has, for example, found that citizens rate the legitimacy of policy making higher when a diverse set of interest groups active on a policy issue are consulted rather than adding priority to one type of interest (Rasmussen and Reher 2023). In line with studies in psychology, multiple sources can increase persuasion, especially when the committee (i.e. coalition) of sources includes members with dissimilar perspectives (Harkins and Petty 1981, 1987).

While previous research has focussed on the effects of lobbying in (diverse) coalitions on policymakers and advocacy success (Dwidar 2022; Junk 2019; Phinney 2017; Strolovitch 2007), the responsiveness of citizens to coalitions of interest groups has hardly been addressed. An exception is Leeper (2013) who tests the effects of both larger coalitions and more diverse arguments on citizen opinions. He only finds limited support for the expectation that larger coalitions have influence on citizens’ opinions. Importantly, he argues that unlike political parties, interest groups are hardly able to provide strong cues to citizens, because they are less salient and established sources. In another experiment, Weber et al. (2012), however, find that ads sponsored by a fictional interest group were more persuasive than those sponsored by political candidates. Put differently, based on existing evidence, it is still unclear to what extent and when interest groups as ‘sources’ can affect the public, and whether it helps them to team up.

In order to address this, we add what we see as a likely case for the effect of information provided by a coalition of interest groups on citizen attitudes. The coalition consists of two actors: the Danish consumer association Forbrugerrådet Tænk (FT) and an association of professionals called ‘Danish IT’ (Dansk IT). While the consumer organisation is an all-rounder on consumer issues, this second organisation is highly specialised and should – already based on its name – signal expertise on IT-related questions such as hacking, data sharing and security. Therefore, we expect that Danish IT, which has nearly 10,000 IT-professionals as members, provides additional credibility on this highly technical issue. By diversifying the (consumer protection and tech savvy) organisational perspectives in the campaign, we expect to boost persuasion of the campaign (cf. Harkins and Petty 1987). Consequently, we hypothesise that the information sent by this coalition of two complementing interest groups will be more effective than information sent out individually by the consumer organisation.

H2: Citizen attitudes and intended behaviour are more likely to be affected by information supplied by a coalition of interest groups, which pools more resources on the issue, than by an individual interest group.

Finally, we draw on existing research to address that the content of the message, i.e. the type of information employed, is likely to affect the effectiveness of the campaign material (e.g. Slothuus and de Vreese 2010; Petty and Cacioppo 1984; McEntire et al. 2015). While the literature uses different terms for such message characteristics including ‘arguments’ and ‘frames’, we follow Leeper and Slothuus’ (2020: 156) reasoning that information-based persuasion occurs when opinion changes in ‘light of new information’, whereas frames emphasise certain aspects of an issue, without themselves providing new substantive information (Leeper and Slothuus 2020: 154).

There are many types of information that could be studied here, and the selection of comparable informational stimuli of different types is inherently challenging. We draw on McEntire et al. (2015) who identify three common messaging techniques for mobilisation by diffuse interest groups as (1) informational, (2) personal and (3) motivational content. However, we conceive of both messages about personal experiences, as well as about quantifiable facts as potential sources of new ‘information’, though of different types. Moreover, we argue that quantifiable facts can be of technical nature, reporting on the frequency of different outcomes, or of subjective nature, drawing on aggregated individual positions (i.e. public opinion).

In our field experiment, the core message on data protection in IoT-products is therefore conveyed in three different ways: (1) in form of a personal individual story about the involuntary collection of data, (2) in a factual manner by focussing on the frequency of security problems and automatic data collection in IoT-products, (3) and in a public opinion-based form that gives information about the share of the population that is worried about data security and hacking. Importantly, these types of messages also resonated well with our partner organisation and fitted the extensive informational material they had prepared for their campaign.

It is practically and theoretically important to tease out whether these different types of information vary in their effectiveness when wanting to persuade citizens. A long-standing debate revolves around whether personal (individual) and/or quantified information is effective in political communication (cf. Aarøe 2011; Marcus 2000; Mérola and Hitt 2016). Based on existing work on political advocacy, we expect the personal content to be likely to provoke an emotional response and therefore to have high potential to affect attitudes and behaviour (McEntire et al. 2015). The factual content, however, is expected to be effective in different ways (see also Larsen and Olsen 2020), namely by signalling high informational resources and increasing source credibility, as has been suggested in the context of the credibility of newspaper articles that use numbers as facts (Koetsenruijter 2011). Finally, we add an additional quantitative informational type, based on aggregate subjective accounts: the public opinion-based content is expected to underline the salience and scope of the problem and tap into the tendency of bandwagoning with a popular opinion (Marsh 1985). Overall, we expect all three information types to have the potential to affect public attitudes and behavioural intent. However, we expect especially the personal and factual messages to be effective, more so than the (so far less studied) public opinion-based message, as hypotheses 3 and 4 summarise.

H3: Citizen attitudes and behaviour are more likely to be affected by personal stories than by opinion-based information.

H4: Citizen attitudes and behaviour are more likely to be affected by factual information than by opinion-based information.

Analysis design

Our research was conducted in Denmark among a representative sample of the population, recruited by the survey company Epinion. Our study measures exposure to a real campaign of the largest Danish consumer organisation FT. This means that the organisation decided on the topic and main message of the campaign that it wanted to communicate to the public, which ensures ecological validity. At the same time, as argued above, the issue of data security in IoT-products was deemed highly suitable for the experiment, because it was a relatively new issue, which does not normally have high salience and which the consumer organisation hoped to inform the public about. Given the experiment was completed before the actual campaign went public, we can reasonably avoid contamination and some of the pre-treatment effects common in many experiments (Slothuus 2016).² Our research design was approved by a University Ethics Board (see Appendices Table B.1 for a summary of ethical considerations in the study).

Design and implementation of the field experiment

To prepare the material for the experimental treatments, the key message of the campaign was communicated using three different types of information: personal, factual and public opinion content in material either sent by the consumer organisation alone (single logo), or jointly with its IT-focused coalition partner (including both logos). The campaign flyers were produced by our partner organisation with the assistance of our research team, and used the organisation’s templates for campaign flyers, thus ensuring they were highly realistic (see Online Appendix B).

To ensure that stimuli material conveyed the same campaign message, but with different types of information to support this message, all flyers were identical in the initial message (‘Protect your personal data and avoid getting hacked’) and presented context (‘[…] Several of the products you use in your daily life may be connected to the Internet and collect information about you.’). Yet, the concrete information to support this claim varied in the different treatment conditions with personal, factual and public opinion-based information of similar length.³ These varying treatment vignettes are displayed Textbox 1 (in English translation). After the informational treatments, all flyers presented identical implications and recommendations for consumers (‘How to prepare yourself’). The full campaign material is included in Online Appendix B.

Textbox 1: Information Type Treatments (translated)

Table

Treatment, personal

As an example, smart TV’s can both share data with third-party companies and be misused by hackers. Kez Garner, who heads a technology consulting firm, has seen several cases of cybercrime. One of her client's children received a smart TV as a gift. But it turned out later that the TV was used to spy on the family. The story about Kez Garner's client underscores the need be aware of the protection of your personal data when purchasing new electronics.

Treatment, public opinion-based

Forbrugerrådet Tænk has recently conducted a survey about products connected to the Internet among more than 1,000 Danes between 18 and 79 years old. The survey found that 56 per cent of the Danes are worried or very worried that the products collect information that is shared with other companies. An even higher share (69 per cent) state that they are worried or very worried that the products may get hacked.

Treatment, factual

As an example, smart TV’s can both share data with third-party companies and be misused by hackers. A test of smart TV’s conducted by the American consumer organization Consumer Reports showed that 5 out of 5 tested smart TV’s collected data on the TV habits of the users. This type of data can be shared with third-party companies in order to target advertisements to users on their phones and computers. The Consumer Reports’ test also found that 2 out of 5 tested smart TV’s had security flaws that allowed a hacker to control the sounds of the TV; its programs and internet connection.

In addition, two placebo treatments were produced where respondents received information about an unrelated issue – pesticides in food products – either sent by the consumer organisation alone or by the coalition (Online Appendix A provides an overview of the treatment groups and control groups). This material also used real information from a previous campaign.

Our study was designed as a field experiment, where respondents received the campaign material independently of the two waves of our survey (pre- and post-treatment) to collect data on the dependent variables (for a similar design compare: Larsen and Olsen 2020). The survey company Epinion was responsible for the practical execution of the survey. Data collection began with a pre-testing survey between 19 September and 14 October 2019 to ask citizens, among other things, about their pre-existing attitudes and consumer behaviour related to IoT products. The survey also requested the consent of participants to be sent further information of a non-commercial character via email without specifically disclosing what the contents of this material would be. 11,436 citizens participated in this pre-testing survey (i.e. wave 1), of which 5174 (45%) gave consent to receive information material.

Within this sample of consenting respondents, different versions of campaign material were randomly assigned between the treatment and placebo groups with material being divided equally between them. This treatment and placebo material was sent out as a flyer via email on 28 October 2019. Notably, this means that our field experiment closely mimics interest groups’ real effort to reach the public, for instance in ads and newsletters. However, instead of testing the effect only on members, we conducted the experiment on a representative sample of the Danish population, which also was the target group of the broad campaign. This allows us to generalise about the effect of the interest group’s campaign on the broader public.

Subsequently, an after-treatment survey (i.e. wave 2) was sent to all participants, including the same questions about their attitudes and intended consumer behaviour with respect to IoT-products. Data for wave 2 were collected between 29 October and 17 November 2019. All respondents received two reminders, and the last reminders were sent on November 13. Of the 5174 respondents who gave consent to receive information material, 3828 also answered the subsequent survey, which gives a response rate of 74% in wave 2. Only after the field experiment was finalised, did the consumer organisation start a publically-oriented campaign on the issue, which involved adds on social media, in traditional news media etc.

Survey measures for the dependent variables (attitudes and behavioural intent)

Our study examines the impact of different versions of the campaign material on both attitudes and intended consumer behaviour. We included three measures of attitudes in both waves 1 and 2 of the survey: The first looks at the respondents’ level of worry that personal data is shared with others when using IoT-products (Attitudes: Worry about data). This is measured on a five-point scale, ranging from 1 = not worried at all to 5 = very worried. The second collects information about the respondents’ level of worry about being hacked when using products connected to the internet (Attitudes: Worry about hacking). As before, this is measured on a five-point scale, ranging from 1 = not worried at all to 5 = very worried. Finally, the third measure of attitudes captures salience by asking citizens about how important they find personal data security when buying electronical products – compared to other factors such as price, design etc. (Attitudes: Salience). This item is measured on a five-point scale, ranging from 1 = much less important to 5 = much more important. These three measures of attitudes were aligned with aims of the campaign to raise awareness about issues of data protection in IoT products. Since the nuances between them are potentially informative, we address their effects separately in the analysis.

Given that the campaign was, furthermore, aimed at mobilising citizens into acting more cautiously in their consumer choices, the surveys also measured the claimed intention to check information on IoT-products. Specifically, we include four measures of intended behaviour, all of which look at the likelihood that the respondent takes different actions the next time (s)he buys a product: First, we measure the reported likelihood to read the product’s privacy policy (Behaviour: Read private policy). Second, we include the reported likelihood that the respondent reads information about other terms of use (Behaviour: Read other product info). Third, we include taking action by asking the shop assistant about the product’s data security (Behaviour: Ask about data security). Fourth, given how wide-spread online-shopping is, we alternatively measure the likelihood of reading online recommendations and tests about the product before buying a product (Behaviour: Read Online Information). In the survey, all these questions were measured on an 11-point scale ranging from 0 = not likely at all to 10 = very likely. Again, given it is practically relevant to address how information-seeking changes in particular, we address these activities individually in the analysis.⁴

Hypotheses tests

We test the effects of the treatments in regression analyses of the changes in our attitudinal and intended behaviour variables between waves 1 and 2 at the individual level. This has the advantage of taking pre-treatment levels into account, and evaluating whether a treated individual was moved by the campaign in the desired direction. To construct these dependent variables, we subtracted the wave 1 observation for the attitudinal and behavioural measures for each individual from the respective wave 2 observation. Where these measures are positive, the individual has, for example, increased worries, salience of the issue, or the likelihood of gathering information about IoT products after the treatment. Where it is zero there is no change, and negative values indicate fewer worries, lower salience, or lower likelihood of information gathering after the treatment.

In order to test the effects of the treatments, we present two complementing types of analyses, each with approximately 3700 observations. First, we use OLS models to estimate the intention-to-treat (ITT) effect, that is, testing for differences in our outcome variables between the treatment and control groups among all respondents that participated in the two waves of the experiment. Notably, this analysis gives the average treatment effect in the whole sample, but it does not take into account that there will be non-compliance, i.e. some individuals were treated with the campaign but did not actually read the email flyer.

Second, we estimate the average treatment effect on the treated (ATT) for those respondents who can correctly recall the campaign material. This analysis takes non-compliance into account, that is, some citizens received the treatment, but were not actually treated, because they did not open or read the email. Of course, ‘actually’ being treated includes self-selection, because some (types of) respondents are more likely to choose to read the material they were randomly assigned (see Online Appendix D for a comparison between compliers and defectors). Therefore, we use an instrumental variable approach and two-stage least squares regressions to estimate the effects on those actually treated (Gerber and Green 2012).

We here measure whether respondents complied, i.e. were actually treated, based on answers to two follow-up questions in our second survey. Respondents were asked both whether they recall being contacted by the consumer organisation and which kind of material they received. Those who identified receiving the correct material were regarded as actually treated. Given this is likely to be affected by selection bias, we use the random assignment of the treatment in the experiment as an instrument for actually being reached with the material, as is common in this approach (see Gerber and Green 2012). The allocation into the treatment group has the advantage of being correlated with whether respondents complied, while we would expect it to be uncorrelated with the regression error since the treatment is assigned randomly. Through the two-step model with this instrument for compliance, our estimations of the average treatment effect on the treated (ATT) therefore adjust for the non-randomness that is introduced by self-section into compliance.

Balance checks and detectable effect sizes

The surveys also collected information about respondents’ demographic and socio-economic characteristics and their knowledge of IoT products, which we used to check whether the random assignment of treatments was successful. We show in Online Appendix D that respondents receiving the two placebo treatments and the six IoT treatments are relatively similar with respect to key background characteristics, even if responses vary somewhat with respect to gender and political orientation. We therefore also present robustness checks of all regressions that control for these covariates in Online Appendix E (Tables E1–E9, respectively). These analyses underline that the findings reported are robust to including these covariates. Furthermore, Online Appendix D shows that our sample sizes (in the different treatment groups) are suited to trace relatively small effect sizes (see Figure D1 and Table D4).

Figure 1. Effect of Treatment versus Placebo (n: 3551–3828) (coefficients from OLS and two-stage least squares regressions with 95% and 90% confidence intervals). The figure displays the ITT effects from Models 1–7 and ATT effects from Models 8–14 in Table E1 in Online Appendix E.

Analysis: the effects of the IoT-campaign on attitudes and behavioural intent

In the following, we probe the effects of the campaign by, firstly, looking at differences in changes in attitudes and intended behaviour for individuals receiving treatment and placebo material. Thereafter, we explore differences in the effects of the treatment conditions by varying the sender and informational content of the campaign material.

General effects of the campaign

To begin with, we assess whether there is variation between the treatment and placebo groups in how individual attitudes and behavioural intent have changed during the campaign. Notably, assessing the differences in responses between these groups allows us to estimate the effect of the intention-to-treat (ITT), that is, the effect of being sent the campaign material. This is a relevant measure for interest groups in practice, as they presumably would like to know whether and how a campaign affects the whole population that they target with a campaign. At the same time, these measures to gauge the causal effects of the treatment will be subject to noise due to non-compliance of some citizens in this treated group. Our analyses therefore also show the average treatment effect on the treated (ATT) for those respondents who can correctly recall the campaign material, while adjusting for potential selection biases in compliance. Note that all full results are shown in table form in Online Appendix E.

Starting with the ITT effects (right side of Figure 1), we see, contrary to our expectations, that there is no significant effect of treatment on any of the three outcome variables measuring attitudes. For two of the measures, i.e. worries about data and perceived salience of IoT products, the effects are not even in the expected direction. These findings underline that attitudes are hard to change by interest groups.

When it comes to intended consumer behaviour, however, there is (weak) evidence in the intention-to-treat analysis that the campaign affected citizens. Here all the coefficients are in the expected direction, and for two of the measures (Behaviour: Read online information and Behaviour: Read private policy), these effects are significant at the 0.10 and 0.05 level, respectively. In line with hypothesis 1, respondents receiving treatment experience an increase of 0.15 points (on the 0–10 point scale) in their likelihood of reading the privacy policy next time they buy a product (ITT: β = 0.15; SE = 0.09; p < 0.10). Similarly, participants treated with campaign material score 0.19 points higher on the scale measuring their likelihood of reading online tests and recommendations next time they buy an IOT product (ITT: β = 0.19; SE = 0.09; p < 0.05).

Judged based on the respondents’ answers in two follow-up questions, it is clear that a considerable hurdle of the campaign has been to actually reach citizens with the campaign flyer – at least according to the respondents’ own memory. In our case, 18% of the respondents in the treatment condition recalled the campaign material correctly.

The left-hand side panel of Figure 1 presents the average treatment effect on the treated (ATT) from the two-stage regression, which takes the non-random nature of non-compliance into account. These as-treated effects are considerably larger than those estimated in the ITT analysis. Citizens that recall receiving campaign material increase their scores with respect to the likelihood of reading the privacy policy and online tests and recommendations about a new product with 0.83 points (SE = 0.50; p < 0.10) and 1.07 points (SE = 0.55; p < 0. 05) on the 0–10 point scale compared to those who received the placebo. At the same time, the confidence intervals for the estimated effects become a lot larger as a consequence of the relatively low share of treated respondents.

In interpreting the share of complying respondents (18% of respondents in the treatment condition), we need to consider that it can also be the case that respondents have indeed been treated, although they cannot consciously recall having read the material from the Danish Consumer Council when answering the follow-up question in the second survey. On the other hand, it is also possible that some respondents report having received the treatment material without having read it. We argue that this uncertainly makes it informative to look both at the intention-to-treat effect and the two-stage model based on this arguably conservative subsection of respondents as actually treated. Notably, while the confidence intervals become wider and point estimates in the two-stage model higher, we see similar trends (with similar significance levels) in both analyses.

Overall, the first conclusion from looking at changes in attitudes and intended behaviour at the individual level between the treatment and control group gives a relatively sobering impression about the possibility of affecting the public: Only for two out of seven relevant dependent variables do we find significant effects of the treatment. And interestingly, it is intended behaviour – rather than general worries about, or salience of the issue – which the campaign was able to affect. While some of the differences in intended consumer behaviour are also relatively small in absolute terms, we see more evidence here that the campaign can tip consumers towards wanting to be more attentive in their future purchasing decisions. The ATT effect sizes (0.83 and 1.07 points on the 0–10 scale) illustrate that an interest group can have a substantial impact in practice provided it succeeds in reaching the participants. An interesting aspect when analysing these differences in intended consumer behaviour is that it does not seem to be fear and emotions about the issue that drive more cautious behaviour. Those treated or actually reached with the campaign material, are equally or less worried than those in the placebo group. However, they report a higher likelihood of intending to check different types of information before buying a product next time. Practically this might mean that concrete calls to action in advocacy campaigns might be more promising than trying to affect general attitudes, perhaps because following them gives citizens a feeling of efficacy (Moe 1981). At the same time, these average effects across all our treatment conditions might also conceal more nuanced effects of the campaign characteristics that we varied in the experiment. To address these, we now attend to characteristics of the sender and message in the next sections.

Effects of sender characteristics: individual versus coalition effort

First, we disaggregate the treatment condition to distinguish between the coalition and individual sender treatments. This analysis nuances our findings by showing that the campaign material sent by the consumer organisation alone actually had a significant effect on three out of four measures of intended consumer behaviour (see the bottom panels of Figure 2). In contrast, the material that was sent jointly with the IT-expert group (Dansk IT) does not have any significant effects on a change in attitudes or intended behaviour compared to receiving placebo material (upper panels of Figure 2).

Figure 2. Effect of Coalition Treatment vs Placebo and Individual Treatment vs Placebo (n: 2226–2413) (coefficients from OLS and two-stage least squares regressions with 95% and 90% confidence intervals). The top panel displays the ITT effects from Models 1–7 and ATT effects from Models 8–14 in Table E2, and the bottom panel the ITT effects from Models 1–7 and ATT effects from Models 8–14 in Table E3 in Online Appendix E.

Looking more closely at the top panels of Figure 2, we see that receiving a joint flyer with logos from the Danish consumer council and its partner Danish IT never significantly affects attitudes or intended behaviour compared to receiving placebo material. It is therefore clear that trying to affect the public as a coalition is no straightforward recipe for success, which backs up tentative findings by Leeper (2013). While evidence in psychology suggests, that a higher number of sources can increase persuasion (Harkins and Petty 1981, 1987), our experiment underlines the risks of cooperation between interest groups in their public campaigning. Whether the IT-group was seen as a less credible source than we expected, or whether the multiple senders confused recipients or diluted the message is difficult to say. Clear is, however, that campaign effects do not hold in the subsample that was sent by this diverse and well-equipped coalition.

Looking at the bottom panels of Figure 2, we see that the treatments sent individually by the consumer organisation do considerably better. Again, we do not find evidence that campaign material can sway attitudes. However, the pictures looks different for measures of behavioural intent. The positive effect of the treatment is here significant at the 0.05 level for three out of the four measures. In the ITT analysis, respondents who were assigned a flyer from the consumer organisation alone scored between 0.21 and 0.24 points higher on the 0–10 scales measuring their intention to read about privacy policy (Behaviour: Read private policy), other product terms of use (Behaviour: Read other product info), as well as online information and tests (Behaviour: Read online information), compared to those who received the placebo. The bottom left hand-side panel shows that in the ATT analyses based on the instrumental variables approach, these effects are as large as 1.13 and 1.32 points for the treated respondents. The fact that these effects are even larger than those presented in Figure 1 reflects that it is the individual, rather than the coalition treatments, that are driving the overall effects.

In order to test hypothesis 2, we specifically compare the effects of the individual and coalition treatments (see Online Appendices, Table E4). They show that the coalition treatment is never more effective than the individual treatment and that on two of the four measures for intended behaviour it even performs significantly worse than the individual treatment. If anything, the individual campaign was thus more effective than the coalition campaign in affecting citizen behaviour, even with what we deemed as a credible and informed partner. This means that hypothesis 2 is not supported.

Message characteristics

Figure 3 presents the results of analyses that examine how effective the different types of information are in changing attitudes and intended behaviour. We do not find any significant differences either when regressing the change in attitudes on the different types of information. For all three types, the impact on changes in our first three dependent variables measuring attitudes is insignificant compared to the placebo effect.

Figure 3. Effect of Different Message Types vs Placebo (n: 1771–1951) (coefficients from OLS and two-stage least squares regressions with 95% and 90% confidence intervals). The top panel displays the ITT effects from Models 1–7 and ATT effects from Models 8–14 in Table E5, the middle panel the ITT effects from Models 1–7 and ATT effects from Models 8–14 in Table E6, and the bottom panel the ITT effects from Models 1–7 and ATT effects from Models 8–14 in Table E7 in Online Appendix E.

Regarding effects on intended behaviour, however, it is mainly the factual content that has significant effects on citizens, whereas the personal and opinion content show no or only one weak effect, respectively. As displayed in the middle panel of Figure 3, respondents receiving a treatment with the factual content (on the frequency of data breaches in smart TVs based on a test study), experience the expected increase in all four measures of behavioural intent compared to respondents that received the placebo. The effects on the respondents’ intention to read the product’s terms of use (Behaviour: Read other product info) and online recommendation and tests (Behaviour: Read online information) are here significant at the 0.05 level. The effects on intention to read the privacy policy (Behaviour: Read online information) and ask a shop assistant about data security (Behaviour: Ask about data security) are significant at the 0.10 level. Regarding effect size, the effects range from 0.19 to 0.29 points in the ITT analysis. When it comes to calculating the effects for ‘compliant’ respondents in the two-stage model, the effects range between 1.10 to 1.66 points on the 0–10 scale. From the perspective of an advocacy organisation, these can be seen as considerable effects that might be unlocked by conducting and communicating a ‘factual’ study about a new issue.

In contrast, the top panel in Figure 3 shows that the message based on a personal story does not have the intended effect. When comparing the effect of campaign material, which communicated the campaign message based on a true case of a family that was spied upon through their smart TV, none of the ITT and ATT effects are significant. Similarly, the bottom panel of Figure 3 indicates that there is limited evidence for a benefit of conveying a campaign message through existing public opinion, in this case fears about personal data being shared without consent, on the behavioural intent variables. An exception is the last behavioural measure of reading online tests and recommendations (Behaviour: Read online information), where we find a marginally significant effect. On this measure, respondents who were assigned a public opinion vignette scored 0.21 higher on the 0–10 scale in the ITT analysis. Among those treated, the ATT effect is 1.22 points.

In order to test the effects of our hypotheses, we compare the effects of the different types of messages (see Online Appendices, Tables E8 and E9). We find no support for the prediction in hypothesis 3 that citizen attitudes and behaviour are more likely to be affected by personal stories than by opinion-based information. There is never a significant difference in the effects of the two types of messages on our outcome measures and the biggest effect sizes even point in the opposite direction. Instead, there is more suggestive evidence for hypothesis 4 that the factual information is more efficient than the public opinion one. On 5 out of the 7 outcome variables, the differences are in the expected direction. Yet, they are never substantial enough to be statistically significant. Instead, the biggest differences are found when comparing the effect of the fact-based and personal, cased-based content. Here the fact-based treatment has a significantly stronger effect for two of the four behavioural outcome measures (at the 0.05 and 0.10 level respectively). In sum, analyses of differences between the different types of messages primarily point to some significant effects of the fact-based information.

Future research should further address this pattern of potentially convincing the public through arguments building on facts as a driver of changes in behavioural intent (see also Larsen and Olsen 2020). One explanation for the null (or sometimes even negative) effects of the case-based content in comparison to other types of information and placebo material could, of course, be that the personal story used in our study was not strong enough (i.e. a family spied upon through their smart-TV). Another is, however, that the use of personal narratives might not work equally well for all group types and all issues. While McEntire et al. (2015) showed that a personal narrative worked well for a human rights organisation working on sleep deprivation as an interrogation technique, the consumer organisation in our experiment clearly did best when using factual information on findings from a test on the frequency of data sharing in IoT products. A key finding in interest group research is that the characteristics of the issue crucially affect lobbying strategies and success (e.g. Klüver 2011; Baumgartner et al. 2009; Rasmussen et al. 2018; Mahoney 2008). Future studies could address in an experimental context in which ways this also holds for the ability of interest groups to affect public opinion with different types of messages on different kinds of issues.

Conclusion

An interest group considering to ‘go public’ to make a difference faces many difficult questions. One of them is whether a public campaign is actually likely to affect public opinion, and/or nudge citizens to act in a different way. Another is whether a potential campaign should be run alone or with partner organisations, and which kinds of messages are best suited to reach citizens. In this article, we provided some answers to these questions based on a field experiment in a situation where a credible, well-known interest group campaigns on a relatively new issue. Other than concrete questions in campaign design, our findings speak to the important broader question of whether public attitudes are endogenous to the activities of interest groups.

Although we arguably picked a likely case for affecting public opinion, because our partner organisation was a high-profile, independent consumer group advocating for the protection of a public good (data security), we found that the campaign was unable to move public attitudes about the issue in the desired direction. Given our large number of observations (>3700), which should allow us to detect even relatively small effects (see Online Appendices), we conclude that public attitudes are generally hard to affect for an interest group. For an interest group, trying to reach policymakers through affecting the attitudes of the public can thus be a long shot. While a possible explanation for this finding might lie in the relatively technical, and thus potentially ‘boring’, characteristics of the issue, it adds to existing evidence that interest groups face more challenges when trying to sway the public (cf. Jungherr et al. 2021; Leeper 2013) than political parties (Slothuus and de Vreese 2010).

On a more positive note, however, our study also highlights the potential educating and instructive role of interest groups. Even though worry about and salience of the issue were not increased by the campaign, it significantly affected how likely citizens were to intend to act more cautiously next time they bought an Internet-of-Things product. Put differently, we find that concrete intended behaviour might actually be easier to change than attitudes (see in contrast: McEntire et al. 2015). This is a relevant finding for interest groups: a campaign with concrete calls for actions that citizen can adopt at low costs (like acquiring different kinds of information in our example), can harness positive effects. At the same time, our findings raise the puzzling question of how the campaign affected intended behaviour without changing attitudes (cf: theories of planned behaviour; Ajzen and Fishbein 1977). We reason that citizen’s desire for efficacy, which has long been seen as a driver of interest group mobilisation (Moe 1981), also plays a role in responses to new information and could explain (intended) behavioural responses. While we cannot test such a potential intermediary mechanism with our data, we hope that future empirical research into advocacy campaigns takes such relationships into account.

Surprisingly, we find no significant effects when adding a second credible expert group on the issue to the campaign. The campaign that was communicated by the diverse coalition of the consumer organisation and the issue-expert did not have significant effects on intended behaviour and was, if anything, inferior to a campaign run by an individual group. This shows that a second source does not simply magnify the message of a lobbying campaign, linking fruitfully to seminal work in psychology (Harkins and Petty 1987, 1981): multiple sources might need to complement each other more explicitly than in the present campaign, for instance with different perspectives and potentially different arguments, in order to persuade the public (see also Leeper 2013). In addition, our findings qualify existing knowledge on the kinds of messages that seem more potent in swaying the public. Unlike McEntire et al. (2015), we find significant effects especially where the campaign’s message was communicated based on facts and not when conveyed through a personal story. We also found no support for the expectation that public opinion-based information would be less effective than either facts or personal content.

While our experiment focussed on changes in intended behaviour of citizens as consumers on the market, we reason their actions can lead to political effects, for instance by incentivizing firms to self-regulate (Malhotra et al. 2019). At the same time, given our results showed that moving public attitudes is difficult, such effects of interest group campaigns through the market-venue might be a fruitful avenue for future research.

Moreover, future studies should address receiver (i.e. citizen) characteristics, that might moderate responses to an interest group campaign, but which our study did not explore. Similarly, given we only focussed on one issue, we could not address how issue characteristics, such as complexity and salience (e.g. Baumgartner et al. 2009; Mahoney 2008) affect the effectiveness of lobbying campaigns in reaching the public. Our field experiment has contributed by showing in a realistic setting of an actual (but at that point unpublished) campaign, that a well-known, credible source can affect intended consumer behaviour on a quite technical issue – especially when acting alone and using factual arguments. We did not, however, include measures of actual participant behaviour, which would involve embedding calls to observable action (e.g. information searches or donations) in experimental designs in the future. Our study illustrates that experimental designs in cooperation with organisational partners are both feasible and highly insightful when wanting to study the effects of interest group campaigns. We hope that similar designs will be adopted in other countries, for other issues and/or interest group types, and with different theoretical foci, in order to add more causal evidence to our knowledge of how outside lobbying works.

Studies of individual campaigns also have potential when it comes to understanding the broader picture of lobbying that takes place in the ‘public sphere’ rather than in ‘the smoke-filled backrooms’ (Kriesi 2009: 154). For those sceptical about the role of lobbying, it might be reassuring to learn that interest groups face clear limitations when trying to manipulate public opinion. For those believing that interest groups can play a positive role in informing the public about important new issues, this can be somewhat sobering. Public attitudes – even at the individual level of citizens reached with a campaign – are not easy to influence by interest groups. Still, as we also showed, carefully designed campaigns have the potential to make citizens resolve to gather additional information on an issue.

Acknowledgements

Earlier versions of this article were presented at the 2020 Annual Meeting of the American Political Science Association and seminars at the Danish Consumer Council in August 2020 and King’s College London in May 2021. We thank all participants, in particular, Marcel Hanegraaff and Tom Holyoke for excellent comments. We are also grateful to Gregory Eady and Asmus Leth Olsen, as well as Wolfgang C. Müller and two anonymous reviewers, for their valuable feedback and suggestions. We would also like to thank Michael Thage Wohlert and his team at the Danish Consumer Council for placing trust in us and participating in our research. Finally, we are grateful to Emilie Kirk and Trine Brigsted Jensen from Epinionglobal for assisting us in executing our two-wave citizen survey, and to Nicholas Buhmann-Holmes and Anna Jakobsen for excellent research assistance.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This work was supported by the ERC Consolidator Grant (864648) and Samfund og Erhverv, Det Frie Forskningsråd (0602-02642B).

Notes on contributors

Wiebke Marie Junk

Wiebke Marie Junk is an Assistant Professor in Public Policy at the Department of Political Science, University of Copenhagen. Her research examines the role of interest groups, firms and expert actors in political decision making. It appears, for instance, in the American Journal of Political Science, Comparative Political Studies, Journal of European Public Policy, and Governance. [[email protected]]

Anne Rasmussen

Anne Rasmussen is a Professor in Political Science at the Department of Political Economy, King’s College London and the Department of Political Science, University of Copenhagen. In addition, she is a Professor II at the Department of Comparative Politics, Bergen University. Her current research focuses on organised interests, policy representation, and social media. It has for example been published in the British Journal of Political Science, Comparative Political Studies, Political Behaviour, the European Journal of Political Research and the Journal of European Public Policy. [[email protected]]

Notes

1 The design of our study and its hypotheses were pre-registered at EGAP (hosted by the Center for Open Science's OSF Registries) before the data was accessible to us. Note that we do not test hypotheses regarding receiver characteristics and time in this article.

2 Our two-wave survey design, where consent to receive some email material is granted in wave 1 might, however, mean that participants are aware that they are being observed. In principle, this could affect their response to the stimulus material, in form of experimenter demand effects (EDEs) or lower responses to the (therefore potentially less ‘realistic’) stimulus. Yet, given that the material topic (and research aim) were not disclosed, and given the otherwise high ecological validity of the campaign material, we do not expect strong pre-treatment effects in this sense.

3 Stimuli choice was based on extensive discussions with our partner, the consumer group, and the aim to keep the main message and calls for action based on the campaign constant. Unfortunately, we did not systematically pre-test stimulus strength beforehand.

4 Table C1 in Online Appendix C shows the correlation between our seven dependent variables. While all are positively correlated, all of the 21 pairwise correlations are <0.53 and 20 of the 21 are <0.36. This indicates that the change in one of the measures is generally only a weak predictor of the others. It is therefore insightful to analyze these changes as separate dependent variables.