Assessing the prominence of interest groups in parliament: a supervised machine learning approach

ABSTRACT

Ascertaining which interest groups are considered relevant by policymakers presents an important challenge for political scientists. Existing approaches often focus on the submission of written evidence or the inclusion in expert committees. While these approaches capture the effort of groups, they do not directly indicate whether policy makers consider these groups as highly relevant political actors. In this paper we introduce a novel theoretical approach to address this important question, namely prominence. We argue that, in the legislative arena, prominence can be operationalised as groups being mentioned strategically – used as a resource – by elected officials as they debate policy matters. Furthermore, we apply a machine learning solution to reliably assess which groups are prominent among legislators. We illustrate this novel method relying on a dataset of mentions of over 1300 national interest groups in parliamentary debates in Australia over a six-year period (2010–2016).

KEYWORDS:

• Interest groups
• prominence
• parliament
• representation

Introduction

The interest group literature accepts as uncontroversial a claim that ‘being noticed by those in power is crucial for organized interests’ (De Bruycker & Beyers, 2015, p. 453). Nevertheless, there has not been a great deal of direct analysis of whether groups are indeed recognised as a relevant resource by policy makers. Instead, work has tended to focus on explaining variations in the involvement of groups (e.g. writing submissions) or on privileged access of groups to policy making venues, such as invitations to sit on advisory bodies or to give oral testimony to legislative committees (e.g. Pedersen, Halpin, & Rasmussen, 2015).

This work has been productive and is highly relevant to understanding the role of groups in policy processes. However, it is different from the analysis of another form of policy engagement, namely group prominence among political elites. Prominence can be defined as the situation where ‘a group has pre-eminence for a particular constituency or viewpoint, and is therefore ‘taken-for-granted’ by a prescribed audience’ (Halpin & Fraussen, 2017). In the context of this paper, we focus on prominence of interest groups among political elites in the parliamentary arena, and seek direct evidence that elected members of the legislature reference groups in political debates. Political elites are unlikely to reference interest groups in their discussions unless they carry some weight in the issue under discussion, and by serving as some kind of placeholder or signifier for an argument or issue perspective. Similar to strategic behaviour in the context of parliamentary votes, policymakers will think carefully about which groups they mention (see Bütikofer & Hug, 2015).¹

For interest groups, being acknowledged by policy makers as a relevant actor, whose preferences should be taken into account, is a valuable asset. If a group is not considered a prominent or key player in a given policy issue, their views risk being overlooked or ignored completely. Whether mentioning a group conveys it is an enemy, an ally, a credible or reputable source of information, political elites presumably reference them because the mere mention of their name conveys something to other policymakers and to the public at large. Furthermore, being referred to by elites in routine policy discussions can be valuable to groups in relation to demonstrating their value to internal stakeholders (i.e. members and boards) and signalling reputation to their peers (i.e. other interest groups). Yet, interest organisations face considerable difficulties in gaining prominence among political elites. Indeed, policy scholars have consistently highlighted how difficult it is to attract the attention of elites: the finite attention span of such individuals means that much will inevitably be ignored (Jones & Baumgartner, 2005; Jones & Thomas, 2012; Walgrave & Dejaeghere, 2017). As elites cannot mention every group, they must make clear choices as to what group – if any – to reference in their arguments and position taking. As will become evident, we are concerned with a non-trivial form of elite attention, what we consider group prominence, namely the selective referencing or ‘name dropping’ of groups by policy elites in the context of them advancing political arguments.

Assessments of group prominence speak to classic debates in political science around the quality of plural democracies (see Lowery et al., 2015 Lowery & Gray, 1994; Schattschneider, 1960; Schlozman, Verba, & Brady, 2012; Truman, 1951; Walker, 1991). While not many interests go completely unorganised, it is certainly the case that few groups go on to become regular and valued participants in national politics (Grossmann, 2012, p. 170). Understanding what shapes this apparent disjuncture between organisation into politics and acknowledgement by political elites is highly relevant for assessments of the functioning of democratic systems. In that sense, our broad approach speaks to classic questions in political science about assessing group power. As Eising notes,

… interest group research faces greater challenges to capture the second and third dimension of power (i.e. agenda setting control, cultural, institutional and ideational biases) than the first dimension, namely the prevalence of actors in public policy decision making (on these power dimensions, see Lukes, 1974) (2016, p. 6).

While by no means perfect, the value of a concept like prominence is that by its very nature it encapsulates – in the context of everyday ‘natural’ speech – an evaluation of the overall relevance of groups to policymakers. That not everyone can be mentioned, that legislators are generally forced to be succinct, and that they are seeking to be as persuasive as possible, means that frequent acknowledgements reveal tacit judgements about relevance.

This paper aims to make two contributions to the existing literature on the policy engagement of interest groups. First, rather than examining the role of groups in a particular policy process, we focus on the extent to which groups are ‘top of mind’ of legislators, and are acknowledged as relevant political actors. Second, we demonstrate how this concept can be operationalised by applying a novel methodology, namely a supervised machine learning approach, which enables us to examine large amounts of text and evaluate not only how often groups are mentioned but crucially in what way they are being talked about by policymakers. Specifically, we train an algorithm to parse out prominent mentions of groups from the corpus of simple group mentions by legislators. In this way, we provide a useful tool for political science research, which may also be applied to assess the prominence of different policy participants (e.g. firms or experts) in other political arenas (e.g. the bureaucracy or the media).

The paper proceeds as follows. In the first two sections we clarify the concept of prominence and outline how we operationalise it in the legislative arena. In the third section we explain how we measure prominence applying a supervised machine learning approach, and introduce our dataset which tracks over 7000 prominent mentions of over 1300 national interest groups in parliamentary debates in Australia over a six-year period (2010–2016). In the analysis, we clarify the distribution of prominence across our population of interest groups, assess differences across group type, and compare our measure of prominence with two more traditional measures of interest group engagement, involvement and access. The conclusion addresses the key implications of our findings and suggests some avenues for future research.

From group effort to elite acknowledgement: interest group prominence

Ascertaining the relevance of different groups within a political system is a longstanding task of political scientists. Yet, this has become increasingly harder to assess as the advocacy landscape has become more densely populated. While it may at one point have been possible for students of interest groups to summarise the landscape by reference to a handful of notable groups, this is almost certainly no longer the case (for a more detailed discussion, see Berkhout, Beyers, Braun, Hanegraaff, & Lowery, 2017). How then might this problem be addressed?

At the aggregate group-system level, scholars have turned to useful proxies of policy engagement. The first, that we call involvement, refers to the choices by groups to take policy actions such as write submissions to consultations or give evidence to legislative hearings. Examples here include studies of written responses to administrative consultations or letters to parliament (e.g. Halpin, 2011; Pedersen et al., 2015). Interest group involvement can be distinguished from the well-established concept of access, which denotes interactions where groups directly meet with elected or unelected officials and exchange information, such as closed hearings or expert groups (Balla & Wright, 2001; Beyers, 2002, 2004; Binderkrantz & Christiansen, 2015; Bouwen, 2002, 2004; Rasmussen & Gross, 2015). Access refers to a situation whereby ‘a group has entered a political arena (parliament, administration or media) passing a threshold controlled by relevant gatekeepers (politicians, civil servants, or journalists)’ (Binderkrantz, Pedersen, & Beyers, 2017, p. 2). As highlighted by Eising (2016), these approaches speak best to the first face of power. Moreover, this approach has tended to privilege a focus on differences in the degree of engagement in the policy process (e.g. varying degrees of involvement or access that groups enjoy), rather than in terms of differences in kind of engagement.

Our focus in this paper concerns a third approach to assessing the policy engagement of interest groups and the relevance of a given group, namely prominence. As indicated in the introduction, questions as to how political elites allocate their attention are often examined in the context of public policy, yet rarely applied to the realm of interest group engagement in the legislative arena. We aim to engage in a similar discussion by shifting our focus from involvement or access to the varying prominence of these organisations in the eyes of policymakers. If elites simply cannot pay attention to all groups, then it surely matters which ones they repeatedly refer to as they argue their case and defend their political views and decisions.

Prominence refers to the taken-for-grantedness a group enjoys, or the extent to which it is top-of-mind among a given audience (e.g. MPs, government officials or journalists). As we clarified elsewhere, since prominence relates to how other actors perceive a particular group, and thus indicates which groups are considered more relevant, it is fundamentally different from concepts like involvement and access, which always relate to the role of groups in policy processes (Halpin & Fraussen, 2017).

Importantly, we distinguish between prominence and visibility. Whereas the visibility of a group refers to any attention it receives from policymakers, prominence refers to attention from political elites that demonstrates the relevance of the group; it is a ‘recognition or favourable notice of a group by policymakers’ (Halpin & Fraussen, 2017). In other words, whereas the notion of visibility mostly captures the frequency of elite attention, prominence is sensitive to the context in which elites mention a particular interest group.

As we will clarify and illustrate below, these mentions can be positive or negative in nature, implying that in some cases reference to statements or research of groups is used to support certain policy positions, yet in other circumstances claims of groups can be used to challenge or criticise particular policy measures. Here, party political dynamics are likely to play an important role. While beyond the scope of this paper, it seems likely to governing parties will mention groups to legitimize their course of action, while the opposition will look for statements of groups that provide ammunition to criticise government policy. Prominence includes the sum of all these mentions, whether they are negative or positive in sentiment; that a group matters sufficiently as to be worth spending speaking time criticising underlines its prominence. Where such referencing accumulates for a specific group, we can conclude that they are highly relevant players in the political system.

In trying to assess the relevance of a set of groups in a given national system, we argue that prominence is an additional – and increasingly valuable – measure. Specifically, it provides a unique indicator as to how elite attention is parsed among groups and which groups are considered (more) prominent. Prominence can thus be considered a scarce resource. Just as scholars broadly accept that access is a ‘policy good’ that groups would like to have (Bouwen, 2002; Truman, 1951), we argue that groups also strive to gain and maintain political prominence.

Does high prominence equal high influence? Similar to other forms of policy engagement such as involvement and access, prominence could be considered as a proxy for influence. Groups that are top-of-mind among legislators appear more likely to have their voices heard. At the same time, it is not unusual for groups to operate under the radar, and let politicians take credit for their policy work and expertise. Therefore, we think the relation between prominence and influence requires more systematic empirical assessment, taking into account the political context (such as mobilisation of other groups and preferences of political parties) and issue-specific factors like salience and complexity.

Operationalising prominence: legislative debates

The way political representatives discuss and debate issues in legislatures matters. As Bara et al. explain: ‘if democratic politics involves the giving and exchange of reasons in public discussion, then the study of how reasons are given becomes important’ (2007, p. 578). Those concerned with the quality of political discussion have used political speech to great effect in assessing the degree of policy deliberation (see Bächtiger, 2013). Others have emphasised the framing and thematic content of debates on specific issues (Bara, Weale, & Bicquelet, 2007). Still others see the value of text in systematically assessing policy positions and ideology of legislators, or preferences of external stakeholders such as interest groups (e.g. Bunea & Ibenskas, 2017; Diermeier, Godbout, Yu, & Kaufmann, 2012; Klüver, 2009; Lowe, Benoit, Mikhaylov, & Laver, 2011). These diverse studies share the same conviction that utilising political texts that are the natural by-product of political activity offers unique analytical insights, in particular to detect ideological positions, political interactions and the specific nature of political conflict (Monroe & Schrodt, 2008, p. 353).

In this paper we focus on the prominence groups have among members of parliament. An obvious viable proxy here might be the mentions of groups – by name – during parliamentary debates by legislators. As it turns out, legislators do mention groups in the course of speeches, motions and other forms of dialogue. Yet, as clarified in the theoretical section, what we are interested in are those occasions where a legislator actively uses the group as a resource in their policy speech, which we consider a crucial distinction between visibility (being mentioned) and prominence (being acknowledged as a relevant player). Measuring the use of groups as a resource in legislative speech is crucial to our claim that prominence provides a useful way to get at the tacit assessments of the relevance of groups, as viewed by political elites.

When addressing parliament, legislators need to swiftly convey a clear message, and they use various props to do so. One such prop is to invoke the name of a group to support and illustrate their argument or to congratulate or affirm the importance of said group for policy making. To be frequently used in such a manner is surely a strong indicator that a group is well established in the political landscape. The following examples underline and clarify the salience of the concept of prominence. Speaking on ‘MATTERS OF PUBLIC IMPORTANCE – Goods and Services Tax’, a Liberal Party Senator made the following statement:

It will stand up and deny it, but it is clear. That mob over there and their minions out there—the Business Council of Australia [BCA], ACCI [Australian Chamber of Commerce and Industry], the business groups—all want a cut to company tax, and how do they want to pay it? They want it to be paid—.²

The BCA and ACCI serve as useful – and necessary – shorthand for the ‘business lobby’ to which this Senator takes umbrage. The same referent groups were used as an endorsement of decisions made. In making a 2015 ‘Ministerial Statement’ to the House of Representatives on road pricing the Minister of Territories, Local Government and Major Projects remarked:

A number of participants in that debate expressed their support, last week, for the direction the Turnbull government has signalled on this issue. Business Council of Australia Chief Executive Jennifer Westacott said that the Business Council welcomes the government's intention to pursue much-needed reform in areas essential for business competitiveness, such as road pricing³.

The Minister went on to mention endorsements from the Australian Automobile Association and the Infrastructure Partnerships Australia. Here, again, name-checking the BCA serves to buttress the Minister’s case that their measures have business support. As both examples illustrate, the sentiment of the speech context in which a group is mentioned can be positive and negative. Given that we expect policymakers to use the claims of interest groups a resource, we expect that this sentiment will vary with their objectives: specifically, whether they are trying to defend or criticise certain policy measures. Yet, from the perspective of groups, it is the sum of all these mentions that demonstrates they are considered relevant actors by political elites.

The way in which the allocation of prominence works – and why it is valuable to groups – is made even clearer when one takes the counterfactual: why did the Minister or the Senator not mention other business organisations such as the Council of Small Business Australia or the Small Business Association of Australia, or some other related organisation? The cut and thrust of parliamentary debate rarely allows a legislator to indulge in a laundry list. And if they do, the choice of such diffuse language is to blunt the impact. So in choosing the few groups that can be mentioned, presumably the Minister passed over such groups because they would not serve as an equally persuasive endorsement of the ministers’ views. As we will explain below, there are clear and discernible variations in the way policymakers mention groups, which enable us to parse out routine mentions from those that demonstrate a group’s prominence.

Relatedly, groups may also be mentioned in the context of providing factual information to the debate. For instance, many groups issue reports or analysis that are picked up by legislators to help agenda set, propose new measures or to criticise existing positions. Take the example of a reference to the Australian Food and Grocery Council (AFGC) in a Second Reading Speech on the Protecting Children from Junk Food Advertising (Broadcasting Amendment) Bill 2010. Greens Senator Rachel Siewert commented:

The advertising industry introduced self-regulation—the Responsible Children’s Marketing Initiative—in January 2009. However, research including the Australian Food and Grocery Council report just this year—January 2011—found that one in five food advertisements in children’s programs were for high-fat, sugar and salt products. The self-regulation is clearly not working to effectively protect children.

Here the report by the AFGC has been used as a source of data and expert knowledge to respond to government proposals. Our straightforward point here is that groups are acknowledged in the discussions of legislators within Parliament. The approach we adopt to operationalising prominence has parallels with work on media coverage of groups. Journalists and newspaper editors know what sources will be most credible with their readers (see Binderkrantz et al., 2017) – and hence seek them out to feature in their news reportage. Considering the interaction between social movements and media systems, Gamson and Wolfsfeld for instance argue that: ‘The media spotlight validates the fact that the movement is an important player. Receiving standing in the media is often a necessary condition before targets of influence will grant a movement recognition and deal with its claims and demands’ (1993, p. 116). We think a similar logic applies in the parliamentary arena.

Measuring prominence: a supervised machine learning approach

In this section we outline the data and methods we use to apply this conceptualisation and operationalisation of group prominence in the legislative arena. We pursue a text-based analysis of the mentions of groups by members of parliament. There is a rich and expanding literature in political science that has capitalised on the increased availability of political texts online – especially formal records of legislative proceedings – and the tools of automated text coding and machine-learning (see Grimmer & Stewart, 2013; Monroe & Schrodt, 2008; Wilkerson & Casas, 2017). A great deal of this work has focussed on the important question of detecting the (changing) policy preferences of legislators and parties (see for instance Lowe et al., 2011). If we consider previous research that has relied on analysis of text from parliamentary debates, a common approach is (automated) content analysis via the use of search queries (to look for (different types of) statements on particular issues (e.g. Bara et al., 2007 van der Pas, van der Brug, & Vliegenthart, 2017;), or mentions of particular actors). Here we take a complementary yet distinctive perspective, namely the prominence of groups in the speech of legislators. Rather than relying on dictionary methods (e.g. sentiment analysis using positive and negative word lexicons), we pursue an innovative use of supervised machine learning to validate our theoretically derived operationalisation of prominence.

Supervised machine learning provides a methodological tool for text analysis that, when applied and validated properly, offers a powerful ‘labor saving device’ (Wilkerson & Casas, 2017, p. 6) that greatly reduces the resources required to analyse large collections of text (Grimmer & Stewart, 2013, p. 1). Here, human coders assign text to predetermined categories in order to create a training set. The algorithm ‘learns’ by finding patterns in the training data and estimating a function to describe the relationship between the input features (i.e. words occurring in documents) and target feature (i.e. human-coded categories). Crucially, the model is trained using k-fold cross validation in order to avoid over-fitting the data, thereby maximising performance on out-of-sample data (see Grimmer & Stewart, 2013, pp. 13–14). Finally, model performance is evaluated on a held-out test set, providing a benchmark of the out-of-sample performance for the final cross-validated model by assessing predicted values (via the model) against observed values (the ‘ground truth’ human-coded data).

While statistical methods focus primarily on theory testing and input variables, supervised machine learning approaches are mostly concerned with explaining outputs (e.g. whether a group is considered prominent or not); which ‘leads researchers to be more concerned with prediction accuracy and less concerned with explanation’ (Wilkerson & Casas, 2017, p. 5). Compared to dictionary methods, supervised learning requires domain-specific and coherent definitions of the studied concept, and can be validated easier by assessing model performance statistics (Grimmer & Stewart, 2013, p. 9). In this way, the concept of prominence set out in this paper is too complex to capture using dictionary-based methods, as the word lists in existing sentiment analysis methods (such as WordNet) are too broad for the particular domain under examination. In line with previous work, this emphasises the value that supervised machine learning brings to political science research: it provides the ability to capture domain-specific concepts into a predictive model, and more importantly provides performance statistics to determine the accuracy and validity of trained models.

Data source

For our research, we utilised the Australian Interest Group Dataset (see Fraussen & Halpin, 2016). This data set draws on the Directory of Australian Associations (DoAA) as its foundation.⁴ The Directory has been published since 1978. Its stated aim is to comprehensively list Australian associations for those professionally engaged in public affairs (including journalists, public servants and political operatives). The data we report here focuses on the 2012 edition.⁵ The DoAA is not something we can work with immediately off the shelf. As its name indicates, it contains a range of organisational types, some of which do not resemble what scholars would conventionally term an interest group. Thus, we took great care in implementing a systematic code-scheme to get us from the directory as published to our estimate of the main population of interest, namely national interest groups. The full 2012 Directory included 4,102 individual entries. We removed all non-national organisations and those that are not interest groups (e.g. corporations, think tanks and sport or leisure groups). After excluding these types of organisations, our remaining sample includes 1,316 national interest groups.

Considering our focus on the legislative arena, we measure prominence by searching for mentions of organisations in the Hansard of the Australian Parliament. Hansard is the official record of proceedings of the Australian Parliament (both upper and lower houses); in the US context it would be similar to the Congressional Record. This unobtrusive approach to data collection means that we capture the way groups are utilised as a resource in the normal routines of the legislative environment. We collected this data for the 43rd and 44th sessions of parliament (28/09/2010 to 27/06/2013, and 12/11/2013 up to 05/05/2016).⁶ The Australian Parliament website does not currently provide data in a wholesale fashion, for example through an API⁷. As a result, we used a programmatic data mining approach to data collection that broadly involved three steps.

Firstly, for each of the 1,316 interest groups we queried the Australian Parliamentary website to obtain search results for full name of each group, within the specified date range. Two queries were made for each interest group, one for the Senate database and another for the House of Representatives database. This is what we hereafter refer to as total group mentions. This is the kind of data that has been used in the past to measure the visibility of groups (Grossmann, 2012). Secondly, we parsed the resulting HTML data to extract the relevant information from each search result, such as the text transcript, date, time, electorate, party affiliation of the speaker, and so forth. The third step involved cleaning, sorting and aggregating the data for analysis. Effectively, this means creating a data set whereby every row is a paragraph equivalent where one of our 1316 groups is mentioned. We discuss the steps in this process in more detail below.

Detecting prominence

Our initial focus was on collecting mentions of one of our 1316 groups. As indicated above, past work discussing prominence has used mentions as a measure (Grossmann, 2012). A similar approach adopted by scholars in the measurement of ‘access’ in the media arena (Binderkrantz et al., 2017; see also Bernhagen, 2012; Binderkrantz, 2005). We argue that mentions alone might be a good method of measuring visibility of groups, but not all mentions amount to prominence. As outlined above, prominence is generated for a group when members of an audience, in our case politicians, consistently use groups as a political resource in their work. This presents us with a crucial empirical task; how to parse out mentions that constitute prominence from those that do not? To identify which mentions indeed imply political prominence, we apply a supervised machine learning approach.

The first step was to train an algorithm which might then be applied to our entire corpus of paragraphs mentioning groups. We implemented a code scheme whereby mentions were coded as prominence when a group’s views were used by a legislator, or where a reference was made to a group as an important participant in the policy process by a legislator. In those instances where a legislator refers to multiple groups in one of these two scenarios, we only considered a mention of a group as prominence if fewer than 10 groups were mentioned. On the latter criterion, the rationale is that prominence relates to scarcity of attention. Where policymakers list more than 10 groups they cannot be said to be singling out particular group. Indeed, the group is probably not being used as a ‘resource’ if it is part of such a long list. Rather, the aim is simply to laundry list the sheer volume of opposition or support. All other mentions are considered not to be prominence.

Two of the authors coded 100 randomly selected mentions according to a pre-determined code scheme. This was implemented and discrepancies discussed in order to finalise the code scheme. A fresh set of 700 randomly selected mentions was selected and the authors coded these using the final code-scheme, with a substantial level of inter-coder reliability (85% agreement, Cohen’s Kappa = 0.725, Krippendorff’s Alpha 0.724). This corpus of mentions was then used to train a supervised machine learning text classifier, using the Support Vector Machine (SVM) algorithm implemented in the RTextTools package for the R programming language (Jurka, Collingwood, Boydstun, Grossman, & van Atteveldt, 2013). The rationale for using SVM and RTextTools was to maximise reproducibility of our approach and make it easier for future researchers to undertake the same or similar analysis.

Broadly, the goal was to estimate a model that, based solely on the text, could correctly classify whether a given mention of an interest group constitutes ‘prominence’ (or not). Thus, we posed the automatic coding of data as a binary text classification problem, where the independent variable is the text paragraph(s) and the dependent variable is the binary coding of ‘prominence’ and ‘non-prominence’. For this research we wanted to estimate a model that would perform well in terms of both precision (true positives/(true positives + false positives)) and recall (true positives/(true positives + false negatives)). In other words, the trained model should not only accurately classify ‘prominence’ mentions correctly, but also return most of the prominence mentions within the dataset. For example, a high precision and low recall score would mean that the model is accurate but only picks up a small fraction of the actual prominence mentions. In this way, we evaluated the model using the f-score metric, which is a weighted average of precision and recall. Our benchmark in this paper was to achieve an f-score of 0.8 or higher, which we regard as a reasonable minimum threshold for using the tool in practice.⁸

A five-fold cross validation technique was used for model evaluation, in order to maximise the validity and generalizability of the model to unknown datasets (in this case parliamentary text without a ‘known’ or manually coded prominence category). The training data were randomly sampled into five equally sized partitions or folds.⁹. Five models were trained and validated, rotating the data sub-samples such that each fold was used exactly once as the validation data and the remaining four folds as the training data. Results from the five models were then averaged to provide a single estimation of model performance. As Table 1 shows, predictive performance on the target category (prominence) was quite good and reliably exceeded the 0.8 minimum benchmark set in this paper.

Table 1. SVM Model performance using 5-fold cross validation.

K-fold model	Precision	Recall	F-Score
#1	0.87	0.85	0.86
#2	0.82	0.96	0.88
#3	0.68	0.90	0.77
#4	0.82	0.85	0.83
#5	0.76	0.93	0.84
Mean score	0.79	0.90	0.84

After training and validating the model, we then applied it to automatically code the remaining paragraph equivalents in the dataset. Further cleaning was required before processing by the algorithm. All duplicate mentions were removed: a case where a group was mentioned multiple times (e.g. 3 times) in a single paragraph would result in a duplicate line of data for each mention. We also implemented a rule whereby paragraph equivalents of less than 15 words were removed from the data. This was on the basis of the project team scrutinising samples of paragraph equivalents at multiple thresholds (e.g. 10, 15 and 30) and deciding that 15 or below provided insufficient context for valid coding of prominence. While we could have retained these paragraph equivalents for the machine learning algorithm to process, they would almost certainly be returned as non-prominent mentions. By removing them as part of the cleaning process we ensure that the efficiency of the algorithm is maximised. The research team verified the results by manually evaluating a random sample of 100 automatically coded mentions, which indicated that the model was operating at the expected level of performance.

Applying machine learning to examine variation in prominence of interest groups during legislative debates

A core aim of this paper is to establish a valid process to measure prominence through text-based mentions of groups during parliamentary debates. Yet, an equally important motivation for pursuing this path is in the potential it holds to scale up the study of prominence by applying this algorithm to large volumes of text. In this final section, to illustrate its empirical application, we report on a corpus of mentions of our 1316 groups across the 43rd and 44th Australian parliaments.

As discussed above, scholars concerned with assessing the attention groups gain from legislators have relied on straightforward mention counts (Grossmann, 2012). As such, in our present context, the ‘raw’ group mentions data – which is the result one would get by simply entering the name of each of our groups in the Australian Parliament website search engine – is a useful comparator with our machine-learning approach.

Our initial search request of the parliamentary web search engine yielded 10,286 mentions for 627 of our 1316 groups. Of our complete set, 689 groups were thus never mentioned in parliament. The mean number of mentions was 7.8, with a minimum of 0 and maximum of 347. The results from our automated coding of prominence identified 7287 prominent mentions. The mean number of prominence mentions was 5.5, with a maximum of 297 and minimum of 0. There is a strong positive correlation between these two measures (.97). This is in line with our expectations, as the raw mentions provided the input for detecting prominent mentions. Yet, we see that in aggregate just over 70 per cent of all mentions are prominent mentions, where groups are deployed as a resource in legislative speech.

While this headline relationship is of interest, what is just as relevant is the extent to which groups vary in respect of the percentage of all mentions that are in fact prominent mentions. We have 168 groups with a prominence rate of 100%, which implies that all their mentions are prominent. We also observe 102 groups for which none of their mentions are prominent. Those groups for which none of their mentions were prominent mentions, or that have 100% of their mentions as prominent, both tended to have rather small number of overall mentions, ranging from 1–10 mentions and 1–26 respectively.

Table 2 reports the % of mentions by % of groups. It demonstrates the very skewed and concentrated nature of prominence. In our data set, 2.4 per cent of groups account for over 50 per cent of all the prominent mentions. That is, just 20 groups account for half of all prominent mentions. Moreover, the top 10 groups account for just over one quarter of all prominent mentions. Similar data is represented graphically in Figure 1, which shows that the distribution of prominent mentions is highly skewed (to aid interpretation all 0 values are removed from the graph). This indicates that the set of groups that members of parliament frequently refer to is rather small and limited to a small pool of ‘top tier’ groups. The highly skewed nature of prominence, with a small core and a much larger periphery, resembles a pattern that is also found in studies that focused on lobbying activity and other forms of group policy engagement (e.g. Heinz, 1993; LaPira, Thomas, & Baumgartner, 2014). A recurring finding is that a minority of groups accounts for the majority of activity, whether it concerns forms of access such as the provision of legislative evidence (e.g. Pedersen et al., 2015), representation in advisory councils (e.g. Fraussen, Beyers, & Donas, 2015), or policy involvement, e.g. in the form of public consultations (Halpin & Thomas 2012; Yackee & Yackee, 2006).

Figure 1. Distribution of prominence (n = 520).

Table 2. Distribution of prominent mentions across groups.

% of groups	n	% of prominent mentions
1	13	32.50
5	66	66.04
10	132	81.30
20	264	93.45
100	1316	100.00

Table 3 reports the top 10 groups in terms of volume of total mentions and prominent mentions. We can see a mix of group types, with all attaining more than 100 prominent mentions over our time period. While several of these groups are most of the time mentioned in a prominent way (with generally more than 85% of their mentions relating to some form of acknowledgement), this does not apply to all groups (see for instance the Australian Workers’ Union and United Voice, two unions which are only referred to in a prominent way about half of the time).

Table 3. Top10 most prominent groups.

Rank	Name	Type	Mentions	Prominence
1	Business Council of Australia	Business	332	297
2	Australian Industry Group	Business	347	294
3	National Farmers’ Federation	Business	273	236
4	Australian Chamber of Commerce and Industry	Business	262	233
5	Universities Australia	Institutional	247	231
6	Law Council of Australia	Professional	210	180
7	Australian Medical Association	Professional	196	173
8	Australian Workers’ Union	Union	273	138
9	United Voice	Union	225	130
10	The Financial Services Council	Professional	139	126

We can also see some variation in prominence when we parse our aggregate data by group type. In Table 4, we report the frequency and per cent composition of the Australian group population, compared to those with prominent mentions. Compared to their share in the population, both unions and institutional groups receive a high proportion of prominent mentions. Professional and service groups have relatively low levels of average prominence – in the case of professional groups, they compose 29 per cent of the population but just 15 per cent of prominent mentions. For business groups and citizen groups, the percentage of prominent mentions approximates the proportion they represent in the broader interest group population.

Table 4. How do prominent mentioned groups compare to broader population?

Type	Population (No. Groups)		Prominence (No. Groups.)		Prominence (Mentions)
	Freq.	Percent	Freq.	Percent	Freq.	Percent
Citizen	210	16	114	22	1265	17
Business	480	37	192	37	3033	42
Union	26	2	21	4	841	11
Professional	385	29	105	20	1121	15
Institutional	25	2	19	4	480	7
Service	133	10	41	8	274	4
Hybrid	57	4	28	5	273	4
Total	1316	100	520	100	7287	100

Overall, the distribution of prominent mentions is not that dissimilar from the proportion of group types in the broader population. Another important question involves whether there is a partisan dimension to this: Are certain group types more prominent among particular political parties? Classic work in political science emphasised the shared role of parties and interest groups as organised channels for the representation of social and economic interests (e.g. Schattschneider, 1948; Key, 1942) and highlighted that the latter often served as formal subsidiary organisations of parties (Duverger, 1954). Whereas for many decades the literature on parties and interest groups has mostly progressed in parallel (Heaney, 2012; Fraussen & Halpin, 2018), this is changing somewhat recently (e.g. Allern & Bale, 2012; Bawn et al., 2012). Recent work on the European Union level has examined the ‘alignment’ in the policy positions of EU parties and types of organised interest (e.g. Beyers, De Bruycker, & Balle, 2015), while others have focused on the interaction between groups and parties in several European countries (e.g. Rasmussen & Lindeboom, 2013).

While groups themselves generally find public partisan support unwise – instead opting for broad bi-partisanship (Binderkrantz, 2015; Halpin, 2015), there is still good reason to posit that political elites will tend to choose sources from among ‘their’ groups. Following that view, one would expect a broad ‘alignment’ between conservative parties – or parties of the political right – and business and professional interests, whereas a similar linkage seems likely between progressive parties – or parties of the political left – and unions and citizen groups.

Our data allows us to parse each prominent mention by the party affiliation of the member of parliament speaking. We have coded each mention by each of the three main parties – Australian Labor Party (ALP), Nationals and Liberals (Coalition) and the Australian Greens (Greens) – with a residual ‘Other’ category for minor parties and independents.

Table 5 reports the prominent mentions by party again utilising our group typology, and highlights relevant differences across parties. If we focus on the three largest parties, we can observe that members of the Coalition refer to business groups 48 per cent of the time they refer to any group, whereas the Greens are most likely to mention citizen groups (28%). Along with the ALP, the Greens also frequently refer to service and hybrid groups. Furthermore, tighest proportion of union mentions are being made by the ALP and the Greens members, while professional groups have a remarkably similar percentage for each party. These findings – pointing to party variations in the prominence of groups – suggest that our approach provides some important insights to scholars of legislative studies and group-party relations.

Table 5. Prominent mentions by party, group type.

	ALP	Coalition	Greens	Other	All parties
Citizen	19 (570)	14 (396)	28 (183)	16 (116)	17 (1,265)
Business	36 (1,066)	48 (1,411)	28 (181)	51 (375)	42 (3,033)
Union	13 (380)	10 (279)	14 (88)	13 (94)	12 (841)
Professional	17 (494)	16 (455)	16 (106)	9 (66)	15 (1,121)
Institutional	6 (179)	8 (245)	3 (22)	5 (34)	7 (480)
Service	5 (146)	2 (63)	5 (29)	5 (36)	4 (274)
Hybrid	5 (140)	3 (78)	6 (36)	3 (19)	4 (273)
Total	100 (2,975)	100 (2,927)	100 (645)	100 (740)	100 (7,287)

Note: Column percentages (due to rounding, column percentages do not always add up to 100), counts in brackets.

Prominence, access and involvement

But how does our measure of prominence compare to a measure like access, which is arguably the prevailing proxy measure of group engagement? Consistent with the convention in the field, we utilise invitations to give oral evidence to Australian parliamentary committees as an arena-specific measure of access (Pedersen et al., 2015). When we examine a straightforward bivariate correlation between our measure of access and prominence, we find the modest positive correlation of 0.59.

Take the Minerals Council of Australia. It is often referred to in the mainstream political press as one of the most influential groups in national politics.¹⁰ Yet, on conventional activity and access measures, it barely registers. Indeed, it has provided a written submission to senate committees on five occasions over the period we cover, and has given oral evidence twice to senate committees – the key mechanism for meaningful formal input into the legislative process. However, it received 106 prominent mentions – the eleventh most mentioned group in parliament.¹¹

This is also borne out in Table 6, which reports the top 10 groups in terms of the volume of oral and written evidence to parliament over the same period (group who are also among the 10 most prominent groups are included are included in bold).

Table 6. Top 10 groups by access and involvement.

Rank	Name	Type	Oral	Written	Mentions	Prominence
Top 10 Groups by Oral Evidence (access)*
1	Australian Industry Group	Business	12	15	347	294
2	National Farmers’ Federation	Business	10	21	273	236
3	Law Council of Australia	Professional	10	26	210	180
4	Australian Council of Trade Unions	Union	9	18	112	78
5	Community and Public Sector Union	Union	9	17	43	21
6	Australian Chamber of Commerce and Industry	Business	8	11	262	233
7	Australian Nursing Federation	Union	8	11	23	11
8	Australian Medical Association	Professional	8	20	196	173
9	Palliative Care Australia	Citizen	7	8	29	26
10	Australian Manufacturing Workers’ Union	Union	7	17	79	48
Top 10 Groups by Written Evidence (involvement)
1	Law Council of Australia	Professional	10	26	210	180
2	National Farmers’ Federation	Business	10	21	273	236
3	Australian Medical Association	Professional	8	20	196	173
4	Australian Council of Trade Unions	Union	9	18	112	78
5	Community and Public Sector Union	Union	9	17	43	21
6	Australian Manufacturing Workers’ Union	Union	7	17	79	48
7	Australian Industry Group	Business	12	15	347	294
8	Australian Psychological Society	Professional	6	13	17	12
9	Uniting Care Australia	Hybrid	4	12	5	4
10	United Voice	Union	6	11	225	130

Note: Several groups had a value of 7 for Oral evidence (access).

The data we present here provides empirical support for the conceptual distinctions made elsewhere between access and prominence (Halpin & Fraussen, 2017). In aggregate terms, the two measures of our concepts are only moderately positively correlated. Yet, as would be expected, we see in some individual cases that particular groups are granted high levels of access (or involvement), and also mentioned prominently by legislators. Still, we see some cases where this is not the case. For instance, we notice that some interest groups focused on health issues (like Palliative Care Australia and the Australian Nursing Federation) enjoy relatively high levels of access, but nevertheless are not among our top 10 of most prominent groups. Furthermore, even though the Business Council of Australia and Universities Australia are two of the most prominent groups in parliamentary debates, they are not among the groups with the highest levels of involvement or access.

Summarising the reported findings, our analysis of prominence of Australian interest groups among Members of Parliament confirms that it is highly concentrated. Put another way, for the majority of groups, being strategically included by Members of Parliament within their legislative speech will not happen. We can conclude from this observation that while many groups exist to have their positions, issues and constituencies viewed as central to political debate by politicians, only a small number of groups are regularly acknowledged as key players.

Conclusion

Ascertaining which interest groups are considered relevant by policymakers presents an important challenge for political scientists. The increasing density of lobbying communities in advanced political systems means that groups will find simply being noticed ever more difficult. Moreover, as the attainment of particularly involvement, yet also occasional access to formal policy making processes, becomes easier, we can expect that being prominent will be an increasingly important asset for groups to possess. Our study contributes to resolving this challenge in two ways. Firstly, we have presented a concept – prominence – that speaks to the imperative for groups to be noticed in a meaningful way by political elites. Secondly, we have developed a method for measuring prominence that takes the linguistic context of parliamentary debates seriously. Moreover, we have developed an approach which enables measurement to be scaled-up to assessments of prominence across entire interest group systems, as well as over time, or at the international level (e.g. Dellmuth & Tallberg, 2017). One could also use this method to assess the prominence of interest groups (or other actors such as think tanks) in a more specific legislative context, such as committees or parliamentary questions (Bailer, 2011; Pedersen et al., 2015).

Our findings show that there is substantial variation in the prominence afforded to groups by Members of the Australian Parliament. In fact, we see that many groups simply do not gain any prominence among elites. The pattern of prominence is highly skewed. This latter observation is consistent with studies more broadly focussed on the way institutions and elites allocate attention. Crucially, we show that our measure of prominence is not highly correlated with other concepts with which we have made conceptual distinctions, specifically access to policymakers.

Our aggregate approach here provides a complementary approach to the issue based analysis of influence and preference attainment (see Baumgartner, Berry, Hojnacki, Kimball, & Leech, 2009; Beyers et al., 2014). We suggest that groups approach their specific advocacy task with the deck already stacked in favour of some over others. We can also draw useful analogies with the studies of groups and the media. In his work on citizen groups in US national politics, Jeff Berry asks ‘Who Counts?’ One approach to answering this question is to refer to mentions in the national political media. His rationale for so doing is straightforward:

The assumption is that beat reporters for these publications [referring to New York Times and such like] have some sophisticated understanding of the issues – or at least the politics surrounding the issue – and that their stories reflect reasonable conclusions about which groups deserve mention in their stories.

Noting that often times, when one aggregates all stories in a similar policy field, just a handful of groups are repeatedly mentioned, he remarks ‘Such judgements are taken to be a relatively accurate reflection of who are the most important spokesmen [sic] for the interests involved’ (Berry, 1999, p. 23; see also Tresch, 2009). Thus, Berry intimates, inferences can be made from the pattern of media mentions in respect of which groups matter. Just as newspaper journalists and editors in effect use professional norms and insider judgement to weed out all but the most relevant groups to reporting on given policy issues, we have strong reasons to surmise that similar strategic processes occur for elected officials when they decide who to reference in their legislative speech.

Like all studies, ours comes with limitations. At this point we have focussed on measuring the outcome – namely group prominence among legislators – but have not offered an account of the micro-foundations that contribute to this aggregate outcome. Precisely this point has been observed with respect to the advancing the literature on policy processing and agenda-setting within political institutions (Jones, 2003; Jones & Thomas, 2013). One approach to this is to probe the way individual political elites – who in aggregate constitute key political institutions, such as legislatures – themselves allocate attention (see examples in Kingdon, 1984; Walgrave & Dejaeghere, 2017). This is an obvious way to extend the work we present here.

Acknowledgement

We wish to acknowledge the financial support from an Australian Research Council Discovery Grant (DP140104097: The Organised Interest System in Australian Public Policy). Earlier versions of this article were presented at seminars at Griffith University (Australia) and the Australian National University, and we thank participants for their helpful comments. We also thank the anonymous reviewers and journal editor for their useful comments, which substantially improved our argument.

Disclosure statement

No potential conflict of interest was reported by the authors.

Notes on the contributors

Bert Fraussen is Assistant Professor at the Faculty of Governance and Global Affairs, Leiden University. His research agenda integrates the organisational design and development of political organisations, notably interest groups, and their involvement in public policy. His work has been published in journals such as European Journal of Political Research, Governance, and Public Administration.

Timothy Graham is Postdoctoral Fellow in Sociology and Computer Science at the Australian National University. His research combines data science with social theory to study online networks and platforms, particularly in the context of public policy and political communication. He has published in journals such as Information, Communication and Society, Information Polity, and Critical Social Policy.

Darren Halpin is Professor of Political Science at the Australian National University. His work focuses on interest groups in the policy process, with particular emphasis on organisational development and democratic participation. His work has appeared in British Journal of Political Science, Journal of Public Policy, Public Administration and Governance. Darren is Co-Editor of the journal Interest Groups and Advocacy.

ORCID

Darren R. Halpin http://orcid.org/0000-0002-2692-0636

Additional information

Funding

We wish to acknowledge the financial support from an Australian Research Council Discovery [grant number (DP140104097: The Organised Interest System in Australian Public Policy)].

Notes

1. While we focus on the mentions of specific individual interest group organisations (that could be considered forms of direct prominence), the concept of prominence could also be applied to particular groupings within society, such as farmers, lawyers, environmentalists, and so on. A comparison with this form of prominence (which could be conceived as a more indirect form) goes beyond the scope of this paper.

2. Transcript can be found at: http://www.aph.gov.au/Parliamentary_Business/Hansard/Hansard_Display?bid=chamber/hansards/4338c56a-c77a-4a12-a868-652df075a3e9/&sid=0188.

3. Transcript can be found at: http://www.aph.gov.au/Parliamentary_Business/Hansard/Hansard_Display?bid=chamber/hansardr/19a0a2a6-673e-4aee-8a78-d39a4d56068c/&sid=0082http://www.

4. We thank Erik Johnson for assisting us with access to the 2012 edition of the DoAA as part of his Comparative Associations Project.

5. These kinds of sources are well used in other countries to good effect. Walker et al. for instance highlight that

in spite of the serious concerns we have addressed about the source’s comprehensiveness and potential biases, the (US) Encyclopedia of Associations is widely recognized as the most inclusive census of national nonprofit associations. As a result, it has been used widely by researchers of various segments as well as the entire national nonprofit organizational landscape, and much of what we know about that landscape depends on the source. We have compiled a list of more than 150 refereed journal articles that utilize information drawn from one or more editions of the source. And, in many of those research reports, those knowledgeable about their own small segments of the associational world the source attempts to chronicle provide testimonials about its utility and comprehensiveness of coverage (2011, pp. 1328–1329).

6. Accessible at http://www.aph.gov.au/Parliamentary_Business/Hansard. Two parameters in the Search function were selected, the date (from = 28/09/2010; to = 05/05/2016), and also the ‘Chamber/Committee’filter. The scraper does two search queries per group, once for House and once for Senate. The total raw mentions is the sum of the results for the two search queries. The list of groups, code for extracting this data and the machine learning approach is available at https://github.com/timothyjgraham/measuring-prominence. Replication data and stata code for all tables and figures is available from the corresponding author.

7. Application Programming Interface.

8. Other model evaluation metrics such as AUC are often regarded as ‘acceptable’ if the score is 0.70 or higher, for example when deploying risk assessment tools for violent crime (see Rice, Harris, & Hilton, 2010).

9. Although 10-fold cross validation is often used, given the relatively small size of the dataset we used 5-fold cross validation.

10. See for instance http://www.smh.com.au/business/lobby-groups-pull-strings-in-halls-of-power-20130816-2s26d.html#ixzz3yPw71p9K.

11. Being prominent may be less important for ‘members’ of interest groups, especially in the case of business associations. Individual companies may not wish to be acknowledged publically as key players in a policy issue, but would be insisting that business associations that they are members of are viewed in such a way by policy makers.