Turnout, government performance and localism in contemporary by-elections

ABSTRACT

Pored over by the media and used as crucial testing grounds for new candidates and campaign techniques by political parties, by-elections – or special elections – are comparatively under-studied in contemporary academic analyses. These elections, held outside the normal election cycle enable local voters to replace a departing representative, but the way in which they vote has often been dismissed as being driven by national factors. In the British case, despite the attention such contests bring, understandings of the way in which people vote in contemporary by-elections are based largely on data from over forty years ago. This article represents a substantial updating of the literature on by-elections, using data on 148 contests from the British context to examine whether pre-existing assumptions from the literature hold: namely the importance of national and local factors. It finds that by-elections no longer represent useful routes by which defeated MPs can re-enter Parliament. For both turnout and party vote share at these contests, a combination of local contextual and national factors impact by-election results, whereas contest specific factors such as having a local candidate only make a difference in a few cases.

Introduction

“This is a by-election, it’s mid-term, it’s a protest. That’s what happens in by-elections” (BBC 2013) So said David Cameron, Prime Minister at the time of the 2013 Eastleigh by-election at which his coalition partners, the Liberal Democrats retained the seat they had held for 19 years (on which occasion, co-incidentally, the party had gained it in yet in another by-election), with UKIP pushing the Conservatives into third place. By-elections, sometimes known as special elections, are contests held between national elections to fill a vacant seat. As Cameron’s quote demonstrates, the results of such elections are sometimes side-lined as freak occurrences; occasional blips in electoral behaviour that tell us little about the state of politics in a country more widely – contests where voters act irrationally and on occasion unpredictably. Yet in some circumstances, by-elections have shifted the fortunes of both governments and party leaders; by-elections in the October 1974 parliament eliminated the government’s majority in 1977, and similarly by-elections (alongside defections) vapourised Major’s majority in the 1992 parliament. Poor performances by governing parties are expected features of by-elections, but conversely opposition parties – particularly smaller parties such as the Liberal Democrats – are expected to perform well (Cook and Ramsden 1973). More fundamentally, they are crucial for the constituents in the vacant seat – to elect a new representative and ensure their continued representation on the national political stage.

By-elections occupy a strange, dichotomous place in British politics; pored over by the media and used by political parties to test new candidates or campaigning techniques, but comparatively understudied and under analysed in contemporary academic research. The 1970s to the 1990s saw much academic interest in by-elections (see for example Cook and Ramsden 1972; Stray and Silver 1980; Norris 1990), with much of the literature emphasising the role of national issues and governmental performance in explaining their results. However, since then by-elections have largely escaped further analysis, classified as second-order elections (Reif and Schmitt 1980) – although the appropriateness of this will be explored in this piece. Evans (2012) and Stewart (2019) are two recent exceptions, although their studies concentrate on recruitment and a historical case study respectively. Surridge’s recent update (2021) explores what three recent by-elections tells us about post-Brexit electoral volatility, but does not place this in a wider discussion of such contests. As such, our contemporary understanding of voter behaviour in these contests – habitually repeated in the media when these contests occur – relies on historical assumptions and analyses, using data that concluded with Thatcher’s premiership, rather than questioning whether this is still appropriate for understanding contemporary British voter behaviour. In analysing general elections, vote share changes are reported in relation to the previous general election, even in in seats where there has been a by-election in the intervening years, airbrushing it from analysis. Yet by-elections are the source of a great deal of focus and speculation in the media, with their triggering recorded and their results unpicked, particularly reflecting on their impact on the national parties and what they may tell us about governmental popularity. To put this into context, ahead of the Hartlepool by-election in May 2021, over 9,000 articles on the contest were published in British newspapers. Kirkland (2020) also shows how the prominence of press reporting on by-elections can create important narratives about the government. Parties focus their attention on by-elections too; in many cases they may flood seats ahead of a by-election with activists, spend not insignificant amounts of money on the campaign and treat local voters to a succession of party VIPs visiting to drum up support. For example, in the 2008 Glenrothes by-election, caused by the death of Labour’s John MacDougall, the SNP and Labour spent a combined £195,000 on the campaign (Electoral Commission 2019). Both the media and parties therefore demonstrate a considerable degree of enthusiasm for by-elections, an enthusiasm requiring further exploration through contemporary academic analyses of these contests.

The article firstly explores what existing literature on by-elections from across the globe identifies as the key explanatory factors in their results and produces deductive hypotheses for exploration in the rest of the piece. The literature shows that there is a key division between the literature which expects national factors to determine by-election results (crossing both the literature on second-order elections and referendum theory), and that which identifies the importance of local factors (both contextual and contest-specific). It also extends the literature on by-election to explore the literature on scandal. Using data on 148 United Kingdom by-elections from 1979 to 2022, the article then establishes contemporary patterns of seat change and incumbency, alongside changing patterns in the causes of such contests and the presence of local candidates. The article then proceeds to test turnout and party/incumbent vote shares at by-elections against a range of local contextual, local contest-specific and national factors to determine the extent to which contemporary patterns of voter behaviour at by-elections fit with expectations from the existing literature. As such, this article represents a substantial updating of knowledge on voting behaviour at by-elections in contemporary British politics and beyond.

Understanding by-elections

Individualised elections outside the normal electoral cycle are a particular feature of electoral systems using single member constituencies, although studies of by-elections also exist for multi-member systems, including New Zealand (Mckenzie 2009) and Ireland (Gallagher 1996). In understanding voting behaviour in such contests, there are two partly overlapping theories. The first is the delineation between first and second-order elections, originating in Reif and Schmitt (1980). Here, higher levels of turnout in first order elections (such as national elections) are explained by voter perceptions that they hold greater value than second-order elections, which typically experience lower turnouts, and are determined primarily by voter responses to national issues. Reif and Schmitt specifically classify by-elections as second order contests (8), alongside regional elections and local elections.

The simplicity of the first versus second order contest categorisation risks dampening the nuances in voting behaviour between different types of second-order elections, including by-elections. For example, as Heath et al. (1999) identify, British voters appear more engaged with local elections than European Parliament elections. Schakel and Jeffery (2013) also argue that regional elections do not always fit easily into the second-order definition. In countries with relatively recently introduced regional institutions, such as the United Kingdom, voters increasingly feel able to look beyond the national government in apportioning blame (see Dafydd and Badanjak 2020; Middleton 2019; Leon and Orriols 2019) in second-order elections. The over-simplification of the first versus second order delineation applies to by-elections as well; instead of the lack of authority of officeholders elected in second-order contests as Reif and Schmitt (1980, 9) propose, constituents voting in by-election contests are electing a representative to the highest directly elected office in the United Kingdom. If an MP elected at a by-election stands at the next general election, they automatically enter a first order contest. As such, Reif and Schmitt’s assumptions of lower turnouts and overwhelming national factors in contests such as by-elections deserves reconsideration.

The second theoretical underpinning for by-election behaviour is referendum theory (Mughan 1986, 1988 in the UK, but also evident in discussions beyond – see Scarrow 1961 for Canada), which overlaps with the second order understandings of by-elections in the respect that it also emphasises the state of national politics at the time of the contest as an explanatory tool. Here, a by-election provides voters with an opportunity to feed back to the government on their performance – a contest held in a time of high unemployment and high inflation is likely, for example, to see a particularly bumpy night for the government party’s candidate. Knotts and Ragusa, in their study (2010) of US special elections have seen a nationalisation in voter behaviour since the 1970s. Anderson and Ward (1996) go so far as to characterise British by-elections as “barometer elections” through which government performance can be examined. Feigert and Norris (1990) test competing theories, including one specific to the local campaign, but find that in the British (and Canadian) cases, the referendum theory approach is the most convincing. Across the existing British studies (see for example Pollock 1941; Butler 1949, 1973; Mughan 1986) is that “by-election results tend to exaggerate a government’s unpopularity” (Stray and Silver 1980, 270) – the party comprising the national government almost universally tends to do badly at by-elections. Not all by-elections are equal; however, the relevance of the point in the electoral cycle at which they occur is identified by Butler (1973) and picked up subsequently by Stray and Silver (1980, 1983). A by-election taking place at the point of the mid-term slump may display different results for the governing party than if it were to take place close to a general election. This echoes similar findings on European Parliament elections (Reif, Schmitt, and Norris 1997), where results were shaped by the point in the national cycle at which the elections fell. Mughan (1986) also finds some support for this, with better results for the governing party at points at which they were doing better in national polls. This referendum theory approach is not universally supported, and even Mughan (1988) admits that it has relatively poor explanatory power when examining by-election results. As Norris (1990) highlights, such concentration on national factors means that other potentially valuable explanations have been side-lined – particularly that which focuses on the local community and political context. As such, the first two hypotheses for this paper amalgamate the overlapping positions of second order contests and referendum theory:

H1: Turnout is lower at by-election contests than general elections.

H2: National factors shape by-election voter behaviour.

In opposition to the nationalised basis of explanation conceived in the second order/referendum theory approach, other research (Norris 1990) indicates the importance of local factors. What local factors comprise is by no means universally supported – in his evaluation of the referendum model Mughan (1988, 36) tests a range of local factors including the number of parties and marginality that he terms “contest specific”, but a more useful understanding of the local is to isolate contextual and contest specific factors under the term local. Contextual factors are those that frame the contest itself – marginality, previous support, tenure, whereas contest specific factors are the active shaper of the contest – the number of parties, the reason for the by-election and whether candidates are local. Specifically in relation to by-elections, Norris and Feigert (1989, 128) identify the “stable dealignment” transition occurring in British voter behaviour from the 1970s onwards making by-elections increasingly responsive to local contest-specific factors, including candidates and the campaign. There is additional evidence of the importance of local factors in by-election contests comparatively, as Gaddie, Bullock, and Buchanan (1999) indicate that they outweigh the importance of the national in the US, also supported by Feigert and Norris’ comparative study (1989) which finds additional evidence in the Australia context. As such, the article also explores the importance of local factors in by-election voting with H3 – additional sub-hypotheses associated with this hypothesis are described below:

H3: Local factors (contextual and contest-specific) shape by-election voter behaviour.

Delving into the contextual dimension of local factors, it can relate to the departing MP’s tenure: Mughan (1986) finds that tenure – and ergo their personal vote – shapes by-election results. However, one important local contest-specific factor relating to the departing MP is the reason for the triggering of the by-election. Although Norris (1990) hint at causal differences impacting voter behaviour, there has been no further disaggregation and examination of the causes of by-elections. Parliament identifies four triggers for British by-elections (Priddy 2019): the resignation or death of the incumbent, an MP being declared bankrupt, mental illness preventing an MP from performing their duties and finally an MP being convicted of a serious offence. An innovation to by-elections was introduced in 2015, with the Recall of MPs Act; under this Act, if an MP is convicted of an offence, suspended after a Standards committee review or convicted for fraudulent or misleading allowances (Johnston and Kelly 2020), a petition is opened in that MP’s constituency, with a by-election triggered if more than 10% of the electorate sign. To date, two by-elections – Brecon and Radnorshire and Peterborough, both in 2019 – resulted from a successful petition. Besides these factors identified officially by Parliament, in practicality an MP may trigger a by-election to take up another role, to be re-elected on a point of principle, to gain legitimacy following their defection to another party or in the face of negative publicity regarding their behaviour.

Yet little has been done to explore the impact a by-election’s cause has on the result, so we must look to other literatures to indicate what to expect. In terms of scandalous departures, there is a growing body of research that explores the electoral impact of scandal, with Praino, Stockemer, and Moscardelli (2013) finding that it encourages turnout from challenger supporters, but also in voters turning away from governing parties (Pattie and Johnston 2012). Scandal may encompass a politician’s personal life or allegations of corruption (or both – see Dobratz and Whitfield 1992), and there is mixed evidence that this can have a significant impact on the way that people vote. Abramowitz (2001) saw particularly marked patterns of voting in the 1998 mid-terms that matched opinion-taking on the Clinton/Lewinsky scandal, although Fischle (2000) found these largely reflected pre-scandal opinions of Clinton. However, as Funck and McCabe demonstrate (2021), we should be cautious of drawing too linear a relationship between scandal and voter behaviour, as information and partisanship can be interacting influences. Time heals though – as Miller Vonnahme (2014) shows voters can forget and (in the case of party supporters) forgive scandal.

In the United Kingdom, Farrell, McAllister, and Studlar’s (1998) study of Conservative sleaze ahead of the 1997 election indicates that British voters punish moral and financial scandals equally – whereas moral scandals have greater electoral traction in the US (Welch and Hibbing 1997). Albeit in general elections, Farrell, McAllister, and Studlar (1998) indicate that the electorate punish the party nationally rather than specifically in that MP’s constituency. Pattie and Johnston’s examination of the 2009 MP expenses scandal (2012), while focusing exclusively on financial misdemeanour, affected the likelihood of voting Labour more so than it did the Conservatives; the actual electoral impact of the scandal was less than expected. While there have been no studies to date of scandal specifically at by-elections, it is therefore plausible that a scandalous departure as a contest-specific local factor may depress both turnout and vote share for the successor candidate.

H3a: By-elections caused by scandal see comparatively lower turnouts than those triggered by other causes.

H3b: By-elections caused by scandal see the incumbent party perform worse than at those triggered by other causes.

Local contest-specific factors can also relate to the incoming candidates. One of the few more recent studies on by-elections, Evans (2012) finds that being local is perceived by parties to be a distinct advantage in candidate recruitment for such contests. More than half of the main party’s candidates in the contests she covers were able to be defined as local. The wider literature (Childs and Cowley 2011; Arzheimer and Evans 2014) also emphasises the importance of the local in party actions and explanations of voting. As Campbell and Cowley (2014) demonstrate, voters tend to prefer candidates with ties to the local area, and candidates are at pains to stress this, arguably in the hope of behavioural localism (Campbell, Cowley, Vivyan and Wagner 2019); the appreciation of particular issues facing communities and the ability of a local MP to defend them. As such, voters should react more positively when candidates standing for their vote are local:

H3c: Local candidates standing for the incumbent party perform better than non-local candidates

By-elections in contemporary British politics

To examine contemporary voter behaviour in by-elections, this article uses data on 148 such contests held in Britain over the last eleven parliamentary terms; from the Manchester Central by-election that occurred just four months after the start of the1979 parliament, to the two by-elections that took place on 23rd June 2022. As such, this is a significant update to the data with which to examine British by-elections. It excludes those occurring in Northern Ireland, due to party system differences. Two by-elections held in the same year (1990) in Bootle are also excluded due to a lack of comparability, being the only seat in the time period under examination where multiple by-elections have been held during the same parliamentary term. In addition, the 2016 Batley and Spen by-election and the 2022 Southend West by-election, both triggered by the murder of the incumbent MPs, were excluded as the major parties did not contest the incumbent party’s candidate (Table 1).

Table 1. By-elections 1979–2022.

Causes
Death of incumbent	88
Resignation	42
Scandal	13
Defection	3
Void	2
Party incumbent
Conservative	56
Labour	85
Liberal Democrat	4
Other	3
Where seat changed hands at by-election	48
Of which reverted back at GE + 1	23
N	148

Note: five seats where by-elections have resulted in seat changes have not at time of writing had a GE+1. As such the figures for reversions relate to a n of 43.

On average there have been 13 by-elections per parliament across the period, but the rate of such contests varies across different parliaments; the most (19) occurred in the 2010 parliament, whereas the 2017 parliament only saw four. The latter figure is perhaps not unsurprising, given the truncated nature of this parliament caused by the snap general election two and a half years later. By the end of 2021, six by-elections had been held in the 2019 parliament, already outstripping this figure.

Causes of by-elections

Disaggregating by-elections into the reasons they are called, the most common reason for holding one across the period is due to the death of the incumbent MP, echoing Norris’ findings from earlier by-elections (1990). However, it is important to note that this proportion has been broadly in decline since 1979; of the 17 by-elections taking place in that parliament, 71% were due to the death of the sitting MP. Yet, since the 2010 election, by-elections are increasingly being caused by other factors, with the death of a sitting MP comprising only between 22 and 33 per cent of such contests. The rate of by-elections caused by MP resignations has likewise fluctuated, comprising 77% in the 2015 parliament but only 11% in the 1992 parliament. Scandal appears to be increasingly important, however, with the earliest MP in the period to resign over scandal being Piers Merchant in 1997. The rise in scandalous resignations may be due to the spread of the internet and social media over the latter period, enhancing public awareness of such behaviour and making an MP’s continued service untenable. While by-elections in the 2010 parliament were particularly “scandalous” at 21%, already in the 2019 parliament 46% of the by-elections so far have been caused by scandals; most notoriously the Shropshire North by-election on 16th December, called after the incumbent Owen Paterson broke lobbying rules and the government caused considerable controversy by trying to protect him. Three MPs (one in the 1979 parliament and two in the 2010 parliament) defected to other parties – which does not automatically trigger a by-election – but the incumbent can trigger a by-election if they perceive value in maintaining their constituency mandate. Only one of the three (Douglas Carswell, defecting from the Conservatives to UKIP in 2014) successfully retained his seat. Two by-elections were called after the general election result was found to be void after inaccuracies in the count and the falsification of information about a rival candidate.

Seat change

More than two-thirds of by-elections do not result in seat change, with the locally incumbent party retaining it at the contest. Perhaps unsurprisingly the Liberal Democrats are particularly successful at causing by-election upsets, gaining 50% of all 48 seat changes at by-elections across the period, echoing Norris (1990). They have taken seats from both the Conservatives and Labour over the years and are well renowned for their strong local organisational capabilities (Cutts 2006). It is also not a given that a seat lost at a by-election will be regained at the subsequent election; of the seats which changed hands at a by-election, 39.5% reverted to the incumbent party at the previous general election (excluding contests where a subsequent general election has not yet occurred). Where seats fail to revert, it is not always the case that the party who gained it at the by-election will be the victor at the next general election. For example, in Littleborough and Saddleworth’s 1995 by-election, the Liberal Democrats won the seat from the Conservatives, but lost it at the 1997 general election to Labour. Considering seat change against the reasons behind by-elections also provides some interesting results. Defections universally have (unsurprisingly) resulted in seat change, with voters following the candidate, not the party. The next highest proportion of seats changing hands occurs in by-elections triggered by scandal at 46.2%, compared to 35.2% change for those caused by the incumbent’s death. 19% of by-elections triggered by resignation lead to seat change. Conversely, void contests have not resulted in any seat change.

Candidates

According to Butler (1973 and appearing elsewhere – see Mckenzie 2009), another key assumption from the existing literature is that by-elections can be useful opportunities for locally incumbent parties to reintroduce former MPs, particularly so-called “heavyweights” into Westminster, either after a period of semi-retirement or after a shock defeat. However, in a more recent examination of by-election candidate recruitment from 1945 to 2010, Evans (2012) finds that political insiders – the category includes former MPs and advisers – do not perform well at by-elections; instead, local candidates perform better. Exploring whether by-elections are still important opportunities to readmit former MPs to Westminster in contemporary by-elections presents another case for updating the literature. The last time a former heavyweight (here classified as having served in a Cabinet or Shadow Cabinet-level position) – or indeed a former MP – was successfully re-elected to Parliament through a by-election was in 1999. Indeed, over the period from 1979 to the end of 2021, only 6% of by-elections have seen the incumbent party choose a former MP to fight; since 1999, only two former MPs have tried to re-enter Parliament via this route. Although in the 1979 parliament we see key examples of heavyweight former MPs re-entering Parliament via by-elections for seats they had no prior connection with, since then former MPs who have been chosen to fight again have been more junior. The most recent former MP to fight a by-election was Paul Williams, who represented Stockton South between 2017 and 2019. Chosen to fight the 2021 by-election in Hartlepool, his selection was controversial – partly because the party appeared to hasten his selection by minimising the application window (Stewart 2021), but also due to dissatisfaction with the way in which he had represented his constituents during the Brexit process (Rea 2021). As such, by-elections no longer appear to be effective ways of reintroducing party heavyweights – or any previous MPs – to Parliament, in contrast to the existing literature. Instead, being local appears to be important in contemporary British electoral politics, and Evans (2012) also gives a sense of the dismay that can greet a centrally parachuted candidate in constituency about to fight a by-election. To explore the extent to which candidates in by-elections are local, any prior associations are identified – whether they have stood in a previous election in the seat, have served on the local council prior to the by-election, grew up in the area and work or have been educated locally – drawing on publicly accessible information from MP biographies and searches of newspaper reporting. To counter for boundary changes that may have occurred, the definition of local here has been extended to cover county lines. An additional measure identifying associations with geographic regions is also incorporated (Table 2).

Table 2. Local and regional connections of by-election candidates, 1979–2022.

	Conservative	Labour	Liberal Democrat
No ties	20.0	9.6	12.6
Local ties	56.2	76.3	65.5
Regional ties	23.8	14.1	21.8
	130	135	119

Particularly with earlier contests in the period under examination, it was not always possible to identify a candidate’s local connections. However, there are some clear patterns. Conservative candidates in by-elections are the least likely to have connections to the region or county. Where they do have connections, they tend to be of the more geographically remote regional type. Labour is the party with the highest proportion of local candidates. Surprisingly, for a party whose electoral fortunes have often seemed closely intertwined with by-elections, the Liberal Democrats have lower levels of local ties; as a smaller party it may be that they are not sufficiently geographically distributed to always field a local candidate in every election. It is worth noting, however, that by-elections can themselves prove training grounds for future MPs. Of recent Prime Ministers, Liz Truss, Tony Blair and Theresa May all fought and lost by-elections before being successfully elected to Parliament.

Turnout and voting in by-elections

As discussed above, by-elections do not always result in seat change and there appears to be an increasing trend towards contests triggered by the behaviour of the incumbent. This section explores patterns of turnout and voting behaviour at by-election contests in further detail, drawing on the hypotheses already described.

Turnout

As H1 describes, both second order understandings and referendum theory posit that turnout in a seat will be depressed in by-elections compared to general elections. Average turnout in by-elections across the period was 49.3%, compared to an average performance in the same seats of 68.3% in the previous general election. The last time turnout at a by-election rose from the previous general election was in Greenwich in 1987. While turnout in Darlington’s 1983 by-election, just 77 days before the general election, was 80%, turnout in Barnsley East in 1996s by-election was just 33.7 with 141 days until the 1997 election. However, falls in by-election turnouts appear relative to pre-existing patterns of turnout in the constituency, with a moderate correlation (.691***), that appears unrelated to the proximity between the by-election and the subsequent general election. In short, turnout will – as expected in H1 – typically fall at a by-election, but it will echo the typical levels of participation in the seat (Figure 1).

Figure 1. Turnout and turnout change at by-elections versus previous general election by cause.

It is plausible that voters may feel more or less motivated to turn out to vote in seats depending on the reasons for the contest. Much of the literature on the impact of scandal on turnout indicates a mixed effect coloured by partisanship, with Praino, Stockemer, and Moscardelli (2013) identifying scandal increasing turnout in certain contests. The two void contests of 1997 and 2011 – unusual occurrences in themselves, being the first since 1923 – had high levels of turnout compared to contests with other causes. Conversely, by-elections held in seats because of scandalous behaviour, experience comparatively low turnouts, with an average of just 45.2%. While looking simply at the turnouts in by-election contests would seem to indicate that there are appreciable differences in turnout according to the cause, examining the percentage point difference in turnout on the previous general election finds negligible variation in turnout across all contests (between −18.5 and −20.2 for all causes except void elections). Again, this reinforces that by-elections reflect the broad patterns of voting behaviour already inherent in constituencies.

To explore turnout in by-elections in further depth, a parsimonious OLS regression model has been constructed, with by-election turnout as the dependent variable. Three sets of independent variables have been included: the first set explores H3 by approximating the local constituency context specifics of the by-election, and as such the local frame within which the contest takes place: the number of years the departed MP served the seat to encapsulate something of the personal vote they may have possessed (see Cain, Ferejohn, and Fiorina 2013), the turnout in the seat at the prior general election, who holds the seat, and a measure of marginality based on the previous general election. The second set of variables examine key local contest-specific factors in the contest in relation to H3; specifically the reason for the by-election (identifying only the two most common causes – death and scandal, where scandal = 1), a binary variable identifying whether the candidates for the parties have ties to the local area and finally how many parties are contesting the by-election. This latter variable is based on Stray and Silver’s (1980) finding that the number of candidates plays an important role regarding splitting the vote, but for the purposes of this model, a log is produced to counter for the multiple additional parties that tend to emerge at by-elections. Lastly, variables measuring national factors are entered to explore H2 as per Mughan (1988); an average of the retail price index for two months preceding the by-election and an average percentage unemployment figure for the same period. Both were sourced from the Office for National Statistics (2022a, 2022b), with the two-month time period selected to provide some lag for the impacts to be felt by voters. Regional figures for these variables may produce a more accurate perspective on the conditions for voters in seats where by-elections were taking place, but data to this level of granularity are unavailable. A binary indicating the party of government and polling figures for the parties (Pack 2022) in the month in which the by-election was held are also present. While constituency-specific polls have been conducted in recent years (Lord Ashcroft 2014), a desire for comparability across the period precluded their inclusion. As part of the model construction, a year variable was also trialled to control for time-trends. This, however, proved to be almost perfectly collinear with the retail price index variable (Table 3).

Table 3. National and local factors in by-election turnout.

Context specific
Tenure	.068 (.106)
Marginality	−.191 (.076) *
Previous turnout	1.108 (.180) ***
Conservative seat	−.667 (2.977)
Contest specific
Reason	2.369 (3.048)
Conservative local candidate	.850 (2.140)
Labour local candidate	−3.623 (2.334)
Lib Dem local candidate	.770 (2.074)
Number of parties standing	1.677 (2.784)
National factors
Inflation	−.011 (.009)
Unemployment	.080 (.934)
Governing party (Conservative = 0, Labour  = 1)	2.341 (3.596)
Conservative polling	.249 (.203)
Labour polling	−.362 (.189)
Lib Dem polling	.243(.246)
Adjusted R^2	.720

Note: Table displays beta values. P < .05 = *, p < .01 = **, p < .00 = ***. Standard errors in parentheses. N = 145. Dependent variable is turnout at by-election. ***p < .001.

The results provide some indications that a mixture of national and contextual factors determine turnout in by-elections, with 72.0% of variation in by-election turnout explained. Firstly – and confirming key expectations from voting behaviour literature in general elections – controlling for other factors, the safer a seat is, the lower the turnout (p < .05). Evidently, even in extraordinary, unplanned events such as this, voters – and by implication the party campaigns themselves – still recognise conventional vulnerabilities. Echoing expectations discussed above, it does also appear that turnout in the prior general election significantly impacts turnout in the by-election. Although none of the local contest-specific variables, including the reason for the contest and whether candidates were local produced any significant results, thereby finding little support for H3a, it would therefore appear that H3 is partly supported – local contextual factors matter in by-election turnout, rather than contest specific factors. Considering the national variables, there is a very slight but insignificant fall in turnout when inflation increases ahead of the contest. The better Labour perform in national polling around the time of the by-election, the lower the turnout, controlling for all else, including whether they were in government at the time. Yet none of these national factors are statistically significant. There is therefore only limited support for H2 evident here – national factors do not significantly matter in explaining turnout at by-elections, rejecting key findings in relation to both second order contests and referendum theory.

Voting patterns

Across the period, the government sees an average fall of 12 percentage points at a by-election compared to less than a percentage point for the national opposition. Of the 48 seats that changed hands at by-elections between 1979 and 2022, just three were instances where the nationally incumbent party gained the seat from another party; the two most recent occurred in Copeland and Hartlepool in 2017 and 2021 respectively, where “red wall” Labour seats were gained by the Conservatives. While this is insufficient to make claims of a shift towards consistently more volatile voting behaviour in by-elections, it does indicate that voters do not consistently punish the government. Mapping the national government party’s performance at by-elections versus the previous general election, there are also indications that a shift may be taking place. While the average performance for the governing party between 1979 and 2005 fell by 14.7 percentage points, from 2005 onwards it has only fallen by 8.8 percentage points on average. It does not automatically follow, however, that the national opposition will be the beneficiary. There have been two parliaments – 2001 and 2017 – where the national opposition failed to increase their vote in any by-elections, and their performance is inconsistent across the period. Instead, confirming existing analyses (Norris 1990), the Liberal Democrats (and their predecessor parties) are particularly strong at by-election contests in contemporary British politics: at every one of the 148 contests in the period under examination they have improved their share of the vote.

However, the web of incumbency in Westminster means that things are more complex than this somewhat simplistic picture paints – particularly, as Table 1 demonstrates, the majority of by-elections have taken place in Labour constituencies. Only a quarter of these by-elections have occurred while the party was in government. By just concentrating on national patterns of incumbency, a vital piece of the puzzle is absent. An increasing value is being placed on local matters in British politics more generally, so incumbency at the constituency level should also be considered.

To explore H2, 3 and the sub-hypotheses in relation to voting behaviour at contemporary by-elections, four further OLS regressions have been conducted, largely based on Table 3 and therefore examining the impact of context specific, contest specific and national factors on voting patterns. The first two models explore voting patterns focusing on incumbency rather than party – as such the dependent variables for these are the vote share received by the national government party at the by-election, and the vote share received by the local incumbent party at the contest. The following three models use party-specific dependent variables, examining the by-election vote shares of the Conservatives, Labour and Liberal Democrats. A few further modifications to the model have been introduced to retain parsimony and avoid multicollinearity. Firstly, previous turnout has been replaced by previous vote share for the party/incumbent identified in the dependent variable. A binary identifying whether the seat is in the current possession of the party/incumbent identified in the dependent variable is also included. Whether the party/incumbent is fielding a local candidate and party/incumbent specific polling data are also slight modifications (Table 4).

Table 4. National and local factors in by-election performance.

	National government	Local incumbent	Conservative	Labour	Liberal Democrat
Context specific
Tenure	.009 (.097)	−.086 (.113)	.026 (.081)	−.050 (.092)	.166 (.126)
Marginality	−.037 (.066)	.174 (.177)	−.084 (.058)	−.252 (.068) ***	.144 (.094)
Previous support	.361 (.089)***	.306 (.295)	.814 (.087) ***	1.102 (.107) ***	1.522 (.164) ***
Own seat	23.636 (2.225)***	-	−.655 (2.810)	−5.538 (3.724)	−26.048 (8.434) **
Contest specific
Reason	1.534 (3.620)	−1.120 (4.517)	1.103 (2.721)	−7.565 (3.464) *	1.508 (4.131)
Local candidate	−3.212 (2.175)	3.084 (2.329)	−1.171 (1.645)	1.241 (2.117)	2.083 (2.597)
Number of parties standing	.650 (2.500)	−6.430 (2.948)*	−1.269 (1.990)	−1.326 (2.590)	4.437 (3.202)
National factors
Inflation	0.12 (.007)	.017 (.007) *	.002 (.005)	.013 (.006) *	−.007 (.009)
Unemployment	.678 (.745)	1.496 (.855)	−.536 (.611)	.844 (.740)	−3.461 (1.164) **
Governing party (Labour = 1)	4.215 (3.158)	−.131 (3.210)	−10.650 (2.286) ***	−10.777 (2.680) ***	−10.919 (3.868) **
DV polling	.560 (.139) ***	.545 (.157) ***	.630 (.126) ***	.626 (.121) ***	1.561 (.212) ***
Adjusted R^2	.716	.457	.787	.815	.685

Note: Table displays beta values. P < .05 = *, p < .01 = **, p < .00 = ***. Standard errors in parentheses. N = 148. Dependent variable is identified party vote share at by-election.

The regressions present mixed results. Firstly, the model exploring national government performance seems to offer a useful encapsulation of patterns at contemporary British by-elections, with 71.6% of variation in vote share explained. There is some support for H3 as both the party’s previous support in the seat and whether the government hold the seat are two context specific factors that significantly impact the vote share of the national government. Indeed, holding all else constant, the performance of the national government at a by-election contest is boosted 23.6 percentage points in seats they hold compared to seats they do not. No contest specific factors offer significant results. There appears to be some support for the importance of national factors as per H2, as like Mughan (1986), polling seems to matter; the government does significantly better in by-elections where it is ahead in the polls, holding all else constant. The picture for local incumbent parties is a little more hesitant, with far lower explanatory power. However, there is an intriguing indication that for local incumbents, an element of the local contest does matter, hinting at support for H3 – as the number of parties increases, the vote share for the locally incumbent party significantly decreases, due perhaps to the fragmentation of the vote. However, H3b and 3c are not supported, with non-significant results for the reasons causing the by-election or whether their candidate is local. Conversely, for the local incumbent party, national factors appear to matter, providing support for H2. As the retail price index rises, the local incumbent party make miniscule (but present) significant increases in their vote share. The better the local incumbent party is performing in national polls also significantly improves their local by-election performance.

Turning to the party-specific dependent variables, the patterns are largely parallel across the parties. Firstly, for the context specific factors, the vote share for all three parties at the previous general election significantly and positively impacts their vote share at the by-election. For Labour only, as the previous majority increases, the party sees a significant decline in their vote share at the by-election, holding all else constant. Where Liberal Democrats are fighting by-elections in their own seats, their performance at the by-election is significantly reduced by 26.0 percentage points. We should therefore modify Cook and Ramsden’s (1973) assumption that smaller parties are expected to do well in by-elections; in the case of the Liberal Democrats, their performance depends on whether they are the locally incumbent party. Intriguingly, for Labour, where a by-election has been triggered by scandal, holding all else constant, Labour vote share declines by 7.6 percentage points when compared to contests triggered by the incumbent’s death. There are therefore indications that context and contest specific factors do matter and H3 is partly supported. There are considerable synergies between the parties when it comes to the national factors in impacting their vote share; notably that as the party increase their polling by one point in the polls leading up to the by-election, both Conservatives and Labour can expect an increase in their by-election vote share of approximately 0.6 percentage points, holding all else constant. The impact for the Liberal Democrats is larger, with a one point increase in their polling ahead of the by-election significantly increasing their vote share by 1.6 percentage points. Echoing the perception of by-elections as a vote on the government expressed in both second order and referendum theory and articulated in H2 as the primacy of national factors, it is notable that the vote share of the party forming the national government significantly falls at a by-election; for both parties, holding all else constant, when they are in government, their party vote share falls significantly by 10.6 (Conservative) or 10.8 (Labour) percentage points. For the Liberal Democrats (where the government binary used indicates Labour), a Labour government significantly reduces Liberal Democrat vote share by 10.9 percentage points. As such, looking at party-specific vote share at by-elections would indicate that a combination of national and context specific factors matter.

Conclusion

By-elections in Britain occupy a purgatory between intense media attention and a comparative lack of interest from contemporary academic studies. Billed by the media as a way of understanding how well the national parties are performing and responded to by parties using them as testing grounds for new candidates and campaigning techniques, academic understandings of voter behaviour in such contests to date is based on evidence often more than seventy years old. This article has taken an important step in updating the state of knowledge on such contests, questioning some outdated assumptions and indicating the continued relevance of others.

Increasingly by-elections are occurring not because of an incumbent MP’s death, but because of their resignation (whether associated with scandal or not). Expectations over who stands are also changing, with by-elections no longer representing a way in which former MPs and heavyweights can be replanted into Westminster. With an increasing focus on the importance of local candidates in voter behaviour, by-elections are no different – candidates do typically have local connections, reflecting the post-modernist move towards the value of locality, although some parties find a higher proportion of local candidates than others. By-elections also serve to enhance existing voting behaviour patterns in the seat, rather than change them entirely; turnout in by-elections may be almost universally lower than in the preceding general election, but seats with lower turnouts in the earlier election see lower turnouts in by-elections too. It is therefore too simplistic to say that the low turnouts in by-elections are due to the contests being second-order.

We should not fall into the easy trap of thinking that by-elections mean nothing beyond a temporary levering of frustration with the government. It does not always follow that governing parties do badly and the national opposition do well. Rather than by-elections being entirely driven by either national or local factors, the state of contemporary British by-elections indicates that it is a combination of the two. When considering turnout at by-elections, local context-specific factors such as the seat’s marginality and previous turnout matter. In understanding who by-election voters are choosing to vote for, again this split between context-specific factors of previous vote share and marginality matter, as do national factors such as which party is in government and how a party is doing in the polls.

Exploring by-election performance of national and locally incumbent parties, a collision of national and local factors matter in determining the result. It is not possible to reject the referendum theory of national issues mattering entirely, but by-elections need to be thought of more cohesively as contributing to the electoral experience of a constituency rather than a temporary loss of sense by voters. Although the contest specific factors such as the reason for the by-election and the importance of local candidates have resulted in little of significance, this does not rule out their importance – more individual-level research needs to be done in respect to what degree causes of by-elections enter the voting decisions of voters before it is dismissed entirely, but there are some initial indications already that participation does vary according to the cause.

Disclosure statement

No potential conflict of interest was reported by the author(s).