More than a digital system: how AI is changing the role of bureaucrats in different organizational contexts

ABSTRACT

The paper highlights the effects of AI implementation on public sector innovation. This is explored by asking how AI-driven technologies in public decision-making in different organizational contexts impacts innovation in the role definition of bureaucrats. We focus on organizational as well as agency- and individual-level factors in two cases: The Dutch Childcare Allowance case and the US Integrated Data Automated System. We observe administrative process innovation in both cases where organizational structures and tasks of bureaucrats are transformed, and in the US case we also find conceptual innovation in that welfare fraud is addressed by replacing bureaucrats all together.

KEYWORDS:

• Public decision-making
• artificial intelligence
• digital welfare system
• bureaucrats
• public sector innovation

View correction statement:

Correction

Introduction

In the digital age, public service delivery is changing in very visible ways when looking at (partially) automated decision-making processes. Citizens receive notifications with no or limited human intervention and AI systems take over routine assessments of, for example, welfare claims. Much of the attention around these changes has been both on the discretion and limited face-to-face interactions of street-level bureaucrats (Lipsky 1980; Bovens and Zouridis 2002; Keiser 2010) as well as on the digital systems themselves as they are integrated into public institutions (e.g. Ransbotham, Kiron, and Kirk Prentice 2015; Höchtl, Parycek, and Schöllhammer 2016). These findings are somewhat disconnected from research that has shown that institutional variety shapes bureaucracies in what is expected from public decision-making and service delivery (Esping-Andersen 1990; Hupe and Buffat 2014). This literature ranges from looking at supervisory support of street-level bureaucrats to the link between institutional goal definition and individual performance (e.g. Brewer and Selden 2000; May and Winter 2007; Keiser et al. 2002).

This paper creates a link between these different sets of literature by defining the implementation of AI applications as a dynamic ecosystem. We look at the organizational context as well as agency- and individual-level factors to highlight innovative changes. This is based on the assumption that digitalization processes disrupt existing processes and ultimately change parameters of state governance and labour relations and that these tensions have further intensified with the growing trend to put in place artificial intelligence (Petropoulos et al. 2019). With this focus, we contribute to the ongoing discussion around, on the one hand, studies that highlight the transformational efforts being undertaken by government when introducing new technologies versus those that point towards a lack of empirical evidence for real public transformations (Coursey and Norris 2008; van Veenstra, Klievink, and Janssen 2011; Pedersen 2018; Tangi et al. 2020). We argue that there have been digitization efforts and a reliance on administrative data in the past, but the growing use of data-driven systems and algorithmic processing now suggests a new service delivery regime is in place. This is due to the fact that various AI technologies not only change the technical fields but also affect communication channels, decision-making functions and mechanisms as well as levels of control and discretion (Erdurmazli 2020). Thereby, we acknowledge that these dynamics are both driven by how the technology is designed as well as how it is integrated into the existing organizational set-up. We, however, focus in this paper on the integration into the organizational context and pay limited attention to system design.

We are particularly interested in identifying barriers and drivers that contribute to or hinder the transformation process and ask how does the implementation of AI-driven technologies in public decision-making in different organizational contexts impact innovation in the role definition of public bureaucrats? This question is explored by bringing together different strands of literature on public management and public sector innovation with the goal of offering comparative insights into system changes and the structures in which they are deployed. By defining innovation as an output variable of the process, we are able to zoom in on the changes in role definitions of bureaucrats specifically. Given the contextual nature of these changes, comparative studies are rare, however they contribute to our understanding to what extent national factors influence ways of innovation (De Vries, Bekkers, and Tummers 2014).

The case selection follows the Kuipers et al. (2014) rationale for public sector innovation by using discrepancies among majoritarian and consensus-based systems and the type of changes they exhibit as a starting point. Based on this, we study two cases that fall into these two categories: The case of the State of Michigan (US, majoritarian system) where the Automated Fraud Detection System (MiDAS) was implemented and the Dutch Childcare Allowance case (consensus-based system). Looking at both these cases, we find that in the US case, AI system implementation led to roughly one-third of the Unemployment Agency staff that dealt directly with citizens being let go and – after criticism – new roles needing to be created to provide oversight over the system and the data connected to it. In the Dutch consensus-based case, we find incremental changes affecting the organization structure while radical changes occurred in the work of bureaucrats that led to faulty decisions around childcare allowance. Based on these observations, we see new organizational structures and coordination of human resources in both cases, which is defined as administrative process innovation. In Michigan, we also see a conceptual innovation in that there is a new way of addressing the problem of fraud, which is that an AI system can handle the process more accurately, faster and effective than a human. These observations speak to a more radical innovation in the latter case and more incremental innovation in the Dutch case, however going forward more refined vocabulary might be needed to identify the innovative changes that are happening.

The paper is structured as follows, section 2 reviews theoretical concepts around the role and function of bureaucrats in (digital) administrative structures as well as research on the integration of algorithmic applications into existing administrative structures and what this means for re-defining the role of bureaucrats in increasingly digital public organizations. Section 3 contains the research approach and highlights legal and public documents consulted for the analysis. The following section dives into the two cases of the Dutch Childcare Allowance and Michigan’s Automated Fraud Detection System. Both cases are discussed and analysed in section 4. The final section concludes the paper.

Theory

Public sector innovation

The theoretical space at the intersection of organizational set-up and technical systems is occupied by different streams of research. ‘Non-system perspectives, such as NPM, emphasize organizational boundaries and siloed functions’ (Laitinen, Kinder, and Stenvall 2017, 6). In contrast, we focus on the integration of a new technical system, the organizational context and bureaucrats using it. This connects to the debate on the intersection of agency, structure, and technology (Orlikowski 1992; Giddens 1994) as well as the idea that service systems are relational and consist of learning interactions (Held 2006). These relationships create an environment where innovation occurs. In other words, the dynamic intersection of (new) technologies, organizational, and service set-up as well as those implementing the system create incremental or more radical innovations. Incremental innovations refer to minor changes to existing products or services, whereas radical innovations replace existing products or services ‘by introducing a new service or implementing a new delivery method’ (Wagner and Fain 2018, 1207). In recent years, this distinction of incremental and radical innovation has been further refined, and De Vries, Bekkers, and Tummers (2014) define four categories that target innovation in the public sector:

• Process innovation: focused on the improvement of the quality and efficiency of internal and external processes

• Administrative process innovations, which include organizational practices, new organizational structures and coordination of human resources;

• Technological process innovations where new elements are introduced into an organizational production or service operation system;

• Product or service innovation where new public services or produced are created;

• Governance innovation including new forms and processes to address specific societal problems; and

• Conceptual innovations that occur in relation to new concepts or even new paradigms to reframe the nature of a specific problem and its possible solution.

For innovation to emerge in public structures, scholars identify both organizational and individual-level characteristics as influential. Those include long-term planning and governmental support by senior management as well as fostering competencies within the workforce by investing in human capital, education and training at all levels (Agolla and Van Lill 2013; Wagner and Fain 2018).

In the context of innovation and change in the public sector, Kuipers et al. (2014) identify outcomes as one of five factors in the literature on innovation in the public sector (next to context, content, process, and leadership). The content of change concerns the systems and the structures in which they are deployed. Outcomes include how a change affects behaviour (Armenakis and Bedeian 1999). The behaviour and attitude of the professionals working with an AI system may change because of the introduction of the system. This may be intentional and be an objective of the innovation but may also be an unintended or unforeseen outcome. AI applications, like many other ICT innovations, are often regarded as (additional) support for making processes in organizations more effective and efficient. In other words; AI may facilitate organizations to develop superior capabilities, impacting the performance of the organization. This is in line with the process view as described by Pang, Lee, and Delone (2014).

Looking at how these dynamics evolve in the context of new technologies, and specifically artificial intelligence service systems, research shows that (technology) infrastructure is entangled with institutional dynamics. New technologies have increased capabilities in public organizations, but often not in areas planned or anticipated (Kraemer and King 2006; Benunan-Fich, Desouza, and Andersen 2020). According to a Delphi study by Benunan-Fich, Desouza, and Andersen (2020), the most important IT innovations in recent years are AI applicants as well as big data analytics. Looking at the implementation of such technologies, Magnusson, Koutsikouri, and Paivarinta (2020) find that there is misalignment among the strategic and operative layers. In addition, there is a lack of feedback loops among those layers, making it unlikely for public organizations to improve or change processes as they are being implemented.

This is in line with research that emphasizes how the implementation of information technologies in public sector organizations is a complicated process, and where change emerges incrementally (Meijer and Bekkers 2015). Yet at the same time, AI, and specifically data analytics, are often presented as capable of fundamentally or radically changing public services or policy (Höchtl, Parycek, and Schöllhammer 2016). Yet, even though at the context level the impact of AI is deemed a radical change, at the level of specific applications it could just be part of a continuous process to improve operations. The public sector is not unique in this; according to Vidgen, Shaw, and Grant (2017) many organizations are rather reactive, trying to put data to use, without necessarily being strategic about it. They argue that data analytics – to be actionable – should be treated as an ecosystem, in which data assets, technologies, and the value propositions should co-develop with the organization in which they are deployed, and with the required expertise and capabilities (Vidgen, Shaw, and Grant 2017). This presents a gap between, on the one hand, the reactive or incremental approach that organizations take, and, on the other hand, the fundamental change that the introduction of this type of technologies presents. A focus on the incremental changes (or ‘maintenance’ of existing practices) may underappreciate the fundamental character of the implementation of some AI applications. It is this tension between the potential for fundamental change brought about by these digital systems and the fact that they are deployed in existing processes and structures, that interests us going forward.

Role and function of bureaucrats in administrative structures

There are early indications that the role of bureaucrats is changing in an increasingly digitized and automated environment. Past research suggests this has to do with a multitude of factors that are often only indirectly linked to the technology or digital process itself, but rather factors that can be traced back to the organizational and supervisory structure as well the experience, culture, and actions of bureaucrats as individuals and in groups (Bovens and Zouridis 2002; Jansson and Erlingsson 2014). Some scholars have looked at the more fundamental effects of digitalization on how public sector organizations work (Frissen 1989; Snellen and Van de Donk 1998; Zuurmond 1994). For instance, Jane Fountain (2001) described a virtual state in terms of what elements of bureaucracy change due to the rise of information systems. For individual bureaucrats specifically, authors, such as Bovens and Zouridis (2002), imply that software will replace certain tasks or even street-level bureaucrats altogether. This paper goes beyond this claim related to the tasks, discretion and legitimacy of street-level bureaucrats and looks for a shift in the role of bureaucrats due to changes in agency-level characteristics that impact the vertical and horizontal relationships of bureaucrats. Generally speaking, these factors can be divided into agency-level factors and individual-level factors that work in concert (Brewer and Selden 2000).

In general, ‘implementing organizations provide organizational, managerial, and administrative imperatives that shape what happens at the operational level of service delivery’ (May and Winter 2007, 455; Keiser et al. 2002). Research on organizational change points towards the role of public managers, who implement a series of actions that change the organizational logic – from arranging information and training to clarifying the vision (Brewer 2005; Fernandez and Rainey 2006; Ashaye and Irani 2019). In short, management within public organizations matters and managers are an important determinant of agency performance through their presence in the workforce and their contributions in the decision-making process. Concerns already then arose around downsizing, specifically the management positions within public organizations that contribute to individual-level performance through guidance related to skills, group dynamics as well as motivation. In addition, Brodkin (1997) finds that ‘caseworker responsiveness to client needs was at least partly contingent on prevailing management pressures’ (Ibid, 16). Further, political attention has been shown to affect caseworker’s policy emphases. ‘Caseworkers are more willing to diverge from national goals when it is clear that their immediate political principles endorse that divergence’ (May and Winter 2007, 469). In short, ‘institutions shape social actors’ cognition by conferring identity – that is, by selecting the factors that are to be considered relevant in making decisions’ (Keiser et al. 2002, 555).

At individual level, there has been a discussion around the role of discretion for bureaucrats, and in particular, street-level bureaucrats, in an increasingly automated environment. Discretion is here understood as the ‘ability to make responsible decisions, individual choice or judgement’ (Sandfort 2000, 730). This point has been raised several times throughout the years as decision-making procedures became more and more streamlined. Brodkin (1997) describes the process in the 1990s when US state agencies established a complex system of incentives and demands that led caseworker’s attention to paperwork and documentation of every decision-making step and compares it to the changes in the 1980s where those same verification procedures were used to slow down or speed up processes for welfare applicants. In essence, discretion was redirected.

In this setting, some foresee new bureaucratic roles that correspond with digital processes. Bovens and Zouridis (2002) suggest that there will be three groups of employees: ‘(1) those active in the data-processing process, such as system designers and the legislative specialists, legal policy staff, and system managers associated with these processes; (2) management and those controlling the production process; and (3) the “interfaces” between citizens and the information system, such as public information officers, help desk members, and the legal staff charged with handling complaints and objection notices on behalf of the organization’ (Ibid, 180). This also means that system designers, legal policy staff, and IT experts are the new equivalents of street-level bureaucrats (Bovens and Zouridis 2002; Reddick 2004). However, more recent research highlights that even though face-to-face interactions are being phased out of public service delivery – a defining characteristics of street-level bureaucrats (Lipsky 1980) – individual bureaucrats will have a role in influencing policy implementation and making decisions about citizen-clients because applicants often do not fit into eligibility criteria (Keiser 2010). Thus, algorithms remain decision-support tools rather than become agents in their own right (Steen, Timan, and van de Poel 2021). In fact, the EC (2019) sets out the principle that ‘AI systems should follow human-centric design principles and leave meaningful opportunity for human choice’ (EC 2019, 12).

Overall, past evidence speaks to the fact that individual public bureaucrats still play a role in decision-making processes, however that their role has slightly shifted, and discretion has been redirected. This has to do with the fact that instead of bureaucrats dealing with the bulk of decisions concerning citizen-clients, they are responsible for exceptions or cases that need additional information, whereas system designers and IT specialists optimize the digital system linked to those applications that can be processed (semi-)automatic. Keiser (2010) highlights that bureaucratic decisions will be influenced by information about what other (digital) actors in the broader governance structure are doing. Hence, bureaucrats might make decisions based on what they anticipate will be confirmed by superiors and/or the digital system. In addition, Jorna and Wagenaar (2007, 189) find that ‘informatization does not destroy operational discretion, but rather obscures discretion’.

To conclude, it is apparent that not necessarily data and analytical techniques pose the greatest challenge (van Zoonen 2020), but rather the organizational context they are embedded in and the new coordination and management processes that individual bureaucrats have to work or perform in. Additionally, much of the research treats agency- and individual-level dynamics separately and thereby overlooks the effects that larger organizational changes have on the role of individual bureaucrats (Pandey and Wright 2006). In this set of literature, there are also first indications of agency- and individual-level factors enhancing each other. That means that once digital applications have been introduced, ‘the pressure to centralize the organization, to formalize the legal regime, and to standardize the work will increase’ (Bovens and Zouridis 2002, 181). Finally, dramatic policy changes often translate into few changes on the front lines (Destler 2017) but might change decision-making considerations among bureaucrats and slightly shift decision-making responsibilities by, for example, only handling exceptions rather than standard cases.

Integration of algorithmic applications into existing administrative structures

In line with recent work on digital transformation in the public sector, we therefore seek a more integrative view on the phenomenon we study, as digitalization changes bureaucratic and organizational culture (Mergel, Edelmann, and Haug 2019). Also, elsewhere in the literature on ICT in government, a systems perspective − or integrative or holistic view – can be found (Meijer and Bekkers 2015). For example, Fountain (2001) work on technology enactment emphasizes how organizational, institutional arrangements, rules and routines determine how the capabilities of IT are used.

Whilst we agree that the change induced by technology is socio-technical, this does not necessarily imply that changes in organizational routines and professional practices are an intended – or even acknowledged – consequence of the introduction of technology. As Meijer and Bekkers (2015) argue, citing work by Dutton and Frissen, ICT might help reinforce existing practices and interests, and strengthen the bureaucratic context in which novel systems are used. This argument is based – in part – on how ICTs depend on or even strengthen formalization (Frissen 1999). This may, however, not be the case for AI applications; whereas they might formally be positioned at a specific place in the process, their inscrutability for professionals might result in boundaries of the systems role and functionality being less clear in practice than in design.

Taken together, these sets of literature (one on public sector innovation, the other on the roles of bureaucrats in administrative structures) highlight relevant points for looking at the application of an AI system in a public welfare context. Research suggests that this process has to be understood in a holistic manner by analysing the ecosystem of (semi-)automated bureaucratic work – including the data, technology, organizational culture and capabilities (Vidgen, Shaw, and Grant 2017). This ecosystem can be disentangled to zoom in on individual- and agency-level factors. For the latter, it shows that the tasks and decision-making procedure of bureaucrats changes as they are less of an intermediary or translator between citizens and bureaucratic system (Jansson and Erlingsson 2014) and more the ‘interfaces’ between citizens and the information system (Bovens and Zouridis 2002). At agency level, research reveals that organizations are hardly capable of identifying bigger changes once digital systems are being introduced and mostly perform reactively in adjusting organizational practices (Bertot, Jaeger, and McClure 2008; Norris and Reddick 2013).

Combining both, we make the proposition that in both expected and unforeseen ways, AI affects not only the work of bureaucrats but also the content of the work and the context in which it is done. In this paper, we explore this claim by seeking to understand in what way the changes in bureaucrats’ work are innovative. These aspects are captured in the model presented in Figure 1.

Figure 1. Conceptual model.

Figure 1 depicts the relationship described in earlier research between the implementation of technology and its effect on organizational changes in the public sector. Public sector innovation is defined as the introduction of new elements into a public service through new knowledge, a new organization, and/or new management or processual skills, which represents discontinuity with the past (Osborne and Brown 2005; De Vries et al. 2014). According to De Vries, Bekkers, and Tummers (2014) these can range from process to conceptual innovation. The organizational context influences the implementation process and ultimately the changes occurring. In addition, agency- and individual-level factors play a role. This links to the set of literature that points towards shifts in decision-making procedures and the changes that result in the interrelations of individual agency and responsibilities of bureaucrats.

Research approach

In our theoretical lens we identify several factors that might affect whether the introduction of an AI system leads to public sector innovation. We seek to empirically explore this conceptual model in two cases studies from different contexts because whether the introduction of an AI system is treated holistically or incrementally depends on the context in which it is introduced. To inform our case selection, we loosely follow Kuipers et al. (2014)’s argument that majoritarian systems may exhibit more top-down reform, which could lead to more radical change, and consensus-based systems would rather display more bottom-up and incremental changes. In incrementalism the system-level change may be missed or misunderstood. A top-down approach may lead to a more integrative view on the change, potentially risking a misalignment with implementation. Without seeking to claim that majoritarian or consensus-based systems systematically lead to different innovation outcomes, we use this distinction to ensure that both potential dynamics are covered in our research design. We selected two cases for our exploratory analysis: one on the Dutch Childcare Allowance and one on Michigan’s Automated Fraud Detection System. The childcare allowance case is based in The Netherlands, which is considered a consensus-based system. The Michigan case is based in the United States of America, which is comparatively majoritarian (Kuipers et al. 2014). Both cases show similarities in that they have been widely publicized and scrutinized by public bodies, such as auditor reports in the Michigan case and parliamentary hearings in the Dutch case. In both instances, problems were traced back to the AI system and its use in the welfare context. Both are high-profile cases that have been extensively covered officially, leading to the public availability of very detailed information, that for other situations might not have been realistic given that this research covers a topic deemed sensitive at many organizations.

Data collection for the Dutch Childcare Allowance case is based on official reports and hearings. The case has been widely reported on in the Media and was extensively documented and analysed by supervisory bodies (the Data Protection Authority and the Auditor General) and the Dutch Parliament through a formal investigation into the case. The analysis of this case is based on publicly available reports and an analysis of the 19 public hearings that were done as part of the investigation of the matter by the Dutch Parliament. Some hearings were with high-ranking civil servants involved in the case. The hearings were taken under oath and took place within an eight-day period in the fall of 2020 and took a total of 45 hours. We analysed the transcripts of the public hearings, searching for segments that were on the (IT) system(s), including but not limited to the fraud detection model, the fraud ‘blacklist’ FSV, or referenced algorithm(s), data, the ways of working of the CAF team (the team tasked with identifying potential fraud-facilitators) as supported by the system (terms included ‘professionals’, ‘employees’, ‘public servants’), and the management thereof. These segments were not necessarily limited to the official that was questioned in a hearing, but also the enquirers, who were Members of Parliament and in their questions sometimes introduced relevant information available to them in their role. This is secondary data of high quality and relevance as these interviews were taken under oath and fully transcribed. Out of the 19 public hearings, the transcripts of six of the interviews were discarded as they did not yield relevant text segments. From the 13 interview transcripts that were included, a total of 69 text segments were extracted based on the search terms and (after further evaluation) on fit with the topic under study (e.g. sometimes ‘system’ would refer to the social security system or legal system rather than a digital system, so those segments were not included). These segments varied in length between one sentence to over half a transcribed page, but usually were about a paragraph long. Using this dataset of 69 text segments and the public reports we conducted an exploratory analysis and came to an extensive case description. This description was the basis for our analysis, which was corroborated using an anonymized interview with two subject matter experts in the government, and with the findings of an investigative journalist (reported in Frederik 2021). Our final dataset included the following data: 69 segments from 13 public interviews conducted by the Parliamentary Committee (Kamer 2020), an extensive report by the Dutch Data Protection Authority (Autoriteit Persoonsgegevens 2020), and a Dutch newspaper article citing from internal intranet/chat messages by bureaucrats from within the agency (Jonker 2021), an interview with two subject matter experts at the Ministry, a detailed report in a book by an investigative journalist (Frederik 2021). The interviews used in the case description in this paper are referenced by the name of the interviewee (as they were public hearings) and interview quotes were translated automatically to make them as neutral as possible.

The Michigan Integrated Data Automated System (MiDAS) has also been widely discussed not only in the Media, but also in legal proceedings, such as a class action lawsuit, in Auditor General Reports and new Legislation has been passed to accommodate changes in how the Michigan Unemployment Agency deals with MiDAS in connection to (fraud) claims. The analysis of the case is based on these different, publicly available documents, including: Legislation (House Fiscal Agency 2015), a Class Action Lawsuit (Bauserman v. Unemployment Ins. Agency 2017), two Auditor General Reports (OAG 2016, Ringler and CPA 2016) and an AI Now Institute Report (Richardson, Schultz, and Southerland 2019). These documents were analysed by focusing on changes in how cases were handled within the system, what role and effect the AI system had in this process and how this transformed the tasks and role of bureaucrats involved in the decision-making process.

Cases

Dutch childcare allowance

In the Netherlands, there is a childcare allowance, which is handled by a special department (‘Toeslagen’) of the Dutch Tax and Customs Administration (from here on: the agency). This allowance covers part of the costs of professional daycare and preschools. The amount covered depends on the income of the parents. Parents have to apply for these benefits and the agency pays it either to them directly or to the daycare centre. In the aftermath of a big fraud case concerning another type of benefit, also handled by the same Toeslagen department, fraud detection became important for the political and administrative leadership of the agency. Although there were signals for a while, in 2019 it slowly became clear that tens of thousands of parents were unfairly classified and treated as potential fraudsters, as small administrative errors led to parents having to pay up to tens of thousands of Euros, leading to loss of homes, work and other major life impacts. This led a Parliamentary committee to study the scandal, which ultimately led to the resignation of the Dutch cabinet Rutte III in January 2021.

Digital systems play an important role in this case. First because this agency, built for collecting taxes, had to implement the function of paying out benefits in a period of austerity. The systems played an important role in the attempt to make the department more efficient. Our case focuses on the introduction and use of a risk-classification mode that was introduced with a specific team (CAF) to address systematic fraud. This model was an AI system, more specifically a supervised learning model (interview Weekers). The model is based on an algorithm that was fed with examples of correct and incorrect applications in order to learn to ‘recognize’ risky ones. The model gave risk-scores and the applications or modifications that received the highest risk scores were selected to be processed by a professional (Autoriteit Persoonsgegevens 2020). In cases with higher risk scores, CAF would inform the department that decided on the benefits, which may lead to pay-out of benefits being stopped and reclaimed, and individual bureaucrats then further investigate, typically by asking for additional proof. Even failing to − for instance – show a receipt for a low amount of money could lead to the agency reclaiming the entire allowance. In practice, this affected the most vulnerable parents mostly; those with the highest debt and lowest incomes (Kamer 2020).

The system was an innovation born out not just efficiency seeking but also to meet a political demand to stop fraud. Yet the stated purpose and role of the system was described differently by different people involved: introducing checks early on, identifying new risks, identifying fraudulent facilitators, identifying fraudulent parents, supporting the primary process by prioritizing potential fraud dossiers for further human scrutiny, supporting the claim handling process, to counter human biases, or to introduce barriers for potential fraudsters (interviews Snels, Tuyll, Veld, Weekers, Wiebes). The use also changed: from supporting a search for fraud facilitators, to identifying parents, to real negative outcomes for citizens: ‘a hunt for facilitators, as it was called. As a fraud team, you apparently could not get your finger on that at a certain point. Then you make the step to: maybe I can find it with the parents. You then treat them from the outset as: they may have committed fraud’ (interview Van Tuyll).

The misalignment between perceived benefits or goals of the system and the actual intended role in the process, led to an innovation process creep that places the system in a context that it wasn’t designed to support, and vice versa. When turning to the role of bureaucrats and the organization, we find that this introduces a gap between the impact on formal role definition and the actual role change induced by the innovation.

Important here is that there are two teams: first the CAF team tasked with investigating fraud, but that was not tasked with looking into individual parents nor with deciding on benefits pay-out. Second, the team that was tasked with the allowance substantively, deciding on benefits for individual cases. Yet the system obscured this difference: ‘the model itself can deduce that some questions […] and it [the model] contains all the questions that are asked to, for example, receive an allowance’ (interview Snel). Consequently, the system gave the CAF team a bigger role, partially because the substantive team started to look at the system-identified cases differently, professionals started looking at those people differently: ‘You treat them from the start as: they may well have committed fraud. Then you will search very deeply’ (interview Van Tuyll). This leads to a shift in the work of the bureaucrats. ‘The shift in that balance was due to the fact that it became possible to carry out certain checks at the gate, so that a number of checks could already be carried out before something was provided’ (interview Weekers).

As a result there was a big difference in what the process formally uses this system for (detecting possible fraud) and what effect it has (everyone picked up by the system was seen and treated as potential fraud based on even small administrative errors). The individual bureaucrats in the substantive team formally still had a stable role in deciding on how to deal with potential fraud cases (Autoriteit Persoonsgegevens 2020). However, a high risk-score in itself now has a potential effect on the allowance payment, even when no fraud is established (yet). What is striking here is that the 100 cases with the highest risk score are presented to bureaucrats for checking. This means that high risk is not absolute but relative (Autoriteit Persoonsgegevens 2020). To what extent this was associated with a real high level of risk was impossible for the bureaucrats to scrutinize, as the bureaucrat on the case had no access to information on what factors led to a certain risk score (Autoriteit Persoonsgegevens 2020). They thus do not know why a case was classified as risky, signalling a shift of knowledge from the professional to the system. Furthermore, the bureaucrat is not offered the information needed to subsequently readjust their own role in the new situation (Autoriteit Persoonsgegevens 2020) and citizens have no idea how the decision came about (interview González Pérez).

Because there was a formal separation of roles between the fraud team and the team granting allowances, the management also assumed such a separation and failed to detect how the system shifted this (interviews Blankestijn, Blokpoel, Uijlenbroek). Yet even though the CAF was not tasked with deciding on benefits for individuals (interview Blokpoel), when it came up with a score, the substantive team further operated under the assumption of (potential) fraud and reversed the burden of proof, impacting the service process significantly even if the system had no direct role in it. There are even examples that the CAF itself took a decision (interview Blokpoel).

In terms of the bureaucrats’ discretionary space, this formally exists for the substantive team but the informal use of it (e.g. communicating or acting on bureaucrats’ views or impressions that a case was a matter of error rather than fraud) were suppressed by system outcomes (interview Veld). The CAF team outcomes thus de facto got a greater role, obscuring discretion. And because the formal picture did not change, the organization was also unable to see how much larger and more intrusive the role of the AI system had become, which had a major inhibiting effect on informal bureaucrats’ channels and on feedback mechanisms.

The interviews disclose that the organizational structure, management, and culture influenced the perceived discretionary freedom of civil servants (perceived to be minimal, even though it was formally there), inhibited functional feedback channels and limits the capacity for organizational learning (interviews Cleyndert, Palmen-Schlangen, Snel, Van Tuyll, Veld). Uijlenbroek (former director-general at the agency) also stated that they were ‘pressed into the straitjacket of those systems and structures, so that you can no longer make the real substantive assessment’. There was a strong organizational incentive to detect fraud, which provides a disincentive to look critically at system outcomes (interview Snel).

Zooming out to the higher officials and political arena, the use of AI is seen here as an instrumental choice at policy implementation (interview Wiebes). It was set up hastily and the procedures, work instructions and archiving were not or insufficiently adjusted, and there was too little expertise on various levels (interview Cleyndert).

Perhaps unsurprisingly, bureaucrats experience a big gap with the policy arena on the topic. At the ministry and at the political level, fraud and incident control were very important, leading to performance indicators regarding these. Yet, in the operational reality, a professional faces high work pressures, poor ICT systems and a pressure to deliver results, (Jonker 2021, citing public servants working in the agency). The cited internal messages (Jonker 2021) paint a picture of an organization driven by efficiency and effectiveness, also in terms of combating fraud, but without the organizational structure needed. The system and model led to unintended shifts in the role of bureaucrats, without clear policy, training, or explicated consideration about that shift.

Michigan’s integrated data automated system (MiDAS)

In the US, several states have started to use data mining techniques for automatic fraud detection in, for example, the food stamp program or unemployment insurance. However, this automation has proven to be unreliable as the Michigan case shows.

The state of Michigan implemented an automated unemployment insurance system with the goal of reducing operating costs and target fraud in unemployment insurance claims called the Michigan Integrated Data Automated System (MiDAS). The system proceeded to identify 26,882 fraud cases between March 2014 and 2015 as well as looked retroactively at claims in the six years prior (Behringer 2016).

The implementation of the system stems from the goal to modernize the Unemployment Insurance Agency (UIA) as well as streamline the benefit process for unemployment due to financial trouble of the State of Michigan and the bad shape of the UIA. ‘Even before the Great Recession, Michigan was in financial trouble. Unemployment was hovering over six percent in the years leading up to 2008, while incomes were stagnating compared to the rest of the country. When the recession struck, government revenues fell sharply, leading the state to cut more than $3 billion in spending between 2009 and2011’ (De la Garza 2020). In this context, the State of Michigan turned to Fast Enterprises, LLC that designed the Michigan Integrated Data Automated System (MiDAS) for $47 million (Richardson, Schultz, and Southerland 2019; De la Garza 2020). MiDAS was programmed to mainly improve efficiency by determining unemployment eligibility, track case files and “intercept income tax refunds for those ‘automatically selected by the system’, according to a 2013 Michigan Licencing and Regulatory Affairs Department memo” (De la Garza 2020). It replaced a 30-year-old mainframe system previously in place (OAG 2016).

This fully automated way of accessing unemployment claims as well as identifying fraudulent behaviour led to roughly 48,000 fraud accusations against unemployment insurance recipients − a five-fold increase from the prior system (Gilman 2020). Some Michigan residents on food assistance, for example, were automatically disqualified in the system and received the notice: ‘You or a member of your group is not eligible for assistance due to a criminal justice disqualification … Please contact your local law enforcement to resolve’ (Richardson, Schultz, and Southerland 2019). ‘Those individuals were sent an online questionnaire with pre-loaded answers, some of which triggered an automatic default finding against them. Automatic determinations of fraud also occurred if recipients failed to respond to the questionnaire within 10 days, or if the MiDAS system automatically deemed their responses unsatisfactory’ (Richardson, Schultz, and Southerland 2019, 20). Following this, tax funds were seized, wages garnished, and civil penalties applied in some cases and overall, costing claimants $56.9 million in fines (Behringer 2016). This led, in turn, to appeals, which were repeatedly denied by the system and some citizens turned to legal assistance. ‘Lawyers working on these cases soon discovered a disturbing trend: the state was frequently unable to provide evidence to support MiDAS’ fraud accusations’ (De la Garza 2020). This practice was upheld until 2015, even though lawyers raised concerns that MiDAS might be behind the false accusations without proper (human) checks in place.

Given these discrepancies, an audit was performed in 2016, which found that in only eight percent of the appeals actual fraud was detected, and 64% of claims had to be reversed or dismissed, while 22% were remanded by UIA (Behringer 2016; OAG 2016). The Office of the Auditor General (OAG 2016) also highlights a gap in how the system was managed internally, by saying that there was ‘a lack of reviews to ensure that privileges granted to database users were appropriate for the users’ job responsibilities’ (Ibid, 13) and that ‘MiDAS is a commercial off-the-shelf tax and benefit system that has been customized for the State of Michigan’ (Ibid, 29). Based on the audit, the state legislature passed a law in 2017 requiring UIA to make fraud determinations manually. In other words, prohibiting that fraud determinations could be solely made by the MiDAS program. A court settlement in the same year ruled that UIA had to review all MiDAS fraud determinations made between October 2013 and August 2015. ‘To date, Michiganders affected by MiDAS have received more than $20 million in refunds’ (De la Garza 2020).

Zooming in on the role of bureaucrats in the process of implementing the MiDAS system at the UIA, several points are relevant to highlight. First, with the implementation of MiDAS, 432 employees were laid off – roughly one-third of UIA’s staff. These layoffs reduced the number of employees working directly with customers from about 260 to 184 (Behringer 2016; Richardson, Schultz, and Southerland 2019). This led to UIA not being reachable: ‘During the 2016 audit, staff had failed or were unable to answer 89 percent of the calls placed to the agency, and 29 percent of callers who did get through later hung up while waiting on hold. A follow-up audit in late 2018 and early 2019 showed continued problems. As many as 79 percent of calls went unanswered, and callers who were able to get through hung up 28 percent of the time while on hold’ (Oosting 2020). The goal of issuing fraud claims without human intervention was reversed in October of 2015 when the Bipartisan House Bill (HB) 4982 was introduced and prohibited the UIA from making fraud determinations solely by computer program (Behringer 2016). Second, the audit emphasizes ‘the need for periodic security training specific to MiDAS data for UIA’s employees and its partners’ (OAG 2016, 9). This is linked to the fact that user accounts by employees were not managed well – this means that some users still had accounts while no longer being employed by UIA as well as some users had access rights that did not align with their job duties or position within UIA (OAG 2016). This speaks to a lack of expertise and training regarding the system itself as well as security gaps to protect the data in the system. In a follow-up audit in 2017, the UIA highlighted that ‘UIA has established a procedure for collection staff to document PACER [Public Access to Court Electronic Records] inquiries within the MiDAS system. These inquiries will be audited by management through the use of PACER activity reports. Additional staff was brought on board to ensure UIA is complying with documenting its searches of PACER’ (Ringler and Cpa 2017, 13). This implies that, at least for this specific function in the MiDAS system, additional staff was hired that performs a new function, which is largely focused on compliance. Finally, a recent report shows that not only did UIA hire additional bureaucrats to fill new tasks arising from monitoring the MiDAS system, but also ‘brought on contractors to answer claimant phone calls, conduct fact finding, and resolve simple adjudication issues’ − in particular to meet demands during COVID-19 times (LEO 2020).

Discussion

To summarize the main findings in both cases, the Michigan case depicts AI implementation as a means towards issuing fraud claims without human intervention. This led to one-third of agency staff being laid off and a lack of oversight and review of individual case decisions. The implementation led to rationing away bureaucrats and ‘standard’ positions in favour of the automated system – mostly targeting front-line positions. The decisions were taken fully automated from data analysis to issuing letters to (potential) claimants. The audit report criticizes limited training available to the remaining bureaucrats on how to handle the system and how to securely handle (access to) the data. In fact, as criticism grew, the UIA hired new bureaucrats with new functions largely around monitoring the system as well as involving third parties to keep up with the system-created demands. UIA was also legislated to add human control into the process to check the decision of the automated system before forwarding it to citizens.

In the Dutch case, we see that bureaucrats still play a relatively similar role in terms of decisions being made on fraud, however the AI system implementation led to a different type of work for which the respective departments did not have the required staffing. By working on a set of cases with a relatively high-risk score based on indicators not revealed to them, the content of bureaucratic work changed fundamentally, whereas the organization and its context treated it reactively as just analytics, without acknowledging the strategic shift. The overall goal underlying this was largely that of efficiency and effectiveness in combating fraud.

Hence, both cases are very similar when it comes to the goals linked to the implementation of both AI systems, which is to streamline the process – making it more efficient and cost-effective – as well as identify fraud more reliably. But, in the Michigan case, this goal goes one step further by aiming for a very limited or even no human intervention in the process to the point that staff is let go for specific tasks and replaced by an automated decision-making system, resulting in major changes to how bureaucratic work is defined.

Based on these observations, we see new organizational structures and coordination of human resources in both cases, which is defined as administrative process innovation. In Michigan, we also see a conceptual innovation in that there is a new way of addressing the problem of fraud, which is that an AI system is able to handle the process more accurately, faster and effective than a human. In the Dutch case, the absence of a clear conceptual and organizational innovation is striking, even though the AI innovation had implications. There was no formal change in discretionary space, organizational structure, and control and management to accompany the innovation. In that sense it was seen as an incremental innovation in one of the organization’s processes. Yet below radar, the system represented a substantial innovation with big effects on the organizational outcomes. Even though the innovation was confined by policy and management, the effects on the organization were not articulated. The fraud department did not have a role in service provision but provided an AI-supported identification of potential fraud. Sometimes they acted on that, but oftentimes another department did, operating on that signal and oblivious to how that came about, what value it had, and how it affected their very core task. Instead of investigating admissibility of benefits claims for the cases identified by the self-learning model, they assumed fraud (incentivized by the context) and entered a different process.

The cases further demonstrate a lack of larger government realignment around public service delivery supported by AI, even though in both societal and academic debate, AI is seen to be fundamentally reshaping governance. In the practice of the cases, little is left of this fundamental perspective; the introduction of AI remains task-focused and driven by considerations of efficiency or effectiveness. As a result, the implementing organizations neglect the important bigger questions that were on the table already, notably Mergel, Rethemeyer, and Isett (2016) point on how the work shifts towards technocratic operations, requiring an explicitly discussed rebalancing rather than organic drift. Yet it is this organic drift that we witness in these cases. The cases demonstrate what Pang, Lee, and Delone (2014) call a process view: AI facilitates the organizations to develop capabilities to perform. Yet they also identify a complementary perspective, which concerns how ordinary organizational processes also need to co-develop with the innovation, for it to be able to lead to performance gains by an organization that developed practices and structures in conjunction with IT developments (Pang, Lee, and Delone 2014). This perspective partially explains what the organizations in our cases missed.

This is problematic as, in line with e.g. the work of Bovens and Zouridis (2002), we find systems replacing certain tasks. Yet, the shift in the role of bureaucrats resulting in the introduction of artificial intelligence to complement or support human intelligence, appears to go almost unnoticed, and the organizational context did not change with it. This negatively impacts the informal discretionary space, as AI-supported information is shared among professionals without being accompanied with their uncertainties (originating from, e.g. the model, data quality, interpretation). This does not only obscure discretionary space but also provides a barrier to organizational learning (Jorna and Wagenaar 2007). That leaves us with, on the one hand, a theoretical narrative about the fundamental change AI brings, warning about how it is harmful to define the value of AI only in functional terms (Redden 2018), and clarity on the fact that how algorithms are used - and how this affects professional usage – is essential (Matheus and Janssen 2020; Sun and Medaglia 2019). On the other hand, the cases demonstrate that this sometimes happens implicitly, that AI is embedded in regular processes, without professionals fully grasping the consequence of what their decisions mean in a process where advanced, hard to understand, algorithms are used (Van der Voort et al. 2019). The experts within the organization that normally should be able to identify the more profound effects of implementing an AI system feel restricted by the goals set by managers, the organization and policy makers. This is clear in especially the Dutch case, where professionals did not experience discretionary freedom and experts did not find effective mechanisms for internal feedback on the system’s effects. The control function of management to experts that Liff and Andersson (2021) describe, seems to apply here. Furthermore, institutional pressures may further complicate how innovation is perceived, or not in the organizational context (De Coninck et al. 2021). Organizations seem to be still in that reactive stage, where they try − and fail − to grapple with the shifts in daily operational processes brought about by the digital systems. In that context, a deep-rooted systems-wide change is beyond the capacity of the organizations at that point.

Taken together, these findings are limited to the point when AI system are integrated into a public organization with a specific focus on changes in bureaucratic role definition. In essence, the interaction between the social and the technical system. This implies that we do not address the characteristics of the AI system in a more detailed manner and how potentially design features play a role in the dynamics laid out.

Concluding remarks

In this paper, we explore the research question of how the implementation of AI-driven technologies in public decision-making in different organizational contexts impacts innovation in the role definition of public bureaucrats. We offer a first exploration of comparative AI system implementation with a focus on public sector innovation. More specifically, we look at the role definition of bureaucrats to highlight an understudied dimension of this line of research. Both cases show that a new digital system meets a complex organizational set-up with agency and individual-level dynamics that are overlooked as automated decisions cut through some of the horizontal and vertical relationships of bureaucrats. These preliminary insights point towards variations of innovation − some more obvious than others. In the Dutch case, innovative changes could be witnessed within bureaucratic tasks, where in the US case more wide-sweeping transformations around bureaucratic work were made. The context of consensus versus majoritarian systems may play a role here as the former is associated more with bottom-up innovation where AI might impact primary processes more fundamentally than how its introduction is intended or perceived at the top. However, more research is needed in the context of both systems to untangle the factors involved and how they influence each other. We also find that the ambitions linked to AI system implementation play a role for the definition of bureaucratic roles, as the US case highlights a framing of AI outperforming bureaucrats on fraud cases and thus a replacement of public officials with automated decision-making. In the Dutch case, the tasks and ambitions were pushed by policy and politics, with the AI technology remaining in a supportive role. Changes were thus focused on bureaucratic tasks and the higher management were all but oblivious and ignorant to its substantial character, which was therefore not accommodated in organization, discretionary freedoms, interactions between public sector professionals, or policy. The study further raises questions around the potential impact of design features and processes of the AI system that were not covered but might play a role in system adoption and integration.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Correction Statement

This article was originally published with errors, which have now been corrected in the online version. Please see Correction http://dx.doi.org/10.1080/14719037.2024.2356444

Additional information

Notes on contributors

Sarah N. Giest

Sarah N. Giest is an Associate Professor at the Institute of Public Administration (Leiden University). Her research focuses on public digitization processes and data use within government. In this line of research, she looks at the role of data and the decisions of individuals handling this data at different points in the public decision-making process. Her recent work revolves around utilizing data-driven technologies for sustainability and welfare policies.

Bram Klievink

Bram Klievink is a Professor of Digitalization and Public Policy at the Institute of Public Administration (Leiden University). He studies the interplay between digitalisation and government. How digital innovations challenge established practices and institutions of public governance and how these innovations can be used to support good and effective governance, is the core of his research.