Algorithmic decision-making and system destructiveness: A case of automatic debt recovery

ABSTRACT

Governments are increasingly relying on algorithmic decision-making (ADM) to deliver public services. Recent information systems literature has raised concerns regarding ADM’s negative unintended consequences, such as widespread discrimination, which in extreme cases can be destructive to society. The extant empirical literature, however, has not sufficiently examined the destructive effects of governmental ADM. In this paper, we report on a case study of the Australian government’s “Robodebt” programme that was designed to automatically calculate and collect welfare overpayment debts from citizens but ended up causing severe distress to citizens and welfare agency staff. Employing perspectives from systems thinking and organisational limits, we develop a research model that explains how a socially destructive government ADM programme was initiated, sustained, and delegitimized. The model offers a set of generalisable mechanisms that can benefit investigations of ADM’s consequences. Our findings contribute to the literature of unintended consequences of ADM and demonstrate to practitioners the importance of setting up robust governance infrastructures for ADM programmes.

SPECIAL ISSUE EDITORS:

• Patrick Mikalef, Aleš Popovic, Jenny Eriksson Lundström and Kieran Conboy

KEYWORDS:

• Algorithmic decision-making
• Destructive systems
• Organisational limits
• Systems thinking
• Case study

1. Introduction

“ … our capacity as humans to create highly complex systems is not always matched by our ability to organize and control them in the face of most conceivable conditions, let alone inconceivable ones.” (Oliver et al., 2017)

The widespread availability of digital technologies in society has generated an unprecedented amount of data ‒ enabling organisations to build powerful algorithms to act autonomously on behalf of humans and make decisions based on the body of data (Newell & Marabelli, 2015). Algorithmic decision-making (ADM) is producing notable efficiency benefits in various fields, including healthcare, energy, and security, just to name a few application areas. Governments have been quick to set up ADM programmes¹ to reap efficiency gains and promote transparency in their internal operations and in serving citizens.

Yet, implementing ADM programmes successfully comes with unique challenges. Transferring decision-making agency from humans to algorithms may result in loss of critical thinking and domain expertise within organisation (Jussupow et al., 2021; Mayer et al., 2020; Strich et al., 2021). It is argued that increasing algorithmic agency (Baird & Maruping, 2021) requires that organisations must develop human-machine configurations that ensure that humans remain in the decision-making loop (Asatiani et al., 2019; Grønsund & Aanestad, 2020). Further, many complex algorithmic models lack transparency, which makes their operating logic hard to understand (Faraj et al., 2018). Such opacity requires organisations to anticipate potential unintended effects and put various safety measures in place to prevent them (Asatiani et al., 2020, 2021).

If not properly designed and implemented, ADM programmes can generate both organisational and societal consequences that are undesirable and a type of “disbenefit” (see Zwikael & Smyrk, 2015). Discrimination resulting from reliance on personal data digitalisation and algorithms (Favaretto et al., 2019) represents a particularly worrying disbenefit that threatens human dignity and autonomy (Leidner & Tona, 2021). For instance, the use of ADM in immigration enforcement (Mcdonald, 2019) and university entrance exams (Hao, 2020) for applicant screening, and in justice systems for recidivism prediction (Buranyi, 2017), has promoted socio-economical and racial discrimination, causing controversy and discreditation of the ADM programmes. Such consequences have the potential to aggravate existing social issues through promoting inequality and discrimination and call into question a government’s ability to protect and serve its citizens (Ananny, 2016; Boyd & Crawford, 2012). The resulting negative disruptions may be significant enough to outweigh the benefits of ADM.

Systems that negatively disrupt the society around them and erode the integrity of the implementing organisation have been said to be socially destructive (Baskerville & Land, 2004; Drummond, 2008). Such systems may be technically successful and provide short-term benefits. However, by realising negative outcomes for the organisation’s stakeholders, socially destructive systems may ultimately jeopardise the very existence of their host organisation.

Due to the wide range of possible application areas of ADM and the rapid advancement of the technology, ADM programmes possess significant potential for social destruction (Canhoto & Clear, 2020; Newell & Marabelli, 2015; O’Neil, 2016). This is especially true in the government context because affected stakeholders often include large numbers of citizens who cannot opt-out from the programmes. These potential problems prompted the Dutch government to stop using algorithms in fraud detection (Henley & Booth, 2020). Yet, many other government organisations are pursuing opportunities to utilise ADM without a clear understanding of its costs, benefits, and risks for stakeholders. Such problematic implementations have emerged in social services in the United States, the United Kingdom, Australia, and India (Pilkington, 2019). These countries have used ADM to automatically detect and collect presumed overpayments from welfare recipients, a practice that has been criticised for using debatable and spurious data sources and calculation methods in a way that hurts society’s most vulnerable members (Pilkington, 2019).

This situation calls for a better understanding of how such systems lead to socially destructive outcomes. Despite multiple real-life cases of destructive ADM systems, the literature on such effects is largely conceptual. In general, empirical accounts of systems that initiate a socially destructive cycle, harming the organisation’s stakeholders and eroding the organisation from within, are rare. The few examples that exist (Baskerville & Frank, 2004; Drummond, 2008) discuss the Information Systems (IS) characteristics that give rise to the destructive effects but give little attention to affected stakeholders, the processes by which the effects take place, or the dynamics that sustain or cease them. Against this background, we ask: How do socially destructive government ADM systems arise and what mechanisms sustain or constrain them?

We address this research question through a rich case study analysis of the Australian government’s debt recovery programme, dubbed “Robodebt” in the popular media. Robodebt was expected to follow a straightforward process for debt collection – an algorithm would compare welfare recipients’ records with income data from the tax office, work out the amount of overpayment, and pursue refunds where relevant. However, the system received mounting criticism for producing invalid debt estimates and causing unfair treatment of thousands of people, which resulted in citizen distress (possibly even suicides) (Medhora, 2019b). Its “debt-averaging method” was declared illegal by the courts, and consequently the government was forced to retire it and return the unlawfully collected money to citizens (Hayne & Doran, 2020). The case has already attracted academic interest regarding the scheme’s legal basis (Carney, 2018, 2019a) and societal implications (Park & Humphry, 2019; Whelan, 2020). It has not, however, been examined from an IS perspective.

Studies on socially destructive systems point to the importance of probing for limiting factors in the technology and its organisational setting (Drummond, 2008). Thus, our inquiry is informed by the concept of organisational limits, a perspective that emerged as a suitable sensitising device during the research process. The organisational limits framework recognises that organisations are limited by their members’ cognition, managerial policies, technological capacity, and environmental conditions (Farjoun & Starbuck, 2007; Oliver et al., 2017). It suggests that the interplay among these different limits helps to explain why organisational activities may trigger unintended consequences. A further perspective that informs our investigations is that of systems thinking (Burton-Jones et al., 2015; Checkland, 1999a), as it sensitises one to consider various positive and negative feedback loops amongst the component parts of a socio-technical system resulting in unintended consequences as an emergent property of the system.

Our study responds to calls for research on the negative societal impacts of ADM (Rossi 2021; Markus, 2015, 2017; Newell & Marabelli, 2015) by investigating an empirical case of a socially destructive ADM system. We develop a research model with a set of four generalisable mechanisms for explaining how an ADM programme’s negative effects are initiated, sustained, and constrained. We call these mechanisms “directing change with limited vision”, “limiting sociotechnical agency”, “dismissing destructive consequences”, and “generating a societal response”. Second, we demonstrate how the combination of a systems approach with the perspective of organisational limits helps to generate explanations for the unintended negative consequences of IS in general.

2. Theoretical background

In this section, we first explore the nature of ADM and its unintended consequences within the context of the broader IS literature. We then discuss unintended negative consequences and destructive systems and move on to explain the chief features of the organisational limits perspective, informed by a systems thinking approach. The two perspectives utilised together guide our theory development.

2.1. Algorithmic decision-making in organisations

Despite the various benefits of ADM, emerging information systems and management literature has discussed the potential negative effects of leveraging the technology, particularly for individuals and society (Boyd & Crawford, 2012; Markus, 2017). For one, ADM works by quantifying a human’s life to generate data (Constantiou et al., 2015; Galliers et al., 2017; Marjanovic & Cecez-Kecmanovic, 2017), which, on the one hand, offers personalised services to individuals but, on the other, restricts their options and choices. This trade-off raises questions regarding individuals’ freedom of choice (Zuboff, 2015, 2019). Second, some advanced algorithms operate as a black box, hiding their inner workings and decision-making processes from human decision-makers (Asatiani et al., 2020, 2021; Faraj et al., 2018). This opacity leads to a difficulty in establishing accountability, assessing the accuracy and robustness of output generated, and a lack of trust in such technologies (Goldenfein, 2019; de Laat, 2018). Third, the use of algorithms to profile individuals – sometimes inadvertently based on their race, ethnic group, gender, and socio-economic status – can lead to biases and discrimination (Chouldechova, 2017), which raises wider ethical concerns (Boyd & Crawford, 2012). Favaretto et al.’s (2018) systematic review on such discrimination reveals that most studies on the topic are conceptual and discuss the potential risk of discrimination with ADM systems. As such, empirical accounts of actual bias and discrimination caused by ADM systems are still relatively few, limiting understanding of the processes and mechanisms that introduce bias into the systems as well as those that drive the implementation of discriminative systems.

The practice of ADM is new for many government organisations. Many organisations still lack good IT practices in developing, testing, and maintaining such algorithms in a way that is beneficial for a broad range of stakeholders. In this paper, we focus on the mechanisms that lead to the creation of ADM systems that cause negative unintended consequences to a degree that can be described as destructive. By doing so, we respond to the recent calls (Rossi, 2021; Markus, 2015; Newell & Marabelli, 2015) for IS researchers to explore the potentially negative consequences of using data and algorithms on individuals, organisations, and society, particularly in cases of “non-responsible” use (Newell & Marabelli, 2015, p. 9).

2.2. Negative unintended consequences and destructive systems

Several studies of IT’s unintended negative consequences have explored the undesired side-effects of otherwise beneficial IT systems. The national school test results sharing portal called My School in Marjanovic and Cecez-Kecmanovic’s (2017) study contributed to transparency but was criticised for causing various unintended effects such as student discrimination and shifting teachers’ priorities in the wrong direction. Moreover, Tim et al. (2018) show how a social media platform that helped to build a sustainability movement also caused the spread of misinformation in a community. Davis and Hufnagel’s (2007) study on automating the work of fingerprint technicians shows that efficiency benefits brought about by the new system also caused cognitive dissonance among the technicians by obscuring the work process. Similarly, implementing artificial intelligence to substitute human loan consultants’ tasks and responsibilities in a bank alleviated the consultants’ work pressures but at the same time the resultant loss of autonomy negatively disrupted their role identities (Mayer et al., 2020; Strich et al., 2021). The employees’ fears of losing their professional statuses were warranted – systems that improve processes by automating analysis and decision-making have been found to gradually erode their users’ expertise (Dowling et al., 2008; Rinta-Kahila et al., 2018; Sutton et al., 2018).

Further, many organisational systems that improve knowledge sharing and performance have been found to foster insidious errors (Ash et al., 2004; Jussupow et al., 2021; Harrison et al., 2007). In other cases, undesired effects stem from unethical IT use (Charki et al., 2017). Finally, literature on technostress has examined the stress-inducing qualities of IT (Ayyagari et al., 2011; Maier et al., 2015). Overall, the above studies discuss systems that are not strictly speaking harmful but that present trade-offs that need to be considered via a negotiation of values and priorities.

On the other hand, systems that result in few benefits, but over time cause consequences that are overwhelmingly negative, are described as “socially destructive systems” by Baskerville and Land (2004) who show how such systems can harm, and even destroy an organisation. These authors provide a case study of an electronic briefing system implemented in a naval command headquarters that did not accurately capture deputies’ intricate social communication activities, and “elements of the organizational structure gradually and unintentionally began to come apart. Had the system survived, it might have destroyed the entire organizational structure” (p. 281). Baskerville and Land see destructive processes as enabled by the different IS characteristics that facilitate destructive effects and inhibit constructive ones. Drummond (2008) gives a further example of a destructive system, with a case study of a company that automated its insurance claims processing. The system yielded short-term benefits but ended up being destructive in the long run. The system in question simplified clerks’ work tasks and increased efficiency in simple claim cases but it was unable to handle complex ones, which ultimately led to plummeting customer satisfaction and decreasing profitability.

Destructive effects in these cases appear to be rooted in the principle of requisite variety (Ashby, 1958), i.e., a system’s ability to meet the number of environmental states in which it operates. For instance, in Drummond’s (2008) case an automated system that was supposed to “cut processing times and improve accuracy” turned out to be “outwardly destructive” (p. 181) because of its inability to handle the complexity of various customer cases, and the managerial decision to minimise human oversight that made it “difficult for staff to check their work” (p. 179). Similarly, the IT in Baskerville and Land’s (2004) study was unable to capture and mediate the complexity of the social system in which it was implemented, highlighting the challenges with sociotechnical change when implementing new IT. While these cases have focused on the destructive characteristics of the implemented IT and its effects on the host organisation, governmental ADM programmes tend to have more far-reaching implications as they concern a wide range of stakeholders (Markus, 2017; Someh et al., 2019), including citizens who may not be able to opt-out of the programmes. This invites us to consider the dynamics of the broader “systems of systems” (Markus, 2017, p. 239) in which such destructive ADM programmes are deployed and sustained.

In the next section, we turn to literature on systems thinking and organisational limits that offers a framework for explaining such dynamics.

2.3. Organisational limits framework for unintended consequences and destructive systems

Baskerville and Land’s (2004) work on destructive systems was informed by general systems theory (Boulding, 1956) and systems thinking (Checkland, 1999a), which have been influential in studying and developing information systems over a long period (Burton-Jones et al., 2015). Ideas central to systems thinking are: (i) the notion of emergent properties for the system as a complex entity within its environment (e.g., unanticipated consequences); (ii) that sub-systems will be nested within encompassing systems in a layered structure (e.g., a software program within a larger sociotechnical system); and (iii) processes of communication (e.g., data input and reports output) and control (e.g., algorithmic decision-making that takes into account signals from feedback loops) (Checkland, 1999b). Central to the systems thinking approach is the idea of purposeful action, as it recognises that people (e.g., system owners and managers) aim to achieve goals with IS congruent with their own worldview (e.g., an economic imperative, Zuboff, 2019). Due to its high relevance to the field and its ability to address interactions of stakeholders across various analytical levels, the systems approach continues to be encouraged by IS scholars (Burton-Jones et al., 2015; Marjanovic & Cecez-Kecmanovic, 2017; Markus, 2017).

Our early analysis of the Robodebt case led us to a systems thinking approach, especially given the salience of the negative feedback loops that were observed amongst component parts of the system. Understanding the issues within these component parts pointed us to consider different types of organisational limits (Farjoun & Starbuck, 2007; Oliver et al., 2017), a perspective which has proven useful in explaining failures and is congruent with a systems thinking approach. Thus, our framework for analysis combines the systems thinking approach with the organisational limits perspective.

The systems thinking approach has several ideas that overlap with the organisational limits perspective. Organisational limits are recognised for the organisation and its environment and for component sub-systems within the organisation. Further, the systems and sub-systems are established for purposeful activity and are expected to communicate with each other and respond to signals from other parts of the system. The organisational limits perspective shows that organisations’ operational capabilities and capacities are constrained by limits that exist both within (endogenous) and outside (exogenous) of the organisation (Farjoun & Starbuck, 2007; Oliver et al., 2017).

Organisational members, such as strategic decision-makers, are cognitively limited in their imagination, perception, memory and foresight, which constrains the members’ ability to recognise, react, interpret, and respond to events or behaviours (Farjoun & Starbuck, 2007). These limits give rise to cognitive biases that may cause decision-makers to act irrationally; for instance, they may avoid risks in choices that involve sure gains and seek risks in choices that involve sure losses (Kahneman & Tversky, 1979). Managerial limits are products of managerial decisions, actions, and policies, manifested in organisational structures, hierarchies, budgets, and internal codes of conduct (Farjoun & Starbuck, 2007). They reflect formal organisational arrangements and coordination mechanisms that are aimed at providing predictability in organisational outcomes (Oliver et al., 2017), for instance, by defining work roles and areas of responsibility.

Technological limits are of special interest to IS researchers. Technology contributes to an organisation’s capacity to operate, representing an endogenous limit to the organisation, beyond which it is possible to expand through further investments in technology development or acquisition. However, technology is limited by its design and by the capacity of its software and hardware, constraining its user either intentionally or unintentionally (Oliver et al., 2017). Technological limits can be assessed by specifying and examining the different components of an IT artefact (Moeini et al., 2020), which in the case of ADM artefacts typically include a decision-making algorithm, data, and an interface that represents the decision-making outcomes or processes. For instance, decision-making algorithms are inherently limited in that they lack the mindful, context-sensitive processing capabilities that humans possess (Salovaara et al., 2019). Limits in the environment represent exogenous constraints that come from outside the organisation, such as limits set by available technologies, current legislation, and prevailing societal norms. In Europe, the General Data Protection Regulation (GDPR) is an example of a legislative limit that sets bounds on organisations’ use of ADM in their operations, including rules relating to “a right for explanation” and “a right to be forgotten” (Goodman & Flaxman, 2017).

Organisational failures with destructive effects may eventuate via a cascade effect where exceeding one type of limit may lead to exceeding other types (Oliver et al., 2017). For instance, an organisation may breach its own policies (whether unintentionally or deliberately), or those of a labour union, by having its employees work prohibitive overtime, which breaches managerial (internal policy) and environmental (labour union rules) limits. The overtime may cause employees to exceed their cognitive limits as they continue to work with no adequate rest and recovery, which may result in them making errors that have negative consequences on stakeholders. The interacting nature of limits is in line with the emergent view of systems thinking, further suggesting that the two perspectives can be employed in a complementary manner.

Overall, little is known about how limits materialise and are exceeded beyond safety-critical environments (e.g., aviation, Oliver et al., 2017) and how they manifest in socially destructive systems. Understanding this issue is important in the digital age as organisations are encouraged to promote a culture of experimentation and to fail and learn fast. As such, experimentation opens new opportunities, but also puts organisations at risk of applying digital technologies when they lack adequate experience and expertise to properly design, implement and use the technology. Indeed, the understanding of how decision-making tasks can be delegated from humans to algorithms responsibly is only starting to emerge (Baird & Maruping, 2021; Grønsund & Aanestad, 2020). Against this background, we take a systems perspective on organisational limits as a sensitising device to examine how socially destructive government ADM systems arise and the mechanisms that sustain and cease them.

3. Research method

3.1. Case study setting: Centrelink’s Robodebt

To shed light on our research question, we examine the case of a controversial, ADM-driven, welfare-overpayment detection and collection scheme introduced by Centrelink in Australia. Centrelink is the Australian Government’s master programme that distributes social security payments to citizens. It provides a range of government payments and social support services for various citizen groups, including retirees, the unemployed, families, carers, parents, people with disabilities, Indigenous Australians, students, apprentices, and people from diverse cultural and linguistic backgrounds.² Centrelink operates within the Department of Human Services (DHS),³ which is responsible for the service delivery of social policies developed and implemented by the Department of Social Services (DSS). In addition to Centrelink, DHS governs the Australian healthcare programme Medicare, and together with the Digital Transformation Agency (DTA), co-manages the myGov internet portal through which citizens can access government services online. The Australian Tax Office (ATO) is responsible for tax collection and provides Centrelink data about citizens’ income. Figure 1 illustrates the organisational locations of the agencies central to our study.⁴

Figure 1. Organisational structure of the Australian welfare system.

The payments central to our case are welfare support payments intended to help citizens get by during a period of unemployment. Citizens’ eligibility for welfare support is defined by their income, and, thus, if their employment situation improves, increased income may reduce or remove their eligibility for support payments. Information regarding one’s income, however, does not flow in real-time to Centrelink, and it remains citizens’ responsibility to report any increases in earnings. If they fail to do so (for whatever reason), they may receive more income support payments than they are eligible for and become debtors, thus the term “welfare overpayments”.

Reducing the national debt has been an objective of Australian political parties for years. One debt-reduction strategy has been to reduce and recoup welfare support overpayments. Australia has a controversial history of “demonising” the long-term unemployed (Hutchens, 2021), with political parties voicing claims that some citizens intentionally “defraud” the welfare system by collecting more payments than they are eligible for. This issue has been framed as a threat to the system’s long-term sustainability, and it has resulted in pursuit of alleged welfare fraudsters by means of prosecution (Wilcock, 2014, 2019). Over the years, measures to curb non-compliance have increasingly utilised ADM technology. Until 2016, the use of ADM had been limited and involved human oversight (Centrelink and the Data-Matching Agency, 2011).

Centrelink’s cost-benefit analysis of employment and welfare benefit data found that over 860,000 recipients of government benefits had discrepancies in their accounts between 2010 and 2013 (Belot, 2017; Karp, 2019). According to these calculations, in most cases the discrepancy would have resulted in welfare recipients owing an average of $1400 (AUD) to the government (Belot, 2017). Thus, in July 2016 Centrelink implemented the Online Compliance Intervention (OCI)⁵ programme, an automated debt-calculation and -collection scheme, which is referred to by the popular media as “Robodebt”. Between July 2016 and October 2019 the government issued a total of $2 billion (AUD) in debt notices to 700,000 current or former welfare recipients (hereafter, citizens), out of which $640 million was successfully recouped (Medhora, 2019a).

The OCI system drew data from two different government systems, one belonging to Centrelink and another one to ATO. These two systems recorded citizens’ income data in different formats – while Centrelink’s system applied fortnightly figures, ATO’s stored annual income data. OCI averaged a citizen’s earnings reported to ATO over a series of fortnights, matched them with received welfare benefits, and based on the matching, calculated potential overpayments. Because the formula used fortnightly averages, instead of actual earnings in the fortnight in question, it has been criticised for inaccuracy that leads to exaggerated or even false inflation of debt. Due to inaccurate and spurious debt-identification methods, even government employees have not always been able to explain how a given debt has been calculated and whether it reflects reality (Skinner, 2019). As a result, a significant number of citizens received a “Robodebt notice” that did not reflect what they actually owed to the government (whether they owed anything at all or not).

In 2017, OCI came under scrutiny by the Commonwealth Ombudsman and the Senate. These inquiries recommended significant revisions be made to the programme and the Senate inquiry called for OCI’s immediate suspension until its fundamental issues were resolved. While some changes were made, public calls for terminating the scheme persisted. On the 17th of November 2019, the government announced the suspension of OCI’s key functions, and a review of all debts raised using the programme. A week later, OCI was deemed unlawful by a state court (Carney, 2019b), and in 2020 the government faced a class action raised by over 10,000 affected citizens (Henriques-Gomes, 2020a). The class action was settled on the day of trial with the government agreeing to financial reparations. The total value of the settlement was $1.2 billion, including repayments of illegally raised debts, interest on money paid by victims, and legal fees (Snape, 2020).

It should be noted that OCI was just one part of a very large information systems architecture run by Centrelink. Our chosen case of Robodebt occurred in an organisation that has considerable depth and length of experience in managing large and complex information systems (see Phillipson, 2017). The reports of Robodebt’s illegality and harrowing consequences suggest that the Australian government’s ADM programme not only failed to deliver benefits, but also resulted in destructive effects for different stakeholders. The government suffered financial losses and serious reputational damage.

3.2. Data collection

Publicly available qualitative data represents a rich, yet under-utilised source of knowledge for IS research (Ghazawneh & Henfridsson, 2013). The Robodebt case has been the subject of wide and comprehensive public discussion and scrutiny, with much material available on the internet. The Ombudsman’s and Senate’s inquiries, as well as various submissions to such inquiries, have been published online. The case has received extensive online news media coverage, which prompted DHS to publish responses on the department’s online Media Hub. A collective action emerged on Twitter (#NotMyDebt), that turned into a website (https://www.notmydebt.com.au/) and a public discourse on how citizens were affected by Robodebt. Table 1 provides an overview of the data collected for analysis. In total, 114 documents were analysed, representing 684 pages.

Table 1. Summary of data sources

Type of data	Description	Documents	Pages
Investigation reports	The Acting Commonwealth Ombudsman: “Centrelink’s automated debt raising and recovery system – A Report About the Department of Human Services’ Online Compliance Intervention System for Debt Raising and Recovery” June 2017	1	113
	The Acting Commonwealth Ombudsman: “Centrelink’s Automated Debt Raising and Recovery System – Implementation Report” April 2019	1	45
	The Senate Community Affairs References Committee: “Design, scope, cost-benefit analysis, contracts awarded and implementation associated with the Better Management of the Social Welfare System initiative” June 2017	1	155
Submissions to inquiry	Submissions to the 2017 Senate Inquiry by: The Australian Council of Social Services (ACOSS) The Community and Public Sector Union (CPSU) The Department of Human Services (DHS) #NotMyDebt Victoria Legal Aid	5	129
Class action online information	Gordon Legal: “Robodebt Class Action” Outline and FAQ sheet of the class action against the Commonwealth Government, retrieved from Gordon Legal’s website in April 2020 (https://gordonlegal.com.au/robodebt-class-action/)	2	11
News articles	Articles in ABC News	56	121
	Articles in The Australian	17	24
	Articles in The Conversation	13	49
Media releases	Media releases by DHS, retrieved from the DHS Media Hub Archives	17	27
Total		114	684

We drew on these rich data sources to study how the destructive consequences of the scheme unfolded. Our data sources collectively provide a strong basis for theorisation. First, the inquiry reports are based on comprehensive investigations of the department’s practices, the implementation process, how the debt collection process changed as a result of ADM implementation, and how citizens, employees and other stakeholders were affected by the programme. Second, our data provides a multi-stakeholder perspective, representing the views of the government, DHS, Centrelink employees, citizens, independent journalists, legal experts, and academics. Appendix A provides a detailed description of the data sources.

3.3. Data analysis

We followed the procedure outlined by Gioia et al. (2012) where a researcher starts the inquiry in an inductive manner but transitions into a more abductive mode in which “data and existing theory are now considered in tandem” (p. 21). The analysis was carried out through a four-stage process.

First, we inductively read the collected materials to identify key events in the case and to construct a timeline of relevant events and a case narrative (see Appendix B). At this stage, no themes or subthemes were anticipated a priori as we let the data do the talking.

Second, we conducted open coding (Strauss & Corbin, 1998) on the material based on the key themes and foci that emerged from the first reading. The initial round of open coding was conducted by two research assistants under the authors’ supervision. One of the authors simultaneously inductively coded a portion of the data so that the initial coding outcomes from different coders could be compared. The coding was reviewed by two of the authors who discussed the codes among themselves and with the initial coders iteratively until an agreement on the first-order coding was reached. This stage resulted in 129 codes. The observation that OCI was kept operational over years, despite the numerous negative feedback signals, pointed us to consider a systems thinking approach.

In the third stage, we asked “whether the emerging themes suggest concepts that might help us describe and explain the phenomena we are observing” (Gioia et al., 2012, p. 20). We thus relied on our knowledge and existing literature to analyse and develop concepts that explained the data. Based on this we developed theoretically informed second-order concepts, some of them similar to those discussed in previous literature (e.g., bias; myopia; economic imperative). This step is where the organisational limits (Farjoun & Starbuck, 2007; Oliver et al., 2017) perspective emerged as a key organising framework. As we travelled between data and literature, we became aware that many of the key problems in the OCI implementation reflected limits, whether technological, organisational, cognitive, or environmental. Using systems thinking and organisational limits combined as a theoretical frame of reference and sensitising device, we reflected on the way in which the second-order concepts represented or related to limit types, violations of limits, or consequences of limits. While we did not allow prior theoretical concepts and assumptions to restrict our interpretations, they helped us to structure and make sense of the data. Focussing on the deep structure underlying the first-order codes and the similarities and differences between them, we reduced the first-order codes to 12 abstract second-order themes informed by the lens of organisational limits.

Finally, we distilled the second-order concepts to a set of six aggregate dimensions. This allowed us to create a data structure (see Figure 2) that connects the first-order codes to the second-order concepts and to their aggregate dimensions. Using this data structure, we revalidated the final concepts against the underlying data and established a clear chain of evidence between raw data, the emerging concepts, and the aggregate dimensions (Gioia et al., 2012). This step involved a sensemaking process in which we strived to understand how the different limits, their components and their interactions resulted in various consequences. Through multiple iterations, we went back and forth between the data and the literature, having conversations among the authors and revising the data structure, until a shared understanding was reached.

Figure 2. Data structure.

The logic of temporal bracketing (Langley, 1999) helped us to organise the events of interest into a process with four distinct phases (see Figure 3 below).

Figure 3. Phases of a destructive system.

We compiled the relationships amongst the different concepts in each phase into a table with evidence from the data (Appendix C). This understanding culminated in a conceptual model (Figure 4) that represents the mechanisms of a public sector ADM implementation that gave rise to significant undesired consequences.

Figure 4. The emergence, maintenance and delegitimization of a socially destructive system (the grey-shaded area demarcates a phase that comes notably later than others).

4. Findings

Our analysis shows how a set of interconnected organisational limits resulted in an ADM programme with destructive effects, how the programme was nevertheless sustained, and how it ultimately was delegitimized. These mechanisms are visualised in a dynamic model (Figure 4). Next, we elaborate on the model’s key concepts and their relationships via a narrative arranged according to the four phases (Figure 3).

4.1. Phase I: ADM programme initiated with limited managerial vision

DHS’ ADM programme was driven by a strategy of the department’s top management.⁶ As part of the government’s 2016 campaign promise, they pledged to crack down on “welfare fraud” by ramping up the use of ADM to fully automate the detection and collection of welfare overpayments. ADM was seen as a “more bespoke way of dealing with people’s arrangements”, that would ensure “that mistakes are minimised” (The Conversation, 16/5/2019). Escalating automation was expected to balance the budget by clawing back $2.3 billion in welfare overpayments and provide evidence of the government’s claim of being responsible fiscal managers.

4.1.1. Top management’s limited vision

The first instance of organisational limits that we observed in the timeline of our case study was top management’s limited vision that made them unable or unwilling to foresee and critically evaluate the ADM programme’s impact on providing social welfare services. This limited managerial vision was formed through interactions between managerial and cognitive limits at the political level of strategic decision-making – managerial limits as a political worldview – that were inscribed in the government’s campaign promises and formalised agendas and policies, which channelled down to individual decision-makers’ cognition, constraining their vision. Limited managerial vision was characterised by two aspects of the government’s approach: welfare-critical ideology and economic imperative (see the data structure shown in Figure 2).

First, the top management’s welfare-critical ideology exhibited an overall critical view towards welfare services. This attitude was evident in significant resource cuts that DHS, as the main welfare agency, experienced during the years preceding OCI: “Over the past five years, the professional and technical capacity of the department has been severely eroded. There has been a significant reduction of permanent staff … ” (CPSU, 2017, p. 14). The ideology viewed welfare non-compliance (framed by the government as “welfare fraud”) as a significant societal problem, Minister Alan Tudge stating: “we’ll find you, we’ll track you down and you will have to repay those debts and you may end up in prison” (ACOSS, 2017, p. 4).

Second, the ADM programme was driven by an economic imperative: an algorithm was implemented to automatically calculate debts and dispatch debt notices to citizens primarily “in a bid to save money” (ABC News, 5/4/2017). This imperative resulted in tunnel vision (i.e., “the tendency to focus exclusively on a single or limited goal or point of view”⁷), as meeting financial targets of cost savings and money recollection became DHS’ primary objective that trumped any other concerns. For instance, when a risk management plan was made for OCI, it recognised only the risk of insufficient resources, overlooking the scheme’s “potential impact on service delivery, customer experience or reputational damage” (Ombudsman, 2017, p. 24). The economic imperative entailed taking a narrow view of legal and ethical issues, while the Australian Council of Social Services (ACOSS) warned that the promised welfare crackdown “could lead to significant hardship for vulnerable people affected if it results in more automated or aggressive debt recovery approaches” (The Conversation, 28/6/2016).

4.1.2. ADM programme

Top management’s limited vision initiated an ADM programme that can be characterised as having limited human agency, a limited ADM solution, and lack of best practices.

First, the new ADM programme meant changed work limits for the agency’s staff (i.e., agency-level managerial limits), compared with previously defined routines and responsibilities in the debt-collection and customer-service processes (inscribed in the official pre-OCI process descriptions; see Centrelink and the Data-Matching Agency, 2011). The new, redesigned process shifted responsibilities for making decisions and performing work tasks from humans to an ADM artefact, resulting in a work system with notably limited human agency. The change was aligned with the tunnel-vision perspective described above: work was delegated to the cheapest labour as algorithms became responsible for calculating potential overpayments and citizens became responsible for validating the algorithm’s calculations. Human agency was limited in three ways: minimising human oversight, reversing the onus of proof, and requiring citizens to self-service.

Minimising human oversight entailed full automation of debt-collection processes and put the machine at the centre of the previously human-centred and largely manual process. Even before OCI, human case workers had leveraged a data-matching system to identify potential debtors,⁸ but they had also “manually checked [the data-matching system’s] information for accuracy and contacted the recipient and/or their employer to clarify the information” (Senate Committee, 2017, p 15). With OCI, humans were left out of the debt-identification and debt-collection loop. The automated system independently estimated welfare overpayments and sent debt notification letters to citizens without human scrutiny. There were no longer any checks of accuracy with the recipient or employer.

Reversing the onus of proof of debts handed the task of verifying “whether or not a purported debt exists” from Centrelink to citizens (Senate Committee, 2017, p. 19). The previously manual detailed case analysis was performed because Centrelink legally assumed the responsibility of establishing the existence of a debt before pursuing it with a citizen and provided support for citizens who called wanting to work out discrepancies. With the new work task configuration, OCI “effectively shifted complex fact finding and data entry functions from the department to the individual” (Ombudsman, 2017, p. 23), requiring citizens to acquire old bank statements or salary receipts from their previous employers if they did not agree with the debt claim.

Requiring citizens to self-service represented another means of pursuing cost savings, by limiting DHS’ employees’ interaction with citizens. Staff were instructed to redirect citizens to the online self-service portal in any debt-notification related matters even if they would have been able to help the citizen over the phone.

In sum, managerial limits in process design removed the mindful human involvement that had previously characterised the complex process of identifying and raising debts, and transformed the role of the ADM artefact from a humans’ decision-support tool to being the main decision-maker.

Second, the ADM solution suffered from technological limits that inhibited its effectiveness: it was a severely limited ADM solution that produced inaccurate and difficult-to-justify debt decisions. The technological limits concerned OCI’s inconsistent data, simplistic algorithm, and complex interface.

The data that was provided to OCI’s algorithm as a basis for its decision-making was inconsistent mainly for two reasons. First, the two data sources, Centrelink and ATO, recorded data in different and incompatible formats (fortnightly vs. yearly). Moreover, the name of the same employer was sometimes recorded in a different way between the two databases, e.g., if a customer had made a spelling mistake when declaring their income to either party. These inconsistencies critically limited the extent to which the raw data could serve as a basis for conclusions about a given customer’s welfare debt as “the information to enable an accurate debt assessment to be made” was insufficient (Senate Inquiry, 2017, p. 42).

The algorithm that estimated potential overpayments used simple averaging to match the two data sources and could not account for citizens’ unique work-history circumstances. Citizens on welfare support payments often have fluctuating work hours and salaries due to casualised work, which the algorithm was not able to consider and would thus return inaccurate estimates. As such, “the discrepancy letters issued and the subsequent debts raised [were] a form of speculation” (Victoria Legal Aid, 2017, p. 7). Such a simplistic logic limited the ADM artefact’s ability to represent the reality of ones’ debt situation.

Finally, OCI’s citizen interface, materialised physically in the debt letters and digitally in the myGov portal, was limited in its ability to explain the basis of the debt calculations and the overall debt-recollection process. The Ombudsman’s report (2017) stated, “[t]he letter did not include the 1800 telephone number for the compliance helpline. It did not explain that a person could ask for an extension of time or be assisted by a compliance officer if they had problems” (p. 9). Further, it noted that “the OCI system does not clearly state it uses the averaging method or explain this may be inaccurate in some cases” (p. 12). In addition, the online portal was criticised for being complex and hard to use, further “black-boxing” the workings of the system.

In sum, the ADM artefact was limited in that it did not ensure data consistency, did not take into account the complex reality of citizen’s circumstances, and could not explain adequately what it was doing.

Finally, top management’s limited vision resulted in the design of the ADM programme being plagued by a lack of best practices. Being predominantly informed by a welfare-critical ideology and an economic imperative, “the strategy was rolled out due to government pressure even when concerns with the process were being expressed” (CPSU, 2017, p. 4). Centrelink employees who raised red flags before OCI was implemented had their warnings dismissed by the department management: “Many members stated that concerns were raised during the design process but were simply ignored … ” (CPSU, 2017, p. 13).

Centrelink did not involve relevant stakeholders, such as DTA, ATO, legal experts, and domain specialists, in the development of OCI. DTA had been “locked out” from the process, although its involvement could have helped to prevent the problems that emerged later: “If you are doing silly things like trying to match mismatching data sets, then we [DTA] would call that out very early in the process … ” (ABC News, 3/3/2017).

As such, little testing or piloting was conducted when implementing the system: “We asked DHS whether it had done modelling on how many debts were likely to be over-calculated as opposed to undercalculated. DHS advised no such modelling was done”. (Ombudsman, 2017, p. 8).

4.2. Phase II: biased decisions at scale trigger a destructive cycle

When implemented, OCI generated both intended and unintended effects. With the OCI system operational, the scale of operations increased significantly. Yet, as the scale of operations went up, so did errors, especially among specific citizen groups.

4.2.1. Intended effects

OCI increased the scale of operations markedly: “DHS estimates it will undertake approximately 783,000 interventions in 2016–2017 compared to approximately 20,000 compliance interventions per year under the previous manual process” (Ombudsman, 2017, p. 5). As intended by top management, OCI generated revenue in the form of recollected “debts” and in doing so appeared to help them deliver on their promises. DHS reported that by the end of 2018, “approximately $865 million in savings had been raised for all income data matching measures announced in the 2015–16 Budget” (DHS Media Hub, 8/2/2019).

4.2.2. Unintended effects

The ADM programme produced decision outputs that were riddled with errors and had a biased effect on vulnerable cohorts. Limits imposed by incompatible data sources and the decision-making algorithm’s inability to reconcile them was especially harmful for populations with a volatile income and numerous previous employers. The ADM programme raised incorrect debts without considering vulnerable individuals’ sensitive situations. When these technological limits were coupled with reduced human agency in a way that ignored best practices, the recipe for a destructive system was set. Further, the debt notices requested a response from citizens in a relatively short timeframe (another managerial limit set by the programme), and if citizens did not respond in the time given, debts were automatically deducted from their welfare payments or referred to external collection agencies. This situation initiated a destructive cycle of increasing distress among affected citizens and growing work disruption among Centrelink employees.

Algorithmic bias embedded in the debt letters triggered widespread citizen distress that manifested both mentally and financially. Receiving debt notifications triggered mental reactions of “anxiety, stress and, for some customers with unidentified vulnerability, crisis” (Ombudsman, 2017, p. 24), even leading to “suicidal ideation” in some cases (#NotMyDebt, 2017, p. 12). Many citizens were in financially fragile situations and some were considered “vulnerable”, meaning that they had complex needs due to factors such as mental illness, substance abuse, or domestic violence. With the onus of proof reversed, citizens struggled to find evidence to waive the debt: “ … unlike Centrelink, individuals have no legislative power to demand payslips. It may not even be possible to obtain payslips in some circumstances … ” (ACOSS, 2017, p. 8). This problem resulted in financial struggles for citizens who were compelled to pay the purported debts despite their difficult financial situations. The distress was aggravated by the system’s poor interface, which made understanding and contesting debts especially hard for people who have lower access to technologies like online portals. The difficulty of challenging debts and the short time given to respond meant that many citizens “who did not believe they owed a debt … paid it because some found it too difficult or too stressful to challenge the purported debt … ” (Senate Committee, 2017, p. 4). Such “unintended” payments contributed to the intended effects of increased revenue.

Citizen distress triggered a response at scale from the affected citizens who contacted the department in droves. This outcome resulted in an interaction between managerial and cognitive limits at the department level, as its limited human resources had to cope with a surge of distressed citizens. The overflow of distressed citizens and the employees’ newly constrained role capacity (e.g., they were directed to push citizens to self-service), resulted in employee work disruption that manifested in work overload, inability to help citizens, and reduced morale and wellbeing.

Work overload was inevitable as the department was not ready for such a response: “ … DHS underestimated the difficulties people would have using the OCI system and the demand for its call centre and in-person services this would generate … there were insufficient resources directed to telephone services” (Ombudsman, 2017, p. 18). This caused an increased work burden on DHS staff: “ … the OCI program will continue to put pressure on workloads for an already stretched DHS workforce” (CPSU, 2017, p. 7).

Inability to help citizens made things worse. Some staff had received no information or training prior to OCI’s go-live: “One Customer Service Officer stated that ‘I had no idea about this initiative until I heard about it on TV and complaints started coming in from customers.” (CPSU, 2017, p. 15). Therefore, “[s]taff were often unable to resolve discrepancies because they were not allowed to rectify errors” (CPSU, 2017, p. 12) and “[c]ustomers had problems getting a clear explanation about the debt decision and the reasoning behind it” (Ombudsman, 2017, p. 2). The inability of Centrelink staff to help citizens resulted in a vicious cycle in which already distressed customers became more distressed when failing to receive help after contacting the department.

Employee morale deteriorated due to the increased work burden from the surge of citizen contacts, frustration from the inability to address citizens’ concerns, and a sense of insecurity due to citizen aggression. “Risk of increased customer aggression and stress that have affected the health and safety of DHS employees … many staff have ended up leaving as a result of not being able to handle the stress of OCI” (CPSU, 2017, p. 15).

4.3. Phase III: escalation of commitment sustains the destructive system

Despite the obvious negative effects, OCI was increasing revenue for the government, which fuelled an escalation of commitment that sustained the troubled programme in the face of mounting criticism. Escalation of commitment limited top management’s cognition and manifested as deflecting critique and as managerial myopia, as decision-makers defended the programme and opted for short-term measures that did not address the programme’s core problems.

Deflecting critique occurred via top management’s initial neglect, and later denial, of OCI’s flaws: “It was not until the problems with the OCI became public that they were even acknowledged” (CPSU, 2017, p. 12). In response to the initial controversy, Human Services Minister Alan Tudge stated: “The system is working and we will continue with that system” (ABC News, 11/1/2017). In response to accusations of reversing the onus of proof to citizens’ disadvantage, DHS top management insisted that the automatically sent OCI letters were not debt letters but calls for citizens to work with DHS to explore whether a debt exists: “To refer them as debt letters is factually incorrect” (DHS Media Hub, 2/10/2018). Yet, “[a]ccording to lawyers that appeared before the [Senate] committee, there may be no basis in law for the department to demand that a recipient demonstrate they do not owe a purported debt” (Senate Inquiry, 2017, p. 83). Moreover, while individual appeals to the Administrative Appeals Tribunal⁹ were consistently ruled in plaintiffs’ favour and Centrelink accepted the rulings, the agency never “applied them to cases not taken to the tribunal” (The Conversation, 17/9/2019). Top management kept publicly defending OCI, although the vast majority of DHS employees thought it should be abandoned (CPSU, 2019, p. 2).

Managerial myopia (i.e., decision-makers’ short-sightedness or “lack of imagination, foresight, or intellectual insight”¹⁰), was reflected in top management’s short-term oriented measures when modifying the OCI system in response to the mounting criticism. Although top management was already pressured to implement modifications in the system after the first wave of negative press in early 2017, the Senate’s and Ombudsman’s investigations found the changes inadequate. The system received a number of helpful modifications between 2017–2019, including more informative debt notices, improved online interface, mechanisms to filter complex cases for manual processing, and employees’ expanded role capacity. These changes did little to address the public outcry, because they did not rectify the system’s core problems of speculative calculation of debts and reversed onus of proof.

Thus, although a negative feedback loop of unintended effects informed top management’s decision-making, so did a positive feedback loop of increased revenue from collected “debts”. The limited vision that had initiated the programme guided top management’s attention towards the intended effects while diluting the influence of the negative feedback loop of unintended effects (represented as a dotted arrow from unintended consequences to top management’s escalation of commitment in Figure 4). The resultant escalation of commitment inhibited major corrective measures to the ADM programme (such as restoring human oversight and reviewing debt decisions), allowing the destructive cycle to continue.

4.4. Phase IV: delegitimization process halts the destructive system

The shaded area in Figure 4 represents a delegitimization process by which the destructive system was halted. It only occurred after multiple iterations of the destructive cycle had taken place, and thus this process appears at a significantly later point in time than the creation and sustaining of destructive effects. This process was slow initially due to a lack of legal frameworks and structures to support citizens in challenging automated decisions (i.e., environmental limits) and given the novel and unprecedented nature of these issues. Governance activation refers to a constellation of governance and legal mechanisms that were mobilised in response to the negative effects of OCI, culminating in a formal court ruling that forced the system’s decommissioning.

Due to the lack of legal support infrastructures and oversight mechanisms in the broader environment, there were insufficient checks and balances to prevent the continued use of the destructive ADM programme. Many citizens who wanted to challenge or contest DHS decisions received little help to do so, with Centrelink making no legal support available to citizens and with community legal centres lacking capacity: the increased demand for legal services “placed extraordinary pressure on Community Legal Centres and Legal Aid Commissions” (ACOSS, 2017, p. 12).

As the stream of negative outcomes mounted, however, this created sufficient pressure to trigger governance activation. First, activists involved in the #NotMyDebt online movement organised to help citizens challenge their debts. Ultimately, governance activation took the form of a legal challenge and a class action against the government. A court ruling against OCI in late 2019 finally delegitimized the already widely discredited ADM programme. In anticipation of this outcome, the government had halted the programme a week before the judicial process, after having defended it for three years. The onus of proof was shifted back to Centrelink: “Centrelink has therefore belatedly decided to freeze the Robodebt System as it currently operates” (Gordon Legal, 2020).

It is noteworthy to mention that top management’s tunnel vision persisted despite the court ruling: “The Minister for Government Services Stuart Robert played down the changes and did not apologise for past errors under the system. ‘The government makes no apologies for fulfilling our legal obligation to collect debts with income from clients and of course, with wider debt collection” (The Conversation, 19/11/2019).

Later the government declared that it would relinquish its OCI-based debt claims and return all debts already collected under the scheme: “On 29 May 2020, the Commonwealth announced that it now accepts that its Robodebt System was unlawful and that it will refund 470,000 debts raised under the system to 373,000 people” (Gordon Legal, 2020).

4.5. Negative outcomes for the government

Sustaining the programme under such criticism damaged the reputation of Centrelink, DHS, and the government: “[robodebt] has severely tarnished the reputation of the [Centrelink] organisation” (CPSU, 2017, p. 2). The Australian government’s ruling party received heavy public criticism due to OCI and the way in which they responded to the critique. The programme eroded public trust in the government: “The cost to the government’s reputation for integrity is incalculable” (The Conversation, 16/5/2019), and undermined confidence in its ability to manage social services: “public confidence in the social welfare system was severely impacted” (Senate Committee, 2017, p. 2).

Later, the government did apologise for OCI: “Prime Minister Scott Morrison expressed ‘deep regret’ for hardship caused by Robodebt” (Hitch, 2020). Still, the class action proceeded: “The Representative Applicants are continuing the case because they do not believe that the Commonwealth’s announcement satisfactorily deals with all of the legal issues raised by the Robodebt System” (Gordon Legal, 2020). The number of plaintiffs in the class action continued to grow and the government agreed to a settlement worth $1.2 billion, on the day the trial was due to start, as noted above. Although the programme was launched with financial savings in mind, it resulted in economic disaster for the government with no financial gain and significant costs. In the class action settlement proceedings, a Federal Court Justice described the programme as “a shameful chapter in the administration of the commonwealth social security system and a massive failure in public administration” (Schelle, 2021).

4.6. Cascading of limits and the mechanisms of a destructive ADM system

Our analysis points to a cascade effect where one type of limit leads to the emergence of other types of limits. None of the limit types alone was responsible for the spectrum of destructive outcomes observed. Rather, the constellation of issues across different types of limits compounded and cascaded over each other, with some limit types amplifying the effects of others. In response to our research question, our analysis of these cascade effects results in four mechanisms that explain how socially destructive government ADM systems arise, are sustained, and are constrained.

We refer to the first mechanism as directing change with limited vision as it shows how limits to the strategic vision result in corresponding limits in the sociotechnical configuration of the programme. In our case, top management’s limited vision (managerial and cognitive limits) focused solely on achieving a crackdown on welfare non-compliance and revenue generation. This limited vision set in motion implementation of an ADM programme that was developed with inadequate resources and expertise. The programme relied on a limited ADM artefact (technological limits) and minimised human agency (managerial limits), preventing the detection and compensation of the technology’s deficiencies.

The second mechanism is called limiting sociotechnical agency, and it shows how limits to sociotechnical agency and technological capability trigger a cascade of various types of unintended effects. The abovementioned technological and managerial limits in the ADM programme resulted in its inability to meet the domain’s (social welfare payments) requisite variety and pushed DHS to operate beyond its limits. The increased scale of intended effects (gained revenue) afforded by ADM automation greatly amplified the scale of social destruction.

The third mechanism is referred to as dismissing destructive effects. It explains how the path dependency from the initial limited vision that set forth the ADM programme resulted in top management’s escalation of commitment (cognitive limit). In particular, the economic imperative inherent to the limited vision resulted in heightened attention to the intended effects on one hand, and a lack of attention to the much more significant unintended effects on the other. As the influence of the corrective feedback loop of negative outcomes was diluted, the destructive system was sustained.

The final mechanism, generating a societal response, reflects a process by which an initially passive environment (here, society) over time becomes increasingly active and ultimately stops the destructive cycle. The accumulation of destructive effects forced governance activation in the form of a legal precedent that established environmental limits in what some had previously considered a grey area, halting the destructive cycle.

5. Discussion

In this paper, we address the question of how socially destructive government ADM systems arise and what mechanisms sustain or constrain them. Socially destructive systems are those that can harm or even destroy an organisation and represent extreme examples of unintended negative consequences of IT use. This question arose against the background of a relative neglect of IS theory development for substantial negative consequences of IT in general and ADM in particular (Markus, 2017; Newell & Marabelli, 2015).

We investigated our research question through a case study of an Australian government organisation’s automatic welfare debt recovery programme. The case study is striking and revelatory. The many negative and destructive effects attributed to the programme were far-reaching, damaging both citizens and the organisation’s employees. The destructive effects ultimately harmed the government too, both financially and in terms of its reputation and citizen trust. A case where a government programme was suspended only because citizens engaged in legal actions is possibly unique. Yet, various other destructive systems are still operational and are likely to emerge.

Our analysis provides illuminating evidence of how organisational limits manifest beyond safety-critical contexts. Oliver et al.’s (2017) study of an airplane crash was set in a closed system, i.e., an airplane, which provided the researchers a “laboratory-like” setting for studying how limits instantiate and affect each other. Our case, in contrast, examined an open system that does not entail immediate risks of death to humans involved but does involve a higher degree of complexity and ambiguity due to its openness. The delegitimization process described in our case demonstrates how a space with relative ambiguity of limits or entire lack of limits can experience a governance activation where previously ambiguous or lacking limits are drawn and solidified in response to persistent destructive effects.

Our study answers recent calls for empirical research on unintended consequences of ADM (Newell & Marabelli, 2015; Markus, 2015; Rossi et al., 2021) and offers contributions to both theory and practice.

5.1. The mechanisms of socially destructive systems

The model in Figure 4 could be generalised to other ADM and potentially other technology initiatives in the public sector and beyond, given that the key components of the model are recognisable in other work and our findings can be expected to hold more widely. Our contribution is that our model shows how these components interact and give rise to severe unintended consequences as an emergent phenomenon. Next, we reflect on each of the identified four mechanisms.

Directing change with limited vision connects the driving strategic vision behind the IT programme to its resultant sociotechnical configuration. The evaluation of IT implementations’ goals and outcomes has long been acknowledged to be affected by values and ideology (Lyytinen & Hirschheim, 1987). Managerial vision, characterised by ideologies and imperatives, shapes IT programme’s priorities, which exerts impact on actors’ agencies, work processes, and their surrounding structures. Zuboff (2019) specifically warns about the economic imperatives that guide many companies’ use of data and algorithms, arguing that such imperatives “disregard social norms and nullify the elemental rights associated with individual autonomy” (p. 11). With Robodebt, a welfare-critical ideology and an economic imperative caused a sociotechnical change that cut costs by diminishing employees’ agency and redefining work processes in a way that shifted complex work to “cheaper labour” (ADM and citizens). This insight provides an illuminating perspective to Marjanovic & Cecez-Kecmanovic’s (2017) findings, as it reveals that the My School system too was born from a limited managerial vision: one characterised by an ideology with careless appreciation of transparency and a tunnel-vision like imperative that prioritised full transparency above other considerations, such as context-sensitivity and privacy.

Limiting sociotechnical agency is specific to the context of ADM in that it assumes that decision-making tasks can be delegated to an IT system with some degree of agency (Baird & Maruping, 2021). The mechanism suggests that a sociotechnical (human-machine) configuration (Grønsund & Aanestad, 2020) that minimises human agency and maximises algorithmic agency can create unintended negative outcomes at scale if the ADM system is unable to satisfy the principle of requisite variety in its context. This is comparable to Drummond’s (2008) case where an IT system’s inability to handle complexity caused destructive effects and raised questions about “how to combine technical and human capability” (p. 183) to avoid such outcomes. Robodebt’s issues with requisite variety were captured in more granular terms by reflecting on the ADM artefact’s limits—simplistic algorithm and inconsistent data— and their specific destructive implications in the broader system in which the sociotechnical sub-system operated. The mechanism identified in our case reveals how an ADM system produces discrimination at scale when humans are removed from the loop, creating unforeseen havoc that previous anecdotal literature has warned about (Boyd & Crawford, 2012; Favaretto et al., 2019; Newell & Marabelli, 2015) but that has not been adequately captured in prior empirical research.

Dismissing destructive consequences connects limited managerial vision to escalation of commitment, a phenomenon observed across a number of fields of study including IS (see Keil et al., 2000). Although an organisation would be expected to discontinue the use of unethical IT if the risk of legal intervention grows (Charki et al., 2017), in our case this did not happen due to escalation of commitment. The phenomenon arises in situations where decision-makers continue to commit resources to a failed or failing course of action (Staw, 1981). Staw points to a number of explanations for this phenomenon, including external justification, where individuals faced with an external threat or evaluation are motivated to prove to others that they were not wrong in earlier decision-making. Moreover, choices that involve sure losses have been found to increase risk-seeking behaviour (Kahneman & Tversky, 1979), which can incentivise decision-makers to defend a clearly flawed course of action if there is a chance that it would lead to lower losses than alternative choices. These explanations are pertinent to our case, where decision-makers, fixed on a specific ideology and political worldview, remained committed to a course of action, with an unwillingness to admit to error so as to avoid possible political damage, despite clear signals that the IT system had destructive consequences. Further, the concept of managerial myopia has been observed in previous research on managers’ tendencies to prioritise short-term profit over long-term interests (Stein, 1988; Vuori & Huy, 2016), and our case indicates it is one possible symptom of escalation of commitment.

Finally, generating a societal response suggests that if the unintended consequences of IT are significant enough, environmental entities are likely to begin delegitimization processes. This also happened with the My School system that triggered a governance activation via a public controversy and two Senate inquiries. However, that system has remained operational. This suggests that in order for a governance activation to lead into delegitimization and subsequent termination of the system that creates negative consequences, those consequences have to be severe enough, i.e., undeniably destructive and threatening to the implementing organisation’s integrity, which was the case with Robodebt. Nevertheless, the lengthy period between the first delegitimization efforts and the ultimate court ruling suggests a lack of legal infrastructures, a finding that is in line with Carney’s (2019a) analysis of the same case. Moreover, it is possible that government organisations may be less likely to respond to calls for programme change by regulatory and oversighting bodies, compared with private sector organisations.

In sum, the four mechanisms contain various elements that have appeared in prior work. The contribution of our model is to synthesise them into a research model for explaining an ADM programme’s severely negative unintended consequences. The generalisations from our model can be investigated in further work.

5.2. System limits approach

As a second theoretical contribution, we find that the core characteristics of systems and limits thinking helped us to flesh out the mechanisms by which an IS became destructive and how negative effects eventuated and were sustained. We thus propose jointly considering systems thinking (e.g., Burton-Jones et al., 2015; Checkland, 1999b) and the organisational limits perspective (e.g., Farjoun & Starbuck, 2007) augments understanding and analysis of how IS may lead to severe negative unintended consequences. We refer to this approach as a “system limits approach”.

Identifying limits in the component parts of a sub-system helps to explain the system’s ability to satisfy the principle of requisite variety, i.e., to meet the complexity of the environment in which it is implemented (Ashby, 1958). While the inability of an IS to do so has been identified as a key contributor to destructive effects (Baskerville & Land, 2004; Drummond, 2008), consideration of technological limits within the component parts of the IT artefact allows more accurate understanding of the specific causes that inhibit sufficient requisite variety (e.g., inadequate data and a simplistic algorithm as sources of technological limitedness that inhibits meeting the principle of requisite variety). Further, our systems-oriented approach reaches beyond the immediate organisational context of IS use (e.g., a department of a government organisation) to consider the wider systems wherein the organisational work systems reside (e.g., the public sector or the society). Components within the overall system, such as those related to government executives, legal frameworks, and societal actors, and their dynamic interactions with other components and sub-systems (feedback loops) help to explain how destructive systems may be allowed to continue for long periods despite their negative effects.

As Checkland (1999b) suggests, IS come into being as a result of purposeful action by stakeholders, and this purposiveness will reflect the worldview and goals of the commissioning stakeholders (e.g., an economic imperative). This worldview may limit the cognition of the stakeholders responsible for an information system, blinding them to the negative messages received in feedback about the system’s performance and result in escalation of negative consequences. In sum, we argue that a system limits approach provides a powerful analytical tool for modelling unintended consequences and destructive systems.

5.3. Practical implications

As the use of ADM at scale by the public sector is a relatively new phenomenon, the appropriate legislative, governance and normative infrastructures are still emerging and hence are unlikely as yet to offer sufficient checks and balances required to ensure these systems are operating in line with societal expectations and norms (Gillespie et al., 2021). Combined with the ability of these tools to rapidly scale and impact many thousands of citizens, this creates a fertile arena for highly destructive effects, as portrayed in our study. The Robodebt case has revealed critical deficiencies in legal frameworks and oversight mechanisms regarding government ADM use in Australia (Carney, 2019a). For instance, although the two Ombudsman inquiries had scrutinised various aspects of OCI, they left the legal grounds of reversing the onus of proof largely unaddressed.

This situation highlights the importance of ensuring effective governance and oversight functions are operating within the deploying organisation to compensate for this lack of external governance and legal oversight. Deploying organisations can proactively adopt best-practice frameworks for ensuring ethical use of ADM, including ensuring appropriate human oversight, ensuring the identification, mitigation and monitoring of risks to stakeholders before and during deployment (e.g., impact assessments, ethical review boards), checking the accuracy and robustness of algorithmic outcomes prior to deployment, and ensuring effective independent investigation processes in response to complaints (Gillespie et al., 2020).

Our study warns managers against considering ADM as a silver bullet that will yield automatic benefits when implemented in a sociotechnical system. The need for robust governance frameworks around ADM indicates that to do this successfully, governments need to develop sufficient capacity and competency to run ADM programmes in a responsible manner. These considerations suggest that managers should have sufficient domain sensitivity and willingness to invest not only in technical capacity but also in human resources.

5.4. Limitations and future research

This study has a number of limitations that suggest future research opportunities. First, our single-case study design enabled us to study the ADM phenomenon in great depth, but it limits our ability to generalise the findings, especially beyond the government context. Future research could probe into private sector applications of ADM and explore whether ideologies and imperatives manifest differently, and what kinds of effects they may exert on the sociotechnical system. Moreover, we acknowledge that our study relies on secondary data. Some data sources channelled stakeholders’ views directly through multiple interview quotes and rich descriptions that allowed us to probe DHS staff’s work disruption and the department’s managerial policies (e.g., the CPSU submission), technological limits (e.g., the Ombudsman’s reports), and citizens’ perspectives (e.g., the #NotMyDebt’s submission). However, as we did not have direct access to government decision-makers, we had to rely on proxies and interpretation when distinguishing limits at a strategy level. Therefore, while we contend that the collected data provides a sound basis for the model and theory generation we propose, our study’s findings could be complemented with first-hand data collection (e.g., conducting interviews with government decision-makers). Another potential extension of our study would be to trace the decisions made and actions taken when designing and modifying the ADM system (whether OCI or some other destructive system) by interviewing developers and other key personnel, which could shed light on counterproductive dynamics during the development and implementation process.

6. Conclusion

Our study employed a case study to develop a model of how the use of algorithmic decision-making in the public sector can lead to a destructive sociotechnical system, causing harm to citizens and reputational and financial damage to a government. The study demonstrates the advantages of a system limits research approach, which combines the perspectives of systems thinking and organisational limits, for studying negative unintended consequences of information systems and the dark side of algorithmic decision-making.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Notes

1. A programme, as opposed to a project, is either a large-scale project or a project and its continuance beyond the delivery point to the point where corporate management may take prime responsibility for benefits realisation (Musawir et al 2017). Our case study is seen as a “programme” in the latter sense.

2. https://www.servicesaustralia.gov.au/individuals/centrelink

3. The department has been known as Services Australia since May 2019, but we use the previous name in this manuscript because the majority of the focal events occurred under that name.

4. A simplified illustration, where we have included only the agencies that have key relevance for our study.

5. Enhanced versions of OCI were later rolled out under the names of Employment Information Confirmation (EIC) and Check and Update Past Information (CUPI), but we refer to the programme as OCI throughout the article for clarity’s sake.

6. Top management refers to ministers that belonged to a government during the time of these events. The government was known as the “the Coalition”, as it was an alliance of two political parties.

7. https://www.lexico.com/en/definition/tunnel_vision

8. It was recently revealed that already prior to OCI, averaging-based data-matching procedure had been used as the last resort in cases where additional information could not be retrieved from citizens or employers. However, the proportion of such cases was relatively small compared to OCI, which dramatically scaled up the use of the averaging method. (Henriques-Gomes, 2020b)

9. “The Administrative Appeals Tribunal provides independent merits review of a wide range of administrative decisions made by the Australian Government.” (www.aat.gov.au)

10. https://www.lexico.com/en/definition/myopia

11. The Conversation is one of Australia’s largest independent news and commentary sites that has expanded internationally since its launch in 2011. It is an independent source of news and views sourced from the academic and research community. (https://theconversation.com/au/who-we-are)

Appendix A. The Data Sources

Below, we discuss the data sources selected for analysis.

Inquiry Reports, Submissions, and Class Action: We collected the three major inquiry reports on OCI and selected five submissions made to one of the inquiries. First, the report published by the Commonwealth Ombudsman in April 2017 examines the concerns raised about OCI. The investigation was informed by material obtained from individual complaints, discussions with other oversight bodies, and meetings with DHS and various community stakeholders. Second, the report by the Ombudsman published two years later in April 2019 evaluated the extent to which DHS and DSS had succeeded in implementing the 2017 report’s recommendations. Third, an inquiry by the Australian Senate’s Community Affairs References Committee published in June 2017 scrutinised various aspects of OCI based on 156 submissions by public bodies, NGOs, and individuals.

To gain a more comprehensive view on particular aspects of the scheme, we also collected submissions made by five key stakeholder groups: DHS, the Community and Public Sector Union (CPSU), ACOSS, community advocacy group known as #NotMyDebt, and Victoria Legal Aid. These submissions deepened our understanding of the events from the perspective of each of the key stakeholders in the case. While DHS’ report provides an official account from the department and the government, the CPSU collected data directly from the department’s operational staff, shedding light on how OCI affected their work. ACOSS provides an external domain expert view, #NotMyDebt reflects the experiences of affected citizens, and Victoria Legal Aid drove the court case that determined OCI was unlawful. Finally, we collected online material published by Gordon Legal, a law firm that raised the class action against the Commonwealth Government on behalf of the Robodebt victims. Gordon Legal’s website (https://gordonlegal.com.au/robodebt-class-action) encompasses an outline of the motivations behind the class action and legal advice for citizens affected by OCI.

Media Articles: To ensure a balanced and representative perspective, we searched the online versions of two of Australia’s major national media outlets: the Australian Broadcasting Corporation (ABC) News and The Australian. These outlets are broadly viewed as representing liberal and conservative views, respectively, providing a balanced view of the politically sensitive Robodebt case. We supplemented these media articles with content from The Conversation,¹¹ which provides in-depth expert academic perspectives on the phenomenon. We collected all written news items and articles related to OCI published in these outlets from January 2017 (when the OCI system first appeared in the writings of these news outlets) to December 2019 (after OCI was deemed unlawful in a state court and consequently suspended by the government).

Government Media Releases: We collected all DHS “Media Hub” statements concerning the OCI programme between 2017–2019, as well as DHS responses to media reports. These typically aimed to correct news media reports’ claims that DHS saw as misleading.

Appendix B. The Timeline of Events

Figure B1 depicts the timeline of relevant events related to Robodebt between 2015 and 2019 that we identified from our dataset. The upper side of the horizontal axis represents actions taken by the government and DHS; the lower side depicts the relevant stakeholders’ responses to OCI.

Figure B1. The timeline of events.

Appendix C. The Concepts and Their Relationships

Table C1. Phase I: ADM Programme Initiated with Limited Managerial Vision

Theoretical abstractions	Empirical observations	Exemplary quotes and evidence from data
Top management’s limited vision (welfare-critical ideology) → ADM programme	Underfunding of social welfare services causing stress on the department’s capacity. Welfare non-compliance seen as a significant injustice that needs to be rectified by using ADM.	The Government’s ongoing cuts to jobs and budgets and their failure to invest in public services have taken their toll not just on the public service, but also on the communities which rely on those services. The problems with OCI, more commonly known as the robodebt debacle, are the latest illustration of this. (CPSU 2017, p. 3) The Government considers that Australians expect the welfare payment system to be fair. People should receive payments for which they are eligible: no more and no less. People receiving more than they are eligible to receive undermines the long-term sustainability of the welfare system. (DHS, 2017, p. 1) On December 5 2016, the Minister for Human Services Alan Tudge appeared on national television to discuss Centrelink’s “welfare crackdown”. The Minister stated: “we’ll find you, we’ll track you down and you will have to repay those debts and you may end up in prison” (ACOSS 2017, p. 4)
Top management’s limited vision (economic imperative) → ADM programme	Government officials want to balance the budget by going after alleged welfare non-compliance. Top management pressures DHS to rush to implement ADM with inadequate resources and without adhering to best practices. Top management’s economic imperative causes Centrelink to offload their own work tasks to customers.	The Government knew Centrelink’s debt recovery program would incorrectly tell clients they owed money if human oversight was reduced, but continued to do so in a bid to cut costs. (ABC News, 12/1/2017) 55.5 per cent of those surveyed said there was not enough staff to do the job, the same percentage who felt the strategy was rolled out due to government pressure even when concerns with the process were being expressed. (CPSU 2017, p. 4) Every organisation makes their own decisions about staffing. Bottom line though is that someone dreamed this up, someone approved it, someone executed it and someone heard the pleas for help, and nothing was done about it”. (ABC News, 6/1/2017) The Centrelink employee who spoke to 7.30 said staff were sceptical about the suggestions that the data-matching system could be refined to stop debt notices being sent out in error. “It is as if Centrelink has simply decided that their prized data-matching program is now too hard, too time consuming and too costly”, the employee said. “So let’s outsource it to the customer – they will be so overwhelmed that they will give up”. (ABC News, 12/1/2017)

Table C2. Phase II: Biased Decisions at Scale Trigger a Destructive Cycle

Theoretical abstractions	Empirical observations	Exemplary quotes and evidence from data
ADM programme → intended effects (revenue)	ADM programme generates revenue for the government by increasing the scale of debt-collection operations	“Using the newly-developed streamlined intervention approach, the savings now far outweigh the cost of undertaking the activity and far outweigh the cost of the overall measure”, the response said. (ABC News 12/1/2017) As reported in response to Questions on Notice at Senate Estimates last year, as at January 312,018 approximately $865 million in savings had been raised for all income data matching measures announced in the 2015–16 Budget. (DHS Media Hub 8/2/2019)
ADM programme → unintended effects (citizen distress)	ADM programme discriminates vulnerable people by producing biased decisions Automation produces biased decisions at scale Biased decisions at scale cause citizen distress	The Department has reversed the Onus of Proof to vulnerable, under-resourced people and provided virtually no guidance, let alone assistance, in achieving what is required of them. (#NotMyDebt 2017, p. 41) In our view, the use of existing VIs and geographical and language data to identify customers for staff assisted intervention is sensible and appropriate. However, we are concerned the existing vulnerability data may not cover all vulnerable people for the purposes of the OCI. We also note there are limitations to using VIs as a tool for identifying vulnerable groups … (The Ombudsman 2017, p. 20) There are also reports of doubling of income where the employer name provided to Centrelink is marginally different to that given to the ATO. This is leading to false notices of overpayment. At least 49,000 inaccurate discrepancy and debt notices have been generated as a result … (ACOSS 2017, p. 5) Although the data matching might appear to be similar to what was done previously, the vastly increased scale of OCI and reduction in individual human assessment means that small errors or omissions have a much more significant effect. (CPSU 2017, p 8) Our social security legislation holds that something is a debt only if proven. But robo-debt victims are not given the courtesy of the detail. They are given the figure that an automated algorithm arrives at but not shown the “working out”. Many are frightened into paying off debts that may be wrong or non-existent. (The Australian, 2/9/2019) In a complaint letter Ms G wrote to DHS in December 2016, she states she received the debt notice a few days before she gave birth to her child. She said she then had a sick newborn baby in hospital and was only able to start dealing with the debt five weeks later. In the meantime, it had already been referred to debt collectors. (The Ombudsman 2017, p. 10)
Citizen distress → employee work disruption	Affected citizens cause work overload by contacting Centrelink Employees unable to help citizens Disruption reduces employee morale	… higher than expected call volumes to DHS non-compliance numbers … higher than expected call volumes to compliance and debt recovery numbers (The Ombudsman 2017, p. 24)Federal Labor and the main public service union have warned Centrelink staff are “desperately overstretched” and struggling to cope with the public backlash over a flawed debt recovery scheme. (ABC News, 10/1/2017) … DHS’ communication and training strategy for staff was not adequate. DHS’ messaging and instructions to front line staff did not have the visual and process detail that may have helped them better understand the customer experience of the OCI system. (The Ombudsman, 2017, p. 18) “Staff had no idea these letters were being sent out until they started calls about it. It’s only when it all blew up in the media that we started getting some limited information about this debacle”. (CPSU 2017, p. 16) “I am so sad – I used to be so proud to work for DHS. Now I wonder where this joy and sense of pride has gone and realise it isn’t me – I still have that – but I realise DHS doesn’t want it anymore and doesn’t want anyone else who has it either” (CPSU 2017, p. 24) Centrelink staff will launch two weeks of rolling industrial action from February 13 in response to the Government’s controversial debt-recovery scheme. […] CPSU national secretary Nadine Flood said budget cuts and close to 5,000 job losses had made it more difficult to provide quality services to the public. “Our members are doing this because they care about the quality of the services they provide, which is why workers dealing with sensitive clients such as those being dragged through the robo-debt crisis will not be taking industrial action”, she said. (ABC News 6/2/2017)
Employee work disruption → citizen distress	Employees’ inability to help amplifies citizen distress	Key problems customers experienced were … not getting a clear explanation about the debt decision and the reasoning behind it … staff not having sufficient knowledge about how the OCI system works. (The Ombudsman 2017, p. 15)7.30 also reported on Miranda Debeljakovic, who last year received an $8,920 debt notice from Centrelink. The debt was later found to be inaccurate and was reduced twice – first to just $2,797, and later to $2,117. (ABC News 12/6/2019)

Table C3. Phase III: Escalation of Commitment Sustains the Destructive System

Theoretical abstractions	Empirical observations	Exemplary quotes and evidence from data
Top management’s escalation of commitment (deflecting critique) → ADM programme	Top management deflects critique by disputing the programme’s flaws	Mr Tudge, who this week returned from holidays, rejected assertions people were being issued with unfounded debt notices or having difficulties in updating their details. He told the ABC the Government would retain the system, rejecting reports it was flawed. “The system is working and we will continue with that system”, he said. (ABC News, 11/1/2017) Public discussion continues to present false accounts of the Department of Humans Services’ online compliance review system … We have confdence in the system and how it is helping us conduct compliance activity more effiently. (DHS Media Hub, 25/1/2017) Robo-debts have been routinely overturned as lacking a legal foundation when appealed to the first level of the Administrative Appeals Tribunal. Although the rulings have always been accepted by Centrelink in the individual cases taken before the Tribunal, Centrelink has not applied them to cases not taken to the tribunal. (The Conversation, 17/9/2019) The Minister for Government Services Stuart Robert played down the changes and did not apologise for past errors under the system. “The government makes no apologies for fulfilling our legal obligation to collect debts with income from clients and of course, with wider debt collection”. (The Conversation, 19/11/2019)
Top management’s escalation of commitment (managerial myopia) → ADM programme	Top management makes refinements to the programme but does not address its core flaws	Senator Siewert said the changes were welcome but not enough. “As far as many people are concerned it’s like moving chairs on the Titanic for a better view, whereas what people want is a completely different system and they don’t want the automated system, certainly not how it’s being implemented even with the changes”. (ABC News, 9/3/2017) Stuart Robert, the Minister for Government Services, said: “This program has been refined a number of times over the years. This is a further refinement this morning … We’ll continue to use income averaging, with other proof points, as the basis to identify the possibility of a debt. The key refinement is that income averaging, plus proof points, will be used to crystallise or finalise the debt”. (ABC News, 19/11/2019)

Table C4. Phase IV: Delegitimization Process Halts the Destructive System

Theoretical abstractions	Empirical observations	Exemplary evidence from data
Delegitimization process (lack of legal support infrastructures) → ADM programme	Lack of legal support infrastructures inhibits legal actions Ombudsman’s inquiry does not address the legal grounds of the programme, inhibiting changes	[Community Legal Centres and Legal Aid Commissions] have already had their funding cut and face further cuts of around 30% from July this year. The major concern is that people affected by the program and those who have other Centrelink matters may not be able to get the legal assistance they need because these centres lack capacity to handle the increased caseload. (ACOSS 2017, p. 12) The Commonwealth Ombudsman has independently and exhaustively reviewed this program and found it is reasonable and appropriate to ask people to explain discrepancies identified from data-matching with other agencies. (DHS Media Hub, 17/8/2019)
Delegitimization process (governance activation) → ADM programme	NGOs help affected citizens to fight off debts The Senate Committee recommends the programme to be put on hold OCI challenged in courts	#NotMyDebt formed following widespread community concern at the Government’s automated debt collection process using the Online Compliance Intervention (OCI) program in late December 2016. In just three months our community-built and volunteer run website has welcomed 512 unique stories from individuals who have received an Overpayment Discrepancy Letter or the subsequent Centrelink Payment Demand Letter. (NotMyDebt 201, p. 2) This week Tasmanian welfare organisations ramped up support for those being sent incorrect debt notices by Centrelink. Several organisations have pooled $12,000 to help community legal centres assist those caught up in the controversy. (ABC News, 20/1/2017) The committee recommends the Online Compliance Intervention (OCI) program should be put on hold until all procedural fairness flaws are addressed, and the other recommendations of this report are implemented. (The Senate Committee 2017, p. ix) The robo-debt scheme is now facing a major legal challenge. Victoria Legal Aid has launched two federal court cases challenging the validity of the automated debts raised against their clients. (ABC News 27/6/2019) In late November 2019 Gordon Legal filed the Robodebt Class Action with the Federal Court of Australia. The essence of the Robodebt Class Action is that the Robodebt system is unlawful. (Gordon Legal 2020)