Algorithmic decision-making and system destructiveness: A case of automatic debt recovery

Figures & data

Figure 1. Organisational structure of the Australian welfare system.

Table 1. Summary of data sources

Type of data	Description	Documents	Pages
Investigation reports	The Acting Commonwealth Ombudsman: “Centrelink’s automated debt raising and recovery system – A Report About the Department of Human Services’ Online Compliance Intervention System for Debt Raising and Recovery” June 2017	1	113
	The Acting Commonwealth Ombudsman: “Centrelink’s Automated Debt Raising and Recovery System – Implementation Report” April 2019	1	45
	The Senate Community Affairs References Committee: “Design, scope, cost-benefit analysis, contracts awarded and implementation associated with the Better Management of the Social Welfare System initiative” June 2017	1	155
Submissions to inquiry	Submissions to the 2017 Senate Inquiry by: The Australian Council of Social Services (ACOSS) The Community and Public Sector Union (CPSU) The Department of Human Services (DHS) #NotMyDebt Victoria Legal Aid	5	129
Class action online information	Gordon Legal: “Robodebt Class Action” Outline and FAQ sheet of the class action against the Commonwealth Government, retrieved from Gordon Legal’s website in April 2020 (https://gordonlegal.com.au/robodebt-class-action/)	2	11
News articles	Articles in ABC News	56	121
	Articles in The Australian	17	24
	Articles in The Conversation	13	49
Media releases	Media releases by DHS, retrieved from the DHS Media Hub Archives	17	27
Total		114	684

Figure 2. Data structure.

Figure 3. Phases of a destructive system.

Figure 4. The emergence, maintenance and delegitimization of a socially destructive system (the grey-shaded area demarcates a phase that comes notably later than others).

Figure B1. The timeline of events.

Table C1. Phase I: ADM Programme Initiated with Limited Managerial Vision

Theoretical abstractions	Empirical observations	Exemplary quotes and evidence from data
Top management’s limited vision (welfare-critical ideology) → ADM programme	Underfunding of social welfare services causing stress on the department’s capacity. Welfare non-compliance seen as a significant injustice that needs to be rectified by using ADM.	The Government’s ongoing cuts to jobs and budgets and their failure to invest in public services have taken their toll not just on the public service, but also on the communities which rely on those services. The problems with OCI, more commonly known as the robodebt debacle, are the latest illustration of this. (CPSU 2017, p. 3) The Government considers that Australians expect the welfare payment system to be fair. People should receive payments for which they are eligible: no more and no less. People receiving more than they are eligible to receive undermines the long-term sustainability of the welfare system. (DHS, 2017, p. 1) On December 5 2016, the Minister for Human Services Alan Tudge appeared on national television to discuss Centrelink’s “welfare crackdown”. The Minister stated: “we’ll find you, we’ll track you down and you will have to repay those debts and you may end up in prison” (ACOSS 2017, p. 4)
Top management’s limited vision (economic imperative) → ADM programme	Government officials want to balance the budget by going after alleged welfare non-compliance. Top management pressures DHS to rush to implement ADM with inadequate resources and without adhering to best practices. Top management’s economic imperative causes Centrelink to offload their own work tasks to customers.	The Government knew Centrelink’s debt recovery program would incorrectly tell clients they owed money if human oversight was reduced, but continued to do so in a bid to cut costs. (ABC News, 12/1/2017) 55.5 per cent of those surveyed said there was not enough staff to do the job, the same percentage who felt the strategy was rolled out due to government pressure even when concerns with the process were being expressed. (CPSU 2017, p. 4) Every organisation makes their own decisions about staffing. Bottom line though is that someone dreamed this up, someone approved it, someone executed it and someone heard the pleas for help, and nothing was done about it”. (ABC News, 6/1/2017) The Centrelink employee who spoke to 7.30 said staff were sceptical about the suggestions that the data-matching system could be refined to stop debt notices being sent out in error. “It is as if Centrelink has simply decided that their prized data-matching program is now too hard, too time consuming and too costly”, the employee said. “So let’s outsource it to the customer – they will be so overwhelmed that they will give up”. (ABC News, 12/1/2017)

Table C2. Phase II: Biased Decisions at Scale Trigger a Destructive Cycle

Theoretical abstractions	Empirical observations	Exemplary quotes and evidence from data
ADM programme → intended effects (revenue)	ADM programme generates revenue for the government by increasing the scale of debt-collection operations	“Using the newly-developed streamlined intervention approach, the savings now far outweigh the cost of undertaking the activity and far outweigh the cost of the overall measure”, the response said. (ABC News 12/1/2017) As reported in response to Questions on Notice at Senate Estimates last year, as at January 312,018 approximately $865 million in savings had been raised for all income data matching measures announced in the 2015–16 Budget. (DHS Media Hub 8/2/2019)
ADM programme → unintended effects (citizen distress)	ADM programme discriminates vulnerable people by producing biased decisions Automation produces biased decisions at scale Biased decisions at scale cause citizen distress	The Department has reversed the Onus of Proof to vulnerable, under-resourced people and provided virtually no guidance, let alone assistance, in achieving what is required of them. (#NotMyDebt 2017, p. 41) In our view, the use of existing VIs and geographical and language data to identify customers for staff assisted intervention is sensible and appropriate. However, we are concerned the existing vulnerability data may not cover all vulnerable people for the purposes of the OCI. We also note there are limitations to using VIs as a tool for identifying vulnerable groups … (The Ombudsman 2017, p. 20) There are also reports of doubling of income where the employer name provided to Centrelink is marginally different to that given to the ATO. This is leading to false notices of overpayment. At least 49,000 inaccurate discrepancy and debt notices have been generated as a result … (ACOSS 2017, p. 5) Although the data matching might appear to be similar to what was done previously, the vastly increased scale of OCI and reduction in individual human assessment means that small errors or omissions have a much more significant effect. (CPSU 2017, p 8) Our social security legislation holds that something is a debt only if proven. But robo-debt victims are not given the courtesy of the detail. They are given the figure that an automated algorithm arrives at but not shown the “working out”. Many are frightened into paying off debts that may be wrong or non-existent. (The Australian, 2/9/2019) In a complaint letter Ms G wrote to DHS in December 2016, she states she received the debt notice a few days before she gave birth to her child. She said she then had a sick newborn baby in hospital and was only able to start dealing with the debt five weeks later. In the meantime, it had already been referred to debt collectors. (The Ombudsman 2017, p. 10)
Citizen distress → employee work disruption	Affected citizens cause work overload by contacting Centrelink Employees unable to help citizens Disruption reduces employee morale	… higher than expected call volumes to DHS non-compliance numbers … higher than expected call volumes to compliance and debt recovery numbers (The Ombudsman 2017, p. 24)Federal Labor and the main public service union have warned Centrelink staff are “desperately overstretched” and struggling to cope with the public backlash over a flawed debt recovery scheme. (ABC News, 10/1/2017) … DHS’ communication and training strategy for staff was not adequate. DHS’ messaging and instructions to front line staff did not have the visual and process detail that may have helped them better understand the customer experience of the OCI system. (The Ombudsman, 2017, p. 18) “Staff had no idea these letters were being sent out until they started calls about it. It’s only when it all blew up in the media that we started getting some limited information about this debacle”. (CPSU 2017, p. 16) “I am so sad – I used to be so proud to work for DHS. Now I wonder where this joy and sense of pride has gone and realise it isn’t me – I still have that – but I realise DHS doesn’t want it anymore and doesn’t want anyone else who has it either” (CPSU 2017, p. 24) Centrelink staff will launch two weeks of rolling industrial action from February 13 in response to the Government’s controversial debt-recovery scheme. […] CPSU national secretary Nadine Flood said budget cuts and close to 5,000 job losses had made it more difficult to provide quality services to the public. “Our members are doing this because they care about the quality of the services they provide, which is why workers dealing with sensitive clients such as those being dragged through the robo-debt crisis will not be taking industrial action”, she said. (ABC News 6/2/2017)
Employee work disruption → citizen distress	Employees’ inability to help amplifies citizen distress	Key problems customers experienced were … not getting a clear explanation about the debt decision and the reasoning behind it … staff not having sufficient knowledge about how the OCI system works. (The Ombudsman 2017, p. 15)7.30 also reported on Miranda Debeljakovic, who last year received an $8,920 debt notice from Centrelink. The debt was later found to be inaccurate and was reduced twice – first to just $2,797, and later to $2,117. (ABC News 12/6/2019)

Table C3. Phase III: Escalation of Commitment Sustains the Destructive System

Theoretical abstractions	Empirical observations	Exemplary quotes and evidence from data
Top management’s escalation of commitment (deflecting critique) → ADM programme	Top management deflects critique by disputing the programme’s flaws	Mr Tudge, who this week returned from holidays, rejected assertions people were being issued with unfounded debt notices or having difficulties in updating their details. He told the ABC the Government would retain the system, rejecting reports it was flawed. “The system is working and we will continue with that system”, he said. (ABC News, 11/1/2017) Public discussion continues to present false accounts of the Department of Humans Services’ online compliance review system … We have confdence in the system and how it is helping us conduct compliance activity more effiently. (DHS Media Hub, 25/1/2017) Robo-debts have been routinely overturned as lacking a legal foundation when appealed to the first level of the Administrative Appeals Tribunal. Although the rulings have always been accepted by Centrelink in the individual cases taken before the Tribunal, Centrelink has not applied them to cases not taken to the tribunal. (The Conversation, 17/9/2019) The Minister for Government Services Stuart Robert played down the changes and did not apologise for past errors under the system. “The government makes no apologies for fulfilling our legal obligation to collect debts with income from clients and of course, with wider debt collection”. (The Conversation, 19/11/2019)
Top management’s escalation of commitment (managerial myopia) → ADM programme	Top management makes refinements to the programme but does not address its core flaws	Senator Siewert said the changes were welcome but not enough. “As far as many people are concerned it’s like moving chairs on the Titanic for a better view, whereas what people want is a completely different system and they don’t want the automated system, certainly not how it’s being implemented even with the changes”. (ABC News, 9/3/2017) Stuart Robert, the Minister for Government Services, said: “This program has been refined a number of times over the years. This is a further refinement this morning … We’ll continue to use income averaging, with other proof points, as the basis to identify the possibility of a debt. The key refinement is that income averaging, plus proof points, will be used to crystallise or finalise the debt”. (ABC News, 19/11/2019)

Table C4. Phase IV: Delegitimization Process Halts the Destructive System

Theoretical abstractions	Empirical observations	Exemplary evidence from data
Delegitimization process (lack of legal support infrastructures) → ADM programme	Lack of legal support infrastructures inhibits legal actions Ombudsman’s inquiry does not address the legal grounds of the programme, inhibiting changes	[Community Legal Centres and Legal Aid Commissions] have already had their funding cut and face further cuts of around 30% from July this year. The major concern is that people affected by the program and those who have other Centrelink matters may not be able to get the legal assistance they need because these centres lack capacity to handle the increased caseload. (ACOSS 2017, p. 12) The Commonwealth Ombudsman has independently and exhaustively reviewed this program and found it is reasonable and appropriate to ask people to explain discrepancies identified from data-matching with other agencies. (DHS Media Hub, 17/8/2019)
Delegitimization process (governance activation) → ADM programme	NGOs help affected citizens to fight off debts The Senate Committee recommends the programme to be put on hold OCI challenged in courts	#NotMyDebt formed following widespread community concern at the Government’s automated debt collection process using the Online Compliance Intervention (OCI) program in late December 2016. In just three months our community-built and volunteer run website has welcomed 512 unique stories from individuals who have received an Overpayment Discrepancy Letter or the subsequent Centrelink Payment Demand Letter. (NotMyDebt 201, p. 2) This week Tasmanian welfare organisations ramped up support for those being sent incorrect debt notices by Centrelink. Several organisations have pooled $12,000 to help community legal centres assist those caught up in the controversy. (ABC News, 20/1/2017) The committee recommends the Online Compliance Intervention (OCI) program should be put on hold until all procedural fairness flaws are addressed, and the other recommendations of this report are implemented. (The Senate Committee 2017, p. ix) The robo-debt scheme is now facing a major legal challenge. Victoria Legal Aid has launched two federal court cases challenging the validity of the automated debts raised against their clients. (ABC News 27/6/2019) In late November 2019 Gordon Legal filed the Robodebt Class Action with the Federal Court of Australia. The essence of the Robodebt Class Action is that the Robodebt system is unlawful. (Gordon Legal 2020)