Centrelink’s compliance program

1. Introduction

The Australian Human Rights Commission (the Commission) welcomes the opportunity to make this submission to the Senate Community Affairs References Committee (the Committee) regarding its inquiry into ‘Centrelink’s compliance program’.

This inquiry follows a previous report of the Committee on various aspects of the Better Management of the Social Welfare System initiative, published in 2017.¹ The Committee will consider the operation of Centrelink’s automated compliance and outsourced debt collection practices, referred to in this submission as the ‘Centrelink Debt Program’.  This program has also sometimes been referred to as ‘Robodebt’.²

This submission does not address each item in the Terms of Reference. Rather, it focuses on how automated debt collection, and in particular the delivery and management of the Centrelink Debt Program, engages human rights. It draws on community consultation conducted by the Commission in the course of its project on human rights and technology (the Project), including an in-depth consultation that concluded earlier this year.³ A number of submissions received by the Commission in this first phase of consultation referred to Centrelink’s Debt Program.

2. Background to this submission

The Commission’s project on human rights and technology is examining the impact of new and emerging technologies using a human rights framework.

The use of new and emerging technologies, particularly in the context of government decision making and service delivery, is increasing at a rapid pace. Specifically, artificial intelligence (AI) is increasingly being used in government and non-government decision-making systems. While AI is not a term of art, it is widely used to refer to a cluster of technologies and techniques that include machine learning, some forms of automation and neural-network processing.

The Project aims to advance human rights protection in a context of unprecedented technological change. ⁴ One of the key issues being investigated by the Commission is the way in which human rights are engaged by the use of AI technologies, and in particular decision making which involves use of AI.

Because there is not universal agreement about the precise meaning of AI, there is some debate among experts about whether the Centrelink Debt Program is truly an example of decision making that uses AI. It could be described also as algorithmic decision making. This is an important debate, because clear delineation of the boundaries of the relevant technologies is necessary to regulate effectively in this area. Nevertheless, that detailed question does not need to be addressed for present purposes, and it suffices to observe that the Centrelink Debt Program raises many of the concerns that arise in respect of automated decision-making systems that unquestionably use AI—something reflected in many submissions to the Project.

In a forthcoming Discussion Paper, the Commission will propose reforms to ensure the Australian Government and others comply with international human rights law in decision making that uses AI.

3. The Centrelink Debt Program

The Centrelink Debt Program was established in July 2016 by Centrelink, within the Department of Human Services (DHS). It commenced using a new online compliance intervention (OCI) system for raising and recovering debts. The use of this automated system has been referred to by some as ‘Robodebt’, primarily because the system led to debt recovery letters being automatically generated by a computer program.

In its 2017 inquiry into the OCI system, the Commonwealth Ombudsman described the operation of the automated system as follows:

The OCI matches the earnings recorded on a customer’s Centrelink record with historical employer-reported income data from the Australian Taxation Office (ATO). Parts of the debt raising process previously done manually by compliance officers within DHS are now done using this automated process. Customers are asked to confirm or update their income using the online system. If the customer does not engage with DHS either online or in person, or if there are gaps in the information provided by the customer, the system will fill the gaps with a fortnightly income figure derived from the ATO income data for the relevant employment period (‘averaged’ data).⁵

The deployment of the Centrelink Debt Program, which used this OCI system, resulted in a very large increase in the scale of DHS’s debt-raising and recovery process. Using the manual system of identifying discrepancies, DHS estimated it would make around 20,000 compliance interventions per year; in 2016–17, DHS estimated it would undertake approximately 783,000 interventions. ⁶

When the Centrelink Debt Program commenced operation, it became apparent that the ‘averaged’ data process was resulting in the generation of inaccurate debt notices, ⁷ which has a particular impact on a number of recipients who were already marginalised. ⁸ The Committee, in its 2017 inquiry, noted in its final report that it had received evidence of ‘many personal accounts of the stress and distress’ the Centrelink Debt Program had caused recipients. ⁹

Both the Committee’s inquiry and that of the Commonwealth Ombudsman identified that many of the problematic aspects of the Centrelink Debt Program related to how the system was rolled out. This included, for example, the lack of information about the Debt Program, and difficulty accessing information about how to challenge or seek review of a debt nominated in a debt recovery letter. ¹⁰

4. Human rights implications of automated decision making by government 

4.1    Relevant human rights

The use of an automated decision-making system as a means of collecting debts relating to social security entitlements engages a range of human rights. Most relevantly, the right to social security is protected by Article 9 of the International Covenant on Economic, Social and Cultural Rights (ICESCR). ¹¹ As a party to ICESCR, Australia must fulfil this right by establishing a social security system, within the government’s maximum available resources, to support access to social security support without discrimination. ¹² While everyone has the right to social security, nation states should give special attention to those ‘who traditionally face difficulties in exercising this right’. ¹³

In addition, governments must ensure that eligibility criteria for social security benefits are ‘reasonable, proportionate and transparent’. ¹⁴ Further, any ‘withdrawal, reduction or suspension’ of social security benefits should be circumscribed and ‘based on grounds that are reasonable, subject to due process, and provided for in national law’. ¹⁵

The right to social security has been recognised as an enabling right, supporting the realisation of a range of human rights in the ICESCR and other human rights treaties, such as the right to an effective remedy, ¹⁶ provision of child care and welfare, ¹⁷ right to health, ¹⁸ right to work ¹⁹ and right to an adequate standard of living. ²⁰ In addition, social security plays an important role ‘through its redistributive character … in poverty reduction and alleviation, preventing social exclusion and promoting social inclusion’. ²¹

Any system that arbitrarily interferes with people’s social security entitlements will be likely to interfere impermissibly with the ICESCR rights discussed above. 

Automated decision-making systems can also engage a number of other human rights. A particular problem that can arise in this context is known as ‘algorithmic bias’. Algorithmic bias has been identified in automated decision-making systems, leading to errors that unfairly disadvantage people by reference to their race, gender and other protected attributes. ²² This can amount to unlawful discrimination and interfere with a number of human rights protected in international and Australian law—most obviously, the right to equality and non-discrimination. ²³

Depending on the context in which an automated decision-making system is deployed, it can also engage other human rights as well. For example, where such a system is used to make decisions in the criminal justice system, this could engage a range of additional rights, such as the right to a fair trial, the right not to be arbitrarily detained and the right to equality before the law. ²⁴

For a variety of reasons, including the proprietary nature of software or computer programs procured to make or support government decision making, it is often unclear to someone who is subject to a decision made using an automated decision-making system how the decision was made—especially how factors are weighed in reaching the decision. This opacity can limit an individual’s right to receive reasons for a decision that affects them. This procedural right is protected, for example, by the right to obtain a remedy for breaches of human rights and the right to a fair trial.

It should be noted also that automated decision-making systems that do not include provision for rigorous human oversight of the decision-making process, and the decisions actually being made, are more prone to error. In particular, such systems are more likely to make arbitrary decisions, because these systems are not truly ‘intelligent’, in the sense that humans are intelligent. Automated decision-making systems are constrained by their programming, and in systems that use machine learning, they are constrained also by their ‘training data’, replicating any errors contained in the data set. Therefore, such systems cannot identify or self-correct many forms of systemic or particular error.

4.2    Submissions to the Project related to the Centrelink Debt Program

The Commission’s Project is ongoing.  However, submissions responding to the Commission’s Issues Paper have discussed the human rights implications of AI, including as adopted and deployed in government decision making and service delivery.²⁵  A number of those submissions expressly identified concerns with the adoption of the Centrelink Debt Program.  These concerns include:

• the impact of automated debt collection on people who are already marginalised or vulnerable, with the risk of entrenching discrimination, and contributing to further disadvantage and social exclusion
• inadequate information regarding how debts are calculated by the algorithm supporting the automated process, and the basis on which those decisions are reached
• inadequate information regarding how a decision can be challenged or reviewed
• lack of planning in the program and/or policy development phase regarding the particular needs of vulnerable debt notice recipients, many of whom experienced significant difficulty in investigating or challenging the debt or the amount calculated, including those with a cognitive impairment or a mental health condition
• a lack of transparency regarding the process by which Centrelink adopted and rolled out the automated technology used in the Centrelink Debt Program
• lack of continuing evaluation and monitoring, as well as an absence of human oversight of the program.

The concerns raised in these submissions are reflected in the formal inquiries into the Centrelink Debt Program. The Commission notes, in particular, the Committee’s conclusion in 2017 that ‘the system was so flawed it was set up to fail’, with an absence of procedural fairness at all stages of its rollout.²⁶ Issues identified in the Committee’s 2017 report reflect the concerns raised in submissions to the Project, including: a lack of consultation with vulnerable stakeholders; the lack of a testing phase for the program website; the failure to carry out a risk assessment before the process started; and a failure to evaluate the accuracy of the Centrelink Debt Program’s outputs.

The matters identified above suggest there are real concerns that a number of human rights, including the right to social security, may have been impermissibly limited by the deployment of the Centrelink Debt Program. The Commission urges the Committee to consider the human rights impact of Centrelink’s Debt Program, as outlined above, and the extent to which these issues have been addressed or remediated, in the present inquiry. 

4.3    Principles for automated decision making by government

The use of automation and similar technologies is rapidly increasing in the public sector to deliver social services and support administrative decision-making, both in Australia and overseas. Centrelink’s Debt Program is a good example of this shift. It is also an example of how problems with the delivery and management of an automated system can undermine public confidence in the use of these types of technology by government. 

There is growing concern regarding how increasing the use of technology to automate debt collection, or the use of automation and other technologies in similar contexts, will affect the human rights of individuals, particularly those who are already disadvantaged, vulnerable or marginalised. The UN Special Rapporteur on extreme poverty and human rights, for example, is currently preparing a thematic report to the UN General Assembly on digital technology, social protection and human rights.²⁷ Evidence is emerging that shows how automated decision making, used in the delivery of social services ranging from social security to child protection, can have the unintended impact of entrenching disadvantage, with a disproportionately negative impact on minority and vulnerable groups.²⁸

As discussed above, the Commission’s Project is considering, among other things, how to protect human rights in the context of automated decision making. Issues the Commission is likely to address in its forthcoming Discussion Paper and final report include:

• how to safeguard procedural fairness in automated decision making by government
• the importance of human oversight, including monitoring and evaluation of government use of automated decision-making systems
• promoting accountability and transparency regarding automated decision making by government
• the impact of AI and related technologies on vulnerable and marginalised groups, particularly those already facing barriers to digital inclusion.

The Commission will release its Discussion Paper later this year, and, following further public consultation, a final report in 2020.