Senate Community Affairs Committee
The Australian Human Rights Commission (the Commission) welcomes the opportunity to make this submission to the Senate Community Affairs References Committee (the Committee) regarding its inquiry into ‘Centrelink’s compliance program’.
This inquiry follows a previous report of the Committee on various aspects of the Better Management of the Social Welfare System initiative, published in 2017.1 The Committee will consider the operation of Centrelink’s automated compliance and outsourced debt collection practices, referred to in this submission as the ‘Centrelink Debt Program’. This program has also sometimes been referred to as ‘Robodebt’.2
This submission does not address each item in the Terms of Reference. Rather, it focuses on how automated debt collection, and in particular the delivery and management of the Centrelink Debt Program, engages human rights. It draws on community consultation conducted by the Commission in the course of its project on human rights and technology (the Project), including an in-depth consultation that concluded earlier this year.3 A number of submissions received by the Commission in this first phase of consultation referred to Centrelink’s Debt Program.
The Commission’s project on human rights and technology is examining the impact of new and emerging technologies using a human rights framework.
The use of new and emerging technologies, particularly in the context of government decision making and service delivery, is increasing at a rapid pace. Specifically, artificial intelligence (AI) is increasingly being used in government and non-government decision-making systems. While AI is not a term of art, it is widely used to refer to a cluster of technologies and techniques that include machine learning, some forms of automation and neural-network processing.
The Project aims to advance human rights protection in a context of unprecedented technological change. 4 One of the key issues being investigated by the Commission is the way in which human rights are engaged by the use of AI technologies, and in particular decision making which involves use of AI.
Because there is not universal agreement about the precise meaning of AI, there is some debate among experts about whether the Centrelink Debt Program is truly an example of decision making that uses AI. It could be described also as algorithmic decision making. This is an important debate, because clear delineation of the boundaries of the relevant technologies is necessary to regulate effectively in this area. Nevertheless, that detailed question does not need to be addressed for present purposes, and it suffices to observe that the Centrelink Debt Program raises many of the concerns that arise in respect of automated decision-making systems that unquestionably use AI—something reflected in many submissions to the Project.
In a forthcoming Discussion Paper, the Commission will propose reforms to ensure the Australian Government and others comply with international human rights law in decision making that uses AI.
The Centrelink Debt Program was established in July 2016 by Centrelink, within the Department of Human Services (DHS). It commenced using a new online compliance intervention (OCI) system for raising and recovering debts. The use of this automated system has been referred to by some as ‘Robodebt’, primarily because the system led to debt recovery letters being automatically generated by a computer program.
In its 2017 inquiry into the OCI system, the Commonwealth Ombudsman described the operation of the automated system as follows:
The OCI matches the earnings recorded on a customer’s Centrelink record with historical employer-reported income data from the Australian Taxation Office (ATO). Parts of the debt raising process previously done manually by compliance officers within DHS are now done using this automated process. Customers are asked to confirm or update their income using the online system. If the customer does not engage with DHS either online or in person, or if there are gaps in the information provided by the customer, the system will fill the gaps with a fortnightly income figure derived from the ATO income data for the relevant employment period (‘averaged’ data).5
The deployment of the Centrelink Debt Program, which used this OCI system, resulted in a very large increase in the scale of DHS’s debt-raising and recovery process. Using the manual system of identifying discrepancies, DHS estimated it would make around 20,000 compliance interventions per year; in 2016–17, DHS estimated it would undertake approximately 783,000 interventions. 6
When the Centrelink Debt Program commenced operation, it became apparent that the ‘averaged’ data process was resulting in the generation of inaccurate debt notices, 7 which has a particular impact on a number of recipients who were already marginalised. 8 The Committee, in its 2017 inquiry, noted in its final report that it had received evidence of ‘many personal accounts of the stress and distress’ the Centrelink Debt Program had caused recipients. 9
Both the Committee’s inquiry and that of the Commonwealth Ombudsman identified that many of the problematic aspects of the Centrelink Debt Program related to how the system was rolled out. This included, for example, the lack of information about the Debt Program, and difficulty accessing information about how to challenge or seek review of a debt nominated in a debt recovery letter. 10
The use of an automated decision-making system as a means of collecting debts relating to social security entitlements engages a range of human rights. Most relevantly, the right to social security is protected by Article 9 of the International Covenant on Economic, Social and Cultural Rights (ICESCR). 11 As a party to ICESCR, Australia must fulfil this right by establishing a social security system, within the government’s maximum available resources, to support access to social security support without discrimination. 12 While everyone has the right to social security, nation states should give special attention to those ‘who traditionally face difficulties in exercising this right’. 13
In addition, governments must ensure that eligibility criteria for social security benefits are ‘reasonable, proportionate and transparent’. 14 Further, any ‘withdrawal, reduction or suspension’ of social security benefits should be circumscribed and ‘based on grounds that are reasonable, subject to due process, and provided for in national law’. 15
The right to social security has been recognised as an enabling right, supporting the realisation of a range of human rights in the ICESCR and other human rights treaties, such as the right to an effective remedy, 16 provision of child care and welfare, 17 right to health, 18 right to work 19 and right to an adequate standard of living. 20 In addition, social security plays an important role ‘through its redistributive character … in poverty reduction and alleviation, preventing social exclusion and promoting social inclusion’. 21
Any system that arbitrarily interferes with people’s social security entitlements will be likely to interfere impermissibly with the ICESCR rights discussed above.
Automated decision-making systems can also engage a number of other human rights. A particular problem that can arise in this context is known as ‘algorithmic bias’. Algorithmic bias has been identified in automated decision-making systems, leading to errors that unfairly disadvantage people by reference to their race, gender and other protected attributes. 22 This can amount to unlawful discrimination and interfere with a number of human rights protected in international and Australian law—most obviously, the right to equality and non-discrimination. 23
Depending on the context in which an automated decision-making system is deployed, it can also engage other human rights as well. For example, where such a system is used to make decisions in the criminal justice system, this could engage a range of additional rights, such as the right to a fair trial, the right not to be arbitrarily detained and the right to equality before the law. 24
For a variety of reasons, including the proprietary nature of software or computer programs procured to make or support government decision making, it is often unclear to someone who is subject to a decision made using an automated decision-making system how the decision was made—especially how factors are weighed in reaching the decision. This opacity can limit an individual’s right to receive reasons for a decision that affects them. This procedural right is protected, for example, by the right to obtain a remedy for breaches of human rights and the right to a fair trial.
It should be noted also that automated decision-making systems that do not include provision for rigorous human oversight of the decision-making process, and the decisions actually being made, are more prone to error. In particular, such systems are more likely to make arbitrary decisions, because these systems are not truly ‘intelligent’, in the sense that humans are intelligent. Automated decision-making systems are constrained by their programming, and in systems that use machine learning, they are constrained also by their ‘training data’, replicating any errors contained in the data set. Therefore, such systems cannot identify or self-correct many forms of systemic or particular error.
The Commission’s Project is ongoing. However, submissions responding to the Commission’s Issues Paper have discussed the human rights implications of AI, including as adopted and deployed in government decision making and service delivery.25 A number of those submissions expressly identified concerns with the adoption of the Centrelink Debt Program. These concerns include:
- the impact of automated debt collection on people who are already marginalised or vulnerable, with the risk of entrenching discrimination, and contributing to further disadvantage and social exclusion
- inadequate information regarding how debts are calculated by the algorithm supporting the automated process, and the basis on which those decisions are reached
- inadequate information regarding how a decision can be challenged or reviewed
- lack of planning in the program and/or policy development phase regarding the particular needs of vulnerable debt notice recipients, many of whom experienced significant difficulty in investigating or challenging the debt or the amount calculated, including those with a cognitive impairment or a mental health condition
- a lack of transparency regarding the process by which Centrelink adopted and rolled out the automated technology used in the Centrelink Debt Program
- lack of continuing evaluation and monitoring, as well as an absence of human oversight of the program.
The concerns raised in these submissions are reflected in the formal inquiries into the Centrelink Debt Program. The Commission notes, in particular, the Committee’s conclusion in 2017 that ‘the system was so flawed it was set up to fail’, with an absence of procedural fairness at all stages of its rollout.26 Issues identified in the Committee’s 2017 report reflect the concerns raised in submissions to the Project, including: a lack of consultation with vulnerable stakeholders; the lack of a testing phase for the program website; the failure to carry out a risk assessment before the process started; and a failure to evaluate the accuracy of the Centrelink Debt Program’s outputs.
The matters identified above suggest there are real concerns that a number of human rights, including the right to social security, may have been impermissibly limited by the deployment of the Centrelink Debt Program. The Commission urges the Committee to consider the human rights impact of Centrelink’s Debt Program, as outlined above, and the extent to which these issues have been addressed or remediated, in the present inquiry.
The use of automation and similar technologies is rapidly increasing in the public sector to deliver social services and support administrative decision-making, both in Australia and overseas. Centrelink’s Debt Program is a good example of this shift. It is also an example of how problems with the delivery and management of an automated system can undermine public confidence in the use of these types of technology by government.
There is growing concern regarding how increasing the use of technology to automate debt collection, or the use of automation and other technologies in similar contexts, will affect the human rights of individuals, particularly those who are already disadvantaged, vulnerable or marginalised. The UN Special Rapporteur on extreme poverty and human rights, for example, is currently preparing a thematic report to the UN General Assembly on digital technology, social protection and human rights.27 Evidence is emerging that shows how automated decision making, used in the delivery of social services ranging from social security to child protection, can have the unintended impact of entrenching disadvantage, with a disproportionately negative impact on minority and vulnerable groups.28
As discussed above, the Commission’s Project is considering, among other things, how to protect human rights in the context of automated decision making. Issues the Commission is likely to address in its forthcoming Discussion Paper and final report include:
- how to safeguard procedural fairness in automated decision making by government
- the importance of human oversight, including monitoring and evaluation of government use of automated decision-making systems
- promoting accountability and transparency regarding automated decision making by government
- the impact of AI and related technologies on vulnerable and marginalised groups, particularly those already facing barriers to digital inclusion.
The Commission will release its Discussion Paper later this year, and, following further public consultation, a final report in 2020.
 Senate Standing Committee on Community Affairs Design, scope, cost-benefit analysis, contracts awarded and implementation associated with the Better Management of the Social Welfare System initiative (21 June 2017) https://www.aph.gov.au/parliamentary_business/committees/senate/community_affairs/socialwelfaresystem/Report/c02
 Senate Standing Committee on Community Affairs Terms of Reference for the inquiry into Centrelink’s compliance program (August 2019) https://www.aph.gov.au/Parliamentary_Business/Committees/Senate/Community_Affairs/Centrelinkcompliance/Terms_of_Reference
 Information about the Commission’s major project on human rights and technology can be found on the project website, tech.humanrights.gov.au, and in the Issues Paper: Australian Human Rights Commission Human Rights and Technology Issues Paper (July 2018), at https://tech.humanrights.gov.au/consultation.
 Commonwealth Ombudsman Centrelink’s automated debt raising and recovery system: A report about the Department of Human Services’ Online Compliance Intervention System for Debt Raising an Recovery (April 2017), 1 at http://www.ombudsman.gov.au/__data/assets/pdf_file/0022/43528/Report-Centrelinks-automated-debt-raising-and-recovery-system-April-2017.pdf
 Commonwealth Ombudsman Centrelink’s automated debt raising and recovery system: A report about the Department of Human Services’ Online Compliance Intervention System for Debt Raising an Recovery (April 2017), 5 at http://www.ombudsman.gov.au/__data/assets/pdf_file/0022/43528/Report-Centrelinks-automated-debt-raising-and-recovery-system-April-2017.pdf
 Errors and discrepancies arose when an assumption was made about income, and, consequently, incorrect information being included in the OCI’s calculation: see Senate Standing Committee on Community Affairs Design, scope, cost-benefit analysis, contracts awarded and implementation associated with the Better Management of the Social Welfare System initiative (21 June 2017), [2.85]-[2.101], https://www.aph.gov.au/parliamentary_business/committees/senate/community_affairs/socialwelfaresystem/Report/c02
 See, for example, Australian Council of Social Service, Submission 31 to the Senate Standing Committee on Community Affairs inquiry into the Design, scope, cost-benefit analysis, contracts awarded and implementation associated with the Better Management of the Social Welfare System Initiative (March 2017), at https://www.aph.gov.au/Parliamentary_Business/Committees/Senate/Community_Affairs/SocialWelfareSystem/Submissions
 Senate Standing Committee on Community Affairs Design, scope, cost-benefit analysis, contracts awarded and implementation associated with the Better Management of the Social Welfare System initiative (21 June 2017), [1.23] https://www.aph.gov.au/parliamentary_business/committees/senate/community_affairs/socialwelfaresystem/Report/c02
 See Senate Standing Committee on Community Affairs ‘Chapter 3 – Communicating with Centrelink’ Design, scope, cost-benefit analysis, contracts awarded and implementation associated with the Better Management of the Social Welfare System initiative (21 June 2017) https://www.aph.gov.au/parliamentary_business/committees/senate/community_affairs/socialwelfaresystem/Report/c02
 International Covenant on Economic, Social and Cultural Rights, opened for signature 16 December 1966, 993 UNTS 3 (entered into force 3 January 1976) art 22; Universal Declaration of Human Rights, GA Res 217A (III), UN GAOR, 3rd Sess, 183rd Plen Mtg, UN Doc A/810 (10 December 1948) Art 22.
 UN Committee on Economic, Social and Cultural Rights General Comment No. 19: The right to social security (Art 9 of the Covenant), 39th session, UN Doc E/C.12/GC/19 (4 February 2008), .
 Ibid .
 Ibid .
 Ibid .
 International Covenant on Civil and Political Rights, opened for signature on 19 December 1966, 999 UNTS 171 (entered into force 23 March 1976), Art 2(3).
 International Convention on the Rights of the Child, opened for signature on 20 November 1989 (entered into force 2 September 1990), Art’s 3(2), 18(3).
 International Covenant on Economic, Social and Cultural Rights, opened for signature 16 December 1966, 993 UNTS 3 (entered into force 3 January 1976), Art 12(1).
 International Covenant on Economic, Social and Cultural Rights, opened for signature 16 December 1966, 993 UNTS 3 (entered into force 3 January 1976), Art 6.
 International Covenant on Economic, Social and Cultural Rights, opened for signature 16 December 1966, 993 UNTS 3 (entered into force 3 January 1976) Art 11.
 UN Committee on Economic, Social and Cultural Rights General Comment No. 19: The right to social security (Art 9 of the Covenant), 39th session, UN Doc E/C.12/GC/19 (4 February 2008), .
 For detailed analysis of the human rights implications of AI technologies, including the phenomenon of algorithmic bias, see, for example, Mark Latonero, ‘Governing Artificial Intelligence: Upholding Human Rights & Dignity’ (Data&Society, 2018); Mark Hodge and Dunstan Allison-Hope, ‘Artificial Intelligence: A Rights-Based Blueprint for Business,’ (Working Paper No 2, BSR, 2018); Filippo Raso et al, ‘Artificial Intelligence & Human Rights: Opportunities & Risks’ (Berkman Klein Center Research Publication No 2018-6).
 International Covenant on Civil and Political Rights, art 2. See also: Convention on the Elimination of All Forms of Discrimination Against Women, opened for signature 18 December 1979, Treaty Series, vol. 1249 (entered into force 3 September 1981) arts 11 and 14(2)(e); International Convention on the Elimination of All Forms of Discrimination, opened for signature 21 December 1965, 660 UNTS 195 (entered into force 4 January 1969) arts 5(e)(i) and (ii); Convention on the Rights of Persons with Disabilities, GA61/106, 61st sess, 106th plenary meeting, A/RES/61/106 (24 January 2007), Art 27.
 See, for example, Australian Human Rights Commission Human Rights and Technology Issues Paper (July 2018), 29.
 Senate Community Affairs References Committee, Parliament of Australia, Design, Scope, Cost-Benefit Analysis, Contracts Awarded and Implementation Associated with the Better Management of the Social Welfare System Initiative, 107.
 United Nations Office of the High Commissioner, ‘Call for submissions: Thematic report to the UN General Assembly on digital technology, social protection and human rights’ (Web Page, 2019) <https://www.ohchr.org/EN/Issues/Poverty/Pages/CallforinputGADigitalTechnology.aspx>
 See, for example, Virginia Eubanks, Automating Inequality (St Martin’s Press, 2018).