\n
![](\"https:\/\/cdn0.tnwcdn.com\/wp-content\/blogs.dir\/1\/files\/2017\/12\/blackbox-796x419.jpg\")
<\/p>\n [ad_1]
\n<\/p>\n
The Hague District Court in The Netherlands yesterday ordered the Dutch government to halt its use of a black box AI system designed to predict welfare fraud. The ruling was issued over privacy concerns and is being heralded as a civil rights victory by activists and privacy advocacy groups.<\/p>\n
The particular program, System Risk Indication (SyRI), uses government data to predict whether welfare recipients are \u201clikely\u201d to commit welfare fraud. In use for years, the Dutch government has asserted that it poses no undue threat to citizen privacy.<\/p>\n
However, after numerous privacy advocacy groups \u2013 including the United Nations \u2013 expressed concerns over the what amounts to an automated criminal investigation of welfare recipients in specific low-income neighborhoods, the Court declined to see things the state\u2019s way.<\/p>\n
Read: We gave US police, ICE, and CBP AI without regulations in 2016: Now it\u2019s 1984<\/em><\/p>\n Per a press release from the Hague District Court (translated via Google Translate):<\/p>\n The court concludes that the SyRI legislation in its current form does not pass the test of Article 8, paragraph 2 of the ECHR. The court compared the objectives of the SyRI legislation, namely to prevent and combat fraud in the interest of economic well-being, with the violation of the private life that the legislation makes.<\/p>\n According to the court, the legislation does not meet the \u2018fair balance\u2019 that the ECHR requires in order to be able to speak about a sufficiently justified violation of private life. Regarding the deployment of SyRI, the legislation is insufficiently clear and verifiable. The legislation is unlawful because it violates higher law and is therefore non-binding.<\/p>\n<\/blockquote>\n What\u2019s important here is that the court didn\u2019t use GDPR regulations to come to its conclusions. Instead it derived its conclusions from the European Convention on Human Rights, which calls for any technology used to potentially profile people to strike a fair balance between privacy and necessity.<\/p>\n The Dutch government submits that it\u2019s crucial to the state\u2019s well-being that citizens maintain trust in the social security system and thus has defended its use of profiling software the same way governments in the US, UK, and countless other nations have: with the \u201chuman in the loop\u201d loophole.<\/p>\n By claiming there\u2019s a \u201chuman in the loop,\u201d the government avoids being accused of using completely autonomous systems to investigate and prosecute people. As Tech Crunch\u2019s Natasha Lomas reports:<\/p>\n GDPR\u2019s Article 22 includes the right for individuals not to be subject to solely automated individual decision-making where they can produce significant legal effects. But there can be some fuzziness around whether this applies if there\u2019s a human somewhere in the loop, such as to review a decision on objection.<\/p>\n<\/blockquote>\n By dismissing this notion and declaring such an invasion of privacy a human rights violation, the Hague District Court has set a precedent by which this loophole can finally be circumvented.<\/p>\n The Dutch government refused to reveal to the court how SyRI worked, where it got its data, or how it determined risk citing concerns that criminals would learn how to exploit the system were it to divulge such information.<\/p>\n Privacy advocates say this amounts to the government viewing every citizen as \u2018not innocent\u2019 and preemptively conducting criminal investigations without disclosing probable cause or evidence for such an investigation. Of even greater concern was the fact that the Dutch government only used SyRI in a handful of low-income neighborhoods, many of which are considered havens for poor immigrants.<\/p>\n Human Rights Watch declared the ruling a global victory in a statement released today:<\/p>\n By stopping SyRI, the Court has set an important precedent for protecting the rights of the poor in the age of automation. Governments that have relied on data analytics to police access to social security \u2013 such as those in the US, the U.K., and Australia \u2013 should heed the Court\u2019s warning about the human rights risks involved in treating social security beneficiaries as perpetual suspects.<\/p>\n<\/blockquote>\n \n Published February 6, 2020 \u2014 21:19 UTC\n <\/p>\n<\/p><\/div>\n\n
\n
\n