At Rise of AI 2022 in Berlin, Carla Hustedt, head of the Digital Center at Stiftung Mercator, argues that the real risk from AI is not a sentient, rebelling superintelligence but already powerful humans using the technology as a tool to consolidate power and deepen inequality. She groups the human causes of 'evil AI' into three categories, illustrated with case studies: AI used for openly harmful purposes, citing China's mass surveillance and detention of Uyghurs since 2013 through systems like the Integrated Joint Operations Platform; AI used for profit at the expense of human rights, citing Frances Haugen's revelations that Facebook knowingly steered young users toward toxic content; and AI used with good intentions but badly, citing the Dutch tax authority's self-learning fraud-detection system, which wrongly accused about 26,000 families, disproportionately from migrant backgrounds, and triggered the resignation of the Dutch cabinet in 2021. Hustedt then presents solutions she funds through philanthropy: more diverse tech teams, civil society organizations such as AlgorithmWatch, transparency registers like Amsterdam's algorithm register, and the EU's risk-based AI regulation.
Key takeaways
- Carla Hustedt argues the real AI risk is not a rebelling superintelligence but powerful humans using AI to consolidate power and deepen inequality.
- China's Integrated Joint Operations Platform aggregates data such as blood type and electricity usage to flag and detain Uyghurs, part of mass surveillance since 2013.
- Frances Haugen's leaked documents showed Facebook knew its algorithm steered young users toward toxic content and chose not to act because it would reduce engagement.
- A self-learning Dutch tax-fraud detection system wrongly accused about 26,000 families, treating dual citizenship as a risk factor, and its scandal brought down the Dutch cabinet in 2021.
- Hustedt points to Amsterdam's public algorithmic system register and the EU's risk-based AI Act as concrete models for making automated systems more accountable.
Detailed notes
Reframing the question: humans, not sentient AI, are the risk
Hustedt opens by rejecting the science-fiction framing of AI as a sentient threat, arguing instead that the real danger is that already powerful humans will use AI as a tool and a weapon to further consolidate their power, deepening inequality in ways that threaten democracy. Drawing on her academic background as a political scientist studying inequality, she organizes the rest of the talk around three human causes of 'evil AI': using it for openly problematic purposes, using it for profit over human rights, and using it with good intentions but in harmful ways.
AI used for harm: mass surveillance of the Uyghurs
She describes how, since 2013, the Chinese government has subjected an estimated 13 million Uyghurs and other Turkic ethnic minorities to mass arbitrary detention and political and religious oppression, which she calls the largest internment of an ethnic minority since the Third Reich. Citing research by Human Rights Watch, she describes the Integrated Joint Operations Platform, China's main mass-surveillance system, which aggregates data such as blood type, height, package orders and electricity usage and alerts police when it flags something as suspicious. She notes internet freedom has declined globally for 11 consecutive years according to Freedom House, as more countries adopt this model of digital authoritarianism, even as social media remained a vital connective tool during the pandemic.
AI used for profit over human rights: Facebook's algorithms
Hustedt cites Frances Haugen, a former Facebook product manager, who revealed that Facebook knew its algorithm steered young users toward toxic content, for example directing a user searching for healthy recipes toward pro-anorexia content on Instagram, and that Mark Zuckerberg decided against measures to limit this because they would also have reduced engagement. She argues social media companies are not interested in weakening democracy any more than tobacco companies want people to die, but both profit from behavior they know is harmful.
Good intentions, bad outcomes: the Dutch child benefits scandal
Hustedt describes a system the Dutch tax authorities used from 2013 for five years to detect fraudulent childcare benefit claims, which wrongly accused about 26,000 parents, many demanding repayment of tens of thousands of euros. Dutch data protection authority research found the self-learning system baked in existing racial inequality, for example treating dual citizenship as a fraud risk factor, and was optimized to minimize missed fraud cases at the cost of many false positives. Social workers reviewing flagged cases were given no explanation of how the system reached its decisions. The scandal, kept hidden for a time by the ministry, led to the resignation of Mark Rutte's cabinet in 2021.
Solutions: diversity, civil society and regulation
Hustedt argues that because humans cause these problems, humans can also fix them. She points to tech companies halting government sales of facial recognition after research, including by MIT researcher Joy Buolamwini, showed it performed worse on women and people of color; to initiatives like She Transforms IT promoting diversity in tech teams; and to civil society groups such as AlgorithmWatch, which she funds, whose annual Automating Society report documents AI use across Europe. She also cites the Alexander von Humboldt Institute for Internet and Society's research sprints, backed by German President Frank-Walter Steinmeier as patron, and Amsterdam's public algorithmic system register as a transparency model. On regulation, she welcomes the European Commission's risk-based approach to the AI Act, which applies stricter oversight to systems affecting access to education, healthcare or childcare benefits while leaving room to experiment with lower-risk systems.
Q&A highlights
Asked about biased coding, given that most coders are white or Indian males, Hustedt says bias can enter at multiple points, the goal set by an institution, the training data, the code itself, or how an output is interpreted by a human such as a social worker, so identifying the actual source of a problem must come first. She argues that deliberately introducing friction between development and deployment helps surface these issues, and that many technical solutions for de-biasing data and code already exist; the real problem, she says, is ensuring they are actually used.
Quotes
I believe that we should fear that the already powerful humans will use this technology as a tool and as a weapon to further consolidate their power.
Carla Hustedt
It's humans who are the problem behind evil AI, it's also humans who can solve it.
Carla Hustedt
Solutions are never entirely technical, we're actually dealing with socio-economic systems.
Carla Hustedt
Questions answered in this talk
How does China use AI to surveil the Uyghur minority?
According to Carla Hustedt at Rise of AI 2022, China's Integrated Joint Operations Platform aggregates data such as blood type, height, purchases and electricity usage and alerts police when it detects something it flags as suspicious, as part of mass arbitrary detention of an estimated 13 million Uyghurs and other Turkic minorities since 2013.
What happened in the Dutch child benefits (childcare) scandal?
According to Carla Hustedt at Rise of AI 2022, a self-learning fraud-detection system used by Dutch tax authorities from 2013 wrongly accused about 26,000 families, disproportionately those with migration backgrounds, of fraud, partly because dual citizenship was treated as a risk factor, a scandal that led to the resignation of Mark Rutte's cabinet in 2021.
What is the EU's risk-based approach to AI regulation?
According to Carla Hustedt at Rise of AI 2022, the European Commission's proposed AI Act applies different levels of regulatory restriction depending on how much risk an AI system poses to human rights and society, allowing more freedom to experiment with minimal-risk systems while imposing strict oversight on systems affecting areas like education, healthcare access or childcare benefits.