Rise of AI Conference 2022 · Berlin · Meta Stage

Is AI Evil?

Carla Hustedt · Stiftung Mercator

At Rise of AI 2022 in Berlin, Carla Hustedt, head of the Digital Center at Stiftung Mercator, argues that the real risk from AI is not a sentient, rebelling superintelligence but already powerful humans using the technology as a tool to consolidate power and deepen inequality. She groups the human causes of 'evil AI' into three categories, illustrated with case studies: AI used for openly harmful purposes, citing China's mass surveillance and detention of Uyghurs since 2013 through systems like the Integrated Joint Operations Platform; AI used for profit at the expense of human rights, citing Frances Haugen's revelations that Facebook knowingly steered young users toward toxic content; and AI used with good intentions but badly, citing the Dutch tax authority's self-learning fraud-detection system, which wrongly accused about 26,000 families, disproportionately from migrant backgrounds, and triggered the resignation of the Dutch cabinet in 2021. Hustedt then presents solutions she funds through philanthropy: more diverse tech teams, civil society organizations such as AlgorithmWatch, transparency registers like Amsterdam's algorithm register, and the EU's risk-based AI regulation.

Key takeaways

  • Carla Hustedt argues the real AI risk is not a rebelling superintelligence but powerful humans using AI to consolidate power and deepen inequality.
  • China's Integrated Joint Operations Platform aggregates data such as blood type and electricity usage to flag and detain Uyghurs, part of mass surveillance since 2013.
  • Frances Haugen's leaked documents showed Facebook knew its algorithm steered young users toward toxic content and chose not to act because it would reduce engagement.
  • A self-learning Dutch tax-fraud detection system wrongly accused about 26,000 families, treating dual citizenship as a risk factor, and its scandal brought down the Dutch cabinet in 2021.
  • Hustedt points to Amsterdam's public algorithmic system register and the EU's risk-based AI Act as concrete models for making automated systems more accountable.

Detailed notes

Reframing the question: humans, not sentient AI, are the risk

Hustedt opens by rejecting the science-fiction framing of AI as a sentient threat, arguing instead that the real danger is that already powerful humans will use AI as a tool and a weapon to further consolidate their power, deepening inequality in ways that threaten democracy. Drawing on her academic background as a political scientist studying inequality, she organizes the rest of the talk around three human causes of 'evil AI': using it for openly problematic purposes, using it for profit over human rights, and using it with good intentions but in harmful ways.

AI used for harm: mass surveillance of the Uyghurs

She describes how, since 2013, the Chinese government has subjected an estimated 13 million Uyghurs and other Turkic ethnic minorities to mass arbitrary detention and political and religious oppression, which she calls the largest internment of an ethnic minority since the Third Reich. Citing research by Human Rights Watch, she describes the Integrated Joint Operations Platform, China's main mass-surveillance system, which aggregates data such as blood type, height, package orders and electricity usage and alerts police when it flags something as suspicious. She notes internet freedom has declined globally for 11 consecutive years according to Freedom House, as more countries adopt this model of digital authoritarianism, even as social media remained a vital connective tool during the pandemic.

AI used for profit over human rights: Facebook's algorithms

Hustedt cites Frances Haugen, a former Facebook product manager, who revealed that Facebook knew its algorithm steered young users toward toxic content, for example directing a user searching for healthy recipes toward pro-anorexia content on Instagram, and that Mark Zuckerberg decided against measures to limit this because they would also have reduced engagement. She argues social media companies are not interested in weakening democracy any more than tobacco companies want people to die, but both profit from behavior they know is harmful.

Good intentions, bad outcomes: the Dutch child benefits scandal

Hustedt describes a system the Dutch tax authorities used from 2013 for five years to detect fraudulent childcare benefit claims, which wrongly accused about 26,000 parents, many demanding repayment of tens of thousands of euros. Dutch data protection authority research found the self-learning system baked in existing racial inequality, for example treating dual citizenship as a fraud risk factor, and was optimized to minimize missed fraud cases at the cost of many false positives. Social workers reviewing flagged cases were given no explanation of how the system reached its decisions. The scandal, kept hidden for a time by the ministry, led to the resignation of Mark Rutte's cabinet in 2021.

Solutions: diversity, civil society and regulation

Hustedt argues that because humans cause these problems, humans can also fix them. She points to tech companies halting government sales of facial recognition after research, including by MIT researcher Joy Buolamwini, showed it performed worse on women and people of color; to initiatives like She Transforms IT promoting diversity in tech teams; and to civil society groups such as AlgorithmWatch, which she funds, whose annual Automating Society report documents AI use across Europe. She also cites the Alexander von Humboldt Institute for Internet and Society's research sprints, backed by German President Frank-Walter Steinmeier as patron, and Amsterdam's public algorithmic system register as a transparency model. On regulation, she welcomes the European Commission's risk-based approach to the AI Act, which applies stricter oversight to systems affecting access to education, healthcare or childcare benefits while leaving room to experiment with lower-risk systems.

Q&A highlights

Asked about biased coding, given that most coders are white or Indian males, Hustedt says bias can enter at multiple points, the goal set by an institution, the training data, the code itself, or how an output is interpreted by a human such as a social worker, so identifying the actual source of a problem must come first. She argues that deliberately introducing friction between development and deployment helps surface these issues, and that many technical solutions for de-biasing data and code already exist; the real problem, she says, is ensuring they are actually used.

Quotes

I believe that we should fear that the already powerful humans will use this technology as a tool and as a weapon to further consolidate their power.

Carla Hustedt

It's humans who are the problem behind evil AI, it's also humans who can solve it.

Carla Hustedt

Solutions are never entirely technical, we're actually dealing with socio-economic systems.

Carla Hustedt

Questions answered in this talk

How does China use AI to surveil the Uyghur minority?

According to Carla Hustedt at Rise of AI 2022, China's Integrated Joint Operations Platform aggregates data such as blood type, height, purchases and electricity usage and alerts police when it detects something it flags as suspicious, as part of mass arbitrary detention of an estimated 13 million Uyghurs and other Turkic minorities since 2013.

What happened in the Dutch child benefits (childcare) scandal?

According to Carla Hustedt at Rise of AI 2022, a self-learning fraud-detection system used by Dutch tax authorities from 2013 wrongly accused about 26,000 families, disproportionately those with migration backgrounds, of fraud, partly because dual citizenship was treated as a risk factor, a scandal that led to the resignation of Mark Rutte's cabinet in 2021.

What is the EU's risk-based approach to AI regulation?

According to Carla Hustedt at Rise of AI 2022, the European Commission's proposed AI Act applies different levels of regulatory restriction depending on how much risk an AI system poses to human rights and society, allowing more freedom to experiment with minimal-risk systems while imposing strict oversight on systems affecting areas like education, healthcare access or childcare benefits.

Mentioned

mass surveillance, facial recognition bias, algorithmic transparency, risk-based AI regulation, Automating Society report, digital authoritarianism

Keep exploring

More from Carla Hustedt

More from Rise of AI Conference 2022

More on Ethics & Trust

Transcript

Auto-generated transcript, not edited. · 3,676 words

Show the full transcript

our our next guest is a front runner in digitization of our society and chris boos gave us a very good start and and it's a great interlinkage here um Carla Hustedt she's head of um the digital center of the mercado stiftung an independent foundation and there she is dealing with topics around ethics of algorithms digital public so she's talking about the or she's also researching and dealing with the public aspects of ai and chris also mentioned the society and our our human attributes into ai and i think that's a largely important topic she's also dealing with climate and digitization and how to push the energy vendor so how to solve our climate crisis and um now well let's welcome Carla Hustedt and with her speech is ai evil hi everybody also hi to everybody in this stream um this is not the best spot right after the coffee break when everybody's enjoying for the first time meeting again getting into conversations um but i'm really happy to be here and to be here physically and

to actually look into faces and not just at my computer is a.i evil if you would pose this question to a group of science fiction writers the majority would probably answer yes we all know these stories about sentient ai taking over control and it's very easy to get lost in these fantasies of super intelligent robots rebelling against the human creators and trying to kill humans i think however that these kind of images these characters tell us a lot about human fears but very little about the actual dangers of modern technology like artificial intelligence i believe that instead of fearing that ai will become sentient that it will become super intelligent and rebel against humans i believe that we should fear that the already powerful humans will use this technology as a tool and as a weapon to further consolidate their power and that this trend will lead to further inequalities and that can be threatening for our democracies which is why i won't talk about this guy today but instead i will talk about us

humans and something that wasn't said in the introduction is that i grew up with technology my dad had a computer store when i was small and my first job was actually formatting floppy disk so i was surrounded by technology very from an early stage and i believe in all the positive potentials of technology but i also started in academics and i started my research on inequalities i'm actually a political scientist so i look at the different ways human structures institutions reproduce inequalities and how we can work against it so what i want to talk about today is the human causes for evil ai and i've created a categorization and i've brought some examples for all the three categories which are humans using ai for what you could call evil purposes problematic purposes purposefully destroying democratic structures humans using ai for profit but also putting profit over human rights and humans using ai with good intentions but in a really bad way and we heard a lot about china today and how it's starting to take

over the ai market but something we haven't really talked about is how since 2013 the chinese government is subjecting 13 million oygours and other turkic ethnic minorities to mass arbitrary detention to political oppression oppression of religion and freedom of movement is also suppressed and what we're witnessing in china and what maybe a lot of us are not seeing is actually that this is the biggest internation of an ethnic minority since the fall of the third rise and a lot of this is actually happening with the help of modern technology and in particularly ai there's a lot of research by ngos nonprofit organizations like human rights watch that have for example analyzed the integrated joint operations platform which is the main system for mass surveillance in china and which is particularly used for the oppression of the oegor minority and what human rights watch found is that this system aggregates data on information like blood type height of people what kind of packages they order or the electricity usage and when it seems it when it

finds something suspicious it alerts the police and they act and um in many of these countries what we are seeing right now is how the original promise of the internet as a force of liberation as a force for human rights is actually turned around and what scares me the most is actually that we're seeing more and more countries adopting this model of digital authoritarianism across the world according to freedom house the internet freedom has actually decreased now in 2021 for 11 years in a row in many of these countries social media are actually the only place where people can get access to neutral information where they can connect where they can organize for social purposes but also for opposition for political purposes and particularly in the corona pandemic we've all profited from social media it allowed us to stay in contact despite restrictions to stay informed and like i said i think digitization is never particularly bad or good it's always we always need to look at the grey tones and the same goes for

social networks like facebook instagram twitter and so on you see on the right here francis hogan she's an american computer scientist and she's a former product manager at facebook and she just revealed a lot of information on social networks because at the very latest in server relations we know not only how social media can be harmful for us as individuals but we also know that they know and that they don't do much about it hogan revealed that facebook has known for years that its algorithm is actually steering young people towards toxic content for example if a young person is googling for healthy recipes the instagram algorithm after a while starts to show them content that is related to anorexia and horgan also showed that facebook has known this information and she published documents from meetings with mark zuckerberg where he decided not to take measures that would limit the spread of this toxic content and the spread of misinformation because at the same time these measures would have also limited interaction social media companies are

not per se interested in the downfall or the weakening of democracy just like tobacco companies are not interested in people dying but we still need to understand that they make profit with our addiction they make profit in the words of francis haagen with hate and with anger and they sometimes accept the downfalls and the weakening of democracies if that means they can make more profit in those two cases we see how the great advancements of ai can be threatening for human rights and for democracy but in many of the cases that i've analyzed in the past years the problem is actually not that ai is particularly performing particularly well but it's the opposite it's ai being pretty stupid and even more problematic it's humans not being able to detect that stupidity because they believe in the strong intelligence which is why the last example i brought to you um actually comes from the netherlands where in 2013 the danish at the the dutch tax authorities started to introduce a system to detect wrong applications for

child um like social money for for children for parents and to detect fraud and the system was in use for five years before it came to public attention that the system had actually wrongly accused 26 000 parents for having received social security money child child service money and so on for a lot of these parents this meant that they received letters stating that they owe sometimes ten thousands of euros to the government and what makes all of this even more problematic is that it was revealed that a lot of the families were actually from already um social like problematic backgrounds people who had less access to financial means and more people with migration background were affected and there's research now from the from the dutch data protection authority that looked at how the racial inequality that was already existent in the system before was baked into the self-learning algorithmic system that was used by the tax authorities and what they found was many causes one of them being that for example a dual citizenship was

identified as a risk factor for fraud another problem they found was that the system was optimized to not miss any case of fraud at the expense of accepting quite a lot of cases of false positives and the dutch minister the the ministry tried to hide these mistakes for quite a while but then at the end the scandal led to the step down of the whole cabinet of mark rutten in 2021 so this shows how big the backlash against such a system once it becomes public can actually be what i find particularly interesting and in a way scary about this case is that whenever a case was flagged by the system you actually had social workers in place who had to also check the case manually but they were not given any explanation whatsoever how the system worked and how it came to its risk or so we literally had to deal with a black box system here and this is actually of all the three the one that sometimes worries me the most because it

happens to me so many times that i'm sitting on panels and people are saying oh maybe democracies aren't fit for the digital age maybe we can't do it and cases like this actually lead to this kind of question they make people lose trust in technology and in democracy the good news since it's humans who are the problem behind evil ai it's also humans who can solve it and luckily as a somebody working in philanthropy i'm not just paid to analyze the problems but also to fund solutions and i'm an optimist so i brought to you some examples that actually give me hope that we can get this problem under control as long as we all work together people from business people from civil society people from science politicians it's great to see you all in this room because i really think that we can only do this with joint effort these are the headlines from i think it was 2020 after it became public that facial recognition is very good in detecting the faces of

white men and not so good at detecting the faces of people of color or women in addition a lot of people had raised the problematic impact of facial recognition technology technology for democracy so a couple of tech companies actually reacted and said we will no longer sell facial recognition to governments and of course this doesn't mean that china will now stop using facial recognition they're actually developing systems that are particularly good at identifying the faces of uigor minorities but i still like this example because it actually shows how civil society and scientists who actually brought this problem to attention can work together with businesses and can make a difference another way that i think tech companies can help us create better technology for democracy is by making their teams more diverse bring more different perspectives to the table people who will help identify these problems at an earlier stage and i'm very proud to be part of an initiative that's called she transforms it which actually brings together leaders from science from business from civil

society to yeah actually raise the the awareness on the need for more diversity in tech and to also make role models visible and i really like this treat by valavchar who's a chief digital evacus at salesforce because it shows that when we're looking for role models in tech we sometimes don't just need to look around us but sometimes we can even look in the past and we just need to remember that computer science actually used to be a very female working place as i said before a lot of these problems only get public attention because of the endless work of civil society and most of the problem projects that i found are actually ngos civil society organizations that write reports that dig really deep that do research on why these problems happen and they try to raise awareness and who you see on the right here is joy bullarini some of you might by now know her from netflix there's a documentary about her work she and i think that was like five years ago

when people weren't really talking about this topic in europe yet she published she did a ted talk actually on the biasness of facial recognition technology she's herself a researcher at mit focusing on on facial recognition and she had realized that those systems that she was working with couldn't detect her own face and she created a database to find out whether there was a problem with her face or whether there was maybe a systematic issue and she realized that the leaders and facial recognition technology at that time ibm phase plus microsoft all had problems with detecting the faces of women and people of color on the right here you see one of the organizations that we're supporting institutionally their growth their organizational development algorithm watch they're one of the front runners i would say in europe working on these topics and you can see the team here they've grown a lot in the last year which makes me really happy because they've played a key role in raising awareness on how these developments are not just

happening in the us and china but actually over here in europe and they publish once a year the automating society report in my last job at the battalion system i had the pleasure to work on this report with them where they brought together 16 journalists from different european countries to do research on cases of ai application across europe they did a great comic book on this topic as well so i can highly recommend it we're also giving them funding to take part in policy processes as you know there's a lot of discussion right now on the european AI Act and something that we are seeing is that these political discussions the meetings with politicians meps are largely dominated by big tech that a lot of money is flowing into lobbying in this industry that actually by now big tech is spending more money on lobbying in brussels than the automobile industry or the pharma industry so what we're trying to create is a counterweight i'm not saying they shouldn't have a say but it should

be a diverse debate so we're trying to create a counterweight of civil society voices there's a lot of things that we actually don't know yet when it comes to the societal effect of ai which is why of course science needs to play a role not just in the development of ai but also in looking at how it affects our society and coming up with solutions and what you see here is a picture from a project that we funded it was actually one of the first in our program at the alexander from homeworld institute which is a research institute focusing on societal questions of digitization here in berlin and it was a cooperation with the digital asia hub and the berkman client center where they brought together scientists from different countries different disciplines to work in also quite i would say modern research methods research sprints very applied cases to come up with yeah solutions for example how to make ai transparent how to avoid cases like the one in the netherlands or the question of

how you can actually democratize decisions made in social networks and we are very lucky to have frankfurter steinmeier our president as a patron in this project which i think is very important because we need political attention for these topics we need politicians to understand that this is a topic that will shape the world we live in our future which brings me to the last stakeholder group politicians government what can they do and since i talked so negatively about what happened in the netherlands i thought i would also bring a positive example from there what you see here is a screenshot from amsterdam's algorithmic system register and what they did is the city of amsterdam created a website quite simple actually where they publish cases they make it available for researchers but also for every citizen and they describe what purposes they use the systems for if they've worked with scientists maybe what kind of risk assessments they've made and so on so they actually make it transparent in the true meaning of the word they're

not just publishing code that nobody can deal with but they're trying to explain the thinking behind applying the system and i think this is a really great example and it's not that hard to adopt and duplicate in other places governments of course don't just use technology they also regulate it and i know that a lot of people from the business sector startups are right now watching the process that is happening in europe also maybe with a little bit of fear of over regulation we heard a lot today how europe is really behind and we all know that um yeah just regulation alone won't solve it so we will need modern regulation that promotes both innovation allows spaces for trying out things but still puts human rights the protection of human rights in the focus so i was really happy when i saw that the european commission developed a risk-based regulation ayosha who is here in the audience who was part of the german parliament's ai on cad commission that had also recommended this this system

where there's different levels of regulation depending on the risk an ai system poses for human rights and society so there will be a lot of space for trying out things when your ai system ends up in the bottom of the pyramid and it's a minimal risk but there will be a lot of restrictions for systems that are at the top of the pyramid at unacceptable risk i think however that we can all agree that if a system decides over access to education who gets which treatment in a hospital or child care benefits that it should actually be applied to a lot of oversight accountability so i'm quite happy seeing like the developments that are happening here i just had a chat with tanya who will sit on the panel that will follow right after my speech before and what i liked was she's going to say a little bit more about that later that she's looking at possible futures because what i did today is i talked a lot about the risks and the causes

behind the risks and i think that's important that we become aware behind about the the power dynamics behind these problems but we also need to come up with new ideas we cannot leave the discussion of what kind of world we're going to live into big big tech platforms and authoritarian regimes so i'm optimistic that we can limit the evil ai that we can actually make sure that it's used for good purposes for the strengthening of democratic structures if we work together if we don't get lost in the fantasies of super evil intelligent ai rebellion against humans if we understand that solutions are never entirely technical that we're actually dealing with socio-economic systems and if we all work together if we open ourselves for conversations also between different sectors disciplines and countries so i look forward to having a couple of more chats in the coffee break later thank you carla very much we are running a bit out of time but i have one more question to you because um the one thing i was

missing is in terms of ethical and digi digital society is this topic of biased coding so do you have a perspective on bias coding and my background is of course that 70 of coders are white males and then the rest is indian males so what about bias coding i mean i think that when we look at examples where there's bias in the outcome and i mean i've presented some like facial recognition systems where the problem lies in the data set or the the dutch example actually we have there's like multiple sources sometimes it's the goal that was set by for example the government sometimes it's the data itself sometimes it's the code sometimes it's the way an output is actually interpreted by for example a social worker so the first step would always be to look at what is the source of the problem but of course to also put in some friction maybe between the development and the implementation status and i know we want to do this all really fast and there's a

lot of potential that we can still use but sometimes some friction is good because it helps us yeah to find these issues so that's really what i'm hoping for and to be honest there's a lot of solutions already out there on how to make data less biased for example or how to de-bias the code we just need to make sure that it's actually used and that's the problem i see at the moment

Rise of AI 2027 · Berlin · May 2027

Be in the room for the next one.

Ten years of talks started as conversations in one curated room in Berlin. The next edition brings 300 selected leaders together in May 2027.