our our next guest is a front runner in digitization of our society and chris boos gave us a very good start and and it's a great interlinkage here um Carla Hustedt she's head of um the digital center of the mercado stiftung an independent foundation and there she is dealing with topics around ethics of algorithms digital public so she's talking about the or she's also researching and dealing with the public aspects of ai and chris also mentioned the society and our our human attributes into ai and i think that's a largely important topic she's also dealing with climate and digitization and how to push the energy vendor so how to solve our climate crisis and um now well let's welcome Carla Hustedt and with her speech is ai evil hi everybody also hi to everybody in this stream um this is not the best spot right after the coffee break when everybody's enjoying for the first time meeting again getting into conversations um but i'm really happy to be here and to be here physically and
to actually look into faces and not just at my computer is a.i evil if you would pose this question to a group of science fiction writers the majority would probably answer yes we all know these stories about sentient ai taking over control and it's very easy to get lost in these fantasies of super intelligent robots rebelling against the human creators and trying to kill humans i think however that these kind of images these characters tell us a lot about human fears but very little about the actual dangers of modern technology like artificial intelligence i believe that instead of fearing that ai will become sentient that it will become super intelligent and rebel against humans i believe that we should fear that the already powerful humans will use this technology as a tool and as a weapon to further consolidate their power and that this trend will lead to further inequalities and that can be threatening for our democracies which is why i won't talk about this guy today but instead i will talk about us
humans and something that wasn't said in the introduction is that i grew up with technology my dad had a computer store when i was small and my first job was actually formatting floppy disk so i was surrounded by technology very from an early stage and i believe in all the positive potentials of technology but i also started in academics and i started my research on inequalities i'm actually a political scientist so i look at the different ways human structures institutions reproduce inequalities and how we can work against it so what i want to talk about today is the human causes for evil ai and i've created a categorization and i've brought some examples for all the three categories which are humans using ai for what you could call evil purposes problematic purposes purposefully destroying democratic structures humans using ai for profit but also putting profit over human rights and humans using ai with good intentions but in a really bad way and we heard a lot about china today and how it's starting to take
over the ai market but something we haven't really talked about is how since 2013 the chinese government is subjecting 13 million oygours and other turkic ethnic minorities to mass arbitrary detention to political oppression oppression of religion and freedom of movement is also suppressed and what we're witnessing in china and what maybe a lot of us are not seeing is actually that this is the biggest internation of an ethnic minority since the fall of the third rise and a lot of this is actually happening with the help of modern technology and in particularly ai there's a lot of research by ngos nonprofit organizations like human rights watch that have for example analyzed the integrated joint operations platform which is the main system for mass surveillance in china and which is particularly used for the oppression of the oegor minority and what human rights watch found is that this system aggregates data on information like blood type height of people what kind of packages they order or the electricity usage and when it seems it when it
finds something suspicious it alerts the police and they act and um in many of these countries what we are seeing right now is how the original promise of the internet as a force of liberation as a force for human rights is actually turned around and what scares me the most is actually that we're seeing more and more countries adopting this model of digital authoritarianism across the world according to freedom house the internet freedom has actually decreased now in 2021 for 11 years in a row in many of these countries social media are actually the only place where people can get access to neutral information where they can connect where they can organize for social purposes but also for opposition for political purposes and particularly in the corona pandemic we've all profited from social media it allowed us to stay in contact despite restrictions to stay informed and like i said i think digitization is never particularly bad or good it's always we always need to look at the grey tones and the same goes for
social networks like facebook instagram twitter and so on you see on the right here francis hogan she's an american computer scientist and she's a former product manager at facebook and she just revealed a lot of information on social networks because at the very latest in server relations we know not only how social media can be harmful for us as individuals but we also know that they know and that they don't do much about it hogan revealed that facebook has known for years that its algorithm is actually steering young people towards toxic content for example if a young person is googling for healthy recipes the instagram algorithm after a while starts to show them content that is related to anorexia and horgan also showed that facebook has known this information and she published documents from meetings with mark zuckerberg where he decided not to take measures that would limit the spread of this toxic content and the spread of misinformation because at the same time these measures would have also limited interaction social media companies are
not per se interested in the downfall or the weakening of democracy just like tobacco companies are not interested in people dying but we still need to understand that they make profit with our addiction they make profit in the words of francis haagen with hate and with anger and they sometimes accept the downfalls and the weakening of democracies if that means they can make more profit in those two cases we see how the great advancements of ai can be threatening for human rights and for democracy but in many of the cases that i've analyzed in the past years the problem is actually not that ai is particularly performing particularly well but it's the opposite it's ai being pretty stupid and even more problematic it's humans not being able to detect that stupidity because they believe in the strong intelligence which is why the last example i brought to you um actually comes from the netherlands where in 2013 the danish at the the dutch tax authorities started to introduce a system to detect wrong applications for
child um like social money for for children for parents and to detect fraud and the system was in use for five years before it came to public attention that the system had actually wrongly accused 26 000 parents for having received social security money child child service money and so on for a lot of these parents this meant that they received letters stating that they owe sometimes ten thousands of euros to the government and what makes all of this even more problematic is that it was revealed that a lot of the families were actually from already um social like problematic backgrounds people who had less access to financial means and more people with migration background were affected and there's research now from the from the dutch data protection authority that looked at how the racial inequality that was already existent in the system before was baked into the self-learning algorithmic system that was used by the tax authorities and what they found was many causes one of them being that for example a dual citizenship was
identified as a risk factor for fraud another problem they found was that the system was optimized to not miss any case of fraud at the expense of accepting quite a lot of cases of false positives and the dutch minister the the ministry tried to hide these mistakes for quite a while but then at the end the scandal led to the step down of the whole cabinet of mark rutten in 2021 so this shows how big the backlash against such a system once it becomes public can actually be what i find particularly interesting and in a way scary about this case is that whenever a case was flagged by the system you actually had social workers in place who had to also check the case manually but they were not given any explanation whatsoever how the system worked and how it came to its risk or so we literally had to deal with a black box system here and this is actually of all the three the one that sometimes worries me the most because it
happens to me so many times that i'm sitting on panels and people are saying oh maybe democracies aren't fit for the digital age maybe we can't do it and cases like this actually lead to this kind of question they make people lose trust in technology and in democracy the good news since it's humans who are the problem behind evil ai it's also humans who can solve it and luckily as a somebody working in philanthropy i'm not just paid to analyze the problems but also to fund solutions and i'm an optimist so i brought to you some examples that actually give me hope that we can get this problem under control as long as we all work together people from business people from civil society people from science politicians it's great to see you all in this room because i really think that we can only do this with joint effort these are the headlines from i think it was 2020 after it became public that facial recognition is very good in detecting the faces of
white men and not so good at detecting the faces of people of color or women in addition a lot of people had raised the problematic impact of facial recognition technology technology for democracy so a couple of tech companies actually reacted and said we will no longer sell facial recognition to governments and of course this doesn't mean that china will now stop using facial recognition they're actually developing systems that are particularly good at identifying the faces of uigor minorities but i still like this example because it actually shows how civil society and scientists who actually brought this problem to attention can work together with businesses and can make a difference another way that i think tech companies can help us create better technology for democracy is by making their teams more diverse bring more different perspectives to the table people who will help identify these problems at an earlier stage and i'm very proud to be part of an initiative that's called she transforms it which actually brings together leaders from science from business from civil
society to yeah actually raise the the awareness on the need for more diversity in tech and to also make role models visible and i really like this treat by valavchar who's a chief digital evacus at salesforce because it shows that when we're looking for role models in tech we sometimes don't just need to look around us but sometimes we can even look in the past and we just need to remember that computer science actually used to be a very female working place as i said before a lot of these problems only get public attention because of the endless work of civil society and most of the problem projects that i found are actually ngos civil society organizations that write reports that dig really deep that do research on why these problems happen and they try to raise awareness and who you see on the right here is joy bullarini some of you might by now know her from netflix there's a documentary about her work she and i think that was like five years ago
when people weren't really talking about this topic in europe yet she published she did a ted talk actually on the biasness of facial recognition technology she's herself a researcher at mit focusing on on facial recognition and she had realized that those systems that she was working with couldn't detect her own face and she created a database to find out whether there was a problem with her face or whether there was maybe a systematic issue and she realized that the leaders and facial recognition technology at that time ibm phase plus microsoft all had problems with detecting the faces of women and people of color on the right here you see one of the organizations that we're supporting institutionally their growth their organizational development algorithm watch they're one of the front runners i would say in europe working on these topics and you can see the team here they've grown a lot in the last year which makes me really happy because they've played a key role in raising awareness on how these developments are not just
happening in the us and china but actually over here in europe and they publish once a year the automating society report in my last job at the battalion system i had the pleasure to work on this report with them where they brought together 16 journalists from different european countries to do research on cases of ai application across europe they did a great comic book on this topic as well so i can highly recommend it we're also giving them funding to take part in policy processes as you know there's a lot of discussion right now on the european AI Act and something that we are seeing is that these political discussions the meetings with politicians meps are largely dominated by big tech that a lot of money is flowing into lobbying in this industry that actually by now big tech is spending more money on lobbying in brussels than the automobile industry or the pharma industry so what we're trying to create is a counterweight i'm not saying they shouldn't have a say but it should
be a diverse debate so we're trying to create a counterweight of civil society voices there's a lot of things that we actually don't know yet when it comes to the societal effect of ai which is why of course science needs to play a role not just in the development of ai but also in looking at how it affects our society and coming up with solutions and what you see here is a picture from a project that we funded it was actually one of the first in our program at the alexander from homeworld institute which is a research institute focusing on societal questions of digitization here in berlin and it was a cooperation with the digital asia hub and the berkman client center where they brought together scientists from different countries different disciplines to work in also quite i would say modern research methods research sprints very applied cases to come up with yeah solutions for example how to make ai transparent how to avoid cases like the one in the netherlands or the question of
how you can actually democratize decisions made in social networks and we are very lucky to have frankfurter steinmeier our president as a patron in this project which i think is very important because we need political attention for these topics we need politicians to understand that this is a topic that will shape the world we live in our future which brings me to the last stakeholder group politicians government what can they do and since i talked so negatively about what happened in the netherlands i thought i would also bring a positive example from there what you see here is a screenshot from amsterdam's algorithmic system register and what they did is the city of amsterdam created a website quite simple actually where they publish cases they make it available for researchers but also for every citizen and they describe what purposes they use the systems for if they've worked with scientists maybe what kind of risk assessments they've made and so on so they actually make it transparent in the true meaning of the word they're
not just publishing code that nobody can deal with but they're trying to explain the thinking behind applying the system and i think this is a really great example and it's not that hard to adopt and duplicate in other places governments of course don't just use technology they also regulate it and i know that a lot of people from the business sector startups are right now watching the process that is happening in europe also maybe with a little bit of fear of over regulation we heard a lot today how europe is really behind and we all know that um yeah just regulation alone won't solve it so we will need modern regulation that promotes both innovation allows spaces for trying out things but still puts human rights the protection of human rights in the focus so i was really happy when i saw that the european commission developed a risk-based regulation ayosha who is here in the audience who was part of the german parliament's ai on cad commission that had also recommended this this system
where there's different levels of regulation depending on the risk an ai system poses for human rights and society so there will be a lot of space for trying out things when your ai system ends up in the bottom of the pyramid and it's a minimal risk but there will be a lot of restrictions for systems that are at the top of the pyramid at unacceptable risk i think however that we can all agree that if a system decides over access to education who gets which treatment in a hospital or child care benefits that it should actually be applied to a lot of oversight accountability so i'm quite happy seeing like the developments that are happening here i just had a chat with tanya who will sit on the panel that will follow right after my speech before and what i liked was she's going to say a little bit more about that later that she's looking at possible futures because what i did today is i talked a lot about the risks and the causes
behind the risks and i think that's important that we become aware behind about the the power dynamics behind these problems but we also need to come up with new ideas we cannot leave the discussion of what kind of world we're going to live into big big tech platforms and authoritarian regimes so i'm optimistic that we can limit the evil ai that we can actually make sure that it's used for good purposes for the strengthening of democratic structures if we work together if we don't get lost in the fantasies of super evil intelligent ai rebellion against humans if we understand that solutions are never entirely technical that we're actually dealing with socio-economic systems and if we all work together if we open ourselves for conversations also between different sectors disciplines and countries so i look forward to having a couple of more chats in the coffee break later thank you carla very much we are running a bit out of time but i have one more question to you because um the one thing i was
missing is in terms of ethical and digi digital society is this topic of biased coding so do you have a perspective on bias coding and my background is of course that 70 of coders are white males and then the rest is indian males so what about bias coding i mean i think that when we look at examples where there's bias in the outcome and i mean i've presented some like facial recognition systems where the problem lies in the data set or the the dutch example actually we have there's like multiple sources sometimes it's the goal that was set by for example the government sometimes it's the data itself sometimes it's the code sometimes it's the way an output is actually interpreted by for example a social worker so the first step would always be to look at what is the source of the problem but of course to also put in some friction maybe between the development and the implementation status and i know we want to do this all really fast and there's a
lot of potential that we can still use but sometimes some friction is good because it helps us yeah to find these issues so that's really what i'm hoping for and to be honest there's a lot of solutions already out there on how to make data less biased for example or how to de-bias the code we just need to make sure that it's actually used and that's the problem i see at the moment