Rise of AI Virtual Chat 2020 · Berlin

Trustworthy AI: Opening the Black Box

Beril Sirmacek · Jönköping University

In this Rise of AI Virtual Chat from May 2020, host Fabian Westerheide talks with Dr. Beril Sirmacek, assistant professor at Jönköping University, about what trustworthy AI means and how explainability fits into it. Sirmacek illustrates trust using an analogy of an alien being told an animal is a cat: trust can come either from a system's track record of good results or from an explanation of the reasoning behind a decision. She structures trustworthiness and ethics into three subtopics: GDPR and data privacy, data security, and responsible use, and describes explainability as one mathematical tool within that larger field, not the whole answer. Using examples including elderly-care robots, cancer-stage classification and credit scoring, she discusses named tools such as LIME and SHAP, the trade-off between model complexity and explainability, and the case for hybrid systems combining feature engineering with smaller neural networks. She also addresses AGI, EU AI regulation discussed in the European Commission's recent white paper, and audience poll results on who should be responsible for AI's impact and who should define the values it acts on.

Key takeaways

  • Sirmacek frames trust in AI as coming from either a system's track record of good results or from explaining the reasoning behind a decision.
  • She divides AI trustworthiness and ethics into three subtopics: GDPR/data privacy, data security, and responsible use, with explainability as one tool supporting the third.
  • German law currently does not allow leaving an elderly person alone with a home-care robot because a machine "doesn't know that you exist."
  • She names ELI5, LIME and SHAP as concrete, open-source explainability tools that show which features drove a model's decision.
  • She argues for hybrid systems combining older feature-engineering methods with smaller neural networks to balance accuracy against explainability.

Detailed notes

Defining trust with the alien-and-cat analogy

Sirmacek explains trust in AI through an analogy: if an alien with no knowledge of Earth asks a human what an animal is and is told "it's a cat," the alien can only trust the answer either because the human/system has a track record of good decisions, or because the human explains the reasoning, for example listing features like fur, ears and a tail that define a cat. She notes that a trained system faces the same problem with a type of cat it has never seen before, and may be uncertain.

Elderly-care robots and the limits of current law

Discussing a company developing indoor robots to help elderly people at home, for example carrying medication and water, Sirmacek says such robots cannot yet be mass-produced and placed with elderly people unsupervised because AI systems are not yet trustworthy enough, and notes that in Germany there is a law against trusting a machine in this way because "a machine doesn't know that you exist."

Three subtopics of trustworthiness and ethics

Sirmacek structures trustworthiness and ethics into three areas: GDPR and data privacy (whether a system transparently discloses what data it collects, why, and how it is stored); security (whether collected data and models are protected from theft, for instance through data engineering that encodes data into a form that is unusable even if stolen); and responsible use, which she calls the least defined of the three, covering whether a system explains why it reached a given decision. She frames explainability as one mathematical tool that supports responsible use, not the whole of trustworthy AI, comparing it to fitting "how you say it's a cat" by pointing to ears and a tail.

Explainability tools and the accuracy-explainability trade-off

Sirmacek names concrete explainability tools, including ELI5 ("explain like I'm 5"), LIME and SHAP, describing them as downloadable, open-source mathematical methods that show which features or training-data patterns drove a model's decision, for example highlighting dark borders in a cancer-stage image classification. She argues there is a trade-off: more complex neural networks can achieve higher accuracy but are harder to explain, and suggests a hybrid approach that combines older feature-engineering methods with smaller neural networks can yield systems that are both more robust and more explainable.

Regulation, GDPR, and self-diagnosing systems

Responding to a question about EU AI regulation following the European Commission's AI white paper, which proposed risk categories with national authorities expected to begin work by the end of the year, Sirmacek says she believes some AI-specific laws will eventually be necessary to protect people, while acknowledging researchers' and companies' fears of restrictions similar to those introduced by GDPR. On GDPR limiting innovation compared to China or the US, she argues constraints can drive new solutions, citing an example of designing smart-office sensors that detect the presence and activity of people without identifying them, to comply with rules against workplace camera surveillance. She also discusses AGI as full domain-transfer intelligence not yet achieved, and describes a vision for industry-4.0-style self-diagnosing AI systems that can identify which of their own neurons or layers caused a low-confidence or incorrect decision.

Q&A highlights

Asked how a regulator should audit a credit-scoring model for fairness, Sirmacek says the same three subtopics apply: public disclosure of data collection, evidence of security measures, and mathematical explanation of individual decisions, for example whether a rejection was driven by a feature like a year of salary history or by a comparison to other applicants' apartments. She acknowledges that even mathematically valid explanations may not be good ways to communicate rejections to customers, and mentions widely reported cases of systems rejecting certain demographic groups, such as Black applicants, more often because of biased historical training data, though she says it is unclear whether this is accurate. Closing polls of the live audience showed 50% believing AI should always be regulated and 50% saying "it depends," a majority holding the deploying company responsible for AI's impact rather than government or the AI itself, and only 9% trusting society at large, rather than companies or government-nominated expert groups, to define the values AI should act on.

Quotes

How can I trust you? I'm an alien, I don't know anything about this world, and you tell me that it's a cat.

Beril Sirmacek

A machine doesn't know that you exist.

Beril Sirmacek

Questions answered in this talk

What does trustworthy AI mean?

According to Dr. Beril Sirmacek at this Rise of AI Virtual Chat in 2020, trustworthy AI covers whether a system can be trusted either because of a proven track record of good decisions or because it can explain its reasoning, structured around three subtopics: GDPR/data privacy, data security, and responsible use.

What is the difference between trustworthy AI and explainable AI?

Beril Sirmacek explains that explainability is a mathematical tool within the broader field of trustworthiness and ethics, used to show why a model reached a decision, but it is only one part of responsible use, not the full definition of trustworthy AI.

What tools exist to make AI models explainable?

Beril Sirmacek names ELI5 ("explain like I'm 5"), LIME and SHAP as open-source, downloadable mathematical methods that show which features or training-data patterns a model relied on to reach a given decision.

Mentioned

LIME and SHAP as open-source

Keep exploring

More from Beril Sirmacek

More from Rise of AI Virtual Chat 2020

More on Ethics & Trust

Transcript

Auto-generated transcript, not edited. · 8,384 words

Show the full transcript

welcome to the rise of AI in virtual chat series my name is Flavia invest Ida I'm an entrepreneur and investor with a passion for artificial intelligence as in every week I'm inviting an interesting AI personality to my show and in this episode my guest was dr. Beryl somatic she isn't professor for AI and we talked about the topic of explainable ER and trustworthy eye it was a very lively interactive session with plenty of questions of the live audience so watch it to learn more what trustworthy actually is and how we can code ethics into the machines welcome to the next rise of AI virtual chat so first I would like to say hello to everyone out there I don't know wait for you it's a rainy day here in Berlin Germany we have plenty of guests from all of the world as it's normal for us it's I would like you to at your you know just say hello the chat saying where you're from what you do share your LinkedIn profile if you want

to feel free to connect with everyone do you know we all do this because we like content we like to socialize and we might like to increase our network and what we all have in common here that this today is a topic which is of our interest some have a deep insight into this some have a professional curiosity others are just here to learn and observe so I will share my LinkedIn for example to feel free to to share yours no advertising you know don't double off three tentpoles else just feel free to say hello so before we start with our guests I would like to announce something is that we have started a new webinar series I believe that AI is the most important technology of our times and that millions of people need to learn skills digital skills to get along with AI either to to up plying the daily business or to become an expert like a kindergarten teacher AI ethnics is AI control or doing AI do diligence you know not

only technical but on a higher level every company has AI experts today or has someone who isn't responsible for this so if you want to increase your knowledge and we have a new webinar series and you can get them on rise of AI or someone from my team can share the link here it's 3 times 6 hours program if wants to and it's perfect if you don't know how to code so it's important this is a iPhone manager so air for CEOs this is if you don't know how to code you don't have to but after the course you at least understand all the technical stuff from all your friends or from your colleagues and it's the wonderful entry in this business and so one of the courses will be also explainable AI something which will be touch because this is the big question you know what is the eye and how can we control it do we have to control it and if we have to how what are the technical possibilities and I

will talk with this about this topic today with berelson magic and you will ask or can ask questions all the time there is a feature in this tool to ask questions so just use it also see the pulse wave I have created a couple of questions before you and you see the poll so you could get a little bit off of the mindset ask questions use the polls show you Lincoln already see it's wonderful hello Jesse hello today is a halogen see hello Simon and now let's let me introduce you to our guest be real she was born in Istanbul but her parents were of the military so she travelled all around the world he lived in Germany Italy Netherlands and continued and she started to study electronic and communication engineering in Turkey and then she started to work in Germany for the German Aerospace Center and at the Technical University Munich she also did her habilitation so you know what you need to become a professional so in osnabrück by the way uh sbrick

is really good for cognitive science in the eye and then she continued to teach in the netherlands and delve and to enter and today just before coruña hits europe she moved to a sweden because she's now an assistant professor here at young pen university and the swedish people please excuse me if i pronounced MPEG not perfectly so she's there now teaching Aryan image recognition on top of being a professor or a teacher she's also the inventor of form a they are so she had invented a tool which you can use image recognition to to recognize crops and to do you know in the food and agricultural space she's also the founder of the school of the eye and she's an animal rights activist and she is a speaker at rights of the eye that's why I know her that's why I know her for a couple of years and her work and I decided she's the right person to talk to my about trustworthy AI and saying this now let me bring her online hi

I'm there oh very nice very nice to know that everyone is there I cannot see you and I see your chat I'm so happy that your dairy and welcome everyone good first questions for you what if they come what I mean you just moved to Sweden and everything changed did anything in your routines change due to Corona yes definitely we are working at home I mean I'm working at home since the beginning of the March and just moved here and started working at home it's a little bit strange situation but under the end I was already teaching a lot on YouTube on all nine webinars so it's not very unusual to me to teach things online so I'm repeat that I've got this opportunity I can do my work mine I think we are the lucky group of people of course some people cannot do their works at home they need to be in public I hope you stay safe and we're mask if possible and protect yourself as much as possible be safe everyone

and in sweetness a little bit more relaxed do you have any limitations most well and they've got signs everywhere like don't close more than two meters to each other like they put signs everywhere but people are generally ignoring these signs and there's no policy looking if they are close to each other or not and is it good or not it's in the general discussion at the moment globally everybody is discussing if it's good or not and some people here claim that it's good idea because we've got a strategy to spread the virus as much as possible that's the strategy to develop immune system to it but that's got danger that they accept that there will be people dying in the meantime we are spreading the virus but they say we will develop immunity because they are not expecting that there will be a vaccination coming in very soon so they say there is no point to lock down the economy so that's a kind of different idea than other neighbors of this country Norway is

doing very much strick lock down and doing a different regulation they've got less debts at the moment but in long terms in Sweden they claim that it will be different results and none of us know that at the moment I'm personally trying to take care of myself staying at home and when I'm going outside for shopping groceries I hear my mask okay so let's jump into a very unclear and complicated topic because it's like a buzzword we've discussed before in the comb what do you understand under trustworthy AI what does it actually mean yeah let's say that you're an intelligent system you're an intelligent human we can also compare it with AI you're on the street okay Fabian I'm an alien mm-hmm I came in front of you you met me and I met you I'm an alien I don't know anything about this world mm-hmm all right and you're an intelligent system something passes by next to us I ask you what is this and then you answer it saket okay all right how

can I trust you I'm an alien I don't know anything about this world and you tell me that it's saket yes ha ha how can I trust you and why should I trust you obviously you can trust me or let's say nobody on earth said it's a cat - why should I trust and how can I trust having like that so if you human to human I have learned to trust you I look at your eyes I have the same observation maybe it's so so is it do you mean that I should trust you because you are doing other intelligent decisions as well and so I trust that you're the system you you you are the intelligence system the system is reliable it gave good results earlier so it's must be a good result given good results now also or do you try to explain me why you made this decision would you would you try to gain my trust by explaining why you say that saket maybe the logical and reason so since I'm

a human I believe in logic and I will explain you hey we defined this as a cat this is an animal it has fur it has okay can cut it that's how I describe a cat and maybe you have a different trim for it maybe you see it different but then we can maybe agree on the same term and say okay in your worst this is a cat right that's great so you already started explaining how you made the decision in your previous learning when your parents are raising you updates a the circuit dislocate dislocate so your system is trained by yet features and you developed some ideas on what these features should look like in order to say it's a kit but what happens if you see a kit which you didn't see before maybe even you will dot right if it's a ket or not yes so that's the situation with our systems right now and we want to trust on them and we try to get gained its trust by asking them

please explain me why you made this decision and even though if you explain it doesn't help me to trust on your new decision maybe the new type of kit which passed right now is not similar to your past learning yet definition and even you are doubtful you are not confident enough then as an alien how can I trust on you so this is the situation in our systems now and I was talking to someone who is a company and they are making robots to use indoors and this robot is meant for taking care of elderly assume that an old person is at home he is sick and the ideas for instance in this lockdown situation if it's possible the elderly people don't go out and robots helps them to carry their drugs water and helps to meet a lettings why not there is a lot of robotics at the moment in factories doing very fancy stuff why not do this at home and then he they developed already some robots but they are doubtful if

if it's good time if it's really the time that we can trust and put the robot in a group elderly home along with an elderly person what happens if an unexpected situation is seen and robot X unexpectedly how we can trust on robots to leave along with an old person I I would say today we can't I know that in Germany it's not legal and in Germany there's a law you can't trust the Machine because a machine doesn't know that you exist so it's like old law yeah I agree it's we bought the robots to help elderly people and then we don't know if you can trust them or not um how did they solve it so you say they produce computer salt still they cannot leave the robot production like in making a mass production and sell in all homes that's not in this state and probably it will not be in that state very soon because we need to develop our AI systems in a trustful way to do so there are also

some explainable AI mathematics are going on probably in the series of yours I see that you've got some explaining belay eye topics further more coming so maybe you will go deeper but there are some mathematical methods already trying to explain why this system made this decision just like you you say it why you define the kit you say it's because there is a tail because there's a here there's these ears you made this decision series and liked it the system AI architecture at the end of given decision it also says which of these characteristics are seen to make this decision so is explaining Billy is subset of trustworthy AI is it how can i I compared the truth trustworthy I and explaining Billy I because both are like buzzwords in the industry are not really defined so very good explained is one step it's it's a mathematical tool to gain a little bit more trust but inside of trust topics and the ethics that you mentioned in in the title of this talk I think

there are three major categories so trust and ethics let's say it's the larger field and explain ability methods is one mathematical technique that we can try to help through this field but it's not explained in the field I mean the few trustworthiness and ethics is a large fields and explained ability is one tool within that we can use mm-hmm and inside of trustworthiness and ethics topics I think there are three major subtopics one is gtp our data compliance data privacy yeah so is my system that did I develop in terms of trust and ethics is it also in respect and privacy of the individuals so let's say you made this home robot and left with elderly is it explaining you telling you when you are putting the robot there which data is going to be collected why it's going to be used however it's going to be stored so that should be transparent with the user is your data collection and processing and how you do things have your share things is it transparent they

shown to user in in the field of trust this gdpr rules is one of the things okay and the second subtopic is security okay it's also this data use agreed with user that you are going to collect and that they agreed you don't their privacy rules but inside of the security area are you protecting this data that you collected okay I agree that you collect my data but are you able to secure this data that somebody else doesn't steal it or you're not selling to somewhere else unconsciously also so how secure is my data collection and how secure also your decision-making systems where your models are stored in a cloud maybe in your system are you able to secure it so within the big field of trustworthiness and ethics I see this gdpr I see this security and third thing is the responsible use responsible use okay and the third thing is the responsible use are you using this data are you processing it and sharing the results responsibly so we tend distrust for two

things I see three major topics one is gtp are when the security and one is the responsible use and it's the still discussion of what is really responsible use or not I think it's little bit open area I think you also puts it into poll here we can we we can try to answer all together so how do I so privacy security responsible use I understand intellectual but how do I do it technical in the technical area respecting the data privacy video data engineering to try to I mean it's on both we become transparent about what data we are collecting and we are also doing data engineering to encode the data to another dimension that it doesn't look like the original data anymore so we do some data in and we encode the data maybe this could be the engineering or picture who do you mean as we we the developer we developers developers okay and how do you control it I mean do you believe there should be control so as you say I'm

an aesthetic trustworthy system is if I the programmers say hey your data is private your data is secure and the user is responsible but this is just me claiming it's like a marketing slogan how can you check on this well it's difficult to check indeed so you claim that my data is my data privacy is respected you are not sharing my data and you explain me exactly which data you collect for which purpose that should be enough for GDP our rules when you come to keep this transparently and if you drill the agreement of GDP are we've got already in Europe rules for to deal with I think we are secure at the first step about the privacy part and the second step to keep this data in your servers for instance that you collected you agreed with me what you collect you colleague in the servers and how to secure that in other people doesn't attack and get my data or get your models of which process this data and then it's the problem

of security I mean people try to do firewalls but this time it's difficult to really really be secure so that's not really my major area but even the zoom environment is highly attacked these days you know zoom booming so it's a little bit still open area to know how to protect but again if the engineers really collect the data in a way we also converts the data to a new dimension in code stem then even the data is stolen they cannot understand the data that they still and still you can do some engineering to know that even if your data is attacked and stolen still they cannot understand what it means okay so we're here one question bye-bye Robin and I agree with the question I had the same he asked how can we make the whole development deployment validation publishing the whole process transparent so I would understand how can we make it accessible for external and internal audits so either you have someone in your department who is head of AI he needs

to control all his teams or all the code or someone outside you know like an auditor has to do it so how can you make this and not only data but the whole AI system accessible for for monitoring yeah generally the explained ability to helps to understand how this system is working if you share the model architecture and weights you you will not be able to explain how these decisions are being done so explain ability helps you're explaining how your architecture is given response when you say disick it what it is looking at inside of the picture maybe it only sees ears and sees a cat then you're explainable to metals will show that this architecture looked at these ears therefore this decision is taken so explaining how the decision is taken and also relating it to training data so in which training data i-i've seen these information highly and it also gives an opportunity to know that maybe in some cases if you are missing some cats it are not detected well you see

that there are features in this data which could not be related to training data set so you you can know how to balance your training data set the explainable method also goes back to correcting the balance in the training data sets to retrain and gain more information so what is in explaining bility tool this is something already existing kind of purchased this or do I have to build this on my own yeah there are already some methods like le 5 explained me as if I'm 5 so it's a it's the name of the methods or lime our shaped like they're already these are shortened names for the algorithms there are already mathematical methods that people can go to a github and download the methods and they can use it to explain how your algorithm is making decision and in the big picture of transport enos and ethics I enlisted three top topics the privacy of the data security of data and responsible use may be inside of responsible use we can say that we want

to use this system responsibly by giving also why we make this decision for instance there's a software making decision on cancer stage and it looks at a picture it says this is cancer stage 3 yes why this message decision is taken is it because of some age scene or is it because color is it because of some pattern fluctuation these methods explainable methods are helping to show what this model is given response and given this output art so showing the highest activations of the model to show you why does this is taken and I'm thinking it from a business perspective first so if I have this cancer example as good so so I get the report hey you have cancer stage three in this area and then I get there like a second report which is explaining why like the reasoning part exactly the reason impart tells that I made this decision cancer stage three because most of the time in the training dataset I've seen these dark borders and I see the same dark

borders here now so explain and nimble part will explain you like what you say to know how how can I trust you that did saket you say there's a fear there's a till and these explainable metals likewise are talented I've seen in training data sets dark borders in cancer stage three and I see the same dark border here in in this new data therefore I classified it it's explaining you why I think that's the part of the big picture of the responsible use of AI so explained ability is just a tool within these big trust worthiness and it takes picture but it's also useful to retrain is to know if we be we've got ample imbalance data if you need to retrain the data or we need to remake the architecture but where I see the big picture should go is that we have to first of all develop the neural network a knowledge of ourselves so much further someday hopefully we will achieve a GI right or yes the priest I'd show general intelligence

I don't say our to show super intelligence no that's why I'm a step ahead your monster and so you know artificial superintelligence is something higher than human capacity or our common capacity words well course well the singularity everything yeah but with AGI we are talking about in general art she'll intelligence like human I mean you can wash the dishes you can cook meal and you can also repair your car for instance you've got intelligence to overcome different topics and now when we use AI we make a robot which only wash the dishes so this domain transfer of the intelligence and the AGI is not there yet so hopefully someday we will issue and this AGI will be able to be trained with very big dataset to learn from so many people what I've seen in medical area especially is that when Rami talked about this cancer stage and for instance somebody came to you you are the doctor and I always use you because I cannot see anybody else or the fine I'm fine I

came you and somebody showed their skin and you're a doctor you are going to decide which cancer stages you say it it's cancer stage before and you need to get this with surgery of you so it needs to be removed immediately and another person that the same person went to another doctor and he says this is cancer stage 2 it's not so urgent so you can live with it go back to home just keep observing if it's developing further or not and the person is just confused and I've seen really tables of people patients went to different doctors and they scored the same cancer at the same time periods differently once ate this is stage 2 one says this is stream 3 one says four or five and the person is totally confused which stage it is and what should be done should I go to operation should I be just relax and sit at home what should I do and assume that there is in even the doctor doctors do not agree with each

other so if you collocate training data set to train an AI to make the decision making you're collecting data from a group of doctors and to group of doctors are not agreeing with these decisions and your AI system is only a biased to the pool yeah to the trained group but hopefully if we achieve to AGI what really we really want is to train this HDI in very large group of people to learn this fluctuation and explain why this fluctuation is there how it's happening and train the neural network in a sensitive way and hopefully if we achieve to this AGI which explainable met and explained ability methods we have to use it differently later in the future we want to use this explained ability tools to learn back let's say there's an AGI who can make decision better than 10 different doctors done if we use explained ability tools when you've got a picture of a cancer explain me why you classify it as stage 3 then the explainable tools of this AGI tells

you because I looked at these patterns because I looked at this color because I looked at this that you missed and that could be an opportunity to learn back ok with plenty of questions is already counting seven so we'll try to bring them in yes a lot of good questions so first questions for this explain ability is from Martin Hallenbeck and he says from his view explainable eyes more than a tool it is a capability so for him it is a method technique plus a process exactly what do you think do you agree with him I do agree with him is between this trustworthiness and it takes pictures explain ability is a tool that we can try to know if we can use this ai responsibly if we can show our results with stare and confidence ratios why I've got confidence and which to which data sets its bias to a most so I explained why I came up with this decision but it's a tool it's not the answer okay yeah we have another

good questions by daughter but I will push this a little bit behind because it's a huge one an easy one by Rama chef asking can you please name again some explainable ability tools know where to book the most look for them uh yeah I mentioned any five it's a it should be I hope I'm not pronouncing wrong it must be e li5 explain explain me like I'm five like something if you google you will find it so it's it's meant for explaining why you made this decision as if it's explaining five-year-old so explain me like I'm five and one metals is so common lime its L AM II and lime is also developed the further you gots you will find some few newer versions of it and one name I mentioned is shape and it's spelt as s a s H ap I will write on the chat box here and the other one I just saw someone said there is IBM open scale and dream quark I honestly I don't know any one of

them but okay but at least not so nothing very sorry and for everyone who wants to have a longer list or detailed list just ping barrel on LinkedIn or on one of her social media feeds and she can give you more for this let's jump to the eyes it's hard to decide it's hard to decide let's jump on the highest voltage question currently yes well yes it's by daughters but it says hey I'm a philosopher that deals with ethics in autonomous vehicles decision-making process you would like to know your opinion on creating ai ai laws in Europe so should we humans create laws for machines and sure should we cooperate with robots in the process hmm and there's another one which is going to this direction about competitive edge indeed appearance so just an introduction the European Union has the Commission has released the white paper an AI couple of weeks ago when they want to rank AI and five category levels and the highest level they want to have under control so what do

you think about generating laws which are dedicated to AI now this could be a good idea because of our responsibility to humans but of course that's also worrisome to companies who who are using AI and their researchers always like can I still do my job can I still serve to people in the way that I started with for instance GDP are even was already affecting all of us suddenly our if we've got services we needed to update a lot of things and the people are fearing now if we go through again or to do we are we going to have some restrictions to not able to do our business or our research but eventually I think we will come to a stage to agree with some laws and there must be some laws to protect human about being an armed nature how it's going to use the AI that we use in businesses in industry was not major enough in some years ago and we even didn't think of rules laws but now I think

it's time to start thinking about it and start discussing what those role laws should be and to that end of the year we are supposed to have the first laws so it's in the white paper that at the end of the year in November national authorities should start with something very complicated topic they're interesting topic and also doctor asked if we should cooperate cooperate with robots in the process honestly I don't understand how we can cooperate with them do you say the self-driving car example it's like some people understand the self-driving car like the car should drive and I should be able to eat my sandwich and read my book and it should drive by itself and so it's not cooperation with a robotics system when they say so in my point of view yes we should be in cooperation and especially for giving feedback to robots so the robot or the intelligent system should be getting feedback from human that should be the major corporation and the system intelligent system in my point of

view should be a lifelong learning with reinforcement learning we say to adapt to new conditions unseen conditions for instance there is a new type of cats we said even you topped and then somebody else comes to you this is a cat because of this this also exists all you say and aha you learn something new and then you recognize that new type of kids easier next time just like wise robots and human should work together in terms of getting immediate feedback and to lifelong improved systems to adapt to new conditions otherwise the system of the robot will be either outdated at some point or will not be adapted let's say you trained yourself driving car all the time in Sweden where there are mountains and stuff and I Drive to Netherlands and suddenly everything is flat and they're comfortable and stuff so it's not an environment which is seen it has to relearn with some feedback from the user the cooperation is definitely needed and the self-driving car example I think the car should be

getting feedback from the driver the driver shouldn't just eat sandwich and read a book it must be a cooperation it doesn't mean we as a user also have responsibility then definitely definitely we have this responsibility to give good feedback and this leads me to the next question by Xavier and javi says in some air attics there's always a trade-off it's fair - versus a curse city it's security versus privacy whether he says worth the trade-off in terms of explaining explaining bility so I understand it can explain the ability be a competitive edge like a different station can in the eye which is maybe not so accurate but therefore trustworthy be a selling argument for company yeah definitely definitely that's a good question and it's really a trade of the neural network more complex it's become it can represent a higher consciousness higher intelligence but a network gives more definite answer more better accuracy but maybe you cannot explain how this decision is taken sake and in this area I find it useful to go back

to little bit old-school methods and to improve our systems improve the neural networks not by adding more layers and getting more context and going to unexplainable way the solution might be going to hybrid solutions like using the old-school metals together with neural networks what I mean is before neural networks are so widely used for decision making we used to feature engineer maybe some of the audience know we used to code things by hand like say how to recognize the kids first you look for ears and the tail and we didn't train a neural network to do this we used feature engineering to look for this specific kind of ear specific kind of tail light feature be hard coded it and then on top of it may be a small neural network can make the final decision if we go to a hybrid way doing things with future engineering and done on top of it doing put an s-1 neural network instead of very complex one we might achieve to a robust system and also explainable

system okay it must be used all the time we don't know we get so many good questions and I try to rank them a little bit ok so the next one is from Robin and he asked if you are an auditor or regulator what information of a machine what information of a machine learning model has to be collected and be provided let's say he takes the example of credit worthiness let's say I'm a regulator and I want to check if your algorithm at your bank for credit worthiness is fair and ethical and trustworthy what do I have to ask for what can hand me over or you know how do I start this process yeah I think we should go through this major three subtopics that I mentioned in this area you will hold it in your company website for instance you have to write openly with these recent rules what data you collect where you store it and for which purpose you collect that data for instance for simple newsletter I'm Dylan sending in

the website I have to write down uncle I'm collecting your email address and name for this purpose and I use it for this distance so this is the GD P R we have to write it down openly what data where and why so this is the first rule and you have to show to this Authority that you mentioned that you have it you you showed it to your customers that you have it on your website and customers can see it and the second part is the security how do you secure this data and it's a tricky part to explain you might get a clot but how do you know it's not going to be hacked so maybe you have to write a report of your engineers that they're doing some feature engineering to encode your data and make it understandable even if it's Hecht so that could be a report from your engineers that you can provide to the authorities that the security rules are provided and the third part is responsible use that it's

still in discussion are you also showing why you made the decisions when manual given the results for instance the highly used explained ability topic is rejecting the credit cards in the bank as one person makes an application for a credit cards and he gets it and the another person makes a credit card application he doesn't get a credit card so there's an AI looking at their data and doing process and saying that you cannot get it you can get it and this is the black box that we have right now and in terms of ethics and responsibility we have to also be communicated with this person why we have rejected okay because Robin continues what what about the informations which are generated through the machine learning model so if we take the rejection it's the rejection is generating an output like yes no and hopefully can explain it how do you provide so you would say it's like a report or like an API or like an interface which is then giving the response of

the explained explanation so the model has to explain why it rejected it's it's much direct to say to which features it was sensitive to in making this mature decision maybe it can say I was looking at lost one year of celery and therefore I made this decision or it can say you're living apartment looks like the apartments of people who didn't make payments earlier that's why I rejected it so in mathematical way that's the way to explain and these mathematical tools that that's the way that we can explain my decision is made like this but I'm not sure in the in the business side if it's the way to communicate with the people it might be also a question if I'm making your rejection by looking at your apartments and I say that are your apartment looks like there are other people's apartments who didn't make the payments that's why you are rejected is it the way is it the good way to communicate with a customer that's also not known yet we don't know

exactly how to communicate this decision-making we we have this it's all over the news sometimes that for example black people get rejected way more often because you know historical they didn't pay their bills therefore the system has learned they don't pay the bills maybe it's true maybe it's not true it's it's very tricky thing and drew is asking questions to create trustworthy I it should be a self defensive system so how can we make AI resistant to AI poisoning attacks is it is it about adversarial attacks or is it is it still and AI architecture and how it works right he also says it's not true AI but I agree if we start now the discussion what's AI what's machine learning that's a very different topic but I like the question is it shouldn't say I or a system of they defend itself should it be able to hide say okay that was wrong data injected someone tried to man you need be biases how do we generate something the feeling for the AI to

say this is right or wrong maybe not to defend so I wouldn't word it liked it but to repair itself like something like that which you know we are going into industry 4.0 at the moment and one of the major changes expected in industry 4.0 is that machines I'm not talking about software and AI at the moment the machines in the factories should be able to identify their own error so for a new machine failed in industry its it shouldn't just say an error message it should also say I am given error because the heat at this point is over three santé grad degree is more than what I want no it should identify exactly the problem to you that's what we expect in industry a 4.0 and likewise what I expect is that if an AI system is not given results with high confidence it should detect in its structure maybe which neurons to remove or which layers to remove or what layers to add so it might also be self-conscious of its structure

and where it's given the responses activations so and it must able to say this neuron is was given higher activation that's why I was given this error so like the industrial system it must self identify its own error in the in in the structure mm-hmm another question from Robin in this we would touch it briefly I mean gdpr it's considered a limitation for us to work freely I mean we know China doesn't have GDP or they they're free with data which is fostering their AI deployment is the same about the United States so if we start having more laws on AI and this will block innovation and where do you see what is your opinion on this like on that one hand we want to have a strong industry and other hand a I which is trustworthy and responsible I know it's very frustrating but also we have to acknowledge that in every challenge there is an opportunity and we need to use our creativity to create opportunities in the current situation currently they say

that in offices we cannot install cameras it's not possible to make recordings of the people where people can be identified and know what they are doing that's not suitable at the moment and I was looking for a smart office application where we would need cameras to know exactly what people are doing so we needed to change the sensor and instead of camera we need to use another sensor which gives response when there is a person but we cannot identify who it is so we need to create a new measures of these sensors and create new algorithms and create some intelligent innovative ways to identify number of people and their activity inside of this office without drilling the gdpr rules there is always an opportunity and it could if it works robustly that could be a new business opportunity ok I feel good questions by Claudia so at the end he asked for instance if any I is much better in identifying cats than a human yes but it uses different features than we used to

so in explaining it in terms we don't understand exhale trust the system which is better than us but we don't understand that it is better than us the validation of something better than us is really difficult so it he wants to validate that this AG ice performance is better than the general public's decision-making it's really hard to say we can of course use a well Dacian data and see it's doing as much as human but we cannot say it's doing better than human maybe in that case we can use a just the cancer example give a label data from different doctors and see the AGI is that he mentioned is still doing when the data set is labeled by different different people and still it's doing very good and if we achieved that done it's an opportunity to learn back that the if the AI explains us how it made the decision then we know how to judge a new cancer picture better than we did before I mean I agreed let's stay with the

Ken example for a second so the AI says this is a cat I agree but it says it's the cat occurs it moves faster jump more it uses a term like acceleration or something I can't measure myself so my capabilities as a human limited to understand the reasoning by nei yeah and so how can I trust the system part is doing validation and could be like so many cats images knew different cats images could be seen and if it's still able to give a good decision or not but it's hard to say if if it's like doing as much as human or can or better than human I think it has to be tested in very long-term to see if it's given good response or not one thing that I want to mention is something that we don't know when human is making decision sometimes we are using other features then we explain it but listen we call them intuition yes and for instance doctors when they classify the cancer they can explain why they

selected its cancer stage 3 like because this looks red because this looks sharp edge and bla bla they can explain their decision but there might be other features that they grep and we don't know exactly what and they are even not avert themselves so that might be the intuition of the doctor developed and by looking at other features what could be maybe the person is walking slowly or I don't know look sleepy the eyes are may may be looking like sleepy maybe the doctor is also considering some other outside features when it's he is making those decision maybe he recognized that the person has less energy than how he should be in normal okay so I'm looking the time because I know you have to have to give a lecture and you have to continue so I like to summarize first that is a very complex topic and we just briefly touched it a little bit of it and I know I have to talk way more with your barrel for this it's song I

would like to read out for a second the results of the polls today because they indication - so a police question one should AIB regulated and they have 50% said yes and the other 50% it depends so it means everyone says regulation is needed it just needs to be classified a little bit more and the other question I asked is who is responsible for the impact of the AI that's what the AI is doing and the majority said the company the owner company so they are responsible for what the AI is doing interestingly 12% still said the government and no one said the I itself because I might you is that one day I will you know this entire life ottomans organization will be self-reliant self like like a game bhai will be like an entity and the last poll asked is who should define the bell use the i should act on and 36 percent said the company using dei so the client at the end so I purchased the AI and I can

calibrate how they I will act and forty five percent should said it's an expert group nominated by a government so we trust interestingly only nine percent said Society so we do not trust in this poll society to make this decision we may be trust expert group higher may be expert like you like me in other fields so barrel thank you very much for being here today thank you so much or answering the questions on this poll and also asking your questions to me and very nice to talk to you very and so many thanks for your time everyone yes if you still have a couple of minutes you can hang out in the chat to answer of one of the unanswered course yeah I will try I'll say goodbye to see thank you yeah thank you bye-bye and so for everyone else we have next week another retro Chet I tried to do it in a weekly algorithm so next week we have made antics so we have the two founders of Murr antics which

is decimals and avian one is a very successful serial entrepreneur the other one is a researcher and it will be very different than today today was a great conversation and by the way thank you for the questions really good questions and next time we will speak about business models and AI driven business month so at the end how can you make money with AI my name is Fabian Vesta Heidi I think you for attending today I thank my team in the background hanging Network hopefully I will see you next week again it was a pleasure as usual good bye

Rise of AI 2027 · Berlin · May 2027

Be in the room for the next one.

Ten years of talks started as conversations in one curated room in Berlin. The next edition brings 300 selected leaders together in May 2027.