um because we now spoke a lot about how we could build a vc fund based on ai um how we can do ai trading and yesterday we already touched a topic around regulation what is really what is ai really doing do we really understand what's happening there um and this box this black box of ai can we actually unbundle it and i'm very happy that professor dr beryl i have to read the story sir machek i hope i pronounced it well from the yucapang university in sweden is here with us she has done tremendous work in all different fields industrial fields but also machine learning ai statistics and she will help us shed some light on this topic i hope you're ready and welcome and let's start thank you very much for your nice welcome and you pronounce my name perfectly thank you for that and good afternoon everyone it's a great pleasure for me to meet you at this virtual event surprise of ai in 2020 i hope it's real event again next year
to meet each other i'm berin i'm a scientist working on computer vision and ai as i'm very nicely introduced thank you very much for that i'm pursuing my academic career but besides i'm developing robust algorithms for industry as well in both academic work and industrial work it's extremely important to know what we are doing with our ai or machine learning models and to explain whether we can trust on decisions of these models or not when we are using them either in scientific work or in industrial work therefore we have to develop trustworthy ai and in other words we have to know how we can develop ai that we can trust this is going to be my topic today and i would like to have a short discussion with you in my short time to address these topics on our human needs why we we see poor trust uh in all areas of our lives and our daily needs but are also industrial business needs of trust and how can we develop trustworthy ai or is
it possible at all to develop trustworthy ai and what kind of mathematical tools do we have to uh see whether we can trust on our model or not and finally what's coming next i would like to have a discussion with you a little bit brainstorming including my ideas and imagination what could be waiting for us in the field of trustworthy ai we human beings are social creatures we do not only have physical needs like food and drinks water and we also need uh other uh emo we also have other needs uh to survive and abraham must know in 1943 he's written a paper called a theory of human motivation and he suggested a very well-known muslims pyramid and in this pyramid he described human needs for our survival the largest layer in the bottom corresponds to the uh layer which we need most for survival and then their importance are reduced uh when you go upper when the slides are getting smaller and as you see here after food and water our humans basic survival
needs are safety security and which we can also summarize as trust assume that a baby who is born uh who can find food and water but not in an environment that he or she can trust and feel safe baby will die and the adults are not different than a small baby but we are just in little bit bigger bodies but our human needs are just the same like a baby beyond uh our physical needs we need to trust in all our areas of our lives not only close relationships but also our businesses when we are collaborating working together under the same roof and assume that you're a developer and you want to sell a model or deploy it on a cloud and make money with it so other people can use and pay you monthly with subscription fee for instance this model gives a result is it safe to use is it trustworthy can other people really rely on its decision what it says and uh do they really get what they pay for uh
and does it also work ethically in ethical rules so these kind of questions opened up as the field of trustworthy ai so ai is we know um already there i mean we are already used so i've got star for one good news and one bad news good news is that ai is already in all areas of our lives and the bad news is that ai is already in all areas of our lives so assume that you applied for a credit card and it's rejected this comes from a result of a ai algorithm and you want to know why who is going to answer that question and who is going to explain how this model is given the decision is it really ethically rejected or not assume that you are in a doctor office and doctor analyzed your ct scans blood measurements with an ai algorithm which is already being used in hospitals and the doctor says only way to cure you is to have a surgery can you really trust you will ask why how
this decision is taken somebody needs to explain us how these models are working in order to trust and let's say you made a job application you've seen some other speakers in the beginning of the day i heard job applications are these days are also going through ai models because there are so many applications to scan and select a good candidate and that decision is taken and let's say you are not accepted to the position and you want to know why so the question today is uh can we really trust if ai is making decision for us ai is really regulating all ways of our lives i will start with some examples of how these models are affecting our lives and then i will show you some tools that either developers or the business owners or people who want to know how ai is working can use to know whether ai could be trustworthy or not i start with a story that's latonya sweeney um she's a nice lady as you see she heard an exciting
uh news from her colleagues her colleagues said when we search your name on search engines we only get some advertisement at site uh if latanya is arrested get the truth about latanya is latanya is uh trustworthy and they really want to know what's uh in her story bringing her uh these advertisements and latonya was a data scientist she was very curious about what's going on with it and she wasn't only a scientist but she was also a harvard university professor in this area so she decided to make a a good analysis on how these engines are working and she found out that uh her name is classified uh in a black minority group which are also in the past associated with some uh events uh which ended up with harassment and they therefore these engines are bringing up results like is latin is arrested or not but not for the other colleagues so can we really rely on such a model to deploy somewhere and to use is it ethical to use another issue happened
at a police office when they used uh pictures of the people to decide whether they've got risk to do further crime or not and this ai model stayed high risk for bernard this is a real story and a low risk for dylan uh and they let uh both of them go and d long three days later ended up in police station again but bernard had in no issues later so how this decision model is working on is it looking at long hair tattoos or is it really looking on some facial expressions it has to explain us what it is looking at what is this focusing it's to rely on the results at the ends maybe you heard that examples uh people added some little noise on pictures epic classification didn't work it ended up classifying it as something else again a panda is classified like another animal just a little bit noise is added even human eye cannot distinguish the noise but uh this changes the decision assume that you you're using this computer vision
algorithm on a self-driving car and suddenly the sensor became a little bit dirty and it cannot give the right decision like any one more these were bad working examples let's see a nice working example but still needs explanation somebody developed a model of discrimination of wolf and husky and at the end they saw that the model was working with 99.9 percent accuracy and they were really surprised both and husky look like each other so much they shouldn't give very high accuracy why it's working very nicely so they also even if the result is very good we still have a question whether we can trust or not so high performance of an algorithm doesn't mean that we can hide the trust because we still need explanation of why we can't trust on this high performance therefore there has been many studies of explainable ai but as xai we are hearing it more often in last two three years maybe before that it was mostly called interpretable artificial intelligence algorithms and it's been searched for almost 10
years at the moment the idea is we need some mathematical tools let's say we've got some training examples then we trained our ai model then a new data came uh from a real environment and the model which is trained by training data gets new example gives a decision and what xai is trying to do to answer why this decision is taken then we go to a next step the next step is to explain this why depending on the audience so who is asking the question of why if it's a developer who is asking why maybe we need to explain it uh more theoretically about layers what's going on in these layers so is there a bias issue maybe you need to retrain it this way but if it's an audience who is the end user somebody waiting for a bank card maybe we have to explain the decision making process differently either by simplifying the neural network or making more examples from how similar examples in the training set looked like therefore the decision is
taken so explainable ai algorithms chosen choosing process is really dependent on who is asking the question of why who is the audience to whom we need to explain this why and when we want to explain why we will we have developed mathematical tools to uh decide whether we can trust or not well maybe we can trust by looking at this we try to uh get uh ideas about training data set about a model itself and about the input where we are looking at in this input when they are making our decision in order to explain our model little bits a little bit to open up the data set i'll just give an example of uh my own story i bring my dog to a vet and then uh vets says i have to make this vaccine to your dog and then i ask why and then she says for instance because every other dog has this vaccine so and then i have an idea whether i can trust on this answer or not if she
explained me differently i got this this uh blood report and then this shows me that this has to be cured with this vaccine maybe if it's explained differently i can't trust so uh this how it is explained to us also brings us to the point of trust whether we can trust on explanation or not and this is never a mathematical thing i mean uh xai methods are explaining training data the model itself where we are looking at input uh they are explaining this mathematically but there is no uh quantitative value to uh discuss trust so the answer that the vet has been given might be less trustworthy for me but maybe more trustworthy for other person there's no quantitative issue about a trust this is the end user the audience must discuss nevertheless we are opening the black box a little bit we are bringing little bit transparency by our x ai methods so opening up the transparency is possible by using some interpretability approaches in the mathematical mathematics and choosing the approach as we
said depends on the audience who is asking the question who is the end user we have to pick one method of explainability depending on the audience and depending on how much transparency is needed is is does need to be high level or does it need to be very detailed maybe it's a developer then we need to explain very very detailedly and what problems we are focused on what mathematical methods could be used in this development environment for instance are possible to be used in this environment and whether the methods of explainability are used during the training process or after maybe model is pre-trained you cut it but you have to explain it afterwards uh so depending on which of these problems are occurring we have to pick an xai algorithm and then the xai algorithms are also classified as whether they explain the entire model uh or not and whether they can split all the neural network layers and two components and explain or not and whether they are focusing on the uh how the
algorithm is trained or how the algorithm is functioning in real life or not i listed some mathematical methods which are frequently used but it's impossible to explain all of them right now if you are interested if you are a developer please pick the slides and you search them further i will just give you some examples one of the most used example is shop or shapely features which list the feature importance then we can decide uh whether the algorithm when it's decision making looking at the right thing or not or what it is looking at we've got a titanic dataset classification example where uh we listed uh we classified whether the passenger died or not then we looked at the features of the passengers and listed their importance in the decision making whether he or she died or not and then we saw that mostly females survived and the males died so we saw that in all this cabin number ticket price age of the person all these features are looked but which one which feature
is the most important in that it's the gender whether it's female or not so that explains a little bit how the decision is made even if it's very high dimensional data set and this is also applied pixel level to images to explain which pixels are contributing to the decision-making for instance if the dog is classified as a dog which pixels contributed to this decision making process when this decision making process in pixar level is looked uh to the husky and wolf example that we looked in the beginning of the presentation they saw that the ai which was working with 99.9 percent performance was looking at the background and if it sees snow it was classifying it as a wolf if it sees no snow it was classifying the animal as husky so that's that's how it was working with very high performance because all the both uh examples were coming with snow but it wasn't looking at the animal at all when it's destroyed making so high performance those doesn't mean that the ai model
is working well we have to use xai methods to know whether we can trust or not so ai can tell us xai methods can tell us a lot about whether we can trust or not whether there is a bias or ethical issues in this algorithm or not and have to fix it because if we know how the decision is made where it's looking at we can also fix it later so satisfied customer satisfied developer hopefully what is coming next these are just my ideas but i'm just imagining that this xai process will be real time some someday it will be like the ai model will explain to user real time why the decision is taken and the user will give feedback i don't trust how you give this decision maybe you didn't look at the writing and then ai maybe retrains itself at that way or looks at other features and then improves the results maybe gives a new decision and explains again i'm assuming that this loop will be real time in the future
maybe getting feedback by uh verbally and improving the model real time and what we've seen to wrap it up uh simple model doesn't mean that uh it's low performance or more complex model doesn't mean that it's high performance and higher accuracy doesn't mean that we can trust more we still need to explain why this high accuracy is taken and xai at the end comes with both advantages and disadvantages it might also end up with some security leaks of course if we explain something which need doesn't need to be explained uh at that point so that's all for my presentation and thank you very much i hope you reach me on website on twitter on my newsletter youtube and very nice to meet you here thank you thanks a lot berry very much on time i can see your researcher and i guess we will have some questions from the audience but maybe let me start by first question what do you think will be the main drivers in the future for understanding or the the
need to understand ai is it the regulation that you see or is it really more understanding the outcome of the models better training the models better maybe creating better products out of ai models creating better products i would say because in ai the at this moment we've got generalization problem we are training algorithms with a data set then we want to sell the product to somewhere else where it sees totally different data set that we even do not imagine and we expect the algorithm to work then if it's not working we need to explain why what it was looking at why it was working good in this place and not there let's assume that a self-driving car uh which is developed with uh within germany with all examples taken in germany all houses and streets learned for germany and you sold the car to you know the uh place just in the netherlands i mean neighbor country i tell you that it will not work with the same accuracy because our buildings are all red
color and all uh different kind of cycling roads and it will not see the same data set so it's important to know by developers when we are making decision how we are making decisions what it is looking at and is it trustworthy or not yeah i once learned from tesla that they roll out i think their models in several areas at several times because exactly what you just said our friend fabian has a question he might be in the room it seems like he wants to start with an explainability test and he wants to understand what technical expertise does he need for that maybe he should be a student of you well it depends on the audience we said if the audience is a person who wants to know how the algorithm is making decision we can use uh methods which either simplifies the decision-making process instead of showing the 100 layers of neural network it can be shown by a simple decision tree i looked at this then i looked at this then i
looked at this and this decision is taken just to simplify the model we can use such metals or just like the future importance which features very important in this decision making process for instance you applied for a job and you are rejected maybe somebody can use this feature important test and say that your publications were not enough therefore you are rejected uh maybe the future importance in that case could be helpful but if you're a developer then you've got a different story you have to maybe explain in deeper level levels maybe you have to really see if it's finding a wolf if it's really looking at the animal if it's really looking at the fur uh not the background then how can i rearrange my training sets so it can learn that even if there is no snow on the background so it still recognizes the animal maybe you have to do some tricks in your training process so depending on the audience the methods and explaining ways could be different but the purpose is
to communicate with anybody if it's someone who doesn't know any programming we still need to be able to communicate with xai tools this was what we were looking at when we are making this decision he might call you tomorrow be careful um there is a question uh from eberhaj nebel sorry um is actually also from fabian but imaginable had a question to that um how can regulation be technically implemented more like a tiff or a gfpr self-responsibility or like an inspection standard tough question and i believe it's still open question for all europe at the moment we've got some uh local tears and teams national teams and also teams collaborate in netherlands and germany all also collaborating within all europe to try to decide uh what are the ethical rules and uh how to regulate those rules and how to describe even the word trust how do you describe trust it's uh too much uh personal uh and these are not mathematically described values so we are trying and still trying to define this i
think it's also interesting if you look at the level we think the ai should work we discussed it for accounting automation and we had an ai that was much better than the human being but the mistakes were different so you are better than a human being but your mistakes are weird and i guess the same can happen with autonomous driving so actually the car is much better but then it runs over a child so obviously that's something we don't want so how do you define that well um and when you um when you look more at the at the training of a model for explainable ai is it is it then possible from your point of view because that's something my startups really care for to say hey the model is not working we have bad results we can't build a good product we are missing this specific type of data we are missing this specific type of i don't know feature engineering approach and then we can make it happen so do we actually
get predictable models definitely this xl story started with the um military in in america uh so darpa project in in united states has started xai developments because they were trying to classify camouflage tanks with a vision algorithm and then they ended up with a algorithm to classify camouflage tanks with 99.9 performance again it was almost working perfectly and then they asked the question how this can work this well i mean even the human eye cannot see this camouflage tank there how the ai can see it this is how aix ai story started in the beginning then they found out that in the ai they found out that in the training data set all the camouflage tank examples were taken in cloudy scenes and all the rest of the images were taken in bright scenes and the ai model was looking whether there is cloud or not so it was nothing related to a camouflage tank again and this is just like a wolf and husky story but this is how the xii developments have started
and then they figured out we don't trust on uh performance numbers it might say 99.9 performance but we also need something else maybe we need less performance but we need some explanation uh then we can decide whether we can trust on it right now or not if the self-driving car is telling us now i'm driving and giving the decisions uh myself but at this moment it's getting dark and i don't trust my decisions a lot maybe we can drive a little bit more carefully and shared the attention with the with the car but if you've got some values about how much the trust car has on itself at that moment maybe we feel a little bit more relaxed to change the radio and a little bit uh become uh drive more relaxed in that point yeah getting here in a traffic jam i also felt like self-driving cars would be amazing so i can just relax and don't look at the traffic thanks a lot beryl for your time thanks for all your insights and
um maybe maybe i i posted your your website in the chat maybe some people can ask some questions and reach out to you thank you very much very nice to speak to you and hope to be there again later bye great