and now we come to at least to our final speaker and um i'm really happy in the beginning i i mentioned um that i think everything can be put under the umbrella of uh trust in ai and that we just talked about so many topics around ai and leads to a better understanding and i think this is the the yeah base for trust and now i have mathematican here who is who has the focus in statistics and optimizations and at the age of 22 he founded neurocat and the company is at the forefront of quality and compliance in the field of ai i have florence gressner here and welcome to the stage and a warm applause all right thanks hi uh thanks for the introduction um my name is florence i'm co-founder and ceo of neurocat and for nearly five years now we specialized in evaluating and mitigating risk for safety and security critical ai systems and in over 25 projects with customers together we implemented some testing strategies and enabled them enabled their mission-critical
use cases like autonomous driving and push these topics forward and so we want to enable trust in ai but it's not that we are afraid that there will be some singularity or that they will replace humanity at some point we just believe we can unleash the full potential of ai when we are providing tests and when the society can trust it and we can put it to the next level and enable more use cases also in security and safety critical domains and this is what we're trying to push forward we want to push the limits of ai and unleash its full potential and to enable that we want to empower these pioneers who want to bring the new use cases into mission-critical ai systems however now the classical security critical and safety critical domains now get extended there will come some regulations that are also will be decided on this year this is the AI Act for example and they published a taxonomy where they described what use case has risk and how much risk
has it um and this is defined for the entire um european union and they are like um unacceptable risks like something that is against the values of the european um person there are high risk applications and lower risk applications that are rather going in the entertainment domain or in the chatbot domain where some transparency is necessary but it does not provide a lot of risk to any person and just to name the high-risk applications because these now get very much extended and are the critical infrastructures so everything related to oil gas water and transportation in general um also um scoring systems that enable you to get a job or a credits or decide on that um are high risk applications and law enforcement in general so looking at the proof and seeing if it is sufficient not to prove but looking at evidence and looking if it's enough for law enforcement border control and biometrics and public services in general and everything that was safety critical before they act is also published so it's publicly
available the draft at least and for all these use cases for the high risk use cases there are no requirements set that are needed to be followed by the ones who are in these exact use cases and i just scratched out the one that is like unreasonable risk and the ones who don't have to beat but for all these who have uh high risk application and going in these use cases and have to follow these requirements they have to test the risk in a systematic manner and for every use case and for every relevant scenario they have to evaluate the risk and later in this talk i will want to show you how it's done you have to have a strategy and to process how to mitigate risk if you find a defect how can you overcome this and provide transparency and interpretability through explanations in the system itself and some other topics that i just want to name to be a complete but i don't want to focus on in this talk is a
whole gate data governance around it explaining why the data is sufficient and so on a documentation of every architecture and logging every event that it's um pushing through and implementing human oversight that humans can always intervene into these high-risk use cases but like i mentioned i just want to focus on the testing and mitigating risk because this is what we focus on the last five years because we were specializing on safety critical and security critical use cases so basically um with our customers together we created test strategies and implemented them for them and these already included like okay first you have to define some quality requirements what do you want to achieve how good the system has to be in the relevant environments but then you come to the point where you very quickly have to define a test approach where you describe how you can measure risk and how you can do a defect react that is nothing else than risk mitigation and up to the last point of the automation strategy so if
your tests are not automated then eventually they get just by budget cuts um scratched out and you're not concerned with it so automation is always a very important thing especially in ai so starting with a quality requirement what we are doing is first define the statistical confidence that is necessary um so in the automotive domain it has to be some ninth after the 99 of confidence in order for a system a safety critical system to be deployed defining this will give you just some insights how much test data you have to use and you have to select the metrics the risk metrics and the measures you want to follow and want to measure your systems against and define the relevant scenario your system should work under so should it work under rain under noisy conditions or under even dirty data and real world applications and setting these qualities thresholds based on the metrics that you have to exceed this is basically also everything what we are doing here is mentioned in the AI Act and
you have to select these things we have to set probabilistic thresholds and you have to follow these and these rules will become mandatory for those relevant um use cases so second one is the testing approach and here i just want to make a key difference to you normal ai application and risky ai applications because the way you measure the performance and measure the success of it is pretty much different and i really hope i don't lose more audience because there will be some formulas but i try to make it very very easy on that so basically you will have some data and we already realized there is some relevant data so you have you defined relevant conditions and relevant scenarios your system should work in so i just for the easiest to make it easy easier just distinguish between two scenarios you have clean data in your training set and you have some dirty data that you will experience in the field when you deploy your system and basically the probability that these data points
you are exposed to them like these dirty data points or these edge cases real edge cases probability might be very very low and you clean data you experience like um like every day you've experienced in your training set so and here is also the key point uh where this is a problem for high risk applications so you defined your performance metric because you had to do it in the quality requirements and then at some point in order to measure the performance you just apply this performance metric on every single data point so what's the problem here for high risk applications we see the dirty data is basically like this pink data point the probability is very low so for a performance metric it is not important and so if you optimize for this it is not important to capture this little tiny data point that is very improbable but probably it's a relevant condition or relevant scenario you have to follow so now what is the contrary to risk so basically risk is severity times
exposure so how severe is something so how much damage it can create and how probable it is that it will um happen in the real world so if we are looking at this one point it might be very improbable but if it's a very severe scenario and a very severe conditions for example a pedestrian on the street you might want to capture this and you might want to mitigate this risk so performance measures are not sufficient to estimate risk at this point so um basically just in comparison it is uh contradicts itself the risk and performance basically contradicts itself because performance cannot cover um all these points because rare scenarios and edge cases um are just thrown another bus so oops so what is important and what we pushed our customers towards is um decoupling all the relevant scenarios and looking at every scenario and separately and this requires you to have a significant amount of data for every relevant scenario and this leads you to a point that you are first able to measure
like the severity times exposure what the definition of risk is and also in the end you are able to push your decision boundary much much forward that you're having a significant amount of data in all relevant scenarios and you are not um like all these points are very important to you because severity can um like these every data point can have severe consequences um but what i didn't tell you is how do you get to these data points and how do these look like for ai applications so basically the hazardous threat of ai system is there's a huge difference between the training data and the real world so in the training data data might be very much clean and in the real world some noise might appear like rain weather conditions or camera noise that can just spontaneously appear in in the field basically and um the second point and just one point is not only for automotive also for nlp systems only a strap of characters also these kind of corruptions these kind of
noise that make data dirty that you have to be concerned with if it's safety critical the second point is the first one was more safety related now also security related if there is an attacker that wants to tamper with your system or seize any kind of advantages the attacker in an ai system is very much freely to do so because there is an active community of adversarial threats out there that get published very frequently that tell you how to put very imperceptible changes to your data point and then tamper with a model and manipulate its decision for upload filters this would be very much severe because you can just have some inappropriate content put some imperceptible noise over it upload it and you will trick any ai system that might lead to it so these are just how these data points are looking like but how do we get there how can we expand our data set and augment our data set such that these also get represented in a way that the the model
itself have to treat them as relevant data points and these happen with test methods so for the first part for the safety critical part and corruptions are test methods that are just augmenting data for relevant scenarios so they put basically on the clean data set some overlays like weather conditions like snow rain or dirt on the lens or a normal camera noise on it you can extend your data set and can you can use it for testing and the second one is the adversarial threats this is a very attacks this is a very active community they are published very frequently and this is basically a cat and mouse game between attacker and defender and in the end the deployer of a system just has to decide what kind of budget an attacker has to exceed in order to tamper with the model and how much computation it has to go into that so this is the testing approach basically this were the test methods and why performance metrics are not sufficient in order to measure
risk especially high risk and the third thing is defect triage and so you have to have a systematic approach how you if you find a defect and if you find a scenario failing how are you dealing with it in the end so first i mean we realized with vanilla performance metrics you're not able to detect any defects because the defects are not there for very rare edge cases and so first you have to test against these relevant scenarios you have to decouple them and test against single every single one of them that you defined before then you evaluate if your quality requirements are met and for every of these relevant scenarios and you minimize the risk by if a quality requirement is not satisfied for one scenario you have to feedback the data that was not doing well into it and optimize the risk measure instead of just the performance measure and in the end you need to update the architecture and the procedures in order to to do so so this is a very
complicated part and we are not concerned with that we are only concerned with the testing and providing the test methods so this is in the creativity of the developers of the i system itself but we provide some metric that you can optimize against and then test again and see if you are fulfilling all the risk requirements and um like this sounds like a very easy pro process and very straightforward but this is also not just an abstract thought so um i don't want to scare more audience here but basically a a very profound this is a really it's not an abstract thought it's a really concrete optimization problem one can follow i don't want to go too much into detail about this how to select these architectures but it is possible and you can optimize for risk instead of just optimizing for performance so now the automation strategy i already mentioned that it's a very crucial and important part and if you're outsourcing anything to humans you might experience some budget cuts and then what
happens which kind of tests do you want to execute and which of them not so automation is very crucial and important especially for very automated development workflow of ai so the ml ops workflow that is very commonly used for ai development follows a quite standardized process so in the beginning you're having some data processing annotating the data cleaning out the data and after them you are selecting your architecture and your training procedures and your hyper parameters such that you can just train your model and hope that you will find one that is good enough to satisfy your quality requirements there comes a evaluation part or the validation part in the ml ops life cycles that are also out there and these platforms it's heavily concentrated on performance metrics but performance metrics do not capture high-risk applications and if you are satisfied with the model you can put it into this into the deployment and we from our part we put all our experience in creating testing strategies and to our customers and enabling mission critical
use cases into our software platform we call it 8-kit and 8-kit basically contains all these relevant test methods where you can test against these different threads it provides explainability methods and is pretty much very much automated and such that you can just implement it into ml ops workflow and measure risk reliably so we call it a kit until you make it the game in order to minimize risk so it's easy to integrate and you can use it in your automated workflow because all the methods are automated and highly scalable it is your robustness and explainability expert that selects your test methods for you and keeps you up to date with the standards and the risks that you're experiencing and what i mentioned before basically you're able to specify your quality requirements your relevant scenarios and just put them in and define the pipeline you want to execute in your automated workflow select the scenarios if it's some lens flare effects some lighting effects or some weather condition or sensor noise and we will provide you
with an actionable report with the metrics that will help you to also estimate risks for your high risk applications and if you find you're not sufficiently good and your your risk is not mitigated enough all the data we produced while testing all these corruptions and all these adversarial attacks can be feedback into your training process and you can targetly improve the risk and minimize risk with that and makes us all in a very automated workflow feedback the data that doesn't work retrain on it and and hope you will make it at some point so what did we see so basically there will be regulations that define us with new um that define new safety critical use cases um we saw them they are in a taxonomy and also requirements for them are satisfied but we can take a lot of things we learned in the past of safety critical components and use cases because we can just reuse everything that you have to have in a test strategy so you have to test the risk
evaluated for every relevant condition and scenario you have to mitigate the risk and have a systematic approach towards that and third you need to provide transparency and interpretability i was not focusing my presentation here but this is also something we are providing and equipping you with and this will be able to satisfy the AI Act the air act will be decided on this year then it will go into a triage and into a um one year of testing phase and then it will become mandatory so in not more than two years we will have mandatory requirements for these critical use cases and yeah getting acquainted with the topic of high-risk applications if you are in this is very crucial also starting now that you're not surprised when the regulation comes and this was basically my take on enabling trust in ai and how to get these abstract regulations into concrete and test strategies and i'm very interested to learn what's your testing strategy and are you maybe in a high-risk application domain thanks a lot
for the intention and i'm looking forward to questions yes thank you florence really well done i come to you um so i have one question um how do you evaluate the risk in a strong ai i think this is always a key word for everyone and still not really existing but um yeah have you thought about that can you tell me what you mean by a strong strong ai like ai that is maybe you you can do it better than me but uh like a developing in an intelligence um independent from humans um like autonomous driving seems like that yeah yeah autonomous um yeah like um like a for general ai if it's in a safety critical domain so first of all i don't believe that we are very close to it yes um we looked at a lot of use cases and we were always able to find a failure in these systems so there's not these um these intelligence insights that we need to compare to human intelligence so i believe we're a
bit further away but i believe if it would exist we would still be able to um manipulate the system and optimize for noise but just this noise must then look to us like um like uh like will also confuse us then at the point like if it's the ai is good enough and we find noise to it i believe it can only be real general ai and real human-like if it already also would confuse us then yeah um and another one um you were talking about like testing the risk and at least you also mentioned transparency i think that when you tested it it's also important to to make create a transparency to the audience right and i think most of the people they are still afraid of ai and i also always handle with that um but what do you think why are they and still so afraid is it that they don't understand it um and how can we change it yeah i believe the society is afraid of ai because there were
so many use cases failing in the past i mean there's a website called tesladeath.org.com and where there's a huge list of people dying from the autopilot because there was not sufficient testing being done there are a lot of cases chat boss getting racist when they're exposed to to humanity so i believe we prove society that we have to be afraid of it and if we don't test it before and then society figures out how bad it is then how should they trust it if we can not trust it by ourselves so i believe and there are a lot of examples that showcase why we don't trust it and i believe only was profound testing before you deploy it you can really generate trust in society yes i think we also had some questions you uh okay start where you want um two questions first the whole testing thing i know it's coming from the european union but we germans like this you know validation testing having tuff temple on top on this um so what
is your feeling will other europeans pick this off will we set standards you know like will be be there the schreiber as usual um so the german versus the rest because it's international and then the second is what is like your business model in terms of my first thought is all the consultancies out there they are now having great business integrating your solutions and others because they they know the process in the companies they know where to tackle it they have to go in um so i'm trying to figure out who will do all this testing which we all under underestimate because every piece of software now needs at least reconsidered if it's falling under this yeah um so the first question i thought was two questions the first question is who will elite this and i believe we have a huge problem we have certification institutes and they made a lot of money by just stamping um like cars and so on and providing a seal of approval and they would have a really
hard time rethinking their business model because now there will be just a few persons a bunch of programmers that sits there and use these tools and we are seeing that these certification institutes have a very hard time if it's tiff um or diegra to rethink that and there are platforms like microsoft and again outside of europe that also are able to provide the same thing and so maybe we will even lose this game if we don't wake up and able to rethink everything and the second question was in the direction of who will execute the testing in the end so at least what i'm hoping to do and also generating partnerships with is like of course the consultancies will be able to to audit and test these requirements and um certification institutes if they wake up and um but like i mean just i want to mention one thing like there are some um innovative um products also in certification institutes as a tooth it for example and creates also some smaller standards that one
can use and specify the requirement with and but they all need a tool and we are trying to put our tools into all of these our tool into all of these things and people like yeah it will be an independent party that has to measure this but it has to be an independent provider who generates the tools so basically um it's it's twofold answer maybe to this question one quick question you had one slide uh where you introduced the different areas where uh eu regulations are coming up and i was missing health care um as a health care health care anything around health care i was wondering if is that due to this already being highly regulated or um what's your perspective i didn't really mention this but like in the end there was also like one bullet point called safety uh safety components and everything that was safety critical before is still safety critical um afterwards so i didn't explicitly mention it in this section but um i i would see it and at
least how i would interpret it i would see it in this corner and in this category okay um i got my coming out i'm german and i'm a consultant for ai so i just wanted to know and because of the certification when do you think there will be a kind of deen or easel certification for trustable and no risk ai yeah this is uh yeah i mean first of all there must be regulation before you can provide this seal of approval of course you can also provide some seals that are some kind of specification where you say okay these are my quality requirements and um independent institute tested it against it so these seals can come before but i believe here only when the regulation is there and when we look at the AI Act maybe there will be some um standards that will exceed or like be faster than this but i believe at the day when it becomes mandatory and even it's a transition period before there will be products and there will
be seals that you will be able to to apply so many questions really cool so you will still be here right and the others hopefully they are also here um and i'm happy to connect later um so thank you for your presentation florence thank you very much you