Rise of AI Conference 2022 · Berlin · Applied AI Stage

Trust in AI: Is Your AI Compliant and Explainable?

Florens Gressner · neurocat

At Rise of AI 2022 in Berlin, Florens Gressner, co-founder and CEO of neurocat, argues that trust in AI has to be built through systematic risk testing, not assumed. Speaking on the Applied AI Stage, he explains that neurocat has spent nearly five years evaluating and mitigating risk for safety- and security-critical AI systems, running more than 25 customer projects including autonomous driving use cases. He walks through the European Union's then-pending AI Act, which classifies AI applications by risk and imposes testing, risk-mitigation and transparency requirements on high-risk categories such as critical infrastructure, credit and job scoring, law enforcement and biometrics. His central technical argument is that ordinary performance metrics fail high-risk AI systems because they average over all data and dilute rare but severe edge cases; he defines risk instead as severity times exposure and argues each relevant scenario, including corrupted or adversarially manipulated data, must be tested separately. He describes neurocat's automated testing platform, built to integrate into machine-learning pipelines, and closes by predicting the AI Act will become mandatory within about two years and answering audience questions on strong AI, public trust, certification and healthcare.

Key takeaways

  • Gressner defines AI risk as severity times exposure, arguing rare but severe edge cases matter even when performance metrics show them as negligible.
  • The pending EU AI Act classifies AI use cases by risk and imposes testing and transparency duties on high-risk categories.
  • High-risk categories he lists include critical infrastructure, credit and job scoring, law enforcement, border control, biometrics and public services.
  • neurocat tests AI systems against corrupted data (weather, sensor noise) and adversarial attacks separately, rather than relying on one aggregate performance score.
  • Gressner argues automated testing is essential because manual or non-automated tests tend to be cut when budgets tighten.

Detailed notes

neurocat and the coming AI Act

Gressner introduces neurocat, which he co-founded at age 22, as a company that has spent nearly five years evaluating and mitigating risk for safety- and security-critical AI systems, running more than 25 customer projects including autonomous driving. He frames the goal not as fear of a singularity but as unlocking AI's potential in safety- and security-critical domains once society can trust it. He then describes the European Union's AI Act, to be decided that year, which classifies AI use cases by risk: unacceptable risk, high risk, and lower-risk cases such as entertainment or chatbots that mainly need transparency. High-risk categories include critical infrastructure such as oil, gas, water and transportation, scoring systems for credit or employment, law enforcement, border control and biometrics, and public services, alongside anything already treated as safety-critical. For these, the Act requires systematic risk testing for every relevant scenario, a strategy for mitigating defects, and transparency and interpretability through explanations, plus data governance, documentation and human oversight, which he mentions but does not cover in the talk.

Why performance metrics can't measure risk

Gressner distinguishes performance from risk: a performance metric applied across all data points treats a rare, 'dirty' edge case, such as unusual weather or sensor noise, as unimportant simply because it is improbable, even if that scenario is a pedestrian on the street. He defines risk as severity times exposure, meaning how much damage a scenario could cause and how likely it is to occur, and argues performance and risk can point in opposite directions because rare, severe scenarios get averaged away in ordinary performance measures. His proposed fix is to decouple relevant scenarios and test each one separately with sufficient data, rather than scoring a system on one aggregate metric.

Sources of risk: corrupted data and adversarial attacks

He describes two ways real-world data diverges from clean training data. Safety-related corruptions include weather conditions, camera noise or corrupted character strings in NLP systems, which can be simulated by overlaying clean data with effects such as snow, rain or dirt to expand test sets. Security-related adversarial threats come from an active research community that regularly publishes imperceptible changes to data capable of manipulating a model's decisions, for example to trick content-upload filters; Gressner calls this a continuous 'cat and mouse game' between attackers and defenders in which a deployer must decide how much attacker effort the system should be designed to resist.

Defect triage and automated testing

For handling failures, Gressner describes testing each relevant scenario individually against defined quality requirements, feeding back data from failing scenarios into retraining to optimize for risk rather than only performance, and updating architecture and procedures as needed, though he notes neurocat's own focus is on providing test methods rather than fixing architectures. He stresses that automation is essential because non-automated tests tend to be cut under budget pressure, and situates this within a standard MLOps workflow, from data processing and model training to an evaluation stage that today is heavily weighted toward performance metrics that miss high-risk failure modes. He describes neurocat's testing platform as integrating these risk-testing and explainability methods into that pipeline in an automated, scalable way, letting a team define quality requirements and relevant scenarios, generating an actionable risk report, and feeding problematic data back into training for targeted improvement.

Q&A highlights

Asked how to evaluate risk in strong AI, Gressner says he does not believe general AI is close, noting that in every use case his team tested they were able to find a failure, and argues that if a system were truly general and human-like, adversarial noise capable of fooling it would likely confuse humans too. Asked why society fears AI, he points to a history of failures, including a website tracking deaths linked to Tesla's Autopilot and chatbots that turned racist, and argues trust can only be earned through thorough testing before deployment. Asked who will actually carry out this testing, he says certification institutes risk losing relevance if they do not adapt their business models, that large non-European platforms could otherwise fill the gap, and that neurocat aims to supply the underlying tools to consultancies and certification bodies as an independent provider rather than acting as the certifier itself. Asked why healthcare was not listed among the high-risk categories, he says it falls under the broader 'safety components' category that carries over prior safety-critical rules. Asked when a formal certification for trustworthy AI might exist, he says regulation has to come first, though interim, voluntary quality seals could appear before the AI Act becomes mandatory.

Quotes

Risk is severity times exposure: how severe is something, and how probable it is that it will happen in the real world.

Florens Gressner

Only with profound testing before you deploy it can you really generate trust in society.

Florens Gressner

We looked at a lot of use cases, and we were always able to find a failure in these systems.

Florens Gressner

Questions answered in this talk

What counts as high-risk AI under the EU AI Act?

According to Florens Gressner at Rise of AI 2022, high-risk categories under the then-pending EU AI Act include critical infrastructure such as oil, gas, water and transportation, credit and job scoring systems, law enforcement, border control, biometrics, public services, and anything already treated as safety-critical.

Why can't standard performance metrics measure AI risk?

According to Florens Gressner at Rise of AI 2022, performance metrics average results across all data points, so rare but severe edge cases get diluted; he defines risk as severity times exposure and argues each relevant scenario must be tested separately to capture it.

Why is society afraid of AI, according to Florens Gressner?

According to Florens Gressner at Rise of AI 2022, society is afraid of AI because of past failures, including deaths linked to Tesla's Autopilot and chatbots turning racist, and trust can only be rebuilt through thorough testing before systems are deployed.

Mentioned

EU AI Act, risk as severity times exposure, adversarial attacks, defect triage, MLOps, explainability, data corruption testing

Keep exploring

More from Rise of AI Conference 2022

More on Enterprise AI & Adoption

Transcript

Auto-generated transcript, not edited. · 4,550 words

Show the full transcript

and now we come to at least to our final speaker and um i'm really happy in the beginning i i mentioned um that i think everything can be put under the umbrella of uh trust in ai and that we just talked about so many topics around ai and leads to a better understanding and i think this is the the yeah base for trust and now i have mathematican here who is who has the focus in statistics and optimizations and at the age of 22 he founded neurocat and the company is at the forefront of quality and compliance in the field of ai i have florence gressner here and welcome to the stage and a warm applause all right thanks hi uh thanks for the introduction um my name is florence i'm co-founder and ceo of neurocat and for nearly five years now we specialized in evaluating and mitigating risk for safety and security critical ai systems and in over 25 projects with customers together we implemented some testing strategies and enabled them enabled their mission-critical

use cases like autonomous driving and push these topics forward and so we want to enable trust in ai but it's not that we are afraid that there will be some singularity or that they will replace humanity at some point we just believe we can unleash the full potential of ai when we are providing tests and when the society can trust it and we can put it to the next level and enable more use cases also in security and safety critical domains and this is what we're trying to push forward we want to push the limits of ai and unleash its full potential and to enable that we want to empower these pioneers who want to bring the new use cases into mission-critical ai systems however now the classical security critical and safety critical domains now get extended there will come some regulations that are also will be decided on this year this is the AI Act for example and they published a taxonomy where they described what use case has risk and how much risk

has it um and this is defined for the entire um european union and they are like um unacceptable risks like something that is against the values of the european um person there are high risk applications and lower risk applications that are rather going in the entertainment domain or in the chatbot domain where some transparency is necessary but it does not provide a lot of risk to any person and just to name the high-risk applications because these now get very much extended and are the critical infrastructures so everything related to oil gas water and transportation in general um also um scoring systems that enable you to get a job or a credits or decide on that um are high risk applications and law enforcement in general so looking at the proof and seeing if it is sufficient not to prove but looking at evidence and looking if it's enough for law enforcement border control and biometrics and public services in general and everything that was safety critical before they act is also published so it's publicly

available the draft at least and for all these use cases for the high risk use cases there are no requirements set that are needed to be followed by the ones who are in these exact use cases and i just scratched out the one that is like unreasonable risk and the ones who don't have to beat but for all these who have uh high risk application and going in these use cases and have to follow these requirements they have to test the risk in a systematic manner and for every use case and for every relevant scenario they have to evaluate the risk and later in this talk i will want to show you how it's done you have to have a strategy and to process how to mitigate risk if you find a defect how can you overcome this and provide transparency and interpretability through explanations in the system itself and some other topics that i just want to name to be a complete but i don't want to focus on in this talk is a

whole gate data governance around it explaining why the data is sufficient and so on a documentation of every architecture and logging every event that it's um pushing through and implementing human oversight that humans can always intervene into these high-risk use cases but like i mentioned i just want to focus on the testing and mitigating risk because this is what we focus on the last five years because we were specializing on safety critical and security critical use cases so basically um with our customers together we created test strategies and implemented them for them and these already included like okay first you have to define some quality requirements what do you want to achieve how good the system has to be in the relevant environments but then you come to the point where you very quickly have to define a test approach where you describe how you can measure risk and how you can do a defect react that is nothing else than risk mitigation and up to the last point of the automation strategy so if

your tests are not automated then eventually they get just by budget cuts um scratched out and you're not concerned with it so automation is always a very important thing especially in ai so starting with a quality requirement what we are doing is first define the statistical confidence that is necessary um so in the automotive domain it has to be some ninth after the 99 of confidence in order for a system a safety critical system to be deployed defining this will give you just some insights how much test data you have to use and you have to select the metrics the risk metrics and the measures you want to follow and want to measure your systems against and define the relevant scenario your system should work under so should it work under rain under noisy conditions or under even dirty data and real world applications and setting these qualities thresholds based on the metrics that you have to exceed this is basically also everything what we are doing here is mentioned in the AI Act and

you have to select these things we have to set probabilistic thresholds and you have to follow these and these rules will become mandatory for those relevant um use cases so second one is the testing approach and here i just want to make a key difference to you normal ai application and risky ai applications because the way you measure the performance and measure the success of it is pretty much different and i really hope i don't lose more audience because there will be some formulas but i try to make it very very easy on that so basically you will have some data and we already realized there is some relevant data so you have you defined relevant conditions and relevant scenarios your system should work in so i just for the easiest to make it easy easier just distinguish between two scenarios you have clean data in your training set and you have some dirty data that you will experience in the field when you deploy your system and basically the probability that these data points

you are exposed to them like these dirty data points or these edge cases real edge cases probability might be very very low and you clean data you experience like um like every day you've experienced in your training set so and here is also the key point uh where this is a problem for high risk applications so you defined your performance metric because you had to do it in the quality requirements and then at some point in order to measure the performance you just apply this performance metric on every single data point so what's the problem here for high risk applications we see the dirty data is basically like this pink data point the probability is very low so for a performance metric it is not important and so if you optimize for this it is not important to capture this little tiny data point that is very improbable but probably it's a relevant condition or relevant scenario you have to follow so now what is the contrary to risk so basically risk is severity times

exposure so how severe is something so how much damage it can create and how probable it is that it will um happen in the real world so if we are looking at this one point it might be very improbable but if it's a very severe scenario and a very severe conditions for example a pedestrian on the street you might want to capture this and you might want to mitigate this risk so performance measures are not sufficient to estimate risk at this point so um basically just in comparison it is uh contradicts itself the risk and performance basically contradicts itself because performance cannot cover um all these points because rare scenarios and edge cases um are just thrown another bus so oops so what is important and what we pushed our customers towards is um decoupling all the relevant scenarios and looking at every scenario and separately and this requires you to have a significant amount of data for every relevant scenario and this leads you to a point that you are first able to measure

like the severity times exposure what the definition of risk is and also in the end you are able to push your decision boundary much much forward that you're having a significant amount of data in all relevant scenarios and you are not um like all these points are very important to you because severity can um like these every data point can have severe consequences um but what i didn't tell you is how do you get to these data points and how do these look like for ai applications so basically the hazardous threat of ai system is there's a huge difference between the training data and the real world so in the training data data might be very much clean and in the real world some noise might appear like rain weather conditions or camera noise that can just spontaneously appear in in the field basically and um the second point and just one point is not only for automotive also for nlp systems only a strap of characters also these kind of corruptions these kind of

noise that make data dirty that you have to be concerned with if it's safety critical the second point is the first one was more safety related now also security related if there is an attacker that wants to tamper with your system or seize any kind of advantages the attacker in an ai system is very much freely to do so because there is an active community of adversarial threats out there that get published very frequently that tell you how to put very imperceptible changes to your data point and then tamper with a model and manipulate its decision for upload filters this would be very much severe because you can just have some inappropriate content put some imperceptible noise over it upload it and you will trick any ai system that might lead to it so these are just how these data points are looking like but how do we get there how can we expand our data set and augment our data set such that these also get represented in a way that the the model

itself have to treat them as relevant data points and these happen with test methods so for the first part for the safety critical part and corruptions are test methods that are just augmenting data for relevant scenarios so they put basically on the clean data set some overlays like weather conditions like snow rain or dirt on the lens or a normal camera noise on it you can extend your data set and can you can use it for testing and the second one is the adversarial threats this is a very attacks this is a very active community they are published very frequently and this is basically a cat and mouse game between attacker and defender and in the end the deployer of a system just has to decide what kind of budget an attacker has to exceed in order to tamper with the model and how much computation it has to go into that so this is the testing approach basically this were the test methods and why performance metrics are not sufficient in order to measure

risk especially high risk and the third thing is defect triage and so you have to have a systematic approach how you if you find a defect and if you find a scenario failing how are you dealing with it in the end so first i mean we realized with vanilla performance metrics you're not able to detect any defects because the defects are not there for very rare edge cases and so first you have to test against these relevant scenarios you have to decouple them and test against single every single one of them that you defined before then you evaluate if your quality requirements are met and for every of these relevant scenarios and you minimize the risk by if a quality requirement is not satisfied for one scenario you have to feedback the data that was not doing well into it and optimize the risk measure instead of just the performance measure and in the end you need to update the architecture and the procedures in order to to do so so this is a very

complicated part and we are not concerned with that we are only concerned with the testing and providing the test methods so this is in the creativity of the developers of the i system itself but we provide some metric that you can optimize against and then test again and see if you are fulfilling all the risk requirements and um like this sounds like a very easy pro process and very straightforward but this is also not just an abstract thought so um i don't want to scare more audience here but basically a a very profound this is a really it's not an abstract thought it's a really concrete optimization problem one can follow i don't want to go too much into detail about this how to select these architectures but it is possible and you can optimize for risk instead of just optimizing for performance so now the automation strategy i already mentioned that it's a very crucial and important part and if you're outsourcing anything to humans you might experience some budget cuts and then what

happens which kind of tests do you want to execute and which of them not so automation is very crucial and important especially for very automated development workflow of ai so the ml ops workflow that is very commonly used for ai development follows a quite standardized process so in the beginning you're having some data processing annotating the data cleaning out the data and after them you are selecting your architecture and your training procedures and your hyper parameters such that you can just train your model and hope that you will find one that is good enough to satisfy your quality requirements there comes a evaluation part or the validation part in the ml ops life cycles that are also out there and these platforms it's heavily concentrated on performance metrics but performance metrics do not capture high-risk applications and if you are satisfied with the model you can put it into this into the deployment and we from our part we put all our experience in creating testing strategies and to our customers and enabling mission critical

use cases into our software platform we call it 8-kit and 8-kit basically contains all these relevant test methods where you can test against these different threads it provides explainability methods and is pretty much very much automated and such that you can just implement it into ml ops workflow and measure risk reliably so we call it a kit until you make it the game in order to minimize risk so it's easy to integrate and you can use it in your automated workflow because all the methods are automated and highly scalable it is your robustness and explainability expert that selects your test methods for you and keeps you up to date with the standards and the risks that you're experiencing and what i mentioned before basically you're able to specify your quality requirements your relevant scenarios and just put them in and define the pipeline you want to execute in your automated workflow select the scenarios if it's some lens flare effects some lighting effects or some weather condition or sensor noise and we will provide you

with an actionable report with the metrics that will help you to also estimate risks for your high risk applications and if you find you're not sufficiently good and your your risk is not mitigated enough all the data we produced while testing all these corruptions and all these adversarial attacks can be feedback into your training process and you can targetly improve the risk and minimize risk with that and makes us all in a very automated workflow feedback the data that doesn't work retrain on it and and hope you will make it at some point so what did we see so basically there will be regulations that define us with new um that define new safety critical use cases um we saw them they are in a taxonomy and also requirements for them are satisfied but we can take a lot of things we learned in the past of safety critical components and use cases because we can just reuse everything that you have to have in a test strategy so you have to test the risk

evaluated for every relevant condition and scenario you have to mitigate the risk and have a systematic approach towards that and third you need to provide transparency and interpretability i was not focusing my presentation here but this is also something we are providing and equipping you with and this will be able to satisfy the AI Act the air act will be decided on this year then it will go into a triage and into a um one year of testing phase and then it will become mandatory so in not more than two years we will have mandatory requirements for these critical use cases and yeah getting acquainted with the topic of high-risk applications if you are in this is very crucial also starting now that you're not surprised when the regulation comes and this was basically my take on enabling trust in ai and how to get these abstract regulations into concrete and test strategies and i'm very interested to learn what's your testing strategy and are you maybe in a high-risk application domain thanks a lot

for the intention and i'm looking forward to questions yes thank you florence really well done i come to you um so i have one question um how do you evaluate the risk in a strong ai i think this is always a key word for everyone and still not really existing but um yeah have you thought about that can you tell me what you mean by a strong strong ai like ai that is maybe you you can do it better than me but uh like a developing in an intelligence um independent from humans um like autonomous driving seems like that yeah yeah autonomous um yeah like um like a for general ai if it's in a safety critical domain so first of all i don't believe that we are very close to it yes um we looked at a lot of use cases and we were always able to find a failure in these systems so there's not these um these intelligence insights that we need to compare to human intelligence so i believe we're a

bit further away but i believe if it would exist we would still be able to um manipulate the system and optimize for noise but just this noise must then look to us like um like uh like will also confuse us then at the point like if it's the ai is good enough and we find noise to it i believe it can only be real general ai and real human-like if it already also would confuse us then yeah um and another one um you were talking about like testing the risk and at least you also mentioned transparency i think that when you tested it it's also important to to make create a transparency to the audience right and i think most of the people they are still afraid of ai and i also always handle with that um but what do you think why are they and still so afraid is it that they don't understand it um and how can we change it yeah i believe the society is afraid of ai because there were

so many use cases failing in the past i mean there's a website called tesladeath.org.com and where there's a huge list of people dying from the autopilot because there was not sufficient testing being done there are a lot of cases chat boss getting racist when they're exposed to to humanity so i believe we prove society that we have to be afraid of it and if we don't test it before and then society figures out how bad it is then how should they trust it if we can not trust it by ourselves so i believe and there are a lot of examples that showcase why we don't trust it and i believe only was profound testing before you deploy it you can really generate trust in society yes i think we also had some questions you uh okay start where you want um two questions first the whole testing thing i know it's coming from the european union but we germans like this you know validation testing having tuff temple on top on this um so what

is your feeling will other europeans pick this off will we set standards you know like will be be there the schreiber as usual um so the german versus the rest because it's international and then the second is what is like your business model in terms of my first thought is all the consultancies out there they are now having great business integrating your solutions and others because they they know the process in the companies they know where to tackle it they have to go in um so i'm trying to figure out who will do all this testing which we all under underestimate because every piece of software now needs at least reconsidered if it's falling under this yeah um so the first question i thought was two questions the first question is who will elite this and i believe we have a huge problem we have certification institutes and they made a lot of money by just stamping um like cars and so on and providing a seal of approval and they would have a really

hard time rethinking their business model because now there will be just a few persons a bunch of programmers that sits there and use these tools and we are seeing that these certification institutes have a very hard time if it's tiff um or diegra to rethink that and there are platforms like microsoft and again outside of europe that also are able to provide the same thing and so maybe we will even lose this game if we don't wake up and able to rethink everything and the second question was in the direction of who will execute the testing in the end so at least what i'm hoping to do and also generating partnerships with is like of course the consultancies will be able to to audit and test these requirements and um certification institutes if they wake up and um but like i mean just i want to mention one thing like there are some um innovative um products also in certification institutes as a tooth it for example and creates also some smaller standards that one

can use and specify the requirement with and but they all need a tool and we are trying to put our tools into all of these our tool into all of these things and people like yeah it will be an independent party that has to measure this but it has to be an independent provider who generates the tools so basically um it's it's twofold answer maybe to this question one quick question you had one slide uh where you introduced the different areas where uh eu regulations are coming up and i was missing health care um as a health care health care anything around health care i was wondering if is that due to this already being highly regulated or um what's your perspective i didn't really mention this but like in the end there was also like one bullet point called safety uh safety components and everything that was safety critical before is still safety critical um afterwards so i didn't explicitly mention it in this section but um i i would see it and at

least how i would interpret it i would see it in this corner and in this category okay um i got my coming out i'm german and i'm a consultant for ai so i just wanted to know and because of the certification when do you think there will be a kind of deen or easel certification for trustable and no risk ai yeah this is uh yeah i mean first of all there must be regulation before you can provide this seal of approval of course you can also provide some seals that are some kind of specification where you say okay these are my quality requirements and um independent institute tested it against it so these seals can come before but i believe here only when the regulation is there and when we look at the AI Act maybe there will be some um standards that will exceed or like be faster than this but i believe at the day when it becomes mandatory and even it's a transition period before there will be products and there will

be seals that you will be able to to apply so many questions really cool so you will still be here right and the others hopefully they are also here um and i'm happy to connect later um so thank you for your presentation florence thank you very much you

Rise of AI 2027 · Berlin · May 2027

Be in the room for the next one.

Ten years of talks started as conversations in one curated room in Berlin. The next edition brings 300 selected leaders together in May 2027.