At Rise of AI 2022 in Berlin, Florens Gressner, co-founder and CEO of neurocat, argues that trust in AI has to be built through systematic risk testing, not assumed. Speaking on the Applied AI Stage, he explains that neurocat has spent nearly five years evaluating and mitigating risk for safety- and security-critical AI systems, running more than 25 customer projects including autonomous driving use cases. He walks through the European Union's then-pending AI Act, which classifies AI applications by risk and imposes testing, risk-mitigation and transparency requirements on high-risk categories such as critical infrastructure, credit and job scoring, law enforcement and biometrics. His central technical argument is that ordinary performance metrics fail high-risk AI systems because they average over all data and dilute rare but severe edge cases; he defines risk instead as severity times exposure and argues each relevant scenario, including corrupted or adversarially manipulated data, must be tested separately. He describes neurocat's automated testing platform, built to integrate into machine-learning pipelines, and closes by predicting the AI Act will become mandatory within about two years and answering audience questions on strong AI, public trust, certification and healthcare.
Key takeaways
- Gressner defines AI risk as severity times exposure, arguing rare but severe edge cases matter even when performance metrics show them as negligible.
- The pending EU AI Act classifies AI use cases by risk and imposes testing and transparency duties on high-risk categories.
- High-risk categories he lists include critical infrastructure, credit and job scoring, law enforcement, border control, biometrics and public services.
- neurocat tests AI systems against corrupted data (weather, sensor noise) and adversarial attacks separately, rather than relying on one aggregate performance score.
- Gressner argues automated testing is essential because manual or non-automated tests tend to be cut when budgets tighten.
Detailed notes
neurocat and the coming AI Act
Gressner introduces neurocat, which he co-founded at age 22, as a company that has spent nearly five years evaluating and mitigating risk for safety- and security-critical AI systems, running more than 25 customer projects including autonomous driving. He frames the goal not as fear of a singularity but as unlocking AI's potential in safety- and security-critical domains once society can trust it. He then describes the European Union's AI Act, to be decided that year, which classifies AI use cases by risk: unacceptable risk, high risk, and lower-risk cases such as entertainment or chatbots that mainly need transparency. High-risk categories include critical infrastructure such as oil, gas, water and transportation, scoring systems for credit or employment, law enforcement, border control and biometrics, and public services, alongside anything already treated as safety-critical. For these, the Act requires systematic risk testing for every relevant scenario, a strategy for mitigating defects, and transparency and interpretability through explanations, plus data governance, documentation and human oversight, which he mentions but does not cover in the talk.
Why performance metrics can't measure risk
Gressner distinguishes performance from risk: a performance metric applied across all data points treats a rare, 'dirty' edge case, such as unusual weather or sensor noise, as unimportant simply because it is improbable, even if that scenario is a pedestrian on the street. He defines risk as severity times exposure, meaning how much damage a scenario could cause and how likely it is to occur, and argues performance and risk can point in opposite directions because rare, severe scenarios get averaged away in ordinary performance measures. His proposed fix is to decouple relevant scenarios and test each one separately with sufficient data, rather than scoring a system on one aggregate metric.
Sources of risk: corrupted data and adversarial attacks
He describes two ways real-world data diverges from clean training data. Safety-related corruptions include weather conditions, camera noise or corrupted character strings in NLP systems, which can be simulated by overlaying clean data with effects such as snow, rain or dirt to expand test sets. Security-related adversarial threats come from an active research community that regularly publishes imperceptible changes to data capable of manipulating a model's decisions, for example to trick content-upload filters; Gressner calls this a continuous 'cat and mouse game' between attackers and defenders in which a deployer must decide how much attacker effort the system should be designed to resist.
Defect triage and automated testing
For handling failures, Gressner describes testing each relevant scenario individually against defined quality requirements, feeding back data from failing scenarios into retraining to optimize for risk rather than only performance, and updating architecture and procedures as needed, though he notes neurocat's own focus is on providing test methods rather than fixing architectures. He stresses that automation is essential because non-automated tests tend to be cut under budget pressure, and situates this within a standard MLOps workflow, from data processing and model training to an evaluation stage that today is heavily weighted toward performance metrics that miss high-risk failure modes. He describes neurocat's testing platform as integrating these risk-testing and explainability methods into that pipeline in an automated, scalable way, letting a team define quality requirements and relevant scenarios, generating an actionable risk report, and feeding problematic data back into training for targeted improvement.
Q&A highlights
Asked how to evaluate risk in strong AI, Gressner says he does not believe general AI is close, noting that in every use case his team tested they were able to find a failure, and argues that if a system were truly general and human-like, adversarial noise capable of fooling it would likely confuse humans too. Asked why society fears AI, he points to a history of failures, including a website tracking deaths linked to Tesla's Autopilot and chatbots that turned racist, and argues trust can only be earned through thorough testing before deployment. Asked who will actually carry out this testing, he says certification institutes risk losing relevance if they do not adapt their business models, that large non-European platforms could otherwise fill the gap, and that neurocat aims to supply the underlying tools to consultancies and certification bodies as an independent provider rather than acting as the certifier itself. Asked why healthcare was not listed among the high-risk categories, he says it falls under the broader 'safety components' category that carries over prior safety-critical rules. Asked when a formal certification for trustworthy AI might exist, he says regulation has to come first, though interim, voluntary quality seals could appear before the AI Act becomes mandatory.
Quotes
Risk is severity times exposure: how severe is something, and how probable it is that it will happen in the real world.
Florens Gressner
Only with profound testing before you deploy it can you really generate trust in society.
Florens Gressner
We looked at a lot of use cases, and we were always able to find a failure in these systems.
Florens Gressner
Questions answered in this talk
What counts as high-risk AI under the EU AI Act?
According to Florens Gressner at Rise of AI 2022, high-risk categories under the then-pending EU AI Act include critical infrastructure such as oil, gas, water and transportation, credit and job scoring systems, law enforcement, border control, biometrics, public services, and anything already treated as safety-critical.
Why can't standard performance metrics measure AI risk?
According to Florens Gressner at Rise of AI 2022, performance metrics average results across all data points, so rare but severe edge cases get diluted; he defines risk as severity times exposure and argues each relevant scenario must be tested separately to capture it.
Why is society afraid of AI, according to Florens Gressner?
According to Florens Gressner at Rise of AI 2022, society is afraid of AI because of past failures, including deaths linked to Tesla's Autopilot and chatbots turning racist, and trust can only be rebuilt through thorough testing before systems are deployed.