At the Rise of AI Summit 2020, Beril Sirmacek, assistant professor in computer science and informatics at Jönköping University, argues that trustworthy AI means opening up the black box of machine learning models rather than trusting them on performance numbers alone. She frames trust as a basic human need alongside food and safety, citing Abraham Maslow's 1943 theory of human motivation, and argues that businesses deploying AI face the same requirement: users will only pay for a model's decisions if they can trust them. She illustrates the problem with concrete cases: search engines linking the name Latanya Sweeney to arrest-related ads because of racial bias, a recidivism-risk model that misjudged two defendants named Bernard and Dylan, adversarial noise that turns a panda into a misclassified animal, and a wolf-versus-husky classifier that reached 99.9 percent accuracy by detecting snow in the background instead of the animal. She presents explainable AI (XAI) methods such as SHAP feature importance as tools for inspecting training data, models and inputs, stressing that the right explanation depends on who is asking and that trust itself has no quantitative measure. She closes by imagining real-time XAI feedback loops between AI systems and their users.
Key takeaways
- Sirmacek frames trust as a basic human need on the same level as safety, applying Maslow's hierarchy of needs to why AI decisions require explanation.
- A search engine linked data scientist Latanya Sweeney's name to arrest-related ads due to bias in how her name was classified.
- A risk-assessment model rated defendant Bernard as high risk and Dylan as low risk, yet Dylan was rearrested days later and Bernard was not.
- A wolf-versus-husky image classifier reached 99.9 percent accuracy by detecting snow in the background rather than the animal.
- Explainable AI traces back to a DARPA project where a camouflage-tank classifier was actually distinguishing cloudy from bright sky conditions.
Detailed notes
Trust as a basic human need
Sirmacek opens by connecting the need for trustworthy AI to Abraham Maslow's 1943 theory of human motivation. In Maslow's pyramid, safety and security, which she summarizes as trust, sit directly above food and water as basic survival needs; a baby who cannot find a safe, trustworthy environment will not survive, and adults need trust in the same way in relationships and in business. She applies this to AI directly: a developer who wants to sell or deploy a model on the cloud needs users to trust that its results are safe, reliable and ethical before they will pay a subscription fee for it.
Real-world failures behind the trust problem
She lists everyday decisions already made by AI: credit card approval, medical diagnosis from CT scans and blood tests, and job-application screening, each raising the question of who explains a rejection. She tells the story of data scientist and Harvard professor Latanya Sweeney, who found that search engines linked her name to arrest-related ads because the algorithm had classified her name as belonging to a Black minority group associated in past data with such searches. She describes a police risk-assessment model that rated a defendant named Bernard as high risk and one named Dylan as low risk, with Dylan re-arrested days later and Bernard causing no further issues, without the model explaining what features drove the decision. She also cites adversarial-noise examples where imperceptible pixel changes make a panda classifier output a different animal. As a case of a well-performing but unexplained model, she describes a wolf-versus-husky classifier that reached 99.9 percent accuracy; despite the visual similarity of the two animals, she argues that high performance alone does not establish trust.
Explainable AI as an audience-dependent toolkit
Sirmacek traces explainable AI (XAI), previously called interpretable AI, back roughly ten years, and describes its basic workflow: a model is trained, applied to new real-world data, and XAI methods are used to explain why a given decision was reached. She stresses that the choice of method depends on the audience: a developer may need a technical explanation of bias inside specific neural network layers, while an end user waiting on a bank-card decision may need a simplified decision tree or similar training examples instead. As one tool, she cites SHAP feature importance, illustrated on a Titanic survival dataset where gender turns out to be the most important feature in the decision, ahead of cabin, ticket price or age. Applied at pixel level to the wolf-and-husky classifier, the same kind of method revealed that the model was reacting to snow in the background rather than to the animal's fur. She adds that trust has no quantitative value: using her own story of a vet recommending a vaccine, she notes that the same explanation can feel more or less trustworthy depending on how it is delivered, a judgment left to the listener.
Origins of XAI: DARPA and camouflage tanks
Sirmacek traces the origin of the XAI field to a DARPA project in the United States that trained a vision algorithm to classify camouflaged tanks and reached 99.9 percent accuracy, a result so good that researchers asked how the system could see what even a human eye could not. Investigation showed the model was not detecting tanks at all: in the training set, all camouflage-tank images happened to be taken in cloudy scenes and the rest in bright ones, so the model was classifying sky conditions, not tanks. She presents this as the moment researchers concluded that a high performance number alone cannot be trusted without an explanation of what the model is actually using to decide.
What's next: real-time explanation loops
Looking ahead, Sirmacek imagines XAI becoming a real-time process in which a model explains its decision to a user as it is made, the user gives feedback such as disagreement with the reasoning, and the model retrains or looks at other features and explains again, forming a continuous feedback loop. She sums up that a simple model is not necessarily lower performing and a complex model is not necessarily higher performing, and that higher accuracy does not by itself mean more trust; the accuracy still needs an explanation. She adds that XAI brings both benefits and risks, since explaining a model can also expose information that creates security leaks if it reveals details that should not be disclosed.
Q&A highlights
Asked what will drive better understanding of AI, Sirmacek points to the generalization problem: a self-driving car trained only on German houses and streets will not perform at the same accuracy once sold into the Netherlands, because buildings, colors and road types differ. Asked what technical expertise is needed to run an explainability test, she ties the answer to audience: a non-technical person can be shown a simplified decision tree or feature-importance ranking, while a developer needs deeper methods to check whether a model is really looking at an animal's fur rather than the background. Asked how regulation, such as GDPR-style rules, could be technically implemented, she says it remains an open question across Europe, since national and EU teams are still trying to agree on ethical rules and even on how to define trust mathematically.
Quotes
Good news is that AI is already in all areas of our lives and the bad news is that AI is already in all areas of our lives.
Beril Sirmacek
High performance of an algorithm doesn't mean that we can hide the trust.
Beril Sirmacek
Simple model doesn't mean that it's low performance, or more complex model doesn't mean that it's high performance.
Beril Sirmacek
Questions answered in this talk
Why does high accuracy not mean an AI model can be trusted?
According to Beril Sirmacek at the Rise of AI Summit 2020, a model can reach very high accuracy while relying on the wrong signal, as in a wolf-versus-husky classifier that hit 99.9 percent accuracy by detecting snow in the background rather than the animal itself.
What is explainable AI (XAI) and where did it come from?
According to Beril Sirmacek, XAI is a set of mathematical methods, developed over roughly ten years and previously called interpretable AI, that explain why a trained model reached a decision; she traces its origin to a DARPA project where a camouflage-tank classifier turned out to be detecting cloudy versus bright skies rather than tanks.
Why did a search engine associate Latanya Sweeney's name with arrest-related ads?
According to Beril Sirmacek, data scientist and Harvard professor Latanya Sweeney found that search engines had classified her name as belonging to a Black minority group historically linked in the data to such searches, causing arrest-related ads to appear next to her name.