In this Rise of AI Virtual Chat from May 2020, host Fabian Westerheide talks with Dr. Beril Sirmacek, assistant professor at Jönköping University, about what trustworthy AI means and how explainability fits into it. Sirmacek illustrates trust using an analogy of an alien being told an animal is a cat: trust can come either from a system's track record of good results or from an explanation of the reasoning behind a decision. She structures trustworthiness and ethics into three subtopics: GDPR and data privacy, data security, and responsible use, and describes explainability as one mathematical tool within that larger field, not the whole answer. Using examples including elderly-care robots, cancer-stage classification and credit scoring, she discusses named tools such as LIME and SHAP, the trade-off between model complexity and explainability, and the case for hybrid systems combining feature engineering with smaller neural networks. She also addresses AGI, EU AI regulation discussed in the European Commission's recent white paper, and audience poll results on who should be responsible for AI's impact and who should define the values it acts on.
Key takeaways
- Sirmacek frames trust in AI as coming from either a system's track record of good results or from explaining the reasoning behind a decision.
- She divides AI trustworthiness and ethics into three subtopics: GDPR/data privacy, data security, and responsible use, with explainability as one tool supporting the third.
- German law currently does not allow leaving an elderly person alone with a home-care robot because a machine "doesn't know that you exist."
- She names ELI5, LIME and SHAP as concrete, open-source explainability tools that show which features drove a model's decision.
- She argues for hybrid systems combining older feature-engineering methods with smaller neural networks to balance accuracy against explainability.
Detailed notes
Defining trust with the alien-and-cat analogy
Sirmacek explains trust in AI through an analogy: if an alien with no knowledge of Earth asks a human what an animal is and is told "it's a cat," the alien can only trust the answer either because the human/system has a track record of good decisions, or because the human explains the reasoning, for example listing features like fur, ears and a tail that define a cat. She notes that a trained system faces the same problem with a type of cat it has never seen before, and may be uncertain.
Elderly-care robots and the limits of current law
Discussing a company developing indoor robots to help elderly people at home, for example carrying medication and water, Sirmacek says such robots cannot yet be mass-produced and placed with elderly people unsupervised because AI systems are not yet trustworthy enough, and notes that in Germany there is a law against trusting a machine in this way because "a machine doesn't know that you exist."
Three subtopics of trustworthiness and ethics
Sirmacek structures trustworthiness and ethics into three areas: GDPR and data privacy (whether a system transparently discloses what data it collects, why, and how it is stored); security (whether collected data and models are protected from theft, for instance through data engineering that encodes data into a form that is unusable even if stolen); and responsible use, which she calls the least defined of the three, covering whether a system explains why it reached a given decision. She frames explainability as one mathematical tool that supports responsible use, not the whole of trustworthy AI, comparing it to fitting "how you say it's a cat" by pointing to ears and a tail.
Explainability tools and the accuracy-explainability trade-off
Sirmacek names concrete explainability tools, including ELI5 ("explain like I'm 5"), LIME and SHAP, describing them as downloadable, open-source mathematical methods that show which features or training-data patterns drove a model's decision, for example highlighting dark borders in a cancer-stage image classification. She argues there is a trade-off: more complex neural networks can achieve higher accuracy but are harder to explain, and suggests a hybrid approach that combines older feature-engineering methods with smaller neural networks can yield systems that are both more robust and more explainable.
Regulation, GDPR, and self-diagnosing systems
Responding to a question about EU AI regulation following the European Commission's AI white paper, which proposed risk categories with national authorities expected to begin work by the end of the year, Sirmacek says she believes some AI-specific laws will eventually be necessary to protect people, while acknowledging researchers' and companies' fears of restrictions similar to those introduced by GDPR. On GDPR limiting innovation compared to China or the US, she argues constraints can drive new solutions, citing an example of designing smart-office sensors that detect the presence and activity of people without identifying them, to comply with rules against workplace camera surveillance. She also discusses AGI as full domain-transfer intelligence not yet achieved, and describes a vision for industry-4.0-style self-diagnosing AI systems that can identify which of their own neurons or layers caused a low-confidence or incorrect decision.
Q&A highlights
Asked how a regulator should audit a credit-scoring model for fairness, Sirmacek says the same three subtopics apply: public disclosure of data collection, evidence of security measures, and mathematical explanation of individual decisions, for example whether a rejection was driven by a feature like a year of salary history or by a comparison to other applicants' apartments. She acknowledges that even mathematically valid explanations may not be good ways to communicate rejections to customers, and mentions widely reported cases of systems rejecting certain demographic groups, such as Black applicants, more often because of biased historical training data, though she says it is unclear whether this is accurate. Closing polls of the live audience showed 50% believing AI should always be regulated and 50% saying "it depends," a majority holding the deploying company responsible for AI's impact rather than government or the AI itself, and only 9% trusting society at large, rather than companies or government-nominated expert groups, to define the values AI should act on.
Quotes
How can I trust you? I'm an alien, I don't know anything about this world, and you tell me that it's a cat.
Beril Sirmacek
A machine doesn't know that you exist.
Beril Sirmacek
Questions answered in this talk
What does trustworthy AI mean?
According to Dr. Beril Sirmacek at this Rise of AI Virtual Chat in 2020, trustworthy AI covers whether a system can be trusted either because of a proven track record of good decisions or because it can explain its reasoning, structured around three subtopics: GDPR/data privacy, data security, and responsible use.
What is the difference between trustworthy AI and explainable AI?
Beril Sirmacek explains that explainability is a mathematical tool within the broader field of trustworthiness and ethics, used to show why a model reached a decision, but it is only one part of responsible use, not the full definition of trustworthy AI.
What tools exist to make AI models explainable?
Beril Sirmacek names ELI5 ("explain like I'm 5"), LIME and SHAP as open-source, downloadable mathematical methods that show which features or training-data patterns a model relied on to reach a given decision.