In this Rise of AI Virtual Chat from August 2020, host Fabian Westerheide talks with Kristian Kersting, professor for AI and machine learning at TU Darmstadt, about what Kersting calls the "third wave" of AI: combining reasoning, learning and acting through hybrid, knowledge-driven systems. Kersting explains GPT-3's excitement as the first sign of a system handling many tasks it was not explicitly trained for, describes deep learning as "differentiable programming," and argues the field needs a dedicated high-level programming language in which learning algorithms are first-class citizens, so domain experts like biologists do not need to specify network architectures themselves. He describes his own research combining probabilistic programming, statistical relational learning and deep networks, and his work leading the planned Hessian cross-university AI institute, which he calls in the talk the “Tensor Institute” (following, he explains, a naming progression from a “Parameter Institute” to the Vector Institute), bringing together universities across the state. On ethics, he argues bias findings in AI models mirror human bias rather than being an AI-specific problem, and rejects blaming AI for autonomous weapons rather than the humans and game theory behind weapons development. He is skeptical of the term AGI and says true intelligence means being a robust general problem solver, not an optimal one.
Key takeaways
- Kersting says GPT-3's significance is handling many tasks it was not explicitly trained for, not any single new technique.
- He argues deep learning is best understood as "differentiable programming," where gradients tune a sketched algorithm's parameters using data.
- He calls for a dedicated high-level AI programming language where learning algorithms are first-class citizens, so non-programmers need not pick architectures.
- He argues AI bias findings mirror human bias and are "a mirror of what we are doing," not a flaw specific to AI.
- On autonomous weapons, he says people should blame the humans and industries building weapons, not only the AI used in them.
Detailed notes
Why GPT-3 excites researchers
Kersting says what stands out about GPT-3 is not any single trick but that, for the first time in the textual domain, a system appears to handle many different tasks, including ones it was not explicitly trained for, which he calls essential for a "truly smart system." He describes deep learning as "differentiable programming": rather than hand-coding every rule as in the first wave of AI, engineers sketch an algorithm with differentiable building blocks and let data and gradients tune its parameters, which he calls the second wave. He explains transformers, GPT-3's architecture, as roughly a differentiable version of nearest-neighbor search using a large dot product.
The third wave: reasoning, learning and acting together
Kersting frames the "third wave" as combining differentiable deep learning with symbolic, reasoning-based approaches, referencing Gary Marcus's paper "The Next Decade in AI" and Yoshua Bengio's talk on cognitive agents. He says his lab works on probabilistic programming and statistical relational learning, replacing parts of database-like tables with deep networks while keeping a gradient-based way to propagate information, building on work with colleagues from KU Leuven and companies including Uber. He argues systems like AlphaGo/Alpha Zero already hybridize search (Monte Carlo tree search) with neural evaluation functions, but that true systems combining reasoning, learning and acting are still early.
A high-level programming language for AI
Kersting argues AI needs a dedicated high-level programming language in which learning algorithms are first-class citizens, so a user can specify a goal, like classifying images and emailing a group, without choosing whether to use an SVM or a deep network. He compares current tools like TensorFlow to "assembler for AI" that is still hard to debug, and connects this to his lab's collaboration with biologists on plant phenotyping who do not want to specify network architectures themselves. He credits companies like Google, through investment in usable frameworks and advertising, with democratizing machine learning the way Linux and later usability efforts spread earlier technologies.
Bias, ethics and blaming AI for weapons
Kersting says a well-known study showing AI language models capture human stereotypes was an important finding, but calls it "a mirror of what we are doing" rather than a flaw specific to AI, and frames the debate over data versus algorithmic bias as central and unresolved. On autonomous weapons, he says he has signed a letter opposing AI in autonomous weapons, but argues people should also blame the humans and industries that build weapons and the game-theoretic logic of mutual deterrence, not treat AI as a convenient scapegoat. He also argues that fears about AGI "getting rid of humans" say more about human psychology than about what a truly intelligent system would do.
What counts as truly intelligent
Kersting says he dislikes the term AGI, since standard AI textbooks never distinguished general intelligence from AI in the first place, and argues machine intelligence should not aim to copy humans but can be usefully inspired by them, noting self-driving cars modeled purely on human driving would not reduce accidents. For him, true intelligence means being a general problem solver able to ask for help and adapt, not being optimal at everything; he argues reproduction-based optimization (evolutionary algorithms) trades optimality for robustness, citing a paper on "why sex is not good for optimization." Discussing his role leading Hessen's planned cross-university AI institute, which he calls the "Tensor Institute," he says a letter of interest from Canada's Vector Institute could enable collaboration with researchers including Geoffrey Hinton and Gary Marcus.
Q&A highlights
Asked where AI will be in 2030, Kersting says his hope is that AI becomes as broadly accepted as the car once became, that climate challenges are addressed with AI's help, and that AI and programming are taught in schools, calling this a matter of societal "enlightenment" as much as academic or industry progress, though he calls achieving it within ten years "challenging." Westerheide notes Kersting supervised the master's thesis of a student who later became known in Germany for criticizing politics and news, without naming him further in the transcript.
Quotes
It's amazing how much they can get out of this data and the huge model that they have.
Kristian Kersting
Failure is part of intelligence in a sense, and that's why we want to have the third way.
Kristian Kersting
We should not just be happy that there's a new player and therefore we can blame her, whatever you want to call AI.
Kristian Kersting
Questions answered in this talk
What does Kristian Kersting mean by the "third wave" of AI?
According to Kristian Kersting at this Rise of AI Virtual Chat, the third wave combines differentiable deep learning with symbolic, reasoning-based approaches so systems can reason, learn and act together, building on prior work by researchers such as Gary Marcus and Yoshua Bengio.
Why is GPT-3 significant according to Kristian Kersting?
Kersting explains that GPT-3 is notable because, for the first time in the textual domain, it appears able to handle many different tasks, including ones it was not explicitly trained for, which he sees as essential for a truly capable system.
Does Kristian Kersting think AI models are inherently biased?
Kersting says at this Rise of AI Virtual Chat that studies showing AI language models capture human stereotypes reflect a "mirror of what we are doing" as humans, so bias is a broader societal and data problem rather than one caused by AI itself.