At Rise of AI 2023, Hans Uszkoreit, Scientific Director at DFKI and co-founder and CTO of Ginance Technologies, asks why the world needs many large language models instead of one truly universal one. He explains that transformer-based foundation models gain their power from self-attention, transfer learning, self-supervised training and emergent functionality, and describes what an ideal "omniglot" model would need: fluency in most of the world's roughly 7,000 languages, where current systems handle only 20 to 50 well, plus complete factual and procedural knowledge. He argues expectations of truthfulness, creativity and consistency conflict with each other and with how these models are trained, illustrating this with contradictory ChatGPT answers about the Chinese Communist Party in Chinese versus English, and with a fabricated claim that a living friend of his had died. He describes a shift from machine learning to "machine teaching," since the core transformer algorithm has barely changed since GPT-1, and outlines two ways to add memory and multi-step reasoning: built-in recurrence, or external agents like AutoGPT. He closes arguing Germany and Europe must build their own large-model capacity through the LEAM initiative.
Key takeaways
- Uszkoreit says the core transformer algorithm has changed little from GPT-1 to ChatGPT; the real shift is in training data and stages, which he calls machine teaching.
- He says current systems cannot do self-driven multi-step or symbolic reasoning, have no memory between prompts, and lack self-awareness or consciousness.
- Asked in Chinese and English about a rejected Chinese Communist Party application, ChatGPT gave a practical, helpful answer in Chinese and a critical, different answer in English.
- ChatGPT told Uszkoreit a friend, a singer, had died in 2021, though she was alive; he attributes this to the model filling a data gap with the statistically likelier answer.
- He argues Germany and Europe must build their own large-model capacity, citing the LEAM initiative and a funding debate over an estimated 350 to 500 million euros.
Detailed notes
Why ask how universal foundation models are
Uszkoreit opens by noting that if today's large language models were truly universal, knowing all languages and knowledge, ethical and reliable, there would be no reason for competing models; the world would just keep improving one shared system. He recounts working with language models before they were large or neural, fine-tuning a BERT model at his own company and later applying GPT very early for summarization at a company producing daily press reviews for large German corporations. He describes transformer models as generative encoder-decoder or decoder-only systems powered by self-attention rather than recurrence, trained through transfer learning and self-supervised learning, which lets the system generate its own training labels. Their defining feature is emergence, functions the system was never explicitly trained for, and what Percy Liang calls homogeneity, the idea of one system giving everyone the same underlying "education."
The omniglot ideal and its limits
He sketches the ideal universal model, which he calls omniglot: fluent in all of the roughly 7,000 languages in the world, including minority languages and jargons, and holding all conceptual, factual and procedural knowledge. Current systems master only about 20 to 50 languages well and degrade sharply beyond that. He pushes back on the expectation that a model should know every fact, noting no human or even a search engine can do that, though conceptual knowledge is learned together with language and cannot be fully separated from it. He adds that the systems are already explanatory to a large degree, including being able to revise an explanation when questioned, though not always correctly.
Truthfulness versus creativity
Uszkoreit argues there is no single truth for a model to report, noting that Xi Jinping also demands truthful systems, and that creativity and strict truthfulness are in direct tension: a fully truthful system cannot also freely invent fiction. He illustrates the risk with an anecdote: he asked ChatGPT about a friend, a singer in New York, and it correctly described her life but then stated she had died in 2021, which was false. He attributes this to the model acting as a probability machine that, lacking recent data during a gap in her public appearances, produced the statistically more likely continuation.
Inconsistency across languages and prompts
He describes asking ChatGPT, in Chinese, why an application to join the Chinese Communist Party was rejected, and receiving detailed, practical advice; asking the same question in English produced a critical answer calling the party dogmatic and authoritarian, with no path to appeal. He argues language models are also inconsistent across differently phrased prompts because they are trained on an enormous, internally inconsistent body of text, and questions why anyone should expect full consistency from a system when no human is fully consistent either.
Machine teaching, not censorship
Uszkoreit argues restricting a model's training sources, as China does by limiting access to services like Wikipedia in favor of its own state-approved encyclopedia, is the wrong response, since it leaves companies such as Baidu and Tencent with much less data and an attempt to build a "clean" filtered version of Common Crawl failed outright. His main message is that AI has entered the paradigm of machine teaching: the transformer algorithm itself has changed little from GPT-1 through GPT-3 to ChatGPT, so the real differentiator across pre-training, behavior training via human feedback, domain-specific training and fine-tuning is what and how the system is taught.
Missing ingredients for AGI, and Europe's stake
On AGI, Uszkoreit says today's systems do probabilistic, implicit learning but cannot do self-driven multi-step or symbolic reasoning, have no memory between prompts beyond re-feeding prior dialogue, and show no self-awareness, consciousness or intentionality. He sees two paths to add these: building memory and recurrence directly into models, as LSTMs once had, or attaching external agents such as AutoGPT that connect a model to tools like search engines or calculators for multi-step planning, warning that intentionality and harmful behavior could then emerge from the surrounding agent rather than the model itself. He closes on Germany and Europe, arguing they must master this technology themselves and citing the LEAM initiative for a large European AI model, discussions with the German government and an industry roundtable over funding estimated at 350 to 500 million euros, and notes that many authors of major AI papers are European. In Q&A, he stresses that multilingual models are necessary precisely to cover under-resourced languages, such as Maltese with about 500,000 speakers, and argues human cognition's core lies in the interaction between associative and explicit memory, something current text-only systems could approach more closely by learning from richer data such as film and action sequences, and eventually through embodied, sensor-equipped robots.
Quotes
We have entered the paradigm of machine teaching.
Hans Uszkoreit
It's a probability machine. It's more probable that she is dead than she is alive.
Hans Uszkoreit
No human being can be consistent.
Hans Uszkoreit
Questions answered in this talk
Why do large language models give different answers in different languages?
According to Hans Uszkoreit at Rise of AI 2023, when he asked ChatGPT in Chinese why an application to join the Chinese Communist Party was rejected, it gave practical, helpful advice, but the same question in English produced a critical answer calling the party authoritarian, showing that model responses are not consistent across languages or prompt phrasing.
Why do language models sometimes state false facts confidently, such as claiming a living person has died?
According to Hans Uszkoreit at Rise of AI 2023, when he asked ChatGPT about a friend who is a singer, it correctly described her career but falsely said she had died in 2021; he attributes this to the model acting as a probability machine that fills a gap in recent data with the statistically more likely continuation rather than a verified fact.
What does Hans Uszkoreit mean by the paradigm of "machine teaching"?
According to Hans Uszkoreit at Rise of AI 2023, since the underlying transformer algorithm has changed little from GPT-1 to ChatGPT, the meaningful differences between systems now come from how they are taught, through pre-training, behavior training with human feedback, domain-specific training and fine-tuning, rather than from new algorithms.