At Rise of AI 2022 in Berlin, Jonas Andrulis, a serial entrepreneur whose latest company is Aleph Alpha and a former AI R&D manager at Apple, asks what counts as work and how AI is already changing it. He argues humanity's work has shifted from basic survival tasks, now largely automated, into knowledge work, innovation and entertainment, and cites an Open Philanthropy report estimating transformative AI around 2030. He explains large language models as "world models" trained to predict how any piece of human-written text continues, then demonstrates Aleph Alpha's model answering factual questions, translating, and reading a company's quarterly report to answer a free cash flow question. He shows multimodal extensions built at Aleph Alpha's Heidelberg lab, which he says formed part of the basis for DeepMind's Flamingo, including image reasoning, retained world knowledge, and even a made-up word the model produces. He also demonstrates clear failure cases, including inconsistent and language-dependent answers and a wrongly generated image, and closes by predicting that intelligence, unlike cloud infrastructure, will consolidate into general layers owned by a few key players.
Key takeaways
- Andrulis argues human work has shifted from direct survival tasks into knowledge work, innovation and entertainment, and AI is already branching into most of these.
- He cites an Open Philanthropy report estimating transformative AI, capable of doing most current work and driving scientific innovation, around 2030.
- He describes large language models as world models of human language that predict text continuation, and says their ability to do translation or factual retrieval surprised him.
- He demonstrates that the same model gives different, culturally shaped answers depending on prompt language, and gives inconsistent answers to near-identical questions.
- He predicts that unlike cloud infrastructure, AI intelligence will consolidate into general intelligence layers owned by a small number of key players.
Detailed notes
From survival work to knowledge work
Andrulis opens by tracing how human work moved from direct survival tasks, such as gathering food and building shelter, to work built around knowledge: collecting and preserving knowledge, making it accessible, innovation, and entertainment. He argues AI is already branching heavily into at least two of these three areas, with innovation still an open question. He cites a report from Open Philanthropy that defines transformative AI as technology able to do most of what is currently called work, potentially reorganize the global balance of power, and do the work of human scientists, and says the report estimates this arriving around the year 2030.
Language models as world models
Andrulis describes language models as systems trained on essentially all human-written text, from books to chats to scientific papers, learning to predict how any cut-off passage would continue, similar to phone keyboard or search-engine autocomplete. He argues this simple training objective produces something he calls a world model, not of the physical world but of the structure of human language, which he says encodes knowledge, experience and what people care about because language itself was built to be useful for exactly those things. He demonstrates Aleph Alpha's model completing the prompt "Angela Merkel was born in" with the correct answer, Hamburg, and producing a German translation of an English sentence, noting that a model trained only to predict text continuation turning out to do factual retrieval and translation surprised many people, himself included.
Applying the model to real documents and interviews
Andrulis shows the model being used to search and transform documents: uploading HPE's Q1 2022 quarterly report and asking about the free cash flow guidance for fiscal 2022, to which the model answers a range of 1.8 to 2.0 billion, which he contrasts with older BERT-based extractive question-answering models as far more capable. He also describes copying a question from a print interview, about when people will be able to have a dialogue with an AI the way they were interviewing him, directly into the model with only the two prior question-answer pairs as context, and reading out the model's unedited answer about gradually accepting new technology and early intelligent assistants in medicine and transportation.
Multimodal models: reasoning, world knowledge and limits
Andrulis presents multimodal models built at Aleph Alpha's Heidelberg lab, which he says formed part of the basis for DeepMind's Flamingo, extending the language model to accept prompts combining text and images while keeping its multilingual capability. Examples include correctly describing an image of an apple with an "iPod" label attached, a reference to a known CLIP adversarial-attack case; identifying that a dog or a spilled drink caused a stain on a carpet, which he calls a light form of reasoning; reading and answering about a Black Lives Matter sign even when part of it is cropped out of the image, which he attributes to retained world knowledge; producing an invented word, "shovelzer," for an ambiguous object that sounded plausible to him as a human reader; and reading a hand-drawn treasure map, including implicitly recognizing that a red X marks the treasure and translating its answer into German when asked in German. He also references PaLM explaining a joke involving two trucks and notes he would not have believed two years earlier that AI could handle this kind of humor. Responding to a conversation partner's argument, from a separate prior discussion, that a system is not AGI until it can go into the woods and chop down a tree, he shows the multimodal model correctly explaining how to start a car from a photo of an Audi cockpit and previews an explainability feature showing how strongly each prompt word, such as "key," influenced the answer, suggesting such models could eventually connect to robotics.
Consistency, bias and failure modes
Andrulis demonstrates that the same question about a favorite sports team returns different, culturally plausible answers depending on the prompt language, Bayern Munich in German and Olympique Marseille in French, while GPT-3 answers the German prompt with the New York Giants, which he attributes to its English- and US-centric training data; he stresses these models have no opinions or agency, only likely completions shaped by training data, sometimes invisibly. He also shows the model giving inconsistent answers to near-identical questions about the most beautiful German towns, arguing the models lack the self-consistency people have and should be treated like a web search rather than a source of truth. He closes the failure examples with an image generation prompt for "a horse riding an astronaut" that instead shows an astronaut riding a horse, and a multimodal caption reading "no regrets tattoo" that is almost but not quite correct, both errors he compares to plausible human misreadings of something seen only vaguely.
Closing: infrastructure versus intelligence layers
Andrulis ends by contrasting the pre-cloud era, where infrastructure and intelligence functionality were separate from product-specific solutions, with today, where hyperscalers own a powerful, easy-to-use infrastructure layer while AI intelligence remains fragmented, with everyone fine-tuning their own models. He predicts this will consolidate into general intelligence layers owned by a small number of key players.
Quotes
You should never think of these as having an opinion or an agency, right? They're just giving pattern actions that are likely.
Jonas Andrulis
These word models also have another difficulty with things that are almost known.
Jonas Andrulis
Questions answered in this talk
What does Jonas Andrulis mean when he calls language models world models?
At Rise of AI 2022, Andrulis explains that language models trained to predict how any piece of human text continues end up encoding the structure of human language, including knowledge, experience and values, so he calls them world models of language rather than of the physical world.
What was the Open Philanthropy timeline for transformative AI that Jonas Andrulis cited?
According to Jonas Andrulis at Rise of AI 2022, a report from Open Philanthropy estimates transformative AI, defined as technology able to do most current work and drive scientific innovation, arriving around the year 2030.
Why do Aleph Alpha's models give different answers depending on the language used?
Jonas Andrulis showed at Rise of AI 2022 that asking about a favorite sports team in German returns Bayern Munich while asking in French returns Olympique Marseille, and explained this reflects patterns in the training data for each language rather than any opinion or agency held by the model.