At Rise of AI 2019 in Berlin, Jakob Uszkoreit, a deep learning researcher at Google Brain, argues that supervised learning will not scale to the complexity of problems AI should eventually solve, because it requires ever more complex and expensive labeled data, and because some domains, such as rare conditions in medical imaging, offer too few examples of any single case. He presents self-supervised learning as an alternative inspired by how humans learn by observation, centered on BERT, a bidirectional Transformer model trained by predicting words deleted from large text corpora. Built on the Transformer, an architecture his team began developing around 2017, and on Google's TPU hardware, BERT set new results on a range of natural language understanding benchmarks and slightly exceeded human performance on the Stanford Question Answering Dataset after fine-tuning, despite Uszkoreit stressing it does not understand the world. He then describes early Google research work extending self-supervision beyond text to video, including a model that learns to recolorize black-and-white footage by implicitly tracking objects, and a Berkeley robot-arm project used to train models to predict video continuations and basic physical interactions.
Key takeaways
- Supervised learning does not scale to complex outputs because labeling cost and complexity grow with task complexity.
- BERT is trained by predicting words removed from unlabeled text, then fine-tuned for tasks like question answering.
- BERT combines the parallel Transformer architecture with Google's TPU hardware, enabling large-scale self-supervised pretraining.
- A model trained to recolorize black-and-white video learned to track objects through occlusion without any labels.
- A Berkeley robot-arm project trains models to predict video continuations, learning basic physics and object interactions.
Detailed notes
Why supervised learning won't scale
Uszkoreit argues that as AI systems are asked to produce answers of increasing complexity, such as full scene descriptions rather than single labels, the labeled training data needed becomes correspondingly complex and expensive to produce. He points to medicine, particularly radiology, as an area with a long tail of problems where each specific phenomenon may have very few labeled cases, making the classic supervised approach uneconomic or impossible to extend to every new question. He also dismisses learning purely through interaction with the environment, as in reinforcement learning for board games, as unsuitable for domains such as medical diagnosis where trial-and-error learning on the job is too risky.
Self-supervised learning and BERT
Drawing an analogy to a child learning to tie shoes by watching rather than doing, Uszkoreit introduces self-supervised learning: constructing supervised tasks from unlabeled data. BERT (Bidirectional Encoder Representations from Transformers) is trained by removing words from large text corpora, such as Wikipedia and books, and learning to predict the missing words, forcing the model to learn grammar, semantics and world knowledge simultaneously. Once pretrained, BERT can be fine-tuned with much less labeled data for tasks like question answering, and became the best system on 11 or 12 standard natural language understanding benchmarks, slightly beating average human performance on the Stanford Question Answering Dataset (SQuAD).
Transformers and TPUs as the key ingredients
Uszkoreit traces BERT's origins to two developments: the Transformer architecture, which he says his team started working on around 2017 as an alternative to sequential recurrent neural networks, which computes representations for all words in a sentence in parallel rather than step by step; and Google's tensor processing units (TPUs), custom hardware whose parallel design fits this kind of computation. He notes BERT does not understand the world in a fuller sense, since it has never perceived space, time or the visual world, which motivates extending self-supervised techniques beyond text.
Extending self-supervision to video and robotics
Uszkoreit describes a Google research model trained to recolorize black-and-white video from a single color frame, which implicitly learns to track objects through occlusion and pose changes without any labeled data. His own Berlin-based team is training models on footage of a robot arm at UC Berkeley pushing objects around in a bin of toys, teaching the model to predict video continuations and thereby learn basic physics, object interactions and simple semantics, such as an egg cracking when poured over a pan. The stated aim is to eventually transfer such self-supervised visual representations to supervised tasks like scene captioning and language grounding, echoing the pattern already achieved with BERT for text.
Q&A highlights
Asked about bringing representations closer to human ones, Uszkoreit says machine learning models today perceive a far narrower slice of sensory input than humans and need broadening, while treating time as intimately linked to causal learning. On interpretability, he points to ongoing but still largely perturbation-based research as not yet producing fully actionable tools. He points to Been Kim as a name to look up for interpretability research within the wider Brain team. He confirms BERT learns word embeddings jointly rather than starting from pretrained embeddings like word2vec or GloVe, and cites question-answer relevance scoring as one of the most impactful internal applications of BERT-style models at Google. He cites Yann LeCun's framing that self-supervised learning is the new unsupervised learning, and expects visual self-supervision to eventually exceed supervised baselines given more compute and larger models.
Quotes
The workhorse of the vast majority of today's deep learning and in general machine learning deployments in the wild, namely supervised learning, won't scale.
Jakob Uszkoreit
It doesn't understand the world. It's never actually observed many aspects of the world the way we perceive it.
Jakob Uszkoreit
Questions answered in this talk
Why doesn't supervised learning scale for tasks like medical diagnosis?
According to Jakob Uszkoreit at Rise of AI 2019, supervised learning requires labeled data whose complexity grows with the complexity of the desired answer, and in domains like radiology many specific phenomena occur too rarely to produce enough labeled examples, making it uneconomic or impossible to keep relabeling data for every new question.
What is BERT and how is it trained?
According to Jakob Uszkoreit at Rise of AI 2019, BERT (Bidirectional Encoder Representations from Transformers) is trained self-supervised by deleting words from large amounts of text, such as Wikipedia and books, and learning to predict the missing words, then fine-tuned with labeled data for specific tasks like question answering.
Can self-supervised learning work beyond text, such as for video?
According to Jakob Uszkoreit at Rise of AI 2019, early research shows self-supervised models can learn to recolorize black-and-white video by implicitly tracking objects, and a robot-arm project at Berkeley trains models to predict video continuations to learn basic physics and object interactions, though this visual work does not yet exceed supervised baselines.