At Rise of AI 2022 in Berlin, Sebastian Ruder, a senior research scientist at Google working on language processing and deep learning, argues that current NLP systems and large language models remain accessible mainly to English and a few well-resourced languages, leaving roughly 3 billion speakers of the next 1,000 languages underserved. He shows that multilingual model benchmarks look promising only because they average over the few languages that datasets actually cover, while most languages remain untested. Ruder outlines four directions needed to close this gap: computationally and data-efficient models that use inductive biases and adapter layers rather than ever-larger parameter counts; multi-modal models that draw on speech, radio and video since most of these languages have little text or web presence; evaluation on real-world data instead of translated datasets, which carry translation artifacts and the biases of the original English data creators; and treating languages as a continuum of varieties and dialects rather than fixed monoliths. He closes by describing the need to build data resources together with local speaker communities rather than through centralized annotation alone.
Key takeaways
- About 3 billion speakers of the next 1,000 languages are currently underserved by existing NLP systems, according to Ruder.
- Multilingual benchmark averages overstate progress because most datasets cover only a handful of the world's languages.
- Adapter layers let a single pretrained model support many languages by adding only small per-language parameter sets rather than full retraining.
- Wikipedia covers only around 300 languages, so most of the next 1,000 languages need speech, radio and video data rather than text.
- Datasets built by translating English data inherit translation artifacts and the biases of English-speaking crowd workers, distorting evaluation.
Detailed notes
The scale of the underserved-language problem
Ruder frames the talk around making NLP useful to speakers of the next 1,000 languages beyond English and a few major European languages. He notes the top 400 languages by speaker count each have more than a million speakers, and the next 1,200 each still have more than 100,000 speakers, adding up to about 3 billion currently underserved speakers. NLP research has historically been concentrated in English because it has been the easiest language to source and annotate data for, a picture largely unchanged over the past decade even with recent deep learning models.
Why multilingual benchmarks overstate progress
Ruder shows a benchmark his team proposed in 2020, averaged across nine tasks and 40 languages, showing steady improvement from early multilingual BERT models toward human-level performance. He cautions this average is distorted because most datasets only cover a handful of languages, leaving most cells in a language-coverage table red rather than green. Comparing multilingual models against English-centric models such as GPT-3, LaMDA and PaLM by share of non-English training data, he finds that even as models have grown larger they have not become more inclusive of the world's languages.
Computational and data efficiency
Because many low-resource languages also face practical constraints such as higher mobile data costs in poorer countries, Ruder argues that training ever-larger models is not the answer. Efficiency is needed in data (learning from very little data by incorporating inductive biases, since transformers alone need huge amounts of training tokens), in space (using adapter layers, small bottleneck parameters added to a frozen pretrained model so only a small parameter set needs to be stored per language), and in time (lower latency at training and inference). He gives the example of subword tokenization splitting words like the German equivalent of "excitement" in ways that do not match actual morphology, a problem that is worse for morphologically rich languages in Africa and Southeast Asia.
Multi-modality beyond text
Ruder notes that most of the next 1,000 languages have little text or web presence; Wikipedia covers only around 300 languages. Data for these languages instead exists in handwritten documents, undigitized books, lexicons, radio stations and YouTube content, and speakers often prefer spoken to written communication, sometimes without standardized orthographies. He describes joint training of text and speech models using unlabeled web and Wikipedia text alongside speech data from YouTube and labeled ASR data, and mentions a Google benchmark covering around 100 languages for speech recognition and speech translation.
Evaluation, translation artifacts and language varieties
Ruder criticizes the common practice of evaluating multilingual models on datasets created by translating English data, which produces "translationese" that differs from natural language and inherits the biases and topics of the original English-speaking crowd workers. He argues for building datasets in partnership with local speaker communities and researchers, and for human-AI collaborative annotation approaches given how few annotators exist for many languages. Finally, he argues languages should be treated as a continuum rather than fixed monoliths, citing varieties of Chinese, European and Brazilian Portuguese, and Indonesian languages with large lexical variation, and calls for models robust to different dialects and registers rather than forcing users into a perceived standard.
Q&A highlights
Asked about the origin of his interest in scaling NLP to more languages, Ruder traces the broader research lineage to Cold War-era machine translation efforts to decipher Russian messages, and says large models are now reaching a point of practical usefulness. Asked about the scale of annotation needed, he says it is hard to estimate and recommends prioritizing high-value tasks like translation, speech recognition and speech synthesis while combining targeted annotation with self-supervised learning on existing data. Asked about common sense reasoning, he says recent models already exhibit forms of common sense he would not have expected five years earlier, but still struggle with reasoning over information that is only implicit rather than explicitly stated in text.
Quotes
We are by far not talking about small languages, but all of these have very large speaker communities and potential users or customers.
Sebastian Ruder
Questions answered in this talk
Why do large language models still underperform for most of the world's languages?
According to Sebastian Ruder at Rise of AI 2022, models are trained and evaluated mostly on English and a small set of well-resourced languages, and even as models have grown larger they have not become more inclusive of the roughly 1,000 languages with significant speaker populations but little available data.
What are adapter layers in multilingual NLP?
Sebastian Ruder explains at Rise of AI 2022 that adapter layers are small bottleneck feed-forward parameters added to a pretrained model while keeping the original model's parameters fixed, allowing a single pretrained model plus small per-language adapters to serve many languages efficiently.
Why are translated datasets a problem for evaluating multilingual AI models?
Sebastian Ruder states at Rise of AI 2022 that datasets created by translating English text into other languages produce "translationese" that differs from natural language, and inherit the biases and topics of the English-speaking crowd workers who created the original data, distorting how well models actually serve real speakers.