Rise of AI Conference 2022 · Berlin · Applied AI Stage

Scaling NLP to the World’s Languages

Sebastian Ruder · Google

At Rise of AI 2022 in Berlin, Sebastian Ruder, a senior research scientist at Google working on language processing and deep learning, argues that current NLP systems and large language models remain accessible mainly to English and a few well-resourced languages, leaving roughly 3 billion speakers of the next 1,000 languages underserved. He shows that multilingual model benchmarks look promising only because they average over the few languages that datasets actually cover, while most languages remain untested. Ruder outlines four directions needed to close this gap: computationally and data-efficient models that use inductive biases and adapter layers rather than ever-larger parameter counts; multi-modal models that draw on speech, radio and video since most of these languages have little text or web presence; evaluation on real-world data instead of translated datasets, which carry translation artifacts and the biases of the original English data creators; and treating languages as a continuum of varieties and dialects rather than fixed monoliths. He closes by describing the need to build data resources together with local speaker communities rather than through centralized annotation alone.

Key takeaways

  • About 3 billion speakers of the next 1,000 languages are currently underserved by existing NLP systems, according to Ruder.
  • Multilingual benchmark averages overstate progress because most datasets cover only a handful of the world's languages.
  • Adapter layers let a single pretrained model support many languages by adding only small per-language parameter sets rather than full retraining.
  • Wikipedia covers only around 300 languages, so most of the next 1,000 languages need speech, radio and video data rather than text.
  • Datasets built by translating English data inherit translation artifacts and the biases of English-speaking crowd workers, distorting evaluation.

Detailed notes

The scale of the underserved-language problem

Ruder frames the talk around making NLP useful to speakers of the next 1,000 languages beyond English and a few major European languages. He notes the top 400 languages by speaker count each have more than a million speakers, and the next 1,200 each still have more than 100,000 speakers, adding up to about 3 billion currently underserved speakers. NLP research has historically been concentrated in English because it has been the easiest language to source and annotate data for, a picture largely unchanged over the past decade even with recent deep learning models.

Why multilingual benchmarks overstate progress

Ruder shows a benchmark his team proposed in 2020, averaged across nine tasks and 40 languages, showing steady improvement from early multilingual BERT models toward human-level performance. He cautions this average is distorted because most datasets only cover a handful of languages, leaving most cells in a language-coverage table red rather than green. Comparing multilingual models against English-centric models such as GPT-3, LaMDA and PaLM by share of non-English training data, he finds that even as models have grown larger they have not become more inclusive of the world's languages.

Computational and data efficiency

Because many low-resource languages also face practical constraints such as higher mobile data costs in poorer countries, Ruder argues that training ever-larger models is not the answer. Efficiency is needed in data (learning from very little data by incorporating inductive biases, since transformers alone need huge amounts of training tokens), in space (using adapter layers, small bottleneck parameters added to a frozen pretrained model so only a small parameter set needs to be stored per language), and in time (lower latency at training and inference). He gives the example of subword tokenization splitting words like the German equivalent of "excitement" in ways that do not match actual morphology, a problem that is worse for morphologically rich languages in Africa and Southeast Asia.

Multi-modality beyond text

Ruder notes that most of the next 1,000 languages have little text or web presence; Wikipedia covers only around 300 languages. Data for these languages instead exists in handwritten documents, undigitized books, lexicons, radio stations and YouTube content, and speakers often prefer spoken to written communication, sometimes without standardized orthographies. He describes joint training of text and speech models using unlabeled web and Wikipedia text alongside speech data from YouTube and labeled ASR data, and mentions a Google benchmark covering around 100 languages for speech recognition and speech translation.

Evaluation, translation artifacts and language varieties

Ruder criticizes the common practice of evaluating multilingual models on datasets created by translating English data, which produces "translationese" that differs from natural language and inherits the biases and topics of the original English-speaking crowd workers. He argues for building datasets in partnership with local speaker communities and researchers, and for human-AI collaborative annotation approaches given how few annotators exist for many languages. Finally, he argues languages should be treated as a continuum rather than fixed monoliths, citing varieties of Chinese, European and Brazilian Portuguese, and Indonesian languages with large lexical variation, and calls for models robust to different dialects and registers rather than forcing users into a perceived standard.

Q&A highlights

Asked about the origin of his interest in scaling NLP to more languages, Ruder traces the broader research lineage to Cold War-era machine translation efforts to decipher Russian messages, and says large models are now reaching a point of practical usefulness. Asked about the scale of annotation needed, he says it is hard to estimate and recommends prioritizing high-value tasks like translation, speech recognition and speech synthesis while combining targeted annotation with self-supervised learning on existing data. Asked about common sense reasoning, he says recent models already exhibit forms of common sense he would not have expected five years earlier, but still struggle with reasoning over information that is only implicit rather than explicitly stated in text.

Quotes

We are by far not talking about small languages, but all of these have very large speaker communities and potential users or customers.

Sebastian Ruder

Questions answered in this talk

Why do large language models still underperform for most of the world's languages?

According to Sebastian Ruder at Rise of AI 2022, models are trained and evaluated mostly on English and a small set of well-resourced languages, and even as models have grown larger they have not become more inclusive of the roughly 1,000 languages with significant speaker populations but little available data.

What are adapter layers in multilingual NLP?

Sebastian Ruder explains at Rise of AI 2022 that adapter layers are small bottleneck feed-forward parameters added to a pretrained model while keeping the original model's parameters fixed, allowing a single pretrained model plus small per-language adapters to serve many languages efficiently.

Why are translated datasets a problem for evaluating multilingual AI models?

Sebastian Ruder states at Rise of AI 2022 that datasets created by translating English text into other languages produce "translationese" that differs from natural language, and inherit the biases and topics of the English-speaking crowd workers who created the original data, distorting how well models actually serve real speakers.

Mentioned

and inherit the biases and topics of the English-speaking crowd workers who created the original data

Keep exploring

More from Rise of AI Conference 2022

More on Research & Frontier Models

Transcript

Auto-generated transcript, not edited. · 4,897 words

Show the full transcript

and now i would like to welcome our next speaker Sebastian Ruder welcome to the stage and another warm welcome sebastian is a senior research scientist at google and he his main focus is on language processing as well as machine learning and deep learning and now i'm curious to learn a bit more about scaling nlp to the words languages with its challenges and opportunities and yeah now it's your stage thanks so much and hello everyone it's my pleasure to be able to talk to you today oh yeah awesome thank you and yeah i think we actually have a good good segue from the previous talk because my talk mainly is going to be about how we can unlock the benefits of language technology in general and of large language models in particular not only for english or a few european languages but for most of the world's languages so the main question at the heart of my talk is how we can make the current mlp systems and current models that we have available accessible and

useful to the speakers of the next thousand languages in the world and just to give you a bit more more context on that if we look at the next 400 languages or the top 400 languages with most speakers in the world all of these have more than one million speakers and going beyond that if we look to the next 1 200 languages with most speakers each of those languages still has more than 100 000 speakers so we are by far even though a thousand languages sounds like a huge amount we by far not talking about small languages but all of these have very large speaker communities and potential users or customers so we might be um that we could serve with our language technology and ultimately looking at these 1000 languages in total these make up about 3 billion speakers that are currently underserved or that are not able to use existing technology as it is currently available today and for a bit more broader context traditionally research in natural language processing nlp has been

typically done in english because in english it's a language of science and it's typically where most of the data was easiest to create and to annotate and practice and this picture this was the case 10 years ago and hasn't changed dramatically even if you look at recent literature in the field that uses deep learning and machine learning based approaches there is kind of a more recent trend that trains these large language models that you've already heard about before in an increasing number of languages in a multilingual setting and these models you can think of are basically um kind of analogous to models that are only trained on english like gpd3 now instead of just training them on a single language like english these are being trained on around 100 different languages typically on data from the web or on wikipedia and to give you a bit of sense of how progress in this area of training these large scale multilingual models have looked like you can see here the performance on a benchmark that we

proposed in 2020 for assessing the generalization ability of these large-scale multilingual models and as is the case with a lot of recent research in nlp on training and pre-training these large models we typically want to evaluate them not only on a single task or a single application but on benchmarks that cover many different tasks many different capabilities at the same time so this so the numbers that you see here are basically the average over nine different tasks and with each task covering a number or among a set of 40 different different languages and here if you just look at this picture we can generally see quite an optimistic or promising trend in that we've seen kind of from the earliest large scale multilingual models like multilingual birds which were basically just multilingual extensions of models that were very successful for english we can see kind of steady improvement on this benchmark getting edging ever closer to human level performance or at least performance achieved by human annotators on these very specific downstream tasks for natural

language processing so just by looking at this figure of progress we might be able to or we might speculate that we might be getting close to actually training very well performing and training multilingual models that will actually be useful and beneficial for many languages of the world however just looking at this average really provides only a distorted picture of progress because many of the tasks that are available in this benchmark only and in this benchmark as well in most datasets that are used in nlp research and practice only cover very few languages among them so here you can see on the right side the language coverage of many standard datasets on the top with the 14 languages of this benchmark on shown with each row and you can see that only for a few tasks you actually have green green cells meaning that this language is covered by the data set for a few of the data sets with most languages actually not being being covered by only one or two of these data sets

so just by looking at kind of the performance on existing data sets or benchmarks we can only tell that we are doing pretty well and we are training models that can do well on languages with large amounts of resources like english or like other european languages or chinese but we can make a much weaker statement about most of the world's other languages with which have much less data available to date and specifically if we look at languages with a lot fewer data such as measured in terms of the number of wikipedia pastors which is kind of a standard resource for training these large-scale models we can see again here mostly red so most datasets actually do not even cover most of these languages and so taking a step back from these multilingual models and looking at them in relation to perhaps the mainstream of nlp research which focuses on training or utilizing large scale models that were trained in english you can see here on this diagram basically the representative recent models both multilingual and

trained on english in terms of the fraction of non-english training data that was used when retraining them so the multilingual models will be very much at the top because most of the data is going to be in other languages and only a small subset of the training data may maybe in english while most of the english based models are going to be at the bottom because either they are exclusively trained in english or only cover a little bit of data in other languages and so you can see here models like gpd3 that was mentioned before but also more recent models like lambda or pound from google which fall under this more english-centric category so in general you can really identify basically two streams of current lp research models that are particularly and dedicatedly trained in this multilingual setting and the mainstream of more english-centric models and so far unfortunately or i think the unfortunate observation here is that even the more recent and as models have grown larger they have not really become more inclusive

of the world's languages so they're still mostly overly focused on english even for the largest recent models and for the for the remainder of the talk i want to outline a couple of um opportunities and challenges to hopefully over time um as a community both in research but also an application that we can consolidate these two research streams on pre-training large models so that we can ultimately not only train models just for english and specific models for other languages but that we can train models that are able to serve hundreds or even the next thousand languages with the same model and on that note in terms of talking about challenges and on opportunities kind of the main facets i think that are generally important when talking about these settings in machine learning and ai in general um regards kind of the models and the architectures we want to build but also the data that we want to feed into these models and specifically on the modeling side i'm going to be talking about training more

computationally efficient models and also models that cover and span multiple modalities while on the data side i want to emphasize the importance of evaluating on real world relevant data and covering data that includes different language varieties so on the computational efficiency side i've talked so far about these languages having a little amount of data in practice available so often these languages are also referred to as low resource languages because they have few data sets that are machine readable and that are available for training with current models but in practice the the settings where we would like to apply models or use applications that will be able to serve speakers of these languages also typically have other constraints so for instance as you can see on this diagram here the cost of mobile data is very different across different countries and particularly for poorer countries or countries in africa there is a comparatively higher cost of mobile data which makes it difficult for instance to deploy larger models on devices or perform on-device computing as well

so in order to really preserve these languages we must need and serve these languages we must be able to take these computational constraints into account as well and the overall direction of just training larger and larger models even though these may be useful for some applications ultimately like having a model that contains 1 trillion or 10 trillion parameters even though that might be very multilingual in the end might not actually be able to be useful for actual customers in the real world so in this note i really think as a community and in general we need to work on designing and training models that are more efficient along different dimensions and in particular efficiency both comes in being more efficient in terms of the data that we're feeding into the models and that we can use for training these models because in practice trying to annotate or obtain data for up to a thousand languages is very expensive or even infeasible in practice particularly looking at different practical applications so we really need to design

models that can learn with very few data as possible and here it can be become increasingly important to try to incorporate inductive biases into the models so even though in in general kind of in nlp and in ml in general the trend really has been going towards making models as general as possible so from recurrent models that still have kind of this recurrency biases we've moved to transformers which are much more general function approximators and which can learn much more complex relations from just data directly and we've seen the power of these models when you feed into them millions or billions of token of training data the amazing feeds and capabilities that they can acquire by being provided with huge amounts of data but because we do not have access to huge amounts of data for most of the world's languages we actually need to encore encode and incorporate more prior knowledge into these models again and here just to highlight one one potential example you can see at this table the tokenization or the

segmentation that kind of a state-of-the-art model assigns to the word excitement in different languages and typically these models like bird or gpd3 as well have a subword-based component as an initial input preprocessing step that splits input words kind of based on a frequency model that is learned previously from large amounts of data and you can see here that for words like excitement in english that might be represented very often in the data because english language data is typically quite overrepresented the model might assign a single word to that word in vocabulary but for other words that occur less in the data like for from different european languages or languages that have more and more themes these words might be split along different boundaries that again the model has learned based on frequency and in practice most of these if you look for instance at the german word might uh not actually really consist with actual morphology or with how you would intuitively try to split this word into its different components or more themes so

here we and this is even a larger problem for languages that are morphologically rich like many in africa or southeast asia as well where we simply don't have a lot of data and where the segmentation models are much less accurate to secondly talking about space efficiency i've mentioned that these huge models are simply not not useful not practical for many of these settings so trying to learn language specific parameters is i think quite an important direction in this area and as one potential approach there is this paradigm of using adapter layers which you can kind of see in illustration of here which are basically small bottleneck parameters consisting simply of a feed forward down and an up projection that are added to an existing pre-trained model and the nice thing about these parameters is that you can keep the entire parameters of the pre-trained model fixed and only learn these very small amounts of additional parameters so in practice if you want to have a model that can perform well on different tasks you only

need to store a single copy of this pre-trained model and only a small set of additional adapter parameters and finally in terms of time efficiency we also need more efficient uh models that are more that have lower latency and can perfor faster at inference time or also during training the other aspect i want to emphasize is the need for multi-modality if we look towards the next thousand languages the data that we've mostly been using so far training these multilingual and monolingual models for english and other languages is data from the web or from wikipedia all in the form of text but looking at the next thousand languages most of these are actually not available or have very little web presence wikipedia itself only covers around 300 languages and about the similar number are represented on data on the internet as well so these languages however are by no means data poor the data is just in different formats that we so far have been avoiding or ignoring in terms of processing it with our models

so the data for these languages might be contained in handwritten documents in books that so far haven't been digitized in lexicons that contain translations to other languages or in completely different formats many languages have vibrant radio stations and a lot of different radio stations in their country as well as strong communities and other platforms like youtube online and for instance you can see here just some examples of different youtube music videos in some of these next thousand languages and in addition for many languages as well it's more common to actually speak them to write to write in them maybe people have not grown up with using computers or using smartphones or the the countries might have little interpenetration or many of these languages might also have non-standardized autographies so the way things are spelled might vary from speaker to speaker so people generally prefer the spoken word so in general in order to make progress really for these next thousand languages we really need not only to train models that are multilingual but also multi-modal

and that are able to use data from these diverse range of different sources and just to highlight the current um two kind of current approaches or directions in this area the first approach is training kind of jointly training text and speech models um so both using kind of huge amounts of unlabeled text data from wikipedia and from the web but also at the same time using speech data from youtube or other sources that are available kind of in the unsupervised setting here as well as using kind of labeled data that is available for for training asr models kind of in a joint fashion in this multi-modal setting and to go hand-in-hand with these multi-lingual multimodal models we also need new benchmarks that are able to evaluate text and speech models across different languages and that cover many different languages as well and here's just one example of a recent benchmark that we worked on at google that covers these multi-modal text and speech-based models where the benchmark itself covers around 100 different languages in its

component tasks and covering things like speech recognition but also speech translation for instance the third aspect i want to highlight here and one kind of limitation of the current existing evolution landscape that these models are being evaluated in is the frequent use of translations for evaluating molecular models because obtaining actual natural language text in many languages at scale is very difficult and very expensive existing data set creators typically resort to using translation so translating an existing english data set to a wide range of different languages and in practice this has a number of different side effects the first one being that the language that is produced through translation which is known as translation ease actually differs in many subtle and not so subtle aspects from what is more regularly known as natural language so language that is kind of uttered and produced by speakers kind of in natural environment so if we evaluate data sets that are only produced through translation we're actually not able to get a really completely accurate reflection of how these

models would fare in the real world and the additional and i think perhaps bigger issue is that by translating existing data sets these data sets inherit the biases in the viewpoints and perspectives that are encoded by the people who created the original english based data which are typically crowd workers based in the us or other english-speaking countries so these data sets might actually not reflect the the concepts the the queries or the areas of interest that are actually relevant and important to the speakers of these other languages so in order to really make progress for for training and actually deploying re models that can be useful and relevant in the real world to actual speakers we also need to be able to design challenging data sets that are practically relevant for evaluation in these different languages and here in particular i want to emphasize because again it's simply not feasible for a single company or a single entity to annotate data in a thousand languages potentially so you really need to be able to build

and work more closely with local speaker communities and particularly researchers and practitioners that are working and natively speaking those languages and work with them more closely in actually building these resources that are relevant in in their local context and the other aspect is for the speakers and the annotators that we have available we also need to design more effective and efficient means to actually use their expertise and their language knowledge for annotation because for many for many languages it's very difficult to find annotators in practice and there may only be very very few annotators that we might have access to so we really would like to maximize the benefits that we can gain from using their time and here in particular there are both strategies like active learning but given kind of the capabilities of large-scale models which can also generate different forms of data we can imagine kind of different human and ai collaborative designs to kind of jointly annotate data where the ai model might propose different ways or different topics for generating

the data that might be further refined by human animators and lastly the the final aspect that i want to highlight is not just looking at languages as monoliths as kind of independent single entities that are that are separate and do not relate to each other but thinking of languages more as a continuum and as something that is maybe also in practice not often not very clearly defined so kind of a famous quote is that language is just a dialect with an army and a navy so in practice languages are most often a product of the socio-political environment and many many varieties that are not considered as separate languages are still spoken by tens of millions of speakers like different varieties of chinese for instance which are not considered and not treated currently separately in existing research but which have by themselves speaker communities that rival many other languages like the ones we have in europe here and similarly um portuguese as well portuguese is typically treated as like as a single language as well even

though portuguese portuguese has a spoken portugal and brazilian portuguese are very different as well and you would really like to have brazilian portuguese speakers in particular would really like to have systems that can actually serve and deal with their particular variety of portuguese and this extends to more local and other forms of dialects which also have large amounts of lexical variation among them such as for indonesian languages so here again we don't really ultimately i think the goal should be that we don't want to force users to adopt a variety or like a perceived standard setting of a language they may not be completely comfortable with or that may not actually be the language they would prefer speaking in but that we would try to meet them actually where they are in the language that they really feel most comfortable speaking and in the most relevant setting to them and on that note i think they're still particularly in this area a lot of a lot of improvements and a lot of work that needs

to be done as a first step we can think of simply making such information about dialect and styles and other registers more easily available in current data sets and beyond that so for instance here's a table with some information about the written language variation for different local indonesian languages which have very different standardized photographies and ultimately we should really aspire to train models that are robust to different language varieties not only when we feed data in different languages or language varieties into them but that can also generate data in different varieties such as for translation or if you want to converse with a dialog assistant for instance so with that i just want to end here again with these four kind of tenets or i think important directions that i outlined and i hope i was able to give you kind of a sense a bit of where current research is at in this area of multilingual nlp and about some of the areas that we still need to make progress in yeah thanks a

lot for your attention thank you sebastian really really interesting i always love when when something that seems natural and becomes so technical so that you uh yeah try to analyze a language and i was just wondering where does the original idea come from that you started scaling nlp in the worst language what was the the origin was it does it come from chat board or conversational ai or coding um language or where does this come from um you mean my like my personal interest in this direction or generally kind of this way when you know when you know it um sure i mean i i well i think it's quite a natural setting um i maybe i think the the earliest traces that you can trace that to might be to the cold war which is where um kind of a lot of like machine translation technology had its beginnings um being able trying to decipher uh like russian messages uh which led kind of the first developments of like very very basic than um

english russian um kind of translation systems and so ultimately i think there's um i think like in ai and like language in particular i think always the kind of goal or this hope to be able to like understand other cultures and be able to converse with people um kind of from like many different languages at the same time so i think it's something that has been kind of very very core to like kind of research or like curiosity um and i think uh i think it's encouraging that recently with these like large scale models i think we're now getting kind of to a point where we can actually see um like practical improvements or where we can see like translate methods actually being uh useful yeah to be used in practice in many languages or yeah yeah interesting and we we were just talking before and you already mentioned that i mean thousands of languages that means a long term uh project right so a lot to do are there any other questions in the

audience yes sam for your talk uh do you have any idea how the like how long the scale and the scale needed for these annotations and if any shortcuts exist like if certain things can be translated and then maybe there are some like cross language translations that can be used to help scale things um yeah yeah i think that's a great great question um i think uh yeah well at this point i think it's still kind of hard to um how to estimate or hard to like speculate what what is ultimately um needed um to like uh to really completely fulfill on this black vision i think the the best thing that we can we can do at this point is to um kind of identify or prioritize the tasks that are most useful those are things like um translation like speech recognition or speech synthesis and try to collect kind of high quality data for as many of these languages as we can and at the same time so kind of this very dedicated

task driven data annotation combined with trying to leverage data that is already available that can be learned from using similar supervisor and sewers methods and combining these two different sets of kind of sets of data um to train uh better models ultimately so i think yeah we both need uh to work on the data annotation side but also in terms of like learning from from more yeah from existing available data and trying to make that more useful great thank you another one yes you're in the middle of course for many applications one is needing something like common sense so now you just have language and it has imagination as it was so and you can figure out how to put the common sense in yes yeah i think a common yeah common sense as i think as like hans mentioned also in this talk is i think a big uh big research area and a big research direction in the in the general community um i think it's something that like if you had asked

me like five five years ago looking at what these recent models can do they actually are already able to exhibit uh like aspects of certain sense that i wouldn't have um of common sense i wouldn't have expected them to have like five years ago so i think even without explicitly including certain certain common sense of world knowledge these models are able to do surprisingly well and things that we previously considered to require common sense for solving them but of course uh there's uh many different forms of reasoning that these models still cannot do or particularly when it comes to capturing uh kind of things that are implicitly represented and not explicitly mentioned in in text or data these i think are still things that these models have a hard time and struggle with and oh yeah that still requires i think a lot of a lot of methodology or trying to think and design ways to actually make them perform better or incorporate this knowledge in some way into these models um yeah i think

you you will still be here a bit right oh yes for for further questions uh so thank you for being here uh i would like to take this one okay yeah so bigger pause to sebastian

Rise of AI 2027 · Berlin · May 2027

Be in the room for the next one.

Ten years of talks started as conversations in one curated room in Berlin. The next edition brings 300 selected leaders together in May 2027.