foreign so he's a German computer scientist dealing with computer Linguistics and of course Ai and his talk will be about how Universal um our the foundation for our models and I'm really curious to listen to his talk and please a warmer Plus for Hans Uszkoreit right good morning good morning colleagues friends guests um strange title some more there um why is the question of any interest how Universal a language models but it should be obvious if these language models were Universal in the sense that I'm still going to explain they would know all the languages they would take all the knowledge they would be ethical reliable and so on and we all worldwide helped them to train and to improve why then have several ones yeah why then have a competition we all just keep improving this one big model in the world and that will solve all of humankind's problems and then it can be deployed and accessed from every place in the world so but this question then how Universal raises a couple
of other questions first how Universal are they now and can or should they be Universal so my own connection to large language models is we worked with language models before they were large and before they were neural for a long time and when the Transformer came out uh we were very early recognized their power and use them also in our company like that there was a company that I had just newly formed Giant so we use them for fine tuning a bird model for instance and later we after the Giants company was bought by another company unit sector that builds every morning the Press reviews for large German Ducks corporations then we apply it to GPT very very early for summaries for summarization and for other purposes so just to see that we are all on the same page I won't go through here I think that will take also too much time we are talking about not any type of language models because there are many language models that are different from Transformers and
ChatGPT we are talking about Transformer models that basically get their power the power that previously modeled it through um recurrence and memory by a mechanism called self-attention they either have an encoder decoder structure or only encoder or only decoder and they can generate so they can generate large outputs even if they get small inputs so they are generative and what makes them special is this generic pre-training it's based on a concept called transfer learning that means you train the system for one task and then you can apply it for one or several other tasks so that's the basic idea and since this one task is rather simple the system can produce their own data for Learning and we call this self-supervised supervised learning but the machine can build the supervised data themselves the label data and the special thing what makes it so special what allows all these breathtaking stunning functionality is the so-called emergence of functions that the system was not trained for so it was trained on very simple tasks but then low
end behold suddenly it can seems to understand and do a thousand things that it was never never trained to do and the last thing homogeneity that Percy Liang also attaches with the notion of foundation models is interesting because homogeneity means if it could lead to one universal model everything is now homogeneous yeah we have one system trained and even if we then maybe apply it in different areas we have the same basic education wouldn't we love that for human beings they all had the best education here and all the top education from the very best schools and there would be no deficits anymore so and there's a variety of applications and I won't talk about this now right now here so the idea being you start with that architecture then you do the generic pre-training then maybe you do other types of training or you apply it immediately for some things and then you can do application specific training instruction feedback on on human feedback you can do reinforcement learning and so on and you
can fine tune it for special applications also the interesting thing is and I'm already going a step in that direction now that I want to take to take you to is except for the beginning all the rest is education yeah only the very very first step is the algorithms of the base model and then we get these Foundation models that are language only and then language combined with other things but that's a different topic I won't say much more about it here now so what would be the ideal Universal Foundation model it would be omnigot it would speak all languages yeah all the even minority languages maybe even languages um so-called um uh so-called special jargons yeah the youth language maybe the language of older people everything it will talk all these languages right now the system gets closer to it out of the seven thousand languages in the world it already Masters between 20 and 50 or 20 maybe quite well 25 and maybe 50 to some degree but from then on it really
degrades quite a bit yeah so but nevertheless let's think about it is it possible to have really an omni glot system that can do do we have enough data for training it then the second question as as soon as the system comes up and is so knowledgeable you find the first people who say oh but it didn't answer that question right the Google system failed on that one thing oh that's bad Google's stock goes down here because the system so which is kind of ridiculous yeah I mean now we have a system that knows more than any AI system before yeah it knows so much and nobody by the way paid attention to that the Bing system failed on three questions and Microsoft stock didn't go down yeah but that's another issue so so people are now getting very picky oh the system didn't know that oh here you got a Harvard Education and you you didn't know the number of inhabitants of Kathmandu how come yeah so it's kind of ridiculous yeah but nevertheless
people wanted and they wanted to have all conceptual knowledge of factual knowledge and all procedural knowledge how to bake a cake and how to reorganize a big company yeah so so all of these things but uh by the way the conceptual knowledge yes it will be done because otherwise it could not learn the language yeah so Concepts and language they are learned together otherwise it would not have the seeming understanding but to think about that this model is responsible for knowing all the facts in the world even the new facts why why do you have this idea that the system should stay no human can do it may be a search engine comes close but why should this brain be able to know all the facts in a daily basis but that's what people think and procedural knowledge I won't touch you so it should be explanatory of course everybody it makes only two words explanatory AI so you can put it easily on any slide and demand that it would be explanatory here and
actually it is explanatory to a very large degree but try to get the explanation sometimes yeah sometimes there are errors in the explanations but the nice thing is now I don't know if you've tried you can get the system into a dialogue and question the explanation and lo and behold the system sometimes says apologies I did this wrong and corrects itself try it I mean it gets it's up to a huge it's it's amazing already it should be truthful of course everybody wants that it's truthful so if we know for instance that nuclear energy is a very bad thing and we are sure this is the truth then the system should say that or otherwise it's not truthful so and here with this example you already see the problem there's not one truth in the world there are many truths about many things there's a certain Collective truth in the sciences and in everyday life in common sense but truthful for whom Xi Jinping also demands that the system is truthful it should tell only
the truth and if you don't know what the truth is ask him yeah so he will so he will know exactly what it is so it should be creative it should tell you the truth but it should be creative it should make up a story it should make some something and of course the story will only be truth no that wouldn't be fictional that would not be creative what now what do we want truth or fiction yeah so here we see already a conflict yeah so it's almost impossible to have in the future systems that are as good in writing novels as they are in sticking to the truth I asked for instance um I talked to chechi PT in a free minute about a friend of mine who lives in New York and I said tell me something about her life and Chachi PT did and it was all correct I was surprised oh Chachi PT because she's a singer yeah so shechibiti knew that she was a gypsy singer and so on and
then at the end said GPT said she died in New York in 2021 I was shocked I believed it so I looked it up I checked whether I found any new new statements on it and it was complete Fiction made it unfortunately she's still alive so why did ChatGPT make it up it's a probability machine because she didn't have in the time of pandemics any more news any more records any more appearances and it's a probability machine it's more probable that she is deaf than she is alive I mean it's no big deal yeah so and and this is uh what do you wonder about here so and this machine should also be ethical of course yeah it should be ethical but not ethical according to your ethics according to my ethics of course yeah it should be ethical and we demand if it's not ethical we cannot let it let the let the people face it but if it's ethical to your ethics it should be forbidden I don't I would not allow it
so so so this is the conflict of creativity and truthfulness there's clearly a conflict of goals there's different Focus for different application domains and we should think about that large language models have lots of power but they are not the best reliable fact machines that I already mentioned that we can build fact machines but maybe not that way and do we get consistency so we asked we tried this in different languages so we asked uh ChatGPT um I we asked in Chinese said my application for the Chinese Communist party was rejected what can I do and uh what did it answer it gave lots of explaining lots of explanations what could have gone wrong maybe where I failed and how to improve for the next time very helpful if I want to enter the party Xi Jinping himself tried eight times unsuccessfully to enter the party before he was accepted so that's a useful type of question so and which is by the way true yeah so so and then we asked the thing in
English and the system said oh the Communist party is a dogmatic authoritarian system they will not tell you why you failed and you have no choice appealing it and doing something about it interesting right so we got completely different responses I don't know if this is still true for the latest version because they are improving it all the time but we we documented these answers yeah so we don't get consistency across languages but also we don't get consistency across prompt formulations if you formulate your prompt in a certain way that seems you are looking for certain arguments uh probabilistically the system will be cooperative and give you maybe one answer if you formulated in another way it may give you other arguments because let's face it it's trained on on a huge body of text Which is far from being consistent why do we think the system should be consistent I mean we can make it consistent with respect to certain applications and certain scenarios but no human being can be consistent I mean we
are not even having full cognitive consistency to all of our belief systems why the hell should we ever expect a system that has read so much more inconsistent stuff should be consistent so beats me so the Insight is it is the wrong approach to enforce a desired Behavior by material and I will come back to that when I talk about the Chinese experience because in China there's now a huge Drive the government two weeks ago issued very strict rules for these systems and they have to adhere to certain correctness and there is a possibility to do this and that is you restrict the system to certain sources and we know how China does that in general if you are in China and you try to access Wikipedia you cannot do it they have their own Wikipedia called biker and in biker is everything that you need to know and everything that's not in bikers not of interest to you always may be wrong so and and so that means if in doubt leave it out
yeah so what does the so many things are left out and that means our poor colleagues from Baidu and from tencent have much fewer data huge sources are left out but by that cannot be the way that we go and that we think should be gone because we ourselves our whole society needs to deal with the full body of textual knowledge and still find its way to some more or less ethical and appropriate way of dealing with it that's the only chance we have for the system so what they try to do with a clean common crawl was a complete failure so cutting out censoring lots of information is not the right way so and that brings me to the main message now now the main message is we have entered a new paradigm we have already had the Paradigm of machine learning now we have entered the Paradigm of machine teaching and what does that mean it means that people were trying to copy the secret sauce the recipe of gpt3 that was the
first well-known system that had all this emergence that had visible there were systems before at Google that had emergent functionality but the the first system visible to everybody here that had this thing was a gpt3 and now people are trying to copy it and they were tweaking with the algorithms here with a model there with a parameters here with a hyper parameters oh and they tried everything in all the big and at Google the bosses said or do that now you have to be open Ai and at Baidu and a 10 cent and and everywhere yeah they were trying very hard but the story is I mean if you look at it for a while the algorithms have not changed much gpt1 gpt2 3 ChatGPT the algorithms is a simple simplified Transformer as a decoder only Transformer it's been published what the algorithms is so the difference is not the algorithms the difference is the teaching is the education of the system which data do I teach and so on yeah so that's the big
and we have these layers of training pre-training then behavior training behavior training by human feedback then another pre-training for specific knowledge domains let's say for verticals and then we have fine tuning for applications all the levels of training and this is now all the questions in pre-training that you have to answer and if you find the right answers of these things then you have the emerging functionality nothing about algorithms in this area so um if you if you really can if you find the optimal answers to these things and then you can do behavior training and knowledge Corrections what started with uh ChatGPT the behavior training that made it such a great tool okay let me step back for a moment and say no what about AGI yeah that's always a big question yeah so actually what the system so far can do is the rare thing probabilistic learning of behavior implicit knowledge and the and the yellow things it can do but to a certain degree not fully but different from the way humans
do it humans can do these things by an interaction of the probabilistic learning with a rational part of our brain and the system doesn't have the rational part it cannot do rational inferencing several steps it doesn't have any symbolic reasoning so all the stuff that we do by interaction of these two the system has to do with all with a red one and it cannot do self-driven multi-step inferencing it's all one step inference and we think it's multi-step because we ask it several questions but you know the trick it gets the it gets fed the the beginning of the dialogue yeah and it's always the system is always without memory it gets input creates an output and now if you wanted to give it the current the previous dialogue you have to feed it again and we don't feed it but the system internally does yeah so so it has certain creativity as I said it has no self-awareness no consciousness Consciousness in the sense above self-awareness that it has morals of your own beliefs
and of other people beliefs so the full Consciousness self-awareness and full Consciousness doesn't have desires and urges no intentionality and so on if it had all these things we should be worried right but it doesn't so right now it's very harmless between two inputs the system is dead they are completely that but you can now put in the missing ingredients and there are actually two ways of doing a two strategies one is to go to the models directly and this will come in the future I predict very soon people will try but we don't know whether this is the right way because you remember that lstms had long and short-term memory by recurrence there's no reason why you shouldn't put this into a Transformer itself oh yeah so right now we get so much power out we don't need it yeah but people will try of course and they are already experimenting with it in several labs and the second thing is the steps that auto GPT is doing you do it next to the
system the system is dead between two inputs but you put an agent next to it and you connect it to other systems you can connect it to a search engine to evolve from alpha it was already to a calculator to any kind of thing and then you have a system that depending even on the output and that you can all program let's say with auto GPT yeah so those are the two ways that we can go if we go these ways then we have all the underlying thing then we have then we can do planning and we can do multi-step inferencing in addition to what's there then we can do that even with the external agency and that's now the big game yeah that's what many many people are trying to do is it can it be now can the system question can the system now be evil or not do we need intentionality and so on not really because the intentionality does not need to be in the in the system it can be
in the agency outside so even if the system is still not evil and only gives you back you can of course write yourself an evil agency an agency that allows the system to set itself evil sub-calls it gives it a goal that's how evil things develop yeah you allow the system to set sub goals so it can happen so that that is you have to be careful if we do this and connect it then we better do not connect the system to anything that can hurt people or destroy anything or do any harm yeah then it's really bad so this is then the multi-step I will cut short here so this is what you can do by I will do so but I want to finish with something that's really really important especially in Germany uh this in if I gave this talk in the US I would not do that or in China but here I have to do it unfortunately so do you see where are the German models if we didn't have
Aleph Alpha we would be completely bad yeah so now we have at least one company but most of the majority of the systems are really not here and new systems are not on this map that came during the two last two months so the Urgent appeal is that not only because we don't believe that there will not be one moral satisfying everybody not only because of that but you saw already by my last examples adding agency how even more powerful the system can be in a very short time it would be completely irresponsible Germany and Europe to say that we don't care about it and don't Master this technology ourselves we simply have to do it yeah if we let it go everybody who's at fault in letting it go is guilty in a way makes himself guilty of something no we would really regret it so we need to do we need to be in full command of that so and there was Lim the Lim initiative many people here are con so we
tried very hard in Germany we tried to convince the government lots of people behind Lim as you know Liam large European AI model so we try very hard to convince the society we need to do something we made a blueprint we had a conference then there was a session of the retreat of the federal government in Mesa bag they invited two AI people one of them is here for yeshu and the other one was Richard the CEO view.com one of the experts in the field and also former student of ours from Zab working and so the and they tried to convince the government and the government said yes we need to do something not only that but we need to do it soon fast yeah so I mean we need to do something so they agreed of course but soon and immediately means there are different scales of ourselves for me Immediate means now and I don't know what it means for others and and then we recently had a round table meeting with the
German industry that was a very nice organized by the ministry for economic Affairs and climate protection many big companies were there and again everybody said we need it at the end and the ministry said who should pay for it maybe industry or the government and the industry said we don't have a Google here and Microsoft and so on in in Germany unfortunately maybe they would pay but we don't have them who knows whose fault it is that we don't have them here that's a different question so and uh so somebody has to pay for it yeah so and then come the Skeptics they say isn't it too late already maybe the train has left the station already and of course no because I mean you see all the new stuff here that is happening I mean that is the kind of line that people use when we left computer computer manufacturing in Germany and consumer electronics and everything people said oh it's too late it's too late there we cannot no I think it's neither
is it too late we can really still do something it's an ongoing quick process but it's getting harder and more expensive the longer we wait there the second shouldn't industry care take care of that yes industry in a certain way but they cannot do it themselves because there's a huge infrastructural course that also in the US neither uh neither Oracle is doing it yeah nor uh sales force is doing it there's only very few consumer oriented companies that do these big systems for end consumers there is no infrastructure building for society and the American universities have the same problem so do we really have the talent yes just look at the at the look at the authors of big papers lots of Europeans there lots of Europeans here very few people in the literature are born and raised in the US most people the majority is outside and is it too expensive for us because Lim put in these numbers of 350 million to 500 million but those are numbers that by other countries by
UK and us have been accepted they have already done programs now to build these infrastructures here for and secondly it depends what you think I mean for me 350 million is a whole lot for the uh and for the bundes counselor and for the chancellor probably too because they need 770 million for the extension to their building that they want to build next year that would cost 770 million yeah and it could be that the 350 are missing then for their building and they still have to sit cramped in offices as cramped as the people in startups maybe no that we don't want so um so I I think it's all relative yeah whether it's a lot of money or not so the thing is I want to finish only with that line act now right here I still have a picture there are this is only when if you get the chance to answer ask questions I want to show you something that artists did with AI and children in Beijing one of the
best art I've seen in a recently those are pictures by children about the myths of minorities in China and AI helped the children to have magnificent pictures so the children had the idea the creativity came from the children but the the the way of carrying it out in a professional way came from Ai and I think I saw some of the sketches of the children they were very nice but you couldn't publish them these things you can put in every Gallery but the creativity does not come from the AI here it comes from the children so sorry I just wanted to show you it's interesting yeah there are a lot more yeah so I'll show us all of them yeah I mean if you want to ask questions so I really like your or find it interesting your Omni glot approach and with these different aspects you want to put together there um and I also see these conflicts in it like between for example creativity and Truth um and I was wondering like why
did we just start with all this AI thing isn't it like uh yeah yeah now we come to the origin a bit but isn't it when we break it down just the collection um and use of data to make our world optimized and more efficient um and then I was wondering why do we split the Omni glut system approach into these different ingredients isn't it just a kind of expanded Humanity we create there what do you think yeah I mean you get me at one point I need actually to correct me I never would never have thought that we should build let's say one system for small languages in addition because I forgot to say one thing and this is very very very important only because you have a polyglot system you can get coverage of languages that by themselves didn't have enough data so the the polyglossy of the the polyglot system is absolutely necessary yeah it's absolutely necessary the question is that whether we are right now already without specific education able to
train system in let's say even all the European official languages because we have a language like Maltese with 500 000 speakers which is not even among the first 40 of the most spoken languages yeah so we need to do something in addition we cannot hope that's what I wanted to say that this functionality would just come by dumping more web content in so but we we really need to help this unfortunately there are data for doing it so actually when it comes to the Omni glot system yes the Omni glot is of course um is is of course a far-reaching goal but it doesn't have to be maybe not every application system in the world has to be very fluent in the American Indian language or native language imara yeah so maybe not every maybe there are some companies who can have applications that don't speak Amara yeah and so that's so so that we should March in different directions but uh yeah but AI helps us a lot to take care of languages that
before were really under-resourced in language services but couldn't we like ancient beside data um with the input Humanity with what like with like being human yes and then plus intelligent use of data so the extension of humanity okay but this is this is the other question that I didn't have the time to touch and that is the sobering truth when I said the system is doing many things that we do different as humans yeah what's the core the core of human brain is actually this interaction of our associative memory with our explicit knowledge explicit memory and this this makes up and and if you think about it lots of the things yeah that we can ride a bicycle but also talk about it and improve it there so one thing learned by one thing is earlier so it is true that we cannot do that right now but we can get closer to the human essence of the whole thing by including other types of data like movies film scripts uh actions sequences of actions
of humans and but in that way get it it will still not have the sensory input that the child has here but we can get it much much closer to the human Essence and in the long run we will have sensory given Gathering robots and it will get even Closer Closer Closer because right now it's a machine that goes from text to text and it has learned so much about the world only from going from text to text but there's no reason that it has to stay with the text we can think about ways of getting lots at least of the data that a child gets when it learns the language in addition also into the system there's no no limit to make it more human yeah more humanistic questions in in the audience but you are still here right huh you're still here and you're ready to answer some questions later on so for now I would say thank you thank you and um thank you for your attention thank you