and one german ai researcher is here in person as well this is professor dr Hans Uszkoreit and we are really happy to have him back he's a constant uh and very well-known guest also at the Rise of AI conference he is specialized in language and knowledge technologies and um was based in beijing and i recently learned hans that you have still some boxes in beijing sorry that i have one some boxes some okay just to be polite the second stage is opening now sorry both in parallel so i recommend you because it's overcrowded here too to move over to give them the opportunity to open to and then now you can continue but give them the people one minute to go yeah you both can start it's wonderful i was so soaked in your sons yes you will need this um hans you are thinking a lot about super intelligence you are an expert on ai strategies china u.s europe so you've worked on three continents and also researched on three continents so what what
about commercialization i mean i hear a lot about research and you are scientific director of many many entities as well so how about commerce commercializing ai what is your perspective on that yeah although i started as a real researcher and research leader actually we have done startups since 2000 several ones i did several ones in germany that i started and also some in china even in china in the united states i never started the startup but many of my friends and colleagues have so i know the scene there as well yeah so so this is also one of your focus areas actually yes i i have a huge list of your many many functions and titles you are having at the moment um um but today you want to talk about large ai models in a global perspective that's true so we are really curious to hear your perspective and i'll be back after your presentation give him a hand warm welcome to hanzo yeah thank you for this wonderful opening and for the introduction
today i would like to talk about two things actually and bring them together one is the general worldwide competition in artificial intelligence especially between the so-called ai superpowers and our situation in germany and the second one is a new development in ai these very large ai especially language models that they were with us already for a while but in the public they were discussed quite a bit after gpt 3 was announced and shown in 2020 not went through the press so gpt-3 is only one of these many models and gpt stands for generative pre-trained transformer and i will in a moment just describe what that means and it's a large language model and it was the third generation by of this gpt model by open ai so um there as i said there were others earlier ones like birth by google switch transformer wu dao so those are neural networks that know language the patterns of language so well that they can do many things they can translate they can generate they can summarize text
they can answer questions and so what makes them special is that they are very big and the way they are trained so they have billions sometimes even trillions of neurons working neurons actually of parameters that are set by the training but that roughly corresponds also to the number of nodes and they permit long inputs not only individual short sentences up texts up to 2048 words or even longer in the case of gpt3 and they can also produce very long outputs and the newest of these models in addition to language also have learned explicit knowledge from knowledge graphs or multimodal data images or couplings of images and and pictures images and and and language so we know this is what the the miracles of ai i can look at the something like this and then describe it here so i can now also go the other way around there are already systems that take a language input and give you an image out that they either find or even construct and what's special now about these
language models their input symbol vector is something that is either a speech recording a sentence in german a sentence in english and so on and they are trained on training datas those are these label data data and so for instance something that extracts relation may extract the relation volkswagen has acquired a europe car from a sentence like on the left but it could also translate the sentence in the case of these large language models or it may summarize it make it shorter and it can do many other things can answer questions about it and so on so one of the central ingredients of these large language models is something that allows us to link parts in the sentence that are different from each other you know also for images there are mechanisms that link things in an image that are next to each other for instance layers of convolution layer networks and here in this case we call it a tension mechanism and the attention mechanism can work either by feeding the sentence piece by
piece in and then keeping parts in memory keeping those important parts in memory that may link our candidates for linking to a later material like in these sentences here where you need to do this or in the case of these newest language models you can feed in the entire sentence even long text like up to 2000 words at the same time and the system then learns from these stretches these attentions themselves and there's a mechanism called self-attention that learns and produces additional layers that link kind of points that require these connections with later material i cannot in during this talk explain exactly how it works the pre-training actually works in embarrassingly simple ways it is not really that the system has to perform very difficult tasks what we need is some tasks in which the data can be labeled automatically that's called self-supervision it's supervised learning so you label data but the machine labels these data and it does it for instance by just taking out words of a of a text or leaving sentences
and texts uncompleted and let the system complete the system then during learning brute force tries to fill this task fill in the left out words or complete sentences and it does it because it's not a child it's very patient it doesn't get tired it doesn't cry it does it billions of times without complaining and in the end the weights of the neural system are set in a way to minimize the errors that the machine made in doing so so and then at some point the machine can do miraculous things you give the following input to gpt 3 the importance of being on twitter just like a title jerome k jerome that's an author 150 years or long ago in a humorist three men in a boat you may know yeah by jerome k jerome and then you say london summer 1897 and now you let gpt3 produce a text for that and it produces a long text a very complex text and it does so it describes twitter which jerome kj rome has never seen
not knew about in the style that jerome would write so it said it's a curious fact that the last remaining form of social life in which the people of london are still interested is twitter so yeah so he writes funny things i was struck with this curious fact when i went on one of my periodical holidays to the seaside and found the whole place twittering like a starling cage even this thing twitter ring like a starling cage this is absolutely amazing creative way of dealing with language and writing this yeah so looking at it you may think this machine must have true intelligence and even the creators of this machine like ilya satsukova the uh chief scientist of open ai after a while stared at these things he should know better because he understands the way very well but he said i think this thing has some form of human cognition has real cognition in there of course it doesn't yeah it doesn't understand a single sentence but it does things that humans can only
do by understanding so it can also answer questions like what is the name of the russian mercenary group it says wagner group and then sometimes it gives answers that are even too long yeah you have to tell the machine then to make a charter like uh where's the wagner group in syria who's the leader and then it rambles off about dmitry utkin where he is and so on so but you can make the machine be shorter or longer yeah you can so the gpt3 model and similar models have met many strengths they are very good at text generation at summarization question answering translation but they're not very good in real multi-step inferences when we do inferences that take several steps logical steps they cannot do it they can do if they have not seen already the answer they cannot calculate in several reasoning steps but otherwise it's striking so what was so special i told you there were models like this before but then is it the size now more data more training more parameters
this is gpt 3 compared to the ones before and i'll come back to exactly that point what does it make is it really the size yes in part it is yeah it's the amount of data because earlier ones were trained in similar ways but what has happened since gpt3 this is the big one this has happened now gpt 3 is at the left side and those are the newer ones uh there are newer ones in china in the u.s huawei and china wu dao in beijing open nvidia did new new ones at google and now the good thing even a german one yeah so i don't know if jonathan rules is he here ah there he is so congratulations to aleph alpha for this big step forward so we now have but in general as you can see this is the first and only one germany in order to be fair is not really in most places 99.9 of german companies and research units don't have access and not the means of doing that yeah
that's a singularity so we are all used to american big companies being attached to ai and in china there are similar companies to these we know this is big and ai but then what's now the comparison between china u.s and europe i gave whole talks on it but i found that this survey by the center for data innovation that did an analysis arriving at almost exactly the conclusion i will would have come up with it compared the three main powers according to six criteria talent research development hardware adoption and data and it gave the following numbers 44.6 and so on as a as a sum of these points given to the individual criteria and the us leads in talent research development and hardware china leads adoption and data data is pretty clear why an adoption i will come back to this in a moment in the european union even with more than 500 million people is not as good as the others although we have a very strong but this is maybe a little unfair
but right now europe is falling back but if you look really at the application areas you will see that europe is not doing so bad in a certain subpart of ai that doesn't get the same popularity as internet aia and consumer electronics which is manufacturing enterprise administration what is called enterprise ai there will be several talks uh today about this topic and so coming back to china the special thing about china is that ai has ended daily life when i lived there there was a food brought by a robot to the people in our building there were traffic tickets in beijing written completely by ai there was the traffic and hangzhou directed or controlled by ai there was a complete artificial this woman this anger woman in tv reading every day she moves and reads but she's artificial she's completely not a human being there is now in shenzhen the first taxis not as a test they already can be used they can be called and they don't have a driver and so there are
these many things that you experience when you are there why are they so fast and doing it yeah so that's the question let's go back to research for a moment and then come back to applications yeah go to research in in research indeed that's what we saw being in ai ourselves at conferences china is now dominating in numbers the conferences but now you can ask those are numbers by the way that were compiled here they come from other sources but they were compiled by colleagues of ours in stanford called the so-called ai index led by my former boss at sri by ray perot and they they've been doing this for a while it's a wonderful resource you should use it the ai index in stanford so china is doing but are these papers of good quality let's see how often are they cited who now we see when we look at how much they are cited china is not doing so well anymore right it's and especially now in the time of pandemics it went
down so but careful careful look at the next slide chinese professors and scientists they get points and jobs for journal papers not so much for conferences let's look at journal papers in journal papers again china is leading but now look at citations oops china is still very much ahead so don't jump too careful jump too fast to conclusion that this is all of lower quality lots of it is lots of fast publications but some of it is also very solid look at patent filings for instance china we read it in the papers china is filing all these patents at our company we needed to file patents for things that we would never patent in europe because the government wanted us to do that so we needed to file these patents but those are the patents granted did you see those are the patents filed china see china united states and those are the patents actually granted quite a difference right so the chinese government found out that lots of this patenting does not really uh
hit the spot and is ill-guided and they are now trying to counterbalance it by a more reasonable publishing policy but patenting policy let's come now to the weaknesses of europe and germany why are we weak and it's not so much in basic research i would come to that many brilliant ideas coming from basic research from europe but it is actually in taking it not only commercializing taking it to applications our problems are in three areas one is the last area of application research the other one is the first part of commercialization risk capital the one the first one is true success in application research true success we have many projects but not true success and the last one is scaling in the market which is more in industrial it has more to do with the fragmentation of the market the relatively bigger size of the american and the chinese market and so on the other two are important and the first one i would claim is the most most most important and i would claim
that the german government up to this point has not understood why the u.s is so successful there but chinese government has so this is a simple fact i know by talking to people from chinese government they have understood it very well and i know by talking to people from german government i'm pretty sure they have not understood it so the first one this is the important true success in application research so how did the u.s solve the problem of this last piece of mile yeah that after the comp first you have the competition in true research publication of papers people want to become professors they publish a lot they get cited and so on why do they fail on this last piece because it is a huge effort of collective effort and you need to have competition in kpis people need to be interested in doing it you need competitive research that is especially funded by the government industry cannot fill that and the us solved it by originally solving a different problem namely the
problem to become the technological leader in the cold war in 57 when the sputnik shock came and america was shocked that the that russia was becoming became very successful they started a very funding scheme first with arpa darpa the defense advanced project research project agency later others followed and also they have regular research like nsf but you will see later that this is not the important part so and the darpa success stories include the internet itself which was called arbornet before windows mouse hypertext gps conversational personal assistant siri that was my former group at sri was first darpa project and then became a spin off conv automatic machine translation the people who later worked at google facebook microsoft many of them were first in diaper projects actually even earlier they were in our german projects but then they went to and then autumn autonomous land vehicles drones everything came out of darpa why because they have the following ingredients of the success recipe and the problem is all of them are needed you cannot take
half of them but to tell you the truth the german research funding of applied research fails in all of these all of these unfortunately the first one is formulation of strategic technology goals oh yeah we have a research strategy oh we come back at that i will tell you later which is different then have competitive r d projects not say say oh this would be double funding if two projects do the same no they have to do the same we need three or four doing the same thing in order to become good yeah it's silly to think you should fund the same thing only once it's absolutely silly it's taking competition out of this type of research the third one is to have infrastructure projects big ones that lead to gps to the internet or for instance i'll come back to this to large language models then procurement is very important the government buys applications not just funds research they are buying darpa and department of energy department of health are buying things from small
companies and that helps the small companies procurement then consequent funding until the goals are reached in in europe and in germany they would say oh we have funded this topic for a while now we need to fund something else yeah we cannot always fund the same stuff here we have to switch and then the people who have worked on that go to the us and continue their work there and highly competent planning and management by experts darpa is inviting for all of their program experts who stay with darpa for a while they they will not follow because they go back to science they cannot be corrupted by the military but they can also be not corrupted by the by the science because they are not for that time they are not taking any uh grants themselves so this is very important but look at this thing this is really mind-boggling here down there you see the national science foundation but this is ai contracts in the u.s in 2021.
you see how little is from the research part all of these things are darpa procurement health and so on it's all kind of very directed government funding this makes the difference and then of course you get more private investment germany is much bigger that's that's for i mean that's clear that we all know so but couldn't we do it as well yes of course we could let's take three american success stories three examples autonomous driving in the 80s daimler was leading yeah and in volkswagen they were really leading in autonomous driving was never carried out to the end because for industry it was too early and government didn't care too much darpa had the autonomous vehicle program the autonomous driving program so that somebody like sebastiantrun a german went to the us and as a stanford professor became very good he won the darpa contest and later he started at google to build up this area machine translation the person who started google translate was those bronze he came from my group he hired franz
oh he had also worked on our projects in germany and then the maharaja bruder the brothers who came from the german projects from our projects in workmobile mainly at ahn at alan and so and then they started machine translation and now we have germans at microsoft at google everywhere deep learning the two most widespread types of neural networks are convolution networks and lstms by jurgen schmidt uber and sepulcher jorgen is here today to go to his talk so but it's it was adopted and it's in almost all applications it came from germany yeah but the business was not made in germany take the three Turing Award winners of their three europeans who were drawn to the us and i did the following joke i took the 50 most cited or the the 50 people who i recommend most often to my students to read up to deep learning you know the whole history of deep learning 50 people schmidt of course jurgen is among them and many others and now how many are from the
u.s the blue ones are from the us but the us is leading right they're all coming from europe they're more chinese and they're more more more canadian maybe than u.s americans yeah so this shouldn't be i mean if you if you work in this field it doesn't come as a surprise so this is brain brain the flow of ideas happens mainly through brains but also through scientific literature through collaboration and international collaboration shows the flow of ideas because if the flow goes slowly from one country to the other you will see that people also cooperate there from both sides this is cooperations between different countries united kingdom and china and so on germany appears only once we are not so cooperative we have many people but are not so we are once with the united states not surprisingly well you see corporations what's missing up there it says excluding u.s and china could it be because the us is so much against china that they cut off the cooperation this is u.s and china up
there and those down there are the others so the u.s is not as silly as some people in europe stopping and ending this cooperation but they are doing it every day in order to gain from it that's pretty clear so then already we mentioned the national ai strategies the first one in the us then in china but in europe fabian already mentioned we are much stronger right we don't have only one strategy we have many we are much better many of these strategies in germany we had a strategy in 2018 angela merkel at the uh digital announced three billion for ai and we said oh that will be interesting i wonder how they spend it because actually i'm now maybe too old i don't really believe in the effectiveness of the way of spending on applied research that is right now in done so i was thinking oh three billion or i hope they don't waste it i should have said oh three billion they'll spend it there but i was afraid so then in
2020 angela merkel promised another 200 another two billion in uh november 2021 thomas attelberger uh free democrat i need to finish right a free democrat but i'll do very fast a free democrat uh bundestags upgrading the parliamentarian as the government and 300 million by that time were spent and even up today it's not more than one billion so the whole thing is but in the meantime china went further they had the next gen they had the five-year plan and the innovation competition act and in each of them ai is number one yeah for the for all the uh things so our urgent need is we in germany urgently need an updated plan and a real strategy with goals kpis here and real competitive research project managed supervised by experts and we need infrastructure because right now nobody can do these things that's why at the german ai association we got together and started a an initiative called large european ai models in which we are now pressing together with 400 ai companies 400 ai companies
and many many other supporters look at all these supporters from industry we are trying to put pressure on and say let's create an infrastructure that gives us at least a chance to compete and companies like aleph iver who is the forerunner is also part of this initiative so let me finish then we are working in three steps uh pilot building first in a in a project that's already been funded but that's much too little then we need the compute infrastructure it's still not the five billion that's needed yeah that angela merkel but it's more money and then the creation together with our friends in other european countries so let me finish with an upshot and outlook a new generation of large ai models offered great prospects really everybody knows and novel applications that were not possible before usa and china have a clear lead in international competition also in this field europe with germany we are falling further behind if we wait we are really falling further behind all the time but we still have
immense chances but we need public support maybe not as much as in china and the us but at least what's promised yeah at least and we need to try new models of ai innovation infrastructure and new funding instruments that are informed by the success stories in the us and the question is in the time of ukraine war pandemics and many other things is the german government able to grab this opportunity or won't they that's my question that's a good question i have the same stay with us thank you so much han very insightful and um i mean i think we all have this question in mind how can we stop the brain the brain drain and you had some recommendations a lot about money investments and so on is there anything else that you no no but but but you mentioned the right things uh first of all in in in germany we have a law called the bethesda lungs for both so many uh many research centers are not allowed to pay people what
they would get in other countries because our salaries have to be in line with the salaries of civil servants in the ministries so you can never get the good people that's one thing you need to the second thing is if you offer competitive environments in which people not just projects in which you compare your results with a project plan that was made three years ago but where you compared really with others yeah where you have real competition people like in darpa projects people thrive yeah they like to work in them you know not because of darpa has nothing to do with the military it has to do with and the third thing is you need of course an infrastructure so many of the creative ideas in open ai and in at google facebook and so uh also at huawei and baidu they could not have come up if you have to apply for compute time at a big high performance computing center yeah you need this kind of and if we have these ingredients and
i think we can keep our best people and maybe even attract people from other parts of the world because europe is a good place to live right that was pretty clear i mean that you really believe in our ability in europe to keep up with the us and china i hope so yeah i think it's possible i i think we have to do it it is not impossible yeah it is but there is no nature of law that says that the u.s has to be so much better than ourselves thank you so much chance give him a warm applause you