good afternoon it's the end of a very pecked day a day full of new information networking and good talks and I appreciate it very much that some of you still have the stamina and the power to stay for this last talk what I want to do is to share with you some observations that we made during the last one or two years mainly the last 12 months and to extrapolate into the future and try to alert you to a new development that you may already see yourself but I hope um I can sharpen your vision a little bit Yeah and say a little more about it it's not about educating people on technology but it is a new field that is emerging that's a field of making technology smart and educating technology in a systematic way so those are the models that we have talked about a lot today they were mentioned a lot let's jump to a slightly different picture wow here we see to the left the American models to to Chinese and
then then on the right hand side we see some European models that came out uh of uh the last two three years in in in in uh Europe and what makes them different are they just doing the same as the American models just not quite as good yeah because Europe is a little smaller and we're a little older or is there something different we have to ask ourselves because these European models are not on the top of the leaderboards they are not doing bad but they're not on the top so today Peter Zarin of Silo AI he gave uh a vision that we also had at the company nonionic and at many other circles namely that we have to find our special space um for AI in Europe that is probably not with the mass products not with the b2c AI but more with a B2B AI more adapting all these wonderful New Gen AI Technologies to the needs of Industry much more closely and much faster than our American colleagues so there would be
a chance yeah so then what's holding us up is Alf Alpha Silo nonionic our models uh the the the models of Mistral are then now running at our largest companies and fulfilling already the needs much better than the American models not really so what's missing here so and this is exactly the topic because we are on the right way but we are not there yet so I don't have to go through again what's so special about these Foundation models here so uh the first time in AI we have models uh that can perform many tasks for which they are not trained because they have basic knowledge they have basic skills and we get what we call the emergent functionality the emerging functionality they seem to understand us when we tell them to do something they were never trained to do like summarize this text in Italian for 10year old children they do but they were never trained for that so that's a new stuff yeah so what can they do and what can't they do
and how do they get trained and how do they get adapted to do much more than they have done so far and what I want to argue for not just argue for I want to sh share an observation that there needs to be a new discipline in a way uh to perform this task and this is built on some observations that I want to share with you um in a moment some observations that show that the bottleneck is not better algorithms is not more computer science not even faster machines the bottleneck is right now the training the education of the models that's the big bottleneck so the models have strengths and weaknesses I don't go through here now what's good about them they have very good language proficiency better provenly better than human beings and they have lots of conceptual knowledge and but they are lacking for lots of things smaller languages and so on new effects they don't know because they've been trained some months ago and uh and they are lacking very specialized
knowledge and I will give some examples for this before just to see that we are on the same line what we are talking about when we talk about pre-training now the very very best models so take for instance uh GPT 4 that you are all using uh eight times 220b parameters Moe so several mix of experts several big models and what we find in there if we if there were books it would be 100 million books but there are only a few books in that means a few 10,000 maybe so not 100 million most of it is not books and most of it is opportunistically uh collected web content and some of it is very good some of it is maybe not as good so here is the very Central starting point the very Central observation that we start out from and is the following that since 20 uh since 22 since 2022 all the big jumps that we seem to experience from I put on GPT but you can also take the Google systems
or llama 2 llama 3 all the big changes are not algorithms is not that new methods in machine learning have been designed or new architecture it's all versions of Transformers and not even the most powerful ones software of the best systems is not even the most powerful Transformer models yeah but but the power is in the data that came in and in the beginning it seemed to be that there is no better data than more data but after a while people saw this is not really the case better data there are better data uh also qualitatively so my own experience so we used bird already very early then we used uh tested gpt3 and others for the Lim and then in the company nonionic we tested many more models and we already tested not just language proficiency but also how uh relevant is the knowledge uh for industry or let's put it this way people from industry told us they said oh the systems are wonderful in medicine in law until we use them for
our own knowledge domain and then suddenly they not so good anymore what's wrong here how come and it's easy to see how come uh among the freely available um data that for instance open AI found in common crawl and crawled themselves on the web or in things like the archive um I will come to show show you more concrete uh data sets um among all of them certain areas are very well represented ah I will show you and others are not so so there are but there are three important developments right now one is the one that I'm talking today about but there are two more and this will be two different talks that we can do another time uh agency and memory the big big next turtle and then extensions to other types of data let's say movies and other sound data so but I talk about the first one today only so our observations were first of all deficits for smaller languages and that's not surprising many people have talked about this another
one is that you don't get consistency the system will not give you if you ask it in Chinese will not give you the same political answers than if you ask it in English that we can prove we proved it's interesting fact interesting um then sources of hallucination I won't talk about hallucination but just say a little bit on it because it has to do with it yeah my argument is really uh and I cannot talk about it at detail is that if we solve the problem of machine education lots of the hallucination will go away just disappear why will the machines then be suddenly discrete and not non probabilistic no stay probabilistic they will not be discreet in knowledge but we can uh I give you some questions I give you some questions uh then you will find out yourself when did GPT 4 hallucinate a 13th month of the year when did it hallucinate another day of the week why not if you ask it a lot about uh daily politics did it invent
a new chancellor of Germany no it didn't it may have told you the wrong one if it's trained a while ago yeah sometimes it mixes things up so the reason is really the more secure the knowledge is the more statistically it is secured by redundant knowledge not just single one the better it becomes and usually it hallucinates when the demand to fulfill the format that you ask it let's say a biography a product plan something when this demand requires fulfilling it in a statistically close way approximating the target demands that it says something about uncertain ground where it's not certain so he asked it give me the products of some Japanese company and it makes up some products why because it didn't have enough secure ground on that so the system for instance made up deaths of people the former boss of dfki my former boss Wolf Gang valster told me J GPT thinks I'm dead I looked up my own biography and it said I died in 21 why and I had the same
experience with a friend of mine she's a singer in New York and one day I searched for biographic knowledge and it told me that my friend is dead I was shocked I was really shocked I mean w g was not shocked yeah because he is he is a self-aware person he knew that he was still alive but I was shocked so I immediately tried to find out from Friends whether my friend Sandra had really died and then she was still alive I was relieved Wolf Gang wer had retired a little before and the system didn't see as much Publications and things of him anymore and it was simply more probable that he was dead and alive and my friend during the pandemic time didn't have new records new no new albums and didn't have any any any uh performances and the system triy to finish the biography in a probabilistically correct way okay so but that's just a little deter so coming back to the main H going back first sorry I forgot one thing
because the last thing is the important the insufficient specialized competence in certain areas so the system knows a lot about medicine knows a lot about law about computer science but when comes to special topics in chemical engineering or in Machining even even things like this yeah the system fails so why so it's manufacturing Industries they don't want to use texts but they work on physical objects so they at first glance they say why do I need language uh technology um but in the end uh they do and and uh there are so many applications I will not go through with you through all of them um to the left you see applications that you do already with pre um gen ai ai so that's um normal um supervised Training Systems prediction classification and so on robot movement and so on so for this you don't need language techn technology right uh language models maybe not I come back to that so in the middle is a little bit of text manuals product descriptions spefications and
so on yet there it can use language technology and to the right is all the very difficult stuff producing program code um talking like chatbots talking to the knowledge basis and then the lower right Corner this is the most complex stuff problem solving helping the company in product planning production planning uh solving very difficult optimization problems that require different disciplines and the system will be will be very good at this because the system has a much bigger brain uh the problem there for instance in doing the product planning for a new type of Auto Automotive or automobile takes in Germany 4 and a half to 5 years in China 2 and a half why does it take so long because there is nobody who knows all the areas so there need to be people from regulation from Material Science from production planning from all the different areas need to come together designers and so on and it's very very complex so you think you have it and then you see oh it's too expensive to
produce that way we have to change it again and so on it takes a long time and this is a place where language models that are knowledgeable about all these different things can really help but only if they are trained in automotive specialization not yeah the brain capacity is not enough it needs to be trained in that so L lar language I jump over that here so it's actually all that stuff that's now in red and the big one can also be done on large language models but now some of you may ask why the hell would I go and do normal Quality Maintenance image recognition with a big foundation model why would I do prediction of demand or uh uh supply chain monitoring with a large language mode uh I can do that with a traditional AI model just fine and you're right right now it's not necessary and it's not needed but it will be why will it be because language models become very cheap become much cheaper and if you put your
data about demand for for demand forecasting yeah you take the demand figures of the last years in normally you would just do time serious uh uh calculation on it yeah and then do predictions but if you do it this way and you tell the machine what you put in the machine knows what type of systems there are it knows oh in summer months there's a different demand you don't have to tell the system was a summer month or you have to the system will know and say oh there was a dip but this was during a financial crisis at was because it can get all the connections so in the end we will be surprised that even lots of tasks that right now we do on Specialized AI systems in the future will be done as fine-tuning on systems that at first glance seem to be too smart to do it right now it's a question of money of of cost but it will happen I mean we can be sure why start with a
dumb system if you have a SM system and do even a simple task okay so and down where the red circle is this is the kind of stuff that you can only do if you have agent systems if you have multi-step reasoning which the systems now don't have yeah so but this is a different talk so this is the Paradigm of machine teaching machine learning is clear you give a problem you have a problem and then you get a set of available data and you have to find the algorithms for solving it but this is not what was going on in jumping from jat GPT to gpt2 and three and so no it was the same algorithms the same system it was only the data and the way things were taught and then people had all these this is the pile yeah the pile one of these uh things and you can see now Pub met Central to the on medicine archive all these things very big where's all the rest of Industries yeah I
cannot see it here so you can see already that the system it's because the it the the the pre-training data were collected opportunistically uh because um common crawl does not log in and ask Springer signs for permission to download a book that's missing so uh it's very opportunistically and most of uh business of uh industry is not covered but we know that it is possible and this is now machine teaching uh op mayi showed us if you know exactly in which aptitude tests the system is not performing very well and you take the additional material then you get here between 3.5 and 4.0 GPT get the following improvements so that's machine teaching machine teaching uh this is uh multilinguality machine teaching is also this is a very interesting paper but we did similar um uh analysis for maintenance and reliability people checked whether uh the uh the systems uh gp4 are already as good as people and here in this case they are but actually it's a textbook maybe at the undergraduate level yeah it's
a bachelor textbook so basically the systems in industrial knowledge are as good as Bachelor sometimes a little better but never as good as the best uh expert that they will collaborate with and that eventually they may substitute so that means there's a long way to go except in some areas in medicine because all the pub met stuff is in there and so on yeah so this is another of these examples but I need to rush up now so we are going for well educated technology the main elements of this education is very simple what what do we want to identify what's the knowledge how by which means what are the data that we can find for learning this how do we teach it the sampling the weighing the training curriculum learning how to feed it and so on to set the learning parameters and so and then the main important thing test testing it did it work testing it during the whole education process just like an education so what we need is actually we
need a discipline that merges education information science AI uh edu educational psychology for instance other fields I jump over this and finish with a conclusion um oh no maybe I go ah I have a few seconds uh this this is the this is what we worked on in nonionic in the company niic uh that we had to leave or close down for reasons that I do not want to discuss now um so we collected certain data in order to build vertical models before we go to the specialized models of let's say BMW or Volkswagen uh we tried a step in between try to get as much Automotive data or Health Data together and this is the way that also Bloomberg did with there Bloomberg did a very smart way in training uh their special uh large language models on financial texts it was very good one so then let me finish now so for the first time now we have the possibility to have a system that can combine in one single artificial brain Knowledge
from so many different areas and lots of knowledge lots of knowledge but why doesn't it have it why does it hallucinate so much why it doesn't have all of this industrial knowledge because right now the people who are doing it are algorithm people there are people in in in in in in the uh who go to the newps conferences yeah who do algor but they are not the people who know how to get the knowledge in there and how to test it that BMW or Volkswagen is really needing and how to put it in and they have no idea where to find that knowledge because it's not on the common crawl yeah and they don't know how to get it so there's a new discipline that has to emerge that brings all of these things together bringing together all the knowledge and the skills and also tests the system so testing the systems the same way that humans are tested but there are very interesting differences but that would be the topic of another talk
and I should finish now thank you very much