and Elizabeth uh for this panel. Guys, we are really in the home stretch uh now. Um so last last well Fabian will will speak in in the end but the last keynote is uh from someone who is more or less inventor at the rise of AI. Um he taught students um in the major universities of Stuttgart, Saarland and Stanford. Um Professor Hans Uszkoreit will tell us beyond algorithms and please give a warm welcome to our good friends professor Hans Uszkoreit. Thank you. Yes, thank you so much for the introduction. Uh what a day of information. Lots of interesting compacted stuff on the next generation of AI and the current generation of AI and most of it is on algorithms.
So why would I say that there is an end to that? What's beyond algorithms? Yeah. Why should we stop? No. No. It's not that algorithms will stop. But I want to concentrate really today on a revolution that is not really dependent on new algorithms that will happen anyway. Yeah. So that's my field. But for the moment, for the time being, let's get back to algorithms. So those are those have been the major um or some of the major neural network architectures before 2017. U many of them I I mean you know them yeah they have been behind things like Alexa and Google translate and and so on. uh very powerful things and then 2017 the transformer came that is what most of the talks today were on so actually large language models generative AI that started with a transformer what it brought in new was this um uh mechanism of self attention multi head self attention uh but it lost memory for instance yeah that we already had on LSTMs but it brought in parallelization
which was not possible with the recurrent networks. Yeah. So that's a it's a give and take. So and now uh algorithm research is really concentrating on two major areas. I will point out two both of them and then leave the algorithms and talk about my my favorite topic. So one of them is that was mentioned today so many times is it became clear that these systems themselves yeah they can be asked the neural networks the large language models the transformers they can give smart replies but they have no agency they cannot themselves do any planning and automating so the agency needs to be done outside outside of the AI which also has some advantages advantages antages and disadvantages.
So most of the research now in applications is in this wide field of agent. Yeah. And usually this is now a very abstract way. Yeah. So there is then a nonAI agent usually there can be some AI in it but usually it's mainly part the core of it is not really AI that accesses one or several LLMs but can also access databases do rag uh do uh look up other things look up a secondary LLM and compare them uh work with knowledge graphs and different apps and so on. So this is this agent systems. Yeah, those are the ma majority of applications somehow fall in there. Most of them are simpler and the next big thing and actually Pascal today she talked about it without mentioning it.
She only said oh we are building these agents. What she didn't really say is that meta is one of the places on earth where people now work on the new generation. What comes after the transformer, namely combining a system that has the power of uh self attention bringing together in a huge context window learning from learning dependencies uh over long stretches of context automatically plus memory and agency. If you have memory and agency then you can also build in uh action models planning. Yeah, with agency and so then you can do other things. You can have situation models and in the end maybe a world model if you get data also from the world or from movies or whatsoever.
So this is the next architecture. So the algorithm research is in these two areas. Most people 99% work on the agent architectures or on the embedding of the LLMs now in industrial or other societal applications. And then there are some people who are working on the next generation. So now back to non-algorithms. That was about algorithms. Yeah. Let's make a thought experiment. Yeah. I mean just think about it. Think about for a moment there would be a ban on algorithm research. From tomorrow on is a death penalty for any new algorithm. So people stop doing algorithm thought experiment. Will AI stop in being exactly what it is now or will it still develop? What do you think?
I think it will still develop and I will point out why because it is actually not the algorithms right now. I will point this out that drives progress right now because nobody is working on the new models of tomorrow yet. So let's get briefly. So those this is the data volume for GPT4. You know some of the numbers. Yeah. Uh the important number here for me today is not the parameters but the tokens used for training because they would be equivalent to 100 million books. But actually there's not even one million books in this system. Yeah, probably we may be mistaken. We don't know. But if there is maybe maybe one or two million but not so many.
So what happens? Why don't they put in all the books there? I come back to that point. Very important to note. So uh then we have the new llama 4 uh a little bigger in parameters and now 30 million tokens. That's equivalent to 216 million books of 200 pages each. Yeah. So about but this time Meta does not want people to know exactly how many books are there in there because they get sued already over books. Yeah, they got several lawsuits now. So they but the the idea is that probably lib genen with 7 million books is in there probably also book three maybe books two and so there's a couple of book collections in there maybe 8 9 million.
We don't know. So the observation now is that since 2022 most major progress that we are talking about is not due to new algorithms because what is open AI using it's an simplified version of the transformer here and Google were cranking out the Google people more and more algorithms algorithms going to the major conferences and the open AI people collect ed more and more data more and more data and put them in and for a while they were winning the race until now Google is catching up and also doing this game of bringing in more more data. Yeah. So it's actually the teaching more data not the new algorithms that make the progress. So when machine learning is the following it's uh the following discipline.
It says you have a problem and then you have available data. now find the best algorithm for solving for learning from these data and be able to deal with new data for classification translation summarization whatever yeah so then is machine teaching is a different discipline it's a discipline given an algorithm maybe the most powerful transformer how can you teach this with data not just the the quality of the data but also how you feed them how you benchmark back how you test uh how you how you increment how you check and so on so the whole teaching thing yeah so machine teaching so it's actually a new transd discipline I will not say so much about it but the discipline is not there as you know there are but which l which computer science course many of you have done computer science yeah which computer science course teaches you this is it one two or three or data no none people in computer science don't learn about copyright, about how to deal with publishing houses,
where how where the disciplines are, what's the categorization of information. That's information science, a discipline that has a bad name, has gotten a bad name for no reason. So, this is a little example that is only achieved actually by, you know, many of you know this thing. That's a difference between GPT 3.5 when it comes to certain aptitude tests in fields. Yeah. Uh undergraduates take these aptitudes placement tests and what was achieved in GPT4 by picking the right training material next time so that the system would be better. There's no new algorithm in there. All this progress is only done by different data. So and there are many examples like the chain of thought that's clearly m much of the chain of thought thing is not reflected in the machine learning algorithm.
Yeah it is actually a way of presenting and learning data and learning steps or the deepseek reinforcement learning jump. Yeah, it's machine teaching, not machine learning. Or for instance, the recent result where people discovered why the systems are so verbose, overly verbose. It has to do with reinforcement learning. They get a penalty on the percentage of words that are wrong, that are bad. Yeah. And if you put out more, you reduce, of course, the proportion that's smaller. That was discovered as an afterthought. some people were able to prove that. Yeah. So, it's quite interesting because you didn't limit it. So, it's all teaching. It's education. It's not algorithms or the recent research that um uh Pascalong who talked this morning did on hallucination where she where she pointed out rightly so that lots of stuff that was called hallucination are actually just missing knowledge and the real hallucination is a much smaller set.
And what we discovered is also you can almost get rid of hallucination. Have you ever heard of a LLM that hallucinated a 13th month of the year? No. Because there's enough evidence. It will never happen. So if you have more secured knowledge, you get rid of hallucination. Don't worry about it too much. So this is this part and additional types of data will come in. That's not only part here is I have to be careful. It's not only machine teaching because very often you need to find substitutes for the tokenization for the vectorzation of the data. So it goes here's already some algorithm work also needed. Okay. But now come back to my point. So for many many centuries millennia lots of researchers and other people gathered knowledge the knowledge of mankind and you can assume that's in many libraries.
So for a long like it was in the monasteries later in libraries uh distributed now it's lots of it is on the web. Yeah that's the knowledge of all mankind. How many books are there? Question. So Google uh books they did a calculation. They gave not just a rounded number in a very exact number which I thought was crazy because nobody can say it with they came in a lot 14 years ago they counted up to 129 million books. Now every year between two 2.7 million and 3 million are published. So that means we are now more than 150 million books ever written. Yeah. So that's by asking studying history that's the best of all of our knowledge.
150 maybe a little more million books were ever published. Yeah. So but nowadays people don't go to libraries that often but lots of knowledge has been stored in big knowledge repositories and knowledge about different things. And this knowledge together with the books and other stuff there is important like regulations, patents, standards that are not there, enterprise, periodicals, they really all that stuff together is the knowledge of our civilization. Another thought experiment thought experiment for a moment. Um, from tomorrow on all of our children are raised by robots and they are not allowed any access to knowledge. They are raised by robots. Very nice robots. They get enough food. Don't worry. Yeah. So the the the kids are treated very well.
They get they can play all day. Yeah. They don't have to learn anything. They play all day. What would happen after a generation after all of us are dead? It's the end of civilization. There's they they would have to rediscover language which works. By the way, there's experiments, the Nicaragua experiments. There's experiments un not not not unethical experiments. There were people uh disabled children who developed their own language. So children develop a language within already one uh generation, a simple language, not yet a full-blown language. So they would come up with a completely new language and it would take them probably as long as it took us. There's no way why it should be faster because if the robots don't tell them anything.
Yeah. So that means it's the end of everything. It's the end of civilization everything. So this stuff is very important. So the knowledge of human civilization and there's a little bit of that dropping into our little heads. So each of us can about by any expert that reads who reads many many books can get about 100,000th part of the human knowledge maximum probably more like a 300,000 part because it's too much. Yeah. So each of us so we cannot solve many things and we need to get together do interdisiplinary research and committees and in order to because our brains are too small. So short detour on the books and then I come back to that point.
So um llama 3 and four they got sued now by authors and these authors said you are training our books you are making. So some of these the first one codes is a political writer kind of Sarah Silverman is a comedian and Adam Neville is a horror fiction writer horror fiction and especially Adam Neville said I take them to court because I heard one meta person say that a book can maximally improve the performance of the system by 0.06%. 06%. Which is absolutely crazy. I don't know who said this. I mean, he should be punished. He should be fired for that tomorrow from Meta because it's not right. And also, it leads him on the wrong way.
So then he said, "I want 0.06% of all the money that Meta makes with this model." Actually, he was dumber than that. He said, "I want 0.0.6% 6% of the money that was invested in it which is okay but it doesn't matter he's a horror writer so you cannot what can you expect so so and uh this is the this this is an interesting story right so people are trying so instead of authors coming and saying oh if the knowledge of all mankind will be concentrated in one big AI system I want to be in please take my books in these people they say oh I want to make money now I want to but actually if you really calculate uh if make the for yourself if you calculate how much do you think open AI made yeah so far with a model and then subtract all the costs that AI had for computing for their people and take the little margin and then divide it by the number of pages by the 13 trillion
words the people would not get much so I think it's an illusion But uh it's interesting. So but and there's right now an interesting paradox on the I call it the knowledge paradox on and the knowledge paradox is the following in every field. We measured it for instance with a master thesis uh together with data from a big car company um in automotive. It doesn't matter whether you take automotive or any large field. If you take a test, if a human takes a knowledge test in that field will always lose against the big systems because nobody knows so many fields. Even automotive engineering is too big. The battery person does not know enough about the gear shift and about the engine.
So we are losing. But in any real concentrated fields, human experts are better. And moreover the systems are usually not reliable and good enough for knowledger tasks in that field. So they are even if they have all this broad knowledge yeah they would win every competition and TV show but when it really comes to fulfilling the not yet they are not yet good enough so they need to be postrained. Oh that's the automotive that we did I won't want to bother you. So how can you then make the model know more and get it in? So let's take the three things in the middle. You do supervised fine-tuning. You already know you can do some reinforcement learning.
That's not really on input output pairs. This is more to tune and improve. Not really to teach it new knowledge but to improve the returnment. And then you can do continued pre-training which is very difficult because usually very few models give you the checkpoints that you can go back and continue the pre-training also. It's very very very risky. If you do it wrong, you can get catastrophic forgetting and the system because with new conflicting biases it destroys already learned knowledge. And so you have the costs of having your own instance plus the risk of ruining everything. Yeah. And and losing all your computation. So most people are too scared to try it. But there are some some people trying it with smaller things in the academic area for other languages and so on.
And then you could do adaptive pre-training. So what's the one on the left? That's interesting. That is if you do not want to change the the model because the context window has been has become so big why not teach the model for a nanocond and make it very knowledgeable in my field. Yeah. For nano because right now you can get up to 2 million uh or 32,000 with GPT4 2 million with Gemini 2 and then you teach it and after that the system has forgotten. This is also very useful if you have confidential information that you don't want the system to memorize. Yeah. So that's also good for that. And actually there are people doing it.
There's a company for instance that fe and I are uh doing some investing some help. We are helping them assisting them doing that. Uh they do this quite a bit. So they teach for teaching the model facts, concepts, formats and extending the prompts and then you need a mechanism that does the teaching on the fly. That's not easy because you cannot teach it all of the field in only if you in every prompt put in all the automotive engineering books. That would be too much. But you need to find that part of knowledge that's needed for that task. And actually they are quite good with that. And so how would we call this is but now they give it a funny name they call it dynamic hybrid language model they say the model together with all this prompting libraries and so templates filling it they are using ontologies knowledge graphs very clever company very very clever company very good customers already very successful in a short in a short time grew very quickly uh they
uh but the name the name is of course not Right? This is not a model that comes out but it it sounds good right hybrid uh it sounds very good the model uh so and I will not talk much about domain adaptive pre-training this is really an interesting thing how you how you get models but let me tell you one thing there is probably a business idea for the big model producers that they have not gotten into yet and this is uh providing the possibility probably even on premise or in a private cloud to do um under their guidance. Yeah. The people who produced the base system helping a big corporation to do this kind of adapt adaptation.
Yeah. Adapting the model and doing because they can do it better than anybody else and they could sell that because the company themselves they don't need it every day. Yeah. So they why should they hire all the staff doing it? So this is a very good business idea for the for the future. Uh but so now the question another question uh we talked about the knowledge of the the of civilization and we saw it is already it fits already if we had it if if we had the possibility of getting all the right books and paying for it because that's of course why GPT why openai didn't do it. Yeah, because it's too expensive. It didn't pay for the books and neither did llama uh generally.
So, but if somebody would do it, would this system stick with that knowledge or could it like humans develop more knowledge? So, we know that there are researchers who don't do any own experiments and who don't go to Africa or to the space in order to gather new data who only work on secondary research. they produce new research out of the results of others and the measurement. Yeah, lots of stuff can be done that way in almost every field. Some fields are better, some others. Yes, I think when we if we get the system uh in a kind of chain of thought or multiple call slow way of getting and doing it. Yes. I mean, I tried it myself in a field that I'm not very good at.
Yeah. And it discovered things that neither the internet nor the system before had. Yeah. That were only discovered because I asked it certain questions and it discovered new knowledge. So there is lots of stuff to be gained that way. Yeah. So lots of stuff. Um and let me finish now. Oh yeah, I should finish. Um some notes on the impact. I I hope that you already saw it yourself. If not, I will tell you. Yeah, it there's a huge impact here if the if you really get the compression of human knowledge into such a system and it becomes more reliable and it's trained on good data much better data and is much more reliable. It will change everything because thought experiment think already it's there.
It's there you can reach it every day. Everybody would use the system. Right now we we are already everybody is using already the imperfect system. But then everybody everybody would use it every time. So all students all stu everybody in school that would be the or authoritative authoritative source and authoritative we have to be careful because some countries would make it authoritarian instead of authoritative. Yeah. They would say uh we have aligned the system to the ideology of our country. Uh you have to use it never write anything else. Yeah. So it's a danger in there too. It's a danger because our system should reflect uh the multiplicity of uh ideas and opinions. It should be a multiple viewpoint system.
But if you can do it right now it is. Yeah. It always gives you arguments pre and for and against. for instance, right now it is. And now think about the following thing. Um, today a robot does not need an onboard brain. The brain can be some other place, right? Doesn't because it's connected to the brain. I mean, we don't need an onboard. No, we need because we don't know how to get the nervous system be connected. But maybe future medicine will connect us to an outside brain. Yeah. So if our Yeah. If if our brain gets smashed or so maybe we get our brain outside but right now we cannot do it but every every system can what does that mean?
If that system is commodity affordable commodity every little thing even my toaster can be smarter than me. Yeah. So I mean think about it. It's cheap. Every thing in the world Yeah. Every robot, everything in the world can be as smart as all of mankind. Have the combined and this will be such a revolution. Every robot will be so super smart and know everything. I mean, think about it. Think it through. This will be the real revolution and it doesn't require any new algorithms. So, okay. So, new and so what would it mean? The little bit of knowledge that we have still new knowledge that is not yet in there will be very expensive because that is the only thing that we have still for ourselves.
Every new knowledge will be immensely expensive. Yeah. So that will come out of that. Yeah. Will be so and then the last thing sorry for the 10 seconds. Yeah. the uh sorry so uh also we should rethink and and I think in automation people who worry about automating things with AI are already starting this process. We have to rethink what our real value is as human beings. Our we thought that our value is that we can do differential diagnosis that we can stylistically that we can have a stylistic tople translation that we can write a wonderful report or application. No, all that stuff can be done by the machine better. If it can play chess and go, why shouldn't it write good reports?
If it can it can write poems and do everything. No, all of these and still we don't have a system that replaces a full job. Yeah. Almost no job is in full replaced. Why? Because we overestimated our knowledge which can be copied and we overestimated our performance in selected tasks that are the most ch because they were the most challenging. We took them. Yeah. Because they were good selectors. But actually today our ability to handle with angry customers and manage exceptions and talk to people and keep others happy that no system can do. So it's a completely change of what is important. What really makes our real value is not whether we can compose or write a good compose.
Yeah, maybe there's aesthetics but whether we can uh write a good report or do a good differential diagnosis. So that's really important. Yeah. So it's our holistic capacity to deal with unforeseen situations because if you have a model trained at uh let's thought experiment thought experiment for a moment. Yeah. Think think about the following. for 10 years. We put a system at every DUX 40 CEO uh desk and the system gets learns everything what the CEO read and did and what what the decisions were. Will the Yeah. Will the system be able to tomorrow take over the CEO role? No. Why not? We live in a dynamic world. The next morning Trump has raised the tariffs.
the people do go on strike. Uh there's a shortage. There's a we live in a completely dynamic world. Therefore uh okay now I finish I get to say so so therefore our real our real value is in dealing with a dynamic and complex world and with other people. Yeah. And doing that that's our thing. Not doing the kind of stuff that we are tested for in school. Thank you very much for your attention. Thank you, Hans.