we have the next speaker coming up and it is jonas andrulis and um he is a serial entrepreneur and his latest company is called alep alpha former ai r d manager at apple very interesting i think and the forefront of development was transformative ai so you are focusing on computer vision and language processing and called your speech the last project and i'm really curious to hear what that is thanks thank you yeah this is a little bit the the vision that christian ordered um so if you um paid attention or if you were here this morning on the talk that huns and and chris gave this is a little bit of a deep dive into that um and what i want to kind of think together with you about is what is the meaning of work what does how a human life have revolved around and is ai maybe getting in position to change a bunch of that so basically oh this is not the first slide this is not the first slide still not
the first slide this is the first slide um what is work so when we look back into the the history of mankind of humanity right we have been organizing ourselves around working together about doing something together and and sharing our skills and uh contributing together together fighting for survival right fighting against famine fighting against illness building um houses to protect ourselves from from nature and this was at the beginning is pretty straightforward like if you're a monkey and you collect some roots and then so you have something to eat and you don't starve the next day and then you can change that maybe for something that somebody else did so this starts pretty straightforward and for the longest time of humanity this has been a struggle for survival life was incredibly hard for pretty much everybody like even a king a few hundred years ago had it significantly worse than probably almost everybody now um and this was basically driven a lot by agriculture right it's like this nature and and the energy that comes
from the sun and lets crops grow and then we harvest those um and so this is basically work in a way and then everybody pretty much building a shelter so we don't get wet um still pretty forward right straightforward so why are we all getting paid for what we do all day none of none of us i'm assuming is doing any of that and so how do we justify the money we're taking home what's what's happening here and what's happening here is that we are shifting a lot of the the value we are creating away from these like super like down to earth and super uh kind of at the bases um work to more creative work more information-based work and this you could categorize into like knowledge work collecting knowledge preserving knowledge making knowledge accessible innovation right in inventing things uh like the wheel here um and what's the thing on the left entertainment right and because we are we have the basic survival covered we know there will be a snickers bar tomorrow
that we can eat and we know that there's kind of like a house we can come home to um we've shifted a lot of the work we're doing and a lot of the value creation that we have into these three fields and we call them knowledge work and that's pretty interesting and maybe this maybe you can come up with other categories you wanna we wanna nail like the composition of knowledge work down to um and what's interesting from an i perspective now is that ai is is branching heavily into at least two of the three we still there's still some question marks on innovation there's um there's a report from open philanthropy and this report is estimating the timeline for transformative ai and transformative ai in the definition of open philanthropy is basically ai technology that can do most of what we should today call work can potentially reorganize the global power balance and it can also do the work of human scientists so it can drive innovation we're not there yet when you ask
open philanthropy about their timeline they are estimating that we're somewhere around 20 30 we can get that so what does that have to do with our work what does it have to do with what what hans talked about this morning is that we are seeing a lot of movement here on language models so so what our language models in case you haven't heard the talk this morning is they're basically just looking at all the language that humanity has ever created like all books all chats all scientific papers and they're asking themselves the question if i cut this text off at any at any position and i'm trying to guess how it might continue so that's pretty much pretty similar to what you have in your cell phone right you can basically start typing something in your cell phone and then your cell phone will give you suggestions or google search here on on the right hand um for how this could continue um and when you when you're familiar with these kind of google search
things and you you would ask yourself immediately why is that so groundbreaking right so if i start with good then i get the suggestion good morning so i'm saving a bunch of time but why is that groundbreaking in the sense of knowledge work well it turns out that these models not only learn like patterns of words that are next to each other they basically learn the structure and the world of all human language i've been calling them world models for a long time and this is say not to to young schmidt hooper who is here as well right because they are also in a way world and all the world models but not world in the sense of our physical world but in the world of all human language and you can basically say their prediction they're predicting the continuation of all of human language that exists so basically and we have built language to be useful for all of knowledge like language is built invented by us even right to be purpose optimal for
everything that matters to us like knowledge but also ourselves our experience what we care about all that is part of the structure of the high dimensional distribution of language in a way and if we are building a world model that understands that then we have a tremendously powerful tool so and i'm going back now because the first slide is supposed to come now this one right so this is our mo well model we you've seen it this morning and on on hanzo slide um we're at least on on the same chart as the us right we're not leading but at least there's a european company that's on the same chart and so basically if you now use our language model and this is on on the top hand side you see a prompt and on the bottom side you see a completion so you're asking the model what's the most likely completion of this text and so if your text is angela merkel was born in then the model again gives you a probability distribution
over tokens which in this case it's pretty confident that hamburg is the right answer which luckily it's correct but you can also do something else you can also say english the future will be wild and then deutsch for for german and then you get a translation so basically a model that has just been trained on on what you've seen earl earlier on kind of like these uh cell phone order complete by accident and this was a surprise to many and myself included is able to do a bunch of things that you would consider to be knowledge work like doing a translation right retrieving factual knowledge about the world that's something that we would not have expected a model that's basically just a stupid deep learning model that we throw a bunch of data at would be able to solve and what that means right so here's an like another example and of course what we're we're building we've built this model and what we're trying to do is we're trying to not only write cool
papers about it and cool blog posts but we're also trying to use this model to transform the our work and to be offer foundational technology for an industrial revolution and this is just an example this is like when you when you go to our website you can register for our playground you get some free credits you can try this out by yourselves and you can upload like a word document and this is the quarterly uh report from hpe um q1 2022 and then you can ask questions about this document and i'm asking question here about the free cash flow guidance for 22 what's the free cash flow ends and then the model answers fish fiscal 22 free cash flow guidance is in the range of 1.8 to 2.0 billion and if you're familiar with what qa models have done in the past right there is like bird-based extractive qa models those were neat at the time but this is like a far cry from that this is like significantly more powerful and there are some
examples that but that i haven't brought today but where the model is answering questions about things that are only really there between the lines so there's there's no clear mentioning where you could extractively just point to tokens in your evidence document but you basically have to understand what the author in this case i tried it out with is like on on short stories right so it's like almost a lyrical finesse that it's it's able to offer so what you can do is this is one part of knowledge work search and transform and recently that there was an uh like an interview in in frankfurt argument it's supposed to be a um transparent background but uh you can imagine the the white box as transparent and so this is there was an interview and the last question of the interviewer or alex ambrose was um when will i be able to have an interview like i'm having with you now with an ai right so of course what i did is i copied the question into
our ai model and basically hit submit and the answer that the ai model gave without any training without any further interference from my side was that's an interesting question i believe that in the next few years we will be able to have a dialogue with an ai but not in the same way as today we have to get involved and accept the technology in order to understand what it can do because there will always be new technologies that will change our world and also turn it back again there's already first approaches for intelligence assistance in the fields of medicine or transportation as well as of course for robots themselves but i think these are only small steps in the way for human ai interaction so that was the answer straight out of the machine the only thing i did was i fushot biased it with the answers of the two questions before that right so i can be like question n minus two answer question n minus one answer question n and that was basically
it so that's creating innovating knowledge right so with all the interviews we're all having wouldn't it be great if we had just an ai that would do all the interviews for us how much time would we save that would that would be awesome um so one of the inventions that came out of our heidelberg lab is multimodal models and of course there have been multimodal models before but we dialed it up a notch and that was kind of the basis also for what uh deepmind did with flamingo now it's basically we enhance the language model we have like our language model speaks five languages so basically what we said is how can we enhance the capabilities such that the prompt can be any combination of text or images right so basically we want to retain all the functionality of the language model but we also want to be able to understand images and here are some examples right so the the green text here is um what the model completed in a certain prompt and
if you are familiar with the clip discussion on address adversarial attacks right then you might find this funny i do um this was an example where the clip model from open ai got quote unquote attacked uh to for full transparency i argued at the time that this wasn't an attack but actually underspecified behavior um about an apple that had a label on it that says ipod and so if you basically ask our model what it is seeing here then it tells you precisely that an apple with a label on it that says ipod okay so that's cool but it's not just a captioning model i mean these have been done before captioning models you can also do something like that where you ask the same question on two different images you can ask what caused the mess on the carpet and so the model will tell you either the dog or the carpet was stained by a spilled drink based on visual observation right so this is kind of like reasoning in a way right
it's a light form of reasoning it's not what would make kind of the critics of conceptual thinking and deep learning happy but it is reasoning from observation in a way and also what we also have seen from these world models is world knowledge so on the left hand side the sign is clearly visible and if i ask the model about its contents it can read it back to me it can read it consists black lives matter and if i cut the sign off so that the model cannot read the complete sign like half of the sign is missing and i'm asking about the contents of the sign i'm still getting the correct answer how is that i mean we've seen this from the language models it's not it's not a surprise really that these models retain world knowledge so they're able to fill in information that's not in the prompt alone that they have learned from their world understanding um and there even do like there are even some things that that surprised me fundamentally
right this is one of them um this is an interesting image and uh the kind of the model ask what it's seeing says it's the score with a shovelzer on yellow background and then i looked up the word chavezer because i wasn't aware that it exists and it seems it seems it doesn't exist right it seems to be like an inventor word that is somewhat of a combination of shovel and maybe stinger i'm i'm not sure but from a human perspective it sounds right it sounds like maybe that's the shovelser who knows right and the same goes for that um if i'm kind of showing this hand-drawn treasure map to the model and i'm asking it the treasure is buried i'm getting in the old oak tree and if i'm asking in german i had like i haven't brought this example it uh says um that's alte eichenholz so it also kind of also translates if i ask in german and that's pretty interesting because it's implicitly doing ocr and some people have told me
that this is superior to their dedicated ocr models but it also knows that the red x must mean that there's the treasure there because this is not written anywhere right why does the red x mean that it's about treasure of course for us humans perfectly clear we've seen so many treasure maps that we we know what to expect and how to read them um yeah and then you've maybe you may have seen um palm pathway language models explaining jokes right so i've been playing around with some interesting observations explaining those so here you have a two trucks driving behind each other one is full of pallets and the other has a sticker on pallets wanted and so ask about this the model says it shows the irony of the situation a truck driver trying to get his pallet business back on track and ask about this this may be happened because the seagulls are not happy with the new rules um and this is also cool i like this image because it's funny and the
truck is a perfect target for the flames so this is remarkable right it's also remarkable that that palm is explaining jokes but it's also it's pretty interesting that this these are things if you would have shown this to me two years ago and told me that ai is answering these kind of questions i would have not believed it i would have said these are typical human things right these kind of things that we find these these things funny that we enjoy this this humor this is a very human thing and it it's it will be super difficult for ai to ex to explain or to understand these and you can ev every time i say understand somebody in the comments says ai's don't truly understand and i get that right i i'm not against that i'm not claiming that it's the same thing that human understanding is supposed to mean so and then i had a one discussion and and so my conversation partner told me that it's never gonna be agi until it can
get up in the woods and chop down a tree and i think that that's a fair point right and we are always in the information space here and if we are kind of being truly visionary about agi then uh we need to step go step further right we the the ai the agi the true agi needs to be able to help uh fabian out with his uh place in the woods um so let's have a look at that uh so this is again our multimodal model i'm showing it a picture of a cockpit of an audi and i'm asking what is the first step to start this car and the answer is turn on the ignition second thing would be putting the key and turning it clockwise until you hear a click sound from the inside of vehicle indicating that your engine has started up successfully then put into drive gear or reverse if needed for driving forward backward respectively so it has obviously some world knowledge about how to drive a car and that's
cool um but now i actually want to drive when i have it drive the car and one of the things we're currently working on that will be in the api soon tm is an explainable explainability endpoint that can give you the impact of everything in the completion for everything in the prompt and this is a preview here so i'm looking at the impact for the word key so i'm asking the model like this word uh how does that relate to your prompt and so this is kind of the the overlay that you're getting there so this model not only can generate text that can only it can it also has an idea on how this text relates to the text in the prompt or to the image in the prompt so i i don't think it's terribly unrealistic to assume that you could hook this up to robotics in any way right so that's a few steps out it's not super clear how to do this but why not right so um so i think
these world models can not only generate the same representation we have in their input layer so like a text-based world model can generate text or a multimodal world model can can generate multi-modal output but they can also be linked up to our physical world because they have an understanding of our physical world especially when you go into visual domain all right so what are some problems with these models um one of the problems is consistency and bias and um input dependency so here i'm asking our model two things twice the same question and once in german once in in french about its favorite sports team and in ask in german i'm getting bayern munich and ask in french i'm getting olympic marseille right so this is uh of course makes sense in a way right because you're saying that maybe a french person might prefer a french team or like a german person so and in understanding these models you should never think of these as having an opinion or an agency right they're just
giving pattern actions that are likely right so it's not like the model loves football or soccer it's just giving you likely completions um but those heavily depend on the input and um the lower prompt is from gpt3 so gpd3 works all rightish in german and if you prompt gbt3 in german and ask about the favorite sports team you're getting the new york giants and this makes sense right it's an english-centric u.s centric data set so the um world mall the world knowledge the culture the values that we have in these models are impacted by your selection of training data and sometimes in invisible ways so this is somewhat dangerous because those models are so powerful that the results cannot be easily interpreted another problem these models have are consistency so we have a value in function in ourselves that's driving truthful and consistent answers like our character our self-consistency and these models don't have that so i'm asking twice very similar questions the most incredibly beautiful german towns are and the first answer is city
of heidelberg and its castle hortonbook of de tauber with its medieval streets in bavaria you'll find munich and as of all the german cities of varying sizes the most beautiful are i'm getting berlin and hamburg which of course is wrong right but so you you so so you see right these models are not consistent you can't you can't use them as truth machines or you can't expect them to be consistent they don't have agencies and they don't follow goals so you have to in a way understand them as a web search if i would punch into like a web search all men are and the first result would be all men are lazy i wouldn't say the google ai is sexist right i would just say yeah that's somewhere out in the world somebody has written that right and the ai maybe maybe thinks it's a good completion it's a good search result it's not an view the the ai holds or like an agency it has and then this is also funny right so
these word models also have another difficulty with things that are almost known so these are images generated for the prompt a horse riding an astronaut and very clearly this is not an horse riding an astronaut but an astronaut riding a horse and i like to think that this is a mistake that we as humans also like to like to make sometimes if we hear some something in passing if we only see something vaguely right we might interpolate it as something we are aware of something that seems common to us and the same thing can happen with these models right because they have learned that riding is typically done to a horse and not by a horse but it turns out it's possible to get the right thing if you modify the prompt slightly so you can get this or this if you modify the prompt into a horse riding on back of an astronaut or a horse riding on shoulders of an astronaut all right so again right caution word of caution um that sometimes
if you encounter things that are highly unlikely uh in terms of the world structure that the model has learned it may get them wrong and the same happens with our model so this is our multimodal model and it says here no no regrets tattoo on chest for men which is again right almost correct and what a human at first a glance might have also seen um i'd like to call them worst walls traps and for for this reason it's super important to shape the human machine interaction in a way that takes these capabilities and properties into account and the last kind of thing i wanted to have you thinking about is basically so this is the pre-cloud era where we had an infrastructure uh functionality and intelligence functionality that was like algorithms or machine learning models and then the solution with ux and product specific industry solutions and now we are basically at a level where the hyperscalers are owning all the infrastructure layer and it has become incredibly more powerful the features we are
getting out of very simple to use in hyperscaler infrastructure functions is quite remarkable but still the ai the intelligence is is still very fragmented everybody is fine-tuning their own models and tweaking around with solutions and what will happen i would assume is that we are seeing general intelligence layers that will be owned by a few key players all right thanks a lot thank you jonas