I'm gonna start with the good news at least for me the good news are that Google research is now in Berlin in fact that's kind of where we are right behind that big tree on the left quite a nice view I'm super happy and super excited about this we're starting with the brain team working on on fundamental deep learning research and we'll expand that over the coming months and and years but after the good news I have to give you the bad or at least some of the bad and that is that the workhorse of the vast majority of today's deep learning and in general machine learning deployments in the wild namely supervised learning won't scale and it won't scale to the breadth of problems that we wanted to solve and it also won't scale to the kind of answers that we expect from it and this is mostly because what we want these models to produce are effectively structures or answers of ever-increasing complexity and if we want these models in the supervised paradigm
to learn how to produce these answers we have to produce labeled data that has labels of ever-increasing complexity so to give you an example right in the future we might not just want to ask deep neural network to tell us with some certainty what the category of object is that is depicted in a simplistic image no we may want that deep neural network to actually describe in detail the whole scene maybe even in natural language right we want to know all the objects that are present in a scene we want to know how people that are present in the scene relate to the objects how they use it what they say to each other if we are looking at video etc etc right even just designing annotation and labeling schemes for something of that complexity is quite challenging let alone producing those labels producing labelled data sets that have labels of that complexity is becoming difficult and also ultimately extremely expensive this gets aggravated by the fact that there's actually a very long tail of
problems a great example of an area where this is the case is deep learning applications in medicine where we want them to classify and detect all sorts of phenomena and overall in say radiology these phenomena are quite closely related but for every single type of question that we might ask these networks there is only very very few cases some thankfully very few cases in the wild because the disease might be infrequent but also sometimes it's just rare because we haven't had time to annotate it and for every single new type of phenomenon that we want and the classic supervised paradigm we would have to go re annotate data from scratch with yet different kinds of labels and that just becomes uh neck economic and in some cases downright impossible to do now there is an alternative and this is a paradigm oftentimes looked at as reinforcement learning something that works great for board games deep mind has used this for go and chess and pushed machines to play those games at superhuman skill levels and
that is directly learning in interaction with the environment and that works great for cases where you have simulators of your environment and where those simulations are fairly cheap but if those simulations become expensive or if actually interacting with your environment becomes very costly and maybe even risky that whole entire paradigm is is really quite challenged right so imagine you had a deep neural network that does some critical diagnosis and a medical domain learn on the job I don't think you would really enjoy that all too much ok so ultimately that might not be the exact answer to supercede purely supervised learning either so maybe we need to look at another page or take another page from from kind of the book of how humans learn things and be inspired and yet another way and looking at this example for instance this girl doesn't participate in tying her shoe at all right I saw it's all her dad but it's watching maybe sometimes a little bit more attentive than other times but ultimately it's probably learning
quite a lot about how to tie shoes and how that process actually works before the girl will ever have tried maybe it'll continue not doing it because it's you know enjoying how the dad does it and and and so forth but ultimately by the time this girl tries to tie her shoes for the first time she will have actually learned a lot about how this works even without ever having done it and so one interesting question is how much can we actually learn just from passively observing the world not interacting with it just passively observing and recent results in specifically natural language understanding actually show probably quite a bit one one case in particular that stands out or one result in particular one line of work in part color is called Burton now you probably guess that it's not actually the Muppet character but when I talk about Burt the stands for bi-directional encoder representations from Transformers right a bit of a handful okay so taking a step back what we're doing in Burt is
we're training a pretty large deep neural network on a task that falls into the category of self supervisor problems or self supervised learning overall and that's kind of a learning paradigm or a learning paradigm that I think we're going to investigate quite a bit here in Berlin and we'll also see in in many other areas over the next few years popping up beyond just language in in case of in case of Burt what we're doing is we're taking large amounts of text from the internet from other sources say books or Wikipedia is what we actually do in invert and we remove some words from this text we just blank them out and we now ask the model we train the model to regenerate those words for us so we actually create a supervised learning problem from otherwise totally unlabeled data right so there are no labels no human went and artificially labelled this text with with anything but we just created labels namely the missing words and input instances namely the text with the holes
in it just from naturally occurring text and in order to learn this the Steep neural network has to learn a whole bunch of different things at different so-to-say levels of resolution or levels of abstraction right so it might start with simplistic things that are even given through English grammar like look at the very last sentence has made cameo appearances within blank Muppets franchise English grammar suggests it's quite likely but the word in there is either right then on the other hand if you look into the third row of third line it says performed by blank Oz I mean you have to know something about the Muppets to actually know that this person's name the Muppet ears name was Frank but if you don't know that you're not gonna guess that this was Frank so in order to perform this task perfectly the Steep neural network actually has to learn not only a lot about English grammar about English language maybe semantics of language as well but actually also quite a bit about the world and
so it does this basically all at the same time and without us having to control for what it is exactly that it learns right we kind of steer it a little bit through the choice of hyper parameters but ultimately it has to learn all of these things so to recap Bert is trained to do this fill in the blanks tasks on Wikipedia on books and then once we have a model that can do this we take it in its parameters and maybe with the addition of a few more layers or a few additional parameters we fine-tune it and we adapt it towards specific tasks and typically supervised tasks so take for example question answering right we have some set of tens of thousands or maybe even hundreds of thousands of question-answer pairs not nearly enough to learn this problem particularly well in a supervised setting but when we build on what this model had to learn previously in order to perform this fill-in-the-blanks task and then we fine-tune that with the supervised data we actually
get much much better results in particular when Bert came out was the best system on a whole range I think 11 or 12 natural language understanding tasks standard academic natural language understanding best mark benchmarks and and was by quite a margin the best system and in those tasks to date and in particular it actually beat humans on a few of those and so there's a task called squad or Stanford question answering data set and on that one Bert actually performs slightly better when fine tune with the supervised training data than your average human and we'll get back to that in a little bit so to explain maybe a few of the fundamental ingredients that were necessary in order to build a model like Bert and in order to employ a model like Bert in this self supervised scenario I'm gonna have to go well way back into the Stone Age of deep learning right like eons ago in the year 2014 if you wanted to use a neural network a deep neural network to understand
language what you would typically do when confronted with the sentences you would embed the words and then you would run a recurrent neural network over the sequence of words in the order those words appeared in your in your text right so you would start with uh you would embed it and then you would run your R and n for one step and you would obtain some representation right a point in vector space some in quotes understanding of this word then conditioning on that understand you would consume the next word conditioning on on that consume the next word and so on and so forth and while these are nen's in particular in their LS TM flavor were extremely successful in understanding natural language and really kind of pushed the envelope by quite a bit say for example in the quality of Google Translate where we hope maybe even saved somebody's live and and somewhere in the Amazon jungle or so they had a severe disadvantage and that disadvantage is clear actually from the picture from the
simplified picture that I just showed you which is they have to operate in sequence they have to go through each word one by one produce an understanding in quotes then conditioning on that produce another understanding and and so on and so forth now this actually a training time is quite expensive even though it might seem intuitive to us because this is how we hear and how we usually perceive language it is actually making this through sequentiality making a quite expensive to train these are nuns so back in 2017 or so we started working on an alternative to these that we've since dubbed transformers and they're the key idea is that instead of doing this thing step by step you just do it all in one go so you compute a representation and then quotes understanding for each of the words in a sentence at the same time conditioning on every single word right and so you produce all of these at the same time then maybe you run this process again and you refine them
over time but you you do a lot of work in parallel and even just again from this massively oversimplified picture maybe you're getting the intuition that this is actually a lot more work there's a lot more arrows right that's kind of true there is a lot more work done but the big difference is that that work now is done largely in parallel and not in sequence and these highly parallel kind of sets of operations typically fit modern machine learning hardware much much better than long sequences of necessarily sequential computations and that gets us to the second main ingredient for bird and that's what we call TP use or tensor processing units so tensor processing units are Google's custom machine learning accelerator hardware this is a picture of a TPU v1 board a very early one actually that couldn't do any training of neural networks but this this hardware was actually used to serve the first neural or deep Google Translate models back in the day for for running inference and otherwise we couldn't have deployed
Google Translate at that time since then we advanced generations we went to TPU v2 s now we pack a lot more of them onto a board they are getting giant heat sinks must mean the processors are getting faster which is true but we also arrange them in such a way in kind of toroidal configurations that enables us to actually train deep neural networks very efficiently using those since then we have a third generation we're working on future ones the third generation is pretty cool because we use water to cool it and so now we have water pipes and data centers not everybody liked that idea so far it's working well I think but at this point we basically have models that are really well adapted to this type of parallel highly parallel processing units we have two custom hardware that now increasingly is also informed by this model design and those two things together actually allowed us to push models like Bert that you know similar things had been tried before not in exactly the
same configuration but it never quite worked to that extent but with those are in part building on those two main ingredients Bert suddenly and similar models since then opening is GPT two comes to mind for example have really started to to push the envelope in in our ability to do transfer learning and in particular transfer learning in a self supervised setting for natural language understanding understanding applications um so you might now ask this question does does a model like Bert understand language right because I just said it's better than your average human on the Stanford question answering data said so maybe it actually understands language and this might be a philosophical question and I would say the answer is no and the answer is no and in fact the answer is no not at all because it doesn't understand the world it's never actually observed many aspects of the world the way we perceive it right in particular it doesn't understand the visual world right it has has no notion of space also in a
certain sense in the notion of time etcetera etc and while I don't want to go all too deep into kind of the philosophy of all of this one of the things that you might wonder is could we apply very similar techniques now that we have TP use and we build building more and more of them and now that we have transformers and others models that fit these CPUs really well could we employ similar techniques to learn a lot more in a self supervised manner about the visual world or about other ways of perceiving the world that are maybe kind of missing pieces missing puzzle pieces in completing the machines understanding or in making the machines understanding of the world more similar to humans understanding and this is a direction that we would like to investigate quite specifically along with many other groups here in Berlin to give you a little bit of an example what a self supervised task could look like kind of akin to or analogously to to the bird fill-in-the-blank task for
something like the visual world consider this example so this is now a training instance for or basically depicts a training instance for an even larger deep neural network this is a paper published late last year I believe by a different team in in in Google research and the input to the model in this specific case is a pair of a single frame of a video in color and the remainder of the video where all color information has been removed and so it's just black and white and the expected output the label but the self supervised label the artificially produced label is just the original video right so data of this form can be generated basically from any video on YouTube you might not want to have all too many edits and so on and so forth but you can clearly produce those on mas right and this specific example is actually not a training instance this is the video on the right really is an output example from a real model that was trained to
perform this task namely to recolor eyes a black-and-white video given only a single frame in color now if you think about it how would a model how would some deep neural network actually try to perform this task right how would it take the single color frame and the black-and-white video and then turn that into a colored video well probably by identifying what the objects are in this video by tracking them through space and time in this video so saying okay so these arms that you see over there over the different frames even though they're sometimes blurry are always the same arms and so we know what color they had and these shoes are always the same shoes and so we know what color they had and the background stays pretty constant so we can fake they easily transform that color so implicitly maybe this model does this by actually learning how to track objects so how could we test that well we could show the model black and white video and paint one of the
objects and look at how it colorize is that right so this is not an artificial video and these are examples of doing exactly this so we take a black and white video or we decolorize some other video and we now color an object of interest by hand or using some existing method and then we apply the very same model that was trained on the self supervised task beforehand and ultimately what we see is even in cases where the objects overlap were they where they intersect with each other where a foreground and background change and occlusion order changes where the pose changes like in case of those judo fighters and the leftmost image even there this model actually manages to fairly accurately keep the outline of those shapes the same so it has in fact without a single labelled instance learned how to track objects but and this is a really nice example where it's quite carefully designed but a task that previously was purely in the domain of supervised learning or at least largely in
the domain of supervised learning can now be largely trained using data that requires no further human intervention data that just occurs naturally in huge quantities on the web so in in Berlin specifically we don't want to stop there we would like to learn much much more than just tracking individual objects just from observation right so we want to take the self supervised learning from videos and learning for and about videos quite a bit further so we're still a very young team we've only just started our first publications are going to go out to two for review shortly but this is kind of a bit of a preview so what we're doing here is we have a robot actually that's not our robot even though we have similar robots this one is in at the University of california-berkeley and it just moves around more or less randomly all day long and pushes around objects it bumps into a coincidentally in basically this pit full of plastic toys and there's a there's a napkin in there somewhere
and we just film it doing that sometimes also from different viewpoints etc etc and then we take minutes or hour of this footage and we train a deep neural network to predict the future or the kind of the remainder of a video just from the first frame or the first few frames and now that includes a whole range of complex dynamics right so it includes things like lighting right you can say you can see maybe that the lighting and all of these is slightly different and different lighting leads to different shadows for example right so on the second video you see the arm actually has a pretty clear shadow but it's in a slightly different angle than on the first video where the shadow goes a bit more off to the right and not so much forward and you can see that on some of the videos there is actually objects interacting with other objects right it's not only the arm bumping into something but something that maybe falls over etc etc and these are
all examples of you know still basic but physical dynamics or physics dynamics and it's working quite all right in particular because only one of these videos is real so the others are actually all made up by a neural network and I'm at least not exactly clear how quickly I would recognize which one is which so just to recap what we're trying to do here is we want to learn basic physics we want to learn object object interactions we want to learn maybe eventually even things like the semantics of objects what do people typically do with objects so we have an example for instance that's quite nice where the first frame of a video is someone holding an egg over a pan and the model just from YouTube has actually learned that the continuation is that yellow stuff comes out and goes into the pan right so these are kind of in a certain sense almost semantic properties of this egg in combination with a pan because maybe if you saw the egg without the pan
it wouldn't actually predict that it would get cracked right and we want to learn those things just from observation and our hope is that in doing so we can then do the same trick that people do with Bert namely to transfer from the self supervised setting to supervised settings in which we use massively reduced amounts of labeled data to learn really complex tasks such as understanding and describing whole scenes maybe captioning entire scenes but eventually at least in you know the somewhat more distant future we hope to push this to handling multiple modalities at the same time grounding language right really understanding how do humans talk about things that they see in the real world or also how to find things that you might want to see or even how to generate things driven by language right so imagine you have an interactive video editor or other content creation tools where you could use language to change what you want to see or its effect what you actually want to see away from something that
wasn't what you wanted to see but that's still you know fairly distant in the future with that thank you okay I said that the first thing in the morning I have a challenge with my colleague next door that I have more people so I probably have the backup better speakers thank you very much Jacob who has a question mm-hmm favorite person Jacob awesome talk thank you for for that so you said how can we bring the representations that our models learn closer to the ones that we have in our brain and I like that question that it's really really hard I guess but so what's our stance on how much reality do we have to bring into the data generation part do we need time and do we need manipulation like I mean obviously showed us that you do think you need that but and what about the task how much does the task influence the representation so yeah so basically my view on how similar the representations themselves have to be to human representations
I'm not so I'm not so clear on that part I think what I said was more aimed towards the the breadth of sensory input that our machines perceive today is massively massively narrower than what humans perceive right typically we expose ml models only to to a tiny slice of what humans perceive and that we need to broaden for sure whether this might then give rise to representations that are more similar to human representations or not I kind of don't even really care about as long as it works but it might be interesting if they are similar yes absolutely absolutely so there are some super exciting things that you could do if you if you did understand exactly how they relate and how similar they are and so on and so forth coming back to another question of yours because there were quite a few embedded I'm gonna try to piece them back together so one other is time I think a good understanding of time was actually rather essential to pushing pushing ml further there's
a lot of debate currently going on about causality and causal learning and that to me at least is intimately tied to this the human concept of time right and and so I think it's it's a very important topic it's not necessarily at the forefront of what we're trying to do with exactly this work but it's definitely something that I that I think is quite crucial um and then there was another interesting aspect that I forgot to your question my ulation how your task task as an informative a presentation so I actually think that it's one of the in a lot of model-based reinforcement learning work right now making the representations reward informed directly is something that is under emphasized massively underemphasized okay who's got another question first can you help me thank you for amazing talk I just have a short question so you showed us a research in which you fit a first frame of video and the colors video I'm just wondering if there's any research paper where I can read further about
this blog post yeah Bundrick at all Calvin Drake 2018 say it again von Drake gon DRI CK 2018 but I can also give you the full list of references offline wait I'm running so give me back who had another question there you are first then please pass it through to him yeah okay so the standard question about black box mm-hmm the is there a standard way now that people try to peel back especially these visually based systems to try to understand what they're queuing in on more is it's still sort of more of a perturbation based model where you're trying to you know sort of figure out what the model is seeing based on somehow trying to alter it and see what it's looking at yeah ultimately I mean it's a fuzzy thing to say whether or not they're based on perturbations like that or an analyzing kind of the potential for perturbations they still have that flavor to some extent research in that area is is very much evolving and and and it's making
quite a bit of progress I'm still not exactly sure that I think it is today producing truly actionable results and also I don't think we quite have the right tools yet to make it actionable even if even if they even if these insights were very concrete right it's actually difficult for us to to nudge say deep neural networks in ways that are not just modifying the training data in a certain kind of targeted manner and that's definitely still vinaya so quite a bit of an unsolved problem yeah there's quite a quite a few people working on it maybe not so much in Brian Berlin quite yet but in the brain team overall there's a quite sizable team looking into this beam Kim would be an interesting name to look up research from in that area okay I think we've got time for two more questions yeah can you thank you thank you very much for the speech and I have a question in this model that you show us with this Birdman you know when
when you have the black card sentence with all of the connections not only in the in the Arin and you think do you use embeddings like word Tuvok or globe or you skip this face good question we we do learn embeddings but we learned them jointly so we don't use objectives like the ones use and we're Tyvek or and love separately as a pre-processing step we start from scratch with randomly initialized word embeddings and then learn them in a single intense okay who wants to ask another question oh sorry Eris thank you so much for your talk um I have a question on where would you see the biggest impact for the implementation of a project like Bert it's actually a bird internally to Google already has quite a few Bert and Friends so to speak already has quite a few applications the most impactful superlatives are always tricky I think the the one that we've already internally proven to be incredibly impactful around question answering so there Bert models for scoring or for judging
whether an answer is a good answer to a given question our massively beyond what we were able to do before but that's not the only one so still in question answering we now also have models that are inspired by these kind of self supervised pre training schemes so basically Bert's style models that judge whether two questions are the same or mean the same thing or whether they're slightly different and in certain potentially very abstract ways right whether one might be more suggestive than the other and so on and so forth and that's another area where where these models really exceed the state-of-the-art in quality there's many more actually by now there is hardly an NLU task that doesn't improve somewhat at least if you tweak the set up just right and make the model just big enough and so on and so hey sorry I'm sorry how about the visual equivalent of the bird model so cuz I can imagine that for a way no for example like for yourself I think it's it's very
important to get the basic physics but but so far it doesn't work well enough okay so this is really an area of active research where I think there is promise I think there's already some evidence that there will be something there but ultimately it doesn't work yet or doesn't work well enough yet I think the the the colorization work or tracking work that I showed even in wondrich's paper and college paper they're getting very close to supervise baselines but they're not able to exceed them and which is still in a certain sense magical because all the supervised baselines use data that was quite expensive to create but it has been created so why not use it right why tie your arms behind your back I think it's a question of time I also believe quite strongly that it's a question of model size so Bert uses 64 TPU v2s or even VIII's for about four or five days to train a 350 odd million parameter model we use 256 even just for the simplistic robot
pushing thing right and we use those for quite a bit longer time probably we would want to have many more computational resources to throw at this and and before we expect to exceed from this much more undirected kind of self supervised learning before we would expect that to exceed very directed supervised regimes but I think it's just a question of compute budget better understanding of the models in the training paradigm before we get he was one more question I just wanted to quickly ask if this self supervision mhm idea is identical or inspired by frist ins predictive coding paradigm isn't the same or such a different technique I mean in a certain sense it's not it's not a crisp it's not a crisp kind of definition and I think they they fall roughly in a similar category right there's contrastive predictive coding Aaron Bernards work which i think is very much self supervised as well I mean generally the idea I think that the common denominator one way of characterizing it could be which is
I think what young laocoon is saying these days is self supervised learning is the new unsupervised learning I'm not exactly I think there's a distinction the distinction is that we're we're bringing to bear all the all the work that we've done on supervised paradigms by constructing supervised tasks in fairly simplistic ways out of data that occurs naturally or more now I think the way we construct these supervised instances will actually change quite dramatically over the next year's I think there's a there's there's metal learning applications without end actually in in trying to construct these tasks and we're gonna see a lot more there but yeah self supervised it's it's a pretty fuzzy notion I like it a little bit better because it makes it then unsupervised for the specific direction because it makes it clear that we're using the same methods and the same mechanisms as in supervised learning that's just on different data