maybe before I start my talk just connect to what has been discussed previously understanding language it has many reasons why this is hard for a is one of them that has been suggested is you need to you know understand what the individual symbols that we use in language actually mean and so this is the famous AI problem symbol grounding how do you grind ground symbols one of the sort of sort of thinking around this problem has been that you need bodies because bodies enable you to share experience so to mean what it what it you know to understand what it means to I don't know climb a tree you need to actually have climbed a tree if you just manipulate symbols on the internet its initially words connecting together a symbol for climbing and a symbol for tree that doesn't work you're just you know connecting words that you really don't understand what the physical experience of doing it actually means one of the reasons I like doing what we do which is connecting AI
and robotics is that we're giving a eyes bodies for the first time right we're not a software that lives on the internet and manipulate symbols we're and that's what drove us into the field initially we're doing things in the real world so we have a eyes we essentially throw them into the world find some way to teach them to deal with the world that they need to live in and do something useful in our case because we have a company and we need to make money about the yen but you know God has into the field and what interests us is how do you ground symbols how do you learn actions and perceptions that are actually a step towards understanding how to act in this world not the internet okay and I've brought a couple of applications of you know what we can do at this point with industrial robots so most people especially with a Berliner audience don't really know much about production or industrial robotics I guess that's true here so if you
just listen to the political discourse around the topic you would assume you know it's a huge thing angle Americus talks about you know capabilities of the German national economy and it's all about robots and AI and we need to keep the knowledge and we do not you know we must not give the knowledge to the Chinese and when she does that I'm always wondering who's she talking about I mean there's cuca they make industrial robots and there's us I guess and maybe two or three more startups she doesn't know us but you know AI for industrial robotics really hasn't happened today and that's where robotics today goes through movements there that are completely predetermined the technical word to say it is there fully understood geometrically so if you imagine a car production that's something we do a lot in Germany in Ingolstadt where Audie makes I think the q2 or something howdy knows exactly at which point in time a car is we're in the factory and when they move a robot to put a
weld line in they move they just synchronize the movement of the arm to the movement of the car in the factory and both of them coincide at the same place there's really no coordination or sensors between those two elements of the system right so the there's no perception going on with the robot to know where the car is or do anything meaningful that just happened to be at the same place at the same time and that works if your audience completely control everything and you know go visit the factory it's awesome it's a ballet of coordination and it's super silent super concentrated and it is that's the case because they know exactly where everything is and how it will move at any given point in time and then you move one step down to the people who make the parts of these cars and it's not a ballet anymore it's just machines and dudes you know pushing metal between these machines and you don't know where everything is so as soon as you approach a
type of problem in production where you need to find out where something is to do something you're out of luck and you typically today employ a human so if you're want to plug in a cable for instance the only thing traditional robotics without a I could do was the following you get a 3d model of say you want to plug in the familiar lightning connector into your iPhone the only thing you could do if you wanted to do this with a robot was you get a 3d model of the phone you get a very expensive very complicated 3d camera and it gives you a point cloud about know the shape of this thing and then you fit the model into this point cloud very precisely and then you know you know the six dimensions of your intention of this phone and then you orient a robot the same and you make a simple single linear very primitive movements towards it then you just orient the robot as the phone is and then you move forward
and that's it right when we humans plug in cables that's not what we do at no point when I have plugged in a cable in my in my life have I known where this thing is I don't care what I do is I just go towards it somehow and wiggle it and make small corrective movements as I approach and this is precisely the capability that we give to robots with our product me right so this is obviously deep learning based so we have cameras that watch a robot make of movement and we're doing this with a new type of robot that is easier to deploy because it's less dangerous traditional industrial robots are very fast very hard and have completely no perception whatsoever so if you tell them to go from A to B and between a and B you happen to be in it will kill you literally kill you now with Universal robots and some new machines from Coca and ABB we have a new class of robots that will at least notice
you know notice when it drives into you and stop which means it's all of a sudden less dangerous to be close to them and you can actually interact with them and one of the things you can do when you have that capability is you teach them you show them how a good plugging in movement would look like and that's exactly what we do when we train a robot to plug in for instance a cable into a phone we grab the robot by the wrist as you would when you teach a person to you know play tennis or play an instrument and we guide the robot through the movement while our product Mirai watches with a camera - and then after a couple of hours of demonstrations where the robot makes the movement with trainer and us watching the data gets sent off to the cloud and back comes a policy that knows how to make the right movement given any of the images that you show and the robot can make this complex real-time correcting
movement on its own so before I show you an instance of this and I will have to do a great deal ex of explaining of what it is that we're doing there because production is always very special just a general setup of the system you buy a robot any robots we don't make the robots right we're a software company you buy a robot for instance from Universal robots you buy a force torque sensor so you feel the forces on the wrist you get a camera from us and any type of end effector you connect all these and there's a tablet app and it's really simple to use anybody in this room could train a robot to do a very complex movement and to solve a very complex computer vision problem in production it's really we made it super simple to use you connect all these you do two hours of training wait a night try it out typically you iterate this two or three times and you're done you have a very complex real time
controlled vision driven system that knows how to find position and end effector towards a visually perceived goal ok this was all the technical terms in one sentence so we've done we've done we've done that let's maybe move over to the video so I can show you how this actually looks when you do it so while the video starts playing I'm going to explain to you what we're doing there imagine you're a factory and you make fridges refrigerators and you're a premium brand so you don't want them to break the premium brand we're working with here is familiar what you need to do is make sure no coolant leaks out of the cooling system after production because if some coolant leaks out after four week four weeks the fridge will break the way this is done to the restart seem like 12 soldering points on the back of a fridge where copper tubes that have been put in by humans so in nobody knows where exactly they are and how exactly they look where copper tubes
have been connected through soldering so this is what the back of a fridge looks like and this is these are the connections we highlight them now but these connections you don't know exactly where they are you don't know exactly how they look and some poor dude his job is to put a sniffing probe that's a little artificial nose to any one of those soldering points and try to sniff if some coolant is leaking out there nobody wants to do this and of course after half an hour data gets they get very sloppy so Lipper had I've tried to automate this for I think 20 years or so and many many people have failed now with collaborative robots and our product Mirai this was a matter of a week of training a robot and solving it so what you're seeing here is a robot so this is quite close right this is like this big and in reality this is simulation of the sniffing probe we call it in a maze and beyond you can see
why and this is the camera that watches the scene and this is the back of the fridge and with a single RGB color camera we're going through these movements so what we did is for each of the individual joints we trained the robot to find it and touch it no matter what it looks like and no matter where precisely this we know roughly where it is right there's a ball in space where it's going to be because most of these fridges are similar when we know the model but really the the last two centimeters of the movements are guided by the camera and are you know literally movements like we would make them when we plug in a cable or when we touch one of these things and the cool thing is this is real-time so you can really mess with the robot as it tries to touch these things and that's what we're doing in the next 30 seconds right so this robot just tries to touch the soldering point and now I'm just
wiggling it about to show that you know that's all this robot wants to do at this point in time it wants to stay there and sniff it that's that's what's going on so normally robotics people get very excited at this point because that's completely unheard of and nobody's seen it real-time control from cameras with a closed loop control system is not something that anybody outside research is doing and we've just productized it right now I believe we're the only people in the world who can do it especially as a product and then you know while we're messing with anyway I just said okay let's mess with the whole fridge that's not something that would happen in a factory and see if it still performs and finds that and finds the goal so that's what's what's happening here yeah we need to wait for the video to catch up with me right so robot wants to touch and then I decided okay let's let's move that thing about and it just does it I think we'll
see one more we'll see one more example of this moves right yes and now I just hit it hard and it needs to go here in this case right it just approaches and even as we wiggle it while it moves we go there okay the rest is technical stuff and boring so we should go back to the presentation so this is not about sniffing fridges the world market for the sniffing fridge application is maybe 15 installations this is this I mean there's one at leap Amelia there's one at LG there's one at whirlpool we're talking to all of them after we've done that so this is not what it's about right this is a highly horizontal product it's as horizontal as Universal robots themselves are what we do is we take the III to the actual factory record the data on-site and solve whatever the movement problem that they have there is right so we're not unlike you know Siri where Amazon has trained something and deploy the result what we're doing is we're deploying
the learning system take it to the factory record the data there in a very specific manner to the workplace there the lighting conditions you know that there may be loyal in the air and chickens running around in the background you don't know right any of these variants we learn to deal with on-site record the data bring it back to the cloud create a policy install it on the robot and have it work and this works for essentially anything where you need to hand I coordinated movement so the way we sell this is we go to a factory and we go to the production engineer and tell them okay now let's go through the factory and you show me the people you keep around where you didn't automate the people you keep around is exclusively for the ability to coordinate how they move with what they see because all of this could be done with a robot instead of a person so yeah and this is just a technical summary of that here for those of
you who like to see some more videos here are a couple of bar codes where you can't see more videos where we do interesting things this is a case where we plug in Ethernet cables there's I think yeah the first one is just the whole demo where we also show how the training process looks like and the second one is a slightly more industrialized case where we're actually you know why bring up a 19-inch patch panel where of course we in software know where each cable needs to go otherwise we would just do random things but then we drop off the robot in front of the right in front of the right socket and then the mirror system our system does away with the variance you can even hit the 19-inch rack while it's moving and it will still reliably plug in the second one you've just you've just seen we have a couple more of these if you're interested follow us there will be more on the website quite soon we can do this
with essentially all the Universal robots machines the six of them right now three different sizes and then slightly different equipment if we need more speed more power and a bit more danger we can also do this with ABB robots we're in no form bound to collaborative robots it just happens to be the case right now the market is crazy about them and buys Universal robots in large numbers so most of the requests we get today are not for traditionally be fast machines but for non dangerous Universal robots machines towards about the company we've been founded in 2014 were sort of a weird animal were really AI people so you know the discussion about artificial general intelligence and how to make language understanding or understanding anything at all instead of just predicting stuff it's really close to our heart it's really you know long-term what I want to do in my life and this is essentially a vehicle for me to make some money on the side we spent two years between 2014 and 2016 figuring
out how doing AI in Germany makes sense right if you if you do AI and you have a bunch of good people and you have a bunch of algorithms that are that that make sense how you do we do you do something where you learn something from the people around you and it wouldn't make more sense to do in Israel or Silicon Valley or at least in the UK and industrial robotics or production topics are sort of a natural match for us and that's how we ended up doing is we're 20 years people now in Berlin with the sales office in New York we venture backed by project a we have invested in November last year kopelyan and Vito so you know all of these these guys are very good operationally which helps us a lot and these guys obviously are close to just production issues and help us a lot there that's me I talk about robotics and AI on LinkedIn sometimes so if you want to follow me and I'll now be ready
to take any questions you might have questions that start here with two years yes thanks for the talk very interesting stuff and since we are building sensors for robots I also got pretty excited looking at some of those since I would I would talk like in the one of the after the next so so what I'm the capability is pretty strong and I think matching your hand-eye coordination from robust perspective mmm it can be transferred to a lot of other things robots do so how difficult would it be to you know the to do to transfer this mechanism into an autonomous driving system for example because that's something we're looking into so how can that help so you know they're very generic so let's not go to autonomous autonomous driving right away but people have approached us about like harbor cranes this is essentially a forty-year movement everything Jay goes about and placing containers and to removing container ships is a high-end skill that you need Heights right highly trained operators for the algorithms could
totally do it the company alas not we need to focus at least on robotics this is wide enough so I I'm fortunate to say no to people who want to do Harbor cranes and diggers and you know whatever else I have been approached about there will be however a generic cartesian interface that we add to the product where if as long as you don't tell us you're free to you know try to hand or whatever your actuator is actuator camera coordinate essentially any movement and then if it works good for you that's excellent so I believe by the end of the year we will have this generic interface with an API buy one of these you connect whatever your cameras are and you connect whatever your machine is that you want to coordinate and just try it and do that autonomous driving I think is different because it's more complex and layered right these are very these are very monolithic skills I think autonomous driving is typically skills and layers of skills on top of
each other interfering with each other and trying to prevent errors in the autonomous autonomous driving space we're strongly on you know even side cameras work this world is designed by humans for sensors that are human which means RGB not radar or anything else so the most information in pretty much any case where you want to act in the world you get from RGB data the spectrum is there it's usually free nobody messes with it so as long as you can deal with shadows and sunlight and we're getting very close with this stuff I mean most of the applications we do work in direct sunlight as long as you don't you know completely over to it so I think the visible spectrum is just the best sensor modality for anything that interacts with the world with humans in it which is the world great any other question there's one I think's for the presentation I'm Raphael from Simmons okay I have a few questions so one if you're really using reinforcement learning to do this and
if you are how you're handling also the transfer learning for other machines so if you teach one machine how can you transfer that to multiple lines and how you're also running that locally or you providing your own hardware or you're working with third-party hardware so easy to reinforcement learning the answer's no not at all there has been a great deal of excitement around reinforcement learning the problem is for industrial use cases you do not have the day time reinforcement learning we're forced in learning you need to be able to roll out to multiple machines so you know Google X did this I think 2015 or so for like 20 robots in a basement and had them rehearse gripping of random objects what they got out of it was a mildly impressive gripping demo and they spent 4 months and 20 robots on it I cannot propose this to anybody making fridges right they buy a robot I want to start right away so the only way to make reinforcement learning work means simulation so you
essentially you need to do some form of randomization and treat reality as some weird form of the simulation that you've that you've trained in I believe there's something there but this also means you need to build the simulation first in a production context for many of the things that happen in factory nobody has a simulation nobody nobody has even heard of it right could I have come up with this demo where we sniff copper tubes on a fridge half a year ago before I had met the Liebherr people hell no I had no clue that this is going on has anybody you know a meaningful simulation for the contact forces that are going on there absolutely no - so as long as nobody figures out how to just take a snapshot of a room and get a decent physical simulation out of it we have a strong belief that using reality as its own model is the best thing to do and that's what we're doing here we're recording all the data we learned from
on-site air to actual thing transferring between different machines you need to make sure that the variants that you see between individual works work participate workspaces has been oh yeah I'm over has been has been has been shown during training right if you do that you can do it it makes it learning more complex the other way to do it is just make it you know make make to make you new real working stations look exactly the same we are bringing hardware not out of you know we have no passion for this whatsoever it's just we need GPUs there's no GPUs in any production context today so what we're selling this is essentially what we're doing there is we're setting a software license as a little box to the little box is essentially a gaming PC it's that never updates it brings a it brings a GPU and that allows us to do it I believe you know once cement and through a Packard Enterprise and all the big guys have had their IOT Wars in
a couple years we will know what a type of accelerated neural network inference architecture there will be in factories and will be perfectly happy to run on this one right now you need to buy a piece of kit and connect to your robot