Rise of AI Conference 2018 · Berlin

Liability Concerns of Autonomous Machines

Helen Tung · Legal futurist

At Rise of AI 2018 in Berlin, legal futurist Helen Tung explores who is liable when autonomous machines cause harm, using three case studies. In a fictitious 2050 scenario, a person is told by an AI doctor that new medication will extend their life by fifty years, but that sensors predict a 99.95 percent chance of future cancer, heart disease and depression, triggering a higher insurance premium; the same person's autonomous car is then hacked, and after a crash an AI judge finds them fully liable for retaking manual control, a ruling later reduced to 25 percent liability on appeal to a human judge. A second scenario asks whether a robotic dog left 80 percent of an owner's will could hold legal entity status. A real case she describes involved two horse owners disputing which owner's horse a jockey could ride at England's Derby, resolved by judges the morning of the race. Tung then shifts to a more personal argument, invoking Clayton Christensen's theory of disruptive innovation and life-measurement questions, before leading a short guided breathing exercise and closing with a Wizard of Oz metaphor about people already possessing the intelligence, heart and courage that AI debates worry about outsourcing.

Key takeaways

  • In Helen Tung's fictitious 2050 case, an AI judge initially rules a hacked-car driver 100 percent liable, but a human judge on appeal reduces liability to 25 percent.
  • Tung argues predictive health sensors reaching 99.95 percent accuracy could be used under future EU rules to justify higher insurance premiums before a condition even develops.
  • She raises open legal questions about whether an inherited robotic pet could hold legal entity status or be considered to have consciousness.
  • In a real Derby horse-racing dispute Tung witnessed, judges ruled on which owner's horse a jockey could ride just hours before the race began.
  • Tung argues that regardless of how capable robots become, people retain something machines lack: a human heart.

Detailed notes

A fictitious 2050 case: AI doctor, hacked car, AI judge

Tung sets up a 2050 scenario in which an AI doctor tells the protagonist that new medication will extend their life by fifty years, but that nanobot sensors predicting a 99.95 percent chance of future cancer, heart disease and depression will trigger a higher insurance premium under incoming EU regulations. On the drive home the protagonist's autonomous car is hacked; a caller threatens to drive it off a cliff unless bitcoin is transferred. The protagonist refuses, tries to retake manual control, and crashes. Facial recognition and satellite footage bring the case to an AI judge within six weeks, who rules the protagonist 100 percent liable, reasoning that nothing would have happened had they not taken control and that their own intervention caused the collision. On appeal, a human judge shows sympathy and reduces liability to 25 percent. Tung poses this as a question for engineers, economists, politicians and researchers in the audience: how would they decide such a case.

A robo-dog inheritance

Tung describes a second scenario: a robotic dog bought in 2018 that by 2050 fetches, delivers mail and acts as a bodyguard, to which its owner leaves 80 percent of their will, prompting family objections. She notes that people today already leave portions of their wills to pets, citing a real case in Australia, where an executor would typically set up a trust fund for the pet's lifetime. For a robo-dog, she raises open questions: whether it could hold legal entity status, what happens if it is leased for a hundred years and repeatedly upgraded by different companies, who then controls it, and whether it could be considered to have consciousness.

A real dispute over a jockey and a racehorse

Tung recounts a real court case she observed involving England's Derby horse race: a jockey who trained with one horse owner fell out with them and began training with a second owner's horse instead. The first owner sued to prevent the jockey from riding any horse but their own; the second owner argued the jockey was now training with him. The judges concluded the dispute was no more complex than any other case before them and ruled, delivering their decision the morning of the race, just hours before it began, that the jockey could ride the second owner's horse.

What robots cannot replace

Tung shifts to what she calls a more personal argument, describing a talk by Clayton Christensen, whom she calls the godfather of disruptive innovation, about being asked to work an extra Saturday, realizing the request conflicted with his family values, and later developing his theory of disruptive innovation after leaving that job. Tung uses this to argue that no matter how intelligent robots become, people retain something machines do not: a human heart. She leads the audience through a short guided breathing exercise focused on releasing a difficult emotion, then closes by comparing the search for answers about AI to Dorothy's journey in The Wizard of Oz, where the brains, heart and courage the characters seek turn out to already be within them, and mentions her involvement with the Moon Village Association.

Questions answered in this talk

Who is liable when an autonomous car is hacked and crashes after the driver retakes manual control?

In Helen Tung's fictitious 2050 case study at Rise of AI 2018, an AI judge initially ruled the driver 100 percent liable for retaking control of a hacked car, but a human judge reduced that liability to 25 percent on appeal.

Could a robotic pet inherit part of someone's will?

According to Helen Tung at Rise of AI 2018, people already leave portions of their wills to pets today, with executors typically setting up a trust fund for the animal's lifetime; for a future robotic pet she raises open questions about whether it could hold legal entity status or be considered to have consciousness.

What does Helen Tung say humans have that robots do not?

Helen Tung argues at Rise of AI 2018 that no matter how intelligent robots become, people retain a human heart, and she closes her talk by comparing the search for answers about AI to Dorothy's journey in The Wizard of Oz, where the qualities sought were already within the characters.

Mentioned

autonomous vehicle liability, AI judge, legal entity status, disruptive innovation

Keep exploring

More from Rise of AI Conference 2018

More on Ethics & Trust

Transcript

Auto-generated transcript, not edited. · 3,315 words

Show the full transcript

humph thank you so the year is 2050 humanity has mastered the art of machine learning we understand machines inside and out we understand autonomy we know the risks of safety the percentages of accuracy and what is going on or do we my name is Helen Tung as said before I am a legal futurist and what do I do so I work with businesses startups and people like you to look at the legal issues of the technologies that you are currently developing some of the things that you might not know about how you're unsure about it's the things that I work with to give you an idea about eight years ago I started looking at Ratana shipping autonomous vessels at times cars and eight years ago I can assure you many lawyers that I spoke to didn't know what on earth I was talking about somehow Google cars has made it to the scene now everyone knows what Google cars is all about I work with a lot of startups what hapenned for instance in apps

profiler in data future cities in Hong Kong and these startups are a very symbolic and important to me so to help me to help you today I want to talk about three case studies one that is fictitious and two that are real and what I'd like to invite you to do is depending where you're an engineer economist politician researcher is to think of how you would deal with this problem if you were in that situation so that is the first part of my talk the second part of my talk is we're going to go a little bit personal and a bit deeper and to touch upon the subconscious mind and why that is important so let's talk about our first case study so you are the leader of the next spacefaring nation in 250 we have already landed on the moon with our first basic selling colony you work with your clone your clone is very busy managing things in space whilst you're busy on earth you have a lot of Robo assistants they work

wonderfully efficiently and there's very little that you have to really care about on this particular morning having your breakfast and you go to your fridge now the fridge of 2018 is really boring but 2050 your fridge is your dietitian assistant and in fact all your life's medication and also vitamins are all supplied by your fridge they will order everything that you need to keep you going you jump onto your autonomous car you go on the Autobahn and today you're seeing your a eye doctor you go to your doctor oh good morning doctor and our doctor goes oh good morning I have some good news and bad news for you which one do you want to hear first you go well tell me the good news well as you know technology has advanced and we've got these great excellent medication what it can do for you is we extend your life by 50 years so you think to yourself we sort of set that luck six months ago so if I add 50 to 50 I'm

going to aren't getting it a hundred years so I'll leave about two hundred so years that's great so tell me what's the bad news the bad news is because of the nanobots and the sensors we've got in your body we know that there's a 99.95 accuracy that you're going to deserve develop cancer heart disease and depression oh dear so what else is going to happen well a eye doctor goes as you know EU regulations are coming in and because our predictions are so accurate that you are going to have these conditions at 99.95% be prepared I think the EU are going to slap on high insurance premium if your situation doesn't change in the next three months so okay you hear this news and then you go back you jump on your you know your tongue as car you start sinking wait a moment is AI actually helping me what is it hindering me and then lo and behold suddenly you see this flash in the corner of your car beep beep beep you know

what's happening your car is being hacked Oh crisis now what so you've got one of three things you can call the cyber cops you can try and tackle the hacker yourself or you can manually try and outmaneuver your autonomous car so you think quick on your feet ok calling the cyber cops is easy I've got this red button it's emergency button bang but there's a problem there's a time lag fine then you think about the hackers and then you get a friend hello hi I am the hacker in the next 15 minutes I want to transfer I want you to transfer all the bitcoins that you have in your account otherwise I'm gonna drive this car off the cliff and you're never gonna see daylight again the line goes dead you go hell no I'm not gonna transfer any of my bitcoins to anyone I'm going to take control my car and as you imagine it's actually quite hard because your cars already hacked but you somehow managed to swerve on the Autobahn and then

you crash your car into the fence thank God nobody dies there's some scratches but obviously you can imagine the pile behind them the Autobahn and because of face recognition and all the satellite TV everyone knows what's going on so in a particular road traffic accident it usually takes about a year in the English legal system in this particular instance in six weeks you are invited to go to the court and you have to appear before an AI judge you get all your documents through all the photos and you explain to the AI judge look this is what happened to me and the AI judge makes a decision very promptly I'm afraid you're at fault you took controls the car with your hands you were hacked but nothing actually happened you preempted the collision and you collided you're absolutely 100% liable you stand up and you go hey this is not fair I'm the one that's being hacked with anger frustration you appeal the case and then you appear before a human judge and you go

look this is what happened to me and the human judge looks at you as much sympathy and says you know I understand how it's like that could have happened to me so I'll give you a reward of 75 percent you're only liable 25% because of what you did these are the kind of questions that I invite you to think about when you do the work that you do these decisions that you have to make that's the first case no second case scenario robo-dog how many of you own a dog or a cat or a pet okay very loving companions right so in year 2018 you purchased a Robo dog the latest of its kind apps are amazing it reacts totally like your normal dog come 250 they're totally Fister catered their assistant they bring food to you they deliver your mail and in fact they can also be your bodyguard that is how good they are you develop such a great relationship with them that you really feel that by the time when you're no

longer on earth you want to leave them a legacy you want to leave something to them so long behold what you do is you bring out your will and you go I will give 80% of my will to my beloved of robo-dog now obviously when your family finds out they're absolutely livid they're like what is going on now before you start smiling and thinking this is a joke in 2018 and beyond before sorry there are real cases of men and women who have after they pass away dedicate a big portion of their will to their pet dogs in fact there's a case in Australia where when that happened this is what would happen still on earth is the executor will try and do their best to deliver the will they'll probably set up a trust fund of course looking after the pet dog for the lifetime of this dog making sure they're looked after and they're well kept but that would probably the end of it so what would happen in 2050 now this is

where the interesting questions come along the first question I would ask is does this Robo dog have legal entity could we even classify it equivalent to a human being a real dog or would it be part of an entity what if Robo dog was subcontracted or contracted out for a 100 year lease and every time it's actually upgraded by another company who actually has control of Robo dog in is Robo dog really truly your pet and of course the big question is does it have consciousness third case scenario horses and I'm sure all of you have sometimes go horse racing well in England is a very big race called the Derby how many of you know about the Derby right so people dress up look glamorous and have a wonderful time and this is a true story so I went to court and I dealt with my own case and then one day in a corner I saw there were many many people say oh let me have a look what's happening I went into

the courtroom and I saw three judges and as all the parties lined up so the case was as follows party a hide the jockey to ride there for the Darby they trained for a while things were fine and then they fell out they absolutely hated each other's I can't work with you anymore so what happens is the jockey goes to owner B and starts training with B's horse ok time passes and soon the race is coming it's very exciting owner a finds out what's happening was owner B owner B is actually training was my jockey so long behold he goes to court and files a claim says I am not letting this jockey to write your horse at the races and if he's gonna write any horse has to be my horse and then owner B goes of course that's ridiculous this jockey is mine he's riding my horse so the judge listens to the argument back and forth about whose horse it is or the financial implications and how this is going to be

make big news in the sporting world and the judges of this there is nothing too complicated in the court that we cannot deal with and there is nothing too difficult as a judge we cannot decide on and then after that he dismissed everyone he says judgment will be made tomorrow at 8 a.m. and get this the race starts at 11 a.m. so you can imagine no one slept that night because we don't know who's going to write the horse correct well as you know human rationality comes in and they decide to let the horse race as party B so these are the kind of questions that I put to you and many there many many more scenarios which you may want to think about now I'm moving on to the second part of my talk which is a bit more deep and a bit more human now there's someone I want to bring up and you may know of him it's Professor Clayton Christensen he's one of my favorite authors authorities on disruptive innovation in

fact I call him the Godfather of disruptive innovation I met him last year at singularity University and he gave an amazing talk and this is something I want to share with you he talks about his work he was an excessive executive he goes to work the busy nine-to-five job and then one day his boss goes to him hey Chris Christian can you do me a favor can you come to work just this once justice on Saturday he goes well Saturday I've sort of forgotten gage mminton you know it's family time I don't I don't really work on a Saturday oh come on just this once so it goes home talks to his wife and his wife goes well it's family time we don't we don't work on Saturday so he goes back to his boss on his and says sorry boss I can't work on Saturday so his boss goes how about Sunday Sunday will I go to church and that question just this once changed his life he realized that this job was no

longer in line with his values and I thank him for quitting his job because then he went on to become a professor at Harvard and he developed the theory of disruptive innovation he has a book called how will you measure your life and that I really want to share with you he talks about high school reunions college reunions how for some reason in the 20s everyone is young enthusiastic they have the best job in the world the best pay best friends lovely girlfriend boyfriend married you name it and then for some reason in the 30s 40s 50s 60s the tone is different they all come to the reunion oh I hate my boss I hate my job I hate my colleagues I don't get along with my kids I think I'm getting a divorce I'm getting depression I'm addicted why why does this happen just when we're at the peak of our careers there are personal lives start crashing down these are really big questions that I'm posing and what I want to do is

say what can we do that robots cannot do or what do we have that robots do not have and I would say it's this we have a human heart regardless for how intelligent the robots are going to be so what's your permission I'd like to do an activity with y'all and it's what I call a meta meditation it's gonna be very short you don't have to take part in it or you can just listen I know you had a big lunch but if you may just get comfortable in your seat you know put down the things you've got phones writing what not but what I want to do is just for a split moment the three minutes that I have with you I just want you to reconnect with your heart so whenever you're ready I would invite you to shake again be comfortable and you just have to listen that's all you have to do I like to invite you to breathe through your heart three deep breaths one and out two and out

and three and as you're breathing in notice the colors that you see next I want you to think of an emotion that has been on your nerves for a while it could be anger sadness frustration whatever it is and I want you to visualize it and see how big it is and I want you to embrace it into a big ball and what I'd like to invite you to do is I want you to breathe it in when you breathe it in I don't want you to do any caveats or cut it short I want your breathing the whole thing and from the top of your head let it enter your body down your face neck shoulder stomach legs and once it's through your body I want it to go down past your feet all the way down to the core of the earth down down down when it touches the hot core of the earth I want it to come back up up up up through your legs through your body through your stomach

heart shoulders neck face head and when it comes out from the top of your head and you release it out into the universe I want you to say to yourself whatever this emotion is there is enough love to absorb it there is enough love and compassion to deal with it and when you do that that emotion is out of your body whatever that's been holding you back is no longer there as you breathe a few more times notice the changes in the colours breathing through your heart and noticing the stillness filling your heart filling the compassion and feeling how wonderful it is to be a human being so whenever you're ready now you can slowly slowly open your eyes still breathing in through your heart it's a special moment it's a special moment when we're busy sinking running about trying to solve these big questions on AI and all these issues give ourselves that time and luxury to just sit still and be with who we are to realize our call and what we're about

and not to forget that if there's anything that doesn't feel right just in the same way that Professor crepe Clayton Christensen felt to say something hey this is not going in the direction that we want this is not the AI that we want or this is the AI that we want and this is the technology we want to embrace last but not least I want to say a story you've all watched Dorothy from The Wizard of Oz right a great pastime right and in many ways this journey is like Dorothy on The Wizard of Oz on the yellow brick road we have no idea how we're going to reach our destination what we do know it's going to be challenging it's going to be tough and we're going to lose our way and on the journey of finding home and what that means is we're going to meet some interesting people for Dorothy she met the Scarecrow who was looking for a brain on the journey she met the Tin Man who was looking for

a heart and also finally she met the lion who is looking for courage and as you all know all the characters find The Wizard of Oz and they all get to the destination that they want to get to but my message is this everything in Wizard of Oz finding home a sense of belonging having the brains the heart and the courage is actually all within us we all have it we don't need to look outside it's already there but it's a question of can we appreciate it can we realize it and can we share it so I am delighted to be here and to talk to you all and one of my lasting messages is when you look around this room there is all the talent in the world there is all the emotional intelligence and well-being here that we work with each other in making this goal happen at the very beginning of the film I showed you a video of the Moon Village Association which is something I am part of and there

is a really true vision that our generation that we can actually go to the moon it is no longer a sky and the pie idea is no longer a dream it's a reality we want to achieve if it's something that interests you and your wants to take part in it by all means get in touch look on the internet and I would love to engage with you all thank you very much

Rise of AI 2027 · Berlin · May 2027

Be in the room for the next one.

Ten years of talks started as conversations in one curated room in Berlin. The next edition brings 300 selected leaders together in May 2027.