Rise of AI Conference 2019 · Berlin

Virtual Human Agent for Smart City

Atsushi Ishii · CEO at Couger

At Rise of AI 2019 in Berlin, Atsushi Ishii, CEO of the Tokyo-based startup Couger, presents Connectome, the technology behind Couger's Virtual Human Agent, a character with simulated senses, emotion and intelligence meant to become a new, more natural interface than keyboards, touchscreens or voice assistants. He grounds the pitch in four areas of prior work: character AI for games, a learning simulator that Honda uses for autonomous cars and robots, cloud robotics developed with a team from the Amazon Robotics Challenge, and enterprise blockchain for secure data sharing. He plays demo videos of the Virtual Human Agent, named Rachel, operating in augmented reality on a tablet and answering questions live in a German television interview, and explains how it detects and mirrors human facial expressions. Ishii describes an architecture that combines machine learning, which predicts a single best answer from data, with gaming AI, which builds a believable world, plus components for personality, semantic and episodic memory, and emotion. He closes by outlining planned use cases, including gaming, interactive digital signage and in-car assistants, and an upcoming SDK and marketplace for outside developers to build their own virtual human agents.

Key takeaways

  • Couger's Virtual Human Agent, built on its Connectome technology, combines character AI, a training simulator, cloud robotics and enterprise blockchain.
  • Ishii says Honda already uses Couger's learning simulator to generate training situations for autonomous cars and robots.
  • US university research cited by Ishii found patients respond more and for longer to interfaces with a human-like face and body than to voice or text alone.
  • The architecture combines machine learning, which predicts single answers, with gaming AI methods, which are used to make a believable world.
  • Couger plans to release a Virtual Human SDK and marketplace so outside developers can build and sell their own virtual human agents.

Detailed notes

Couger's Background: Four Underlying Technologies

Ishii introduces himself as CEO of Couger, a Tokyo-based startup and creator of Connectome, with a background at IBM and at a company he describes as comparable to Amazon in Japan, and current work leading a national government AI project. He grounds the Virtual Human Agent in four prior technology areas: character AI, where a game character's visuals, voice and behavior act like a human; a learning simulator that generates situations such as an airport, countryside, school or shopping mall so robots and AI can be trained, which Honda already uses for autonomous cars and robots; cloud robotics, where one robot's results are shared to the cloud and reused by other robots, developed with technical cooperation from a team in the Amazon Robotics Challenge; and enterprise blockchain, which Couger is testing to let companies share data securely.

Why a Human-Like Interface, and the Vision of Trustworthy AI

Ishii argues that user interfaces have moved from PC to smartphone to smart speaker, and that the next step is a more natural interface modeled on how humans communicate, even with pets or babies who cannot fully understand language. He frames Couger's goal as building AI people can trust, contrasting it with what he calls black-boxed AI built only by large companies where users do not know the underlying data. The Virtual Human Agent has presented at Stanford University and appeared on NHK, Japan's national public broadcaster.

Demo: Rachel in AR and on German Television

Ishii shows demo footage of the Virtual Human Agent, named Rachel, running in an augmented reality tablet app that detects objects and people in a co-working space and greets them by name, and on a monitor during a live interview with a German TV crew, where Rachel answers questions about where she is from, her hobbies and what she thinks about AI. He cites US university research finding that patients respond more easily and for longer to interfaces with a human-like face and body than to voice-only or text interfaces, and demonstrates the system detecting and mirroring human facial expressions such as surprise or happiness.

Combining Machine Learning and Gaming AI

Ishii explains that standard machine learning predicts a single answer from data, useful for classification, while gaming AI is built to make a world feel real to players. Couger's architecture combines both: machine learning handles input and character behavior, while gaming methods and original research into human cognition drive an internal architecture with input senses (vision, sound, touch, IoT sensors), an unconscious layer, personality, semantic memory (recognizing what an object is), episodic memory (recalling a specific past interaction), emotion, and a decision-making and behavior layer that produces the output: facial expression, body movement, speech or control of other IoT devices.

Use Cases and the Planned SDK Marketplace

Ishii lists use cases already in proof-of-concept with companies: gaming and entertainment, where users can talk to and get more complex situational responses than in games like Pokemon Go; interactive digital signage that can answer questions and suggest information, unlike ordinary one-way signage; and virtual human assistants inside cars, as the car interior becomes more of a private space with autonomous driving. Couger plans to release a Virtual Human SDK so outside developers, designers and AI module creators can build and upload their own virtual human agents and AI components to a future marketplace, and says it has already run meetups, workshops and hackathons with developer communities in the US, Africa, Europe and Asia.

Questions answered in this talk

What is a Virtual Human Agent?

According to Atsushi Ishii of Couger at Rise of AI 2019, a Virtual Human Agent is a character-based AI interface with simulated senses, emotion and intelligence, built on Couger's Connectome technology, intended to replace keyboard, touchscreen and voice interfaces with a more natural, human-like one.

How does Couger combine machine learning and gaming AI in its Virtual Human Agent?

According to Atsushi Ishii at Rise of AI 2019, machine learning handles input and predicts character behavior from data, while gaming AI methods and original human-cognition research build personality, memory and emotion layers that make the world and the agent feel believable to users.

What use cases is Couger testing for its Virtual Human Agent?

According to Atsushi Ishii at Rise of AI 2019, Couger has proofs of concept in gaming and entertainment, interactive digital signage, and in-car virtual assistants, and plans to release an SDK and marketplace for outside developers to build their own agents.

Mentioned

Connectome, Virtual Human Agent, cloud robotics, enterprise blockchain, character AI, semantic memory, episodic memory, SDK marketplace

Keep exploring

More from Rise of AI Conference 2019

More on Enterprise AI & Adoption

Transcript

Auto-generated transcript, not edited. · 2,243 words

Show the full transcript

hi everyone today I'd like to introduce about our technology it's called connect Tom and how it can create new generation interface it's called badger human agent okay let me talk about me I'm at shishi I'm CEO of cougar cougar is tokyo-based startup and also a creator of connect Tom connectome is technology name and my background is I was dropping Raj scales are changing in IBM and also in Roxanne black thing is like Amazon in Japan and also I am leading national government project AI project now and by the way my any sure is AI yeah I explained about AI and I'm AI yeah okay it's our background first one is humor like decision-making AI there is character I and in the project the comma W is like the characters I and sound is the characters auditory and the character is a character act like like human automatically it's very similar to our budget human agent I will explain second why is a learning simulator as you know robot or a I need to run any

kind of situation if they they didn't know the situation it's very difficult for AI so our simulator can create any kind of situation such as airport or countryside or school or shopping mall anything and Honda is already using it for autonomous car or robots and sad one is cloud robotics cloud robotics means if robot a grasp something unveiled the result is going to the crowd and the robot be neighbor fader game it's cloud robotics and there is a top team of Amazon robotics challenges and we provide technical cooperation to the team and so we are very strong for cloud robotics things and last one is it is for using the blockchain as you know now blockchain usage is for the currency right but we are looking for the next usage for the real world if we can we can share data safely between several enterprise it it will be very useful for blockchain so we are testing blockchain on the real world using the enterprise broke chain and it is media achievements and we are

on yeah many kind of media's especially we we had lecture on the Stanford University about our budget human agent it is very unique product and I explained about technology in Stanford University and also zhk is one of the biggest national public broadcast TV and I I'm on the TV and explain about my budget human agent and it is a vision of connectome and it is towards world where human being can trust AI I think this this time AI is Brock boxed and only big player can create so III high accuracy AI but nobody know the data or a is a I trust what not so we want to create trust AI the butter human agent is like a humor like AI agent humor like AI assistant right pradana so why now as you know user interface is changing from PC to smartphone and small speaker and what is the next and we we think the interface is changing to more friendly and naturally right yeah if you use the PC you have to use quick

and keyboard and if you use a smartphone you can just touch and the small speaker you can talk and so it's like I think the best best interface for for human is I think the human because if you talk to the you a pet you just just communicate right right human and of course if you talk to the baby it's it's like if if you you already know baby can't understand you're talking but you try to talk to our like a human so that's a point and this is budget human agent and but your human agent what's on the monitor and at the same time in the AR space AR AR yeah like Pokemon go or something like that and also on tablet and it is on the car you know car will be changed in next two or three years in so Thomas autonomous means Cosby is changing to your private space so budget human agent is expected in the area and let me show you a demo video and it is on the

AR space so please look and it is a tablet good morning Rachel there is a bunch of you know yes uncie know the time and the rectangle is budget human detects the objects yes yes suitcase and detects a suitcase or chairs or something like that yes Rachel come over here yes come over here yes ah thank you let's go to office it works for us Tibet but already tested grass already thank you she's such a drink there is about your human it is find some people in the space co-working space yeah come over here Richard come over here yes and the next one is it works on the monitor it's a little bit different and yeah this movie is actually it's a German TV program is coming to our office and C is from from Maybelline and the German key program asking to the budget humor about ha yeah maybe yeah about her yeah so president hello had sushi alright see recognize me looks like we have a visitor yes nice to meet you I

am Rachel hi Rachel where are you from I'm from Germany you came from far away that's true have you been to Germany no I have not but I will be going there in May if you have any questions about me please ask how old are you that is a secret where are you from I'm from Shibuya Japan what do you do Rachel I am working at couger as assistant I can provide some assistance for you what's your hobby I have started to learn dancing recently Oh can you dance for me okay I will try thank you thank you you are welcome what do you think about a I am I am not ready to answer the question yet I hope you can see it through my growth okay what's your dream I am still in the process to understand the concept of dream I will tell you when I understood it yes this is about your human agent and as Adam video but a human has face and body why so the point is it

is a potential humor like interface and this is yeah some research in the u.s. University just one first one is there are all statistic patients talk to about your humor right interface and the answer is very easy to talk with the face and this one is also in the u.s. research and it is compared to the patch of humor like interface and the only boys interface and that yeah result is response rate and this one's time is very increased with humor likely interface because yes it's more like a humor like a human like with body and face is very important this is at the point emotion expression and this is the budget human can detect some kind of human facial expression surprised happy or neutral and also budget human can express the emotion and it is the input and output is connected so let me show you the damn video and if he is smiting budget human also smile hiii-yah angries yeah but a human also copy it so this is a different point of

our approach this is a typical machine learning AI and as you know if you use machine learning the answer is just one I means if she is chair cars car tables table so if you have big data it's very very easy to predict answer this is a comprehension and specific purpose AI this area is character I and gaming AI this comprehension a is purpose is to predict the answer Barry for called predicts for the answer using the big data and this part is the purpose is how to enjoy the world I you have to have to create word for the users so if you play the game you can you can feel the worthies exist and you can feel it the enemies exist and your teammate is exist so that's why you can enjoy the world so this is very different different so our technology is combining these two kind of different they I machine learning things and the gaming AI so the real input is using machine learning and it's a character's behavior and

making this joint is for Delta AI and this is virtual human architecture overview this is this area is for the input yeah this is like humans humans architecture of course your vision input and sound touch the IOT sensors and this is the output output means humans action humans action is facial expression body expression and the moving talking and also IOT devices any other such AI devices as well and this is yeah big overview this party is input and the architecture is has the emotion part and personality personality means there are load of budget you know this this one is unconscious part unconscious means if you if you see something and the human human is making this young huh which which is which you you focus on focus on and memory is haha words and making this show and behavior 3 is working and yeah let me explain the some yeah sample if you if you see the cut just cut after that the semantic memory is working semantic memory means what is cut and cap

copies the animal and pet or something like that and after that episodic memory is working episodic memory means your memory please cut and if you have good memory with cut the emotion is working the emotion pleasure or happy is working at the same time the budget humor reminds you our role you alone is if but a human's role is officer working a for the receptionist and in that time you are your emotion is happy and very pressure but at the same time but your human control the emotion and after that making decision and the behavior three means how how to how the variation of the action the after that the action is action will be done and this lived one we are using some machine running and this green one we are using the gaming method and gaming algorithm and the light green one we are using original human science and this is very yeah unique technologies and this is use case for batch of human agent and we already running for the POC with

some companies and first one is game and entertainment it is like Pokemon go and Pokemon go is just Pokemon is you can find the Pokemon in the real world but but your human is you can talk to the body human and budget Hindman detects situation more complex and second why is better signage like badger human signage and we already testing this one the different thing is normal digital signage is of course no interaction but battle human data signage you can ask to the our body human and but you humans suggest some new information like them video and third one is virtual human in the car as I mentioned the car industries will be changed for the auto mask our auto mask a means yeah us space is changing to your private moving private room so virtual human is expected in the next interface in the car like a bunch of human basket or something like that and also we are now we are not planning to provide connect on marketplace what is market price as

as I as a damn video we are developing such a human agent right but we already know we need more yeah we we need more kind of much kind of developer and creator or over the world so we will provide the virtual human SDK if we if creator can use SDK you can create any any kind of battle film original budget human agent and write that developer can create your original budget human and also designer can create your creative vision like what rights are your favorite budget human look like and the AI developer also can create AI modules for example the motion detection or natural language or computer vision yeah any module I can upload on the marketplace so so according to this marketplace creator can create your original body human agent and also we are now syncing this week kind of use case but or of course the use case will be created more using you yeah creators because creator can create the original budget human isn't that's a point and we are already

expanded to the development community in the US and Africa and Europe and the Asia and now we already host it yeah the meetups and technical workshop and the hackathon yes and now we we are thinking that we will we will have some virtual human Hakkasan or butter humour workshop and also yeah we will provide the SDK and next year we will open the marketplace and yes and this is our my region yes that's my presentation thank you very much

Rise of AI 2027 · Berlin · May 2027

Be in the room for the next one.

Ten years of talks started as conversations in one curated room in Berlin. The next edition brings 300 selected leaders together in May 2027.