At Rise of AI 2019 in Berlin, Atsushi Ishii, CEO of the Tokyo-based startup Couger, presents Connectome, the technology behind Couger's Virtual Human Agent, a character with simulated senses, emotion and intelligence meant to become a new, more natural interface than keyboards, touchscreens or voice assistants. He grounds the pitch in four areas of prior work: character AI for games, a learning simulator that Honda uses for autonomous cars and robots, cloud robotics developed with a team from the Amazon Robotics Challenge, and enterprise blockchain for secure data sharing. He plays demo videos of the Virtual Human Agent, named Rachel, operating in augmented reality on a tablet and answering questions live in a German television interview, and explains how it detects and mirrors human facial expressions. Ishii describes an architecture that combines machine learning, which predicts a single best answer from data, with gaming AI, which builds a believable world, plus components for personality, semantic and episodic memory, and emotion. He closes by outlining planned use cases, including gaming, interactive digital signage and in-car assistants, and an upcoming SDK and marketplace for outside developers to build their own virtual human agents.
Key takeaways
- Couger's Virtual Human Agent, built on its Connectome technology, combines character AI, a training simulator, cloud robotics and enterprise blockchain.
- Ishii says Honda already uses Couger's learning simulator to generate training situations for autonomous cars and robots.
- US university research cited by Ishii found patients respond more and for longer to interfaces with a human-like face and body than to voice or text alone.
- The architecture combines machine learning, which predicts single answers, with gaming AI methods, which are used to make a believable world.
- Couger plans to release a Virtual Human SDK and marketplace so outside developers can build and sell their own virtual human agents.
Detailed notes
Couger's Background: Four Underlying Technologies
Ishii introduces himself as CEO of Couger, a Tokyo-based startup and creator of Connectome, with a background at IBM and at a company he describes as comparable to Amazon in Japan, and current work leading a national government AI project. He grounds the Virtual Human Agent in four prior technology areas: character AI, where a game character's visuals, voice and behavior act like a human; a learning simulator that generates situations such as an airport, countryside, school or shopping mall so robots and AI can be trained, which Honda already uses for autonomous cars and robots; cloud robotics, where one robot's results are shared to the cloud and reused by other robots, developed with technical cooperation from a team in the Amazon Robotics Challenge; and enterprise blockchain, which Couger is testing to let companies share data securely.
Why a Human-Like Interface, and the Vision of Trustworthy AI
Ishii argues that user interfaces have moved from PC to smartphone to smart speaker, and that the next step is a more natural interface modeled on how humans communicate, even with pets or babies who cannot fully understand language. He frames Couger's goal as building AI people can trust, contrasting it with what he calls black-boxed AI built only by large companies where users do not know the underlying data. The Virtual Human Agent has presented at Stanford University and appeared on NHK, Japan's national public broadcaster.
Demo: Rachel in AR and on German Television
Ishii shows demo footage of the Virtual Human Agent, named Rachel, running in an augmented reality tablet app that detects objects and people in a co-working space and greets them by name, and on a monitor during a live interview with a German TV crew, where Rachel answers questions about where she is from, her hobbies and what she thinks about AI. He cites US university research finding that patients respond more easily and for longer to interfaces with a human-like face and body than to voice-only or text interfaces, and demonstrates the system detecting and mirroring human facial expressions such as surprise or happiness.
Combining Machine Learning and Gaming AI
Ishii explains that standard machine learning predicts a single answer from data, useful for classification, while gaming AI is built to make a world feel real to players. Couger's architecture combines both: machine learning handles input and character behavior, while gaming methods and original research into human cognition drive an internal architecture with input senses (vision, sound, touch, IoT sensors), an unconscious layer, personality, semantic memory (recognizing what an object is), episodic memory (recalling a specific past interaction), emotion, and a decision-making and behavior layer that produces the output: facial expression, body movement, speech or control of other IoT devices.
Use Cases and the Planned SDK Marketplace
Ishii lists use cases already in proof-of-concept with companies: gaming and entertainment, where users can talk to and get more complex situational responses than in games like Pokemon Go; interactive digital signage that can answer questions and suggest information, unlike ordinary one-way signage; and virtual human assistants inside cars, as the car interior becomes more of a private space with autonomous driving. Couger plans to release a Virtual Human SDK so outside developers, designers and AI module creators can build and upload their own virtual human agents and AI components to a future marketplace, and says it has already run meetups, workshops and hackathons with developer communities in the US, Africa, Europe and Asia.
Questions answered in this talk
What is a Virtual Human Agent?
According to Atsushi Ishii of Couger at Rise of AI 2019, a Virtual Human Agent is a character-based AI interface with simulated senses, emotion and intelligence, built on Couger's Connectome technology, intended to replace keyboard, touchscreen and voice interfaces with a more natural, human-like one.
How does Couger combine machine learning and gaming AI in its Virtual Human Agent?
According to Atsushi Ishii at Rise of AI 2019, machine learning handles input and predicts character behavior from data, while gaming AI methods and original human-cognition research build personality, memory and emotion layers that make the world and the agent feel believable to users.
What use cases is Couger testing for its Virtual Human Agent?
According to Atsushi Ishii at Rise of AI 2019, Couger has proofs of concept in gaming and entertainment, interactive digital signage, and in-car virtual assistants, and plans to release an SDK and marketplace for outside developers to build their own agents.