At Rise of AI 2018 in Berlin, Dr. Catherine Havasi, Chief Strategy Officer and co-founder of Luminoso, explains how adding common sense knowledge to machine learning systems helps them generalize from far less data. She argues deep learning's reliance on huge datasets, such as AlphaGo needing about 68 million self-play games without transferring anything it learned to chess, is a core limitation, especially where data cannot be simulated, as with text or robotic picking. She describes how humans generalize from very few examples by using base knowledge and analogy, and presents common sense reasoning as the addition of a general knowledge base to a machine learning system to make it more flexible. She cites research projects such as ShapeNet at Princeton and Stanford, and a game-learning project at City University London, that improved performance with less training data by infusing general world knowledge. She then describes her own project, ConceptNet, a crowdsourced multilingual knowledge graph running since the late 1990s, and how her company Luminoso uses it with transfer learning to help enterprises, including the gaming company Supercell, automate customer service with far smaller amounts of domain data.
Key takeaways
- Havasi argues AI's reliance on huge datasets, such as AlphaGo's roughly 68 million self-play games, is a core reason it fails to generalize across tasks.
- She frames common sense reasoning as adding a general knowledge base into a machine learning system rather than a separate type of machine learning.
- Research projects like ShapeNet and a City University London game-learning study cut required training data by infusing general world knowledge.
- Her project ConceptNet, running since the late 1990s, was the first to use crowdsourcing to build an open multilingual knowledge graph.
- Her company Luminoso combines ConceptNet with transfer learning to move models into enterprise domains with only a few thousand data points, illustrated by a Supercell customer-service case study.
Detailed notes
Why deep learning struggles to generalize
Havasi frames the talk around well-known problems with AI: it needs huge amounts of data, it behaves as a black box, and it does not generalize or transfer knowledge across tasks. She notes deep learning systems for language typically rely on word embeddings, a technique that became popular with Google's word2vec in 2013 alongside tools like Stanford's GloVe and fastText, and that training such systems takes hundreds of millions or billions of data points. Where data is scarce, teams fall back on the Common Crawl web corpus, which knows little about specialized domains such as pharmacovigilance.
The limits of manufactured data
She points to AlphaGo, which had to play roughly 68 million games against itself to reach human-level Go, as an example of a system that cannot take what it learned and apply it to chess. She argues text data cannot be simulated the way game data can, since a system able to simulate language convincingly would not need the simulation in the first place, and that manufactured training data tends to introduce artificial 'tells' that break down in the real world. She gives robotic picking, the problem of a warehouse robot picking an item like a can of soup and boxing it, as a second example: companies running robotic arms over piles of cans repeatedly still cannot generate enough data to solve the problem generally.
How people generalize, and what common sense reasoning adds
Havasi contrasts this with how people learn: children generalize from a very small number of examples, for instance recognizing a giraffe as a hoofed animal from a single sighting by reasoning about its features, and people understand complex concepts, such as data, through analogies to simpler ones, like the common analogy between data and water (flow, streams). She defines common sense reasoning as adding a base knowledge graph about the world into a machine learning system so it can generalize better, describing it not as a distinct type of machine learning but as an addition to it. She notes Microsoft co-founder Paul Allen funded the Allen Institute in Seattle specifically to research common sense, believing it could unlock the next stage of AI progress.
Research examples: robotics, games and sentiment analysis
She cites ShapeNet, a project from researchers at Princeton and Stanford that built general models of small objects such as chairs, planes and cans and infused them into a robotic picking system, which then needed much less training data to work well. At City University London, researchers taught a system general ideas about games before a specific game, cutting the number of simulation sessions needed to learn a new game by five times. She also mentions Cambridge University research showing that adding information about opposites and similarities improved a simple neural network's ability to do sentiment analysis.
ConceptNet, transfer learning and the Supercell case study
Havasi describes running the Common Sense Computing Initiative since the late 1990s, which she says was the first project to use crowdsourcing to build an open knowledge graph, now available as ConceptNet, working across as many languages as Wikipedia. Her company, spun out in 2011 and now called Luminoso, combines ConceptNet with transfer learning to help enterprises automate customer service with far smaller domain datasets, a few thousand data points rather than the hundreds of millions typical deep learning needs. On last year's ACL benchmark, she says word embeddings enriched with ConceptNet outperformed plain word embeddings on word similarity tasks, including across languages. As a case study she describes working with the mobile gaming company Supercell, maker of Clash of Clans, to route and tag customer support tickets that use invented gaming-specific vocabulary, for example the general web-design terms 'hamburger' (a menu icon) and 'carousel' (a swipeable panel), and to surface emerging bugs the support team has no existing workflow for.
Quotes
You can't simulate text data, because if you had a simulation for text data, you wouldn't need a simulation for text data.
Catherine Havasi
Taking knowledge that we know about the world in general and then using that base knowledge to be able to solve more complicated tasks is a type of machine learning called common sense reasoning.
Catherine Havasi
Questions answered in this talk
Why don't deep learning systems generalize well from one task to another?
According to Catherine Havasi at Rise of AI 2018, systems like AlphaGo need enormous manufactured datasets, roughly 68 million self-play games in AlphaGo's case, but nothing they learn transfers to a different task such as chess, because they lack the base world knowledge humans use to generalize.
What is common sense reasoning in machine learning?
Catherine Havasi describes it at Rise of AI 2018 as adding a general knowledge base into a machine learning system so it can generalize from far fewer examples, the way people reason about a new animal or concept using knowledge they already have.
What is ConceptNet and how is it used commercially?
According to Catherine Havasi at Rise of AI 2018, ConceptNet is a crowdsourced, multilingual open knowledge graph her Common Sense Computing Initiative has built since the late 1990s, which her company combines with transfer learning to help enterprises, such as the gaming company Supercell, automate customer service with much less domain-specific data.