At Rise of AI 2023, Rana Gujral, CEO of Behavioral Signals, argues that building systems closer to AGI requires modeling emotion and behavior, not just logic, since human decisions draw on both. He invokes Marvin Minsky's observation that the real question is not whether machines can have emotions but whether they can be intelligent without them, and argues a highly capable but emotionally blind system would function like a psychopath. Behavioral Signals measures emotion and behavior purely from non-verbal vocal cues, tone, pitch and prosody rather than words, producing classifiers for emotions, behaviors such as engagement or politeness, and industry-specific states like duress or coercion. Gujral describes a 'conversational bioprint' built from one to two minutes of audio and a case study at a European bank's debt-collection call center where matching agents to callers by bioprint improved loan-restructuring outcomes by 20 percent. He also describes a model predicting, with 80 to 83 percent accuracy, whether a debt holder will pay from the first 20 seconds of a call, while acknowledging sarcasm and data bias remain unresolved challenges.
Key takeaways
- Gujral argues systems approaching AGI need emotional and behavioral awareness, framing a purely logical but emotionally blind intelligence as functionally like a psychopath.
- Behavioral Signals classifies emotion and behavior purely from non-verbal vocal cues such as pitch and prosody, not from words.
- A 'conversational bioprint' built from one to two minutes of audio was used to match debt-collection agents to callers, improving loan-restructuring outcomes by 20 percent at a European bank.
- A model predicting whether a debt holder will pay from just the first 20 seconds of a call reached 80 to 83 percent accuracy in testing.
- Sarcasm detection remains largely unsolved, and Gujral says any emotion model scoring below roughly 70 to 75 percent F-score is too unreliable for use.
Detailed notes
Why intelligent machines may need emotions
Gujral argues that human decision-making relies on more than logic, drawing on emotions and behavioral cues from the face, words and tone, so building systems that approach AGI requires modeling that process too. He cites a quote he attributes to Marvin Minsky: the real question is not whether machines can have emotions, but whether they can be intelligent without any. Gujral frames a highly intelligent but emotionally unaware system as functionally similar to a human psychopath, and argues that is not an entity people should want to depend on.
Measuring emotion and behavior purely from tone of voice
Behavioral Signals' approach focuses specifically on non-verbal cues in speech, pitch, tonal variance, intonation and prosody, rather than the words spoken, to quantify emotional and behavioral states using neural networks trained on domain-specific data. Its classifiers cover basic emotions such as anger, happiness and sadness, behaviors such as engagement, empathy and politeness, and more advanced, industry-specific states such as customer satisfaction, stress or duress, and signs a speaker is being coerced. Gujral shows a demo of the engine embedded in the Temi companionship robot, used to remind an elderly user to take medication, and situates the company relative to competitors and academic groups focused on facial-expression analysis, such as Affectiva, and university labs at MIT and Stanford.
The conversational bioprint and a debt-collection case study
Gujral describes a 'conversational bioprint,' a vector of roughly 75 to 200 attributes built from as little as one to two minutes of audio that uniquely characterizes how a person converses, similar to a fingerprint. Applied in a call center, it can be used to match callers with the agent most likely to build rapport with them. In a case study at a large, unnamed European bank's debt-collection center, matching agents to clients this way improved loan-restructuring discussion outcomes by 20 percent, which Gujral translates into roughly 1.5 million dollars per agent per year, and about 300 million dollars in total upside across the center's approximately 158 agents.
Beyond Alexa: predicting intent
Gujral contrasts this with today's voice assistants, noting that a sarcastic reply to Alexa is not understood as sarcasm because assistants like it can process language but not the speaker's state of mind, so no real dialogue develops. He shows a demo of a more emotionally aware assistant conversing naturally about research data, and describes capabilities his engine can extract purely from tone in real time: diarization of who is speaking when, gender, and age bracket. He also describes a model built to predict whether a debt holder will pay or not pay, as a binary outcome, from just the first 20 seconds of a call, which tested at 80 to 83 percent accuracy against 1,000 past calls with known outcomes.
Limits: data, bias and sarcasm
Gujral is clear that understanding affect is not a solved problem. Sarcasm, he says, remains largely unsolved and is the hardest case because even humans frequently miss it; he considers any model scoring below roughly 70 to 75 percent F-score too unreliable to use. He identifies the biggest constraint on building new domain-specific models as access to high-quality data, including making sure the data is unbiased and that annotation and labeling meet a high standard.
Q&A highlights
Asked whether the system can detect emotion or sarcasm without knowledge of cultural background, given how differently sarcasm lands even within one country, Gujral says sarcasm-detection models remain poor performers and unsolved. For core emotions, however, he explains the system measures where a vocal signal falls relative to an established baseline rather than applying fixed acoustic rules; that baseline shifts with language, culture, dialect and accent, but once recalibrated, now achievable with about 30 minutes of audio in a new language or domain versus a far larger dataset previously, the same point of inflection from the baseline consistently signals the same emotion. He compares this to how a human brain gradually recalibrates when moving to a new linguistic environment, except the model can do it within minutes.
Quotes
I don't think we really want to create a future where we have machine psychopaths that we depend on.
Rana Gujral
You need to really understand the state of mind, the emotional behavioral state, to actually have a meaningful interaction.
Rana Gujral
Questions answered in this talk
How does Behavioral Signals detect emotion from voice?
According to Rana Gujral at Rise of AI 2023, Behavioral Signals analyzes only non-verbal vocal cues, such as pitch, tone and prosody, not the words spoken, using neural networks trained on domain-specific data to classify emotions, behaviors and industry-specific states like duress or coercion.
What is a conversational bioprint?
According to Rana Gujral at Rise of AI 2023, a conversational bioprint is a vector of roughly 75 to 200 attributes derived from one to two minutes of audio that uniquely characterizes how a person converses, used for example to match call-center agents to callers.
Can emotion AI detect sarcasm across cultures?
According to Rana Gujral at Rise of AI 2023, sarcasm detection remains largely unsolved, but detecting core emotions works by measuring where a vocal signal falls relative to a recalibratable baseline, which can now be adjusted to a new language or culture with about 30 minutes of audio.