foreign and I actually met Rana last night and it's one of the most interesting and insightful startups that I've had or should we say scale UPS really I've had the pleasure of meeting recently and it's a topic that really does blow your mind it's all about behavioral signals and so please welcome Rana CEO of Behavioral signals to the stage to talk about his company and the applications of Behavioral signals thank you thank you all right so let's talk a little bit about a very interesting application of deep learning which is getting inside the thought process the brain of a human especially around understanding emotions behaviors intent and essentially the state of mind which goes a long way in building an AGI which is like a human so we largely use AI for automation for the most part but we are starting to use AI towards making certain decisions for us but when we are trying to do that we're also trying to understand well how does a human brain actually make a decision a human
brain uses terabytes of data towards a single insight and then it uses many insights to eventually arrive at a decision that's not necessarily you know linear linear in terms of computation to sort of how you're sort of building some of the systems today it's very sort of disjointed it's a part of it is sort of not easily understood and so when we're looking at building these systems we have to sort of like you know deconstruct that whole sort of thought process and especially if we are going to now allow AI not to help us make decisions but actually make those decisions for us which is like you know just make it let us know what we need to do and how do we eventually get there and it becomes really key so when when you think about a human I mean humans don't necessarily just rely on logic of course you need logic to make a decision but it it takes a lot more than just simply logic to get to a human decision point
a human typically would take into account a whole bunch of other things such as emotions behaviors you know from the face from words from tone uh among other things and emotions is also a very complicated aspect because you know affect which is a technical term of emotions multimodal it's expressed in different ways and so then when we're thinking about building these uh entities now we have to think about well how do we build these algorithms that are similar to The Human Experience and I get this asked this question a lot which is like okay well I mean but should you be doing that I mean should you be building a capability that allows it's because it's very human thing um a machine to actually have emotions and to understand behaviors um you know is that is that first possible second is that right and um one of the codes you know from Marvin Minsky comes to mind I mean he was one of the founding fathers of AI as some of you might already know
and when he was asked um Can machines have emotions or can intelligent machines have emotions and his answer was the question is not whether intelligent machines can have any emotions but whether machines can be intelligent without any emotions and I think that's a very profound quote um because another way to sort of think about this is that we can rely on two things happening one is that machines are going to get more and more intelligent and in many ways get to a point where they're going to be an order of magnitude more intelligent than machine than humans and second we're going to be relying on these machines quite a bit so then imagine a future where there is this entity that is super intelligent many times more intelligent than a human and we're very very dependent on it and would you like that entity to be emotionally aware um or not because in a human context a human that is very intelligent but has lacks the ability to understand or process emotions is really the
technical definition of that is a psychopath right so I don't think we really want to create a future where we have machine Psychopaths that we depend on so then how do we measure emotions anyways um and one would argue that you know um probably it's not as concrete as say measuring say a blood sugar level in a human but you know we as humans are pretty good at knowing when someone's mad at us uh that's you know pretty easy to do no matter how they actually uh whatever they say that they feel about us we kind of know and so it's this incredible ability that we're all born with which is to understand the subtle nuances in words in tone in body language and facial expressions um that that's something that we just have naturally and we take all of that account to sort of understand how someone's feeling in the moment um humans um you know also sort of uh use as I said emotions and behaviors uh as an essential element of making
a decision it's an essential part of decision making process and we can teach machines to do the same and to do this we first have to sort of build these algorithms that have relevant data uh pertaining to the specific domain that we're trying to apply that engine to that AI space into and it first starts with quantifying and measuring various emotions I mean you take varied speed signals or other modalities and you use neural Nets which is you know what we do to sort of quantify various emotional states um it sometimes takes thousands of utterances to understand a single state of emotion or behavior and then of course behavior is even more complicated because it's context specific um you know why someone's acting in a certain way so you have to build very very complex custom models towards sort of doing that but when you do it you can now build these very enriched experiences that can really change how we live in the future so here's a implementation of Babel signals engine into one
of the partner machines which is temi robotics it's actually a social companion that is really sort of geared towards a single purpose of providing companionship and helping people who are living by themselves especially elderly I think we're missing the audio John it's time for your medication could you please retrieve your pills from the dispenser John please take your pills you know what why don't you uh go somewhere else where you need it like maybe in the kitchen okay I will go to the kitchen good yeah so um some of us have been working on this problem quite uh for quite some time um there are many companies that are focused on various specific modalities of how do you understand emotions or behaviors company called affectiva um that we partnered with they recently got acquired from media Labs they've been focused on facial expressions there are entire Departments of universities in the US that are focused on emotional behavioral State like sale like MIT Stanford Etc our specific Focus as a company is to understand emotions
and behaviors from non-verbal cues of a conversation which is essentially how something is being said not really the words not what you're saying but how you're saying those words which is pitch and tonal variants intonations porosity among other things and from that we can extract a whole wide variety of intelligence such as emotions like anger happiness sadness behaviors such as engagement empathy politeness and then you know the third bucket is an interesting collection of advanced classifiers that are you know more sort of Industry specific like um you know understanding a customer's experience uh or level of satisfaction live in the moment um you're trying to understand if someone's stressed or in some sort of a distress or duress um identifying someone uh potentially uh in someone is in some sort of a control which is you know you're saying something but do you really mean it or someone's making you say something Etc um or an identifying intent or intent to do something or intend to not do something and so those are sort of
you know all those things that are now possible to do purely from assessing the emotional and the behavioral state of a participant one of the implementations is actually looking at a recent interaction audio interaction and then creating what you would call a conversational bioprint this conversational bioprint essentially is an amalgam of a whole wide variety of factors that range from sort of like how fast you speak to various emotional behavioral states that you exude in a conversation and then when you put it together it really uniquely identifies how a person Converses so once you have that information then you can use that to make decisions around who's the best conversational partner for this particular human who's calling in for this particular agenda you know those are the kind of things that you could apply in a call center and when you do that you have a significant lift in the kpis that's a that's a quick view of sort of like you know what that conversational bioprint looks like it's really a vector file of
75 200 attributes that that can be quickly put together from a recent um you know audio interaction and it doesn't have to be long it could be you know anywhere from a minute to two minutes in length and it's it's really uh unique to every person just like a fingerprint is how does it help like you know so one of the implementations uh but applied it in a call center uh for a debt collection scenario in a large EU Bank it happened to be the name has been withheld uh one of the largest uh Banks of that country um and in that situation when we applied it to uh you know the agents that were using this technology they were able to improve the loan restructuring discussions which was the primary agenda between agent and client by 20 so in Stark numbers it was almost a million and a half dollars per agent per year and so with about you know 158 odd agents that call center had it was a 300 million dollar upside
just by bringing the right people together uh which then you know uh creates that tremendous opportunity of building natural Rapport natural affinity and then you know everything falls from that so it is really that powerful if you kind of use it in a proper context so then let's talk about something else right so how's that promise of um the the fact that you know uh we were promised at some point that the virtual assistants are like saying Google homes uh will become a conversational partner how's that going not very well right I mean um I think we all realize that while they're useful to some extent I don't think we can actually have a conversation with these devices it's a much higher bar to actually have a conversation so for example let's say if Alexa asks you to do something and you respond back with a very sarcastic sure Alexa has no clue whether you actually mean yes or no by that word that you're saying uh choosing to use which is sure a human
would for the most part but Alexa won't and so there is sort of like you know the end of the conversation I mean you could ask it to do something but it not necessarily allows you to actually have a dialogue to have a dialogue to have a conversation you really need to do a lot more than say you know just understanding what someone's saying and speaking in the same language which is really the you know nlu and NLP which is really solved for for the most part but you need to understand the state of mind and you need to really understand the State of Mind the emotional behavioral State of Mind to actually have a meaningful interaction to actually say something that the other human can relate with this is what it looks like when you put that have they published results on cancer causality perceptions yet I'm afraid not what am I going to do now foreign maybe I can find results somewhere else I will search further Eureka the international Human Genome Project
has published some preliminary results on the number of study participants and their population diversity is that useful that is fantastic great find Elsa can you upload it would be great to add some numbers already uploaded in your files yeah as you can imagine it's a completely different experience than what you guys have been having with your Alexa right and so but soon enough I mean you'd be having conversations like this with your car um you're definitely definitely the virtual assistants I mean these these entities that really uh not only understand what you're saying um your language your dialect your accents uh but also necessarily how you're saying it and the the emotional behavioral state in which you're expressing that whatever utterance that you're expressing and then using that into how it's interacting back with you and then Things become really really interesting I mean it's an essential building block of AGI um that uh that is an essential human component that you can bring into the AI experience but I think we could do a
lot more than that and this is where sort of like you know now we're sort of into uh pushing the boundaries a lot of what I've shown you are the things that we've successfully done implemented there are companies using it I mean the kpis are being impacted and uh you know companies are getting a lot of value out of it and so they're ready for commercialization but we're also now looking at other aspects of understanding the cognitive state which is identifying and understanding intent and also uh predicting a propensity for a certain action well how do you actually do that so predicting a propensity for a certain action is something also you know known by something uh some other word that you're all very familiar with which is gut a gut check right so sometimes you have an intuition you have a gut you know this is what's going to happen it's not going to happen or you're in a conversation and someone's saying you know I'm going to do this and you're like no
that's not going to happen he's not going to do it or she's not going to do it or um you know vice versa how does that really actually work out well you know where are What are the factors that sort of drive that human brain that Insight of whether someone's going to do something or not and that's really a combination of two things right the first is of course you need to um you know understand what is this conversation about so yeah it's a domain of the conversation you have to understand the nature of the interaction so if you're building an AI model then you have to build a very specialized custom domain towards a custom model for that particular domain so uh so that that is relevant to that interaction so it could be a conversation between say our doctor patients you need to understand the lingo or if it's a conversation between say a debt collector and a you know a debt holder then you also need to understand sort of what that
interaction looks like and build a custom domain so that's something you know we know very well how to do I mean there are that's how where machine learning comes to play you build a specific uh set of uh um you know requirements into a machine learning engine and then you give it the right amount of data and it learns from itself loans on its own but then the more complicated stuff is the second piece is understanding the state of mind and this is where um I think you know uh we had a sort of like a dead end for many years which is well how do you really understand the state of mind and so now using some of these tools um you know some of these different modalities but tone of voice is one of them that we use you can actually get inside uh sort of the context of the conversation which is like you know how how someone's saying something what is the emotional behavioral State and then when you put it
together you can actually predict intent so one of the models that we built um uh really largely for experimentation purposes for a company um they they were looking to see you know if we can predict if a client will buy or not buy or a debt Holder will pay or not pay and so we use some of these early models and they're still sort of you're testing the boundaries on that to make a binary prediction for a debt holder on whether they'll pay or not pay and so we make that prediction in the first 20 seconds of a call and it's a binary prediction which is yes we'll pay or no will not pay not percentage probability but yes or no and that prediction turned out to be 80 to 83 percent accurate uh which was pretty mind-boggling and I mean and it's actually pretty easy to test because all you really need to do is uh take thousand calls from the past you exactly know what the outcome is well don't tell the engine
the outcome you have an engine predict the outcome match it to exactly what happened and it was pretty modeling I mean needless to say we were very surprised and so certainly the company was very surprised and so now we're sort of pushing the boundaries towards uh doing other more complicated things such as understanding um you know the intent to fraud potentially very useful uh in banking Etc but also in law enforcement among other things you know assessing someone's trustworthiness and intend to do something or not do something why does this all even work right so I think that's an interesting question so I want to play the next video actually it's mostly an audio but there's some graphic in there and it's a test I'd encourage you guys to do on your own which is you know go home and find a piece of your favorite audio clip in a foreign language that you absolutely do not understand even a little bit right and uh so presumably preferably from a movie such a scene where
there are multiple actors something's happening there so try to see if you can figure out what's really going on how many speakers you know what is the emotional behavioral state of that's those speakers um can you can you understand maybe the subtle differences between someone's excited or happy or angry and uh you have no other cue other than the tone right so there's no video there's obviously no uh spoken words there's no closed caption and you're just trying to do it from the tone of voice and see how you do um and here's you know um a quick sort of uh audio clip where you know there are five clips from five audio clips from five different movies in five different languages and uh the engines processing it which is like a cheat here right now so that's why you can do this at your home and you could also see if you can sort of follow along okay cool Christmas tonight foreign laughs ladies and gentlemen some described I can hardly force myself to
keep looking at it so hopefully eyes are black and they Lean Like A serp in the mouth is that's kind of v-shapeless saliva dripping from its windless lips and see this those Quivers pulsating the monster or whatever it is can hardly move he's way down by possibly gravity or something of things I think that was the last one so I mean I'm sure most of you did pretty decent um but then again you know it really to do this test properly you don't want to have this sort of graphic in there and sort of really start to see if you can figure this out but we now have been able to sort of take this um you know innate human ability um that we're all born with and put it inside an AI engine that can you know allow a machine to do this as well and we could do this real time I mean the signal signals can be processed and uh extracted real time in the moment uh you could do real-time diarization
which is really understanding you know how many parties in a conversation who's speaking when um you know if multiple parties are speaking so you know speaker one speaker two are now both interacting you could also identify gender um in a conversation of the conversation of the of the participant you know male or female and also age bracket I mean you could sort of you know understand you know uh you know let's say female between 25 and 35 Etc you know uh all of this all of this you can do purely from tonal analysis um you don't need the facial expressions I mean obviously there's no video and you you you don't necessarily even need to process a spoken word and all of this can be done purely uh you know from the tone of voice and it's it's uh really amazing um in terms of sort of where the technology has come through but emotion emotion Ai and behavioral Ai and actually the sort of these building blocks of sort of assessing the uh the
cognitive State of Mind is definitely not a salt problem right so it's it's affect understanding is very very challenging right so you know if I were to ask or take a quick poll into like you know what do you guys think uh a gal think uh you know what's happening out here uh I'm sure there'll be like a you know a bunch of sort of diverse set of uh opinions in terms of what is actually going on in here but when you see the full picture you know um it might surprise you and that's why uh it is uh a complicated problem because the full picture is often missing and it's often sort of like the the area that we lack and but when we have the full picture um you know it things will become even more interesting so the stuff that we need to continue to build is one we have to build Uh custom um the sort of custom models for specific domains and there are many many domains that we see
yet haven't uh explored and the biggest sort of limitation Factor there is high quality data um getting access to the right amount of data you know making sure uh you know it's unbiased which is extremely problematic and also making sure that you know the whole process of annotation annotation and labeling is done up to standards um and then applying it to custom built machine learning models that are trying to one both sort of get getting deeper into the emotional behavioral state but also you know uh really trying to solve for an industry kpi that becomes relevant but there's a lot that is happening and it's coming together really really quickly and I'm really excited to sort of see at some point you know some of these components manage infest themselves itself into some of the AGI that we are going to see very soon that's it thank you guys thank you very much Rana so we do have a couple of minutes and I'll join you on stage to see if we have any questions
from the audience yes please thank you um it's probably two questions and I'm sorry I'm already a bit it's late now so I will try to be very structured but um so is it really that your model can understand donation or or from the Tunisian go to this state of emotion without any knowledge of like a cultural background and why I'm asking that is I know from working in teams talking to my partner with a cultural background my Austrian sarcasm is not understood if I go 20 kilometers away from my hometown kind of and and same goes for UK right and same as so sometimes I'm wondering the tonality then would kind of be like what's the tonality of sarcasm and can you go into this more um nuanced emotions as well or do you have like these four parameters that you have where I say okay these are the rough emotions let's say yeah that's a fantastic question um and um you know certainly very valid because uh obviously um there's a lot of
differences in terms of how we speak depending on what language we're speaking the cultural aspects and the dialects Etc first off before I answer that piece uh I mean sarcasm is extremely hard to do right and that's probably sort of you know uh it's it's uh sort of like the the last barrier I'd say you know the emotional behavioral AI because I mean humans suck at it right I mean how many humans get it uh and so you know I understand having uh maybe an AI do it is a very tall order so um that problem is unsolved I mean you could build sarcasm models but they're poorly performing and poorly in my opinion are anything that you know is an f-score of uh less than 70 75 is poor um because you know if three-fourths of the time you're right and then for the time you're wrong you know okay you know you can't really do much with it and so sarcasm is really really hard to do not something uh we're able to
solve right now but let's talk about emotions and behaviors right and other aspects right so um now the way this works I mean there's a deep technical answer to this but I'm going to keep it simple like when you're trying to measure uh like an emotion signal like anger or happiness uh or let's say anger and excitement because they're relatively similar but also very distinct signals what you're trying to do is you're trying to measure a point uh you know uh where uh you know where's the signal falling uh from a point of inflection of a certain Baseline that you've established for that data set and um when the signal Falls at a particular Baseline a particular point of inflection uh that's usually anger I mean anger is supposed to always fall there and if you find the signal falling there it's anger uh for the most part yeah so so the way that works is so I mean um so that is consistent among all languages all cultures all dialects but obviously the Baseline
shift right so like uh you know the tonal languages and their non-tonal languages and uh so the Baseline keeps shifting and Baseline also gets impacted by uh cultures by dialects by accents by among other things right and so um what we've been able to do is we've been able to recalibrate the Baseline with a very small amount of data right so today I mean we started off with a much bigger data set that we had to use to calibrate it and now we could do it with 30 minutes of audio that's it so if we were to like you know apply these engines to a language set that we've never worked on ever I mean we need 30 minutes of relevant audio in in that domain and the engines calibrate and the magic part of this is that you know once the engines calibrated and the Baseline shifted um doesn't matter which part of the world you come from then you're expressing anger it's the same point of inflection from the Baseline which is pretty
pretty remarkable right and that's sort of how it all works and you know our human brain also does the same thing let's say you're born in a certain country and then you immigrate to a different country and you go down there or forget about that right even if you go from a you know let's say from the south in the U.S and then you start to live in New York I mean your brain's like you know recalibrating which is like you know how someone talking I mean it's different it's still English but you know the brawl and the accents very very different and your brain takes a much longer human brain takes a much longer time to recalibrate and the AI actually can do it with 30 minutes of audio in a few minutes fantastic well thank you Rana and that that's at the end of our another final round of applause for you actually thank you what a great way to end the day thank you very much well that's a wrap guys um
thank you very much for your continued attention and for our lovely speakers for some really thought-provoking uh discussion topics we have a closing statement taking place on The Meta stage and then I think we're open for some drinks I'd like to send a final thank you to the people that have joined us on