Rise of AI Conference 2019 · Berlin · Unicorn Stage

Ultimate AI: Augment, Do Not Replace

Reetu Kainulainen · CEO at ultimate.ai

At Rise of AI 2019 in Berlin, Reetu Kainulainen, CEO and co-founder of ultimate.ai, argues that AI should augment human customer service agents rather than replace them with chatbots. She traces customer service from 1960s call routing through the dot-com era, cloud-based contact centers and, later, instant-messaging channels, noting that 4 percent of OECD employees, an estimated 7 million people in Europe, now work in contact centers, at an average enterprise cost of over 50 million dollars a year with up to 300 percent annual staff turnover. She says many companies tried pure chatbot automation but found the technology too shallow to replace human conversation. Instead, ultimate.ai builds an AI backbone integrated with tools such as Salesforce, Zendesk and Genesys, using deep neural networks trained on historical chat and email data to give agents real-time recommended answers, with a feedback loop that improves accuracy as agents accept or modify suggestions. She describes handling complex languages like Finnish, using clustering to structure unlabeled conversation data, and reports customer results including one user whose agents now take 96 percent of answers from recommendations, and response times cut roughly in half.

Key takeaways

  • Kainulainen argues standalone chatbots failed because they are not conversational enough and mask deeper structural problems in customer service.
  • She says an average enterprise spends over 50 million dollars a year on customer service, with staff turnover as high as 300 percent annually.
  • Finnish alone has around 2,500 inflected forms of a single word, which is why ultimate.ai uses deep learning rather than manual grammar rules.
  • One customer's agents now take 96 percent of their answers directly from ultimate.ai's real-time recommendations.
  • Kainulainen says agents in a test group that had recommendations removed reported feeling slower and more stressed.

Detailed notes

A short history of customer service technology

Kainulainen traces the evolution of customer service from 1965's intelligent call routing, which created the call center industry, through the dot-com boom's addition of email, the cloud era driven by companies like Salesforce, and the later arrival of instant messaging as a consumer channel. She notes that despite decades of adding new communication channels, the underlying way customer service is delivered has not changed: the same agents sitting at desks answering calls, emails and chats. She argues the next big shift will be AI-driven models that change how the people in these roles actually work.

The scale and cost of the problem

Kainulainen says customer service employs about 4 percent of the workforce in OECD countries, roughly 7 million people in Europe by her estimate, and that an average enterprise spends over 50 million dollars a year running its customer service department. She describes the operating environment as highly complex, with agents juggling calls, chats, email, tweets and WhatsApp messages alongside up to 50 different back-end systems, and calls the work highly labor-intensive, citing annual staff turnover as high as 300 percent. She argues this combination produces poor outcomes for consumers and brand damage for companies, and says many companies' attempt to solve it with standalone chatbots failed because the technology was not conversational enough and only masked a deeper structural problem.

Building an intelligent backbone: augmentation and feedback loops

Kainulainen describes ultimate.ai's approach as an intelligent backbone integrated into existing contact-center software such as Salesforce, Zendesk and Genesys. A deep neural network is trained on historical chat and email data and then gives agents real-time recommended answers as they handle multiple simultaneous conversations. A feedback loop captures whether agents accept or modify a recommendation and feeds that back into the model, improving accuracy without manually created training data. Once accuracy is high enough on simple, frequent cases, the system can move those cases to full automation, and because every message is classified, the aggregated data can also show a business its most common issues, agent performance patterns, and which conversations correlate with higher NPS scores or better sales conversion.

Three hard technical problems: language, labeled data and feedback

Kainulainen names three technical challenges. First, complex languages: Finnish, she says, has roughly 2,500 inflected forms for a single word compared with English's singular and plural, so the company avoids manual grammar rules and instead trains deep learning language models on public and customer-specific data to scale across languages. Second, labeled data: customer service databases hold millions or billions of unlabeled conversations, so the company uses clustering algorithms to semantically group similar questions and, especially, similar answers, which cluster more reliably than questions, to build initial training datasets. Third, the feedback loop itself, which she describes as the core of the company: agents validating or editing recommended answers continuously improves the model, which in turn produces better recommendations and more agent usage.

Customer results

Kainulainen presents data from an early customer with about 40 agents showing recommendation accuracy rising with usage, with one customer's agents recently taking 96 percent of their answers directly from the tool's recommendations. She cites an airline customer whose response times roughly halved within two months of deployment. She also reports that conversations handled per agent per hour rose after an initial learning curve, and that a test group whose recommendations were temporarily removed reported feeling slower and more stressed, which she compares to losing Google Maps or Spotify recommendations.

Q&A highlights

Asked how the algorithm learns, Kainulainen explains it is a supervised classifier: agents effectively annotate training data every time they select or edit a recommended answer, avoiding manual annotation. Asked how the company got early enterprise customers to test an imperfect product, she says it hired its first sales team in December, started by asking companies directly to share data under a data processing agreement, and built a first version from that data, favoring "show, don't tell." She lists over 30 enterprise customers across the Nordics, Germany, the UK and Romania, including Zalando and Telia (a further airline customer name is unclear in the transcript). Asked about onboarding speed, she says the initial data clustering step can be completed within a day, results from agent-facing recommendations are visible within the first 30 days, and the first automated cases can often go live within one to two months.

Quotes

Customer service is still between the two humans, and having a relationship with somebody to actually take care of you.

Reetu Kainulainen

The biggest innovation comes when you have these technologies and you understand how the process itself works.

Reetu Kainulainen

Questions answered in this talk

Why did standalone chatbots fail to fix customer service, according to Reetu Kainulainen?

According to Reetu Kainulainen at Rise of AI 2019, chatbots were not conversational enough and sat as a shallow layer on top of a contact center that was still struggling underneath, masking rather than solving the underlying problem.

How does ultimate.ai's AI improve customer service agents' work?

According to Reetu Kainulainen at Rise of AI 2019, ultimate.ai trains a deep neural network on historical chat and email data to give agents real-time recommended answers, with a feedback loop from agent acceptance or edits that continuously improves accuracy.

How much does customer service cost a typical large enterprise?

According to Reetu Kainulainen at Rise of AI 2019, an average enterprise spends over 50 million dollars a year running its customer service department, with some large companies spending significantly more.

Mentioned

deep neural networks, clustering, feedback loop, labeled data, chatbots, supervised learning, customer service automation

Keep exploring

More from Rise of AI Conference 2019

More on Enterprise AI & Adoption

Transcript

Auto-generated transcript, not edited. · 3,215 words

Show the full transcript

hey everybody I'm ready to climb Leinen I'm the CEO and one of the cofounders of ultimate a I'm and we help business you scale their customer service I know it's getting close to lunch but don't worry I'll throw in a few jokes and the time will just fly so let's start with the history of customer service so we think that customer service is really normal to us like it's been around forever but actually it's quite a new thing the guides at the ancient Egypt they didn't have contact centers or call centers exactly in the 60s when there was new invention in technology called intelligent call routing that boomed and created the call center industry so instead of having hundred phone numbers and hundred guys answering the individual phone call you could actually have just one phone number and then hide the people behind that and have a call center to provide customer service and that was in 1965 the second big switch was the dot-com boom so now people had emails and internet adoption was

growing rapidly so call centers became contact centers so now you had calls and put he also had emails those things we all love then the second big change in technology is or was the cloud so mainly driven by players like Salesforce and now companies could buy these highly complicated contacts and the solutions straight from the cloud and just pay a monthly subscription I'm from Finland I moved to Germany and I know there's still a little bit of work in the cloud side but still it's a it's a growing growing business and then in the 2010 to the 8th thanks to the guys of Zuckerberg and their what's up guys we start to have instant messaging as a consumer like a normal consumer behavior and that was also added to these lovely lovely contact centers and and all through our history there's more and more channels being added II easy for companies to have call centers or contact centers and more and more people being employed there and actually today 4% of the employee relation of

oacd countries actually work in contact centers that's a huge number like I come from Finland I think that's over 7 million people in Europe Finland has only 5 million people so from the babies to the elders everybody's just work in a call center but the issue there is that we've been adding new technology we're adding new pipes so it's easier to contact the customer service but we really haven't changed the way customer service is being supplied it's still the same guys sitting in the same desk or box answering your calls emails chats what's ups facebook Messenger editor but we feel that the next big change is gonna be an AI driven models that truly change how the people work within these call centers big problems right now like we heard from Michael earlier first of all the quite costly like on average an average enterprise spends over 50 million every year running their customer service department and that's just an average I don't want to even talk about the numbers that Deutsche Telekom's and bond

spent inter call centers and the complexity is enormous like you can imagine not just having to having the calls the chats to email the tweets and the whatsapp's but also having 50 different back-end systems that you have to operate and try to find the information as the chat is going on as the phone call is going on it's really complex in environment and like said it's really labor-intensive so more and more people work in call centers and like we heard more and more people quit up to 300% annual turnover in staff that's huge like you go around your whole staff you replace your staff three times a year that's highly inefficient and what is the outcome like what is your experience usually in customer service usually the outcomes are actually quite poor which is bad for the consumer but also it's it's a lot of brand damage from the companies itself so how do we solve this problem well knowbut chatbots that's for sure so there's a lot of hype and still is in bots

and automation so a lot of companies i canti to three years ago were born and said hey why don't we just deploy a chat bot that uses NLP then i have the conversations on behalf of the human we don't need humans in customer service anymore in theory and in paper when you do the ROI calculations that sounds really nice but what we have learned is that there's actually a lot of issues in this approach first of all they're not that good the technology is not really conversational customer service is still between the two humans and having a relationship with somebody to actually take care of you and we feel that the bots are just masking large large a bigger problem so usually the bots words too shallow layer somewhere in the corner of your website where the whole call center or contact center is actually struggling behind so this is what we're doing so we're building an intelligent backbone for the contact center so we're going a bit deeper than the bots so just quickly

go through how our solution works we can talk about later I can show you some demos later but we feel like there's three pillars of success being able to solve these issues first like Michael said earlier augmentation we believe heavily in augmentation so how our solution works is that we we have a deep neural networks based a solution that is integrated the existing contacts the solutions like Salesforce Zendesk Genesis then we train the Machine based on hysterical conversational data like Chacho email and then we start to augment the work of the customer service agent so now when you send a message to customer service the agent will get recommended answers from us in real time as they go through the conversation and when you have five to six conversation simultaneously there's always let machine helping the age and reduce complexity and kind of reduce also the stress of how should you handle this case then we like feedback loops so then we can actually just give stuff to the agent but also receive inputs from

the agents so now when the agent gets to recommend the answer and when the agent uses Dancer modifiers to answer sensed answer we automatically can feed that back to the algorithm to validate that yes this was actually a good good way of handling this case so we can grow in accuracy without manually having to create training data and then we can move the automation then we can say like hey your agents are handling this amount of cases per month and our recommendations are accurate in these cases and these actually super simple just two simple questions the customers are asking we should flick on automation so now we can seamlessly slowly and pragmatically move to automation and automate those cases that make sense to automate and finally since we classify every single message that comes into customer service we gain a lot of lot of data on how the whole business is run basically so customer service is such a treasure trove of data like customers are explaining what are their issues what what what is

going well what is going badly and the agents are talking to them every day so imagine being able to pull that and give that data to maybe the head of customer service who can say oh these are mouths our most common issues this is how we handle them this is how the agents are performing this is how individual agent is handling this individual topic versus another one and then oh this is actually a trend happening in our business of one problem we're having that we should communicate back to the let's say the product team that they would fix that problem and then we can move in a higher level of seeing okay which of these conversations lead to your high let's say a higher higher NPS score or in a sales related situation which of these conversations actually convert better and then start to optimize on that and recommend more answers to everybody but there's three hard technical problems heart problems to be able to do that the first one is Finland of course so

there was actually a good question about the dialect we come from Finland and we got the pleasure to start with one of the hardest languages in the world and in call centers when you work with large companies even if made majority of their customer service might be English still they're global companies and they might serve ten different languages twenty different languages thirty different languages and you have to scale so how do you handle complex languages like Finnish I can give an example in English you have basically you have a singular and plural form of a word and that's about it so almost like a key word shirts thing might work in Finnish let's say there were a store has two and a half thousand ways the world inflection changes between in relation to the context so we didn't want to build manual rules how each one of these grammars or languages work so of course major breakthrough or major advancements in machine learning is deep learning and this is what we use as a foundation

technology in our technology stack so now we can just give the algorithm a lot of let's say a publicly available data but also the customer specific data to train the language model then can be highly accurate in languages actually which is deploying Romania so we can scale quite nicely second is the labeled data because like I said in customer service you have millions or in some cases billions of conversations just as a textual form in a database or somewhere and then Desco sales for us it's quite hard to grasp what to do with this data and some let's say in the power statistical Lda unsupervised learning methods hasn't been that good it's kind of gained insight from that data but we'll build is these clustering algorithms it's not just one silver bullet there's a lot of stuff but that can tap into the history cool data and try to semantically separate similar questions and similar answers and actually good tip answers are usually more standardized than the question so it's easier the cluster based on

answers so then we tried to show the customer like hey based on your data we can find let's say here's the 50 most reliable clusters we found for these cases and this is how your agents are giving the answer and we can use that as datasets to deploy the solution second thing is the feedback or third thing is the feedback loop so now that we have the initial model we deploy that we start recommending answers when the agents validate those answers and get value and use answers then we can grow in accuracy when they modify those answers we can add more answers and then we have this nice feedback loop where we can really quickly show better recommendations and when we show better recommendations the agents use more of them which creates more accurate recommendation and now we have a nice feedback loop and this is the core of our whole company creating feedback loops inside the process how customer service works and then creating let's say value-added services like documentation or automation or analytics

on top of that here's just to end this with a few slides so results so this is actually one of our earliest customers over a year ago so we can see a nice trend line of the usage so this is about 40 agents using the tool and the more they use the tool the higher high accuracy the tool gets and then the more they start using that actually nowadays we have users that 96 one of the users just check last week 96 percent of their answers come straight through our recommendations and that's quite insane almost then this is the end result so when we've created a process an algorithm that fits into the process and works with the agents help the agents to answer better and faster we immediately can cut the response times almost in half so this is actually one of the airline customers we have and in months two we have almost over over double over half the response times and this has a major direct impact on the end customer experience

like imagine being being able to handle your customer service issues double the speed that that feels good then kind of the money shot in here is for the business itself so of course there needs to be some benefits there for these large corporations to invest in these solutions so this is metric we're really obsessed about so how many conversations per hour let's say one agent can handle and actually this is you can see it's real data because it goes down sometimes so actually this is really interesting to see because in they started about seven and a half conversations an hour and when they start to use the tool it was a change how they have to work and there was a liberal learning curve so it took them a while to get used to of searching just answers the knack she's just choosing the answers next to the next to the chat or email but then when they became used to the tool it became a status quo for them and their efficiency went up

without them feeling more stressed actually the agent was saying we have this test group that we kind of took the tool away they said that they felt slower and they felt a bit more stressed because they couldn't rely on kind of recommendations and I feel like it's similar if I'll take away your Google Maps or your Spotify and you have to think what what do I want to listen it creates tension and friction so how I want to end this is to basically convey that what we learned for the past three years is that all these state-of-the-art machine learning algorithms and deep learning is it's great and we're investing heavily in the dos but that's that's half of the problem the biggest innovation comes when you have these technologies and you understand how the process itself works like Michael talked earlier how the call center agent or or the sales agent works and what are their problems there and how can you fit these days of our tools into the daily work that's when the

true innovation happens so I have a few minutes left with time for Q&A and then we can all go for lunch thank you guys thanks day 2 so let's start with the questions Tomas I see you and we have to wait for the microphone sorry thank you and no thanks I looked for the presentation I would be interested on actually how your algorithm learns you know usually you gather a lot of data from those conversations but how actually the learning is accumulated well on a higher level it's it's about just creating let's say we have it's a classifier in the end that that is in production the clustering is a different different algorithm but the classifier basically how it learns is that it gets more training data and how we get more training data it's a supervised learning problem so we need somebody to annotate the training data so instead of manually annotating them we use the customer service agents while they work every time they select answer they basically annotate the piece of data

and that's how they let's say I don't want to call it self learning but the active learning happens within the algorithm good great another question there in the back thank you for the presentation so I'm a founder and I build and sold companies already but I have no idea how you got companies to test your product because at some point you must have had a not that perfect product and then you went to some big corporation and said hey just go for it or how did that work right now we have we have over 30 enterprise customers from the Nordics Germany UK Romania soon customers like Zalando orphan air or Telia large stella coasts and well in the early days we actually hired our first sales team last december and everything was just me literally going to the guys and saying hey you want to give us your data and we'll do something about it and that was just for gdpr so they're still doable but right now how we how we basically just do

we always want to show stuff you want to be pragmatic usually what we say hey let's sign and in the a at EPA and then send us some data and then we'll kind of create let's say the first version for you guys and then go forward usually show don't tell that works quite well but it's hard and maybe one thing to add there can you because we was talking about about feedback loops and that could be kind of a mode for your company so strength what I found very impressive I think you have onboarding time or at least time to return on investment awhile for the customer as a KPI so how fast do people see actually the impact of ultimately oh yeah that was one key thing because like we always think about being an AI first company and like it's a different than being a SAS company but it's just create a column that's it if we would just create a count to our customers and give it to them to actually be

really disappointing or destructive almost so you always have to create initial data sets and that's why I've invest still heavily on the clustering part so we can go the clustering itself won't take longer than a day and it will basically comprised of having the initial clusters and we have somebody going through them cleaning a little bit and doing that and it like in it iterative a matter so we can in theory we can do it within the day deploy the customer then it starts creating the recommendations learning from the agent within the first 30 days you can immediately see the kind of the results and how it's working and you can oh actually quite quickly even go to automation because then some of those cases are so obviously simple that you can let's say do the first five or ten automation cases within the first one or two months you

Rise of AI 2027 · Berlin · May 2027

Be in the room for the next one.

Ten years of talks started as conversations in one curated room in Berlin. The next edition brings 300 selected leaders together in May 2027.