Rise of AI Conference 2018 · Berlin

From Statistics to ML to AI

Steffen Wachenfeld · Chief Product Officer, WeQ

At Rise of AI 2018 in Berlin, Steffen Wachenfeld, Chief Product Officer at WeQ, argues that companies should not skip stages when introducing intelligent systems, moving instead from simple statistics to machine learning and only then to what he considers true AI. Drawing on his experience in performance marketing, he walks through the economics of buying ad impressions and explains how a company first improves results with basic counting and A/B testing, which only optimizes one variable at a time, before moving to machine learning that handles many variables at once, powering recommenders and real-time programmatic ad auctions decided within 50 to 200 milliseconds. He calls that stage engineering and machine learning, not AI. He defines AI as a system whose behavior becomes human-like: it observes, learns patterns and then acts on its own, such as detecting fraudulent ad traffic and notifying an account manager without being asked. He closes by saying the useful cases are ones like answering complex analytical questions that would otherwise take a person half an hour, not chatbots that just answer simple FAQs.

Key takeaways

  • Wachenfeld argues companies should progress through stages, from counting to machine learning to AI, without skipping ahead.
  • In his performance-marketing example, roughly a million dollars a day buys about 2 billion ad impressions at 50 cents CPM, breaking even at a 2 percent click-to-install rate.
  • He distinguishes machine learning, such as real-time programmatic ad bidding decided in 50 to 200 milliseconds, from what he considers true AI.
  • His company discards 30 to 40 percent of ad traffic as fraudulent, detected through pattern recognition on bot-driven clicking.
  • He defines AI as a system that observes, learns patterns and then acts on its own, such as flagging bad traffic and notifying an account manager unprompted.

Detailed notes

A hype cycle that plays out inside every company

Wachenfeld opens by polling the room on business versus technical backgrounds, then describes a pattern he sees across industries: new technology is first dismissed, then over-hyped, then settles into realistic use. He says the same cycle plays out among a company's own leadership, with technical staff first pushing the board to adopt AI and later having to explain to the CEO what AI cannot yet do.

Data hunger and the economics of performance marketing

He recalls that as a graduate student the hardest problem was never the algorithms, which were all documented, but getting access to data, such as car braking data for an anti-lock braking idea. He contrasts this with his current world of performance marketing, where a company might spend around a million dollars a day buying roughly 2 billion ad impressions at about 50 cents per thousand (CPM). In his example, a 2.5 percent click rate yields 50 million clicks, of which 2 percent convert to app installs, returning about a million dollars and putting the campaign at breakeven, generating large volumes of ground-truth data on who clicked and who did not.

Stage one: counting and one-dimensional optimization

He argues the first stage of maturity is simple counting: tracking how often an ad placement gets clicks, blacklisting poorly performing IDs, and running A/B tests between creatives. This requires no specialized staff but only optimizes one variable at a time, for example buying cheaper impressions that turn out to be in less visible positions, or picking a banner that performs well for reasons unrelated to the product.

Stage two: machine learning across many variables

The next stage brings in engineers and databases to optimize many variables simultaneously, including automatic feature learning to identify which of thousands of signals (screen size, operating system, geolocation, weather derived from IP and time, and more) matter. He cites recommenders, such as Amazon's purchase suggestions, that can be offered as an API so other sites ask a system what to show, and real-time programmatic ad auctions, where a bidding decision is made within 50 to 200 milliseconds. He is explicit that he does not consider this AI: 'to me it's not AI, that is engineering, machine learning maybe, but it's not AI.' He also describes using pattern recognition to detect and discard fraudulent click traffic, which he says accounts for 30 to 40 percent of traffic his company throws away, and flags the filter bubble problem in recommenders, where a single click on politically charged content leads a system to show only more of the same.

What counts as AI, and where it adds value

For Wachenfeld, a system only becomes AI when it observes, learns patterns and then acts on its own, the way a responsible person managing an account would, for example detecting bad traffic from a source, flagging it with certainty and notifying the relevant account manager without being told to. He says he does not want a chatbot that makes small talk, comparing it to Alexa, which he only ever asks about the weather. Instead he wants a system he can ask complex natural-language questions, such as the change in revenue over the last month for the top ten hotel offers, that a person would otherwise need half an hour and a pivot table to answer, with the system translating the question into a database query. He closes by saying the technical components, natural language processing, pattern recognition and image recognition, already exist, and the real question is picking use cases where they add genuine value rather than replacing simple FAQ bots.

Quotes

To me it's not AI, that is engineering, it's machine learning maybe, but it's not AI.

Steffen Wachenfeld

The biggest problem was not the algorithms, because they were all in the books, but the biggest problem was data.

Steffen Wachenfeld

Questions answered in this talk

What is the difference between machine learning and AI, according to Steffen Wachenfeld?

At Rise of AI 2018, Steffen Wachenfeld says systems that optimize many variables at once, such as real-time programmatic ad bidding, are engineering and machine learning, while he reserves the term AI for systems that observe, learn patterns and then act on their own, the way a responsible person would.

How does programmatic real-time bidding for online ads work?

According to Steffen Wachenfeld at Rise of AI 2018, each ad impression is auctioned in real time, with bidders deciding whether to buy it within about 50 to 200 milliseconds based on data about the device and user.

Why do recommendation systems create filter bubbles?

Steffen Wachenfeld explains at Rise of AI 2018 that once a user clicks or likes content, such as politically charged posts, a recommender assumes they want more of the same and increasingly shows similar content, reinforcing the bubble.

Mentioned

Performance marketing, Programmatic real-time bidding, A/B testing, Feature learning, Recommender systems, Filter bubble, Click fraud detection, Natural language processing

Keep exploring

More from Rise of AI Conference 2018

More on Enterprise AI & Adoption

Transcript

Auto-generated transcript, not edited. · 3,599 words

Show the full transcript

yeah hello everybody my name is Stefan I work for week you as a chief product officer and from a background as you heard I'm a machine learning guys or more on a technical side but I also studied business so I have these two legs business and tech and that that would be my first question to to the audience so that I have a little bit of a feeling who's who's sitting here so if you think of who's more on the business side could you maybe be so kind and and raise your hands okay and who would consider himself being more on the technical side maybe ok ok that's cool that so we have a 50/50 or maybe some even raise their hands twice so I will talk a little bit about something that I observe it's it's a journey how yeah some sort of intelligent processing enters in industry because I think it might be relevant for your industry I guess very few of you might be from advertising industry which is the current company

I work for but it's pretty much I think a pattern that I see in in in multiple industries so you all know this hype cycle if something is new then nobody wants to hear about it is super skeptical whatever talk to your parents about blockchain and you have this effect then half a year later everybody is on this super hype and everybody wants to invest all this money into some cryptic horns or everybody thinks all cars will be speaking next year and the fridges will talk to you and then after some time it becomes a little bit more realistic and I think in the company it's the same you have different people on the board you have a CTO or CEO CFO maybe different guys and they have a different position where they are on this hype cycle so at first maybe the tech guys try to convince the other guys on the board that they totally need to do something and later it's the same tech guys that try to convince the the CEO

that no they cannot do everything with learning an AI so this is what I see and so what is a realistic journey of something or realistic introduction of of technology and yeah this is what my talk is about at first I going to show you a doughnut it's it's the universal representation of data and for those who raise the hands to be more on the technical side I guess some of you have maybe a degree or studied whatever computer science machine learning pattern recognition artificial intelligence and if you did that I did that and I remember that the biggest problem was not the algorithms because they were all in the books but the biggest problem was data so if you wanted to work on any problem like you have a super idea how to improve the anti break system of a car where would you get break data from cars and and try to get it like try to write to BMW or so if they want to give you some some car data they

were probably not so you sit there and you are hungry for data this is pretty much the situation I know from every graduate student and yeah now I'm in advertising where the situation is a little bit different and I'm gonna quickly explain a little bit the very very core business principle and which is at the same time the the optimization criterion in that respect it's like what do we do and then we look at what how to optimize it pretty much the advertising business is very simple there's a person and we are in performance marketing so that means there's people that say I give you a dollar if you make someone install my crazy app or I give you a dollar if you make someone buy my just presented car buddy and so you say okay I I'll try to do that and you buy impressions but just this little banner slots - in the simplest case but you can also buy someone on YouTube - to to create a video or an influencer or

whatever but like at the end of the day you you you buy kind of impressions which is showing something to someone and the average price is maybe 50 cents for a thousand so CPM means for a thousand for for 50 cents you can say hello world to a thousand people and you do that for $1,000,000 because that's like your budget or what your cash flow allows you to spend and you buy 2 billion impressions and that's a realistic number maybe a realistic number for a day so maybe a typical company that is small or medium sized would spend a million dollar like like our company would spend a million dollar a day on buying impressions and they're not or money we do it for for other people and then let's say only 2.5 people click on the the thing so you get so you get a 50 million clicks 2% of those actually download the app so you end up with 1 million downloads and guess what you get 1 million dollar back so that

is pretty much the business and now you made it your breakeven you can do that if you don't pay taxes a pretty long time and now you want to improve that somehow and you say okay what can I do to be profitable to to make my business better and you typically in a small company like a 10 people company 15 people company you have people doing that manually and they will start yeah so your breakeven but also the nature of this is the Paradise yeah because you get from all these impressions you get like ground truth data you know who clicked and who not and you have a lot of data but you want to improve it and so at first in order to be preferable you can just go which is what we do you pick up the phone and you call the guy said dude I need two dollars instead of one dollar for install can we do that and then maybe you agree on one dollar 25 and in that moment you're

profitable next thing is you could buy cheaper impressions like cheaper cheaper places where you where you show that you could optimize your whatever you show to improve the percentages of people that actually click or you can make a better match like what you show to which person or if they go after clicking they go to the App Store you can also improve the descriptions and the images in the App Store to improve all that so that is pretty much the whole business if you if you do that you have a business you can go out and money and it's pretty cool the problem is if you scale it you need a lot of people to do that and and pretty much this is a typical example for many businesses we heard talk from telecom we heard a lot of other talks where there's a lot of repetitive tasks that involve a lot of manual work and and most of this can can maybe be done by by machines but how first of all congratulations if

you're at that point with the numbers I just told you you receive already a lot of data so you have between 10,000 and a million events per second which is all these showings of ads and and the information if someone clicked or downloaded and that that even if you compress it slightly but with all the additional data like where was it from which browser that he has like all the data that you receive you end up with terabytes of data per day so obviously you cannot take a PhD student or or a bachelor student with his notebook and ask him to run his machine learning algorithm on that data it doesn't work so what do you do and what I see is in all the situations where you have that many data abhi telecom or something else there's a journey that that goes across some stages and I would just like to talk about that and maybe let it discuss it with you maybe you have a business maybe you know where you are and

I think you should not skip a stage I think that is the most important thing some business are very manual and then there's this great idea yes let's do AI and they start at the end I think what what the telecom I think what is was it Yan right what the unjust said was that they started directly with natural language processing in order to make a bot talk to customers which is pretty much a pretty-pretty yeah that's a big problem at the end of what you can do so I think the first thing is to stop counting we use counting is very easy and you can just count I show this how many people click I show that how many people click where was it shown was it shown in this certain spot in all these places where you show something they have an ID and you can identify I mean you see okay all always this idea it never nobody ever clicks so you can you can start blacklisting which means I don't want

to show any more on this ID so obviously that is going to be millions of IDs that you need to review and you can use a very simple statistic and you just count and you say ok this is a good idea this is a bad ID in general in average and you optimize for that or you say oh I have for the same thing I have two banners like the red one and the blue one and I see which one performs better and you start the a/b testing which is the natural thing which is stage one which is very easy you don't need any PhD student nothing you can just use any person that has counted before and you can you can do that problem is you just look at one dimension so if you say if you look for example optimize for the price and you buy cheaper you typically the cheaper spots are also not in the nice places so maybe they're not even visible like maybe they are below the fold like

when the page appears or something or you a be test two banners and you find out that the one with a naked lady or the one with the win and I pad performs super good everybody clicks on it but obviously it has nothing to do with your product so you're just optimizing one dimension and this is the the biggest problem also in other industries when you go for that you you typically tackle only one dimension also if you do it manually if you let people do the work typically people optimize only for one damage they can say I look for the largest or best in this aspect but combining this banner at that time for this person that is pretty difficult but it has been researched centuries ago and you can use I think this thing yeah you can you can use machine learning and you learn on multiple various at the same time so machine learning is specialized for that and that is the first time when you really start getting some engineers in

the company building some databases and thinking what are the interesting variables because pretty much all of the time you have too many variables also like if you shown at I we receive I don't know thousand or 10,000 different variables like screen size operating system how many other apps the geolocation like whatever you want and with some of the data you can get additional data if you have the IP address and the time you can get the place and with the place and the time you can get the weather and you can you can pretty much use whatever you want and so it's important to learn which features are relevant so if you know the business you have been in it for 10 years you pretty much have a good guess and be like ok that's not important or this is important but there's also ways to do that automatically which is automatic feature learning methods and they should be applied at this stage it's a very good stage it's a stage as fun and that

has a really good impact on on revenue and also on scalability examples are all these recommenders that you know like you go on Amazon you buy something or even you login of course the system recognizes you and then gives you a recommendation what are the top 10 things that you should look at based on your selling history the problem is always the same you have you're asking a computer system to do a recommendation what will be the most likely thing that you buy the most likely banner that you click on the most likely insurance that will you will purchase and so on plus the next thing is you can provide this as a service as a company you can just say I have a recommender and I create a an API for that that means other people can connect to it so that means you do the work of recommending you observe and recommend and others say I have this offer wall on my cat page and I would like to know what I should

show on this cat page and you asked the computer and then the computer tells you which ads should be shown and that's also what you can buy off-the-shelf from Google like if you have Google advertisement on your on your web page it's pretty much your page ask Google what they should show they they put something on it and there's real time by programmatic buying which is a huge share of the market of how currently advertisements are being bought maybe if you don't know it it's it's an interesting this is actually like a stock exchange you have to imagine where well if you would all turn on your phones now and there would be let's say we all open a page where there's an advertisement you probably would be auctioned off in a on some sort of exchange where for each of you there would be like an ad request saying okay it's an iPhone it's an Android which version and so on and then other people can bid and say okay I bit one cent

to show him and I with two cents to show her an ad I and so on and that happens in real time so within like 50 to 200 milliseconds computers decide on if they want to buy an ad or not let us currently of how today the advertisement works that requires super huge amounts of data it requires machine learning it real-time everything you have only 20 milliseconds or so so sophisticated but to me it's not AI that is engineering it's machine learning maybe but it's not AI so I think the next thing is I have maybe maybe one thing on this thing what youtube or what I typically do or did wrong in the past and what I see others doing wrong is you model the reality like and then ideal world you know how it should be and then you let your algorithm run and then you realize that actually the world is different and then you try to find what is broken and one thing is for example that you have these Chinese

or Russian guys or also maybe Germans I don't know but that have these huge racks of phone and they don't do anything else but clicking adds the whole day like you can program them to just click and download ads and that's when you use pattern recognition because you don't want to pay for that you don't want to have that in your traffic so I think our company and we throw away 30 to 40% of traffic because it's just the empty spots that we we don't use because we consider it bad another problem with this recommenders you know the YouTube thing but it's the same on Facebook is that you get more of the same if you're a dominant Donald Trump fan for example you would think that everybody's a Donald Trump fan because once you liked it once the system thinks you will like it always and give you more of that because you're more likely to click you're more likely to like it you're more likely to share it so it's it is it

creates the problem that it puts us all in these bubbles and that's also the same for advertising and that's where we're far like AI is a completely different topic but there we we're not even able to after I advertise the the fridge that you always want it to you and you bought it you still for two weeks get the advertisement for fridges and you're like dude I just bought the fridge so why do I still get advertisement so there's a very basic promise that need to be solved anyways now III two and a half more minutes now III this is this is where we want to go and what does it mean so for me I is something where the problem is so complex or the behavior is so natural that it becomes human like or even over humanely I don't know what the word is for that that means to me a system needs to be able to observe and learn and to detect some patterns and then act on it which is pretty

much what a human would do and for our a thing it would be really I observe a lot of bad traffic coming from the source and the system texts it is sure and creates some action and maybe even sense the email to the the traffic source or notifies the the related account managers and acts like a human like like a real responsible person would do that isn't the place of managing this account or this traffic source and the next thing is and we heard it before bots yun talked about it from telecom and the other room was also talk about bots what is about a good part for me is i would like to have my account managers talk to my system but I don't need the natural language processing and that it wishes them a good day and so on it's really for me it's a tool that should do things that are difficult and that I want to do easier it's not like the biggest problem of Alexa is that you don't have

a use case like I pretty much asked her how the weather's and then I'm done with her so but you could really make more complex questions which normally a person would need to create an Excel and a pivot create a pivot table and and and work half an hour to answer a question but if you could put that questions in like what is the potential change in the revenues of hotel abroad of our top ten offers in the last month that can be understood that could be translated by the system into the corresponding database query and create a result that you know replaces half-an-hour of a bi person's work or if the system like before observed and acted it would just say ID deactivated the following offers because I believe something is wrong and I notified the partners and it just tells you that or writes you an email or calls you yeah and when we would get there and I'm done with my time I think the components are all there so you have

the components of natural language processing you can do pattern recognition you could also use image recognition in other fields and the point is to me more what do I want and where does it really add value and like this adds to me value but for example the the typical talking bot of my bank or insurance I don't like it because if I call my insurance I have a real problem that's so complicated that the five fa Q's didn't answer it and so that is when I the last person I want to talk to is a bot but like if I'm like ah now I now I have to create this complicated excellent get all the data that's what I would like to talk to a bot and be like dude get me the answer to this and this question and please know how we do it and that's that's it from my side unfortunately yeah I don't know if I if there's a opportunity for questions maybe one question thank you very much already

Rise of AI 2027 · Berlin · May 2027

Be in the room for the next one.

Ten years of talks started as conversations in one curated room in Berlin. The next edition brings 300 selected leaders together in May 2027.