Rise of AI Conference 2024 · Berlin · Applied AI Stage

Are Machines the Better Decision Makers?

Katharina Zweig · Professor at Rheinland-Pfälzische Technische Universität Kaiserslautern-Landau (RPTU)

At Rise of AI 2024 in Berlin, Prof. Dr. Katharina Zweig of RPTU Kaiserslautern-Landau argues that machines can reliably help with only some kinds of decisions, not all of them. She distinguishes four decision types: factual decisions, risk predictions, judgments and singular decisions, and argues AI can, in principle, be checked and trusted for the first two, since their correctness can be verified, but not for judgments, where qualities must be weighed subjectively and justified in words. She demonstrates that ChatGPT 3.5 cannot reliably perform long multiplication despite producing plausible step-by-step explanations, arguing a single failed test is enough to show a system cannot be trusted with a task in general. Using a real case of a user asking ChatGPT to build dynamic pricing for a shop, including an Amazon incident where competing pricing algorithms sold products for one cent, she shows how large language models generate functioning-looking code without asking the requirements questions a human expert would raise. Her conclusion: citizens need basic computer science and AI literacy to check AI-generated solutions rather than trust them outright, since judgment decisions still require human weighing and justification.

Key takeaways

  • Katharina Zweig splits decisions into four types: factual, risk prediction, judgment and singular, only the first two of which are reliably checkable.
  • ChatGPT 3.5 fails at long multiplication on larger numbers despite giving a plausible step-by-step explanation, showing explanations do not prove correctness.
  • A single failed test is enough to show a machine cannot be trusted with a task in general, unlike a human who can be said to 'still learn' from one mistake.
  • An Amazon dynamic-pricing incident, where competing algorithms drove a product's price to one cent, illustrates the requirements questions an LLM never asks.
  • Zweig argues judgment decisions, which require weighing qualities and giving justifications, cannot be delegated to today's statistical AI systems.

Detailed notes

An 8-year-old's decision procedure

Zweig opens with a childhood story: told by her mother to say goodbye to moving neighbors or risk regretting it later, she devised a decision procedure as an 8-year-old: collect available data, list possible decisions, think through consequences, then decide and live without regret. She uses this to introduce the idea that most real decisions cannot use all available data or fully map every consequence, which is why people might want machine help.

Four types of decisions

Zweig categorizes decisions, drawn from her recent book, into four scenarios. Factual decisions and risk predictions are both checkable: for factual decisions, experts can verify the same conclusion intersubjectively; for risk predictions, such as creditworthiness, the outcome can in principle be checked once the future unfolds. Judgment decisions, a concept she borrows from Kahneman, Sibony and Sunstein's book 'Noise', are ones where experts may disagree but not arbitrarily much, such as grading a university exam. Singular decisions, such as those made during the pandemic, lack historical data entirely, meaning AI has nothing to learn from and cannot help.

Testing whether ChatGPT can do factual tasks: long multiplication

Responding to a forum question asking whether ChatGPT can set dynamic prices for a store, Zweig first tests a simpler factual task: whether ChatGPT 3.5 can perform long multiplication. It explains its steps clearly and gets small numbers right, but produces incorrect results for larger numbers despite a plausible-looking explanation. She argues that unlike a child who makes an occasional mistake but 'can still do it', software that fails even once on a well-defined task cannot be trusted to perform it reliably, because there is no way to know in advance which cases will fail. She cites a colleague's term for LLM output as 'intelligible textures', text that reads like an explanation without reflecting genuine understanding of the procedure.

Dynamic pricing and the limits of AI judgment

Zweig identifies the forum user's real request as dynamic pricing, a well-studied area with more than 100,000 academic papers, and argues an LLM cannot know which existing approach fits a specific business without requirements engineering that the user never provided and the model never asked for. She cites an Amazon incident, shortly before Christmas, in which competing dynamic-pricing algorithms each undercut the other by one cent, driving product prices down to one cent until a seller noticed. She contrasts human judgment, which selects and weighs qualities, justifies choices in words, and adapts case by case, with what a machine actually does: predicting what an average programmer's algorithm would output, based on statistical training on past data, without being able to justify the result.

Conclusion: what citizens need to learn

Zweig concludes that factual decisions and risk predictions are reasonably safe to delegate to AI because they can be checked, but judgment and singular decisions cannot be delegated with current statistical methods. She lists what computer scientists actually contribute beyond coding: checking hidden assumptions, requirements engineering, correctness proofs or testing, and checks for safety and fairness. Her recommendation, addressed partly to a representative of the German Federal Ministry of Education and Research (BMBF) in the audience, is to give citizens basic computer science and data science literacy, including Python reading skills, knowledge of quality and fairness measures, and testing strategies, so they can check AI-generated solutions rather than judge machines by the standard of a 9-year-old who merely explains a procedure.

Quotes

A single fail test is enough to show that a machine is not in general able to do the tested task.

Katharina Zweig

Computer cannot judge for you, so you must judge for yourself.

Katharina Zweig

An explanation by an LLM does not mean they know how to do it, they just know how to create the text that feels like an explanation.

Katharina Zweig

Questions answered in this talk

Can ChatGPT reliably do multiplication, according to Katharina Zweig?

According to Katharina Zweig at Rise of AI 2024, ChatGPT 3.5 gives plausible step-by-step explanations for long multiplication but produces incorrect results for larger numbers, and since large language models cannot have learned exact multiplication from examples alone, she argues they cannot do it reliably regardless of how many examples they see.

What are the four types of decisions Katharina Zweig identifies?

According to Katharina Zweig at Rise of AI 2024, decisions fall into factual decisions and risk predictions, both checkable against a verifiable outcome, judgment decisions, where experts may disagree but not arbitrarily, and singular decisions with no historical precedent, which AI cannot support at all.

Why does Katharina Zweig say AI cannot make judgment decisions?

According to Katharina Zweig at Rise of AI 2024, judgment decisions require selecting and individually weighing qualities and then justifying the choice in words, something current statistical AI systems can only approximate by predicting an average past decision, without being able to justify it.

Mentioned

Dynamic pricing, Long multiplication test, Judgment vs. factual decisions, Noise (book), Requirements engineering, AI literacy, Intelligible textures

Keep exploring

More from Rise of AI Conference 2024

More on Ethics & Trust

Transcript

Auto-generated transcript, not edited. · 2,926 words

Show the full transcript

good morning I'm very happy to be here because really I just noticed how much I want to better understand decision making and especially decision making with about and by machines and to tell you why that is I think it's helpful to look into my childhood so when I was about eight our neighbors were moving and it was time to say goodbye but I was playing along in the back yarden and so my mother yelled down to me and asked Nina don't you want to say goodbye to the neighbors and I was like no we're playing here and then she said yeah you know they will be gone and you might not have the time to say goodbye to them you might not see them again and you might regret it and my 8-year-old self thought okay regret so I need to make a decision now that I will not regret how will I do that and so this is the how to not recreate a decision in the version of an 8-year-old collect all available

data neighbors moving away collect all possible decisions saying goodbye or don't saying goodbye think through possible consequences I can play now or I have to get up and say goodbye to them and then make an informed decision deal with the consequences without regret and that was my plan and I did not say goodbye to the neighbors and I never regretted it so of course when you get older you see that the eight-year-old version does not work right so what you see is uh most of the time you can only grab the easily available data and you have to collect the most important possible decisions and then the rest you just hope that everything will go on so given how many decisions we all need to make each and every single day wouldn't it be nice if we could get some help by machines and so for that we first need to talk about what actually is a decision and as a computer science scientist that's very easy to answer of course a decision is a

number right so whatever it is uh choosing your next car um finding out how many children you want to have you can always put it into a number or if you really can't I would not be interested it in in it because I'm a computer scientist and the nice thing about numbers is they can be computed right so the question is can a computer make a decision or at least help you with it and we've been talking about algorithmic decision- making with the help of AI or just pure data science for a very long time and in my last book I tried to find an easy categorization on what kind of decisions we actually make and I think there's only four scenarios of Interest so the first one is we need to make factual decisions right the second one is we would like to have um uh a risk prediction and both of these are facts in the first one experts agree there's a verification process it's inter subjective you should come to the same

conclusion as I do and then we can also check it with respect to the quality of decision- making with risk predictions that's kind of a fact in the future we can check it in the future unless my decision today changes the future so if I do a risk prediction on credit worthiness and you don't get the credit then you can show me that you would have paid it back so some of these decisions will change the future so this is why I've differentiated these two cases then the most interesting decision-making scenario is Judgment so what is a judgment judgmental decision I like this definition very much by caraman siboni and sunstein in their book noise it's a thing where experts can disagree but not arbitrarily much so for example a grade in university is something where I can give I don't know a one minus and my colleagues give uh a three plus we don't have to agree but we shouldn't be a part too much so this is Judgment scenarios and then we have

singular decisions and this is what happened in the pandemics because luckily this was a singular event this is what happens all the time in globalization we've not been there before so this is um a part in which we don't have historical databases and that means you can't get any help from AI there because if we don't have data AI won't be able to help you so now the question is can the AI make factual decisions and these are questions that I get all the time in interviews and the very first thing I say is we shouldn't talk about the AI it's a zoo of systems and it doesn't make any sense there is no consciousness it's not one thing so please be a little bit more specific yeah so that's the wrong question and then I found a very interesting question in a forum where a person asked this can cat GPT help me set prices automatically for products in my store okay that's an interesting question because we have a specific AI system and

we have a specific question but let's start a little bit more simple so in order for chbt giving you prices maybe chpt should be able to multiply numbers right so let's check whether AI or chbt in that uh that regard can do long multiplication this is one English term I had to look up it's multiplic so can chat do long multiplication of 13 * 25 and then it first explains to me what it does first you multiply 13 by five and so on and then it also gave me like a diagram and it looks perfectly well this is uh chbt 3.5 by the way so and then I asked it for the long multiplication of longer numbers and again the explanation was pretty okay the numbers correct yeah everything looking fine but here you see that it does something strange yeah it doesn't it doesn't actually do it correctly so there's something going on and now in this forum I try to make people aware that chbd can't even multiply numbers maybe you shouldn't ask

it to do the pricing for you in your store but what I found was really intriguing I had a lot of discussion because there are two opposing views and the people in The Forum said yeah the machine can explain the procedure so it just made a simple mistake so it is in principle able to do it it just needs more examples but my view as a computer scientist is and now if you have a software that gives one wrong result it's not capable of doing it reliably so it gave the wrong results plus if you look at the technology there's no way in which it could have learned multiplication from just examples it's it's not possible but people didn't want to listen to me so um of course we see some results that are correct from examples that's totally understandable from the technology but more examples will not help fundamentally llms cannot multiply and I thought that is so interesting that normal people that do not have my education as a computer scientist have this

confusion but then I thought about it it's the following confusion I have a 9-year-old if he can explain the procedure to me and just makes a mistake I would still say he can do it he just needs more practice as a computer comp scientists I have to say if your code is not running reliably what it ought to do it's wrong and so I think we need to teach them so we need to teach them to have a Technology based assessment and we need to teach them that if you want to have a reliable system then one single test failing shows you that there is something going on so that is my my main message to you to today that in contrast to humans a single fail test is enough to show that a machine is not in general able to do the tested task and in general means not reliably not in all cases and the problem here is you'll never know in which cases it is and in which cases it is not

moreover an explanation by an llm does not mean they know how to do it they just know how to create the text that feels like an explanation because they've seen it so often so and this is really an article that I would like to recommend to all of you it's called intellig Tex it's in German I'm Sorry by my colleague Yan gor Schneider and he talks about it yeah llms create intelligible textures which we as humans read as explanations and I think that's a very helpful term it's not a text written by someone who's intelligent but we put the intelligence in so our poor little chappie here cannot do multipication yeah so even ma Journey cannot write multiplication so judgments I think cannot be done let's have uh no I'm sorry uh factual decisions have now the um the advantage that at least we can check for it right so a multiplication is a factual decision we could check it and then we can decide whether we want to use AI or any kind of

software for it or not so if you if you want to do a factual decision you can at least check whether a software is able to do it or not and as I said previously with the risk predictions unless your decision now changes the future you can check it in the future and so these are two good categories of decision-making which you can try to use software so but now we'll come back to the more complex question the person asked can Chet GPT help me set prices automatically for products in my store and all of you computer scientists in the room knows know what the person asked for it's called Dynamic pricing right so this user reinvented Dynamic pricing and that is so interesting because it means that llms create accessibility to to coding to a lot of people who have no idea about coding so when I looked it up on Google Scholar there's so many approaches there's more than 100,000 papers on Dynamic pricing with different approaches right so how would a machine

know which of the choices is really the best for you the best choice will depend on specific conditions the code must be correct and meaningful and also the optimization criteria needs to be given by the user so we need to do requirements engineering but the machine will not do that it will just create some code so let's come to one paril that is not often seen if you now have two of these people that want to have a dynamic pricing they don't know that it is called Dynamic pricing the machine creates something and maybe they say something like hauh I want to be cheaper by 1 cent than the lowest price on this item that is a fine strategy unless you have two persons that have the same strategy and this actually happened in Amazon uh a few years ago when they introduced Dynamic pricing and shortly before Christmas they had a lot of products sold for only one cent until the seller noticed so this is really a problem and I think that most

of our citizens are not at all aware of this because the AI is smart and the AI will give me the best solution but it will not even ask all of these questions it will not make them aware of all of these choices they would need to make if they talk to one of us right so here we are at the um judgmental part because we now have different things that need to be understood and weigh so how would we as humans do it if we do a judgment we would try to make a model of the situation how fast do you have your products in the shop how how fast do you need them need to get them out we would collect the data we would do an information selection not all of the data is actually interesting we would determine qualities of each choice so what are the qualities that we want to present Reserve what are the values that we donate um and then we would consider weighing all of these qualities

make a decision and then we would be able to justify with our client why we chose this one and not that one so a prediction of judgment is what the machine does the machine can predict what the average programmer would choose for a dynamic pricing algorithm it would we would give it a selection of a statistical model to do that to learn that we would give it data from the past so this is how Dynamic pricing was done in the last 20 years then we would give it feature engineering we would train it on data it would learn the weight of these features yeah that is somehow similar to the consideration and weighing of qualities then the machine would compute a decision and would not be able to justify it so what what is our contribution that is the human part we can determine the qualities for of each choice individually we can do it here differently than we would do it there and the Machine can never do that the weighing would then also

be individual yes this introduces subjectivity but the machine is not objective either because it depends on all of the things that we design it on the right hand side and we can give a justification and yeah justifications are not perfect sometimes we do not know why we actually choose something but at least we can communicate them in in words that are understandable by another per uh person so my view and this is only a very small part of the argument why I think that llms and other AI systems cannot make judgments in general so situation in which experts do not need to agree in which things need to be weighed qualities need to be weighed if you want to read the longer one you'll have to read my book but the the small part is that if you are in a decision scenario in which you have Jud to judge things machines are not going to help you with the methods we have today they might be later but not with Statistics so in general

I think that factual decisions and risk predictions are quite safe you can try to get help by the machine and for everything else like creativity but if you want to make decisions in a singular setting or a judgmental decision machines are not going to help you computer cannot judge for you so you must judge for yourself and now we need to give all of our citizens this ability we need to help them because computer science is much more than coding but at the moment it looks as if everybody could just ask an llm and it will create the code and that's not true so here is just a part of the things we do as computer scientists we are checking hidden assumptions of a computational method can you actually use it requirements engineering what is your specific situation is that really the best choice for you we can hopefully give you a mathematical proof of correctness if it is an algorithm or at least we can do testing for you and it will be thorough

you might have to look for safety aspects you might have to look for fairness aspects so what needs to be done and I'm happy to have someone from the bmbf here Tina I'm looking at you so what needs to be done clear communication about what llms can and cannot do and I think all of us are working on that some Literacy for Citizens checking Solutions not providing them they only need literacy they do not need to be able to code but they need to be able to read it so basic computer science and data science literacy python coding literacy because that seems to be the choice of the day python is it uh knowledge of quality and fairness measures um also some basic insights into requirements engineering because machines might be better if they know more about the situation so either they need to start asking questions or we need to provide them with more information and very important knowledge of testing strategies and we need to make them aware that they can't judge machines

um to the standards of a 9-year-old yeah so that is not the right way so how to not regret a decision the AI version is only use software where it is technically and reliably able to assist you in your decisions thank you very much

Rise of AI 2027 · Berlin · May 2027

Be in the room for the next one.

Ten years of talks started as conversations in one curated room in Berlin. The next edition brings 300 selected leaders together in May 2027.