At Rise of AI 2024 in Berlin, Prof. Dr. Katharina Zweig of RPTU Kaiserslautern-Landau argues that machines can reliably help with only some kinds of decisions, not all of them. She distinguishes four decision types: factual decisions, risk predictions, judgments and singular decisions, and argues AI can, in principle, be checked and trusted for the first two, since their correctness can be verified, but not for judgments, where qualities must be weighed subjectively and justified in words. She demonstrates that ChatGPT 3.5 cannot reliably perform long multiplication despite producing plausible step-by-step explanations, arguing a single failed test is enough to show a system cannot be trusted with a task in general. Using a real case of a user asking ChatGPT to build dynamic pricing for a shop, including an Amazon incident where competing pricing algorithms sold products for one cent, she shows how large language models generate functioning-looking code without asking the requirements questions a human expert would raise. Her conclusion: citizens need basic computer science and AI literacy to check AI-generated solutions rather than trust them outright, since judgment decisions still require human weighing and justification.
Key takeaways
- Katharina Zweig splits decisions into four types: factual, risk prediction, judgment and singular, only the first two of which are reliably checkable.
- ChatGPT 3.5 fails at long multiplication on larger numbers despite giving a plausible step-by-step explanation, showing explanations do not prove correctness.
- A single failed test is enough to show a machine cannot be trusted with a task in general, unlike a human who can be said to 'still learn' from one mistake.
- An Amazon dynamic-pricing incident, where competing algorithms drove a product's price to one cent, illustrates the requirements questions an LLM never asks.
- Zweig argues judgment decisions, which require weighing qualities and giving justifications, cannot be delegated to today's statistical AI systems.
Detailed notes
An 8-year-old's decision procedure
Zweig opens with a childhood story: told by her mother to say goodbye to moving neighbors or risk regretting it later, she devised a decision procedure as an 8-year-old: collect available data, list possible decisions, think through consequences, then decide and live without regret. She uses this to introduce the idea that most real decisions cannot use all available data or fully map every consequence, which is why people might want machine help.
Four types of decisions
Zweig categorizes decisions, drawn from her recent book, into four scenarios. Factual decisions and risk predictions are both checkable: for factual decisions, experts can verify the same conclusion intersubjectively; for risk predictions, such as creditworthiness, the outcome can in principle be checked once the future unfolds. Judgment decisions, a concept she borrows from Kahneman, Sibony and Sunstein's book 'Noise', are ones where experts may disagree but not arbitrarily much, such as grading a university exam. Singular decisions, such as those made during the pandemic, lack historical data entirely, meaning AI has nothing to learn from and cannot help.
Testing whether ChatGPT can do factual tasks: long multiplication
Responding to a forum question asking whether ChatGPT can set dynamic prices for a store, Zweig first tests a simpler factual task: whether ChatGPT 3.5 can perform long multiplication. It explains its steps clearly and gets small numbers right, but produces incorrect results for larger numbers despite a plausible-looking explanation. She argues that unlike a child who makes an occasional mistake but 'can still do it', software that fails even once on a well-defined task cannot be trusted to perform it reliably, because there is no way to know in advance which cases will fail. She cites a colleague's term for LLM output as 'intelligible textures', text that reads like an explanation without reflecting genuine understanding of the procedure.
Dynamic pricing and the limits of AI judgment
Zweig identifies the forum user's real request as dynamic pricing, a well-studied area with more than 100,000 academic papers, and argues an LLM cannot know which existing approach fits a specific business without requirements engineering that the user never provided and the model never asked for. She cites an Amazon incident, shortly before Christmas, in which competing dynamic-pricing algorithms each undercut the other by one cent, driving product prices down to one cent until a seller noticed. She contrasts human judgment, which selects and weighs qualities, justifies choices in words, and adapts case by case, with what a machine actually does: predicting what an average programmer's algorithm would output, based on statistical training on past data, without being able to justify the result.
Conclusion: what citizens need to learn
Zweig concludes that factual decisions and risk predictions are reasonably safe to delegate to AI because they can be checked, but judgment and singular decisions cannot be delegated with current statistical methods. She lists what computer scientists actually contribute beyond coding: checking hidden assumptions, requirements engineering, correctness proofs or testing, and checks for safety and fairness. Her recommendation, addressed partly to a representative of the German Federal Ministry of Education and Research (BMBF) in the audience, is to give citizens basic computer science and data science literacy, including Python reading skills, knowledge of quality and fairness measures, and testing strategies, so they can check AI-generated solutions rather than judge machines by the standard of a 9-year-old who merely explains a procedure.
Quotes
A single fail test is enough to show that a machine is not in general able to do the tested task.
Katharina Zweig
Computer cannot judge for you, so you must judge for yourself.
Katharina Zweig
An explanation by an LLM does not mean they know how to do it, they just know how to create the text that feels like an explanation.
Katharina Zweig
Questions answered in this talk
Can ChatGPT reliably do multiplication, according to Katharina Zweig?
According to Katharina Zweig at Rise of AI 2024, ChatGPT 3.5 gives plausible step-by-step explanations for long multiplication but produces incorrect results for larger numbers, and since large language models cannot have learned exact multiplication from examples alone, she argues they cannot do it reliably regardless of how many examples they see.
What are the four types of decisions Katharina Zweig identifies?
According to Katharina Zweig at Rise of AI 2024, decisions fall into factual decisions and risk predictions, both checkable against a verifiable outcome, judgment decisions, where experts may disagree but not arbitrarily, and singular decisions with no historical precedent, which AI cannot support at all.
Why does Katharina Zweig say AI cannot make judgment decisions?
According to Katharina Zweig at Rise of AI 2024, judgment decisions require selecting and individually weighing qualities and then justifying the choice in words, something current statistical AI systems can only approximate by predicting an average past decision, without being able to justify it.