At Rise of AI 2024 in Berlin, Nestor Maslej, Research Manager and AI Index Editor-in-Chief at Stanford's Institute for Human-Centered AI, joins Thomas Neubert, founder of the Transatlantic AI eXchange, in a fireside chat that condenses findings from the 2024 AI Index Report. Maslej walks through headline trends: AI systems now exceed human performance on many benchmark tests, training costs for frontier models have risen sharply (GPT-4 an estimated 80 million dollars, Gemini 191 million dollars), and only a handful of companies can afford to build such systems. He argues Europe is not out of the running on large language models even without matching the biggest training budgets, since there is value in making models more efficient. He cites evidence that AI tools speed up work by 26 to 73 percent across studies, that a McKinsey survey found 42 percent of managers report cost decreases and 59 percent report revenue increases from AI, and that Google's AlphaMissense model raised annotation coverage of human protein mutations from 0.1 percent to 89 percent. He closes on survey data showing rising public nervousness about AI in every country sampled, most pronounced in Germany, the US and other Western nations.
Key takeaways
- Training GPT-4 cost an estimated 80 million dollars and Google's Gemini 191 million dollars, per the AI Index.
- Maslej argues Europe can still compete on AI efficiency and latency research without matching the largest training budgets.
- Workers with AI tools finished tasks 26 to 73 percent faster than workers without them across the studies cited.
- Google's AlphaMissense model raised classified human protein missense mutations from 0.1 percent to 89 percent.
- Ipsos survey data shows AI-related public nervousness rose in every country sampled between 2022 and 2023.
Detailed notes
What the AI Index Report is
Nestor Maslej explains that the AI Index was created in 2017 by a committee of AI figures including Jack Clark (a co-founder of Anthropic), the economist Erik Brynjolfsson, and James Manyika, senior vice president of research at Google, who wanted a resource that could tell a compelling, data-driven story of where AI stood. The report, now around 500 pages, covers what AI systems can do, economic trends, which countries are leading on investment, and trends in responsible AI, drawing on a mix of partner data vendors and data collected by the Index team itself.
Benchmark progress and the rising cost of frontier models
Maslej points to Midjourney's evolution from its hyperrealistic Harry Potter prompt in the original V1 (released February 2022) to its most recent version as one illustration of rapid progress, and notes that AI systems are now exceeding human capabilities on many benchmark tests. He attributes much of this to scaling: the 2017 Transformer paper introduced the architecture now used by nearly every large language model, and since GPT-3 (around 2020) it has been clear that feeding models more data improves performance. As a result, the Index estimates GPT-4 cost 80 million dollars to train and Google's Gemini 191 million dollars, putting frontier-model training out of reach for all but a few companies. Asked by Neubert whether this is "game over" for Europe on large language models, Maslej says no: there is still value in building more efficient, lower-latency models, research questions that do not require 200 million dollars to pursue, even if matching the largest state-of-the-art systems is difficult without comparable funding.
Productivity gains and the risk of blind trust in AI
Maslej presents a cross-study comparison showing workers with access to AI tools completed tasks 26 to 73 percent faster than a no-AI baseline. In a separate consulting study, three groups did the same task: one without AI, one told the AI was unreliable and prone to hallucination, and one told the AI was reliable. Both AI-assisted groups outperformed the no-AI group, but the group told the AI was flawed outperformed the group told it was good, because the latter reportedly stopped checking the system's output. Maslej draws the business lesson that companies integrating AI tools must keep humans checking the work rather than trusting the output on autopilot.
AI in science: AlphaMissense
As an example of AI's impact beyond chatbots, Maslej describes AlphaMissense, a Google model released in the past year that classifies missense mutations, alterations of a human protein. Of an estimated 75 million such mutations, human annotators had classified only 0.1 percent before AlphaMissense; the model has since classified 89 percent.
Business returns and declining public trust
Citing a McKinsey survey of managers integrating AI into business functions, Maslej highlights that, averaged across activities, 42 percent reported cost decreases and 59 percent reported revenue increases. He then turns to an Ipsos survey run before and after the release of ChatGPT (2022 to 2023) asking whether AI products make people nervous: nervousness rose in every country sampled, with the lowest optimism, meaning the highest pessimism, in Germany, the United States and other Western nations, while countries such as Indonesia, Thailand and Mexico remained more bullish. Neubert adds that Germany scores well on AI research but, in his view, lags the United States at commercializing research into products, a point Maslej agrees the Index's investment data also suggests.
Quotes
It's still very important, it's still very essential that you keep humans in the loop with how you use AI.
Nestor Maslej
I'm just a humble academic, I write a report and make a lot of fancy charts, I don't know anything about running a business.
Nestor Maslej
I wouldn't say so. I still think there's a lot of value in training these kinds of systems.
Nestor Maslej
Questions answered in this talk
Is Europe out of the race to build large language models?
According to Nestor Maslej at Rise of AI 2024, no: while few players can match the training budgets of the largest US models, there is still a lot of value in making models more efficient and reducing latency, research that does not require hundreds of millions of dollars.
How much did it cost to train GPT-4 and Gemini?
According to the 2024 AI Index Report as presented by Nestor Maslej at Rise of AI 2024, the Index estimates GPT-4 cost about 80 million dollars to train and Google's Gemini about 191 million dollars.
Does using AI tools make employees faster at their work?
According to Nestor Maslej at Rise of AI 2024, a cross-study comparison in the AI Index Report found workers with access to AI tools completed tasks 26 to 73 percent faster than workers without them.