At Rise of AI 2024 on the Applied AI Stage, Thomas Schmidt, founder of AMBER AI & Data Science Solutions GmbH, explains what it takes to build and operate large-scale AI infrastructure, or 'AI factories', for enterprise customers. AMBER grew out of Fluid, a company his co-founder, a professor at the Technical University of Munich, started in 2006 as one of Nvidia's earliest partners; it was acquired by US simulation company Altair in 2016, and Schmidt's team spun out as AMBER in 2022 to focus on AI infrastructure. He argues that GPU-based computing saves rather than wastes power compared to legacy CPU servers, illustrating this with a comparison between a 2012 supercomputer and a 2023 Nvidia DGX H100 system delivering more performance at a fraction of the cost and power draw. He outlines the three components of an AI factory, compute, storage and networking, warns that compute needs are growing 256 times every two years since transformers, and describes the shift toward liquid-cooled data centers as GPU racks reach a thousand watts and several tons per system. He closes by previewing an immersion-cooled AI cloud AMBER is building in Germany with partners Itera and Sustainable Metal Cloud.
Key takeaways
- AMBER grew out of Fluid, founded in 2006 by a TU Munich professor as one of Nvidia's earliest partners, and spun out as a standalone company in 2022.
- Thomas Schmidt argues GPU-based accelerated computing saves power compared with legacy CPU servers, despite AI's reputation for high energy use.
- An AI factory rests on three components, GPU compute, AI storage and networking, and customers often underbudget the latter two.
- Compute demand has grown 256 times every two years since transformers arrived in 2017, up from eight times every two years before.
- AMBER is building an immersion-cooled AI cloud in Germany with Itera and Sustainable Metal Cloud, cutting power needs by about 50 percent.
Detailed notes
From an early Nvidia partner to AMBER
Schmidt traces AMBER's roots to Fluid, founded in 2006 by his co-founder, a professor at the Technical University of Munich and one of the first people working with Nvidia's CUDA on GPU simulation software; Fluid later also sold Nvidia hardware. Fluid was acquired in 2016 by Altair, a US simulation company. Two years before this talk, Altair decided Nvidia's growing push into software (via Omniverse) made it more of a competitor and asked Schmidt's team to pivot away from Nvidia work; instead, in 2022 the team spun out as AMBER to focus on building AI infrastructure for enterprise, startup and government customers worldwide, mainly in EMEA.
Why accelerated computing saves power, not wastes it
Schmidt compares the 2012 Titan supercluster, 20 petaflops at 8,200 kilowatts and roughly 20,000 GPUs, worth about 130 million dollars today, with the 2023 Nvidia DGX H100, which delivers 32 petaflops using only 10 kilowatts, eight GPUs, and under half a million dollars: more performance in eleven years for a fraction of the cost and power. He argues that converting legacy CPU servers to GPU servers cuts costs and power sharply, citing an example where 960 CPU servers costing 10 million dollars can be replaced by two GPU servers costing 800,000 dollars at five percent of the power requirement, and that training a 1.8 trillion parameter GPT-class model in 90 days will take a quarter of the GPUs and power on Nvidia's upcoming Blackwell GB200 generation compared with the current Hopper generation.
The three pillars of an AI factory
Schmidt describes AI factories as resting on three components: GPU compute nodes, AI storage, and internal networking, typically InfiniBand for customers who want to match Nvidia's own reference architecture, which is why Nvidia acquired Mellanox. He says customers commonly underestimate the budget needed for storage and networking, comparing a cluster with weak storage and networking to a race car fitted with toy wheels. He adds that legacy storage systems are not sufficient for AI workloads and lists vendors including IBM, DDN, NetApp and VAST as options. Citing figures he attributes to Nvidia, he notes compute demand grew eight times every two years before transformers arrived in 2017, and 256 times every two years since, with an expected eight- to twenty-times increase in inference demand as AI moves from training into production.
Cooling, weight and the shift to liquid-cooled data centers
Schmidt says new GPUs draw around 1,000-1,200 watts each, generating more heat than air cooling can handle, forcing a shift to liquid and direct cooling; new systems such as Nvidia's B200-based racks can also weigh about three tons per square meter, more than many existing data center floors can support, so this needs to be planned for in advance. For companies that want a fast, complete solution he describes Nvidia's DGX SuperPOD as a turnkey AI factory bundling compute, networking, storage and Nvidia's Base Command software, while noting that custom-built solutions, for example with Super Micro servers, can be more cost-effective but require more in-house expertise. He stresses cluster-management essentials such as health monitoring, GPU temperature and utilization tracking, and usage reporting and chargeback so organizations can see who is using expensive GPU resources.
An immersion-cooled AI cloud for Europe
Schmidt previews a project with partners Itera and Sustainable Metal Cloud to build an immersion-cooled AI factory, with systems submerged in mineral oil inside shipping containers, cutting power requirements by roughly 50 percent by removing the need for fans. The first containers are expected in Germany between September and October, offered as a sovereign, sustainable managed cloud platform. He closes by arguing Europe has strong AI talent but is roughly a factor of ten behind the US and Asia in the compute resources available to that talent, and frames building local infrastructure as essential to keeping it in the region.
Quotes
Accelerated computing is not eating power, it is saving power, we just need to use it on a large scale.
Thomas Schmidt
Six months lost means six months gained by the others.
Thomas Schmidt
Questions answered in this talk
What are the three key components of an AI factory?
According to Thomas Schmidt at Rise of AI 2024, an AI factory needs GPU compute nodes, high-performance AI storage, and fast internal networking such as InfiniBand, and customers often underestimate the budget needed for the latter two.
Does AI computing use more power than traditional computing?
Schmidt argues the opposite: converting legacy CPU servers to GPU servers can cut power use to about five percent of the original requirement while lowering cost, so accelerated computing saves power at scale rather than wasting it.
Why are data centers moving to liquid cooling for AI?
Schmidt explains that newer GPUs draw around 1,000-1,200 watts each and generate more heat than air cooling can remove, while dense GPU racks can weigh about three tons per square meter, exceeding what many existing data center floors can support.