Rise of AI Conference 2024 · Berlin · Applied AI Stage

AI Factories at Scale

Thomas Schmidt · Founder, AMBER AI & Data Science Solutions GmbH

At Rise of AI 2024 on the Applied AI Stage, Thomas Schmidt, founder of AMBER AI & Data Science Solutions GmbH, explains what it takes to build and operate large-scale AI infrastructure, or 'AI factories', for enterprise customers. AMBER grew out of Fluid, a company his co-founder, a professor at the Technical University of Munich, started in 2006 as one of Nvidia's earliest partners; it was acquired by US simulation company Altair in 2016, and Schmidt's team spun out as AMBER in 2022 to focus on AI infrastructure. He argues that GPU-based computing saves rather than wastes power compared to legacy CPU servers, illustrating this with a comparison between a 2012 supercomputer and a 2023 Nvidia DGX H100 system delivering more performance at a fraction of the cost and power draw. He outlines the three components of an AI factory, compute, storage and networking, warns that compute needs are growing 256 times every two years since transformers, and describes the shift toward liquid-cooled data centers as GPU racks reach a thousand watts and several tons per system. He closes by previewing an immersion-cooled AI cloud AMBER is building in Germany with partners Itera and Sustainable Metal Cloud.

Key takeaways

  • AMBER grew out of Fluid, founded in 2006 by a TU Munich professor as one of Nvidia's earliest partners, and spun out as a standalone company in 2022.
  • Thomas Schmidt argues GPU-based accelerated computing saves power compared with legacy CPU servers, despite AI's reputation for high energy use.
  • An AI factory rests on three components, GPU compute, AI storage and networking, and customers often underbudget the latter two.
  • Compute demand has grown 256 times every two years since transformers arrived in 2017, up from eight times every two years before.
  • AMBER is building an immersion-cooled AI cloud in Germany with Itera and Sustainable Metal Cloud, cutting power needs by about 50 percent.

Detailed notes

From an early Nvidia partner to AMBER

Schmidt traces AMBER's roots to Fluid, founded in 2006 by his co-founder, a professor at the Technical University of Munich and one of the first people working with Nvidia's CUDA on GPU simulation software; Fluid later also sold Nvidia hardware. Fluid was acquired in 2016 by Altair, a US simulation company. Two years before this talk, Altair decided Nvidia's growing push into software (via Omniverse) made it more of a competitor and asked Schmidt's team to pivot away from Nvidia work; instead, in 2022 the team spun out as AMBER to focus on building AI infrastructure for enterprise, startup and government customers worldwide, mainly in EMEA.

Why accelerated computing saves power, not wastes it

Schmidt compares the 2012 Titan supercluster, 20 petaflops at 8,200 kilowatts and roughly 20,000 GPUs, worth about 130 million dollars today, with the 2023 Nvidia DGX H100, which delivers 32 petaflops using only 10 kilowatts, eight GPUs, and under half a million dollars: more performance in eleven years for a fraction of the cost and power. He argues that converting legacy CPU servers to GPU servers cuts costs and power sharply, citing an example where 960 CPU servers costing 10 million dollars can be replaced by two GPU servers costing 800,000 dollars at five percent of the power requirement, and that training a 1.8 trillion parameter GPT-class model in 90 days will take a quarter of the GPUs and power on Nvidia's upcoming Blackwell GB200 generation compared with the current Hopper generation.

The three pillars of an AI factory

Schmidt describes AI factories as resting on three components: GPU compute nodes, AI storage, and internal networking, typically InfiniBand for customers who want to match Nvidia's own reference architecture, which is why Nvidia acquired Mellanox. He says customers commonly underestimate the budget needed for storage and networking, comparing a cluster with weak storage and networking to a race car fitted with toy wheels. He adds that legacy storage systems are not sufficient for AI workloads and lists vendors including IBM, DDN, NetApp and VAST as options. Citing figures he attributes to Nvidia, he notes compute demand grew eight times every two years before transformers arrived in 2017, and 256 times every two years since, with an expected eight- to twenty-times increase in inference demand as AI moves from training into production.

Cooling, weight and the shift to liquid-cooled data centers

Schmidt says new GPUs draw around 1,000-1,200 watts each, generating more heat than air cooling can handle, forcing a shift to liquid and direct cooling; new systems such as Nvidia's B200-based racks can also weigh about three tons per square meter, more than many existing data center floors can support, so this needs to be planned for in advance. For companies that want a fast, complete solution he describes Nvidia's DGX SuperPOD as a turnkey AI factory bundling compute, networking, storage and Nvidia's Base Command software, while noting that custom-built solutions, for example with Super Micro servers, can be more cost-effective but require more in-house expertise. He stresses cluster-management essentials such as health monitoring, GPU temperature and utilization tracking, and usage reporting and chargeback so organizations can see who is using expensive GPU resources.

An immersion-cooled AI cloud for Europe

Schmidt previews a project with partners Itera and Sustainable Metal Cloud to build an immersion-cooled AI factory, with systems submerged in mineral oil inside shipping containers, cutting power requirements by roughly 50 percent by removing the need for fans. The first containers are expected in Germany between September and October, offered as a sovereign, sustainable managed cloud platform. He closes by arguing Europe has strong AI talent but is roughly a factor of ten behind the US and Asia in the compute resources available to that talent, and frames building local infrastructure as essential to keeping it in the region.

Quotes

Accelerated computing is not eating power, it is saving power, we just need to use it on a large scale.

Thomas Schmidt

Six months lost means six months gained by the others.

Thomas Schmidt

Questions answered in this talk

What are the three key components of an AI factory?

According to Thomas Schmidt at Rise of AI 2024, an AI factory needs GPU compute nodes, high-performance AI storage, and fast internal networking such as InfiniBand, and customers often underestimate the budget needed for the latter two.

Does AI computing use more power than traditional computing?

Schmidt argues the opposite: converting legacy CPU servers to GPU servers can cut power use to about five percent of the original requirement while lowering cost, so accelerated computing saves power at scale rather than wasting it.

Why are data centers moving to liquid cooling for AI?

Schmidt explains that newer GPUs draw around 1,000-1,200 watts each and generate more heat than air cooling can remove, while dense GPU racks can weigh about three tons per square meter, exceeding what many existing data center floors can support.

Mentioned

AI factory, DGX SuperPOD, Base Command, InfiniBand, liquid cooling, immersion cooling, transformers, inference

Keep exploring

More from Thomas Schmidt

More from Rise of AI Conference 2024

More on Enterprise AI & Adoption

Transcript

Auto-generated transcript, not edited. · 4,831 words

Show the full transcript

um thank you all for actually joining me in this talk um I'm very happy to share our experiences we've had in the last few months it has been an amazing journey we've been on in the last few let's say 18 months since we had the big ChatGPT explosion I'd like to call it um and we've been building a few AI factories in our day um could just continue like this who's Amber actually um Amber is in one form one of the oldest Nvidia Partners out in the market it has been founded in 2006 by one of my co-founders uh Thomas iner who has uh who has always been a professor at the Technical University of Munich uh back then it was called fluid and um Thomas has been one of the first people actually working with Cuda and actually developing on gpus um back then a simulation software and that has been so successful that he actually started also selling Nvidia Hardware back then Nvidia was not the PowerHouse it is today um so he

he started very early in in working with Nvidia on on AI infrastructures and gpus uh in 2016 we or fluo back then has been bought by a company um called Al Al is a US simulation company who were very interested in this GPU code and um uh with that we actually got into the into the big Market of AI globally and uh two years ago alter actually decided oh Nvidia is now getting more of a danger to us because of Omniverse because of a lot of development going on with Nvidia and they asked us to maybe switch and do something else internally but um I and the team at at alter we decided that no we think AI is going to be pretty big AI is going to be pretty important so in 20122 uh we decided to found Amber uh and be a standalone company and since then I I think we have been one of the strongest and biggest Partners in building AI infrastructures for for customers and these customers involved really everything

from from Big Enterprise customers to startups to governments uh facilities Etc globally uh mainly in emia but um we see we see developments going all around the world so that's that's us and then what are we actually talking about a little bit of an intro uh where we are at and and why AI factories are getting more and more important and and I'll just jump into 2006 basically when Nvidia released Cuda um and and started this entire wave uh with with GPU coding um and then of course 2012 alexn we like to call the first contact with AI um and since then it has been a rapid rapid development and growth we we see doubling of sizes of models we see doubling of systems of performance um constantly and Nvidia actually saw that the oems back then they weren't really ready for AI and they they didn't believe in it that much so in 2016 Nvidia decided to create their own system now back then it was basically the first AI Factory in a small

form factor called the dgx the djx1 created by Nvidia purposefully for AI nothing else there were no holes barred it was the fastest thing you could do um and they continue to develop this uh this this solution and we'll see more about it in a second but um what what also pushed uh this entire this entire wave is in 2070s when at 2017 when Transformers hit and since Transformers uh hit the market we've just seen a crazy development basically models have been popping up to do all kinds of things and uh this resulted in 2022 the big jet GPT open AI explosion uh since then it is on everyone's mind everyone is working with AI it is in in the mainstream media everybody at least has heard about it and it's on every agenda in every every Enterprise and uh there has been rapid rapid developments in in all kinds of areas which also resulted in gen Ai and gen AI of course is is a big big uh factor in in every AI strategy

nowadays and it's pushing businesses all around the world in in creating value and that also started a whole bunch of startups with with amazing innovative ideas um getting funds as well who need a lot of resources and all of this results in what Jensen calls a new Industrial Revolution um it is a a a very big thing to say a new Revolution the last Revolution proposed propelled Humanity uh into into the era we are in today and Jensen believes that we will face another leap like this with AI and of course this results in a giant market 100 trillion is is what Nvidia is saying and to cover all of this we need AI factories we need to gather data we need AI factories to work with this data to then basically create intelligence and if we if we take a look at the state of AI generative AI adoption globally uh we actually see and this is one of the favorite things uh Jensen says spend more to save more we see that the

more you spend in in Ai and gen AI especially you the more you get back um most companies report 3.5 of a return of in interest um select few actually really know what they're doing and 5% of them actually get up to eight times of their uh investment back estimates go as far as saying that 33% of all g2000 companies will in some way exploit business models to double their monetization potential with Gen AI or gen and of course we see a lot of development in in software development uh via gen and which which results in a lot of software tests being being written by gen and increasing the software code and usability of software so we know why geni is important and now we take a look at at what AI factories look like back then and what they look like today on the left hand side we see 2012 the Titan supercluster 20 P flops it was an amazing number back in the day um and Al it took an amazing amount of

power 8,200 Kow of power to to run this Beast almost 20,000 Nvidia gpus and this system if we calculate it in money today considering inflation would be worth $130 million us that's that's a lot of money right but let's take a look at at what we get nowadays to cover the same amount of performance 2023 the djx h100 based on the hopper GPU uh the current gen of of of Nvidia um even more performance 32 peda flops only takes 10 kilowatt of power that's that's quite a lot less power right that's that's a huge jump down it's only eight gpus way less gpus costs you less than half a million and that's in 11 years that's quite a jump and that actually takes me to a very very important point about AI factories and the the state of AI Computing in general because you see it on the news all the time AI is heating up our planet AI is is uh it's it's taking so much power it's it's it's in everybody everybody's talking

about it data centers they they need more power than almost anything else um that's true AI needs a lot of power but if we actually take a look at what AI can do and What GPU systems especially can do right if we convert old Legacy Tech CPU servers not every workload can be transferred to gpus but those that can be if we would do that we would actually save a huge amount of power so accelerated Computing is not eating power it is saving power we just need to use it on a large scale and we need need to transform all data centers to actually move away from CPU servers to GPU servers so also of course saving a lot of money right 960 CPU servers are expensive that's that's 10 million uh you can you can cover the same for 800k with just two GPU servers and in my opinion way more important with 5% of the power requirement right that's that's a huge saving and I believe honestly um AI is a key way

in solving huge amounts of problems that humanity is facing it will not solve it by itself but it will give us the tools to solve them and um we should not break break this this amazing progress we have right now because we take a look at the power it requires yes it is a lot of power but it is less power than the old Tech and not only is it less power than if if you move from CPUs to gpus if you actually move with the new generations Nvidia is releasing every year or every two years um even then you're saving a lot of power if you if you just take a look at this if you if you if you take a GPT 1.8 trillion um parameter model and want to train it in 90 days if you want to do it with hopper you will need 8,000 gpus 1,000 systems that's that's that's what we're doing right now it will take you 50 me of power that's that's a lot if we want

to reduce this next year's generation of Blackwell the GB 200 with actually 72 gpus per system which will be liquid cooled but more than this in a second it will take a quarter of the power a quarter of the gpus so it is extremely important to actually use the new tech that is being de developed yes it is it is a challenge in keeping up with all of this but please reach out to us there are life cycle ways to to actually um move from the old generations to the new generations and to make sure you always uh keep this in mind and don't forget power is not for free power is expensive especially in Germany right power is really expensive here so this can actually be a massive factor in your business as well um so yeah that's that's where we are 2022 jet GPT explo explosion lot of stuff happened started happening back then then we had experimentation we see new in ative stuff happening daily we see new models almost daily um

just I think it was yesterday right when jgpt or open AI um released their new model and and there's so much happening it's almost hard to to keep up with everything but what's key is we we're still at the beginning of all of this there's a lot of experimentation going on and that's important but what we're facing in the next years will actually be production we'll move away from just training we will move into inference we will move into using these systems that are being or these models that are being trained and some conservative opinions think that um this will mean an increasing need for computational Resources by at least eight times others say by 20 times that's we we're moving so fast it's hard for anyone to keep up and not only is inference going to be a huge thing but I I mentioned before Transformers were actually a key key aspect in the development of AI and if we take a look at at before Transformers were a thing we saw a an

increase of computational resources needed by eight times every two years that was huge that was huge that was a lot but since then it's 256 times every two years that means you need to increase you need to keep up that needs that that means bigger and bigger clusters bigger and bigger AI factories one system is not enough anymore you need to to increase the Clusters you need to realize I how to build them and how to to operate them as well so that's the Triforce of AI as I like to call it for AI Factories at least um AI factories there are basically let's say three key components are making them there is the GPU notes or acceleration notes however you want to call them if it's not gpus and there's the AI storage a lot of our customers think they can just keep on using their Legacy Storage that they have been using for ages does not work and I'll explain why in a second and then there is the internal networking you need

to connect the systems between each other and uh you can of course also use ethernet and ethernet is making huge jumps and it's getting better and better but um we always say if you want to keep up with the big players and that's Nvidia um then use their reference architecture and that's infin band that is why they got melan melanox a few years ago they are developing it rapidly as well as their own GPU nodes um why are these two things on the bottom and GPU noes on top what Nvidia is always saying um building a huge cluster is like building best and fastest race car but if you then skimp out on AI storage on the networking it's like putting tiny toy wheels on your cluster or on your race car and then expecting it to win a a race does not work you need to keep in mind these three aspects all of them and most customers they forget the bottom too they think about these fairly late and don't expect that there

is a pretty hefty amount of budget that you need as well to cover networking and AI storage of course connecting all of these is software and service there is a lot of uh things you can do on the software side nowadays it's very very custom very individual depends on the project the use case what you want to do and of course there's the service and Invidia is also covering a lot in in that area um what we're seeing more and more is that Enterprise grade service is more important than ever for companies to move if they want to move into production right they need to make sure that their cluster and their their entire inference infrastructure is being serviced in a way that um covers their business and and and does not hinder their progress and their development their training and their production of course and then all of this is built on the data center infrastructure itself and as with many other infrastructures we're facing now a a time where the old infrastructure is

not enough anymore we're seeing that these systems require more and more power they require or or generate more and more heat and all of a sudden a aircooled old um uh old data center will probably not cover the next generation of of gpus we're facing you you have to imagine these gpus they suddenly have requirements of power of a th000 watt and more th 1,200 this is an amount of of heat that will get tough to handle physically uh with a air so we we have to move into into liquid cooling and that is something you need to cover in in your data center infrastructure not only that wait these systems they're really really heavy if you see the new um um b200 systems from Nvidia with 72 gpus per W that's three tons of weight on on one square meter um most data centers will not cover this they they will they will if you ask them to to put your data your your rack in there it will just break through the floor

that will not be very conclusive to your training and inference um so you need to you need to figure this out early right you're not going to build a data center from today until tomorrow this is something you need to plan early to cover this basically and this is basically if you if you if you want if you say I don't want to I don't want to get into it I I just want to buy something and this is the djx superod the AI Factory TurnKey solution of Nvidia it has everything you need it has the compute nodes now with the old um um djx h100 upper generation uh with with all of the networking included with AI storage included with the software base command of Nvidia included with basically the rxs everything is in there if you say I have money and I want to start this is a pretty this is the fastest way to start um and it is it is simple it is something that Nvidia has built themselves over over

their Seline cluster the EOS cluster everything is built on this reference architecture um it's not to say that you have to go with dgx it is simply the easiest and simplest way uh our friends of Super Micro are here with us for example we can build a custom solution with with their servers as well um and and oftentimes it is the more cost-effective solution let's say but um it is it it takes more effort and it takes more experience on the side of the customer even though we will help a lot and we will cover a lot but generally you want to start you want to start quick you want to scale that's also very important you can scale this easily um you can you can go with the superpod it's basically it's it's it's like ordering one product from us then you will get this and even if you have this this super pod or an ictory custom build uh you will need to ask yourself is your AI stack actually optimized um you

will need to ask yourself if you can keep up with the monthly framework churn you will need to ask yourself if you have the people who can who can operate this this cluster um you will need to check if you're using all of the best performance libraries if all of everything is is up to date and running and what's also very important is you will need to check what you want to do yourself what you want to build yourself or what you actually can buy as a service I I don't have the time unfortunately to cover everything that's possible possible or everything that Nvidia is doing nowadays but generally they have a lot of Frameworks they have Nims we've already heard about them today there is a lot of pre-trained and pre-built software a lot of pre-trained and pre-built services that you can use which you don't have to use yourself so you have to think about your use case you have to think about your regulations your requirements and then decide what is actually

what makes sense you don't want to waste time the the business and the market is moving So Fast 6 months lost means 6 months gained by the others now let's take a look at a cluster software and I'll I try to keep this short and simple because it's it's a huge huge thing um and and very individual and you can go into very very different directions but I'm trying to cover the the main things as I see them and uh on top you see a few of the of the buzzword around it that that we see a lot it's Nvidia Enterprise which is basically their entire Suite uh of software Solutions and and service which covers I would say everything you need um and then the key factor around this is base command this is their cluster management um tool which we will take a look at in a second and I also like I really like to work with runi um which now has been acquired by Nvidia but um it it is is

a great workload management tool but the key things is basically infrastructure provisioning workload management and resource monitoring and as I said I have only 5 minutes remaining so I'll be quick about this um why is it so important because basically you want to make sure that all the images on the different servers are being managed that the cluster can be configured remotely um that you can share a common software stack that you can of course also post installation um actually make sure that you can configure all of these this different systems make sure the networking can be configured user authentification GPU settings BMC power settings Etc there are a lot of things that the admins need to take take into account um then you also want to make sure that the utilization of the of the system is overall good and that you want to make sure that the health of the system is overall good so that's why um base command for example gives you the option to to monitor the health of the

entire cluster uh Cloud integration is something we see more and more and it's is more and more important um we see a lot of customers who have the idea or or follow something which I call uh own the base and Spike into the cloud or it's not just me calling it this but a lot of people and and basically the idea is to own your on Prem cluster yourself and for if you for example for two three months or a few weeks see that you need more resources uh to be able to spike into the cloud and and to use the resources there which of course makes it has to auto start start and stop uh when you when you stop you don't want to to run these notes forever in the in the cloud and forget them which is going to be very expensive ET Etc um next then of course kubernetes integration is really important um base command for example also supports multiple operators like run ey Jupiter Prometheus Etc um Jupiter integration

is really important key factor of all of this is you want to make the life of your data scientists as easy as possible um data scientists are very expensive their time is is valuable and you don't want to to annoy them um any more than than you have to so basically everything that makes their life easier makes it easier for them to create value for for your company and then of course uh really important Health monitoring and and Remediation um basically the the the idea here is that with the right software tools you can make sure that any problems that may occur are being recognized early so you don't want to have your cluster break down you want to to face these problems early on before you have any problems in production especially or for example also in training right you don't want your training to stop and then suddenly lose weeks of time and and power and energy um because you you don't have a checkpoint or whatever so it's very important to uh

monitor the health of your cluster as well which base command for example does and um the Nvidia service then helps you actually to cover all of these different things right it's very very complicated to run a cluster efficiently you need to think about all all these things all the time then if you take a a step deeper into Health monitoring as well uh it gets very important these These gpus are also very expensive so you want to make sure that the temperature of the gpus are are is being well capped you want to make sure that the fan speeds or for example the utilization of the of the gpus is is running as you as you want it and the memory Etc that everything is running well and what we're seeing more and more of is actually reporting and chargeback so you as an organization want to make sure that you see who's actually using all of your valuable resources and what is he using them for is is it is it just running randomly

is he is he using it for important jobs what does it cost me to basically uh support these data scientist teams and uh so you want you want a software Tool uh kit that that basically helps you with the reporting and charge pack um yeah as I said earlier storage really important for AI factories you can't just use Legacy Storage uh very very storage agnostic there are a lot of different players out there this is for example a a system by IBM which we love to work with there's also ddn net app vast voco there are a lot of different players uh and it takes takes time and effort to figure out the perfect system for you and your use case um i' I'd like to share this uh these slides afterwards uh but it's basically very important that you have a high performance um storage system to make sure that you're you're getting the most out of your cluster um quick Trends we're facing in the last few months we are seeing specialized AI

networks and we're seeing very very large models that's basically the two developments we're seeing um what we are expecting or Nvidia is also expecting though is that these large models will take care of more and more of the specialized um of the specialized things that that are happening and uh what this also means though is that these large models will have bigger and bigger parameters longer and longer context which means more memory is needed which means the inference system needs to be bigger on the other side as well so Nvidia is expecting that in the future you will actually need these huge cluster systems um to to run the inference as well right now there is still a bit of a divide between training gpus and inference gpus but it's moving more into into one unified and as I mentioned before um data Senter requirements in the future direct cooling is the way to go there is no way around this in the in the future um we need to get ready for it and

then I had a oh can I jump back I had like a 4minute video here and I I knew I wouldn't have the time to play it so if you actually want to see the AI Factory of the future Amber and its partner itera and their partner sustainable metal Cloud um are working on an immersive cooled solution which is really impressive and reduces the amount of power requirement required Again by roughly 50% why because you don't need to have fans running you have all of these systems you see it on the right hand side they're actually immersed in in mineral oil and they are um therefore you you you can get you can move the heat better and um you need less power to cool them and the one picture on the left hand side shouldn't be there but basically it looks like a a shipping container uh with all of these systems being liid being immersed in the mineral oil and um we're getting the first containers in Germany in the next few months

by September October hopefully uh and this will be a managed Cloud platform which you can use which will be Sovereign here in Europe which is very important it will be sustainable and it will it will uh hopefully free a lot of companies from the shackles of the hyperscalers and actually um help us build compute resources here in Germany in Europe where we desperately need them a last word from from my side um there is a lot of work to be done we see globally Europe has amazing talent and this Talent needs resources it needs gpus and we're by a factor of 10 behind the us behind Asia Etc there is a lot of a lot of work we we need to focus on to enable this talent and to keep the talent here and this is one way to do it thank you for your time

Rise of AI 2027 · Berlin · May 2027

Be in the room for the next one.

Ten years of talks started as conversations in one curated room in Berlin. The next edition brings 300 selected leaders together in May 2027.