So, please welcome on stage with me Christian Winterfeldt from Dell who is going to give his speech about can AI ever be green? How we built the AI factory sustainably. >> Thank you. Thank you. >> Ah, give him a round of applause guys. I >> Yes. So, you know, back in the machine room, back to the basics of a data center. Um, yeah. So imagine um right now somewhere in the world a data center is being built that is consuming more energy than a small town not to run hospitals not to run its trans transportation system just to create neuro uh tokens from a neuronet network and and that's the scale of infrastructure revolution we are all in right now and everybody of you is is aware of this in uh revolution.
So the clicker is not working. So maybe we check on this one. No. Yeah, perfect. Now it's working. Um, so imagine and this is exactly the situation we see and you have this discussion nearly every day. This is this is becoming boardroom discussion. This is becoming capacity planning. This has become product planning. How to handle that? And I'm leading the data center business for Dell in Germany, Austria, Switzerland. and I'm talking with lots of executives about these questions. The first is how to scale AI within an organization without destroying the economics because just just consuming AI out of the cloud can be very expensive and this is what we see with customers who started this journey two years ago using always latest models and I had a good conversation with one CIO who was telling me hey Christian AI workloads are working different than other cloud workloads we experienced in the past.
Usually we throw lots of data, lots of uh workloads into the cloud and the more we used the cheaper it became. And with AI it's it's the opposite. So the more user are using this the more advanced the company is becoming in terms of AI the token bill is exploding. So they need to to have different options. So how to go forward? Are we building our own data center? If you do that, how can we scale this and without without breaking the economics and the sustainability? And this is exactly what I want to talk about. How should a proper AI factory look like today in order to be sustainable? And this is kind of weird having AI and sustainability in one center as it sentence, but there's there is a a big um part of engineering around this, a big part of operational efficiency.
And this is what I want to talk exactly about to give you a scale of the energy consumption of data centers right now. um this year and this is one study amongst many studies data centers will consume more than 2,000 terawatt hours by the end of this year and this number has doubled over the last three years and this is double uh this is half of the energy North America is using so we are talking about immense um amount of energy we are consuming with AI data centers but let's first start with a mental shift how data centers are operating in the past. In the past, we we squeezed enormous value out of CPU infrastructure.
So that was very simple. We overs subscribe CPUs. We shared memories. We believed in statistical leverages. And this is why we cloud provider became very famous and could make these very aggressive offers on infrastructure servers. This is why your own data center became super efficient because you overs subscribed your infrastructure and in case somebody will use it maybe you could add something but this economies are are breaking right now and this is what also cloud providers I'm talking to are challenged because you can't this playbook is breaking you can't easily overs subscribe a GPU because GPU memory has a hard boundary because GPU bandwidth has a b hard boundary and if you over subscribe this AI users will immediately notice because latency is is going up or throughput is dropping.
So you need to have other concepts of of running these environments and this is why the token builds are are exploding right now because the the the old economics are not working anymore. So in old times over subscription was your friend today orchestration is your friend and this is what you need to do. So you need to look at it more not not just just putting more expensive service into the data center. You have to see the full stack as an IAI factory. And such an AI factory, this is not just hardware. This is a fine production system where you have data flowing and on the one side models are trained, tuned, evaluated and deployed.
inferencing is running continuously and hopefully a good business outcome is generated by these AI factories and along that part of these pipeline of compute storage networking coding um scheduling the telemetry around this all the models uh who are running on this environment the data uh processing everything needs to run as a yeah I would say as a symphony needs to work together and this is because you don't want to uh run as many GPUs as possible. You want to have a useful output per watt. And this is the the key um element here, a useful output per watt and not what per workload. And if you keep this principle in mind, every design architecture becomes clear.
So let's start with the the physical layer, the energy and power first. um a modern data center right now a more high dense AI rack is consuming more energy than we ever thought a couple of years ago. So the these numbers are pretty old right now. These are 30 to 170 kilowatts. The latest GTC announcement from from Jensen um he announced Vera Rubin and one rack is consuming 300 650 kilowatts. So this is immensive amount of energy and this would have sought absorbed couple of years ago. So you can run dozens of households with that energy. So having that in mind data center architecture becomes crucial. So how you design your facility, how to design your cooling, how you design all the things which are running around this.
And what we see as a big trend is of course liquid cooling. Today 99% of all data centers are still air cooled and but we see on the other side a 20% growth on liquid cooling because yeah it's more efficient to get the energy to get the heat out of the systems. um you reduce the chiller load, you reduce the power to the fans, you are a enable um water cooling um with higher temperatures helps you to reuse um the energy for something else. And on the other hand, you know, you really need to look at the whole system on a rack design. So not just putting servers, storage, networking into a data center, look at full design.
Design your AI factory on a rack scale and then deploy it on rack scale. This is how big organizations are doing this and this is also how small organizations are doing this. And one example here. So um today we we have the perception in Germany AI is not going that fast forward as we all think that that our our organizations enterprise organizations are bit leggers on the on this uh on on this the thing I'm I'm running the data center business for D technology in that area and I can say that we have more and more enterprise customers and small mid-market customers who are running their own AI for sustainability reasons but more for uh for governance reasons for um uh sovereignty reasons on premise and we're adding every quarter a tripledigit number of organizations who are keeping their AI on premise.
So moving this back from the cloud making this on premise and running in an AI factory. This this is true in Germany but we don't do this um with the huge investments. So you hear from from big US companies from APJ companies who are investing hundreds of millions but Germans customers don't do that they're investing in small portions half a million here a million there and expanding these environments and optimizing the whole stack uh on a ra scale as as much as possible to get the maximum output per watt in this environment. So um the real challenge but is not to to keep your um GPU cold. Um the biggest problem is when your GPU is getting idle and this is what we see um very often.
So most organization they orchestrate their infrastructure um and also some some providers we saw that they organiz organize their GPU infrastructure in a way that feels operationally safe. So some clusters for testing, some clusters for for team A, some clusters for team B and you see that the utilization of these environments is not very good. So you see on one hand um training is is running at 30%, inferencing pool is running at 90%, and and some um capacity is absolutely unused. So this is something you definitely need to change because you can oversubscribe, you need to orchestrate. So we are going from stic pools to multi-tenant um scheduling. So what what are you doing? You have a centralized queue with all your demand and you have a centralized scheduleuler like like Nvidia's um run AI.
This is what we are doing together with Nvidia. Three years ago we created the first AI factory together with Nvidia. The whole stake created together with Dell Technologies to to make this happen. So the centraluler is is deciding where to put which workloads. So uh inferencing runs first um training runs when um um once in between where when there are some gaps and then the scheduleuler is saying okay maybe if there are some some bad jobs we are just filling the gaps and this is increasing your utilization of your AI cluster up to 90% and higher. So there you're really getting the the the efficiency this. So this is one examples where we are using um Nvidia run AI and um where a centralized scheduleuler is placing with multi-tenants um scheduling workloads where they belong.
So you see on the one side there are pools are static and you still have three um slots but you still can't use them. with a um multi- um tenant scheduling you are able to just fit another workload, another workload, another workload. And if you see this on an organizational scale, so not every department for itself on an organizational scale AI is becoming more efficient. So um the critical insight here sustainability metrics and um economics they are aligned. So if both get worth. So your sustainability story is going down and uh your economics around your data center is going worse. So utilization and sustainability they're going hand in hand with multiGPU multi-tenency. It's not pretending to have more GPUs than you actually have.
It's about um precision scheduling and this is exactly what an AI factory is doing. The next thing and the the title is um from the GPU to the to the power meter and this is about measuring measuring everything and this seems to be standard. Yes, you should do that. You should measure what happens in your data center but many data center they don't do that. So you need to measure everything within a system. So how much energy are you are using? What are your CO2 footprint per system per workload? Measure all these things. There are hundreds of data points within a system. You you can measure with Dell. For instance, you can use IDRA um open manage enterprise where you can have lots of these parameters for a system or for environment for 10,000 units and more.
And this is exactly helping you where are the top power offloaders, where are the zombie racks which are still using energy but effectively doing nothing. And this is what you need to do in a data center in order to get your electricity bill and your sustainability story. So your electricity bill down and your sustainability story up. So with it you can optimize without this is just a guess. But what are we doing with all that heat? Um the question is if we are treating heat as waste or heat as a resource. So in climates like here in uh in Germany the the solution is obvious. You use it for residential heating. You use it for district heating, you use it for industrial production or for agriculture production.
In some areas, we are supporting agriculture production with AI data centers. And in warmer climates, the solution is is more critical. But in fact, you can use it for desordinization and and other factory um usage. And this is what what exactly is is going to do. If you are in an area with water and power stress, you shouldn't design data centers as just big boxes with giant chillers. You should put this into an ecosystem. Um because with um um with u um without energy reuse, power uses effectiveness is just a um is just half of the equation. So you really need to have a plan how to reuse that energy for your own um company or for the district you are putting the data center in.
And now we we talked a lot about the infrastructure the physics around this. And now this is the the biggest challenge we are facing the biggest energy saver we we can use. So the the smartest what is the what you are never using. So um the chips says armor race gets gets a lots of attention in the industry but not always buying the newest and latest GPUs is the solution for making your ST more efficient. Optimizing um the software stack and we are talking here about the software letter. There are so many areas where you can really improve your AI environment. So you can go to optimization and pruning the way how you're really using the models.
The first uh decision is always which user which model to use. So don't throw uh a 70 billion or bigger model um to to each problem. Sometimes you can really use a very tiny model to solve a a problem within an manufacturing floor if you're creating a digital twin uh beside a manufacturing plant. And this needs to be a very small or it could be a very small small model and make these decision first. And next thing is a more efficient um retrieval argumented regeneration adaptive um batching so that you are uh putting together more prompts into into one batch and makes so your environment even more um more efficient rightsiz context windows or you have a flexible model serving so that you need uh you're using agents to serve just the model which is needed to to do the job.
So there are many areas to improve the architecture here and we have many examples where we helped organizations to just do exactly that step. So here was one customer who was moving the whole environment back from a from a cloud to an on-remise data center and the GPU footprint would have looked like 12 GPUs and we looked at the caching behavior. We looked at the model serving behavior. We looked all all the way down to these um to this optimization and a footprint of 12 GPUs become actually one GPU. We reduce the GPU print by one/10. This this is actually what we saw and this is actually what most customers can expect if they optimize the architecture because usually architectures from a PC perspective are only copied to the next stage to the next stage.
If you're looking at the full software letter, you can really optimize your your footprint in that way. And we as Dell, we have more than 800 engineers who are actually working on that. How to improving model behavior, how to improving models, how to fine-tune, how to do this optimization and pruning. And here's some some optimization and pruning examples here. So you download the papers, see what DA is doing. This is public domain. We share how customers can optimize their stack in that way. The other thing is also um having GPU um um keeping utilized. So the worst thing is if your GPU is waiting for data. So you also have to look at your data path in in the in in your organization.
So um if your GPU is idling because there are no data, they're still consuming 30 40 50% of their um of their energy budget. So the worst thing is to have your most important um resource uh not utilized and um Nvidia is helping here. So in with an AI factory you have Nvidia AI enterprise software which is helping to keep the GPUs utilized using other STS but in in in fact you need to have a look at the good old storage ting. So this is back to the machine room. This is back to how we handle our data. So in the end you really have to look at where to put which data because you couldn't put all your data on the latest and fastest storage.
It's too expensive. It's consuming more most of the energy. So having good old storage tuning so MVME or flash first for all these um active inferencing for KV caches um scale out object for um for training data um for um for for batch jobs you are doing and um yeah the lowest the the cheapest storage tier you should use for for archiving for logging etc. Sounds sounds easy but it's not done in every environment. So this is also helping organizations to make that stack even more efficient. So if if you are planning to run a data center or if you are already running a data center scaling AI workloads these are the five things you should do over the next 18 months.
So first measure everything measure everything within what you're doing the out output per watt um GPU utilization CPU utilization measure everything set clear KPIs and optimize uh this environment and you can easily use AI doing that. So in in the operation phase right now then build a reference pot. Build a reference pot you can easily copy and expand in that way. This reference pot should be in a factory on a ra scale. It should be um water cooled preferently. So having a having a water cooled environment and start with this reference point. So that makes your operation um in the end even more effective. Then move to dynamic scheduling. Um use the tools you already have.
If you're using an AI factory with with an 8, the job is very easy. Take some of the workloads and start with um dynamic scheduling and promise or and see that no um user experience is compromised with that. So force and the most important part of this take your smartest software engineers in the company set clear KPIs and start to optimize the full stack. So really looking at how's the architecture working right now, which models I'm using, what's my model serving strategy within the organization. Set clear KPIs and you will see that you will reduce your GPU need um massively. This is what we see at customers right now. And therefore we see that German customers um especially they're optimizing this like hell.
We are going back after after half a year. Hey, you purchased a big eightway GPU system. You are you're doing this and this. We talked about these use cases. Yeah. are we doing all these uh we have optimized here so the ones who got it are super efficient and then energy costs in Germany are not not counting that much and they are comparing their previous um cloud bill with what they're doing right now and there there is a very positive equation there's a very positive outcome if you are really going down that path and not just throwing more GPUs to a problem having a companywide view and working on this and then finally align the two teams so if you have an infrastructure team in your organization If you have a sustainability team in your organization, hey, let let them speak together.
Let let's align them together on the same goals. And if you've done this, you are far beyond greenwashing. You are far beyond having just a sustainability story. Your organization is working towards that trend. So coming back to my initial question, can AI ever be green? Honestly, AI will always consume a serious amount of energy. This this is this is for fact. So for the next 10 uh 10 years AI will be the single biggest new energy consumer absolutely given. So we will see a massive increase of energy consumption on AI data centers. So the question is not whether AI data centers are going to be built. The question is are we building them intelligently and using all this optimization along the stack and really looking at the valuable outcome.
And if you're then um having a sustainable and a a sovereign infrastructure in your own data center where you're um in control on your own data improving um um the the economics um around this and then you have a real AI infrastructure which becomes more effective more useful and hopefully is getting producing more useful output for the company for the society on than what you're putting in on energy. So, it was very short overview on how the machine room should look like. I'm looking forward to lots of questions from you. Thank you very much. >> Thank you, Christian. And you already gave the prompt. Does anybody have a question from the audience? Ah, here's one.
A long way for for the microphone. So uh first of all thank you for your presentation. Um my question is you mentioned liquid cooling of service. Yeah >> but you don't mention immers immersion cooling service. Um they are more efficient I think and also produce not so much hot air. >> Yeah. >> Uh why you don't mention them? >> So we we have a look at this. We have a very close look into this one. Right now the industry trend is going towards liquid cooling. Um direct liquid cooling on a chip. More components are cooled. Immersive cooling is even more expensive if if you're purchasing this. So the RORI is not there. But I guarantee you if if Michael Dell sees an opportunity there the he will ex exactly jump on this.
But right now we don't see the economics around this. So, a good liquid cooling um with a um yeah heat heat exchanger in the back. Having this uh engineered to your residential um cooling is even more effective today than um than immersive cooling. So, we have a close look at this one. >> Thank you. >> Okay, we have time for more questions. While the microphone travels, I have a question for you. Christian, you said implementing some of these things is fairly easy and your cloud bill looks better. Why are not more people doing it? Like what is the discussion you have with the clients that say okay I don't want to do this? >> It's um we are lagging a strategy in an organization.
So AI is is like a strategy house. You should start with a purpose what you are doing good as an organization and then top down you should decide where to start where to implement AI and um we see lots of pilots. So the last three years was the was were pilots and you you are not building an AI infrastructure just for department or just for an idea you just easily consume um yeah something from the cloud because it's it's easy it's fail fail fast it's it's available it's it's great there are great models but if you are scaling if you are really excelling your AI journey then you have to shift um your your gears and this is what we see right now and as I said um a triple digit number of customers is joining uh this strategy every quarter in Germany.
So we are fairly on a good way that more and more customers go to a sovereign sovereign part. But these these two um parts will still coexist. So there is still a hybrid way of of of uh of doing this. So some models will um still use from the public cloud and some will uh run on premise and if you have a proper strategy how to handle this um then the the overall solution becomes better. Okay. So in the pilot already think of if this works what's like the next step >> how to scale in the future. >> Got it. Cool. So our question is over there. >> Yeah. >> Hi this is Robert from Nvidia.
Thank you for your talk. Really cool. We for fully support like squeezing every bit of performance out of the GPU. So but I mean you said it's easy. I think it's not so easy. what do you recommend uh organizations who want to go on that path and don't have the capabilities yet to really squeeze this uh this performance out of the GPUs yet? >> Yes. So um start with an AI factory and and you are right there there are tens of thousands of combination possibilities if you put all these different tools and and softwares and and and infrastructure pieces together. Start with a AI factory which is a reference architecture which It's a proven proven stack work with Nvidia with the enterprise stack.
So most of the knowledge is already there and um if you don't have your own data center go to a collocation where you already have these these environments. So there there are many ways to go there and over the last two years um we sold more than 3,000 AI factories globally. So there's lots of knowledge how to deploying this. So go go with a big partner. uh doesn't need to be Dell could be but go with a big partner together with Nvidia and and build these build these factories and then you have a pretty much good good approach to get to the 80 90% very quickly time for one more question. Any more questions? Right.
Then maybe one more from me. So you said, you know, it's good for the costs, like you can reduce costs and you can help the environment, but why are you dedicating yourself to this personally? What is it beyond those things that makes you feel like we should all care? >> Yeah. What happens with my data? So it's it's it's basically the um the data I'm using. So I'm um if if you are in control of your data so AI should be there where your data are you should you are not copying all your general ledger all your most serious uh critical information to somewhere to a to a nx data center where GCP AWS or somewhere has has its cloud so because you you want to take more and more advantage out of what you are doing yourself and put this within your own um governance so it's that it's not leaving your data center, that it's not leaving your control, and uh that you're not copying all the data.
So, it's it's about latency, it's about cost, about sovereignty, it's about governance. So, there are more reasons to to think about having a different strategy. >> Thank you so much. Your hand of applause,