um thank you all for actually joining me in this talk um I'm very happy to share our experiences we've had in the last few months it has been an amazing journey we've been on in the last few let's say 18 months since we had the big ChatGPT explosion I'd like to call it um and we've been building a few AI factories in our day um could just continue like this who's Amber actually um Amber is in one form one of the oldest Nvidia Partners out in the market it has been founded in 2006 by one of my co-founders uh Thomas iner who has uh who has always been a professor at the Technical University of Munich uh back then it was called fluid and um Thomas has been one of the first people actually working with Cuda and actually developing on gpus um back then a simulation software and that has been so successful that he actually started also selling Nvidia Hardware back then Nvidia was not the PowerHouse it is today um so he
he started very early in in working with Nvidia on on AI infrastructures and gpus uh in 2016 we or fluo back then has been bought by a company um called Al Al is a US simulation company who were very interested in this GPU code and um uh with that we actually got into the into the big Market of AI globally and uh two years ago alter actually decided oh Nvidia is now getting more of a danger to us because of Omniverse because of a lot of development going on with Nvidia and they asked us to maybe switch and do something else internally but um I and the team at at alter we decided that no we think AI is going to be pretty big AI is going to be pretty important so in 20122 uh we decided to found Amber uh and be a standalone company and since then I I think we have been one of the strongest and biggest Partners in building AI infrastructures for for customers and these customers involved really everything
from from Big Enterprise customers to startups to governments uh facilities Etc globally uh mainly in emia but um we see we see developments going all around the world so that's that's us and then what are we actually talking about a little bit of an intro uh where we are at and and why AI factories are getting more and more important and and I'll just jump into 2006 basically when Nvidia released Cuda um and and started this entire wave uh with with GPU coding um and then of course 2012 alexn we like to call the first contact with AI um and since then it has been a rapid rapid development and growth we we see doubling of sizes of models we see doubling of systems of performance um constantly and Nvidia actually saw that the oems back then they weren't really ready for AI and they they didn't believe in it that much so in 2016 Nvidia decided to create their own system now back then it was basically the first AI Factory in a small
form factor called the dgx the djx1 created by Nvidia purposefully for AI nothing else there were no holes barred it was the fastest thing you could do um and they continue to develop this uh this this solution and we'll see more about it in a second but um what what also pushed uh this entire this entire wave is in 2070s when at 2017 when Transformers hit and since Transformers uh hit the market we've just seen a crazy development basically models have been popping up to do all kinds of things and uh this resulted in 2022 the big jet GPT open AI explosion uh since then it is on everyone's mind everyone is working with AI it is in in the mainstream media everybody at least has heard about it and it's on every agenda in every every Enterprise and uh there has been rapid rapid developments in in all kinds of areas which also resulted in gen Ai and gen AI of course is is a big big uh factor in in every AI strategy
nowadays and it's pushing businesses all around the world in in creating value and that also started a whole bunch of startups with with amazing innovative ideas um getting funds as well who need a lot of resources and all of this results in what Jensen calls a new Industrial Revolution um it is a a a very big thing to say a new Revolution the last Revolution proposed propelled Humanity uh into into the era we are in today and Jensen believes that we will face another leap like this with AI and of course this results in a giant market 100 trillion is is what Nvidia is saying and to cover all of this we need AI factories we need to gather data we need AI factories to work with this data to then basically create intelligence and if we if we take a look at the state of AI generative AI adoption globally uh we actually see and this is one of the favorite things uh Jensen says spend more to save more we see that the
more you spend in in Ai and gen AI especially you the more you get back um most companies report 3.5 of a return of in interest um select few actually really know what they're doing and 5% of them actually get up to eight times of their uh investment back estimates go as far as saying that 33% of all g2000 companies will in some way exploit business models to double their monetization potential with Gen AI or gen and of course we see a lot of development in in software development uh via gen and which which results in a lot of software tests being being written by gen and increasing the software code and usability of software so we know why geni is important and now we take a look at at what AI factories look like back then and what they look like today on the left hand side we see 2012 the Titan supercluster 20 P flops it was an amazing number back in the day um and Al it took an amazing amount of
power 8,200 Kow of power to to run this Beast almost 20,000 Nvidia gpus and this system if we calculate it in money today considering inflation would be worth $130 million us that's that's a lot of money right but let's take a look at at what we get nowadays to cover the same amount of performance 2023 the djx h100 based on the hopper GPU uh the current gen of of of Nvidia um even more performance 32 peda flops only takes 10 kilowatt of power that's that's quite a lot less power right that's that's a huge jump down it's only eight gpus way less gpus costs you less than half a million and that's in 11 years that's quite a jump and that actually takes me to a very very important point about AI factories and the the state of AI Computing in general because you see it on the news all the time AI is heating up our planet AI is is uh it's it's taking so much power it's it's it's in everybody everybody's talking
about it data centers they they need more power than almost anything else um that's true AI needs a lot of power but if we actually take a look at what AI can do and What GPU systems especially can do right if we convert old Legacy Tech CPU servers not every workload can be transferred to gpus but those that can be if we would do that we would actually save a huge amount of power so accelerated Computing is not eating power it is saving power we just need to use it on a large scale and we need need to transform all data centers to actually move away from CPU servers to GPU servers so also of course saving a lot of money right 960 CPU servers are expensive that's that's 10 million uh you can you can cover the same for 800k with just two GPU servers and in my opinion way more important with 5% of the power requirement right that's that's a huge saving and I believe honestly um AI is a key way
in solving huge amounts of problems that humanity is facing it will not solve it by itself but it will give us the tools to solve them and um we should not break break this this amazing progress we have right now because we take a look at the power it requires yes it is a lot of power but it is less power than the old Tech and not only is it less power than if if you move from CPUs to gpus if you actually move with the new generations Nvidia is releasing every year or every two years um even then you're saving a lot of power if you if you just take a look at this if you if you if you take a GPT 1.8 trillion um parameter model and want to train it in 90 days if you want to do it with hopper you will need 8,000 gpus 1,000 systems that's that's that's what we're doing right now it will take you 50 me of power that's that's a lot if we want
to reduce this next year's generation of Blackwell the GB 200 with actually 72 gpus per system which will be liquid cooled but more than this in a second it will take a quarter of the power a quarter of the gpus so it is extremely important to actually use the new tech that is being de developed yes it is it is a challenge in keeping up with all of this but please reach out to us there are life cycle ways to to actually um move from the old generations to the new generations and to make sure you always uh keep this in mind and don't forget power is not for free power is expensive especially in Germany right power is really expensive here so this can actually be a massive factor in your business as well um so yeah that's that's where we are 2022 jet GPT explo explosion lot of stuff happened started happening back then then we had experimentation we see new in ative stuff happening daily we see new models almost daily um
just I think it was yesterday right when jgpt or open AI um released their new model and and there's so much happening it's almost hard to to keep up with everything but what's key is we we're still at the beginning of all of this there's a lot of experimentation going on and that's important but what we're facing in the next years will actually be production we'll move away from just training we will move into inference we will move into using these systems that are being or these models that are being trained and some conservative opinions think that um this will mean an increasing need for computational Resources by at least eight times others say by 20 times that's we we're moving so fast it's hard for anyone to keep up and not only is inference going to be a huge thing but I I mentioned before Transformers were actually a key key aspect in the development of AI and if we take a look at at before Transformers were a thing we saw a an
increase of computational resources needed by eight times every two years that was huge that was huge that was a lot but since then it's 256 times every two years that means you need to increase you need to keep up that needs that that means bigger and bigger clusters bigger and bigger AI factories one system is not enough anymore you need to to increase the Clusters you need to realize I how to build them and how to to operate them as well so that's the Triforce of AI as I like to call it for AI Factories at least um AI factories there are basically let's say three key components are making them there is the GPU notes or acceleration notes however you want to call them if it's not gpus and there's the AI storage a lot of our customers think they can just keep on using their Legacy Storage that they have been using for ages does not work and I'll explain why in a second and then there is the internal networking you need
to connect the systems between each other and uh you can of course also use ethernet and ethernet is making huge jumps and it's getting better and better but um we always say if you want to keep up with the big players and that's Nvidia um then use their reference architecture and that's infin band that is why they got melan melanox a few years ago they are developing it rapidly as well as their own GPU nodes um why are these two things on the bottom and GPU noes on top what Nvidia is always saying um building a huge cluster is like building best and fastest race car but if you then skimp out on AI storage on the networking it's like putting tiny toy wheels on your cluster or on your race car and then expecting it to win a a race does not work you need to keep in mind these three aspects all of them and most customers they forget the bottom too they think about these fairly late and don't expect that there
is a pretty hefty amount of budget that you need as well to cover networking and AI storage of course connecting all of these is software and service there is a lot of uh things you can do on the software side nowadays it's very very custom very individual depends on the project the use case what you want to do and of course there's the service and Invidia is also covering a lot in in that area um what we're seeing more and more is that Enterprise grade service is more important than ever for companies to move if they want to move into production right they need to make sure that their cluster and their their entire inference infrastructure is being serviced in a way that um covers their business and and and does not hinder their progress and their development their training and their production of course and then all of this is built on the data center infrastructure itself and as with many other infrastructures we're facing now a a time where the old infrastructure is
not enough anymore we're seeing that these systems require more and more power they require or or generate more and more heat and all of a sudden a aircooled old um uh old data center will probably not cover the next generation of of gpus we're facing you you have to imagine these gpus they suddenly have requirements of power of a th000 watt and more th 1,200 this is an amount of of heat that will get tough to handle physically uh with a air so we we have to move into into liquid cooling and that is something you need to cover in in your data center infrastructure not only that wait these systems they're really really heavy if you see the new um um b200 systems from Nvidia with 72 gpus per W that's three tons of weight on on one square meter um most data centers will not cover this they they will they will if you ask them to to put your data your your rack in there it will just break through the floor
that will not be very conclusive to your training and inference um so you need to you need to figure this out early right you're not going to build a data center from today until tomorrow this is something you need to plan early to cover this basically and this is basically if you if you if you want if you say I don't want to I don't want to get into it I I just want to buy something and this is the djx superod the AI Factory TurnKey solution of Nvidia it has everything you need it has the compute nodes now with the old um um djx h100 upper generation uh with with all of the networking included with AI storage included with the software base command of Nvidia included with basically the rxs everything is in there if you say I have money and I want to start this is a pretty this is the fastest way to start um and it is it is simple it is something that Nvidia has built themselves over over
their Seline cluster the EOS cluster everything is built on this reference architecture um it's not to say that you have to go with dgx it is simply the easiest and simplest way uh our friends of Super Micro are here with us for example we can build a custom solution with with their servers as well um and and oftentimes it is the more cost-effective solution let's say but um it is it it takes more effort and it takes more experience on the side of the customer even though we will help a lot and we will cover a lot but generally you want to start you want to start quick you want to scale that's also very important you can scale this easily um you can you can go with the superpod it's basically it's it's it's like ordering one product from us then you will get this and even if you have this this super pod or an ictory custom build uh you will need to ask yourself is your AI stack actually optimized um you
will need to ask yourself if you can keep up with the monthly framework churn you will need to ask yourself if you have the people who can who can operate this this cluster um you will need to check if you're using all of the best performance libraries if all of everything is is up to date and running and what's also very important is you will need to check what you want to do yourself what you want to build yourself or what you actually can buy as a service I I don't have the time unfortunately to cover everything that's possible possible or everything that Nvidia is doing nowadays but generally they have a lot of Frameworks they have Nims we've already heard about them today there is a lot of pre-trained and pre-built software a lot of pre-trained and pre-built services that you can use which you don't have to use yourself so you have to think about your use case you have to think about your regulations your requirements and then decide what is actually
what makes sense you don't want to waste time the the business and the market is moving So Fast 6 months lost means 6 months gained by the others now let's take a look at a cluster software and I'll I try to keep this short and simple because it's it's a huge huge thing um and and very individual and you can go into very very different directions but I'm trying to cover the the main things as I see them and uh on top you see a few of the of the buzzword around it that that we see a lot it's Nvidia Enterprise which is basically their entire Suite uh of software Solutions and and service which covers I would say everything you need um and then the key factor around this is base command this is their cluster management um tool which we will take a look at in a second and I also like I really like to work with runi um which now has been acquired by Nvidia but um it it is is
a great workload management tool but the key things is basically infrastructure provisioning workload management and resource monitoring and as I said I have only 5 minutes remaining so I'll be quick about this um why is it so important because basically you want to make sure that all the images on the different servers are being managed that the cluster can be configured remotely um that you can share a common software stack that you can of course also post installation um actually make sure that you can configure all of these this different systems make sure the networking can be configured user authentification GPU settings BMC power settings Etc there are a lot of things that the admins need to take take into account um then you also want to make sure that the utilization of the of the system is overall good and that you want to make sure that the health of the system is overall good so that's why um base command for example gives you the option to to monitor the health of the
entire cluster uh Cloud integration is something we see more and more and it's is more and more important um we see a lot of customers who have the idea or or follow something which I call uh own the base and Spike into the cloud or it's not just me calling it this but a lot of people and and basically the idea is to own your on Prem cluster yourself and for if you for example for two three months or a few weeks see that you need more resources uh to be able to spike into the cloud and and to use the resources there which of course makes it has to auto start start and stop uh when you when you stop you don't want to to run these notes forever in the in the cloud and forget them which is going to be very expensive ET Etc um next then of course kubernetes integration is really important um base command for example also supports multiple operators like run ey Jupiter Prometheus Etc um Jupiter integration
is really important key factor of all of this is you want to make the life of your data scientists as easy as possible um data scientists are very expensive their time is is valuable and you don't want to to annoy them um any more than than you have to so basically everything that makes their life easier makes it easier for them to create value for for your company and then of course uh really important Health monitoring and and Remediation um basically the the the idea here is that with the right software tools you can make sure that any problems that may occur are being recognized early so you don't want to have your cluster break down you want to to face these problems early on before you have any problems in production especially or for example also in training right you don't want your training to stop and then suddenly lose weeks of time and and power and energy um because you you don't have a checkpoint or whatever so it's very important to uh
monitor the health of your cluster as well which base command for example does and um the Nvidia service then helps you actually to cover all of these different things right it's very very complicated to run a cluster efficiently you need to think about all all these things all the time then if you take a a step deeper into Health monitoring as well uh it gets very important these These gpus are also very expensive so you want to make sure that the temperature of the gpus are are is being well capped you want to make sure that the fan speeds or for example the utilization of the of the gpus is is running as you as you want it and the memory Etc that everything is running well and what we're seeing more and more of is actually reporting and chargeback so you as an organization want to make sure that you see who's actually using all of your valuable resources and what is he using them for is is it is it just running randomly
is he is he using it for important jobs what does it cost me to basically uh support these data scientist teams and uh so you want you want a software Tool uh kit that that basically helps you with the reporting and charge pack um yeah as I said earlier storage really important for AI factories you can't just use Legacy Storage uh very very storage agnostic there are a lot of different players out there this is for example a a system by IBM which we love to work with there's also ddn net app vast voco there are a lot of different players uh and it takes takes time and effort to figure out the perfect system for you and your use case um i' I'd like to share this uh these slides afterwards uh but it's basically very important that you have a high performance um storage system to make sure that you're you're getting the most out of your cluster um quick Trends we're facing in the last few months we are seeing specialized AI
networks and we're seeing very very large models that's basically the two developments we're seeing um what we are expecting or Nvidia is also expecting though is that these large models will take care of more and more of the specialized um of the specialized things that that are happening and uh what this also means though is that these large models will have bigger and bigger parameters longer and longer context which means more memory is needed which means the inference system needs to be bigger on the other side as well so Nvidia is expecting that in the future you will actually need these huge cluster systems um to to run the inference as well right now there is still a bit of a divide between training gpus and inference gpus but it's moving more into into one unified and as I mentioned before um data Senter requirements in the future direct cooling is the way to go there is no way around this in the in the future um we need to get ready for it and
then I had a oh can I jump back I had like a 4minute video here and I I knew I wouldn't have the time to play it so if you actually want to see the AI Factory of the future Amber and its partner itera and their partner sustainable metal Cloud um are working on an immersive cooled solution which is really impressive and reduces the amount of power requirement required Again by roughly 50% why because you don't need to have fans running you have all of these systems you see it on the right hand side they're actually immersed in in mineral oil and they are um therefore you you you can get you can move the heat better and um you need less power to cool them and the one picture on the left hand side shouldn't be there but basically it looks like a a shipping container uh with all of these systems being liid being immersed in the mineral oil and um we're getting the first containers in Germany in the next few months
by September October hopefully uh and this will be a managed Cloud platform which you can use which will be Sovereign here in Europe which is very important it will be sustainable and it will it will uh hopefully free a lot of companies from the shackles of the hyperscalers and actually um help us build compute resources here in Germany in Europe where we desperately need them a last word from from my side um there is a lot of work to be done we see globally Europe has amazing talent and this Talent needs resources it needs gpus and we're by a factor of 10 behind the us behind Asia Etc there is a lot of a lot of work we we need to focus on to enable this talent and to keep the talent here and this is one way to do it thank you for your time