Um, so I will welcome three guests on stage next for our next session and this is before we head into lunch. So bear with us and stay awake. Bring all your questions and we will um look at one of the most demanding areas of AI deployment next bringing agentic AI into classified and highly sensitive environments. So on stage Dennis Gisa from HPE, Robert Friedri from Nvidia and Anna Gruntla from deep set will discuss how we can be deploying a Gentic NI in classified domains. Come on stage. Still no chairs wanted. Welcome them with me. >> Hi. You need the kicky. [cheering] Well done. Hi. >> Yeah. Hi everyone. uh thanks for staying even though lunch is already served outside.
Um we appreciate that and uh it probably shows the importance of agentic AI right it's the talk of the town and uh one thing that we often discuss in in our daily lives is how we can bring that to confidential data to maybe even classify data to to secure environments and uh this is what uh today's going to be about um since it's the applied AI stage we also brought some demo of a of a real use case um so yeah stay with us and uh we we have some really tangible things uh in the end. So um yeah, Robert, take it away. >> So to get us into the topic, um let's start. What gave me a blueprint of this idea?
Uh and >> can we have the slides, please? >> There we are. >> Okay. Thank you. Um so what gave me uh the blueprint or a vision for this um for agentic in in government? Uh the agentic uh state was a great white paper for me. It it was a a blueprint of what this could look like and um the question was how how can this become a reality? And uh and then came Davos. And uh Davos gave me the idea of a of a commitment to this kind of sketched out plan. And uh what we see now is the first large scale projects that arose from this plan. Uh one I want to mention is the um Agentic AI hub.
Uh Anna will will also talk a little bit more about this uh this later where um local government and startups are kind of meshed together to uh create this uh uh to implement this uh this vision and also Spark as a as another initiative is uh is not just a pilot it's a it's a large scale uh implementation of of this vision and uh we want to bring this vision also to classified data um and it's certainly more difficult and uh Anna will tell us a little bit more about the challenges um that needs to be solved to get to get there. >> Thank you very much Robert. I'm coming into the middle now. Um so I'm from deep set.
We are an AI company that was founded in 2018. So we're building AI systems since 2018. But most people know us for Haste, our open source framework. It is the um leading European uh open source framework for AI orchestration and we work a lot with customers and a lot in the public sector and defense space and um when we talk with them about Agentic AI or AI solutions in general what we hear is yes but is it BSI certified? Can we really do it? Is it safe? Where is the model hosted? What model are you actually using? how can we scale it if we do it on prem and um a lot of these questions um came to us and we thought okay we are AI orchestration but we have to solve actually the layers below and this is why we started working together um with HPE Nvidia and seconet to come up with an a kind of an out ofthe-box solution that we want to present today that answers those challenges because without answering
those challenges it's nice to have a genetic AI but you don't have it where it maybe needs to be. And with that, I hand it over to you. >> Thank you. Yeah, as Anna said, let's let's look at the team that got together here um to to create this uh end to- end solution. So, first of all, um we have Sikunet as as Anna mentioned, they provide the BSI certified security um in in that um architecture we built. They um provide domain separation, encryption on a hardware level. So we basically can work in really secure and um disconnected environments. We from HPE provide the data management and the uh AI computing infrastructure on uh yeah basically to to work with the data to analyze the data to build um the the AI use cases on on dedicated uh and and selfhosted uh infrastructure basically.
Of course, uh, Nvidia GPUs are are also built into that. Uh, also DPUs, but Nvidia provides more than that. They also, um, provide a lot of, um, software that we use to really optimize local inference and and local usage of of AI tools. Um uh we we'll go into more detail about that but it's for example uh the Nvidia inference microservices where we have uh optimized uh large language models for running locally on on the infrastructure and then um we have deep set with with haststack with the orchestration control plane where we can build but also evaluate and and govern our AI applications um within that stack. So if we put it all together and that's what we built what we are presenting today um we create something um let's call it a a private cloud for AI within a secure environment.
So you can imagine it like a complete solution that features um everything you need to build your own cloud stack basically but within a specified environment. there's the infrastructure, there's a software to use and and make available that um infrastructure components um and also the AI models and the tools to create use cases all within one predefined pre-installed box basically that's the green box on on this uh on this diagram here and uh what the customers need to bring is the data because obviously it's mostly confidential data maybe even classified data that only they have within their premises. So we bring that private cloud to their premises. They connect the data and within a couple of days there users um have the opportunity to create use cases to build AI applications within the safe environment and it's all enclosed um within the sikunet um sena uh cloud security layer.
So it's it's a really encrypted and and separated uh domain of um where the the data lives basically and and resides. Now if we go a little bit more into detail what's included here we have um I mentioned that a couple of times of course the infrastructure uh with optimized compute for um AI workloads with GPUs uh in in uh included of course but also storage if there's uh you know the data that that's processed locally should be um somehow stored there as well. there's Nvidia GPUs and also networking gear to connect all these things internally, right? Um, but on top of that, there's a very cool software layer as well. So, from a users perspective, what the users get is like an app store that they are used from their daily private life basically and they get the same experience or similar experience in this secure environment as well.
So tools like Haststack from DeepSet is available within a a really easy to use environment. They can get there with their browser and start building applications really quickly. We'll show that in the demo in a couple of minutes. And um yeah, on top of that um we we have uh this uh what we call AI essentials um software that um allows not only to uh use applications from that app store but also include um connections to data that is stored somewhere. So um users don't even need to create a copy of the data that they want to work with. they just connect to the systems where the data is actually store that and and they can get going and use that.
And um there's also um as I already mentioned um the Nvidia software stack with for example blueprints um for applications or the uh inferencing microservices that give us really optimized um versions of um the top-notch LLMs for example to to use in our um private use cases. So Robert um if you want to talk about uh the Nvidia um software Yeah. >> So I I just don't want to talk about the model itself like every everybody's talking about models. uh a lot of talk in this conference is about sovereignity and uh it's defined as a kind of scalable or adjustable way of managing risk basically and um a key component of this sovereignty is open source uh and um uh and maybe to share a story from a recent exchange with the Ukrainian um IT of the Ukrainian army is there's the they basically have to follow the strictest way of or the strictest way of of sovereignty.
Um so the talk was about like licensing and what they uh requested also from their providers is uh non-revocable licenses or uh and basically uh open source is the best way of addressing this because there is no license and um Nvidia is also committed to this idea of um of non-revocable licenses and uh and open source. That's why most of our software most of our and all of our models are open source. Um the latest uh one of our models is is Neimotron. It's not just one model. It's a it's actually a family of model. And uh I want to also define a little bit more what we understand as open source because oftenly it's understand as open weights.
So I can download the model and and be good. Uh so this is also part of it. Yes. Um also our models have have open weights and uh it's it's completely transparent but there's more to it from my understanding. uh so I need to go I need to be able to go back and understand this model and in order to go back I also need the training data uh which is why uh Nvidia released trillions of tokens uh of training data for their uh uh which the model was trained with and not just that it's uh it also contains post-training uh data reinforcement learning it contains security traces and uh basically every uh data that goes into the model to uh to arrive where until we release it.
Uh and in order to then work with these models, you also need uh libraries. And these libraries of course need to be open, available and transparent so that you can uh securely work with them. Um uh and to to make it easier, we also release training scripts and and build this into entire blueprints of um like highscale enterprise processes like video search or or rag use cases. And all of this is provided uh from Nvidia open source. And um I brought a little like uh chart with me. It's probably hard to read, but you should check it out. Uh it's uh from artificial analysis uh and they are independently um checking um all models and all kinds of uh AIS for with regard to their uh benchmarks.
It's uh it's from my uh perspective a better way of of measuring model performance. Uh and in that case it's the it's the openness index. I haven't really seen that a lot in uh from from others. And here Nvidia models uh score very high next to others uh other models which are sometimes also a derivative of our models which is uh which is great. And in the end um it's not just this is not just about models and our models. Uh we should think about this as uh as an as more an open source discussion uh and taking this to a model perspective. It gives us uh models that are the right fit basically for uh for for the intended use uh gives us full control over the model and its behavior and uh we heard that a lot today.
uh it also provides uh cost-efficient way of implementing AI and uh uh this implementation of course this is not our focus and uh that's why we again handing over to Anna to show us how this can be implemented also cost efficiently >> perfect thank you so much yeah [clears throat] the best infrastructure is nothing without load right so you need applications and only if we build applications we can um bring AI into action and we can really see the benefits that we've been talking about today already. And um with Haststack um we bring basically the Lego building blocks to build these AI pipelines or these AI systems. And I want to start with a little story.
So um in 2018 when DeepSet was founded, we started directly we were bootstrapped and we started directly working with customers and in the beginning we were training models. So uh fun fact, we trained the first non-English bird model. It's still somewhere on hugging phase with a nice number of downloads. Um and when implementing them at our customers site so one of the first customers was actually Airbus in Tulus. um we realized wait it's not only the model you need a lot around you need the data you need to do something with your data then maybe the model comes into play and then there's an output and this output needs to be fed into something via RP so we realized okay actually this logic that you have have to build pipelines is something that needs to go out to the world and this is when we released haststack and since um I think that was then in 2009 end of 2019 and since then we have continuously building and improving haststack together with our community and
uh in 2021 we built uh an AI platform on top um which makes it very easy to build and monitor AI applications you can evaluate them very well um so basically for enterprises if you have multiple use cases you want to have a governance you're going want to have some sort of control logging monitoring everything that maybe you all I'm sure you all know about. Um, so this is AI orchestration. You need to bring different things together such as models, data, etc. You need to orchestrate it and you need to control it somehow. And um this logic um helps us to build these applications that I've been talking about and they sit basically on top on top of uh all the stack that we we have heard of and um I want to um I have one use case with me of a multi- aent system because with haste you can build a lot of systems right you can build simple rag applications you can build aic rag you can build individual agents multi- aent
systems whatever you want because of its modular uh architecture basically but today I wanted to bring an example um from the from the classified domain and this is a multi- aent system that we called Aida um it is a AI decision agents I have anonymized it in a way um that we can talk about it um today but basically um it is multiple agents that work together to give you um a so-called laga build or a situation analys is and what these three uh different agents do is there is one observation agent and this has a specific skill set. This observation agent works constantly in the background. It's like your your person your observation person that you have employed and is collecting information what is happening in the field all the time with um and there's different um information that is collected.
It's weather data. It's sensor data. It is data about the situation itself. And this information is processed. And when the agent uh thinks this is relevant, it would feed it back into the um other agent, the assessment agent. The assessment uh agent also works in the background. The assessment agent has a certain plan what should happen and it compares the plan with what the reality shows and compares those plans. And if there is a deviation then it would ping the interaction agent and the interaction agent we call it like that. You can call it whatever but um is the agent that directly interacts with an operator and get the operator then gets the relevant information to come to a um yeah to be able to make a decision.
basically pulls together all this information and then you can have real time intelligence basically and can um decide what to do next. What is very very important in those cases is to always have a human in the loop. I would not automate it at this time to be very honest. I think it's very important to always have a person um yeah being there and um making the decision. But how does it um look like uh in practice? I brought a um a little mock a little uh demo for you. I think I have to click twice. Yeah. To start it. >> Yes. >> To start it. >> Okay. So, this is how it would look like then um in in the tooling.
Obviously, it could be via app directly integrated into your system of choice. So, um everything is open and then you see the different agents that collect information. you see um what they do and in the end in the chat window but it can be also any other window you would see um a yeah a suggestion what to do or some information that is relevant for you as a human and you as a human would only see the chat interface everything else just happens in the background um and what we learned about building agent is that um it's super important to give them context um especially uh when it comes to such topics. So you have to make sure you give them enough context uh that they are able um to bring the right data up and that they don't run into loopholes.
Um and um yeah this this is a little demo and now I want to hand over to you. How would we find this now? >> Yeah let's uh let's put these pieces together. Um so um from this um specific tool like like hstack going on step back to the platform that we're working on the where the infrastructure is um basically abstracted for the users so they um will only need to go to a um website with their browser and have all these tools available. Right? That was what we what we promised. So um what you see on the screen here is the interface of that um private cloud for AI um that we were talking about and um you can tell from the menu already that there's um basically everything you need for an entire um life cycle of AI products from data engineering to um training of models if you want to go there but also of course um uh for uh implementing applications using their models and um deploying these models um and
and um I have put together a really quick video to show you the the most important um features how how it will work. So first of all we need to connect some data right. So we go to the data sources and we have um options to connect to um structured data um like databases, SQLbased or even um you know like iceberg these modern um data formats. We can connect these sources. We can give them access rights. So we don't have to build copies of data. We can um basically virtually tap into the original um data sources. We can do the same for uh object stores based on the S3 protocol and also integrate um u yeah file storage.
Now um when we deploy the models we have this model catalog that you see here with models from um Nvidia AI enterprise for example. Uh if we deploy a Neotron model we get an open AI API compatible um interface to that. So applications that can talk to OpenAI in most cases are able to talk to these um without any um any uh yeah other u programming effort. Um we can also scale these models. If we have lots of load we can automatically spin up multiple instances and in the end we can go to the tools and frameworks tab. This is what I called the app store in the beginning, right? Where we have integrated um lots of tools um out of the box that we think uh are useful to get started.
But as you can see in the top right, there's a a little button that says import framework because you can go there and import your own applications as well. So basically everything that runs on Kubernetes um we are able to import here as well. Um if there's a Helm chart, it's a matter of minutes uh to bring in uh your own applications in here. And with that we can basically build this cloud-like environment for users. But all of that is residing in the own data center or in a um at least controlled um environment uh within the secure network um that is is uh basically where all the all the confidential and classified data resides.
So that is uh why we got together in in this group plus saconet to create this solution um that uh will will help us do that cool agentic AI stuff that we were talking about all day um and bring that to confidential data as well. Now with that um maybe look a little bit into what's next right and if you want to give us your thoughts um what what do you think will be the next steps now that we have this available what can we create with it and what do we need to look look for >> we need to go into production so if anyone has a has good ideas uh or um yeah um what applications would make sense here please come up to us um and discuss with us.
We have written it all in a white paper so to also get your input and feedback. Um and I think oh yeah that's a white paper. Um and I think from a a technological perspective what really is um top of my mind at the moment is everything around harness engineering. So um how can you make your agents better? How can you give it more context? How can you guard rail agents? This is something also our engineering teams talk a lot about. Um yeah, stay posted what the next haste releases bring. Um so you can yeah follow us on GitHub obviously but um yeah that's definitely a topic that I think is of big relevance also for classified domains.
What do you think? What's next? Well, for me, I think um of course we need to go into production now uh that we figured out how we can do this in a safe environment, right? And um from my perspective, what we also need to do is use um these platforms to bring together um like a scalable environment, bring more use cases on the same environment because only then it will also be um uh yeah economically useful and and we can scale it and then we will also have uh I think a lot of fun with these uh with these AI environments in in a private space. Yeah, my my closing remarks here is uh Nvidia will provide a big stage for uh demonstrating uh all these uh innovations and and use cases uh in Berlin uh October 20 to October 22.
U GTC uh is coming to Berlin. Our our flagship event will be uh hosted in Berlin uh this year. Uh I'm super proud of it. Um um there's a lot of excitement already. Um and yeah I I think we will all see u implementations of u of this technology at GTC and see you there ladies. >> Thanks >> yes what was next? Are there questions? We have a few minutes from the audience for our panel. you are all keen for a okay lunch hungry >> um a question to um GA is my name I'm um CISO from the energy company uh we have at the moment the pilot phase for many agents uh and I'd like to keep the AI safety for many different reasons risk compliance and so on and I'm concerned that if we have critical uh workflows that are supported by agents uh and they are isolated on prem how do get instant incident response on an uh AI incident on the weekend.
So typical interruption question, how is that sorted? How is AI safety and maintenance brought together in off business hours uh to keep business continuity? >> I mean I mean from my perspective on on on premises doesn't mean it's only accessible when you plug in your computer into the server. So there is a secured network to access uh this uh this data to and I mean there will still be also for Microsoft there will still be an ops team uh especially for uh critical infrastructure like energy and um this uh this ops team will have to be trained to to learn to work with these agents to become more efficient. >> So I don't see it as isolated.
It's just living in a secure environment but it's it's still accessible. But you're raising a good point. So um governance visibility into all these agentic workloads is a really important topic. It's from a uh like um management point of view but also from an operational point of view, right? And in some cases also from a regulatory point of view, you need to show what workloads you have running, how they are performing. Maybe you want to monitor how they u work over time, right? to detect if there's some I don't know some drift that is is uh with declining um accuracy or something like that. So monitoring and and responding to these things is is an important part here.
One more question. Anna, I have one for you. Okay. Because you said that uh in the agent setup that you had that it's not yet automated, but you didn't go into detail. Like is that ever going to be a good idea? Are we on the way to that or is that is the human in the loop always going to be there in these critical places? Um yeah, I think we need a risk based approach like in the UAI act and for critical applications we should have a we have to have a a human in the loop. Uh but I think there's a lot of other applications where we don't need it and where we can automate a lot of things and have multiple agents working together and just like doing workflows or um yeah making stuff that nobody wants to do and just work it off.
But for some I mean this is classified domains can also mean defense and this is a big thing and I'm not sure if you want to automate all of that. >> Is that a German perspective or is that a general perspective? >> That's Anna perspective now. >> Got it. Very good. Okay. We will no longer stand between you and lunch. Thank you so much. We meet here again. Let's do that