yeah thank you for the kind introduction um we've already heard a few talks today and many conversations yesterday and in the past two years basically talking about gen I what it can do what it should be able to do what it can't do but um I think the stage is is called applied stage so maybe we should just apply it and see how it works in the actual production environment so from this point of view um we had SBA already Implement gen at our um clients and I want to give you a little overview how we are doing this at the moment from a concept point of view but also on the execution part so I will detail a use case um rather technically but I hope you can can take something from it so first of all I already had this nice introduction um actually my colleague uh Stefan Kama was supposed to give the talk but sadly got sick on short notice so I wish him a speedy recovery if he's watching um
but I've been working with machine learning for two years now at SVA and also um starting gen um related topics but that's enough for me now actually we're going to start with the with the use case itself when we're working with the clients um we mainly see the same pain points popping up everywhere on a daily basis where you have the use case of actually where you need to retrieve data knowledge which is somewhere hidden in the in the uh in the organization in the cooperation or you need to access um data for production facility for example and then in the end you also need to present it to your stakeholder so the typical workflow as you can see here is for um that a employee gets a task where he usually needs to figure out what he needs to do specifically who he needs to talk to um he needs to gather data which is mostly distributed dur um uh throughout the firm and then analyze it see what insights he can get visualize
it and then present it to the stakeholders who need to be able to do the right decisions based on these um metrics which are presenting and as you can see there are many um tools people can use so the tool landscape is way way too broad by now it's actually growing even faster day by day which means nobody really can handle these tools on an expert level um which makes it necessary that you have um which makes it necessary to consolidate everything under one hood or even automate it if possible so that you do the burden of knowledge um so that you reduce the burden of Knowledge from the the employees and automate it if possible and this is where we come in with our gen approach in that case um the idea is this is not a high level overview that we have a query example um I need to report on a produ production facilities and now we want to get the actual answer to this question and what we are employing is
an agent framework this means we have one or more agents um large language models which basically translate the incoming query into a useful thought um train of thought which then um is able to be executed and this means rather concrete that the agent is figuring out okay what does our client actually want to do and then accessing data sources if needed um different databases SQL Vector database or in in our specific use case um signavio which is a Process Management um product to be able to do this we employ different llms um depending on the actual use case and combining this with the the agent framework enables the architecture basically to to answer the question which the um the client has has been given to it now I want to zoom in into the the heart of this architecture which is a single agent so just remember you can have more than one agent but um One agent itself um can cover a specific domain a specific topic let's say the retrieval of the the
processes um which are in in in the organization so you have different functions which have to be defined and the agent now Maps basically the the query onto these functions so that you can then um execute the functions get the answer and because it's a self referential large language model uh this means it feeds itself in a in a kind of a feedback loop um the answer into itself until it's um yeah happy enough or uh until the answer is good enough for what's actually needed and this allows uh this allows to have an agent um as kind of like the the first stepping point in in order to actually get um the answer you want to but you don't need to only don't you don't need to use only one agent you can use multiple agents that's what I meant um that's because you can think of an agent as an specialist for a specific domain usually you don't have one expert for everything you have experts for rather narrow Fields that's exactly the
same idea here because what you can do now is you can combine um multiple agents for different domains and depending on the query one or even more agents are basically triggered they figure out what needs to be done and then accessing the tools um which are needed for this so you have from architecture point of view you have the agent framework consisting of different agents um which are responsible for different topics and domains in the end for this you need a rather uh rather good conversational large language model um like llama 3 for example um to to be a to be able to handle the conversation with the uh with the client but you need more than one llm as I said in the beginning because in our eyes it's really good um that you don't use one big llm all the time sure you can throw a CH gbt with all its powers onto the problem but it's most of the time it's Overkill if it's even able to do the the task which
you want him to do so for this we um use different llms and specifically um for our clients we are using Watson X um from IBM because there we have a platform where we can select the different llms swap them out swap them in as we as we desire and these Foundation models then are tailored towards specific tasks so this is the main difference you have Foundation models which are good in one thing and maybe bad and other thing but then you simply use another large language model that's exactly the point and small models are more effective cost-wise but also energy-wise so this is really the um the selling point in this especially also for for smaller organizations which really don't need a whole package but rather specific um Foundation models so this really good also IBM um by the way provides some nice models so the the cranite series um where they know exactly these are IBM made um models where they know on what train what data the mods have been trained so
from the governance point of view this is a really really a step ahead compared to many of the the open source or even the the third party U models now in order to access this kind of model catalog you need to use Watson xai which we are using at a client um so on the one hand you can train validate tune and deploy these AI models rather easily you can swap swap them in if you have a new model which is better than the old one you can just replace it for example but there's also a nice uh no code well low code environment uh the so-called prompt lab which I find really intriguing uh when you actually need to work with it because um you don't need to have much knowledge to actually promp tune an an large language model it's rather easy it's it iterative process where you can come to a good um good result rather easily you can just change the hyper parameters basically like the tunic parameters how creedy the
model is you can change uh the maximum amount of togens it has um st stop sequences um and similar thing so this is really good way to to go from from the testing stage to the to the productive stage in the end also all of these llms are accessible via AR rest API this means you can easily integrate them into your existing ecosystem you don't have to use um you don't have to to add a completely new kind of software but uh you can simply use the functional functionality of calling these uh large language large language models via the rest call additionally um so this is Watson x. a additionally there's also Watson x.g governance which covers uh the ml life cycle of um of your large language models uh so it has a model registry where you can have a nice overview of what models you have deployed we we know of clients um who have I don't know like 50 and upwards models in usage but but nobody really knows what's going on
in each specific one because there's no overview so this is an important step but you can also get metrics about fairness bias um how the model behaves during during its deployment and but you also know the model lineage so you can follow the model how it has been changed how it has um on what data it has been trained specifically and this is an important step um in order to comply to the EU AI Act for example so this is uh really an an important step um regarding this so with the building zones set up this way uh now we are coming actually to the execution part I promised before um the idea here is the client should be able to have a conversation I mean this is nothing new but it's a really convenient way for humans to interact with any kind of system simply use natural language for it so what we envision basically is that we have um this kind of conversation at the top is the the question or the input
from the human at the bottom is the response from our framework so for example um somebody needs to report on on production facilities and then we will receive the process um also where it where this this step the client is is specifically in in the process um chain so usually you have many many processes chain together um and there you can uh and there you can then know where you are specifically in this kind of chain and furthermore after you know what process you are at you can tell the llms to gather the relevant information in a nice compact way so that you can actually use it for your for your own or present it right away uh this is good um good to represent the data uh via dashboards for example but you can you can use anything you you like and then you can go even a step further and then ask for a summary about these kind of um dashboarding you do so you you can really um you can really be
flexible with with these kind of llms obviously so this is the main idea this is basically what a client is seeing on the surface how does it look in in in the background on the technical side of things uh for this we have to do kind of a pre-processing step so in this use case if you want to see the the processes which are available in the in the organization we first have to make the available to the llms so this means um you have your data in our case signavio data um which comes in is converted into a cph and then passed into a Json which is a file format um which then again can be converted into text via an llm and at that point you have the processes basically as embeddings um available which means you can use these embeddings uh as you can store these embeddings in a vector database in our case for example elastic search and therefore feed them into your large language model so this makes your processes
available within this kind of context this is a self-built pre-processing pipeline from us which is necessary um to to be able to uh do the next steps because once this pre-processing pipeline is set up we can then actually do the the productive stuff which means we have our query which then gets fed into our multi-agent framework where where you have many many agents kind of arguing and discussing um who needs to do what exactly depending on the query and then once they decide okay what needs to be done in this case it's the process agent um you go to the you call the actual functions the green ones and these are act out or these are um enforced then by the by the agent so you access the the vector database in that case and you can access the processes that way and then you can get back the relevant process data um and return it to the to the customer basically depending on on the task you we are using different um large language
models as I said in the beginning for the multi-agent framework we mainly using um llama Lama 3 by now and for the for the process retrieval we use Granite jet from IBM and here you already can see um yes the use case I'm presenting is rather specific it's focused on the processes but in the end it doesn't matter what kind of agent you are using I mean you need to set it up for your specific use case but implementing um another agent once this pipeline is is up and running is really not that difficult which means you can expand on on this kind of architecture rather easily once it's up and running so the then the step afterwards after retrieving where you are in the in the process step or in a process chain is to generate um dashboards crafts showing the kpis you want to know so this basically the same for a different use case now uh so prepare the relevant information is the is the query which means based on the on
the results we already had finding the right processes um we now employ different agents um first one is getting the the right kpi so it accessing our database in the top right to find the right kpis use them store them calculate them and these are then used in another agent the code generation agent uh which is using star coder for example it's an llm um focusing on code generation that agent generates then python code in our casee for visualizing so then you have a nice chain coming from um coming from the first query from the customer where he wants to know okay what do I need to do where I am in the process structure to okay now I know what to do just but please give me just a summar summarization summarization of everything I need to know and present it nicely so this really speeds up the the process uh in the end so with that I come to the conclusion and also the Outlook um at that part working with our clients
we are seeing a huge interest uh in combining this um gen approach because this is scalable from small organizations to big ones it doesn't matter and as I said the the biggest selling point is that is that is really flexible so once the Baseline is set up once you have established one use case then you can go and add even more rather easily and uh employing whatson X AI we also can swap out the large language models also rather easily in the future um when there are better ones or simply if the governance on compliance side the laws are changing then you can adapt rather easily and and fast the the use case was focused on process retriev basically but I mean it's not really a surprise that you can uh change the topic completely and you go to human resources or customer support where a studies found that you can increase um the productivity or the efficiency up to uh 80% depending on the specific use case obviously so this enforces uh the idea
that that we are going from people doing all the work to people asking a business AI agent or employing an AI agent in order to speeden up uh the the whole processes within an organization and with that I'd like to conclude my talk thank you for attention and I'm open for questions