At Rise of AI 2024 in Berlin, Oliver Iff, data scientist at SVA System Vertrieb Alexander GmbH, presents a technical walkthrough of a generative-AI system his team built for clients to retrieve process knowledge scattered across an organization. He describes a multi-agent architecture in which large language models translate a client query, for example about a production facility, into a chain of function calls against data sources such as SQL, vector databases or the Signavio process-management tool, with each agent specialized for a narrow domain rather than relying on one large general-purpose model. The system runs on IBM's watsonx.ai platform, letting the team swap foundation models, including IBM's Granite series, and access them via REST API, while watsonx.governance tracks model lineage, fairness and bias metrics for EU AI Act compliance. He details the pipeline: converting Signavio process data via BPMN and JSON into embeddings stored in Elasticsearch, retrieving processes through a process agent using Granite, and generating dashboards and Python code through a code-generation agent using StarCoder. He closes citing productivity gains of up to 80 percent in comparable use cases such as HR or customer support.
Key takeaways
- Iff's team builds multi-agent systems where each agent specializes in a narrow domain rather than using one large general-purpose model.
- SVA uses IBM's watsonx.ai to swap foundation models, including IBM's Granite series, tailored to specific tasks for cost and energy efficiency.
- Watsonx.governance provides a model registry, fairness and bias metrics, and model lineage, which Iff links to EU AI Act compliance.
- The pipeline converts Signavio process data into text embeddings via BPMN and JSON, stored in Elasticsearch for retrieval.
- A separate code-generation agent using StarCoder produces Python dashboards and visualizations from retrieved KPI data.
Detailed notes
The recurring pain point: knowledge scattered across the organization
Iff describes a common client workflow: an employee receives a task, must figure out whom to talk to, gather data distributed across the organization, analyze it, visualize it and present it to stakeholders who make decisions from it. He says the tool landscape supporting this has grown so broad that nobody can master it at an expert level, creating a need to consolidate or automate the process and reduce the 'burden of knowledge' on employees.
Multi-agent architecture
SVA's approach uses an agent framework of one or more large language models that translate an incoming query, such as a request to report on a production facility, into a chain of reasoning and function calls against data sources including SQL databases, vector databases or Signavio, a process-management product. A single agent covers a specific domain, mapping a query onto defined functions and feeding its own output back into itself in a loop until the answer is sufficient. Multiple specialized agents can be combined and triggered depending on the query, similar to how human experts specialize in narrow fields.
Foundation models on watsonx.ai and watsonx.governance
The team avoids relying on one large model for every task, arguing this is usually overkill and less cost- and energy-efficient than smaller, task-tailored models. They use IBM's watsonx.ai platform to train, validate, tune, deploy and swap foundation models, including IBM's own Granite series, whose training data is known for governance purposes. The Prompt Lab offers a low-code way to tune hyperparameters like temperature, maximum tokens and stop sequences, and models are accessible via REST API. Watsonx.governance covers the ML lifecycle with a model registry, and Iff notes some clients have 50 or more deployed models with no overview otherwise; it also reports fairness and bias metrics and model lineage, which he ties to EU AI Act compliance.
From query to dashboard: the technical pipeline
Signavio process data is converted via BPMN into JSON, then into text via an LLM, producing embeddings stored in a vector database (Elasticsearch in their setup) so processes are available as context. A client query is fed into the multi-agent framework, which determines which agent and functions to call, for example retrieving process data from the vector database using IBM's Granite model. A separate chain then prepares KPI data and passes it to a code-generation agent using StarCoder, which generates Python code to produce dashboards and charts, and can summarize results for the client. The primary conversational model used is Llama 3.
Questions answered in this talk
How does SVA's generative AI system retrieve organizational process knowledge?
According to Oliver Iff at Rise of AI 2024, SVA converts process data from the Signavio tool into BPMN and JSON, then into text embeddings stored in a vector database, so a multi-agent framework can retrieve relevant process information in response to a natural-language query.
Why does SVA use multiple smaller LLMs instead of one large model?
Oliver Iff explains at Rise of AI 2024 that using one large model like ChatGPT for every task is usually overkill; instead SVA uses IBM's watsonx.ai to deploy multiple smaller, task-tailored foundation models such as the Granite series, which are more cost- and energy-efficient.
How does watsonx.governance support EU AI Act compliance?
Oliver Iff states at Rise of AI 2024 that watsonx.governance provides a model registry, fairness and bias metrics, and model lineage tracking, giving organizations an overview of deployed models that he says is an important step toward compliance with the EU AI Act.