Rise of AI Conference 2024 · Berlin · Applied AI Stage

The EU AI Act: The Good, the Bad, and a Tiny Bit of Hope

Felix Broßmann · Partner at SKAD AG

At Rise of AI 2024 in Berlin, Felix Broßmann, a partner at SKAD AG, gives an overview of the EU AI Act based on his firm's consulting work in AI governance for banks, insurers and startups. He opens by arguing that regulation is needed because generative AI carries real, demonstrated risks: answer quality degrading over time, training data being extractable from models, backdoors surviving safety training, bias in outputs, and chatbots that can be manipulated into damaging statements, such as a car dealership bot offering a car for one dollar or a jailbreak that bypasses safety filters by asking in Zulu. He then explains the Act's risk-based, four-tier structure, including an added layer for general-purpose AI models, and its goals of building trust, harmonizing the internal market and protecting fundamental rights. He details the burden this creates: an 458-page text, high compliance costs, steep fines and unresolved practical questions such as fine-tuning, biometric categorization and AI literacy obligations. He closes with the Act's relief measures for smaller companies and ten concrete recommendations for organizations preparing for enforcement.

Key takeaways

  • The AI Act uses a four-tier risk model, with obligations scaling from minimal-risk transparency duties to strict requirements for high-risk systems.
  • General-purpose AI models such as ChatGPT face an added regulatory layer aimed at systemic risks as models grow more capable.
  • Initial compliance setup can cost around 30,000 euros, with total AI compliance estimated at roughly 17% of AI investment.
  • Fines for violations such as supplying incorrect information to authorities can reach 7.5 million euros or 1.5% of global annual turnover.
  • Microenterprises, SME cost reductions and regulatory sandboxes offer relief, but questions like fine-tuning and AI literacy remain unresolved.

Detailed notes

Why AI needs regulation

Broßmann opens with documented risks of generative AI: academic research shows answer quality can degrade over time, and training data can be extracted from models, including GPT-3.5. He cites an Anthropic paper on backdoors in Claude 3 that persisted through safety training, and a study showing bias in GPT-4 usage in healthcare. He gives two real-world incidents: a DPD chatbot that was manipulated into giving unwanted answers, and a Chevrolet dealership chatbot in Watsonville that a user convinced to offer a new car for one dollar, which was shut down six weeks later. He also demonstrates a jailbreak: ChatGPT refuses to explain how to produce black powder in English and in Gaelic, but complies when asked in Zulu, a loophole he says still worked two days before the talk.

The good: a risk-based framework

The AI Act aims to build trust through a CE-marking-style certification system, comparable to consumer electronics safety recalls, and pursues two stated purposes: a functioning internal market with harmonized rules across EU states, and protection of health, safety and fundamental rights. It uses a four-tier risk model, from minimal-risk systems that only need transparency to high-risk systems that require a full risk management system and documentation of training data and bias. The final version adds a layer specifically for general-purpose AI models, which face stricter obligations and exchange requirements with commissioners as they grow more capable, to address the kind of systemic risks shown by the jailbreak example.

The bad: complexity, cost and open questions

The Act runs 458 pages, versus 78 for the GDPR, and Broßmann says the substantive rules effectively start around page 100. A European Commission study estimated AI compliance costs at roughly 17% of AI investment, with upfront quality-management setup costing about 30,000 euros for companies without existing risk management. Sanctions are steep: supplying incorrect, incomplete or misleading information to authorities alone can cost 7.5 million euros or up to 1.5% of global annual turnover, on top of possible class action lawsuits. He expects a prohibition phase six months after entry into force, with sanctions following twelve months later. Practical ambiguities remain, including whether video-conferencing tools that track speakers count as regulated biometric systems, how fine-tuning can shift a company from deployer to provider status, what counts as sufficient AI literacy under Article 4, and how to measure fairness when different statistical measures of the same dataset disagree.

A tiny bit of hope

The Act includes relief for microenterprises, roughly nine or fewer employees with up to one million euros turnover, offering simplified quality-management guidance, and cost reductions for small and medium enterprises, including a commitment to re-evaluate certification costs. Broßmann points to planned innovation hubs and regulatory sandboxes that ease obligations for use cases such as medicine or ESG. He notes that companies already operating in regulated industries typically already satisfy 50 to 70% of the required risk-management functions.

Ten recommendations

Broßmann closes with concrete steps: accept that the Act exists and is complex and its fines severe; inform yourself and your colleagues beyond headlines; inventory all AI systems in the company, including edge cases that could be high-risk; build an AI governance structure sized to the organization; and use available help, from consultancies to regional AI hubs, to prepare before enforcement begins.

Quotes

That is why we must talk in all honesty about real risks of AI.

Felix Broßmann

The AI act is there, get over it.

Felix Broßmann

Questions answered in this talk

What is the four-tier risk model in the EU AI Act?

According to Felix Broßmann at Rise of AI 2024, the Act classifies AI systems by risk level, from minimal-risk systems that only need transparency to high-risk systems requiring a full risk management system, training-data documentation and bias checks, with an added tier for general-purpose AI models.

How much can EU AI Act violations cost a company?

According to Felix Broßmann at Rise of AI 2024, supplying incorrect or misleading information to authorities alone can cost 7.5 million euros or up to 1.5% of global annual turnover, with further exposure possible through class action lawsuits.

What relief does the EU AI Act offer small companies?

According to Felix Broßmann at Rise of AI 2024, microenterprises with about nine or fewer employees may get simplified quality-management guidance, SMEs may see reduced certification costs, and regulatory sandboxes are meant to ease obligations for use cases such as medicine or ESG.

Mentioned

EU AI Act, GDPR, CE marking, general-purpose AI models, regulatory sandbox, AI literacy, risk management system, jailbreak

Keep exploring

More from Rise of AI Conference 2024

More on Ethics & Trust

Transcript

Auto-generated transcript, not edited. · 3,844 words

Show the full transcript

Thanks a lot. Thanks a lot. I think we are ahead of this. We are ahead of this. So, that's great news because then I can do one little thing I started last year, I think, about in February when giving talks. So, Should I ask this person? Okay, I ask the question. Is there anyone here in the room that never has tried out ChatGPT or any other large language model by himself or herself? Okay, I thought about that one. Who of you is using generative AI on a daily basis? Oh, that's much, but not all. Interesting. And who of you has AI, not only generative, but AI implemented in your own businesses or corporates? More or less the half.

Thanks a lot. So, I'll try whether it works. Yeah, that's great. I'm Felix. I'm pretty glad to be the last speaker standing up here and talking about a very lightweight topic of the European AI Act. I hope I can give you a comprehensive overview, not about all of the rules in the act. I will come to that later, but at least about our experiences that we made in the last year in consulting in regards of AI governance. So, for that one, I have also my good colleague sitting right there over there because we have we at all not only proposing AI tech consultancy, but also AI governance consultancies and we are having projects not only in the financial industry, so banks and insurance companies, but we are also trying to help um startup or the startup ecosystem.

I for myself personally a lot in Hamburg, um but around that also. So, I really like this picture. Let me pour some precious precious water in your great artificial intelligence wine that was presented all day long by today. I will start a little bit different because I will give you an honest overview about why regulation should happen. And then I will give you some of our experiences. So, let us start with the great question. Let us start with why are we regulating? Maybe you felt it by yourself. You're going to ChatGPT and it gives you an answer and the day after it doesn't propose the same answer or neither the quality of the answer that you were trying to think of.

As a matter of fact, you don't have to really control about when that happens, but it is actually academical proven. One matter of fact, there are lots of lots of slides that you can have the slide deck afterwards, so you don't have to take photos at all. So, researchers showed that the declining or the quality degradation, that is a real thing. That isn't made up. Did you notice that one this last April? I see some someone is nodding here. Why is it a problem? I asked me. Yeah, because of that. There are academic papers that showed that you could extract training data out of models. And not only diffusion models, that is proven, but also in a large productive language model, GPT-3.5.

I think it was in the paper here. So, if you are using the free version, as was taught by Ingo before, if you are using free version of ChatGPT, please be aware that all of your precious training data you put in there and OpenAI uses for retraining their models could be extracted at some point. Potentially. So, that is a real risk. I'll give you another one, backdooring. I don't know I didn't heard about that one this year, but in last year we had great guys here talking about it. Backdooring and attacking of especially multimodal LLMs is a real threat and it's a real thing. And there's much more to look at. There are ultra backdoors.

I actually pointed that one out. That was a paper from Anthropic. You know, the big competitor in the US, so Claude 3, Claude 3 open and so on and so forth. They showed that they could propose a model or could propose yeah, an agent that persists through all their safety measures within training. It was a little programming component. And they couldn't get it out with their training measures. Another one, biasing. I really often hear our biasing isn't a thing and racial and so on and so forth. It is a real thing. And the large productive models are biasing as shown in this study in regards of GPT-4 usage in healthcare. Did you notice that one?

It was quite funny. DPD implemented a chatbot and the chatbot got yeah, more or less hacked and then produced answers you don't want to hear. So, reputational risks are a real thing. That was not the only accident. There was another accident. I think that one you could heard of. Yeah, I see I see big thumbs up there. So, what happened there? There was a Chevy. There was a Chevy yeah, um team so in Watsonville and they implemented a chatbot and what some clever user did there was it convinced the chatbot to give him a new brand new car for $1. Some things you can imagine by yourself and what could go possibly wrong if you are doing and proposing a virtual priest.

Six weeks afterwards it was shut down. So, that are real threats. And then another one you can try by yourself when going out of this room after Fabian's talk. Um There are there is a paper and it shows that it is possible by using not too often spoken languages um to yeah, circumvent the safeguards of unproductive being models and I'll show you what they meant. Actually, I tried it for you. You see here ChatGPT was an instance and I asked the English question, please could you give me a way to produce black powder. And what does ChatGPT say? No, I don't like it. Because it's unsafe to give you that information. The paper stated then that you could ask the same question in Gaelic.

And actually what the model produced was no, no, no, no, no, also in Gaelic, no way. And I was about out and about on LinkedIn and ranting about those paper, but then I recognized, hey, they also told in the paper that you should try Zulu. And for those who couldn't see it, this answer actually gives you a recipe for black powder. So, it actually circumvented all the safety measures the big OpenAI given out and you can try it out by yourself because 2 days ago it still worked. That is why we must talk in all honesty about real risks of AI. And that are the obvious ones. Not the nice ones. No, bad ones. That are the obvious ones.

And bad ones? Okay, the Pope is cool. I I give that credit, but but bad ones and can you imagine what is possible with kind of that technology? Actually, the OpenAI 2 days ago showed or at least talked about it in their talk that I red teaming and that they are taking seven and so on. They are pretty aware right now that it is risky what they are producing there. And I'm really glad that I joined the last the last minutes there because if you don't believe me that all that what I was expecting there are leading to uncertainty within the users, within the corporates, within the public administration, then please believe the stand for the AI index because it showed.

And the talk before also stated it. So, let us take a look at the AI Act and maybe think a little bit about the good things that comes with it. So, the first thing is building trust. That's what the politicians say. And yeah, there is something happening on a global basis in terms of regulating AI. It isn't only the EU. There are regulations also in the US and also in China and yeah, they go pretty deep. So, don't get fooled by the media attention around how bad the AI is regulating anything. There are absolutely not the same, but there are regulatory things happening all over the world. In Europe, it's that one. And I brought you some more of that slide so that you can take a look and challenge my um talk afterwards.

So, that's the last the last version of the UI, it's a so-called corrigendum, it's more or less final version that will be uh yeah, that will be um yeah, the the the final version. So, one thing um I would show you is or an idea at least. So, what is laid there is building trust by certification. That's one of the idea also the UI proposed. And you know that, if you are dealing with consumer electronics, you already saw that kind of things. Not only consumer electronics as a whole. So, what the act proposes is more or less also a CE marking. So, that's a good thing. As a consumer, you could hopefully um be safe if you are using CE marked products with AI in it, okay?

I will be honest, it will not help preventing every risk, but as you saw here the same plug charger there yeah was to be recalled because yeah, it actually didn't uh it was a safety risk because it failed in some state, but at least there was an exchange program. And I think that's in the good mind of regulating AI how it could work. Even in if there are new stated new risks afterwards after being deployed into the market, then there should be at least um at least the need for the providers to do something about it. Another thing, protection of rights. There are two main purposes stated in the AI Act. And the first purpose is improve the functioning of the internal market.

It's quite easy if you think about having a different regulation of AI systems in Germany and in Spain and in France and so on and so forth. It wouldn't be a great solution for all of your pretty great AI product ideas. So, therefore, functioning of the internal market. Mark off. And another thing is not only the functioning, but also ensure a pretty high level and standard not only for health and safety, but for fundamental rights. And that is the main idea hinter behind all the things of how it is done. So, what does the I propose? The AI Act proposed. It proposes the following. It's an shape model four layers you you can say of different risks.

So, what's what the um act says is the harder or the more risky your product is and the more you could go into this fundamental rights, the harder you will be regulated. That's the main idea behind all the act and regulatory things in there. And it goes from minimal risk where you only have transparency things to consider and it goes all the way up to high risk where you have to have a risk management system where you have to propose at least in some cases where are the training data is from, is it biased, is it unbiased and so on and so forth. The harder the potential impact of the fundamental rights the harder the regulation gets.

And it's nothing new. We already had that. For example, in medicine. Actually in the last and that maybe a bad thing as well. Actually in the last draft with version and it's now in the final version, there are some or two additional layers. And they have especially built in there for general purpose AI models. ChatGPT, they couldn't put in ChatGPT as a name there, but that's the thing. So, what also imposes there is the the the the the bigger the model is and the more sophisticated the model is, the harder the regulation and the need for exchange with the commissioners to prevent these systemic risks that are within or that lies within those models as shown with the jailbreak before.

That's the main idea and that's a good thing. That was a short part. Now comes the bad things. Boy, that one is a complex one. 458 pages 58 pages full of incredible joy. So, if you can't get to sleep tonight, just pull out the AI Act. I promise you 30 seconds and you fall asleep. Putting kidding aside, it is a very very very complex matter. And the real act so to say starts I think on page 100. And to give you some yeah, something to consider it too, the GDPR. Anyone knows how much pages was the GDPR? The regulation as a whole? 78. So, just from looking at the those numbers, you can understand why I don't go much deeper in how it is really in detail regulating because that what you are seeing here are only the obligations of providers and deployers of AI systems.

And the great Man Hansen does a wonderful job with his AI officer and this overviews he gives there, there are I think at least 20 of those overview pages on what is regulated. It is simply a beast. And as we've heard before, it doesn't stand alone. No, no. There is regulation all around. Not only GDPR, but also if you are in banking, so specialized banking rules further comply, yeah? Complexity is a thing. And with complexity comes debt. Before the first incarnation was out there, was a study from the European Commission Commission um that stated that the overall costs for AI compliance and being compliant with AI regulation will be round about 17 percent of the AI investment costs.

And they stated further that you will have upfront costs if you don't have any risk management at all in your company till now that you will have upfront um quality management costs about 303 30,000 euros. There's another study that states it's only about 100,000, but I think that is a bug. Only for setting up an um risk management system for the AI Act. So, sanctions, we already heard about it. That's another thing. And one thing I want to to propose, it's not only the forbidden practices that are pretty pretty steep. No, no. Look at the first one. The state supply of incorrect, incomplete, or misleading information in response to a request of authorities. If you are doing so, it costs you from 7.5 million up to 1 .5% of your global annual turnover.

That's the thing. Not to mention class action lawsuits that will also get up there. So, one main thing is you don't have time. At least not too much time. Six months after the expected go entry into force in June at least we expect that one. Um there will be the prohibition phase and 12 months beyond that, there will be sanctions coming in or kicking in. Yeah? And then you see the dates there already. And there's another really really bad thing. And you only notice that when you are implementing those kind of things in practice. So, one thing as an example of many many others is that there's a video conferencing tool there and it often times uses um uh or it focuses on the people who are speaking right there.

So, are those biometric matrixes that aren't allowed? Or are they allowed? So, there is AI also in places you don't expect. And it is not only about software, it is also about combined software and hardware products you will have to think about. Whether those here is a high risk or not, you must think about. Another one is fine-tuning. So, in the act it states uh the act itself it says the provider of AI systems has very very very large especially in high-risk cases has a very large obligations to fulfill and the deployer had a smaller set. But, what you can get you can get easily from being deployed to being provider when you are white-labeling a product.

That's enough. And you can also easily get there when you are fine-tuning model because you are uh changing the model in a way. And that is also stated in the act. Or another question. You have to get your head around AI literacy. It is stated in article four. And I really recommend it to to read it and give it to your bosses. Yeah. So, the main question here is what is sufficient? Is it enough to give out a handout? Some lawyers say yes. I personally would say no. And another one. We heard all around non-discriminatory and fairness. But the difficult thing is what is fairness and how do you measure it? And if you're taking a look at the literature, you see here for example, the source is given there uh right down there.

You see here many many different kind of measurements from a data set, it's a German data set of financial data and what I marked you up there is if the measure says it is a fair data set or not. And depending on what measure you use, you come to a different conclusion. And the act says yeah yeah there are some things and we we mean by fairness is that and that and that, but it isn't sufficient to answer that question how in reality in practice it is measured at the end. So, that was the downside, a tiny bit of hope, but only a tiny bit. Microenterprises. Microenterprises are enterprises with I think nine or less um people working there and I think 1 million or up to 1 million global annual turnover.

Actually, they are named in the act itself number 100 60 46. In the um yeah in the in the in in the in the first pages is so to say you can read it there. So, what the lawmakers actually thought of was maybe we should give those small uh companies a a way or a guideline how to implement at least a core quality management without overburdening them. How that in practice then will be implemented I can't tell you right now because we are missing some of the main ingredients. Also, cost reduction is a thing. Not only in regards of micro companies, but also for small and medium-sized enterprises. There are two actually there are two passages in the act in which they are meant not only to pay less for all the certification, that's the idea, but also the commission should re-evaluate the costs.

So, how much is it then a real time and should adjust appropriately for small and medium-sized enterprises. There will be standard procedures. At least we hope that. Also, that you find in the act and there will I hope additional innovation hubs and you will give some money for that for helping provide technical and scientific support in regards of yeah fulfilling those safety measures. And there's one last thing I'd like to mention that is sandboxing and sandboxes. It gets clear quite in a moment why I'm mentioning that. Sandboxes are regulatory sandboxes. They are um they should be built to foster innovation by a little degradation of yeah the the the obligations you have to fulfill before going into market.

Very special very complex kind of material, but that could be especially for use cases in regards of medicine or ESG and so on. There are GDPR yeah little liftings in there and so on. So, sandboxes, remember the name. You will hear about that. And some more good news, there are already um functions that you have established if you are working in a regulated industry and what we saw there is that those corporates they are already regulated that you could fulfill 50 to 70% in the functions that are already existing. You don't have to build a new risk management up there. So, last thing, there is still a little bit room of improvement. And actually, the great guy center he was there today or he is there he has a wonderful blog post out there and you will still have the chance to share your expertise and build up the guidelines that are hopefully answering the questions I mentioned before.

Fine-tuning, you remember and so on and so forth. So, there will be guidelines out there and you could deliver your expertise in building up those guidelines. I actually did so within the sandboxes that was the thing I mentioned last February and I can swear you you can make an impact. And you will make an impact with sharing your expertise. So, I think I have one more minute at least I hope. Okay, that's good. 10 things to take home with you if you don't listen all the time till now. First thing, the AI Act is there, get over it. Full stop. It is complicated. Absolutely. And the fines the that are proposed there are absolutely absolutely high and severe severe.

Stanley is right. You with all your products and you with all your ideas on implementing AI, you have great power. But you should use it responsibly responsible. And the first thing you should do is inform yourself about how the act really is working. Not just the headlines. Get a deep dive. And when you're done with that, share this information with your colleagues. Afterwards, please list up your AI systems. Yeah, take an inventory. And I personally would say of all your AI systems. And not only the um the question on what is the use case and so on and so forth, but all the nitty-gritty information because you don't know if you need it because it could be a potential high-risk system.

Yeah, so get your company together and um gather information as fast as you can. And whether you are huge or small, implement an AI governance structure. Align it to your size. And the last thing, get help. Use all the possible sources that you have. Be it CMS, be it the AI startup hub in Hamburg when I am sitting here or all the other hubs all around, they will propose you help. Use it. Last but not least, okay, that one was an insider. So, I hope you enjoyed till now. If you have any further questions, um then I would say get in contact with me or my colleague, but I always wanted to do that. There is one more little thing.

We actually proposed a study and it will be out by tomorrow. A study on how to scale AI. And the one more thing I always wanted to do it is offer officially released by tomorrow and available for you by today. So, thanks a lot for your attention and I hope you enjoyed the session.

Rise of AI 2027 · Berlin · May 2027

Be in the room for the next one.

Ten years of talks started as conversations in one curated room in Berlin. The next edition brings 300 selected leaders together in May 2027.