Rise of AI Conference 2023 · Berlin

Hacking AI: Data Poisoning and the Cybersecurity Challenge

Mirko Ross · CEO, asvin

At Rise of AI 2023, Mirko Ross, CEO of asvin, argues that the AI industry is repeating the security mistakes the web industry made around 1997-98, when developers trusted any data entered into online forms until SQL injection attacks taught them otherwise. He says AI systems face an equivalent lesson with data poisoning: attackers who manipulate training data or exploit a model's neural-network structure can force a chosen, non-random output, for example making an autonomous vehicle ignore a stop sign. Drawing on research asvin began about four or five years before this talk, well before ChatGPT-driven public debate, he traces the evolution of adversarial attacks from early 2015-era image-recognition exploits, through backdoor attacks planted during retraining of public models, to 2022-era work fooling 3D object recognition from any viewing angle. He demonstrates a practical defense, reducing image bit-depth or smoothing to strip out adversarial disturbance patterns while preserving accuracy, and closes by discussing prompt-based jailbreaks of ChatGPT's content filters and the need for a responsible-disclosure culture modeled on the software industry's, plus scrutiny of AI supply chains such as outsourced data labeling.

Key takeaways

  • Mirko Ross argues the AI industry is repeating the web industry's 1997-98 mistake of trusting all input data by default.
  • Data poisoning forces a model toward a specific attacker-chosen output, such as an autonomous vehicle failing to recognize a stop sign, rather than random error.
  • Reducing an image's bit depth or applying smoothing can strip out adversarial disturbance patterns while largely preserving a model's normal accuracy.
  • A 2022 attack fooled Google's object-recognition system into misclassifying a 3D-printed turtle as a rifle from any viewing angle.
  • Backdoors planted during retraining of public models are a realistic risk when companies outsource data labeling to less trusted suppliers.

Detailed notes

AI's supply chain problem, and why asvin started early

Ross says asvin focuses on cybersecurity in supply chains, including the data and model supply chains involved in building AI: how data is collected and prepared, how models are trained, and how they are securely distributed. He says the company began researching how to compromise AI systems roughly four or five years before this talk, anticipating that AI would run into the same kind of supply-chain trouble the software industry faced around cybersecurity, well before the ChatGPT-driven public debate. He references the White House meeting with Google, Microsoft and OpenAI as evidence that trust and robustness have become central industry concerns, and warns that if the industry does not build stable foundations itself, regulation will step in and limit its possibilities.

What data poisoning is and how it forces a chosen output

Ross explains that data poisoning injects manipulated data into a model at one of two points: during training, for example placing backdoors, or through adversarial attacks that exploit how a trained neural network's statistics work, flipping specific neurons to move the model's output in a chosen direction. He stresses this is not the same as ordinary noise or error; the goal is a specific, attacker-intended output, illustrated by an autonomous vehicle's camera system being manipulated into treating a stop sign as something the car should drive through rather than stop at.

Learning from the early web: never trust your input data

Ross compares today's AI industry to the web industry around 1997-98, when developers had no concept that a name field in an online form might contain a SQL command instead, leading to the first code-injection attacks; the resulting lesson, that no developer should ever trust input data, became standard practice in software development. He argues AI needs the same lesson: because AI development is largely a data-analyst world rather than a hardened software-development one, practitioners are not used to assuming data could be malicious, and need to learn to 'de-poison' data the way web developers learned to sanitize form input.

From stickers on stop signs to 3D objects and clothing

Ross traces the escalation of adversarial attacks: a well-known 2015 example fooled Google's public image-recognition model into misreading a panda as a gibbon with 99.3 percent confidence using a calculated pixel-level disturbance pattern overlaid on the image, a method he calls purely mathematical. Since then, attack patterns have become far smaller and more automated, shrinking to roughly a 16-pixel patch, and researchers have shown stickers placed on physical stop signs can stop autonomous-vehicle camera systems from recognizing them at all. He also cites adversarial clothing, sold online for about 60 dollars, that can make a person invisible to person-detection cameras, misclassifying them as a bird, and adversarial patterns that can defeat facial recognition. The most advanced example he cites is from 2022, where researchers fooled Google's object-recognition system into consistently misclassifying a 3D-printed turtle as a rifle from any viewing angle, by encoding the disturbance into the object's shape and color rather than a flat image.

Backdoor attacks and a practical defense

Ross explains that when companies retrain public models for their own purposes, attackers who can inject manipulated data into that retraining process can plant a backdoor: an input pattern that reliably triggers a specific, hidden output while the model behaves normally otherwise. He demonstrates a defense against pixel-level adversarial attacks: reducing an image's bit depth, for example to six bits, or applying smoothing removes the fine-grained disturbance pattern an attack depends on while leaving normal classification accuracy largely intact, since the attack needs a minimum threshold of encoded disturbance information to work.

Q&A highlights

Asked about fine-tuning and instruction layers on top of models like ChatGPT, Ross says the central risk is the trustworthiness of the underlying data supply chain, including outsourced data labeling and classification work, for example to low-cost contractors, since a company far removed from that labeling process has limited ability to verify it, and injecting a backdoor there is a realistic attack path; if you have not trained a model yourself, he says, you should not fully trust it, though access to a model's structure allows detecting likely backdoors with high accuracy and retraining to reduce their effect. He cites Microsoft's Tay chatbot, trained on Twitter, as an example of an unanticipated attack, a coordinated public campaign to make the bot say offensive things, distinct from data poisoning. On content filters, he shows that ChatGPT's ethics filter blocked a direct request for discriminatory advice but did not catch the same request rephrased as 'write Python code that does that', and separately blocked a joke request about women while allowing one about men, illustrating an inconsistency in how the filter had been tuned. He argues that operators of public AI systems need a responsible-disclosure culture, similar to the software industry's, so people who find these weaknesses report them rather than posting them elsewhere.

Quotes

If you haven't trained the model yourself, don't trust it.

Mirko Ross

Data is never good data, is never something you should trust.

Mirko Ross

Questions answered in this talk

What is data poisoning in AI?

According to Mirko Ross at Rise of AI 2023, data poisoning is the injection of manipulated data into a model, either during training or by exploiting a trained neural network's statistics, to force a specific output the attacker wants rather than random error.

How can adversarial attacks on image recognition be defended against?

Ross demonstrates that reducing an image's bit depth, for example to six bits, or applying smoothing removes the fine-grained pixel disturbance an adversarial attack depends on, de-poisoning the input while largely preserving the model's normal classification accuracy.

Why does Mirko Ross compare AI security to early web security?

Ross argues AI developers today are repeating the mistake web developers made around 1997-98, when no one anticipated that form input could contain malicious code (SQL injection); AI needs the same lesson, that input and training data should never be trusted by default.

Mentioned

data poisoning, adversarial attack, backdoor attack, SQL injection, responsible disclosure, AI supply chain, ChatGPT, content filter jailbreak

Keep exploring

More from Rise of AI Conference 2023

More on Defense & Security

Transcript

Auto-generated transcript, not edited. · 5,755 words

Show the full transcript

foreign okay so on to our next speaker who I have to say is probably one of the coolest startups that I've seen come out of stuttgard for a long time that's a company called asvin please welcome their CEO and the coolest guy in the marketplace Mr micro Ross to the stage the stage is yours thank you thanks um yeah and thanks being here um it's late in the evening and I think it's for purpose to bring the cyber security guys almost at the end of the conference because we are this kind of Show Stoppers for all this nice growth stories in Ai and now um yes now it comes now we are going to the Pains of the industry the repaints okay hacking AI so just let me give you a brief introduction uh what we're doing with Ashwin in Germany um just I need to take it on the right side great um just a short profile so uh what we are doing at Ashwin we are focused on cyber security particularly in Supply

chains and data Supply chains are very important and however if you look at AI there are a lot of Supply chains involved not only data as well the questions as how you're processing your data preparing it for the training then you have the models how to distribute the model securely so there are many Topics in AI which are related to supply chains which have leaded us to the conclusion conclusion that it's very wise to have a look on this and we have started this rather rather early I think it was now four or five years ago where we were thinking about hey we think that the AI industry will get run into a supply chain trouble in the same supply chain trouble like the software industry uh towards cyber security and this was before we had these discussions about chat GPD and all this run so we were starting to really deeply dig into the questions on how can we compromise AI systems and how and what kind of lessons we can get out of

compromising AI systems to milk data to make them more trustworthy at the end and more robust and whatever we are discussing today as an industry and and I've listened a lot of very good applications of AI but we had as well this discussion about AI Act and all this stuff so I mean there are a lot of concerns out there not only in Europe as well in the US if you had seen the meeting the presidential meeting in the white house with all uh the Googles the Microsoft and the open AIS of this world it's very obvious that trustworthy and robustness of this technology is an obstacle if the industry will not getting towards this kind of stable foundations unfortunately regulation is stepping in and regulation means in this case as well limiting all the possibilities we have as an industry so building robust Ai and trustworthy AI will be really the the a very important challenge for the future so we were really early digging in into the questions how we can compromise AI

systems just to get the Lessons Learned on how to build better and more safety secure AI systems there are several research projects we have carried out mainly early starting with data poisoning so the questions how we can inject manipulate the data towards your training your AI training just to force the models to give a prediction we want to have as an attacker and not a prediction which has been intended by the companies so this was really the starting point then of course all these questions about adversial attacks I mean this was getting very popular 2015-17s if you look at the papers 2022 we have really a peak of papers um about adverse role attacks so because many people are now stepping in fortunately because it's so easy to be honest um and you think you can find very very fruitful examples on how to manipulate AI systems by this and um now on a current project we are working on with anti-dot on the tools you can use within your machine learning um development life

cycles and your CI CD pipelines you should have as an AI company how you can integrate the tools just to de-poisoning for example data inputs you're bringing into various machine learning models so this is one scope how we can de-poisoning it why it is so important because obviously if you haven't built trained your AI system it's not trustworthy there's no way to say well there is any trust in there so if you can't fix it at the core you need to take care that you fix it on the data level with the data you're operating with this with the systems um and that's one scope of the projects there so just to give a people who are not part of this like how you can manipulate AI systems uh just giving you a short introduction on well what is the main idea on data poisoning and how it works um usually it's the way that you say well I mean you have a you have trained you're you have your training data you're training the

model you have a very very great idea on what is the allowed um statistical output of your model where your model is in a certain quality level like this red dots are out of the quality level and then your do your training iterations to get the best fit of the accuracy of your model on data poisoning what we are doing is we are injecting data to your AI model on two levels one level could be during the training like placing back doors in your AI models or a second one for the adversal attacks we are just using the way how AI models are working on the neural network statistic um systematic and by that trying to flip certain neurons in your network and then we can just move the output of the model as well in a certain direction which is very important we don't want to move it randomly like that's easy I mean randomly it's just an error that's what you have in your expectation every day when you're working with AI and

you want to train and to increase the accuracy now it should not be done randomly we want to move it really in a certain direction because like we want to go or we want to force your system to make an output we have intended as an attacker it's very clear if you are working with AI systems who are operating as an detective we want to force the system to make a certain prediction and by that to force it to make a certain decision later on some famous uh just giving you some famous examples if you have an automobile autonomous vehicle you have a you have a visual camera system the prediction of the road signs it's not only to manipulate the way how a road sign is recognized we want to manipulate the way how the car is reacting should it stop or should it just go Full Speed Ahead over a stop sign that's that's really a different it's not randomly intended what we want to do because that's just an error we want

really to force the system to go to a certain direction and that's the way uh how you hack AI why it's such a great opportunity to do this it's a great opportunity because unfortunately the AI industry is at the same level like the web industry in around the year 1998 or 1997. I mean we have the same enthusiasm today oh I mean we have the same bus as well like it will change everything blah blah blah for sure uh however but we are dealing with the same attitude then on that time it was amazing of course people were building great applications um we had only a very limited set of tools where we can work with this kind of service and stuff like that but for guys and ladies and others in this room who can a little bit remember the time there there was a certain attitude like you had been really lucky when you have built your first web application you had been proud of that it's very I mean a lot of

knowledge you have to acquire a lot of things and then you have this kind of first web application and in this case it's one of these kind of web applications with it like this Pearl scripts where you have an online form someone was typing in something to an online form the data has been submitted by a post request to your server processed and whatever what happened it was amazing so but nobody was thinking about can I trust what people are entering there because the perception was yeah they will enter their name very good what was the result at the end we had people who were not entering names they were entering SQL commands to the input lines so this was the beginning of code injections to our web server industry and why because the developer had no intention that someone could put anything else than a name on a web form there was no idea that someone would put in their SQL commands or Pro script or whatever so this was a lesson learned we

had to take in the software industry date of the art today in software industries that no developer in this world should trust any input data from any forms he will never trust it I mean if he's doing that to be honest fire him because that's that's not the way the attitude you should work as a software developer so if you're getting data from online form what you're doing you're de-poisoning the data so you're doing Escape commands on SQL injections you're removing from the strings anything which could be maybe harmful that's the way how we build software today software developers now let's have a look on AI AI is not a software developing world of course the iei people developing your your core principles of machines but however it's a data analyst world I mean people working with data people not used to be grown up in this hostile software developer world people who need to take the action or let's say the lesson that data is never good data is never something you should trust

so as an industry we need to take the action um that we need to de-poison in data whatever we are doing so the question is how data analysts are looking on that well you're looking on yeah well I can't trust data but of the question like you have some certain errors in the data Maybe um and then you are applying certain filters but listening on that on a way that you need to de-poison data because it could be manipulated from a really malicious attacker perspective as a new lesson we have to take and with the rise of AI with the rise of AI we will learn it okay just for the people who who who are new to the topic I mean this was one of the first very popular ways showing how adverse role attacks are carried out and it's a really a really early example I think 2015 or something like that applied to the Google public image recognition so what I've shown there was just simple like you can use the Google

model which is an oil Network and you can force it to do very strange outputs obviously and the strange output was like recognize what is on this image and I mean at that time it was 57 to recognize a panda to be honest today the Google model is much better but that time it was let's say 57 to recognize that this is a panda um the idea of this adversarial attack was that I can overlay a disturbance on a picture and the disturbance is this pattern you see here so so it was a very rough method it was like creating the disturbance pattern uh calculating this disturbance pattern into the picture which is just leading into a very like the pattern is a is the mathematical overlay on a pixel which leads into the situation that you can't see the pattern anymore as a human as a human eye because it's just really really a slightly difference on a pixel however for the neurons in the in the in the neural network for Google it

was it has a statistic effect so in the effect is like that certain certain Norms are flipping and they're flipping flipping flipping and at the end for this example it was in 99.3 percent confidence that this is a gibbon and then I say well how can AI be so stupid not to recognize that but at the end this attack was pure mathematic in inside now um this was 2015. now seven eight eight years later because this was very popular of course the way how we can apply adversary attacks and how we think about how to defend it has really progressed first of all the models are better so they're more robust that's number one but however on the attacker side we have done as well improvements on how to apply this kind of adversal attacks so for example one one bad messages like finding this kind of patterns has been very very there was an intensive research on this to build mathematic models uh on creating such patterns so we have really roughly automated it

very good to find the right patterns to carry out adversary attacks and then as well the questions like what is the size of the pattern I mean if you look here it was a pattern full completely over over the picture now we can really shrink the size of the pattern for the disturbance so it's really let's say only 16-bit part or let's say 16 pixels in one side on a quadrant which can be overlaid so so as well the attacks are getting have improved amazingly there and as well the question regarding these public models um how attackers can work with this adversal attacks I mean this was a this is a public model this is an open source model uh which gives which is giving us as an attacker as well as certain New Perspectives on how to compromise the model itself so the first question is okay I can put in now an optimized version of this pattern to flip the model to a very strange image recognition number one but what if I

really want to not only put it into a gibbon if I have a very very great idea of what should be the output when I put a certain image as an input and you as someone who is building the model and who are doing the quality management has no chance to find out that there is any manipulation in this model because it will just be the behavior of the model will completely unsuspicious and the way how we are doing this is our backdoor attacks so this can be applied on models for example if they are public models you take these models and you retrain these models for your own purpose then attackers who are doing data poisoning injection like manipulating your trainer date training data have the ability just to manipulate the retrained model and by that I can do a so-called backdoor attack which means if I know what is the right input pattern I can immediately decide the way which neurons should be taken and used in the network and by that create

exactly the output I want so and this is a very uh as well let's say from the last two three years are very intensive area how to design perfect back doors number one and second how you can mitigate risks of back doors in models and there are as well certain ways how you can mitigate this risk at the end we are all talking about robust AI so giving you a little bit a little bit better understanding of what is the critical critical way of this kind of adversal attacks I mean this is very rough this was the way where people were putting stickers on stop signs just to force the image recognition here to make a wrong prediction and this was very rough because you see the sticker I will show you later on examples where you can't see anything like a sticker or and the models are as well growing very strange and mad but this was very rough at the end if you look like on the result of this um so this

is the camera view of an autonomous car you see one stop sign which is not recognized yet but it has sticker on it if you go more near it would be no more recognized as a stop sign very sure as a person um which will lead into a completely wrong decision of the car at this moment um we talk later on how you can prevent from this kind of attacks but these are attacks you can you can apply this kind of attack on any neural network model which has been not built and designed towards robustness against adversary attacks so this is really this is not that the model has been manipulated it's just a natural way how these models are built and how they are vulnerable as I'm from Barton Wittenberg there's another problem we are just putting the stickers all over the world it's Nettie and Barton Wittenberg so maybe if you get road signs which are this way it's distracting of what kind of the stickers will be adversial attacks and what will

be just the marketing stickers of My marketing agency of of the London Wittenberg and they are as well I think very very cool ideas where you can make money out of this I mean we have a big industry about recognizing people all the surveillance industry recognizing phases uh recognizing people and of course if the systems are built and trained and if there are lacking of robustness as well well I mean they're just doing image recognition and they're trying to classify what they find on on these images so of course there is a very good there are some as well very uh bizarre ways how you can deal with that for example this camera system which has been built to recognize persons it's very easy you can just take this kind of adversial t-shirt printing and you are not getting recognized as a person anymore but as a bird one possibility if you like to acquire such shirts and hoodies here's the online shop which will give you the opportunity to buy it for 60 bucks

and have a look if you go to taking what they will tell you if you if you cross the security gates there for facial recognition we have seen as well some strange things like these people this was as well from there but there's a series there's a serious background of course you can do the same for facial recognition just put on the disturbance patterns to be honest it looks very obvious yes but not if you go to the carnival maybe then you have a good chance that this will be not recognized but there's of course a serious background in this and let's say it in that way this is a very rough way how you want to cheat the systems what I will show you next is really the next level because there you will see that it's uh um there are more now sophisticated ways how you can apply adversary attacks and this was an example lab6 has presented last year what it did is well they said well okay we can do an

adversarial attack but we will do this on object recognition so it's not only to do this on a flat picture it's not only to have a road sign on it we'll do that on 3D objects uh and we will show you that we can fool systems and again it's the Google system which will be fooled here uh to recognize a 3D object with net worth tax and um just showing you the example here so this is a turtle a 3D printed Turtle which is not manipulated so um it will be always with a high perception than recognized as a turtle so if you turn the 3D object generated a 3D manipulated object where the disturbance is part of the shape and the color of this turtle and by that they had the ability to fool the system as well on this 3D 3D recognition with a very very high Precision so you can it's really like you can turn it in any direction and the system will always recognize this not as a turtle but

as a rival so um what this example will tell you is just simple like if you think about oh you how to prevent um to prevent my robust models from being um compromised by adversal attacks be sure that the current state of the research is as well very far progressed like we are as well progressed on compromising your models uh in the way you are progressing on optimizing the way how you're building models so this is really really a very actual threat we are facing and now we are coming back to this kind of online forms remember this 2000 2000 ish online form where people were just putting in their names into Data so the question is like if you are an AI developer if you're a data scientist how can I prevent from such stuff how can I prevent this easy attacks I mean this lab6 taxes it's wonderful to see that you can do it it's like someone is printing an SQL injection into an online form that's the same level of attack

why it's working it's working because of course there's a the model itself can be compromised that's a structural problem of the model um however it works as well because the data stream who's coming into the model is not de-poisoned very simple the poisoning patterns the disturbance patterns are always there they are part of the camera perspective so how you can prevent this kind of very this kind of attacking it's very simple it's like it works only because um to be effective the disturbance must be as well transferred very or encoded very perfectly into the image so let's say it from a mathematical level there's a level of information as in disturbance you have to calculate into the image if the level of information is um is there we can manipulate the model if the level of information of disturbance is under a certain threshold it will be no more effective and now we have the situation that there is a balance between what is the accuracy of your model towards robustness and the way how

this model can be disturbed and there's a certain level where you say well if I reduce the information in the picture I'm reducing as well the disturbance which has been calculated in there and that's a little bit you need to try it out and I mean you need testing but then you will find out there's a level of where the accuracy of your model is not really effectively um yeah effectively changing because you're just reducing the level of information of your input data but you're reducing effectively the disturbance which have been placed by the attackers in there by that the adversal attack will be ineffective so your data is depoisoned like you have escaped out the disturbance however it will have an effect on the accuracy of your model and there's one example it's just simple if you take the original picture of this bird there's the disturbance in there and then what you can do is just reducing the bit depth of this picture like do it as a six big call or four

bit caller three bit two bit or you can do this kind of smoothing the effect is always the same the disturbance pattern is reduced as well like the image is getting reduced another big question is like how this will affect the regular accuracy of your model uh and for example if we take the sixth spit reduction um the models are still very accurate on identifying this picture as a board or a specific bird and however the adversary attack is completely out of order it's de-poisoned there's still a pattern in there but as the pattern has been as well reduced it's no more effective so this is a very simple way of uh de-poisoning your input data and be sure that an attacker can't use an adversal attack okay next thing because I have two minutes left but this was the main part now more the fun part I mean as and everybody is talking about ChatGPT enlargement language models and so on we know a lot about how these models are working and as well

what are the problems um towards them and this is more regulatory perspective like what should a model what is the what is the output level um the system should provide you and what we see is like that you need to implement as well a procedure of responsible disclosures and that's what's happening now on Reddit on Reddit we have perfect boards boards where people are just they're playing around with ChatGPT and trying to find out how can I care how can we cheat all the filters input filters who have been applied on GPT to filter out let's say um The Unwanted more unethical or regulatory critical questions um for the bot and if you're an operator in this kind of public space where you're creating this kind of public input and large input the what you need to implement is as well a culture of responsible disclosure like we have in the software industry as well so that people who are finding the weak spots of your model that they are not using it and posting

in on on any Dark Net forums or whatever know that this people have a place where can put where they can post it publicly because they are proud they have found out something which is super cheaty they're found on the cheat and however that you have the team and the responsibility to to review all these disclosures and then Implement your risk mitigation on that on a certain time and for ChatGPT open my eye doing that very well so there's very a very popular board on Reddit where people are just posting stuff like that like um there's an ethic filter in there of course that you should that the system should not provide you any tips on how to discriminate people best giving you advice how to discriminate people okay if you do it as a direct input it was not working if you say ChatGPT give me a python code who's doing that so the indirect way which the system was not able to get a context on that it was working perfectly so writing

your python code on how to discriminate people however what open AI needs to do is to implement this kind of filter Technologies how ever there's still a lot of buyers with this filter apply system as well and that's why you need to implement it really very good to your um to your teams and procedures that's as well one one example so if you ask ChatGPT to tell a joke about a woman it will tell you no that's not very very nice you should not discriminate anyone because the filter has been set from a certain bias obviously the BIOS was male because if you ask the system to tell a joke about men it's very proud to bring you a joke which is not very funny anyway but however you can bypass easily that buyers and by that uh thank you very much thank you very much America I thought that was a fascinating talk and I do want to see if there's any questions for Marco from the group here please gentlemen in the Striped

chat we are very interesting I have a lot of thoughts and questions here but perhaps like two uh thoughts um you know the knowledge GPT they my understanding is that as you are creating the instructions and you fine-tune the models and so on you're adding something to the model and are you aware of um adversial attacks there like is that how does that how would that look like yeah and maybe quick or maybe you can answer like the main problem with opening Ai and is really the big discussion that open my eye is not open so um there's let's say there's one one critical effect like because these are these kind of large models so the question is what are the training and the data sources of this large model so they they have a supply chain for the data I mean that's what we know for sure just to acquire all this data so what they need to keep ensure is that this Supply chains and suppliers are trusted but if you Outsource to

Kenya for example to some to some cheap keyboard workers and whatever doing the classification this is might critical because it's very far away the way how data is classified and preceded is maybe very far away from open AI as a company in the Silicon Valley building the model so for them it's really challenging to keep a secure supply chain and if I would like to inject the back door in this kind of models the best would be maybe picking one of the suppliers who are doing this classification as this click worker click work on models very cheap but in large amounts so this is really a problem as we can't look into the model to prove it in that way it's still very well to say if you haven't trained the model yourself don't trust it okay I was probably referring more to when you do have church ept4 and you do start including instructions and you add to you add to it so that layer of which you can be fully responsible for there

there's a question you know if you don't have the full overview or if you automate those instructions or prompts you know based on customer feedback yeah but then we have this very very prominent example and I think the industry has learned from that was Microsoft tape I mean they were training today over Twitter we know Twitter is never a good idea and this was a wrong setting of the training let's say the the trainings specification and then they were wondering that they was getting very rude uh within a couple of hours because they were never thought about that there were boards online in the internet where people are just as grouping themselves like to make a crowd attack on a Model we haven't seen any crowd attacks on on ChatGPT yet and maybe I think the model will be as well very robust because it's much bigger than K on its time but however there are always ways of um attacks you don't have in mind like Microsoft hadn't in mind with Tay that people

could go for Reddit and say well let's make a fascist racist really ugly chatbot out of day but however we see when I did have another question now I started to change that to another question but I was thinking of when you have the um if you retrain and introduce a backdoor attacks then basically though the one that's um training the model is having control of the data so I'm not quite sure how that would work other than having a type of Industry spy that starts to introduce Data before the data you know before this train if you have capable Engineers that should be able to see what you that was one question yeah but um I can answer that like if a model is compromised let's say if you assume that a model is compromised by a backdooring if you have access to the model I mean the question do you have access to the model yes or not um you have the ability to find out if there are structure if there are

malicious structural elements in the model you can identify now with a very high accuracy if there are possible backdoors in your model in there that's a good that let's say that's a good news as well like from the current research perspective we have as well the let's say some certain algorithms and tools for the retraining to turn the model back again in a status which is let's say less malicious in this way so where the back door will be no more effective but however the problem is the retraining like you're retraining and de-poisoning uh you need to identify where was the injection of the back door because otherwise it will be just it's like it's like a malware I mean you have applied the malware removal it will become back again with the next iteration now thank you very much I'm mindful that we need to crack on with the next session but thank you very much and for your questions thank you right so from one cyber security expert

Rise of AI 2027 · Berlin · May 2027

Be in the room for the next one.

Ten years of talks started as conversations in one curated room in Berlin. The next edition brings 300 selected leaders together in May 2027.