At Rise of AI 2023, Mirko Ross, CEO of asvin, argues that the AI industry is repeating the security mistakes the web industry made around 1997-98, when developers trusted any data entered into online forms until SQL injection attacks taught them otherwise. He says AI systems face an equivalent lesson with data poisoning: attackers who manipulate training data or exploit a model's neural-network structure can force a chosen, non-random output, for example making an autonomous vehicle ignore a stop sign. Drawing on research asvin began about four or five years before this talk, well before ChatGPT-driven public debate, he traces the evolution of adversarial attacks from early 2015-era image-recognition exploits, through backdoor attacks planted during retraining of public models, to 2022-era work fooling 3D object recognition from any viewing angle. He demonstrates a practical defense, reducing image bit-depth or smoothing to strip out adversarial disturbance patterns while preserving accuracy, and closes by discussing prompt-based jailbreaks of ChatGPT's content filters and the need for a responsible-disclosure culture modeled on the software industry's, plus scrutiny of AI supply chains such as outsourced data labeling.
Key takeaways
- Mirko Ross argues the AI industry is repeating the web industry's 1997-98 mistake of trusting all input data by default.
- Data poisoning forces a model toward a specific attacker-chosen output, such as an autonomous vehicle failing to recognize a stop sign, rather than random error.
- Reducing an image's bit depth or applying smoothing can strip out adversarial disturbance patterns while largely preserving a model's normal accuracy.
- A 2022 attack fooled Google's object-recognition system into misclassifying a 3D-printed turtle as a rifle from any viewing angle.
- Backdoors planted during retraining of public models are a realistic risk when companies outsource data labeling to less trusted suppliers.
Detailed notes
AI's supply chain problem, and why asvin started early
Ross says asvin focuses on cybersecurity in supply chains, including the data and model supply chains involved in building AI: how data is collected and prepared, how models are trained, and how they are securely distributed. He says the company began researching how to compromise AI systems roughly four or five years before this talk, anticipating that AI would run into the same kind of supply-chain trouble the software industry faced around cybersecurity, well before the ChatGPT-driven public debate. He references the White House meeting with Google, Microsoft and OpenAI as evidence that trust and robustness have become central industry concerns, and warns that if the industry does not build stable foundations itself, regulation will step in and limit its possibilities.
What data poisoning is and how it forces a chosen output
Ross explains that data poisoning injects manipulated data into a model at one of two points: during training, for example placing backdoors, or through adversarial attacks that exploit how a trained neural network's statistics work, flipping specific neurons to move the model's output in a chosen direction. He stresses this is not the same as ordinary noise or error; the goal is a specific, attacker-intended output, illustrated by an autonomous vehicle's camera system being manipulated into treating a stop sign as something the car should drive through rather than stop at.
Learning from the early web: never trust your input data
Ross compares today's AI industry to the web industry around 1997-98, when developers had no concept that a name field in an online form might contain a SQL command instead, leading to the first code-injection attacks; the resulting lesson, that no developer should ever trust input data, became standard practice in software development. He argues AI needs the same lesson: because AI development is largely a data-analyst world rather than a hardened software-development one, practitioners are not used to assuming data could be malicious, and need to learn to 'de-poison' data the way web developers learned to sanitize form input.
From stickers on stop signs to 3D objects and clothing
Ross traces the escalation of adversarial attacks: a well-known 2015 example fooled Google's public image-recognition model into misreading a panda as a gibbon with 99.3 percent confidence using a calculated pixel-level disturbance pattern overlaid on the image, a method he calls purely mathematical. Since then, attack patterns have become far smaller and more automated, shrinking to roughly a 16-pixel patch, and researchers have shown stickers placed on physical stop signs can stop autonomous-vehicle camera systems from recognizing them at all. He also cites adversarial clothing, sold online for about 60 dollars, that can make a person invisible to person-detection cameras, misclassifying them as a bird, and adversarial patterns that can defeat facial recognition. The most advanced example he cites is from 2022, where researchers fooled Google's object-recognition system into consistently misclassifying a 3D-printed turtle as a rifle from any viewing angle, by encoding the disturbance into the object's shape and color rather than a flat image.
Backdoor attacks and a practical defense
Ross explains that when companies retrain public models for their own purposes, attackers who can inject manipulated data into that retraining process can plant a backdoor: an input pattern that reliably triggers a specific, hidden output while the model behaves normally otherwise. He demonstrates a defense against pixel-level adversarial attacks: reducing an image's bit depth, for example to six bits, or applying smoothing removes the fine-grained disturbance pattern an attack depends on while leaving normal classification accuracy largely intact, since the attack needs a minimum threshold of encoded disturbance information to work.
Q&A highlights
Asked about fine-tuning and instruction layers on top of models like ChatGPT, Ross says the central risk is the trustworthiness of the underlying data supply chain, including outsourced data labeling and classification work, for example to low-cost contractors, since a company far removed from that labeling process has limited ability to verify it, and injecting a backdoor there is a realistic attack path; if you have not trained a model yourself, he says, you should not fully trust it, though access to a model's structure allows detecting likely backdoors with high accuracy and retraining to reduce their effect. He cites Microsoft's Tay chatbot, trained on Twitter, as an example of an unanticipated attack, a coordinated public campaign to make the bot say offensive things, distinct from data poisoning. On content filters, he shows that ChatGPT's ethics filter blocked a direct request for discriminatory advice but did not catch the same request rephrased as 'write Python code that does that', and separately blocked a joke request about women while allowing one about men, illustrating an inconsistency in how the filter had been tuned. He argues that operators of public AI systems need a responsible-disclosure culture, similar to the software industry's, so people who find these weaknesses report them rather than posting them elsewhere.
Quotes
If you haven't trained the model yourself, don't trust it.
Mirko Ross
Data is never good data, is never something you should trust.
Mirko Ross
Questions answered in this talk
What is data poisoning in AI?
According to Mirko Ross at Rise of AI 2023, data poisoning is the injection of manipulated data into a model, either during training or by exploiting a trained neural network's statistics, to force a specific output the attacker wants rather than random error.
How can adversarial attacks on image recognition be defended against?
Ross demonstrates that reducing an image's bit depth, for example to six bits, or applying smoothing removes the fine-grained pixel disturbance an adversarial attack depends on, de-poisoning the input while largely preserving the model's normal classification accuracy.
Why does Mirko Ross compare AI security to early web security?
Ross argues AI developers today are repeating the mistake web developers made around 1997-98, when no one anticipated that form input could contain malicious code (SQL injection); AI needs the same lesson, that input and training data should never be trusted by default.