Rise of AI Conference 2018 · Berlin

GDPR & AI: The Twain Shall Meet

Labhesh Patel · CTO & Chief Scientist at Jumio

At Rise of AI 2018 in Berlin, Labhesh Patel, CTO and chief scientist at Jumio, explains how the EU's General Data Protection Regulation, taking effect a week after the talk, will reshape machine learning workflows. He describes Jumio's business verifying identities online by comparing a selfie to an ID document using machine learning and human review, which puts the company squarely under GDPR because it handles images, names, addresses and birth dates. He outlines the regulation's two fine tiers, up to 20 million euros or 4% of global revenue, and explains that GDPR now makes data processors, not just controllers, liable for breaches. Patel lays out practical steps toward compliance: encryption, human review of automated decisions under GDPR's right to explanation, configurable data retention, and 72-hour breach notification. He introduces 'compliant machine learning', building privacy and security into every stage of a pipeline, from data acquisition and tagging to training, deployment and evaluation, and explains why data from different customers cannot be commingled to train one shared model under the regulation's data minimization principle.

Key takeaways

  • Labhesh Patel argues GDPR applies to any company processing EU citizens' data, regardless of where its servers are located.
  • GDPR fines reach 20 million euros or 4% of global revenue, and now make data processors, not just controllers, liable.
  • GDPR's recital 71 lets users contest fully automated decisions, requiring machine learning systems to give human-interpretable explanations.
  • Patel calls for 'compliant machine learning' that builds privacy and security into every stage of the ML pipeline.
  • Data minimization rules prevent a processor from training one shared model on multiple controllers' customer data.

Detailed notes

Jumio's identity verification business and why it sits inside GDPR

Patel describes Jumio's two products: authenticating a user's ID document and verifying that the person present in a transaction matches that ID, using machine learning, facial recognition and human review. Because this involves images, names, addresses and birth dates, GDPR applies to Jumio and to any company handling EU citizens' data regardless of where its servers are based, once the regulation takes effect on 25 May.

Fines, controllers and processors

He explains GDPR's two fine tiers: a lower tier of 10 million euros or 2% of global annual revenue for failures like inadequate data security design, and an upper tier of 20 million euros or 4% for specific infringements such as shipping personal data to another country without safeguards. He notes the UK's Information Commissioner's Office hired more than 200 people to prepare for enforcement, and that GDPR newly makes processors, not only controllers, liable for penalties, so the two now have to manage compliance together.

Practical first steps toward compliance

Patel lists concrete steps: baseline data encryption, for example using PCI DSS as a security standard; human review of automated decisions, since GDPR's recital 71 lets a person contest a purely automated outcome such as a denied mortgage application and requires a human-interpretable explanation; configurable data retention so a system can delete a user's data on request; and breach notification, since GDPR's definition of a personal data breach is broader than usual and requires notifying authorities and affected users within 72 hours.

Compliant machine learning across the pipeline

Because GDPR does not mention machine learning directly, Patel reads its right to erasure (Article 17) and right to explanation (recital 71) as requiring 'compliant machine learning': building privacy and security into every stage, from data acquisition, tagging and training to deployment and evaluation, keeping workflows interpretable and ensuring a user's data and its residual influence on a model can be removed on request.

Why customer data cannot be commingled

He explains GDPR's data minimization principle prevents a processor from training one shared model on data from multiple controllers' customers, since a user who consented to improve controller A's product cannot have that data used to improve controller B's product. Human-derived insights, such as a data scientist manually noticing an attribute and coding it into a second model, can be transferred, but a trained model itself cannot, which multiplies the engineering work needed per customer.

Quotes

GDPR is here now, and I think there is going to be a mad scramble to get compliant.

Labhesh Patel

My personal data should not be lingering around in your servers for longer than it is absolutely necessary.

Labhesh Patel

Questions answered in this talk

Does GDPR apply to companies outside the EU?

According to Labhesh Patel at Rise of AI 2018, GDPR applies to any company that processes EU citizens' personal data, even if its servers are based in North America, once the regulation took effect on 25 May 2018.

What are GDPR's fine levels?

Patel explains GDPR has two fine tiers: up to 10 million euros or 2% of global annual revenue for security and design failures, and up to 20 million euros or 4% for specific data protection infringements.

How does GDPR affect machine learning pipelines?

Patel says GDPR requires 'compliant machine learning', building privacy and security into data acquisition, tagging, training, deployment and evaluation, since a user's right to erasure and right to explanation must be honored at every stage.

Mentioned

GDPR, data controller, data processor, right to erasure, right to explanation, data minimization, compliant machine learning

Keep exploring

More from Rise of AI Conference 2018

More on Enterprise AI & Adoption

Transcript

Auto-generated transcript, not edited. · 2,594 words

Show the full transcript

hi guys so my name is lavash patel and i'm the CTO and chief scientist at jr corporation so the conversation that we are going to have today is about gdpr and how it is going to cause a massive change in the AI and the ML workflows that you might be running so a little discreet disclaimer out here I am NOT a lawyer but we have worked with a lot of lawyers because of the kind of workflows we run and the kind of personal data that we have to take care of and we need to be compliant we are going to talk about what we do a Juneau just in the context to give you a context of how all this works together with gdpr we are going to be talking about some of the key elements of gdpr for probably the 0.5% of you who have not heard about gdpr we are going to talk about some things that you can start doing that can help you move towards compliance and I'm going to touch

on compliant machine learning this is something that we have started talking about it's a new way of running your machine learning workflows to be more compliant to any of these regulations so at Union we are in the business of establishing trust online so the usual methods of authentication like passwords or SMS codes or knowledge based authentication zhh they have proven to be vulnerable to attacks and governments regulators enterprises they are looking for new secure and robust ways of definitively establishing users online identities so our sweet spot is connecting our users online and their offline identities we use a mix of machine learning and human experts to quickly verify and authenticate an ID coming through our system the process plugs into the currently existing workflows of enterprise processes and quickly tells the enterprise whether or not incoming ID is fraudulent or not our other product identity verification is about establishing the presence of the actual human behind a transaction so it tells the enterprise that well this person who is actually transacting is the person whose

ID was just sent in and we use biometrics we use facial recognition and a human element to compare a selfie to the picture which is present on your ID so as you see we are dealing with personal data of the wazoo we have images we have names addresses birth dates all kinds of stuff so this puts us right in the midst of gdpr conversations so the essence of gdpr is about protecting the personal data of EU citizens now this applies to any company that transacts with you citizen so even if the company's servers are based out of North America it if there are GU users that are using that service and if that service requires in any way collecting of EU citizens personal data it will come under gdpr purview and this goes into effect on May 25 about a week from now so hopefully everybody out here is already up and running with all the compliance measures it is going to be really expensive to be in non-compliance with GDP are there there are

two levels of fines the lower level is 10 million euros or 2% of your worldwide annual revenue and the upper fine is 20 million euros or 4 percent of the worldwide annual revenue whichever is greater so lower fines they relate to data security standards or design falls within your system there are certain things that you might not have gotten around to implementing that will fall under the lower fine regime upper fine they relate to specific infringement on data protection data security rights of of the data subjects so for example if companies are collecting a user's data or shipping it off to a different country or an international organization that stuff would fall under the upper fine regime now the level of fines are going to be dependent on the severity of the of the infringement it's going to depend on the length duration it's not black and white but what is black and white that the ICO it's not ice here that you're thinking it's this is the Information Commissioner's Office they hired more than

200 people in the last few months and this is all about getting ready for making sure that people comply and and and the enforcement of GDP are the difference the key distinction between controllers and processors in within the GDP our regime is that controllers they determine the purpose and the means of processing of the personal data and the processors are the actual ones who process the personal data now why is this important so up to now it was controllers who had all the responsibility for adherence to regulation and they were the ones who could have been fined for breaches but with this coming in place with GDP our processors are also going to be liable for penalties so now controllers and processors are in this together they have to work together controllers have to manage their processors very closely and processors have to make sure that they also have all the infrastructure for for compliance so how can we start on the journey of getting GDP our compliance well there are some things that we

can start doing number one data encryption so having a base baseline for data security is a good idea if any one of you know about PCI DSS that is a pretty good baseline for data security that can help you to as gdpr compliance from the data security side of things human review is an interesting one there is a recital 71 of gdpr states that some humans can opt out of completely automated workflows so let's say you have a mortgage application machine learning system which comes up comes back with a score for whether or not a user should be given credit well you cannot just say that I'm denying you the mortgage application the user has the right to come back and actually ask you and explain and challenge you on the decision and it cannot be the answer cannot be well the fc7 layer of my cornet says that there the weights artists and so that's why you got denied you have to give a human interpretable answer so if you have a human in

the loop system where there is a human is touching the workflow somewhere you are closer to compliance data retention you're the both the controllers and processors need to have infrastructure in place so that if for some technical reason your system goes down you can respond very quickly get the user data back online very quickly and you need to have a configurable data retention policy so a user should be able to come into your system and say that well delete my data after X amount of time and you need to your system needs to be able to comply with that gdpr is very very strict about the breach notification part of it so the idea out here is that there is a concept of personal data breach within the new regulation and the personal data breach this concept is much wider than the usual types of breach that we talked about within this regulation a personal data breach can happen just by unauthorized access to the data and here also having a baseline data security standards

such as PCI helps you along a lot because then you get a lot of the encryption at rest encryption during transit logging monitoring a lot of these things are already in place if not you might have to do that because that will minimize your chance for you know having a data breach and then complain machine learning GDP error does not specifically talk about machine learning which is very annoying but they have these articles like article 7 articles and 13 which is the right of erasure and recycle 71 which is the right to explanation which which kind of gives you the hint as to what the spirit of gdpr is so with compliant machine learning you want to make sure that all the different steps within the machine learning workflow are compliant to the spirit of gdpr basically this is something that we are working towards and compliant machine learning is all about building and data privacy and security at every stage of the process starting from data acquisition tagging processing model training deployment evaluation that

entire pipeline how do you look at each of these steps with the lens of data privacy and security again article 13 which is the right to explanation makes it difficult because you have to make sure that your workflows are interpretable the right to disappear which is article 17 now your machine learning algorithms need to be flexible enough that if somebody in the system says that okay forget me then you have to have the infrastructure in place so that they are complete their memory residual memory is completely removed from your machine learning pipeline data minimization is another principle that gdpr talks about and the entire idea out here is the footprint of your data should be as small as possible so my personal data should not be lingering around in your servers for longer than it is absolutely necessary so if you're training computer vision model and it takes several weeks to do that and you have to run multiple epochs on the on the data then you have to really think about how to do

how to do that and whether you have the right consent from the user to do that and you cannot really commingle data from multiple customers so this is a very key point and introduces a lot of friction in the current way of doing things so for example let's say you are a data processor and you have a few controllers as your customers so traditionally what you were able to do is you will get data from all the controllers and you can train a machine learning model or an AI model which aggregates all this data and now you have one model and you'll open up a REST API maybe you have it on TF serving or something and you'll be serving out that model to various customers well now it becomes tricky to do that because the controllers the end users of controller a have given right to controller a that it should be the data can may be used for improving the product which is which is being provided by my controller a so now

this user cannot it's their data cannot be used to improve the product for controller B which is what would happen if the processor were aggregating all the data so now you can imagine for each of the steps whether it be the data capture step the pre-processing step training the model step testing evaluation everything you have to figure out a way of doing it n times for for your different controllers and if you do not have a huge data velocity from your controllers from from some specific customers then guess what their models are not going to be their troubles so this introduces a lot of friction in the current ways of doing things and within each step if we look at the step of data capture what is allowed is if I create a model on controller a if I have some human relevant insights I can hard code those insights into model B let's say I train a decision tree on controller a and I find that there is this one particular attribute which reduces

entropy the most now I can if as a human I know that I can go over to the other table where I'm doing model B and explicitly code in that attribute and be done with it if I'm you using human as a conduit for passing this information I am fine I cannot just take the model and start with that model on the second one you cannot really do that so human insights can be transferred and when you're doing the pre-processing every single step of let's say you have a human workforce which helps you in tagging data now when the data is being transported to their workstations it needs to be encrypted the tags that they come up with they need to go into an encrypted store the channels all need to be encrypted end-to-end we want to minimize the data breach because if this human just takes a picture of this particular personal data on their screen and tweets about it well you are in you are in breach and you need to make sure

that this doesn't happen so this might require physical security frisking of these people before they go ahead and start tagging data so a lot more rigor needs to be put in place model training again we usually machine learning you will run multiple epochs over your data and between those efforts the data is just sitting on disk you need to make sure that that process is encrypted and if it is coming from the loud if you have a cloud server where the data is getting stored you need to make sure that the pipes are encrypted so you just need to start thinking about privacy and security at each of these steps and crypting your API keys etc is very important too so in summary they way you want to get started in the journey for moving towards compliance is start with a baseline security measure like PCI DSS or some security standard you might want to have security audits pentesting Billie encryption audits independently verified audits human-in-the-loop workflows are probably good because at least some of

these articles that talk about explanation and interpret ability of whatever you are talking about you are able to provide that in in time also when somebody comes with a specific request that why is this decision taken you don't have infinite time to come back so unless you have a workflow in place that this reply can come back within a few days that might cause problems we talked about complying machine learning we're looking at every step and building in data security and protection within each step can help you get compliant data retention you want to have an independently audited data retention policy where it should be configurable users should be able to come into the system and say well I don't want to want you to keep your keep my data and you should comply with that request and breach notification where you have to once you find out about a breach you have to notify the appropriate supervisory authorities and the data subjects who are actually affected within 72 hours so in summary gdpr is

here now and I think there is going to be a mad scramble to get compliant but in the long run I think it's good because users this will create an environment where users will actually feel empowered and last but not least I'm hiring if you guys are into deep learning computer vision a I go to junior comm drop us a line we are hiring all over the world thanks a lot

Rise of AI 2027 · Berlin · May 2027

Be in the room for the next one.

Ten years of talks started as conversations in one curated room in Berlin. The next edition brings 300 selected leaders together in May 2027.