hello everyone my name is Florian and I will speak today about my experience and Avenue on breast cancer detection and what we achieve in a couple of experiments so I hope that yeah all right so we start with a problem statement if you look a little bit on the breast cancer diagnostic stages there are three of them starting from stage one to stage three in the agnostic we spend a lot of time in stage two which is clinical evaluation diagnostic and and staging typically this stage the stage two is taking something like four weeks could be taking some some cases as well six weeks or more and this is very important because we have to actually go to the stage three of giving the treatment very fast especially if the cancer is actually prolonging and is moving fast so I leave it on on the background on those stages first one when you have an issue first of all you have to do a clinical evaluation right so we have to go to the or
medic doctor and and just have something like you know early evaluation and after that if something goes wrong there you have to go and do a mammography which typically some elements after 44 is just doing regular one-year interval but you have to do this and if then they find something then say okay you have to go to the next step which is a wilt result in some countries they skip the ultrasound step and they go to the next one which is MRI that will give you a 3d perspective you know what was going on there more accurate and if that confirms that you okay there is a tumor there the question is what kind of tumor is Right smiling is welling in what type that's very important and then you go to the next one which is you know you have a surgery a small surgery which is a biopsy to take small piece of the tissue from from there which is typically is guided by by the doctors and then they analyze that into
a lab for three weeks in lucky cases for two weeks but in general it takes three weeks in some other cases for weeks where you have to you know wait for for an answer and in the lab test you get you know the type of the to more you get the grade the receptor neurons and based on these you actually get treatment this is very very important it's a crucial step in detection of the type of the cancer that is there especially for example if it's her2 positive and that's something that the lab tests will actually give now I leave it on the breast cancer diagnostic we see here this is coming from the National Cancer Institute in UK five-year survival rate so early detection you have 90 percent by statistics according to them to in order to be cured and then advanced stages which is no stage three or even four very very few chances as a survival rate and this is very crucial to have a detection earlier right so that if you
look a little bit in the detection part the lab is taking the most of the time of and where you spend a lot of time assuming that all these let's say appointments where the doctors are no well done now I've been here on the stage of the breast cancer if you are having in early detection we have 98 percent survival rate but as the cancer is progressing and you go earlier or you are getting less a chemo or radiotherapy after a while then if the cancer is progressing so fast it can be spread into the lungs into the brain those are typically for the breast cancer the next organs where they spread so it's very crucial you know to cut the say the two more the type of the tumor and to give the right treatment as much as fast as we can can do so one of the question was can we accelerate this process with deep learning can you accelerate part of it at least and one that you know we are spending
in the Box in the histopathology where you have a the tissue you actually look on on the microscope to see you know what kind of cancer is and so on and to detect on the type so to have a let's say a better look on that we have to a little bit step aside to say okay I have this great technology this convolutional neural networks can I apply them what is the legislation around that and this is a code from FDA in the United States defining software as a medical device because ultimate we want to have an algorithm to trained to recognize say in an image a type of a cancer so that falls into the software as a medical device where you can see here the definition the international medical device regulation form and the RF defined software as a medical device as a software intended to be used for one or more medical purpose that performs those purposes without being part of the hardware medical device right so if you take this a
little bit say okay I want to apply this then the question is what is the general context and again you can go ahead and say okay I want to look again to FDA say what you know how they are seeing the applying of those algorithms and here is the product life cycle regulation approach for AI and based on the software-defined medical devices so not only that you have to create a device but you have to maintain that algorithm you have to monitor that algorithm continuously to get all these kind of you know metrics you have to actually track and log anything that then the algorithm will produce so it's not that easy that I'm building an algorithm and then I'm putting in that in the production so you need to have the whole pipeline in place the whole implementation of the a helps to have this alright so a little bit on the infrastructure stack obviously we started with the health care regulations and from there we have to understand you know the data that
we have to handle the governor's around the data people involved policies all these kind of things and then we ought to understand from where those those that are coming so we need a data lag to you know to have those images stored and we can take them from there for training and and so on we have to distribute the computing we need those machine learning algorithms and and at the end we need all these furnace explainable AI were up to understand if we have biases in in the data sets and how we can remove them so all those aspects in general are very complicated for for the doctors for the researchers which are on the medical field to deal with them right and one of the thing was okay can we simplify here something so in order to answer to this we had to take a look a bit and breast cancer diagnostic pipeline in a simplistic view from the image perspective we have first of all like an engine the mammography part MRI and
then we have the biopsy where we have the histopathology in those cases we produce images and metadata the metadata is actually when you have an image from let's say my MRI you have a deacon file format and inside of the header of the file you have a couple of things like the age the sex of the person the unique a variety and and so on so forth so those are metadata which are very important as well to preserve so ultimate we want to capture those images in various stages and then have a deep neural network to train this network to recognize the type of a cancer either we do segmentation of the detection or we do classification as well we have to need to have information the age of the person because that's very important and typically is inside of those specific file formats and ideally we want to give a prediction to the data say well the network can predict that with so much accuracy that this is cancer this type and and and
probably this size and so on so we are concentrating a little bit on the most time-consuming part which is the histopathology part now another thing is those file formats are coming in specific things especially if you if you have more complicated more complex things then again you have to deal with let's say other kind of formats for example brain you have the nifty formats so you have to pre-process those images and then extract from those images the metadata and the PNG files or the JPEG files that we need to train those neural networks and then we actually go further to this stage where we have to do the labeling right so we have to do the labeling process we have to this typical done by doctors and specialists and this is on will create specialized let's say data sets for Mama for MRI histopathology and so on and then we want to actually detect the bias and to remove them from from there if there are and we use a 360 toolkit for example within
IBM and we have a lot of iterations until we get those neural networks on those on those things trained at the end we want to actually have an image we want to classify that image being a tumor a B or being a normal case and then we want to do more things like localizing that having the size of the tumor if possible especially devolution and then we want to extract the context you know the age of the person and so on from the existing metadata that we acquired and then we want to have some api's where we want to say okay we want to have two more prediction and this is the type of the tumor we predict and this is the accuracy this is intersection of Union and all those kind of terms and obviously at the end is the two more description and this is provided to the doctor and we can apply this to mammography MRI histopathology that that will be the goal now from the perspective of earrin so putting this
in the production we would like to put those in Finnish scenarios either running on a server in the data center or either put it in a medical device in a small device in embedded devices on scenarios three and four those are the big two things one which are can run on the other Center and one run in the in the hospitals all right so from the hardware we use a Schneider to system it's an IBM deep learning system the reason we use that is because for complicated things where you have to investigate for example the the cancer which has been spreaded on the tomorrow has been spent on the brain we use better accuracy and more images using this kind of technology you see here an example 250 650 images per second compared with 65 images now from the software perspective we used power I vision and you see left for training and inference and right only for inference and this is a software that is simplifying the whole landscape this is focusing on providing
the best experience of doctors for labeling training without taking care of all the aspects of deep learning and of the infrastructure beneath so I'm going to speak about how we use these things right so the goal is to actually automate the whole pipeline as much as we can with the technology that is available its general available on Prem because we see you know hospitals that they don't want to send data due to regulations into the cloud right so the product itself is actually providing a web interface in a lot of api's where doctors they can go there and find the images label those images do some training by just a couple of clicks for classification of detection and in the future segmentation semantic segmentation and in addition we provide as well informations for if you do inference if you want to test then those models and see you know how they perform you know how good they're so we provide as well some parts of the those models are pre-trained but we can allow as
well things to be added additional on top of the product as you know custom neural network models all right so the experiment we did it was by using the VRI lab Department of Informatics in interest field that were there quite kind to give us their data set for his breakfast called for histopathology this was having more than 10,000 images microscopy images and the goal was to create a neural network model that will identify those classes you can see it here of the breast cancer from the tissues so bending and modeling but as well you know other noses we might know my exactly were interesting to train a natural for mostly for our typical cases where the doctors in the lab they spend a lot of time so we use for the neural network model the classification Google net class activation mapping where in especially on a high resolution on 40 X magnification of those images we can actually identify the part in in a specific class and this is tends to be very very good
in terms of accuracy so a couple of screens here about the experience so you connect us as a doctor to the system you start your loading your datasets creating the categories that you have you see those are the images that we have given on the left side you will see the classes for each image and if you want to view the image and have more deep knowledge and check those images in addition to what has been given and labeled by it some others you can go there and then you can which is just a button you can start a training and the network will start training and then you do this a lot of iterations until you get it to the right accuracy in the initial run we got 87 percent accuracy obviously this was not enough so we said okay can we do something else right can you change something and there is a technical documentation that has to be used very careful in those cases so we used to flip the images to
a horizontal or rotate them with a specific angle to generate more images from what we had to actually train the network so when actually you do this automatically the system will augment the images and you see them here with a from augment so again we train that and we achieved 93% inaccuracy in detecting those classes and that's is the experiment that we have done and we believe that inference in under one second to give you 90% accuracy is close to the human error which is five percent so we still have to present that I think can be solved in in the field there so in the inference when you test this with the validation images you are getting not only the prediction you can actually establish the threshold and you have as well a heat activation mapping so we can give you this map so can't stand what were the most important features in the class that been contributing to the image we classify the way it was so moving further what's next right so
one of the problems that we see there because this is a continuous process is you know how it can be involved a little bit into the mammography part right mammography is a little bit different because of the you don't have color images majority of them so we are just grayscale and that's a challenge for those newer networks to pick them so until now Mike's been wasn't up to 91% accuracy still not enough but we see promising results so one of the thing is to take as well in the consideration this part of the mammographies MRIs as well we are looking as well to do posting feelings and orchestration so when you have something like a patient that and you have the image from the mammogram from MRIs and from the histopathology you can run them as three separate inference processes and then you can do the scoring including the aging the age of the person the sex and and all these kind of things so you can have a less a better prediction technical consideration
the other part especially if you try to consider the cases where we have unfortunately patients that they have let's say lung cancer which is the metastasis and then the that is treated by a doctor which is specialized in lungs and then we have the the doctor which is specialized in the breast cancer and typically have to find the time to speak each other say okay what we do with this passion so for those kind of cases we can actually have a platform where it can use it you know for to do this kind of scoring among you know multiple disciplines by using deep learning in addition we can we are interesting in actually the technology as well lesions so did this can we believe that can be done and move moving these things to smaller devices to embedded GPUs and bedded FPGA is what can be run somewhere on a desktop up to the moment that you have to understand I need to put this in the production what that means you know how I'm
monitoring what those algorithms will do and will predict to advise the doctors in a way or another one and this is go this goes into the air operations how you actually deal with those kind of things not only on building that model that will actually predict something and that model has to be put into the context into the age of the person into the type of the image that has been used where has been used and what kind of position and so on and so forth so we're actually trying to give a platform that can be used for those kind of cases on Prem by the doctors by the clinicians they can train their own network to advise them right to find those cases we don't try to give them a model and say use this model will be the best for you right you build your model based on your experience but based on your labels and the labels they have to be verified as well by other entities to be sure that those
are correctly labeled the way there are because those networks I would say are very dummy if you don't give them the right data they will learn not to predict things that will be accurate in the or will not generalize well in the in the production questions so far yeah we're going to get your microphone hi nice talk I'm puja from nature research I'm just curious to know if you are extending your application to any metabolic diseases like liver fibrosis snaffle D cardiovascular diseases no the answer is no that's why we were trying to include us well the context other kind of things into dog's neural network models as it as a say different pipeline right so one of the thing is we're taking the images we are trying to teach those networks for those kind of let's say typical cases but for our typical cases you have to understand us with the context of the person so we didn't go that far we are trying to validate those this power vision software and how can
be used in such cases are there more questions no oh there's one more thanks for the talk I'm just curious about what is the need of real real-time feedback or you know why do you need real-time analysis for for such a critical cancer detection or or something submitted that right so the way I perceive this is that for example when you have an electronic microscope right and you actually look for the cells on different magnification right as human at certain point you have duds right and typically you speak with one of your colleagues but in general the algorithms that is are displaying you the image are just algorithms so the need of having a solution like this for the people on the field if they give them real-time insights so they can okay so the algorithm is telling me that at 40% for the X magnification I have less a fibroadenoma the question is okay I should go and look a little bit more on a high resolution you know 400 X 4 more in
rather they just say okay no it's okay right and and especially for cases where there are countries where they don't have labs across the country so there's a long queue you know for those kind of cases how you deal with them okay okay we wouldn't have time for one more question if there is one okay here she says coming to you with a microphone so everybody can hear you thank you for the nice talk can you give a little bit more explanation how doctor will use this for example if a doctor has one or two pictures with him or her so they upload to your system and use your data set train data set and then later on check their voices well to a system to make a decision so there are there are two ways so assuming that the system is in the hospital because we we believe that we this system should run in the hospital trained by the hospital based on their experience and the reliable data the doctor has two possibilities
other to integrate let's say the MRI system VI API with the system so it will get integrated via Vialis if they use G or they use other company if you had that we are the parks or because we have an API just send an image and you get the prediction or ii will take the we'll export the image from from decom or from if and then we'll actually upload the image via web interface to the system will then we'll give them a prediction so there are two ways depends of the situation in the hospital