Rise of AI Conference 2019 · Berlin

Breast Cancer Detection with IBM PowerAI Vision

Florin Manaila · Senior AI Architect and Inventor at IBM Systems Hardware Europe / Member of the IBM Academy of Technology

At Rise of AI 2019 in Berlin, Florin Manaila, Senior AI Architect and Inventor at IBM Systems Hardware Europe, presents an IBM proof of technology for detecting breast cancer from MRI and histopathological images using convolutional neural networks. He explains that diagnosis today moves through clinical evaluation, mammography, sometimes ultrasound, MRI and biopsy, with lab analysis of tissue samples alone taking two to three weeks, and that early detection is linked to much higher survival, citing National Cancer Institute UK figures. He frames the work within the regulatory category of software as a medical device and describes an infrastructure stack covering data governance, a data lake, distributed training and bias detection. Using IBM's AC922 hardware and PowerAI Vision software running on-premise in hospitals, his team trained a classification model on more than 10,000 histopathology images, reaching 93 percent accuracy after image augmentation. He describes next steps including extending the approach to mammography, combining multiple imaging modalities with patient context, and building AI operations for monitoring deployed models, while stressing that hospitals train their own models on their own verified, labeled data.

Key takeaways

  • Manaila says histopathology lab analysis alone takes two to three weeks and is the most time-consuming step in breast cancer diagnosis.
  • The team's model reached 93 percent accuracy on histopathology images after augmenting the training data by flipping and rotating images.
  • He frames the algorithm under FDA/IMDRF rules as software as a medical device, requiring ongoing monitoring and logging after deployment.
  • The system runs on-premise in hospitals on IBM AC922 hardware with PowerAI Vision, since hospitals are reluctant to send data to the cloud.
  • Mammography detection is harder than histopathology because images are grayscale, reaching about 91 percent accuracy so far.

Detailed notes

The diagnostic bottleneck

Manaila describes the three stages of breast cancer diagnosis, noting that stage two, covering clinical evaluation, diagnosis and staging, typically takes four to six weeks. Patients move from clinical evaluation to mammography, sometimes an ultrasound, then an MRI for a 3D view, and finally a biopsy whose lab analysis alone takes two to three weeks and determines tumor type, grade and receptor status such as HER2. He cites National Cancer Institute UK five-year survival statistics showing that early detection is linked to survival rates around 90 to 98 percent, compared with much lower rates for advanced stage three or four cancer, and identifies the histopathology lab step as the most time-consuming part of the pipeline, asking whether deep learning can accelerate it.

Regulation and the infrastructure stack

Because a trained algorithm that recognizes cancer in an image falls under FDA and IMDRF definitions of software as a medical device, Manaila stresses that building the model is only part of the work: the product life cycle also requires continuous monitoring, tracking and logging of what the algorithm produces. He outlines an infrastructure stack starting from healthcare regulations and data governance, moving through a data lake for storing images, distributed computing for training, and machine learning algorithms, and ending with explainable AI tools to detect and remove bias in the data sets, using IBM's AI Fairness 360 toolkit.

Building the imaging pipeline

The pipeline captures images and DICOM metadata such as age and sex from mammography, MRI and biopsy/histopathology, pre-processes formats specific to each modality, and passes them to doctors and specialists for labeling into data sets for each imaging type. Trained neural networks then classify, detect or segment tumors, localize and size them, and combine the prediction with patient context to produce a description for the doctor, with intended deployment either on a server in a hospital data center or on a small embedded medical device.

Hardware, software and the histopathology experiment

The system runs on IBM's AC922 deep learning hardware for higher throughput and accuracy, paired with PowerAI Vision software that gives doctors a web interface for labeling, training and inference without managing the underlying deep learning infrastructure; it runs on-premise because hospitals are reluctant to send data to the cloud. Using a data set of more than 10,000 histopathology microscopy images, the team trained a GoogLeNet classification model with class activation mapping at 40x magnification, reaching an initial 87 percent accuracy that improved to 93 percent after augmenting the training images through flipping and rotation, with inference under one second and a heat map showing which image regions drove the classification.

Next steps and Q&A highlights

Manaila describes extending the approach to mammography, which is harder because images are grayscale and currently reaches about 91 percent accuracy, and to combining mammography, MRI and histopathology predictions with patient age and sex for a better overall score, plus a platform for cases needing input from multiple specialists, such as metastasis to the lungs. He also mentions moving inference to embedded GPUs and FPGAs and building AI operations to monitor deployed models. Asked whether the work extends to metabolic diseases such as liver fibrosis or cardiovascular conditions, he says not yet. Asked why real-time feedback matters for a critical diagnosis, he says it gives specialists an immediate signal, for example a percentage likelihood of a finding at a given magnification, so they know when to look closer, which is especially valuable where lab queues are long. Asked how a doctor uses the system in practice, he explains two paths: direct API integration with the hospital's existing imaging system for automatic predictions, or exporting and uploading an image through the web interface to get a prediction.

Quotes

Inference in under one second to give you 90 percent accuracy is close to the human error, which is five percent.

Florin Manaila

Questions answered in this talk

How accurate is deep learning at detecting breast cancer from histopathology images?

According to Florin Manaila at Rise of AI 2019, IBM's experiment using PowerAI Vision on more than 10,000 histopathology images reached 93 percent accuracy after applying image augmentation such as flipping and rotating, up from an initial 87 percent.

Why is early detection so important for breast cancer survival?

According to Florin Manaila at Rise of AI 2019, citing National Cancer Institute UK statistics, early-stage detection is linked to survival rates as high as 90 to 98 percent, compared with much lower rates for advanced stage three or four cancer.

How does a hospital doctor use an AI system like this in practice?

According to Florin Manaila at Rise of AI 2019, the system runs on-premise in the hospital, and doctors either integrate it via API with their existing imaging system for automatic predictions or export and upload images through a web interface to receive a prediction.

Mentioned

software as a medical device, convolutional neural networks, histopathology, image augmentation, class activation mapping, explainable AI, DICOM metadata

Keep exploring

More from Florin Manaila

More from Rise of AI Conference 2019

More on Enterprise AI & Adoption

Transcript

Auto-generated transcript, not edited. · 3,879 words

Show the full transcript

hello everyone my name is Florian and I will speak today about my experience and Avenue on breast cancer detection and what we achieve in a couple of experiments so I hope that yeah all right so we start with a problem statement if you look a little bit on the breast cancer diagnostic stages there are three of them starting from stage one to stage three in the agnostic we spend a lot of time in stage two which is clinical evaluation diagnostic and and staging typically this stage the stage two is taking something like four weeks could be taking some some cases as well six weeks or more and this is very important because we have to actually go to the stage three of giving the treatment very fast especially if the cancer is actually prolonging and is moving fast so I leave it on on the background on those stages first one when you have an issue first of all you have to do a clinical evaluation right so we have to go to the or

medic doctor and and just have something like you know early evaluation and after that if something goes wrong there you have to go and do a mammography which typically some elements after 44 is just doing regular one-year interval but you have to do this and if then they find something then say okay you have to go to the next step which is a wilt result in some countries they skip the ultrasound step and they go to the next one which is MRI that will give you a 3d perspective you know what was going on there more accurate and if that confirms that you okay there is a tumor there the question is what kind of tumor is Right smiling is welling in what type that's very important and then you go to the next one which is you know you have a surgery a small surgery which is a biopsy to take small piece of the tissue from from there which is typically is guided by by the doctors and then they analyze that into

a lab for three weeks in lucky cases for two weeks but in general it takes three weeks in some other cases for weeks where you have to you know wait for for an answer and in the lab test you get you know the type of the to more you get the grade the receptor neurons and based on these you actually get treatment this is very very important it's a crucial step in detection of the type of the cancer that is there especially for example if it's her2 positive and that's something that the lab tests will actually give now I leave it on the breast cancer diagnostic we see here this is coming from the National Cancer Institute in UK five-year survival rate so early detection you have 90 percent by statistics according to them to in order to be cured and then advanced stages which is no stage three or even four very very few chances as a survival rate and this is very crucial to have a detection earlier right so that if you

look a little bit in the detection part the lab is taking the most of the time of and where you spend a lot of time assuming that all these let's say appointments where the doctors are no well done now I've been here on the stage of the breast cancer if you are having in early detection we have 98 percent survival rate but as the cancer is progressing and you go earlier or you are getting less a chemo or radiotherapy after a while then if the cancer is progressing so fast it can be spread into the lungs into the brain those are typically for the breast cancer the next organs where they spread so it's very crucial you know to cut the say the two more the type of the tumor and to give the right treatment as much as fast as we can can do so one of the question was can we accelerate this process with deep learning can you accelerate part of it at least and one that you know we are spending

in the Box in the histopathology where you have a the tissue you actually look on on the microscope to see you know what kind of cancer is and so on and to detect on the type so to have a let's say a better look on that we have to a little bit step aside to say okay I have this great technology this convolutional neural networks can I apply them what is the legislation around that and this is a code from FDA in the United States defining software as a medical device because ultimate we want to have an algorithm to trained to recognize say in an image a type of a cancer so that falls into the software as a medical device where you can see here the definition the international medical device regulation form and the RF defined software as a medical device as a software intended to be used for one or more medical purpose that performs those purposes without being part of the hardware medical device right so if you take this a

little bit say okay I want to apply this then the question is what is the general context and again you can go ahead and say okay I want to look again to FDA say what you know how they are seeing the applying of those algorithms and here is the product life cycle regulation approach for AI and based on the software-defined medical devices so not only that you have to create a device but you have to maintain that algorithm you have to monitor that algorithm continuously to get all these kind of you know metrics you have to actually track and log anything that then the algorithm will produce so it's not that easy that I'm building an algorithm and then I'm putting in that in the production so you need to have the whole pipeline in place the whole implementation of the a helps to have this alright so a little bit on the infrastructure stack obviously we started with the health care regulations and from there we have to understand you know the data that

we have to handle the governor's around the data people involved policies all these kind of things and then we ought to understand from where those those that are coming so we need a data lag to you know to have those images stored and we can take them from there for training and and so on we have to distribute the computing we need those machine learning algorithms and and at the end we need all these furnace explainable AI were up to understand if we have biases in in the data sets and how we can remove them so all those aspects in general are very complicated for for the doctors for the researchers which are on the medical field to deal with them right and one of the thing was okay can we simplify here something so in order to answer to this we had to take a look a bit and breast cancer diagnostic pipeline in a simplistic view from the image perspective we have first of all like an engine the mammography part MRI and

then we have the biopsy where we have the histopathology in those cases we produce images and metadata the metadata is actually when you have an image from let's say my MRI you have a deacon file format and inside of the header of the file you have a couple of things like the age the sex of the person the unique a variety and and so on so forth so those are metadata which are very important as well to preserve so ultimate we want to capture those images in various stages and then have a deep neural network to train this network to recognize the type of a cancer either we do segmentation of the detection or we do classification as well we have to need to have information the age of the person because that's very important and typically is inside of those specific file formats and ideally we want to give a prediction to the data say well the network can predict that with so much accuracy that this is cancer this type and and and

probably this size and so on so we are concentrating a little bit on the most time-consuming part which is the histopathology part now another thing is those file formats are coming in specific things especially if you if you have more complicated more complex things then again you have to deal with let's say other kind of formats for example brain you have the nifty formats so you have to pre-process those images and then extract from those images the metadata and the PNG files or the JPEG files that we need to train those neural networks and then we actually go further to this stage where we have to do the labeling right so we have to do the labeling process we have to this typical done by doctors and specialists and this is on will create specialized let's say data sets for Mama for MRI histopathology and so on and then we want to actually detect the bias and to remove them from from there if there are and we use a 360 toolkit for example within

IBM and we have a lot of iterations until we get those neural networks on those on those things trained at the end we want to actually have an image we want to classify that image being a tumor a B or being a normal case and then we want to do more things like localizing that having the size of the tumor if possible especially devolution and then we want to extract the context you know the age of the person and so on from the existing metadata that we acquired and then we want to have some api's where we want to say okay we want to have two more prediction and this is the type of the tumor we predict and this is the accuracy this is intersection of Union and all those kind of terms and obviously at the end is the two more description and this is provided to the doctor and we can apply this to mammography MRI histopathology that that will be the goal now from the perspective of earrin so putting this

in the production we would like to put those in Finnish scenarios either running on a server in the data center or either put it in a medical device in a small device in embedded devices on scenarios three and four those are the big two things one which are can run on the other Center and one run in the in the hospitals all right so from the hardware we use a Schneider to system it's an IBM deep learning system the reason we use that is because for complicated things where you have to investigate for example the the cancer which has been spreaded on the tomorrow has been spent on the brain we use better accuracy and more images using this kind of technology you see here an example 250 650 images per second compared with 65 images now from the software perspective we used power I vision and you see left for training and inference and right only for inference and this is a software that is simplifying the whole landscape this is focusing on providing

the best experience of doctors for labeling training without taking care of all the aspects of deep learning and of the infrastructure beneath so I'm going to speak about how we use these things right so the goal is to actually automate the whole pipeline as much as we can with the technology that is available its general available on Prem because we see you know hospitals that they don't want to send data due to regulations into the cloud right so the product itself is actually providing a web interface in a lot of api's where doctors they can go there and find the images label those images do some training by just a couple of clicks for classification of detection and in the future segmentation semantic segmentation and in addition we provide as well informations for if you do inference if you want to test then those models and see you know how they perform you know how good they're so we provide as well some parts of the those models are pre-trained but we can allow as

well things to be added additional on top of the product as you know custom neural network models all right so the experiment we did it was by using the VRI lab Department of Informatics in interest field that were there quite kind to give us their data set for his breakfast called for histopathology this was having more than 10,000 images microscopy images and the goal was to create a neural network model that will identify those classes you can see it here of the breast cancer from the tissues so bending and modeling but as well you know other noses we might know my exactly were interesting to train a natural for mostly for our typical cases where the doctors in the lab they spend a lot of time so we use for the neural network model the classification Google net class activation mapping where in especially on a high resolution on 40 X magnification of those images we can actually identify the part in in a specific class and this is tends to be very very good

in terms of accuracy so a couple of screens here about the experience so you connect us as a doctor to the system you start your loading your datasets creating the categories that you have you see those are the images that we have given on the left side you will see the classes for each image and if you want to view the image and have more deep knowledge and check those images in addition to what has been given and labeled by it some others you can go there and then you can which is just a button you can start a training and the network will start training and then you do this a lot of iterations until you get it to the right accuracy in the initial run we got 87 percent accuracy obviously this was not enough so we said okay can we do something else right can you change something and there is a technical documentation that has to be used very careful in those cases so we used to flip the images to

a horizontal or rotate them with a specific angle to generate more images from what we had to actually train the network so when actually you do this automatically the system will augment the images and you see them here with a from augment so again we train that and we achieved 93% inaccuracy in detecting those classes and that's is the experiment that we have done and we believe that inference in under one second to give you 90% accuracy is close to the human error which is five percent so we still have to present that I think can be solved in in the field there so in the inference when you test this with the validation images you are getting not only the prediction you can actually establish the threshold and you have as well a heat activation mapping so we can give you this map so can't stand what were the most important features in the class that been contributing to the image we classify the way it was so moving further what's next right so

one of the problems that we see there because this is a continuous process is you know how it can be involved a little bit into the mammography part right mammography is a little bit different because of the you don't have color images majority of them so we are just grayscale and that's a challenge for those newer networks to pick them so until now Mike's been wasn't up to 91% accuracy still not enough but we see promising results so one of the thing is to take as well in the consideration this part of the mammographies MRIs as well we are looking as well to do posting feelings and orchestration so when you have something like a patient that and you have the image from the mammogram from MRIs and from the histopathology you can run them as three separate inference processes and then you can do the scoring including the aging the age of the person the sex and and all these kind of things so you can have a less a better prediction technical consideration

the other part especially if you try to consider the cases where we have unfortunately patients that they have let's say lung cancer which is the metastasis and then the that is treated by a doctor which is specialized in lungs and then we have the the doctor which is specialized in the breast cancer and typically have to find the time to speak each other say okay what we do with this passion so for those kind of cases we can actually have a platform where it can use it you know for to do this kind of scoring among you know multiple disciplines by using deep learning in addition we can we are interesting in actually the technology as well lesions so did this can we believe that can be done and move moving these things to smaller devices to embedded GPUs and bedded FPGA is what can be run somewhere on a desktop up to the moment that you have to understand I need to put this in the production what that means you know how I'm

monitoring what those algorithms will do and will predict to advise the doctors in a way or another one and this is go this goes into the air operations how you actually deal with those kind of things not only on building that model that will actually predict something and that model has to be put into the context into the age of the person into the type of the image that has been used where has been used and what kind of position and so on and so forth so we're actually trying to give a platform that can be used for those kind of cases on Prem by the doctors by the clinicians they can train their own network to advise them right to find those cases we don't try to give them a model and say use this model will be the best for you right you build your model based on your experience but based on your labels and the labels they have to be verified as well by other entities to be sure that those

are correctly labeled the way there are because those networks I would say are very dummy if you don't give them the right data they will learn not to predict things that will be accurate in the or will not generalize well in the in the production questions so far yeah we're going to get your microphone hi nice talk I'm puja from nature research I'm just curious to know if you are extending your application to any metabolic diseases like liver fibrosis snaffle D cardiovascular diseases no the answer is no that's why we were trying to include us well the context other kind of things into dog's neural network models as it as a say different pipeline right so one of the thing is we're taking the images we are trying to teach those networks for those kind of let's say typical cases but for our typical cases you have to understand us with the context of the person so we didn't go that far we are trying to validate those this power vision software and how can

be used in such cases are there more questions no oh there's one more thanks for the talk I'm just curious about what is the need of real real-time feedback or you know why do you need real-time analysis for for such a critical cancer detection or or something submitted that right so the way I perceive this is that for example when you have an electronic microscope right and you actually look for the cells on different magnification right as human at certain point you have duds right and typically you speak with one of your colleagues but in general the algorithms that is are displaying you the image are just algorithms so the need of having a solution like this for the people on the field if they give them real-time insights so they can okay so the algorithm is telling me that at 40% for the X magnification I have less a fibroadenoma the question is okay I should go and look a little bit more on a high resolution you know 400 X 4 more in

rather they just say okay no it's okay right and and especially for cases where there are countries where they don't have labs across the country so there's a long queue you know for those kind of cases how you deal with them okay okay we wouldn't have time for one more question if there is one okay here she says coming to you with a microphone so everybody can hear you thank you for the nice talk can you give a little bit more explanation how doctor will use this for example if a doctor has one or two pictures with him or her so they upload to your system and use your data set train data set and then later on check their voices well to a system to make a decision so there are there are two ways so assuming that the system is in the hospital because we we believe that we this system should run in the hospital trained by the hospital based on their experience and the reliable data the doctor has two possibilities

other to integrate let's say the MRI system VI API with the system so it will get integrated via Vialis if they use G or they use other company if you had that we are the parks or because we have an API just send an image and you get the prediction or ii will take the we'll export the image from from decom or from if and then we'll actually upload the image via web interface to the system will then we'll give them a prediction so there are two ways depends of the situation in the hospital

Rise of AI 2027 · Berlin · May 2027

Be in the room for the next one.

Ten years of talks started as conversations in one curated room in Berlin. The next edition brings 300 selected leaders together in May 2027.