Rise of AI Conference 2022 · Berlin · Applied AI Stage

Predicting Natural Disasters: Testing the Boundaries of Computer Vision

Sam Edds · Senior Data Scientist, Yelp; Z by HP Global Data Science Ambassador

At Rise of AI 2022 in Berlin, Sam Edds, senior data scientist at Yelp and Z by HP Global Data Science Ambassador, presents a proof-of-concept project testing whether a computer vision model trained only on point-of-view photos of natural disasters can predict on aerial images it has never seen. Using a deep learning model trained on tweet-sourced point-of-view images of wildfire, flood and landslide incidents across categories like buildings, land, infrastructure and water, she evaluates it against a separate MIT lab dataset of low-altitude aerial disaster images. The point-of-view model predicts incidents almost perfectly and places correctly 9-90% of the time on its own data type. Applied out of sample to aerial images, it correctly identifies fire 85% of the time, flood 78%, and landslide 60%, while struggling badly with building types such as residential neighborhoods and house facades, since aerial views show little of the roof and facade detail visible from ground level. She argues the approach is promising for scaling low-cost, pre-trained tools to disaster response and hazard mitigation agencies, while stressing that biased input data always produces biased output regardless of the model used.

Key takeaways

  • A model trained only on point-of-view disaster photos correctly identifies aerial wildfire images 85% of the time, floods 78%, and landslides 60%, despite never being trained on aerial data.
  • The model performs worst on building types (residential neighborhoods, houses, facades) in aerial images because aerial views show little roof structure or facade detail compared to ground-level photos.
  • Floods and landslides are frequently confused with each other in aerial images because both often show dark, sediment-covered or gritty-colored water.
  • Edds proposes adding negative classes, such as fire pit photos, so the model does not misclassify harmless scenes as disasters.
  • The dataset's original wildfire labels were largely unusable, so Edds discarded them and re-scraped wildfire images herself after quality-checking the data.

Detailed notes

The question: can a point-of-view model generalize to aerial images?

Edds frames the project around rising numbers of climate-driven natural disasters, citing recent floods in Germany, landslides in Brazil, and ongoing wildfires on the US West Coast, and asks where computer vision can help. Her research question is whether a model trained and tested only on point-of-view (ground-level) photos of natural disasters can predict correctly on aerial images it has never seen, arguing that a model generalizing across viewpoints would be far easier for disaster response and hazard mitigation teams to scale than building a new model for every scenario.

Data and model setup

The point-of-view training data, sourced from tweet photos gathered by an MIT lab, covers three incidents (wildfire, flood, landslide) and 14 places across four categories: buildings, land, infrastructure and water. The separate aerial dataset, from a different MIT lab, uses low-altitude imagery with the same incidents and places but often shows multiple places per image rather than one dominant scene. Edds notes she personally quality-checked and re-scraped the wildfire images after finding the original labeled data contained only a few usable smoke and fire photos, and flags that roughly 10% of landslide images in the sample were themselves ambiguous even to her as a non-expert.

Results on point-of-view data

On its native point-of-view data, the model almost always correctly predicts the three incident types, aided in part by color cues such as red-orange tones for wildfire, and correctly predicts places 9-90% of the time. It performs well on land and infrastructure but mispredicts residential neighborhoods as houses, and struggles to distinguish lakes from oceans, particularly in flood images where water appearance is distorted.

Results on out-of-sample aerial images

Applied to aerial images without any aerial-specific training, the model correctly identifies fire 85% of the time, flood 78%, and landslide 60%. Edds observes that floods and landslides are often confused with each other because both can show dark, gritty-colored water obscured by sediment. Place prediction on aerial images is weaker overall, with flood scenes often predicted simply as "river" even when a bridge is also present, and inconsistent results between visually similar images that she could not fully explain. The clearest pattern is that the model struggles with all building types (residential neighborhoods, houses, building facades), since aerial views show little roof structure or facade detail compared to ground-level views, though she notes aerial building detection is otherwise one of the easier computer vision problems to solve with dedicated models.

Implications, limitations and next steps

Edds argues the project's value lies in showing that a single pre-trained model could be deployed cheaply and simply, letting agencies upload images to a website and get predictions back on ordinary hardware, rather than requiring a new model per scenario. She notes the project used about 100,000 images, trained in 25 minutes on GPU versus an hour and a half on CPU, and stresses that scaling to millions of images multiplies that cost. She proposes adding negative classes, such as photos of fire pits or people playing croquet, so the model does not misclassify harmless images as disasters, and suggests testing the reverse direction of training on aerial images and predicting on point-of-view ones. Her three closing takeaways: computer-vision models trained on one viewpoint can, with refinement, transfer to another except for hard cases like buildings; pre-trained, easily deployed models are key to real-world scaling; and biased input data always produces biased output regardless of model choice.

Q&A highlights

Asked how to fight bias in such a diverse dataset, Edds points out that most images were taken in summer, lacking seasonal and angle variety, and recommends physically sampling and reviewing data, involving diverse people in reviewing it, and setting guidelines for expected data diversity from the start rather than only discovering bias after a model is built. Asked whether the method could predict other catastrophes such as pandemics, she says her approach would not transfer to something like a pandemic without incorporating health trend data, but expects it could work well for other well-documented natural disasters such as earthquakes and tsunamis given how much data now exists. Asked whether she annotated the 100,000 images herself, she says no: the point-of-view data was human-labeled by the MIT lab, and the aerial dataset was labeled via Amazon Mechanical Turk, for which she applied a confidence threshold and manually checked results, which is also how she caught the unusable wildfire labels.

Quotes

If you put something in that's biased, it doesn't matter how you model it, you will get bias out.

Sam Edds

Questions answered in this talk

Can a computer vision model trained on ground-level disaster photos work on aerial images?

According to Sam Edds at Rise of AI 2022, a model trained only on point-of-view images correctly identified aerial wildfires 85% of the time, floods 78%, and landslides 60%, though it struggled badly with building types.

Why does aerial disaster image prediction struggle with buildings?

Edds explains that aerial views show buildings as small shapes with little visible roof structure or facade detail, unlike ground-level images where features like pitched roofs, windows and HVAC units are visible, making buildings one of the hardest categories to transfer across viewpoints.

How can bias be addressed in a natural disaster image dataset?

Edds recommends physically sampling and reviewing the underlying data, involving people with diverse perspectives in that review, and setting guidelines for expected data diversity (such as season and camera angle) from the start of data collection rather than only after a model is built.

Mentioned

Computer vision, Deep learning, Point-of-view vs. aerial imagery, Explainable AI, Negative classes, Data bias, Hazard mitigation planning

Keep exploring

More from Rise of AI Conference 2022

More on Ethics & Trust

Transcript

Auto-generated transcript, not edited. · 4,193 words

Show the full transcript

and now i would like to welcome our next speaker Sam Edds a warm applause for sam please sam is senior data scientist at yelp and she's also the set by hp a global data sign ambassador we yesterday we talked about that also at the um dinner she was such an enrichment at the event already and she wants to talk about how you can predict natural disasters with testing the boundaries of computer vision and um yeah i would like to hand over to you welcome on stage thank you hi everyone today i'll be talking about predicting natural disasters and testing the boundaries of computer vision so right now we see climate change is increasing every year every year there are more and more natural disasters that are happening and these numbers aren't going down and they're starting to impact all of us from the floods in germany last summer to the landslides last week in brazil to the seemingly constant wildfires on the west coast of the us this is a problem that we are all

now facing and i want to see how we as an ai community can start to have some conversation around this and where is there space and computer vision to potentially help okay so first i'll tell you a little bit about me uh and then i'll discuss the questions we're trying to answer the data set a little bit about deep learning and then we'll look at the results and how this can impact in the public sphere so like law said i am a z by hp global data science ambassador they gave me a wonderful computer with multiple gpus to do this project i'm also a senior data scientist at yelp and i'm a huge explainable ai enthusiast which may seem extremely ironic that i'm talking about something with deep learning today but i've worked in crime and education research and in berlin startups and i think it's really important to think about the data underneath which is also a topic we've heard today if you put something in that's biased it doesn't matter how you model

it you will get bias out so if you are also interested in this topic feel free to approach me later because it's something i really care about okay eventually i will get this done so what is our question today can we predict natural disasters in aerial images when just using a model trained on point of view images so this means that the model has never been trained or tested on aerial images but just on natural disasters from point of view and obviously point of view right doesn't necessarily mean exactly straight on you can look up you can look down but it's much more granular than something that you see from an aerial image and from the air and the impetus for this is if we can take a model that's used in one place and trained on one thing and apply it to many different scenarios it's much better for the natural disaster response teams and also for the hazard mitigation planning teams who would be using this because it's using one thing and then

being able to scale it so with that our training data set i'll use these two phrases incident and place so an incident is the natural disaster that occurred i look at three of them so wildfire flood and landscape and then the place is where this occurred so this could be a bridge street forest there can also be multiple places with an incident as well and we'll see that a lot with the aerial data set so this is what the data art the model is trained on and this data comes from actually tweet photos that people had made and then this data was gathered by an mit lab but i just use the data make my own model based on the point of view images to train and test and then i do the out of sample prediction on the aerial image data so what are the places that we're looking at there's four categories there's buildings land infrastructure and water and these might some of these might seem really granular like street and highway but

we're interested in testing the boundaries of what a model could do so we could aggregate these to something like road infrastructure but let's see how it can do knowing that later some other time not today we could actually aggregate these further and look at them at a higher level so the aerial data set we have is from a different mit lab this is low altitude aerial data and images so this isn't satellite level so there could be differences if you tried to predict out on satellite level uh of images versus these aerial data ones but you can already see compared to point of view you can have a lot more places and things involved maybe now instead of taking up most of the photo the incident is just taking up a small snippet towards the top or the middle or a side and all these differences can really be impactful when trying to predict using something that's trained on point of view versus aerial so we have the same three i've shared my suit a

little too early we have the same three incidents and the same 14 places and the last comment i guess i cannot go back that i wanted to make on that is the fire data i actually scraped myself because when i looked at all the label data which please always whenever you're using data look carefully at the labels and quality check it i saw a few pictures of smoke and fire and that was it so during my quality checks i removed the fire the wildfire data for the aerial images and just scraped it myself online so i shared my secret slightly early i am not a natural disaster identification expert there are maybe 10 percent of photos in the sample that look something like these two i don't know if there is actually a landslide in either of them uh i left them in but i just wanted to point out that sometimes it's also difficult to tell whether images are something or not and so there could be a little more bias in the results

because of images such as these but this is only on the prediction data set so going to now a little bit about deep learning we won't be talking about matrix factorization or anything super thrilling in depth i think the key point is that there is no person making these features the images are put in it's all done under the hood the future extraction is not done by me i'm not saying ah this looks there's this in this part uh both in terms of the there's an algorithm that's weighting different aspects of an image and determining whether or not that's important and also looking at non-linear relationships between high-level features to try to determine what an image actually is and in the end output is a predicted probability for the images such as for the places and incidents and these don't have to equal one you can have this is like a multi-label multi-class problem that we're looking at so just a last quick point on deep learning versus machine learning is that it's a person

with machine learning eventually i will get the clicker down eventually uh there's a person who's behind the machine learning aspects of this who are making these actual features deep learning this is under the hood and as someone who's super interested in explainable ai i always want to think about what is the end use case of this one of the end use cases is that natural disaster teams are going to be able to better understand the extent of a flood and whether there are say bridges or streets or what exactly is happening to this there's not human data involved and so this is actually a use case in which i'm quite comfortable using deep learning and think is a really good potential use case because it doesn't touch on human data and we don't need necessarily that level of explainability so to our results let's first look at how the model does on point of view incidents because this is what is trained on we want to make sure or c doesn't do a good job

or not in these cases so the model almost always correctly predicts all three of our incidents which is great the model also is able to take into account color so for things like wildfire a lot of it may be that kind of red orange yellow aspect and then the others do a pretty good job as well so that we just is like a check box good we want to know how it did but it's not really what we're focusing on the places as i mentioned before uh we get quite granular in what we're trying to test with the places and so the model correctly predicts point of view places between nine and ninety percent of the time it does pretty well on land and infrastructure and then let's look at what it does badly on and try to think about what is the story behind this so for residential neighborhood when i looked at those those are pretty much always mispredicted as a house okay pretty close if we thought of this as an aggregate

building or not uh that would explain it and then for lake and ocean it looks sometimes it's really hard lakes and oceans from point of view especially if the image has flooding it i would not be able to tell that that was a lake uh so i think sometimes it's interesting in the when you have both these places and incidents in one there are certain things especially if you have an incident that is a flood where it's going to be super difficult to tell and correctly predict what that place is so models now it's the most exciting part how does the model do on the aerial data when the model was never trained or tested on aerial images so i would say for instance it's actually quite promising so the model best predicts the aerial view fires correctly 85 percent of the time it gets this right and i think a lot of that might again be due to the color it's pretty easy to pick out the general fire related colors for flood it's

78 percent for landslide uh it's 60 so not super great compared to just guessing whether it's this or not uh but let's think a little bit more about why this could be so whenever something did really well i would say okay and look a little at the images whenever something did poorly i thought let's look at what is there a story behind this and do i see some kind of patterns so floods are often mispredicted as landslides and vice versa it might be difficult to see but in the image on the left there's actually some flooding in the middle but it's pretty covered up by a lot of sediment and then in the image on the right the water is quite dark and these are both flood images but if you see a landslide image with flooding a lot of times the water is also super this like dark grittier color and i think that could be uh why we see this in these cases for the misprediction of fire and landslides i didn't see

as clear of patterns emerged for that so that is the incidence and now let's think about the places so aerial images like i said they typically have many places compared to point of view images and it shows also in the results so again infrastructure and land does pretty well and then there's a bunch that do quite poorly residential neighborhood house and building facade though are all building types and then lake and coast and when we think about this there are a few potential stories that came out of it so again places and flood images are often only predicted as rivers so in this image there actually is a river but there's also a bridge and i always looked at the top two uh they're at least in an aerial image generally two places and then i'd see is at least one of them is the top predicted one one of these if not i dug in a bit to see okay well maybe it's actually predicted still a pretty high percentage it just wasn't the

top one but a lot of times with the flood and the with floods or with rivers it's just predicted as a river other times i'll see something like this and i couldn't really say why one is predicted as a bridge and one is a river to me they both look pretty similar they both have rivers and bridges in them and yet the results are quite different so while there are some times that i was able to look into the results and see a story that made sense there are also times like this where i really couldn't say and i don't the model wasn't really able to differentiate that clearly so one thing that did become a bit clear though is that the model struggles to predict buildings so residential neighborhoods houses and building facades it all did terribly on when looking at the point of view trained and then predicting on these aerial images when you think about looking at a building from a point of view you can notice things like there's a pitched

roof there are windows there are aspects of these external facade there's an hvac there are a lot of details that you might see when you look at a point of view image compared to when you look at it from an aerial point of view there you see the buildings look quite small they also you can't really see much about the roof structure you basically only see the roof you don't really see a lot about the exteriors and i think that makes sense to me that right sometimes point of view and aerial things will look super different and we're interested in figuring out where does it do well and where does it do poorly and we have a clear case of where it does if you train on point of view it's really difficult to predict on aerial images luckily predicting on aerial images finding buildings and aerial images is actually one of the like arguably otherwise easiest uh ai problems to solve and so right in some cases it just doesn't make sense to use

something like a point of view model on some parts of aerial images so when we think about all these results the whole impetus for this project was how can we apply this for public good i see two big areas natural disaster response and hazard mitigation planning and i wanted to to take a model and predict on these point of view images or sorry make a model in point of view images and then predict on the aerial images to see can we put train a model on something and use it in these other scenarios because that is also what is going to most easily scale and make it easier for companies to implement i spent end to end maybe getting the data et cetera maybe 60 hours on this project i also had a super strong computer with two gpu a lot of memory and i also have a background in statistics and data science if you go to a company and you need someone to every time there's a new scenario make a new model

it's going to be super hard to get some kind of public companies to implement this and to get them to actually put something into production what is great is if you have something trained in one area and it works in multiple areas then someone could just upload a file of images for example and then get these predicted scores back oh this is then they can map out the flooding in this specific area and see okay it's these bridges are affected this highway and all it needs to do is all they need to do is upload images to a website that is pre-trained on the model and then they're able to get back results and they're just using a normal computer so there's nothing specialized required in this scenario and that's why i think that's why i wanted to look at how we can use a model in one space and try to apply it in another and so a last point on this is again on the gpu cpu aspect is i train this on

about 100 000 images a lot of times you might want to train a model on many millions of images if it took me 25 minutes with gpu an hour and a half a cpu think about this at a hot broader scale it's going to take a lot more time and effort to run as well and so the more cases in which we can find something that's made in one scenario and applicable to many the better and the easier it is to scale and implement across a wide variety of agencies and with that i will discuss just quickly what are a few things that i could do next because this was kind of a proof of concept to see how does this do overall is it even worth it to try to do this and something i think would be really interesting to add are these negative classes so you could put images of fire pits for example to see okay if you put a fire pit right now it will probably say it will say

this is a wildfire but it's it's actually not right it's just a fire pit and so or someone playing croquet there's no incident there so adding these negative classes and just places with no incidents is one area that i think would be very beneficial for the model because i was trying to protect out on aerial images that have natural disasters but we'd also want to make sure if given a random photo that we don't say ah this is a wildfire that wouldn't make sense so i think that's a big next area to explore and then i think it would also be interesting to see the flip which is if you train on aerial images and predict on point of view once with that i will give you just the key takeaways so i think with refinement computer vision trained on the point of view images can be applied to aerial images there are obviously some cases where it won't make sense things like buildings but generally it seems like with some work this could be

promising another is when thinking about implementation computer vision is computationally expensive but if you have a model that can be easily applied to many settings then you can just use this pre-trained model and people can use it with cpu on their machine and create some kind of web app or some easy way for them to integrate it and not have to actually interact at all and it saves a lot of time and will make it a lot easier to scale with public agencies and natural disaster response and the last thing as with anyone working in data science i would be very sad if i didn't say this what you put in is what you get out so whatever potentially biased data you have if you put that in it doesn't matter what kind of model you use to try to model the data the bias is there and it will perpetuate onto the other side so with that thank you for listening and that is it feel free to reach out on linkedin or talk

to me after thank you sam that was super interesting and at least you you mentioned bias and this is a point which um interests robert mikko from our online audience as well he asked how do you fight bias with such a diverse data set yeah i think i would say that currently there are some points of bias i could pick out to you right now like most of the images are in summer so there's not a variety of seasons i think that there is some angle also that could be there could be more angles involved and i think a lot of the ways in which you can combat bias is just looking at the sources you're using and physically looking at the data obviously you can't use look at every single image i mean you could but it would take a super long time but i think it's super important to always take a sample of your data and try to look at it to see okay does this check out are there other things

i expect and to bring in different people into the room like i might see think of things in one way and you might in another so if we both look at something i think it's always helpful to have diverse opinions because it helps you think of things in a different way and then better be able to combat the biases also often in the end when something is built you can find more biases a bit easier but i think it's always better to try to do that immediately from the start as you're gathering the data is work also with others and really look at it and come up with guidelines for what you think should actually what you would expect and how you think that the data could differ in certain ways that would help make it more diverse and what i also thought is which further qatar catastrophes could you predict with this method do you think i mean i was just thinking about the pandemic do you think there are opportunities in other fields

so i think for so this is trying to look at the natural disasters i think there are a lot of other disasters you could potentially predict things like a pandemic i think based on what i did you wouldn't and when i think about that you would have to do a lot more things with like health trends etc so i think the things like those i couldn't say anything about or i'd just be making it up but i think other natural disasters you definitely can anywhere where there's a lot of data you can probably predict something and for natural disasters it's there are unfortunately so many now that i think a lot of things like earthquakes tsunamis etc you could really predict because there's so much good data actually out there to do so yeah so maybe we have enough data for the hopefully not coming second pandemic to predict are there any other questions in the audience here um to the topic of natural disasters and how to predict it yeah wait a second for

the microphone testing i just had a question about the project that you set up i was curious did you annotate the data yourself because 100 000 images that you had to do that manually because that's what we do in general and i'm just curious ah i luckily so i did not because i've also had to do that for work and it's yeah it's really rough so i used these data that had been gathered in tweets uh by this mit lab and then they had humans label it and then what i did was basically check these labelings and then the other the aerial data set they had humans label and then they also did i'm blanking on aws's name the mechanical turk they also had those labels and some of the mechanical turk labels were pretty bad so i created a threshold basically for the places like all the places had been on the aerial data sets had been labeled with mechanical turk i basically created a cut off which the score had to be well

above and then i did manual checking basically to see do i think this makes sense or not like that's also where the wildfire issue came in and then that's when i went and scraped them myself and decided okay this one yeah the quality is too questionable thank you sam thank you

Rise of AI 2027 · Berlin · May 2027

Be in the room for the next one.

Ten years of talks started as conversations in one curated room in Berlin. The next edition brings 300 selected leaders together in May 2027.