At Rise of AI 2022 in Berlin, Sam Edds, senior data scientist at Yelp and Z by HP Global Data Science Ambassador, presents a proof-of-concept project testing whether a computer vision model trained only on point-of-view photos of natural disasters can predict on aerial images it has never seen. Using a deep learning model trained on tweet-sourced point-of-view images of wildfire, flood and landslide incidents across categories like buildings, land, infrastructure and water, she evaluates it against a separate MIT lab dataset of low-altitude aerial disaster images. The point-of-view model predicts incidents almost perfectly and places correctly 9-90% of the time on its own data type. Applied out of sample to aerial images, it correctly identifies fire 85% of the time, flood 78%, and landslide 60%, while struggling badly with building types such as residential neighborhoods and house facades, since aerial views show little of the roof and facade detail visible from ground level. She argues the approach is promising for scaling low-cost, pre-trained tools to disaster response and hazard mitigation agencies, while stressing that biased input data always produces biased output regardless of the model used.
Key takeaways
- A model trained only on point-of-view disaster photos correctly identifies aerial wildfire images 85% of the time, floods 78%, and landslides 60%, despite never being trained on aerial data.
- The model performs worst on building types (residential neighborhoods, houses, facades) in aerial images because aerial views show little roof structure or facade detail compared to ground-level photos.
- Floods and landslides are frequently confused with each other in aerial images because both often show dark, sediment-covered or gritty-colored water.
- Edds proposes adding negative classes, such as fire pit photos, so the model does not misclassify harmless scenes as disasters.
- The dataset's original wildfire labels were largely unusable, so Edds discarded them and re-scraped wildfire images herself after quality-checking the data.
Detailed notes
The question: can a point-of-view model generalize to aerial images?
Edds frames the project around rising numbers of climate-driven natural disasters, citing recent floods in Germany, landslides in Brazil, and ongoing wildfires on the US West Coast, and asks where computer vision can help. Her research question is whether a model trained and tested only on point-of-view (ground-level) photos of natural disasters can predict correctly on aerial images it has never seen, arguing that a model generalizing across viewpoints would be far easier for disaster response and hazard mitigation teams to scale than building a new model for every scenario.
Data and model setup
The point-of-view training data, sourced from tweet photos gathered by an MIT lab, covers three incidents (wildfire, flood, landslide) and 14 places across four categories: buildings, land, infrastructure and water. The separate aerial dataset, from a different MIT lab, uses low-altitude imagery with the same incidents and places but often shows multiple places per image rather than one dominant scene. Edds notes she personally quality-checked and re-scraped the wildfire images after finding the original labeled data contained only a few usable smoke and fire photos, and flags that roughly 10% of landslide images in the sample were themselves ambiguous even to her as a non-expert.
Results on point-of-view data
On its native point-of-view data, the model almost always correctly predicts the three incident types, aided in part by color cues such as red-orange tones for wildfire, and correctly predicts places 9-90% of the time. It performs well on land and infrastructure but mispredicts residential neighborhoods as houses, and struggles to distinguish lakes from oceans, particularly in flood images where water appearance is distorted.
Results on out-of-sample aerial images
Applied to aerial images without any aerial-specific training, the model correctly identifies fire 85% of the time, flood 78%, and landslide 60%. Edds observes that floods and landslides are often confused with each other because both can show dark, gritty-colored water obscured by sediment. Place prediction on aerial images is weaker overall, with flood scenes often predicted simply as "river" even when a bridge is also present, and inconsistent results between visually similar images that she could not fully explain. The clearest pattern is that the model struggles with all building types (residential neighborhoods, houses, building facades), since aerial views show little roof structure or facade detail compared to ground-level views, though she notes aerial building detection is otherwise one of the easier computer vision problems to solve with dedicated models.
Implications, limitations and next steps
Edds argues the project's value lies in showing that a single pre-trained model could be deployed cheaply and simply, letting agencies upload images to a website and get predictions back on ordinary hardware, rather than requiring a new model per scenario. She notes the project used about 100,000 images, trained in 25 minutes on GPU versus an hour and a half on CPU, and stresses that scaling to millions of images multiplies that cost. She proposes adding negative classes, such as photos of fire pits or people playing croquet, so the model does not misclassify harmless images as disasters, and suggests testing the reverse direction of training on aerial images and predicting on point-of-view ones. Her three closing takeaways: computer-vision models trained on one viewpoint can, with refinement, transfer to another except for hard cases like buildings; pre-trained, easily deployed models are key to real-world scaling; and biased input data always produces biased output regardless of model choice.
Q&A highlights
Asked how to fight bias in such a diverse dataset, Edds points out that most images were taken in summer, lacking seasonal and angle variety, and recommends physically sampling and reviewing data, involving diverse people in reviewing it, and setting guidelines for expected data diversity from the start rather than only discovering bias after a model is built. Asked whether the method could predict other catastrophes such as pandemics, she says her approach would not transfer to something like a pandemic without incorporating health trend data, but expects it could work well for other well-documented natural disasters such as earthquakes and tsunamis given how much data now exists. Asked whether she annotated the 100,000 images herself, she says no: the point-of-view data was human-labeled by the MIT lab, and the aerial dataset was labeled via Amazon Mechanical Turk, for which she applied a confidence threshold and manually checked results, which is also how she caught the unusable wildfire labels.
Quotes
If you put something in that's biased, it doesn't matter how you model it, you will get bias out.
Sam Edds
Questions answered in this talk
Can a computer vision model trained on ground-level disaster photos work on aerial images?
According to Sam Edds at Rise of AI 2022, a model trained only on point-of-view images correctly identified aerial wildfires 85% of the time, floods 78%, and landslides 60%, though it struggled badly with building types.
Why does aerial disaster image prediction struggle with buildings?
Edds explains that aerial views show buildings as small shapes with little visible roof structure or facade detail, unlike ground-level images where features like pitched roofs, windows and HVAC units are visible, making buildings one of the hardest categories to transfer across viewpoints.
How can bias be addressed in a natural disaster image dataset?
Edds recommends physically sampling and reviewing the underlying data, involving people with diverse perspectives in that review, and setting guidelines for expected data diversity (such as season and camera angle) from the start of data collection rather than only after a model is built.