-
Generating Physically-Consistent Satellite Imagery for Climate Visualizations
Authors:
Björn Lütjens,
Brandon Leshchinskiy,
Océane Boulais,
Farrukh Chishtie,
Natalia Díaz-Rodríguez,
Margaux Masson-Forsythe,
Ana Mata-Payerro,
Christian Requena-Mesa,
Aruna Sankaranarayanan,
Aaron Piña,
Yarin Gal,
Chedy Raïssi,
Alexander Lavin,
Dava Newman
Abstract:
Deep generative vision models are now able to synthesize realistic-looking satellite imagery. But, the possibility of hallucinations prevents their adoption for risk-sensitive applications, such as generating materials for communicating climate change. To demonstrate this issue, we train a generative adversarial network (pix2pixHD) to create synthetic satellite imagery of future flooding and refor…
▽ More
Deep generative vision models are now able to synthesize realistic-looking satellite imagery. But, the possibility of hallucinations prevents their adoption for risk-sensitive applications, such as generating materials for communicating climate change. To demonstrate this issue, we train a generative adversarial network (pix2pixHD) to create synthetic satellite imagery of future flooding and reforestation events. We find that a pure deep learning-based model can generate photorealistic flood visualizations but hallucinates floods at locations that were not susceptible to flooding. To address this issue, we propose to condition and evaluate generative vision models on segmentation maps of physics-based flood models. We show that our physics-conditioned model outperforms the pure deep learning-based model and a handcrafted baseline. We evaluate the generalization capability of our method to different remote sensing data and different climate-related events (reforestation). We publish our code and dataset which includes the data for a third case study of melting Arctic sea ice and $>$30,000 labeled HD image triplets -- or the equivalent of 5.5 million images at 128x128 pixels -- for segmentation guided image-to-image translation in Earth observation. Code and data is available at \url{https://github.com/blutjens/eie-earth-public}.
△ Less
Submitted 21 October, 2024; v1 submitted 10 April, 2021;
originally announced April 2021.
-
Bringing Cognitive Augmentation to Web Browsing Accessibility
Authors:
Alessandro Pina,
Marcos Baez,
Florian Daniel
Abstract:
In this paper we explore the opportunities brought by cognitive augmentation to provide a more natural and accessible web browsing experience. We explore these opportunities through \textit{conversational web browsing}, an emerging interaction paradigm for the Web that enables blind and visually impaired users (BVIP), as well as regular users, to access the contents and features of websites throug…
▽ More
In this paper we explore the opportunities brought by cognitive augmentation to provide a more natural and accessible web browsing experience. We explore these opportunities through \textit{conversational web browsing}, an emerging interaction paradigm for the Web that enables blind and visually impaired users (BVIP), as well as regular users, to access the contents and features of websites through conversational agents. Informed by the literature, our previous work and prototyping exercises, we derive a conceptual framework for supporting BVIP conversational web browsing needs, to then focus on the challenges of automatically providing this support, describing our early work and prototype that leverage heuristics that consider structural and content features only.
△ Less
Submitted 7 December, 2020;
originally announced December 2020.
-
Physics-informed GANs for Coastal Flood Visualization
Authors:
Björn Lütjens,
Brandon Leshchinskiy,
Christian Requena-Mesa,
Farrukh Chishtie,
Natalia Díaz-Rodriguez,
Océane Boulais,
Aaron Piña,
Dava Newman,
Alexander Lavin,
Yarin Gal,
Chedy Raïssi
Abstract:
As climate change increases the intensity of natural disasters, society needs better tools for adaptation. Floods, for example, are the most frequent natural disaster, but during hurricanes the area is largely covered by clouds and emergency managers must rely on nonintuitive flood visualizations for mission planning. To assist these emergency managers, we have created a deep learning pipeline tha…
▽ More
As climate change increases the intensity of natural disasters, society needs better tools for adaptation. Floods, for example, are the most frequent natural disaster, but during hurricanes the area is largely covered by clouds and emergency managers must rely on nonintuitive flood visualizations for mission planning. To assist these emergency managers, we have created a deep learning pipeline that generates visual satellite images of current and future coastal flooding. We advanced a state-of-the-art GAN called pix2pixHD, such that it produces imagery that is physically-consistent with the output of an expert-validated storm surge model (NOAA SLOSH). By evaluating the imagery relative to physics-based flood maps, we find that our proposed framework outperforms baseline models in both physical-consistency and photorealism. While this work focused on the visualization of coastal floods, we envision the creation of a global visualization of how climate change will shape our earth.
△ Less
Submitted 12 February, 2021; v1 submitted 15 October, 2020;
originally announced October 2020.
