-
Ensemble score filter with image inpainting for data assimilation in tracking surface quasi-geostrophic dynamics with partial observations
Authors:
Siming Liang,
Hoang Tran,
Feng Bao,
Hristo G. Chipilski,
Peter Jan van Leeuwen,
Guannan Zhang
Abstract:
Data assimilation plays a pivotal role in understanding and predicting turbulent systems within geoscience and weather forecasting, where data assimilation is used to address three fundamental challenges, i.e., high-dimensionality, nonlinearity, and partial observations. Recent advances in machine learning (ML)-based data assimilation methods have demonstrated encouraging results. In this work, we…
▽ More
Data assimilation plays a pivotal role in understanding and predicting turbulent systems within geoscience and weather forecasting, where data assimilation is used to address three fundamental challenges, i.e., high-dimensionality, nonlinearity, and partial observations. Recent advances in machine learning (ML)-based data assimilation methods have demonstrated encouraging results. In this work, we develop an ensemble score filter (EnSF) that integrates image inpainting to solve the data assimilation problems with partial observations. The EnSF method exploits an exclusively designed training-free diffusion models to solve high-dimensional nonlinear data assimilation problems. Its performance has been successfully demonstrated in the context of having full observations, i.e., all the state variables are directly or indirectly observed. However, because the EnSF does not use a covariance matrix to capture the dependence between the observed and unobserved state variables, it is nontrivial to extend the original EnSF method to the partial observation scenario. In this work, we incorporate various image inpainting techniques into the EnSF to predict the unobserved states during data assimilation. At each filtering step, we first use the diffusion model to estimate the observed states by integrating the likelihood information into the score function. Then, we use image inpainting methods to predict the unobserved state variables. We demonstrate the performance of the EnSF with inpainting by tracking the Surface Quasi-Geostrophic (SQG) model dynamics under a variety of scenarios. The successful proof of concept paves the way to more in-depth investigations on exploiting modern image inpainting techniques to advance data assimilation methodology for practical geoscience and weather forecasting problems.
△ Less
Submitted 20 January, 2025;
originally announced January 2025.
-
Using Mutual Information to measure Time-lags from non-linear processes in Astronomy
Authors:
Nachiketa Chakraborty,
Peter Jan van Leeuwen
Abstract:
Measuring time lags between time-series or lighcurves at different wavelengths from a variable or transient source in astronomy is an essential probe of physical mechanisms causing multiwavelength variability. Time-lags are typically quantified using discrete correlation functions (DCF) which are appropriate for linear relationships. However, in variable sources like X-ray binaries, active galacti…
▽ More
Measuring time lags between time-series or lighcurves at different wavelengths from a variable or transient source in astronomy is an essential probe of physical mechanisms causing multiwavelength variability. Time-lags are typically quantified using discrete correlation functions (DCF) which are appropriate for linear relationships. However, in variable sources like X-ray binaries, active galactic nuclei (AGN) and other accreting systems, the radiative processes and the resulting multiwavelength lightcurves often have non-linear relationships. For such systems it is more appropriate to use non-linear information-theoretic measures of causation like mutual information, routinely used in other disciplines. We demonstrate with toy models loopholes of using the standard DCF & show improvements when using the mutual information correlation function (MICF). For non-linear correlations, the latter accurately & sharply identifies the lag components as opposed to the DCF which can be erroneous. Following that we apply the MICF to the multiwavelength lightcurves of AGN NGC 4593. We find that X-ray fluxes lead UVW2 fluxes by ~0.2 days, closer to model predictions from reprocessing by the accretion disk than the DCF estimate. The uncertainties with the current lightcurves are too large though to rule out -ve lags. Additionally, we find another delay component at ~-1 day i.e. UVW2 leading X-rays consistent with inward propagating fluctuations in the accretion disk scenario. This is not detected by the DCF. Keeping in mind the non-linear relation between X-ray & UVW2, this is worthy of further theoretical investigation. From both toy models & real observations, it is clear that the mutual information based estimator is highly sensitive to complex non-linear correlations. With sufficiently high temporal resolution, we will precisely detect each of the lag features corresponding to these correlations.
△ Less
Submitted 16 June, 2021;
originally announced June 2021.
-
Massively Parallel Implicit Equal-Weights Particle Filter for Ocean Drift Trajectory Forecasting
Authors:
Håvard Heitlo Holm,
Martin Lilleeng Sætra,
Peter Jan van Leeuwen
Abstract:
Forecasting ocean drift trajectories are important for many applications, including search and rescue operations, oil spill cleanup and iceberg risk mitigation. In an operational setting, forecasts of drift trajectories are produced based on computationally demanding forecasts of three-dimensional ocean currents. Herein, we investigate a complementary approach for shorter time scales by using a re…
▽ More
Forecasting ocean drift trajectories are important for many applications, including search and rescue operations, oil spill cleanup and iceberg risk mitigation. In an operational setting, forecasts of drift trajectories are produced based on computationally demanding forecasts of three-dimensional ocean currents. Herein, we investigate a complementary approach for shorter time scales by using a recent state-of-the-art implicit equal-weights particle filter applied to a simplified ocean model. To achieve this, we present a new algorithmic design for a data-assimilation system in which all components - including the model, model errors, and particle filter - take advantage of massively parallel compute architectures, such as graphical processing units. Faster computations can enable in-situ and ad-hoc model runs for emergency management, and larger ensembles for better uncertainty quantification. Using a challenging test case with near-realistic chaotic instabilities, we run data-assimilation experiments based on synthetic observations from drifting and moored buoys, and analyse the trajectory forecasts for the drifters. Our results show that even sparse drifter observations are sufficient to significantly improve short-term drift forecasts up to twelve hours. With equidistant moored buoys observing only 0.1% of the state space, the ensemble gives an accurate description of the true state after data assimilation followed by a high-quality probabilistic forecast.
△ Less
Submitted 2 October, 2019;
originally announced October 2019.
-
Rainfall nowcasting by combining radars, microwave links and rain gauges
Authors:
Blandine Bianchi,
Peter Jan van Leeuwen,
Robin J. Hogan,
Alexis Berne
Abstract:
The objective of this work is to provide high-resolution rain rate maps at short lead-time forecasts (nowcasts) necessary to anticipate flooding and properly manage sewage systems in urban areas by combining radars, rain gauges, and operational microwave links, and taking into account their respective uncertainties. A variational approach (3D-Var) is used to find the best estimate for the rain rat…
▽ More
The objective of this work is to provide high-resolution rain rate maps at short lead-time forecasts (nowcasts) necessary to anticipate flooding and properly manage sewage systems in urban areas by combining radars, rain gauges, and operational microwave links, and taking into account their respective uncertainties. A variational approach (3D-Var) is used to find the best estimate for the rain rate, and its error covariance, from the different rain sensors. Short-term rain rate forecasts are then produced by assuming Lagrangian persistence. A velocity field is obtained from the operational radar-derived rain fields, and the rain rate field is advected using the Total Variance Diminishing (TVD) scheme. The error covariance associated to the estimated rain rate is also propagated, and we use these two in the 3D-Var at the next observation time step. This approach can be seen as a Variational Kalman Filter (VKF), in which the covariance of the prior is not constant but dependent on time. The proposed approach has been tested using data from 14 rain gauges, 14 microwave links and the operational radar rain product from MeteoSwiss in the area of Zurich (Switzerland). During the applications the assumption of the Lagrangian persistence appears to be valid up to 20 min (a bit longer for stratiform events). During convective events, the algorithm is less powerful and shorter lead times should be considered (i.e., 15 min). Although such lead times are short, they are still useful to various hydrological and outdoor applications.
△ Less
Submitted 28 October, 2018;
originally announced October 2018.
-
Particle filters for high-dimensional geoscience applications: a review
Authors:
Peter Jan van Leeuwen,
Hans R. Künsch,
Lars Nerger,
Roland Potthast,
Sebastian Reich
Abstract:
Particle filters contain the promise of fully nonlinear data assimilation. They have been applied in numerous science areas, but their application to the geosciences has been limited due to their inefficiency in high-dimensional systems in standard settings. However, huge progress has been made, and this limitation is disappearing fast due to recent developments in proposal densities, the use of i…
▽ More
Particle filters contain the promise of fully nonlinear data assimilation. They have been applied in numerous science areas, but their application to the geosciences has been limited due to their inefficiency in high-dimensional systems in standard settings. However, huge progress has been made, and this limitation is disappearing fast due to recent developments in proposal densities, the use of ideas from (optimal) transportation, the use of localisation and intelligent adaptive resampling strategies. Furthermore, powerful hybrids between particle filters and ensemble Kalman filters and variational methods have been developed. We present a state of the art discussion of present efforts of developing particle filters for highly nonlinear geoscience state-estimation problems with an emphasis on atmospheric and oceanic applications, including many new ideas, derivations, and unifications, highlighting hidden connections, and generating a valuable tool and guide for the community. Initial experiments show that particle filters can be competitive with present-day methods for numerical weather prediction suggesting that they will become mainstream soon.
△ Less
Submitted 13 April, 2019; v1 submitted 27 July, 2018;
originally announced July 2018.
