Particle Filter Optimization: A Bayesian Approach for Global Stochastic Optimization
Authors:
Mostafa Eslami,
Maryam Babazadeh
Abstract:
This paper proposes a novel global optimization algorithm, Particle Filter-Based Optimization (PFO), designed for a class of stochastic optimization problems in which the objective function lacks an analytical form and is subject to noisy evaluations. PFO utilizes the Bayesian inference framework of Particle Filters (PF) by reformulating the optimization task as a state estimation problem. In this…
▽ More
This paper proposes a novel global optimization algorithm, Particle Filter-Based Optimization (PFO), designed for a class of stochastic optimization problems in which the objective function lacks an analytical form and is subject to noisy evaluations. PFO utilizes the Bayesian inference framework of Particle Filters (PF) by reformulating the optimization task as a state estimation problem. In this context, evaluations of the objective function are interpreted as measurements, and a customized transition model based on covariance ellipsoids is introduced to guide particle propagation. This model serves as a surrogate for classical acquisition functions, equipping the PF framework with local search capabilities and supporting efficient exploration of the global optimum. To mitigate the adverse effects of measurement noise, the Unscented Transform (UT) is employed to approximate the underlying mean of the objective function, enhancing the accuracy of particle updates. The algorithm offers notable improvements over existing stochastic optimization algorithms for black-box multi-modal objective functions. First, PFO provides a fully probabilistic definition of particle weights, enhancing adaptability and robustness. Second, PFO integrates exploration and exploitation within a unified Bayesian framework, ensuring a non-zero probability of sampling from unexplored regions throughout the optimization process. This approach contrasts with traditional particle filter methods that are primarily used for state estimation, and heuristic optimization algorithms that lack theoretical guarantees. The novelty of PFO lies in its unique integration of particle filtering with a dynamic search space prediction, offering a theoretically grounded alternative to acquisition functions in Bayesian Optimization (BO).
△ Less
Submitted 20 June, 2025; v1 submitted 5 June, 2024;
originally announced June 2024.
Extracting Global Dynamics of Loss Landscape in Deep Learning Models
Authors:
Mohammed Eslami,
Hamed Eramian,
Marcio Gameiro,
William Kalies,
Konstantin Mischaikow
Abstract:
Deep learning models evolve through training to learn the manifold in which the data exists to satisfy an objective. It is well known that evolution leads to different final states which produce inconsistent predictions of the same test data points. This calls for techniques to be able to empirically quantify the difference in the trajectories and highlight problematic regions. While much focus is…
▽ More
Deep learning models evolve through training to learn the manifold in which the data exists to satisfy an objective. It is well known that evolution leads to different final states which produce inconsistent predictions of the same test data points. This calls for techniques to be able to empirically quantify the difference in the trajectories and highlight problematic regions. While much focus is placed on discovering what models learn, the question of how a model learns is less studied beyond theoretical landscape characterizations and local geometric approximations near optimal conditions. Here, we present a toolkit for the Dynamical Organization Of Deep Learning Loss Landscapes, or DOODL3. DOODL3 formulates the training of neural networks as a dynamical system, analyzes the learning process, and presents an interpretable global view of trajectories in the loss landscape. Our approach uses the coarseness of topology to capture the granularity of geometry to mitigate against states of instability or elongated training. Overall, our analysis presents an empirical framework to extract the global dynamics of a model and to use that information to guide the training of neural networks.
△ Less
Submitted 14 June, 2021;
originally announced June 2021.
