-
Achieving Diversity in Objective Space for Sample-efficient Search of Multiobjective Optimization Problems
Authors:
Eric Hans Lee,
Bolong Cheng,
Michael McCourt
Abstract:
Efficiently solving multi-objective optimization problems for simulation optimization of important scientific and engineering applications such as materials design is becoming an increasingly important research topic. This is due largely to the expensive costs associated with said applications, and the resulting need for sample-efficient, multiobjective optimization methods that efficiently explor…
▽ More
Efficiently solving multi-objective optimization problems for simulation optimization of important scientific and engineering applications such as materials design is becoming an increasingly important research topic. This is due largely to the expensive costs associated with said applications, and the resulting need for sample-efficient, multiobjective optimization methods that efficiently explore the Pareto frontier to expose a promising set of design solutions. We propose moving away from using explicit optimization to identify the Pareto frontier and instead suggest searching for a diverse set of outcomes that satisfy user-specified performance criteria. This method presents decision makers with a robust pool of promising design decisions and helps them better understand the space of good solutions. To achieve this outcome, we introduce the Likelihood of Metric Satisfaction (LMS) acquisition function, analyze its behavior and properties, and demonstrate its viability on various problems.
△ Less
Submitted 23 June, 2023;
originally announced June 2023.
-
Bayesian Optimization is Superior to Random Search for Machine Learning Hyperparameter Tuning: Analysis of the Black-Box Optimization Challenge 2020
Authors:
Ryan Turner,
David Eriksson,
Michael McCourt,
Juha Kiili,
Eero Laaksonen,
Zhen Xu,
Isabelle Guyon
Abstract:
This paper presents the results and insights from the black-box optimization (BBO) challenge at NeurIPS 2020 which ran from July-October, 2020. The challenge emphasized the importance of evaluating derivative-free optimizers for tuning the hyperparameters of machine learning models. This was the first black-box optimization challenge with a machine learning emphasis. It was based on tuning (valida…
▽ More
This paper presents the results and insights from the black-box optimization (BBO) challenge at NeurIPS 2020 which ran from July-October, 2020. The challenge emphasized the importance of evaluating derivative-free optimizers for tuning the hyperparameters of machine learning models. This was the first black-box optimization challenge with a machine learning emphasis. It was based on tuning (validation set) performance of standard machine learning models on real datasets. This competition has widespread impact as black-box optimization (e.g., Bayesian optimization) is relevant for hyperparameter tuning in almost every machine learning project as well as many applications outside of machine learning. The final leaderboard was determined using the optimization performance on held-out (hidden) objective functions, where the optimizers ran without human intervention. Baselines were set using the default settings of several open-source black-box optimization packages as well as random search.
△ Less
Submitted 31 August, 2021; v1 submitted 20 April, 2021;
originally announced April 2021.
-
Quasi-Monte Carlo Software
Authors:
Sou-Cheng T. Choi,
Fred J. Hickernell,
R. Jagadeeswaran,
Michael J. McCourt,
Aleksei G. Sorokin
Abstract:
Practitioners wishing to experience the efficiency gains from using low discrepancy sequences need correct, robust, well-written software. This article, based on our MCQMC 2020 tutorial, describes some of the better quasi-Monte Carlo (QMC) software available. We highlight the key software components required by QMC to approximate multivariate integrals or expectations of functions of vector random…
▽ More
Practitioners wishing to experience the efficiency gains from using low discrepancy sequences need correct, robust, well-written software. This article, based on our MCQMC 2020 tutorial, describes some of the better quasi-Monte Carlo (QMC) software available. We highlight the key software components required by QMC to approximate multivariate integrals or expectations of functions of vector random variables. We have combined these components in QMCPy, a Python open-source library, which we hope will draw the support of the QMC community. Here we introduce QMCPy.
△ Less
Submitted 14 October, 2021; v1 submitted 15 February, 2021;
originally announced February 2021.
-
Efficient Rollout Strategies for Bayesian Optimization
Authors:
Eric Hans Lee,
David Eriksson,
Bolong Cheng,
Michael McCourt,
David Bindel
Abstract:
Bayesian optimization (BO) is a class of sample-efficient global optimization methods, where a probabilistic model conditioned on previous observations is used to determine future evaluations via the optimization of an acquisition function. Most acquisition functions are myopic, meaning that they only consider the impact of the next function evaluation. Non-myopic acquisition functions consider th…
▽ More
Bayesian optimization (BO) is a class of sample-efficient global optimization methods, where a probabilistic model conditioned on previous observations is used to determine future evaluations via the optimization of an acquisition function. Most acquisition functions are myopic, meaning that they only consider the impact of the next function evaluation. Non-myopic acquisition functions consider the impact of the next $h$ function evaluations and are typically computed through rollout, in which $h$ steps of BO are simulated. These rollout acquisition functions are defined as $h$-dimensional integrals, and are expensive to compute and optimize. We show that a combination of quasi-Monte Carlo, common random numbers, and control variates significantly reduce the computational burden of rollout. We then formulate a policy-search based approach that removes the need to optimize the rollout acquisition function. Finally, we discuss the qualitative behavior of rollout policies in the setting of multi-modal objectives and model error.
△ Less
Submitted 18 June, 2020; v1 submitted 24 February, 2020;
originally announced February 2020.
-
Sampling Humans for Optimizing Preferences in Coloring Artwork
Authors:
Michael McCourt,
Ian Dewancker
Abstract:
Many circumstances of practical importance have performance or success metrics which exist implicitly---in the eye of the beholder, so to speak. Tuning aspects of such problems requires working without defined metrics and only considering pairwise comparisons or rankings. In this paper, we review an existing Bayesian optimization strategy for determining most-preferred outcomes, and identify an ad…
▽ More
Many circumstances of practical importance have performance or success metrics which exist implicitly---in the eye of the beholder, so to speak. Tuning aspects of such problems requires working without defined metrics and only considering pairwise comparisons or rankings. In this paper, we review an existing Bayesian optimization strategy for determining most-preferred outcomes, and identify an adaptation to allow it to handle ties. We then discuss some of the issues we have encountered when humans use this optimization strategy to optimize coloring a piece of abstract artwork. We hope that, by participating in this workshop, we can learn how other researchers encounter difficulties unique to working with humans in the loop.
△ Less
Submitted 10 June, 2019;
originally announced June 2019.
-
Bayesian Optimization with Approximate Set Kernels
Authors:
Jungtaek Kim,
Michael McCourt,
Tackgeun You,
Saehoon Kim,
Seungjin Choi
Abstract:
We propose a practical Bayesian optimization method over sets, to minimize a black-box function that takes a set as a single input. Because set inputs are permutation-invariant, traditional Gaussian process-based Bayesian optimization strategies which assume vector inputs can fall short. To address this, we develop a Bayesian optimization method with \emph{set kernel} that is used to build surroga…
▽ More
We propose a practical Bayesian optimization method over sets, to minimize a black-box function that takes a set as a single input. Because set inputs are permutation-invariant, traditional Gaussian process-based Bayesian optimization strategies which assume vector inputs can fall short. To address this, we develop a Bayesian optimization method with \emph{set kernel} that is used to build surrogate functions. This kernel accumulates similarity over set elements to enforce permutation-invariance, but this comes at a greater computational cost. To reduce this burden, we propose two key components: (i) a more efficient approximate set kernel which is still positive-definite and is an unbiased estimator of the true set kernel with upper-bounded variance in terms of the number of subsamples, (ii) a constrained acquisition function optimization over sets, which uses symmetry of the feasible region that defines a set input. Finally, we present several numerical experiments which demonstrate that our method outperforms other methods.
△ Less
Submitted 24 January, 2021; v1 submitted 23 May, 2019;
originally announced May 2019.
-
Orchestrate: Infrastructure for Enabling Parallelism during Hyperparameter Optimization
Authors:
Alexandra Johnson,
Michael McCourt
Abstract:
Two key factors dominate the development of effective production grade machine learning models. First, it requires a local software implementation and iteration process. Second, it requires distributed infrastructure to efficiently conduct training and hyperparameter optimization. While modern machine learning frameworks are very effective at the former, practitioners are often left building ad ho…
▽ More
Two key factors dominate the development of effective production grade machine learning models. First, it requires a local software implementation and iteration process. Second, it requires distributed infrastructure to efficiently conduct training and hyperparameter optimization. While modern machine learning frameworks are very effective at the former, practitioners are often left building ad hoc frameworks for the latter. We present SigOpt Orchestrate, a library for such simultaneous training in a cloud environment. We describe the motivating factors and resulting design of this library, feedback from initial testing, and future goals.
△ Less
Submitted 18 December, 2018;
originally announced December 2018.
-
Sequential Preference-Based Optimization
Authors:
Ian Dewancker,
Jakob Bauer,
Michael McCourt
Abstract:
Many real-world engineering problems rely on human preferences to guide their design and optimization. We present PrefOpt, an open source package to simplify sequential optimization tasks that incorporate human preference feedback. Our approach extends an existing latent variable model for binary preferences to allow for observations of equivalent preference from users.
Many real-world engineering problems rely on human preferences to guide their design and optimization. We present PrefOpt, an open source package to simplify sequential optimization tasks that incorporate human preference feedback. Our approach extends an existing latent variable model for binary preferences to allow for observations of equivalent preference from users.
△ Less
Submitted 8 January, 2018;
originally announced January 2018.
-
Practical Bayesian optimization in the presence of outliers
Authors:
Ruben Martinez-Cantin,
Kevin Tee,
Michael McCourt
Abstract:
Inference in the presence of outliers is an important field of research as outliers are ubiquitous and may arise across a variety of problems and domains. Bayesian optimization is method that heavily relies on probabilistic inference. This allows outstanding sample efficiency because the probabilistic machinery provides a memory of the whole optimization process. However, that virtue becomes a dis…
▽ More
Inference in the presence of outliers is an important field of research as outliers are ubiquitous and may arise across a variety of problems and domains. Bayesian optimization is method that heavily relies on probabilistic inference. This allows outstanding sample efficiency because the probabilistic machinery provides a memory of the whole optimization process. However, that virtue becomes a disadvantage when the memory is populated with outliers, inducing bias in the estimation. In this paper, we present an empirical evaluation of Bayesian optimization methods in the presence of outliers. The empirical evidence shows that Bayesian optimization with robust regression often produces suboptimal results. We then propose a new algorithm which combines robust regression (a Gaussian process with Student-t likelihood) with outlier diagnostics to classify data points as outliers or inliers. By using an scheduler for the classification of outliers, our method is more efficient and has better convergence over the standard robust regression. Furthermore, we show that even in controlled situations with no expected outliers, our method is able to produce better results.
△ Less
Submitted 12 December, 2017;
originally announced December 2017.
-
Active Preference Learning for Personalized Portfolio Construction
Authors:
Kevin Tee,
Michael McCourt,
Ruben Martinez-Cantin,
Ian Dewancker,
Frank Liu
Abstract:
In financial asset management, choosing a portfolio requires balancing returns, risk, exposure, liquidity, volatility and other factors. These concerns are difficult to compare explicitly, with many asset managers using an intuitive or implicit sense of their interaction. We propose a mechanism for learning someone's sense of distinctness between portfolios with the goal of being able to identify…
▽ More
In financial asset management, choosing a portfolio requires balancing returns, risk, exposure, liquidity, volatility and other factors. These concerns are difficult to compare explicitly, with many asset managers using an intuitive or implicit sense of their interaction. We propose a mechanism for learning someone's sense of distinctness between portfolios with the goal of being able to identify portfolios which are predicted to perform well but are distinct from the perspective of the user. This identification occurs, e.g., in the context of Bayesian optimization of a backtested performance metric. Numerical experiments are presented which show the impact of personal beliefs in informing the development of a diverse and high-performing portfolio.
△ Less
Submitted 24 August, 2017;
originally announced August 2017.
-
Robust Bayesian Optimization with Student-t Likelihood
Authors:
Ruben Martinez-Cantin,
Michael McCourt,
Kevin Tee
Abstract:
Bayesian optimization has recently attracted the attention of the automatic machine learning community for its excellent results in hyperparameter tuning. BO is characterized by the sample efficiency with which it can optimize expensive black-box functions. The efficiency is achieved in a similar fashion to the learning to learn methods: surrogate models (typically in the form of Gaussian processe…
▽ More
Bayesian optimization has recently attracted the attention of the automatic machine learning community for its excellent results in hyperparameter tuning. BO is characterized by the sample efficiency with which it can optimize expensive black-box functions. The efficiency is achieved in a similar fashion to the learning to learn methods: surrogate models (typically in the form of Gaussian processes) learn the target function and perform intelligent sampling. This surrogate model can be applied even in the presence of noise; however, as with most regression methods, it is very sensitive to outlier data. This can result in erroneous predictions and, in the case of BO, biased and inefficient exploration. In this work, we present a GP model that is robust to outliers which uses a Student-t likelihood to segregate outliers and robustly conduct Bayesian optimization. We present numerical results evaluating the proposed method in both artificial functions and real problems.
△ Less
Submitted 18 July, 2017;
originally announced July 2017.
-
Bayesian Optimization for Machine Learning : A Practical Guidebook
Authors:
Ian Dewancker,
Michael McCourt,
Scott Clark
Abstract:
The engineering of machine learning systems is still a nascent field; relying on a seemingly daunting collection of quickly evolving tools and best practices. It is our hope that this guidebook will serve as a useful resource for machine learning practitioners looking to take advantage of Bayesian optimization techniques. We outline four example machine learning problems that can be solved using o…
▽ More
The engineering of machine learning systems is still a nascent field; relying on a seemingly daunting collection of quickly evolving tools and best practices. It is our hope that this guidebook will serve as a useful resource for machine learning practitioners looking to take advantage of Bayesian optimization techniques. We outline four example machine learning problems that can be solved using open source machine learning libraries, and highlight the benefits of using Bayesian optimization in the context of these common machine learning applications.
△ Less
Submitted 14 December, 2016;
originally announced December 2016.
-
Preemptive Termination of Suggestions during Sequential Kriging Optimization of a Brain Activity Reconstruction Simulation
Authors:
Michael McCourt,
Ian Dewancker,
Salvatore Ganci
Abstract:
Reconstructing brain activity through electroencephalography requires a boundary value problem (BVP) solver to take a proposed distribution of current dipoles within the brain and compute the resulting electrostatic potential on the scalp. This article proposes the use of sequential kriging optimization to identify different optimal BVP solver parameters for dipoles located in isolated sections of…
▽ More
Reconstructing brain activity through electroencephalography requires a boundary value problem (BVP) solver to take a proposed distribution of current dipoles within the brain and compute the resulting electrostatic potential on the scalp. This article proposes the use of sequential kriging optimization to identify different optimal BVP solver parameters for dipoles located in isolated sections of the brain by considering the cumulative impact of randomly oriented dipoles within a chosen isolated section. We attempt preemptive termination of parametrizations suggested during the sequential kriging optimization which, given the results to that point, seem unlikely to produce high quality solutions. Numerical experiments on a simplification of the full geometry for which an approximate solution is available show a benefit from this preemptive termination.
△ Less
Submitted 13 December, 2016;
originally announced December 2016.
-
Evaluation System for a Bayesian Optimization Service
Authors:
Ian Dewancker,
Michael McCourt,
Scott Clark,
Patrick Hayes,
Alexandra Johnson,
George Ke
Abstract:
Bayesian optimization is an elegant solution to the hyperparameter optimization problem in machine learning. Building a reliable and robust Bayesian optimization service requires careful testing methodology and sound statistical analysis. In this talk we will outline our development of an evaluation framework to rigorously test and measure the impact of changes to the SigOpt optimization service.…
▽ More
Bayesian optimization is an elegant solution to the hyperparameter optimization problem in machine learning. Building a reliable and robust Bayesian optimization service requires careful testing methodology and sound statistical analysis. In this talk we will outline our development of an evaluation framework to rigorously test and measure the impact of changes to the SigOpt optimization service. We present an overview of our evaluation system and discuss how this framework empowers our research engineers to confidently and quickly make changes to our core optimization engine
△ Less
Submitted 19 May, 2016;
originally announced May 2016.
-
A Stratified Analysis of Bayesian Optimization Methods
Authors:
Ian Dewancker,
Michael McCourt,
Scott Clark,
Patrick Hayes,
Alexandra Johnson,
George Ke
Abstract:
Empirical analysis serves as an important complement to theoretical analysis for studying practical Bayesian optimization. Often empirical insights expose strengths and weaknesses inaccessible to theoretical analysis. We define two metrics for comparing the performance of Bayesian optimization methods and propose a ranking mechanism for summarizing performance within various genres or strata of te…
▽ More
Empirical analysis serves as an important complement to theoretical analysis for studying practical Bayesian optimization. Often empirical insights expose strengths and weaknesses inaccessible to theoretical analysis. We define two metrics for comparing the performance of Bayesian optimization methods and propose a ranking mechanism for summarizing performance within various genres or strata of test functions. These test functions serve to mimic the complexity of hyperparameter optimization problems, the most prominent application of Bayesian optimization, but with a closed form which allows for rapid evaluation and more predictable behavior. This offers a flexible and efficient way to investigate functions with specific properties of interest, such as oscillatory behavior or an optimum on the domain boundary.
△ Less
Submitted 30 March, 2016;
originally announced March 2016.
