-
Using the Empirical Attainment Function for Analyzing Single-objective Black-box Optimization Algorithms
Authors:
Manuel López-Ibáñez,
Diederick Vermetten,
Johann Dreo,
Carola Doerr
Abstract:
A widely accepted way to assess the performance of iterative black-box optimizers is to analyze their empirical cumulative distribution function (ECDF) of pre-defined quality targets achieved not later than a given runtime. In this work, we consider an alternative approach, based on the empirical attainment function (EAF) and we show that the target-based ECDF is an approximation of the EAF. We ar…
▽ More
A widely accepted way to assess the performance of iterative black-box optimizers is to analyze their empirical cumulative distribution function (ECDF) of pre-defined quality targets achieved not later than a given runtime. In this work, we consider an alternative approach, based on the empirical attainment function (EAF) and we show that the target-based ECDF is an approximation of the EAF. We argue that the EAF has several advantages over the target-based ECDF. In particular, it does not require defining a priori quality targets per function, captures performance differences more precisely, and enables the use of additional summary statistics that enrich the analysis. We also show that the average area over the convergence curves is a simpler-to-calculate, but equivalent, measure of anytime performance. To facilitate the accessibility of the EAF, we integrate a module to compute it into the IOHanalyzer platform. Finally, we illustrate the use of the EAF via synthetic examples and via the data available for the BBOB suite.
△ Less
Submitted 20 September, 2024; v1 submitted 2 April, 2024;
originally announced April 2024.
-
An Exact Framework for Solving the Space-Time Dependent TSP
Authors:
Isaac Rudich,
Quentin Cappart,
Manuel López-Ibáñez,
Michael Römer,
Louis-Martin Rousseau
Abstract:
Many real-world scenarios involve solving bi-level optimization problems in which there is an outer discrete optimization problem, and an inner problem involving expensive or black-box computation. This arises in space-time dependent variants of the Traveling Salesman Problem, such as when planning space missions that visit multiple astronomical objects. Planning these missions presents significan…
▽ More
Many real-world scenarios involve solving bi-level optimization problems in which there is an outer discrete optimization problem, and an inner problem involving expensive or black-box computation. This arises in space-time dependent variants of the Traveling Salesman Problem, such as when planning space missions that visit multiple astronomical objects. Planning these missions presents significant challenges due to the constant relative motion of the objects involved. There is an outer combinatorial problem of finding the optimal order to visit the objects and an inner optimization problem that requires finding the optimal departure time and trajectory to travel between each pair of objects. The constant motion of the objects complicates the inner problem, making it computationally expensive. This paper introduces a novel framework utilizing decision diagrams (DDs) and a DD-based branch-and-bound technique, Peel-and-Bound, to achieve exact solutions for such bi-level optimization problems, assuming sufficient inner problem optimizer quality. The framework leverages problem-specific knowledge to expedite search processes and minimize the number of expensive evaluations required. As a case study, we apply this framework to the Asteroid Routing Problem (ARP), a benchmark problem in global trajectory optimization. Experimental results demonstrate the framework's scalability and ability to generate robust heuristic solutions for ARP instances. Many of these solutions are exact, contingent on the assumed quality of the inner problem's optimizer.
△ Less
Submitted 11 July, 2024; v1 submitted 3 December, 2023;
originally announced December 2023.
-
A Study of Scalarisation Techniques for Multi-Objective QUBO Solving
Authors:
Mayowa Ayodele,
Richard Allmendinger,
Manuel López-Ibáñez,
Matthieu Parizy
Abstract:
In recent years, there has been significant research interest in solving Quadratic Unconstrained Binary Optimisation (QUBO) problems. Physics-inspired optimisation algorithms have been proposed for deriving optimal or sub-optimal solutions to QUBOs. These methods are particularly attractive within the context of using specialised hardware, such as quantum computers, application specific CMOS and o…
▽ More
In recent years, there has been significant research interest in solving Quadratic Unconstrained Binary Optimisation (QUBO) problems. Physics-inspired optimisation algorithms have been proposed for deriving optimal or sub-optimal solutions to QUBOs. These methods are particularly attractive within the context of using specialised hardware, such as quantum computers, application specific CMOS and other high performance computing resources for solving optimisation problems. These solvers are then applied to QUBO formulations of combinatorial optimisation problems. Quantum and quantum-inspired optimisation algorithms have shown promising performance when applied to academic benchmarks as well as real-world problems. However, QUBO solvers are single objective solvers. To make them more efficient at solving problems with multiple objectives, a decision on how to convert such multi-objective problems to single-objective problems need to be made. In this study, we compare methods of deriving scalarisation weights when combining two objectives of the cardinality constrained mean-variance portfolio optimisation problem into one. We show significant performance improvement (measured in terms of hypervolume) when using a method that iteratively fills the largest space in the Pareto front compared to a näive approach using uniformly generated weights.
△ Less
Submitted 20 October, 2022;
originally announced October 2022.
-
Detection of Hidden Objectives and Interactive Objective Reduction
Authors:
Seyed Mahdi Shavarani,
Manuel López-Ibáñez,
Richard Allmendinger
Abstract:
In multi-objective optimization problems, there might exist hidden objectives that are important to the decision-maker but are not being optimized. On the other hand, there might also exist irrelevant objectives that are being optimized but are of less importance to the DM. The question that arises here is whether it is possible to detect and reduce irrelevant objectives without deteriorating the…
▽ More
In multi-objective optimization problems, there might exist hidden objectives that are important to the decision-maker but are not being optimized. On the other hand, there might also exist irrelevant objectives that are being optimized but are of less importance to the DM. The question that arises here is whether it is possible to detect and reduce irrelevant objectives without deteriorating the quality of the final results? In fact, when dealing with multi-objective problems, each objective implies a significant cost best avoided if possible. However, existing methods that pertain to the reduction of objectives are computationally intensive and ignore the preferences of the decision-maker. In this paper, we propose an approach to exploit the capabilities of interactive evolutionary multi-objective optimization algorithms (EMOAs) and the preference information provided by the decision-maker, to detect and eliminate the irrelevant objectives during the optimization process and replace them with hidden ones, if any. The proposed method, which is based on univariate feature selection, is computationally effective and can be integrated into any ranking-based interactive EMOA. Using synthetic problems developed in this study, we motivate different scenarios in the experiments and prove the effectiveness of the proposed method in improving the computational cost, the total number of objective evaluations, and the quality of the final solutions.
△ Less
Submitted 2 June, 2022;
originally announced June 2022.
-
The Asteroid Routing Problem: A Benchmark for Expensive Black-Box Permutation Optimization
Authors:
Manuel López-Ibáñez,
Francisco Chicano,
Rodrigo Gil-Merino
Abstract:
Inspired by the recent 11th Global Trajectory Optimisation Competition, this paper presents the asteroid routing problem (ARP) as a realistic benchmark of algorithms for expensive bound-constrained black-box optimization in permutation space. Given a set of asteroids' orbits and a departure epoch, the goal of the ARP is to find the optimal sequence for visiting the asteroids, starting from Earth's…
▽ More
Inspired by the recent 11th Global Trajectory Optimisation Competition, this paper presents the asteroid routing problem (ARP) as a realistic benchmark of algorithms for expensive bound-constrained black-box optimization in permutation space. Given a set of asteroids' orbits and a departure epoch, the goal of the ARP is to find the optimal sequence for visiting the asteroids, starting from Earth's orbit, in order to minimize both the cost, measured as the sum of the magnitude of velocity changes required to complete the trip, and the time, measured as the time elapsed from the departure epoch until visiting the last asteroid. We provide open-source code for generating instances of arbitrary sizes and evaluating solutions to the problem. As a preliminary analysis, we compare the results of two methods for expensive black-box optimization in permutation spaces, namely, Combinatorial Efficient Global Optimization (CEGO), a Bayesian optimizer based on Gaussian processes, and Unbalanced Mallows Model (UMM), an estimation-of-distribution algorithm based on probabilistic Mallows models. We investigate the best permutation representation for each algorithm, either rank-based or order-based. Moreover, we analyze the effect of providing a good initial solution, generated by a greedy nearest neighbor heuristic, on the performance of the algorithms. The results suggest directions for improvements in the algorithms being compared.
△ Less
Submitted 19 April, 2022; v1 submitted 29 March, 2022;
originally announced March 2022.
-
Managing Manufacturing and Delivery of Personalised Medicine: Current and Future Models
Authors:
Andreea Avramescu,
Richard Allmendinger,
Manuel López-Ibáñez
Abstract:
With almost 50% of annual commercial drug approvals being Personalised Medicine (PM) and its huge potential to improve quality of life, this emerging medical sector has received increased attention from the industry and medical research, driven by health and care services, and us, the patients. Notwithstanding the power of Advanced Therapy Medicinal Products (ATMPs) to treat progressive illnesses…
▽ More
With almost 50% of annual commercial drug approvals being Personalised Medicine (PM) and its huge potential to improve quality of life, this emerging medical sector has received increased attention from the industry and medical research, driven by health and care services, and us, the patients. Notwithstanding the power of Advanced Therapy Medicinal Products (ATMPs) to treat progressive illnesses and rare genetic conditions, their delivery on large scale is still problematic. The biopharmaceutical companies are currently struggling to meet timely delivery and, given high prices of up to $2 million per patient, prove the cost-effectiveness of their ATMP. The fragility of ATMPs combined with the impossibility for replacements due to the nature of the treatment and the advanced stages of the patient's condition are some of the bottlenecks added to a generally critical supply chain. As a consequence, ATMPs are currently used in most cases only as a last resort. ATMPs are at the intersection of multiple healthcare logistic networks and, due to their novelty, research around their commercialisation is still in its infancy from an operations research perspective. To accelerate technology adoption in this domain, we characterize pertinent practical challenges in a PM supply chain and then capture them in a holistic mathematical model ready for optimisation. The identified challenges and derived model will be contrasted with literature of related supply chains in terms of model formulations and suitable optimisation methods. Finally, needed technological advancements are discussed to pave the way to affordable commercialisation of PM.
△ Less
Submitted 21 May, 2021;
originally announced May 2021.
-
Reproducibility in Evolutionary Computation
Authors:
Manuel López-Ibáñez,
Juergen Branke,
Luís Paquete
Abstract:
Experimental studies are prevalent in Evolutionary Computation (EC), and concerns about the reproducibility and replicability of such studies have increased in recent times, reflecting similar concerns in other scientific fields. In this article, we discuss, within the context of EC, the different types of reproducibility and suggest a classification that refines the badge system of the Associatio…
▽ More
Experimental studies are prevalent in Evolutionary Computation (EC), and concerns about the reproducibility and replicability of such studies have increased in recent times, reflecting similar concerns in other scientific fields. In this article, we discuss, within the context of EC, the different types of reproducibility and suggest a classification that refines the badge system of the Association of Computing Machinery (ACM) adopted by ACM Transactions on Evolutionary Learning and Optimization (https://dlnext.acm.org/journal/telo). We identify cultural and technical obstacles to reproducibility in the EC field. Finally, we provide guidelines and suggest tools that may help to overcome some of these reproducibility obstacles.
△ Less
Submitted 29 June, 2021; v1 submitted 5 February, 2021;
originally announced February 2021.
-
Safe Learning and Optimization Techniques: Towards a Survey of the State of the Art
Authors:
Youngmin Kim,
Richard Allmendinger,
Manuel López-Ibáñez
Abstract:
Safe learning and optimization deals with learning and optimization problems that avoid, as much as possible, the evaluation of non-safe input points, which are solutions, policies, or strategies that cause an irrecoverable loss (e.g., breakage of a machine or equipment, or life threat). Although a comprehensive survey of safe reinforcement learning algorithms was published in 2015, a number of ne…
▽ More
Safe learning and optimization deals with learning and optimization problems that avoid, as much as possible, the evaluation of non-safe input points, which are solutions, policies, or strategies that cause an irrecoverable loss (e.g., breakage of a machine or equipment, or life threat). Although a comprehensive survey of safe reinforcement learning algorithms was published in 2015, a number of new algorithms have been proposed thereafter, and related works in active learning and in optimization were not considered. This paper reviews those algorithms from a number of domains including reinforcement learning, Gaussian process regression and classification, evolutionary algorithms, and active learning. We provide the fundamental concepts on which the reviewed algorithms are based and a characterization of the individual algorithms. We conclude by explaining how the algorithms are connected and suggestions for future research.
△ Less
Submitted 23 June, 2021; v1 submitted 23 January, 2021;
originally announced January 2021.
-
Benchmarking in Optimization: Best Practice and Open Issues
Authors:
Thomas Bartz-Beielstein,
Carola Doerr,
Daan van den Berg,
Jakob Bossek,
Sowmya Chandrasekaran,
Tome Eftimov,
Andreas Fischbach,
Pascal Kerschke,
William La Cava,
Manuel Lopez-Ibanez,
Katherine M. Malan,
Jason H. Moore,
Boris Naujoks,
Patryk Orzechowski,
Vanessa Volz,
Markus Wagner,
Thomas Weise
Abstract:
This survey compiles ideas and recommendations from more than a dozen researchers with different backgrounds and from different institutes around the world. Promoting best practice in benchmarking is its main goal. The article discusses eight essential topics in benchmarking: clearly stated goals, well-specified problems, suitable algorithms, adequate performance measures, thoughtful analysis, eff…
▽ More
This survey compiles ideas and recommendations from more than a dozen researchers with different backgrounds and from different institutes around the world. Promoting best practice in benchmarking is its main goal. The article discusses eight essential topics in benchmarking: clearly stated goals, well-specified problems, suitable algorithms, adequate performance measures, thoughtful analysis, effective and efficient designs, comprehensible presentations, and guaranteed reproducibility. The final goal is to provide well-accepted guidelines (rules) that might be useful for authors and reviewers. As benchmarking in optimization is an active and evolving field of research this manuscript is meant to co-evolve over time by means of periodic updates.
△ Less
Submitted 16 December, 2020; v1 submitted 7 July, 2020;
originally announced July 2020.
