-
Exploring the application of quantum technologies to industrial and real-world use cases
Authors:
Eneko Osaba,
Esther Villar-Rodriguez,
Izaskun Oregi
Abstract:
Recent advancements in quantum computing are leading to an era of practical utility, enabling the tackling of increasingly complex problems. The goal of this era is to leverage quantum computing to solve real-world problems in fields such as machine learning, optimization, and material simulation, using revolutionary quantum methods and machines. All this progress has been achieved even while bein…
▽ More
Recent advancements in quantum computing are leading to an era of practical utility, enabling the tackling of increasingly complex problems. The goal of this era is to leverage quantum computing to solve real-world problems in fields such as machine learning, optimization, and material simulation, using revolutionary quantum methods and machines. All this progress has been achieved even while being immersed in the noisy intermediate-scale quantum era, characterized by the current devices' inability to process medium-scale complex problems efficiently. Consequently, there has been a surge of interest in quantum algorithms in various fields. Multiple factors have played a role in this extraordinary development, with three being particularly noteworthy: (i) the development of larger devices with enhanced interconnections between their constituent qubits, (ii) the development of specialized frameworks, and (iii) the existence of well-known or ready-to-use hybrid schemes that simplify the method development process. In this context, this manuscript presents and overviews some recent contributions within this paradigm, showcasing the potential of quantum computing to emerge as a significant research catalyst in the fields of machine learning and optimization in the coming years.
△ Less
Submitted 6 May, 2025;
originally announced May 2025.
-
Addressing the Minor-Embedding Problem in Quantum Annealing and Evaluating State-of-the-Art Algorithm Performance
Authors:
Aitor Gomez-Tejedor,
Eneko Osaba,
Esther Villar-Rodriguez
Abstract:
This study addresses the minor-embedding problem, which involves mapping the variables of an Ising model onto a quantum annealing processor. The primary motivation stems from the observed performance disparity of quantum annealers when solving problems suited to the processor's architecture versus those with non-hardware-native topologies. Our research has two main objectives: i) to analyze the im…
▽ More
This study addresses the minor-embedding problem, which involves mapping the variables of an Ising model onto a quantum annealing processor. The primary motivation stems from the observed performance disparity of quantum annealers when solving problems suited to the processor's architecture versus those with non-hardware-native topologies. Our research has two main objectives: i) to analyze the impact of embedding quality on the performance of D-Wave Systems quantum annealers, and ii) to evaluate the quality of the embeddings generated by Minorminer, an algorithm provided by D-Wave and widely recognized as the standard minor-embedding technique in the literature. Regarding the first objective, our experiments reveal a clear correlation between the average chain length of embeddings and the relative errors of the solutions sampled. This underscores the critical influence of embedding quality on quantum annealing performance. For the second objective, we focus on the Minorminer technique, assessing its capacity to embed problems, the quality of the embeddings produced, and the robustness of the results. We also compare its performance with Clique Embedding, another algorithm developed by D-Wave, which is deterministic and designed to embed fully connected Ising models into quantum annealing processors, serving as a worst-case scenario. The results demonstrate that there is significant room for improvement for Minorminer, as it has not consistently outperformed the worst-case scenario.
△ Less
Submitted 25 April, 2025; v1 submitted 17 April, 2025;
originally announced April 2025.
-
Optimizing Package Delivery with Quantum Annealers: Addressing Time-Windows and Simultaneous Pickup and Delivery
Authors:
Eneko Osaba,
Esther Villar-Rodriguez,
Pablo Miranda-Rodriguez,
Antón Asla
Abstract:
Recent research at the intersection of quantum computing and routing problems has been highly prolific. Much of this work focuses on classical problems such as the Traveling Salesman Problem and the Vehicle Routing Problem. The practical applicability of these problems depends on the specific objectives and constraints considered. However, it is undeniable that translating complex real-world requi…
▽ More
Recent research at the intersection of quantum computing and routing problems has been highly prolific. Much of this work focuses on classical problems such as the Traveling Salesman Problem and the Vehicle Routing Problem. The practical applicability of these problems depends on the specific objectives and constraints considered. However, it is undeniable that translating complex real-world requirements into these classical formulations often proves challenging. In this paper, we resort to our previously published quantum-classical technique for addressing real-world-oriented routing problems, known as Quantum for Real Package Delivery (Q4RPD), and elaborate on solving additional realistic problem instances. Accordingly, this paper emphasizes the following characteristics: i) simultaneous pickup and deliveries, ii) time-windows, and iii) mobility restrictions by vehicle type. To illustrate the application of Q4RPD, we have conducted an experimentation comprising seven instances, serving as a demonstration of the newly developed features.
△ Less
Submitted 2 April, 2025;
originally announced April 2025.
-
Transfer of Knowledge through Reverse Annealing: A Preliminary Analysis of the Benefits and What to Share
Authors:
Eneko Osaba,
Esther Villar-Rodriguez
Abstract:
Being immersed in the NISQ-era, current quantum annealers present limitations for solving optimization problems efficiently. To mitigate these limitations, D-Wave Systems developed a mechanism called Reverse Annealing, a specific type of quantum annealing designed to perform local refinement of good states found elsewhere. Despite the research activity around Reverse Annealing, none has theorized…
▽ More
Being immersed in the NISQ-era, current quantum annealers present limitations for solving optimization problems efficiently. To mitigate these limitations, D-Wave Systems developed a mechanism called Reverse Annealing, a specific type of quantum annealing designed to perform local refinement of good states found elsewhere. Despite the research activity around Reverse Annealing, none has theorized about the possible benefits related to the transfer of knowledge under this paradigm. This work moves in that direction and is driven by experimentation focused on answering two key research questions: i) is reverse annealing a paradigm that can benefit from knowledge transfer between similar problems? and ii) can we infer the characteristics that an input solution should meet to help increase the probability of success? To properly guide the tests in this paper, the well-known Knapsack Problem has been chosen for benchmarking purposes, using a total of 34 instances composed of 14 and 16 items.
△ Less
Submitted 11 April, 2025; v1 submitted 27 January, 2025;
originally announced January 2025.
-
On the Transfer of Knowledge in Quantum Algorithms
Authors:
Esther Villar-Rodriguez,
Eneko Osaba,
Izaskun Oregi,
Sebastián V. Romero,
Julián Ferreiro-Vélez
Abstract:
The field of quantum computing is generating significant anticipation within the scientific and industrial communities due to its potential to revolutionize computing paradigms. Recognizing this potential, this paper explores the integration of transfer of knowledge techniques, traditionally used in classical artificial intelligence, into quantum computing. We present a comprehensive classificatio…
▽ More
The field of quantum computing is generating significant anticipation within the scientific and industrial communities due to its potential to revolutionize computing paradigms. Recognizing this potential, this paper explores the integration of transfer of knowledge techniques, traditionally used in classical artificial intelligence, into quantum computing. We present a comprehensive classification of the transfer models, focusing on Transfer Learning and Transfer Optimization. Additionally, we analyze relevant schemes in quantum computing that can benefit from knowledge sharing, and we delve into the potential synergies, supported by theoretical insights and initial experimental results. Our findings suggest that leveraging the transfer of knowledge can enhance the efficiency and effectiveness of quantum algorithms, particularly in the context of hybrid solvers. This approach not only accelerates the optimization process but also reduces the computational burden on quantum processors, making it a valuable tool for advancing quantum computing technologies.
△ Less
Submitted 23 January, 2025;
originally announced January 2025.
-
A Tutorial on the Design, Experimentation and Application of Metaheuristic Algorithms to Real-World Optimization Problems
Authors:
Eneko Osaba,
Esther Villar-Rodriguez,
Javier Del Ser,
Antonio J. Nebro,
Daniel Molina,
Antonio LaTorre,
Ponnuthurai N. Suganthan,
Carlos A. Coello Coello,
Francisco Herrera
Abstract:
In the last few years, the formulation of real-world optimization problems and their efficient solution via metaheuristic algorithms has been a catalyst for a myriad of research studies. In spite of decades of historical advancements on the design and use of metaheuristics, large difficulties still remain in regards to the understandability, algorithmic design uprightness, and performance verifiab…
▽ More
In the last few years, the formulation of real-world optimization problems and their efficient solution via metaheuristic algorithms has been a catalyst for a myriad of research studies. In spite of decades of historical advancements on the design and use of metaheuristics, large difficulties still remain in regards to the understandability, algorithmic design uprightness, and performance verifiability of new technical achievements. A clear example stems from the scarce replicability of works dealing with metaheuristics used for optimization, which is often infeasible due to ambiguity and lack of detail in the presentation of the methods to be reproduced. Additionally, in many cases, there is a questionable statistical significance of their reported results. This work aims at providing the audience with a proposal of good practices which should be embraced when conducting studies about metaheuristics methods used for optimization in order to provide scientific rigor, value and transparency. To this end, we introduce a step by step methodology covering every research phase that should be followed when addressing this scientific field. Specifically, frequently overlooked yet crucial aspects and useful recommendations will be discussed in regards to the formulation of the problem, solution encoding, implementation of search operators, evaluation metrics, design of experiments, and considerations for real-world performance, among others. Finally, we will outline important considerations, challenges, and research directions for the success of newly developed optimization metaheuristics in their deployment and operation over real-world application environments.
△ Less
Submitted 4 October, 2024;
originally announced October 2024.
-
Exploring Utility in a Real-World Warehouse Optimization Problem: Formulation Based on Quantum Annealers and Preliminary Results
Authors:
Eneko Osaba,
Esther Villar-Rodriguez,
Antón Asla
Abstract:
In the current NISQ-era, one of the major challenges faced by researchers and practitioners lies in figuring out how to combine quantum and classical computing in the most efficient and innovative way. In this paper, we present a mechanism coined as Quantum Initialization for Warehouse Optimization Problem that resorts to D-Wave's Quantum Annealer. The module has been specifically designed to be e…
▽ More
In the current NISQ-era, one of the major challenges faced by researchers and practitioners lies in figuring out how to combine quantum and classical computing in the most efficient and innovative way. In this paper, we present a mechanism coined as Quantum Initialization for Warehouse Optimization Problem that resorts to D-Wave's Quantum Annealer. The module has been specifically designed to be embedded into already existing classical software dedicated to the optimization of a real-world industrial problem. We preliminary tested the implemented mechanism through a two-phase experiment against the classical version of the software.
△ Less
Submitted 1 October, 2024; v1 submitted 15 September, 2024;
originally announced September 2024.
-
QOPTLib: a Quantum Computing Oriented Benchmark for Combinatorial Optimization Problems
Authors:
Eneko Osaba,
Esther Villar-Rodriguez
Abstract:
In this paper, we propose a quantum computing oriented benchmark for combinatorial optimization. This benchmark, coined as QOPTLib, is composed of 40 instances equally distributed over four well-known problems: Traveling Salesman Problem, Vehicle Routing Problem, one-dimensional Bin Packing Problem and the Maximum Cut Problem. The sizes of the instances in QOPTLib not only correspond to computatio…
▽ More
In this paper, we propose a quantum computing oriented benchmark for combinatorial optimization. This benchmark, coined as QOPTLib, is composed of 40 instances equally distributed over four well-known problems: Traveling Salesman Problem, Vehicle Routing Problem, one-dimensional Bin Packing Problem and the Maximum Cut Problem. The sizes of the instances in QOPTLib not only correspond to computationally addressable sizes, but also to the maximum length approachable with non-zero likelihood of getting a good result. In this regard, it is important to highlight that hybrid approaches are also taken into consideration. Thus, this benchmark constitutes the first effort to provide users a general-purpose dataset. Also in this paper, we introduce a first full solving of QOPTLib using two solvers based on quantum annealing. Our main intention with this is to establish a preliminary baseline, hoping to inspire other researchers to beat these outcomes with newly proposed quantum-based algorithms.
△ Less
Submitted 24 April, 2024;
originally announced April 2024.
-
Solving a Real-World Package Delivery Routing Problem Using Quantum Annealers
Authors:
Eneko Osaba,
Esther Villar-Rodriguez,
Antón Asla
Abstract:
Research focused on the conjunction between quantum computing and routing problems has been very prolific in recent years. Most of the works revolve around classical problems such as the Traveling Salesman Problem or the Vehicle Routing Problem. The real-world applicability of these problems is dependent on the objectives and constraints considered. Anyway, it is undeniable that it is often diffic…
▽ More
Research focused on the conjunction between quantum computing and routing problems has been very prolific in recent years. Most of the works revolve around classical problems such as the Traveling Salesman Problem or the Vehicle Routing Problem. The real-world applicability of these problems is dependent on the objectives and constraints considered. Anyway, it is undeniable that it is often difficult to translate complex requirements into these classical formulations.The main objective of this research is to present a solving scheme for dealing with realistic instances while maintaining all the characteristics and restrictions of the original real-world problem. Thus, a quantum-classical strategy has been developed, coined Q4RPD, that considers a set of real constraints such as a heterogeneous fleet of vehicles, priority deliveries, and capacities characterized by two values: weight and dimensions of the packages. Q4RPD resorts to the Leap Constrained Quadratic Model Hybrid Solver of D-Wave. To demonstrate the application of Q4RPD, an experimentation composed of six different instances has been conducted, aiming to serve as illustrative examples.
△ Less
Submitted 28 June, 2024; v1 submitted 22 March, 2024;
originally announced March 2024.
-
Hybrid Quantum Solvers in Production: how to succeed in the NISQ era?
Authors:
Eneko Osaba,
Esther Villar-Rodriguez,
Aitor Gomez-Tejedor,
Izaskun Oregi
Abstract:
Hybrid quantum computing is considered the present and the future within the field of quantum computing. Far from being a passing fad, this trend cannot be considered just a stopgap to address the limitations of NISQ-era devices. The foundations linking both computing paradigms will remain robust over time. The contribution of this work is twofold: first, we describe and categorize some of the mos…
▽ More
Hybrid quantum computing is considered the present and the future within the field of quantum computing. Far from being a passing fad, this trend cannot be considered just a stopgap to address the limitations of NISQ-era devices. The foundations linking both computing paradigms will remain robust over time. The contribution of this work is twofold: first, we describe and categorize some of the most frequently used hybrid solvers, resorting to two different taxonomies recently published in the literature. Secondly, we put a special focus on two solvers that are currently deployed in real production and that have demonstrated to be near the real industry. These solvers are the LeapHybridBQMSampler contained in D-Wave's Hybrid Solver Service and Quantagonia's Hybrid Solver. We analyze the performance of both methods using as benchmarks four combinatorial optimization problems.
△ Less
Submitted 12 September, 2024; v1 submitted 18 January, 2024;
originally announced January 2024.
-
A Quantum Computing-based System for Portfolio Optimization using Future Asset Values and Automatic Reduction of the Investment Universe
Authors:
Eneko Osaba,
Guillaume Gelabert,
Esther Villar-Rodriguez,
Antón Asla,
Izaskun Oregi
Abstract:
One of the problems in quantitative finance that has received the most attention is the portfolio optimization problem. Regarding its solving, this problem has been approached using different techniques, with those related to quantum computing being especially prolific in recent years. In this study, we present a system called Quantum Computing-based System for Portfolio Optimization with Future A…
▽ More
One of the problems in quantitative finance that has received the most attention is the portfolio optimization problem. Regarding its solving, this problem has been approached using different techniques, with those related to quantum computing being especially prolific in recent years. In this study, we present a system called Quantum Computing-based System for Portfolio Optimization with Future Asset Values and Automatic Universe Reduction (Q4FuturePOP), which deals with the Portfolio Optimization Problem considering the following innovations: i) the developed tool is modeled for working with future prediction of assets, instead of historical values; and ii) Q4FuturePOP includes an automatic universe reduction module, which is conceived to intelligently reduce the complexity of the problem. We also introduce a brief discussion about the preliminary performance of the different modules that compose the prototypical version of Q4FuturePOP.
△ Less
Submitted 27 September, 2023; v1 submitted 22 September, 2023;
originally announced September 2023.
-
Hybrid classical-quantum computing: are we forgetting the classical part in the binomial?
Authors:
Esther Villar-Rodriguez,
Aitor Gomez-Tejedor,
Eneko Osaba
Abstract:
The expectations arising from the latest achievements in the quantum computing field are causing that researchers coming from classical artificial intelligence to be fascinated by this new paradigm. In turn, quantum computing, on the road towards usability, needs classical procedures. Hybridization is, in these circumstances, an indispensable step but can also be seen as a promising new avenue to…
▽ More
The expectations arising from the latest achievements in the quantum computing field are causing that researchers coming from classical artificial intelligence to be fascinated by this new paradigm. In turn, quantum computing, on the road towards usability, needs classical procedures. Hybridization is, in these circumstances, an indispensable step but can also be seen as a promising new avenue to get the most from both computational worlds. Nonetheless, hybrid approaches have now and will have in the future many challenges to face, which, if ignored, will threaten the viability or attractiveness of quantum computing for real-world applications. To identify them and pose pertinent questions, a proper characterization of the hybrid quantum computing field, and especially hybrid solvers, is compulsory. With this motivation in mind, the main purpose of this work is to propose a preliminary taxonomy for classifying hybrid schemes, and bring to the fore some questions to stir up researchers minds about the real challenges regarding the application of quantum computing.
△ Less
Submitted 21 August, 2023;
originally announced August 2023.
-
Solving Logistic-Oriented Bin Packing Problems Through a Hybrid Quantum-Classical Approach
Authors:
Sebastián V. Romero,
Eneko Osaba,
Esther Villar-Rodriguez,
Antón Asla
Abstract:
The Bin Packing Problem is a classic problem with wide industrial applicability. In fact, the efficient packing of items into bins is one of the toughest challenges in many logistic corporations and is a critical issue for reducing storage costs or improving vehicle space allocation. In this work, we resort to our previously published quantum-classical framework known as Q4RealBPP, and elaborate o…
▽ More
The Bin Packing Problem is a classic problem with wide industrial applicability. In fact, the efficient packing of items into bins is one of the toughest challenges in many logistic corporations and is a critical issue for reducing storage costs or improving vehicle space allocation. In this work, we resort to our previously published quantum-classical framework known as Q4RealBPP, and elaborate on the solving of real-world oriented instances of the Bin Packing Problem. With this purpose, this paper gravitates on the following characteristics: i) the existence of heterogeneous bins, ii) the extension of the framework to solve not only three-dimensional, but also one- and two-dimensional instances of the problem, iii) requirements for item-bin associations, and iv) delivery priorities. All these features have been tested in this paper, as well as the ability of Q4RealBPP to solve real-world oriented instances.
△ Less
Submitted 21 August, 2023; v1 submitted 5 August, 2023;
originally announced August 2023.
-
Benchmark dataset and instance generator for Real-World Three-Dimensional Bin Packing Problems
Authors:
Eneko Osaba,
Esther Villar-Rodriguez,
Sebastián V. Romero
Abstract:
In this article, a benchmark for real-world bin packing problems is proposed. This dataset consists of 12 instances of varying levels of complexity regarding size (with the number of packages ranging from 38 to 53) and user-defined requirements. In fact, several real-world-oriented restrictions were taken into account to build these instances: i) item and bin dimensions, ii) weight restrictions, i…
▽ More
In this article, a benchmark for real-world bin packing problems is proposed. This dataset consists of 12 instances of varying levels of complexity regarding size (with the number of packages ranging from 38 to 53) and user-defined requirements. In fact, several real-world-oriented restrictions were taken into account to build these instances: i) item and bin dimensions, ii) weight restrictions, iii) affinities among package categories iv) preferences for package ordering and v) load balancing. Besides the data, we also offer an own developed Python script for the dataset generation, coined Q4RealBPP-DataGen. The benchmark was initially proposed to evaluate the performance of quantum solvers. Therefore, the characteristics of this set of instances were designed according to the current limitations of quantum devices. Additionally, the dataset generator is included to allow the construction of general-purpose benchmarks. The data introduced in this article provides a baseline that will encourage quantum computing researchers to work on real-world bin packing problems.
△ Less
Submitted 29 June, 2023; v1 submitted 28 April, 2023;
originally announced April 2023.
-
Hybrid Approach for Solving Real-World Bin Packing Problem Instances Using Quantum Annealers
Authors:
Sebastián V. Romero,
Eneko Osaba,
Esther Villar-Rodriguez,
Izaskun Oregi,
Yue Ban
Abstract:
Efficient packing of items into bins is a common daily task. Known as Bin Packing Problem, it has been intensively studied in the field of artificial intelligence, thanks to the wide interest from industry and logistics. Since decades, many variants have been proposed, with the three-dimensional Bin Packing Problem as the closest one to real-world use cases. We introduce a hybrid quantum-classical…
▽ More
Efficient packing of items into bins is a common daily task. Known as Bin Packing Problem, it has been intensively studied in the field of artificial intelligence, thanks to the wide interest from industry and logistics. Since decades, many variants have been proposed, with the three-dimensional Bin Packing Problem as the closest one to real-world use cases. We introduce a hybrid quantum-classical framework for solving real-world three-dimensional Bin Packing Problems (Q4RealBPP), considering different realistic characteristics, such as: i) package and bin dimensions, ii) overweight restrictions, iii) affinities among item categories and iv) preferences for item ordering. Q4RealBPP permits the solving of real-world oriented instances of 3dBPP, contemplating restrictions well appreciated by industrial and logistics sectors.
△ Less
Submitted 25 May, 2023; v1 submitted 1 March, 2023;
originally announced March 2023.
-
Towards Improving Exploration in Self-Imitation Learning using Intrinsic Motivation
Authors:
Alain Andres,
Esther Villar-Rodriguez,
Javier Del Ser
Abstract:
Reinforcement Learning has emerged as a strong alternative to solve optimization tasks efficiently. The use of these algorithms highly depends on the feedback signals provided by the environment in charge of informing about how good (or bad) the decisions made by the learned agent are. Unfortunately, in a broad range of problems the design of a good reward function is not trivial, so in such cases…
▽ More
Reinforcement Learning has emerged as a strong alternative to solve optimization tasks efficiently. The use of these algorithms highly depends on the feedback signals provided by the environment in charge of informing about how good (or bad) the decisions made by the learned agent are. Unfortunately, in a broad range of problems the design of a good reward function is not trivial, so in such cases sparse reward signals are instead adopted. The lack of a dense reward function poses new challenges, mostly related to exploration. Imitation Learning has addressed those problems by leveraging demonstrations from experts. In the absence of an expert (and its subsequent demonstrations), an option is to prioritize well-suited exploration experiences collected by the agent in order to bootstrap its learning process with good exploration behaviors. However, this solution highly depends on the ability of the agent to discover such trajectories in the early stages of its learning process. To tackle this issue, we propose to combine imitation learning with intrinsic motivation, two of the most widely adopted techniques to address problems with sparse reward. In this work intrinsic motivation is used to encourage the agent to explore the environment based on its curiosity, whereas imitation learning allows repeating the most promising experiences to accelerate the learning process. This combination is shown to yield an improved performance and better generalization in procedurally-generated environments, outperforming previously reported self-imitation learning methods and achieving equal or better sample efficiency with respect to intrinsic motivation in isolation.
△ Less
Submitted 30 November, 2022;
originally announced November 2022.
-
Analyzing the behaviour of D'WAVE quantum annealer: fine-tuning parameterization and tests with restrictive Hamiltonian formulations
Authors:
Esther Villar-Rodriguez,
Eneko Osaba,
Izaskun Oregi
Abstract:
Despite being considered as the next frontier in computation, Quantum Computing is still in an early stage of development. Indeed, current commercial quantum computers suffer from some critical restraints, such as noisy processes and a limited amount of qubits, among others, that affect the performance of quantum algorithms. Despite these limitations, researchers have devoted much effort to propos…
▽ More
Despite being considered as the next frontier in computation, Quantum Computing is still in an early stage of development. Indeed, current commercial quantum computers suffer from some critical restraints, such as noisy processes and a limited amount of qubits, among others, that affect the performance of quantum algorithms. Despite these limitations, researchers have devoted much effort to propose different frameworks for efficiently using these Noisy Intermediate-Scale Quantum (NISQ) devices. One of these procedures is D'WAVE Systems' quantum-annealer, which can be use to solve optimization problems by translating them into an energy minimization problem. In this context, this work is focused on providing useful insights and information into the behaviour of the quantum-annealer when addressing real-world combinatorial optimization problems. Our main motivation with this study is to open some quantum computing frontiers to non-expert stakeholders. To this end, we perform an extensive experimentation, in the form of a parameter sensitive analysis. This experimentation has been conducted using the Traveling Salesman Problem as benchmarking problem, and adopting two QUBOs: state-of-the-art and a heuristically generated. Our analysis has been performed on a single 7-noded instance, and it is based on more than 200 different parameter configurations, comprising more than 3700 unitary runs and 7 million of quantum reads. Thanks to this study, findings related to the energy distribution and most appropriate parameter settings have been obtained. Finally, an additional study has been performed, aiming to determine the efficiency of the heuristically built QUBO in further TSP instances.
△ Less
Submitted 1 July, 2022;
originally announced July 2022.
-
An Evaluation Study of Intrinsic Motivation Techniques applied to Reinforcement Learning over Hard Exploration Environments
Authors:
Alain Andres,
Esther Villar-Rodriguez,
Javier Del Ser
Abstract:
In the last few years, the research activity around reinforcement learning tasks formulated over environments with sparse rewards has been especially notable. Among the numerous approaches proposed to deal with these hard exploration problems, intrinsic motivation mechanisms are arguably among the most studied alternatives to date. Advances reported in this area over time have tackled the explorat…
▽ More
In the last few years, the research activity around reinforcement learning tasks formulated over environments with sparse rewards has been especially notable. Among the numerous approaches proposed to deal with these hard exploration problems, intrinsic motivation mechanisms are arguably among the most studied alternatives to date. Advances reported in this area over time have tackled the exploration issue by proposing new algorithmic ideas to generate alternative mechanisms to measure the novelty. However, most efforts in this direction have overlooked the influence of different design choices and parameter settings that have also been introduced to improve the effect of the generated intrinsic bonus, forgetting the application of those choices to other intrinsic motivation techniques that may also benefit of them. Furthermore, some of those intrinsic methods are applied with different base reinforcement algorithms (e.g. PPO, IMPALA) and neural network architectures, being hard to fairly compare the provided results and the actual progress provided by each solution. The goal of this work is to stress on this crucial matter in reinforcement learning over hard exploration environments, exposing the variability and susceptibility of avant-garde intrinsic motivation techniques to diverse design factors. Ultimately, our experiments herein reported underscore the importance of a careful selection of these design aspects coupled with the exploration requirements of the environment and the task in question under the same setup, so that fair comparisons can be guaranteed.
△ Less
Submitted 21 November, 2022; v1 submitted 23 May, 2022;
originally announced May 2022.
-
Collaborative Training of Heterogeneous Reinforcement Learning Agents in Environments with Sparse Rewards: What and When to Share?
Authors:
Alain Andres,
Esther Villar-Rodriguez,
Javier Del Ser
Abstract:
In the early stages of human life, babies develop their skills by exploring different scenarios motivated by their inherent satisfaction rather than by extrinsic rewards from the environment. This behavior, referred to as intrinsic motivation, has emerged as one solution to address the exploration challenge derived from reinforcement learning environments with sparse rewards. Diverse exploration a…
▽ More
In the early stages of human life, babies develop their skills by exploring different scenarios motivated by their inherent satisfaction rather than by extrinsic rewards from the environment. This behavior, referred to as intrinsic motivation, has emerged as one solution to address the exploration challenge derived from reinforcement learning environments with sparse rewards. Diverse exploration approaches have been proposed to accelerate the learning process over single- and multi-agent problems with homogeneous agents. However, scarce studies have elaborated on collaborative learning frameworks between heterogeneous agents deployed into the same environment, but interacting with different instances of the latter without any prior knowledge. Beyond the heterogeneity, each agent's characteristics grant access only to a subset of the full state space, which may hide different exploration strategies and optimal solutions. In this work we combine ideas from intrinsic motivation and transfer learning. Specifically, we focus on sharing parameters in actor-critic model architectures and on combining information obtained through intrinsic motivation with the aim of having a more efficient exploration and faster learning. We test our strategies through experiments performed over a modified ViZDooM's My Way Home scenario, which is more challenging than its original version and allows evaluating the heterogeneity between agents. Our results reveal different ways in which a collaborative framework with little additional computational cost can outperform an independent learning process without knowledge sharing. Additionally, we depict the need for modulating correctly the importance between the extrinsic and intrinsic rewards to avoid undesired agent behaviors.
△ Less
Submitted 24 February, 2022;
originally announced February 2022.
-
Focusing on the Hybrid Quantum Computing -- Tabu Search Algorithm: new results on the Asymmetric Salesman Problem
Authors:
Eneko Osaba,
Esther Villar-Rodriguez,
Izaskun Oregi,
Aitor Moreno-Fernandez-de-Leceta
Abstract:
Quantum Computing is an emerging paradigm which is gathering a lot of popularity in the current scientific and technological community. Widely conceived as the next frontier of computation, Quantum Computing is still at the dawn of its development. Thus, current solving systems suffer from significant limitations in terms of performance and capabilities. Some interesting approaches have been devis…
▽ More
Quantum Computing is an emerging paradigm which is gathering a lot of popularity in the current scientific and technological community. Widely conceived as the next frontier of computation, Quantum Computing is still at the dawn of its development. Thus, current solving systems suffer from significant limitations in terms of performance and capabilities. Some interesting approaches have been devised by researchers and practitioners in order to overcome these barriers, being quantum-classical hybrid algorithms one of the most often used solving schemes. The main goal of this paper is to extend the results and findings of the recently proposed hybrid Quantum Computing - Tabu Search Algorithm for partitioning problems. To do that, we focus our research on the adaptation of this method to the Asymmetric Traveling Salesman Problem. In overall, we have employed six well-known instances belonging to TSPLIB to assess the performance of Quantum Computing - Tabu Search Algorithm in comparison to QBSolv, a state-of-the-art decomposing solver. Furthermore, as an additional contribution, this work also supposes the first solving of the Asymmetric Traveling Salesman Problem using a Quantum Computing based method. Aiming to boost whole community's research in QC, we have released the project's repository as open source code for further application and improvements.
△ Less
Submitted 3 May, 2021; v1 submitted 11 February, 2021;
originally announced February 2021.
-
Hybrid Quantum Computing -- Tabu Search Algorithm for Partitioning Problems: preliminary study on the Traveling Salesman Problem
Authors:
Eneko Osaba,
Esther Villar-Rodriguez,
Izaskun Oregi,
Aitor Moreno-Fernandez-de-Leceta
Abstract:
Quantum Computing is considered as the next frontier in computing, and it is attracting a lot of attention from the current scientific community. This kind of computation provides to researchers with a revolutionary paradigm for addressing complex optimization problems, offering a significant speed advantage and an efficient search ability. Anyway, Quantum Computing is still in an incipient stage…
▽ More
Quantum Computing is considered as the next frontier in computing, and it is attracting a lot of attention from the current scientific community. This kind of computation provides to researchers with a revolutionary paradigm for addressing complex optimization problems, offering a significant speed advantage and an efficient search ability. Anyway, Quantum Computing is still in an incipient stage of development. For this reason, present architectures show certain limitations, which have motivated the carrying out of this paper. In this paper, we introduce a novel solving scheme coined as hybrid Quantum Computing - Tabu Search Algorithm. Main pillars of operation of the proposed method are a greater control over the access to quantum resources, and a considerable reduction of non-profitable accesses. To assess the quality of our method, we have used 7 different Traveling Salesman Problem instances as benchmarking set. The obtained outcomes support the preliminary conclusion that our algorithm is an approach which offers promising results for solving partitioning problems while it drastically reduces the access to quantum computing resources. We also contribute to the field of Transfer Optimization by developing an evolutionary multiform multitasking algorithm as initialization method.
△ Less
Submitted 12 April, 2021; v1 submitted 9 December, 2020;
originally announced December 2020.
-
A Coevolutionary Variable Neighborhood Search Algorithm for Discrete Multitasking (CoVNS): Application to Community Detection over Graphs
Authors:
Eneko Osaba,
Esther Villar-Rodriguez,
Javier Del Ser
Abstract:
The main goal of the multitasking optimization paradigm is to solve multiple and concurrent optimization tasks in a simultaneous way through a single search process. For attaining promising results, potential complementarities and synergies between tasks are properly exploited, helping each other by virtue of the exchange of genetic material. This paper is focused on Evolutionary Multitasking, whi…
▽ More
The main goal of the multitasking optimization paradigm is to solve multiple and concurrent optimization tasks in a simultaneous way through a single search process. For attaining promising results, potential complementarities and synergies between tasks are properly exploited, helping each other by virtue of the exchange of genetic material. This paper is focused on Evolutionary Multitasking, which is a perspective for dealing with multitasking optimization scenarios by embracing concepts from Evolutionary Computation. This work contributes to this field by presenting a new multitasking approach named as Coevolutionary Variable Neighborhood Search Algorithm, which finds its inspiration on both the Variable Neighborhood Search metaheuristic and coevolutionary strategies. The second contribution of this paper is the application field, which is the optimal partitioning of graph instances whose connections among nodes are directed and weighted. This paper pioneers on the simultaneous solving of this kind of tasks. Two different multitasking scenarios are considered, each comprising 11 graph instances. Results obtained by our method are compared to those issued by a parallel Variable Neighborhood Search and independent executions of the basic Variable Neighborhood Search. The discussion on such results support our hypothesis that the proposed method is a promising scheme for simultaneous solving community detection problems over graphs.
△ Less
Submitted 30 September, 2020;
originally announced September 2020.
-
Lights and Shadows in Evolutionary Deep Learning: Taxonomy, Critical Methodological Analysis, Cases of Study, Learned Lessons, Recommendations and Challenges
Authors:
Aritz D. Martinez,
Javier Del Ser,
Esther Villar-Rodriguez,
Eneko Osaba,
Javier Poyatos,
Siham Tabik,
Daniel Molina,
Francisco Herrera
Abstract:
Much has been said about the fusion of bio-inspired optimization algorithms and Deep Learning models for several purposes: from the discovery of network topologies and hyper-parametric configurations with improved performance for a given task, to the optimization of the model's parameters as a replacement for gradient-based solvers. Indeed, the literature is rich in proposals showcasing the applic…
▽ More
Much has been said about the fusion of bio-inspired optimization algorithms and Deep Learning models for several purposes: from the discovery of network topologies and hyper-parametric configurations with improved performance for a given task, to the optimization of the model's parameters as a replacement for gradient-based solvers. Indeed, the literature is rich in proposals showcasing the application of assorted nature-inspired approaches for these tasks. In this work we comprehensively review and critically examine contributions made so far based on three axes, each addressing a fundamental question in this research avenue: a) optimization and taxonomy (Why?), including a historical perspective, definitions of optimization problems in Deep Learning, and a taxonomy associated with an in-depth analysis of the literature, b) critical methodological analysis (How?), which together with two case studies, allows us to address learned lessons and recommendations for good practices following the analysis of the literature, and c) challenges and new directions of research (What can be done, and what for?). In summary, three axes - optimization and taxonomy, critical analysis, and challenges - which outline a complete vision of a merger of two technologies drawing up an exciting future for this area of fusion research.
△ Less
Submitted 8 August, 2020;
originally announced August 2020.
