-
Bilevel optimization with sustainability perspective: a survey on applications
Authors:
Giulia Caselli,
Manuel Iori,
Ivana Ljubić
Abstract:
Bilevel optimization, a well-established field for modeling hierarchical decision-making problems, has recently intersected with sustainability studies and practices, resulting in a series of works focusing on bilevel optimization problems involving multiple decision makers with diverse economic, environmental, and social objectives. This survey offers a comprehensive overview of sustainable bilev…
▽ More
Bilevel optimization, a well-established field for modeling hierarchical decision-making problems, has recently intersected with sustainability studies and practices, resulting in a series of works focusing on bilevel optimization problems involving multiple decision makers with diverse economic, environmental, and social objectives. This survey offers a comprehensive overview of sustainable bilevel optimization applications. First, we introduce the main concepts related to the nature of bilevel optimization problems and present some typical mathematical formulations for bilevel pricing problems that cover many of the collected applications. Then, we review the most relevant works published in sustainable bilevel optimization, giving a classification based on the application domains and their association with well-known operations research problems, while briefly discussing the proposed solution methodologies. We survey applications on transportation and logistics, production planning and manufacturing, water, waste, and agriculture management, supply chains, and disaster prevention and response. Finally, we outline a list of open questions and opportunities for future research in this domain.
△ Less
Submitted 22 April, 2025; v1 submitted 11 June, 2024;
originally announced June 2024.
-
The Multi-Commodity Flow Problem with Outsourcing Decisions
Authors:
Elena Fernandez,
Ivana Ljubic,
Nicolas Zerega
Abstract:
We address a new prize-collecting problem of routing commodities in a given network with hub and non-hub nodes, in which the service of the non-hub nodes will be outsourced to third-party carriers. The problem is modeled as a Stackelberg game: there is a major firm (leader) that decides to serve a subset of commodities. The leader aims to outsource first and third legs of transportation services t…
▽ More
We address a new prize-collecting problem of routing commodities in a given network with hub and non-hub nodes, in which the service of the non-hub nodes will be outsourced to third-party carriers. The problem is modeled as a Stackelberg game: there is a major firm (leader) that decides to serve a subset of commodities. The leader aims to outsource first and third legs of transportation services to smaller carriers (who act as followers) by allocating at most one carrier to each non-hub node. The carriers try to maximize their own profits, which are influenced by the leader's offers. The goal of the leader is to determine the optimal outsourcing fees, along with the allocation of carriers to the non-hub nodes, so that the profit from the routed commodities is maximized. The optimal response of the followers must be taken into account, as the followers might refuse to serve some legs in case they are negative or do not maximize their profit. We also study two alternative settings: one in which the outsourcing fees are fixed, and the other one in which the carriers accept any offer, as long as the resulting profit is non-negative. We prove that the set of possible outsourcing fees can be discretized and formulate the problem as a single-level mixed-integer nonlinear program. For all considered problem variants, we prove NP-hardness and propose and computationally investigate several MIP formulations. We study the computational scalability of these MIP formulations and analyze solutions obtained by varying the reservation prices of the carriers. Finally, by comparing the introduced problem variants, we derive some interesting managerial insights.
△ Less
Submitted 4 June, 2024;
originally announced June 2024.
-
Emerging Optimization Problems for Distribution in Same-day Delivery
Authors:
Yuanyuan Li,
Claudia Archetti,
Ivana Ljubic
Abstract:
Same-day deliveries (SDD) have become a new standard to satisfy the "instant gratification" of online customers. Despite the existing powerful technologies deployed in last-mile delivery, SDD services face new decision-making challenges related to the trade-off between delivery cost and time. In addition, new challenges related to environmental issues, customer satisfaction, or fairness arise. Res…
▽ More
Same-day deliveries (SDD) have become a new standard to satisfy the "instant gratification" of online customers. Despite the existing powerful technologies deployed in last-mile delivery, SDD services face new decision-making challenges related to the trade-off between delivery cost and time. In addition, new challenges related to environmental issues, customer satisfaction, or fairness arise. Researchers have explored various approaches to face these challenges in the context of SDD, where stochastic and dynamic data uncertainty plays a fundamental role. In this paper, we carefully review the emerging routing problems and solutions proposed in the existing literature for SDD services. We survey papers related to how to deal with dynamic arrival times of orders, how to allocate time slots to deliveries, how to select the right delivery options, how to design pickup and delivery routes, or how to partition the delivery areas and decide the composition of the fleet. We also formulate and compare models for representative problems elaborating on the pros and cons that might guide practitioners in choosing the most appropriate objectives and constraints. Finally, we sketch challenges and identify future research directions.
△ Less
Submitted 1 October, 2024; v1 submitted 9 May, 2024;
originally announced May 2024.
-
Three Network Design Problems for Community Energy Storage
Authors:
Bissan Ghaddar,
Ivana Ljubic,
Yuying Qiu
Abstract:
In this paper, we develop novel mathematical models to optimize utilization of community energy storage (CES) by clustering prosumers and consumers into energy sharing communities/microgrids in the context of a smart city. Three different microgrid configurations are modeled using a unifying mixed-integer linear programming formulation. These configurations represent three different business model…
▽ More
In this paper, we develop novel mathematical models to optimize utilization of community energy storage (CES) by clustering prosumers and consumers into energy sharing communities/microgrids in the context of a smart city. Three different microgrid configurations are modeled using a unifying mixed-integer linear programming formulation. These configurations represent three different business models, namely: the island model, the interconnected model, and the Energy Service Companies model. The proposed mathematical formulations determine the optimal households' aggregation as well as the location and sizing of CES. To overcome the computational challenges of treating operational decisions within a multi-period decision making framework, we also propose a decomposition approach to accelerate the computational time needed to solve larger instances. We conduct a case study based on real power consumption, power generation, and location network data from Cambridge, MA. Our mathematical models and the underlying algorithmic framework can be used in operational and strategic planning studies on smart grids to incentivize the communitarian distributed renewable energy generation and to improve the self-consumption and self-sufficiency of the energy sharing community. The models are also targeted to policymakers of smart cities, utility companies, and Energy Service Companies as the proposed models support decision making on renewable energy related projects investments.
△ Less
Submitted 25 January, 2024;
originally announced January 2024.
-
Benders decomposition for congested partial set covering location with uncertain demand
Authors:
Alice Calamita,
Ivana Ljubić,
Laura Palagi
Abstract:
In this paper, we introduce a mixed integer quadratic formulation for the congested variant of the partial set covering location problem, which involves determining a subset of facility locations to open and efficiently allocating customers to these facilities to minimize the combined costs of facility opening and congestion while ensuring target coverage. To enhance the resilience of the solution…
▽ More
In this paper, we introduce a mixed integer quadratic formulation for the congested variant of the partial set covering location problem, which involves determining a subset of facility locations to open and efficiently allocating customers to these facilities to minimize the combined costs of facility opening and congestion while ensuring target coverage. To enhance the resilience of the solution against demand fluctuations, we address the case under uncertain customer demand using $Γ$-robustness. We formulate the deterministic problem and its robust counterpart as mixed-integer quadratic problems. We investigate the effect of the protection level in adapted instances from the literature to provide critical insights into how sensitive the planning is to the protection level. Moreover, since the size of the robust counterpart grows with the number of customers, which could be significant in real-world contexts, we propose the use of Benders decomposition to effectively reduce the number of variables by projecting out of the master problem all the variables dependent on the number of customers. We illustrate how to incorporate our Benders approach within a mixed-integer second-order cone programming (MISOCP) solver, addressing explicitly all the ingredients that are instrumental for its success. We discuss single-tree and multi-tree approaches and introduce a perturbation technique to deal with the degeneracy of the Benders subproblem efficiently. Our tailored Benders approaches outperform the perspective reformulation solved using the state-of-the-art MISOCP solver Gurobi on adapted instances from the literature.
△ Less
Submitted 23 January, 2024;
originally announced January 2024.
-
A bilevel approach for compensation and routing decisions in last-mile delivery
Authors:
Martina Cerulli,
Claudia Archetti,
Elena Fernandez,
Ivana Ljubic
Abstract:
In last-mile delivery logistics, peer-to-peer logistic platforms play an important role in connecting senders, customers, and independent carriers to fulfill delivery requests. Since the carriers are not under the platform's control, the platform has to anticipate their reactions, while deciding how to allocate the delivery operations. Indeed, carriers' decisions largely affect the platform's reve…
▽ More
In last-mile delivery logistics, peer-to-peer logistic platforms play an important role in connecting senders, customers, and independent carriers to fulfill delivery requests. Since the carriers are not under the platform's control, the platform has to anticipate their reactions, while deciding how to allocate the delivery operations. Indeed, carriers' decisions largely affect the platform's revenue. In this paper, we model this problem using bilevel programming. At the upper level, the platform decides how to assign the orders to the carriers; at the lower level, each carrier solves a profitable tour problem to determine which offered requests to accept, based on her own profit maximization. Possibly, the platform can influence carriers' decisions by determining also the compensation paid for each accepted request. The two considered settings result in two different formulations: the bilevel profitable tour problem with fixed compensation margins and with margin decisions, respectively. For each of them, we propose single-level reformulations and alternative formulations where the lower-level routing variables are projected out. A branch-and-cut algorithm is proposed to solve the bilevel models, with a tailored warm-start heuristic used to speed up the solution process. Extensive computational tests are performed to compare the proposed formulations and analyze solution characteristics.
△ Less
Submitted 10 June, 2025; v1 submitted 18 April, 2023;
originally announced April 2023.
-
Operational Research: Methods and Applications
Authors:
Fotios Petropoulos,
Gilbert Laporte,
Emel Aktas,
Sibel A. Alumur,
Claudia Archetti,
Hayriye Ayhan,
Maria Battarra,
Julia A. Bennell,
Jean-Marie Bourjolly,
John E. Boylan,
Michèle Breton,
David Canca,
Laurent Charlin,
Bo Chen,
Cihan Tugrul Cicek,
Louis Anthony Cox Jr,
Christine S. M. Currie,
Erik Demeulemeester,
Li Ding,
Stephen M. Disney,
Matthias Ehrgott,
Martin J. Eppler,
Güneş Erdoğan,
Bernard Fortz,
L. Alberto Franco
, et al. (57 additional authors not shown)
Abstract:
Throughout its history, Operational Research has evolved to include a variety of methods, models and algorithms that have been applied to a diverse and wide range of contexts. This encyclopedic article consists of two main sections: methods and applications. The first aims to summarise the up-to-date knowledge and provide an overview of the state-of-the-art methods and key developments in the vari…
▽ More
Throughout its history, Operational Research has evolved to include a variety of methods, models and algorithms that have been applied to a diverse and wide range of contexts. This encyclopedic article consists of two main sections: methods and applications. The first aims to summarise the up-to-date knowledge and provide an overview of the state-of-the-art methods and key developments in the various subdomains of the field. The second offers a wide-ranging list of areas where Operational Research has been applied. The article is meant to be read in a nonlinear fashion. It should be used as a point of reference or first-port-of-call for a diverse pool of readers: academics, researchers, students, and practitioners. The entries within the methods and applications sections are presented in alphabetical order. The authors dedicate this paper to the 2023 Turkey/Syria earthquake victims. We sincerely hope that advances in OR will play a role towards minimising the pain and suffering caused by this and future catastrophes.
△ Less
Submitted 13 January, 2024; v1 submitted 24 March, 2023;
originally announced March 2023.
-
A Brief Introduction to Robust Bilevel Optimization
Authors:
Yasmine Beck,
Ivana Ljubić,
Martin Schmidt
Abstract:
Bilevel optimization is a powerful tool for modeling hierarchical decision making processes. However, the resulting problems are challenging to solve - both in theory and practice. Fortunately, there have been significant algorithmic advances in the field so that we can solve much larger and also more complicated problems today compared to what was possible to solve two decades ago. This results i…
▽ More
Bilevel optimization is a powerful tool for modeling hierarchical decision making processes. However, the resulting problems are challenging to solve - both in theory and practice. Fortunately, there have been significant algorithmic advances in the field so that we can solve much larger and also more complicated problems today compared to what was possible to solve two decades ago. This results in more and more challenging bilevel problems that researchers try to solve today. In this article, we give a brief introduction to one of these more challenging classes of bilevel problems: bilevel optimization under uncertainty using robust optimization techniques. To this end, we briefly state different versions of uncertain bilevel problems that result from different levels of cooperation of the follower as well as on when the uncertainty is revealed. We highlight these concepts using an academic example and discuss recent results from the literature concerning complexity as well as solution approaches. Finally, we discuss that the sources of uncertainty in bilevel optimization are much richer than in single-level optimization and, to this end, introduce the concept of decision uncertainty.
△ Less
Submitted 24 January, 2023; v1 submitted 29 November, 2022;
originally announced November 2022.
-
Mathematical Programming Formulations for the Collapsed k-Core Problem
Authors:
Martina Cerulli,
Domenico Serra,
Carmine Sorgente,
Claudia Archetti,
Ivana Ljubic
Abstract:
In social network analysis, the size of the k-core, i.e., the maximal induced subgraph of the network with minimum degree at least k, is frequently adopted as a typical metric to evaluate the cohesiveness of a community. We address the Collapsed k-Core Problem, which seeks to find a subset of $b$ users, namely the most critical users of the network, the removal of which results in the smallest pos…
▽ More
In social network analysis, the size of the k-core, i.e., the maximal induced subgraph of the network with minimum degree at least k, is frequently adopted as a typical metric to evaluate the cohesiveness of a community. We address the Collapsed k-Core Problem, which seeks to find a subset of $b$ users, namely the most critical users of the network, the removal of which results in the smallest possible k-core. For the first time, both the problem of finding the k-core of a network and the Collapsed k-Core Problem are formulated using mathematical programming. On the one hand, we model the Collapsed k-Core Problem as a natural deletion-round-indexed Integer Linear formulation. On the other hand, we provide two bilevel programs for the problem, which differ in the way in which the k-core identification problem is formulated at the lower level. The first bilevel formulation is reformulated as a single-level sparse model, exploiting a Benders-like decomposition approach. To derive the second bilevel model, we provide a linear formulation for finding the k-core and use it to state the lower-level problem. We then dualize the lower level and obtain a compact Mixed-Integer Nonlinear single-level problem reformulation. We additionally derive a combinatorial lower bound on the value of the optimal solution and describe some pre-processing procedures and valid inequalities for the three formulations. The performance of the proposed formulations is compared on a set of benchmarking instances with the existing state-of-the-art solver for mixed-integer bilevel problems proposed in (Fischetti et al., A New General-Purpose Algorithm for Mixed-Integer Bilevel Linear Programs, Operations Research 65(6), 2017).
△ Less
Submitted 2 May, 2023; v1 submitted 27 November, 2022;
originally announced November 2022.
-
On SOCP-based disjunctive cuts for solving a class of integer bilevel nonlinear programs
Authors:
Elisabeth Gaar,
Jon Lee,
Ivana Ljubić,
Markus Sinnl,
Kübra Tanınmış
Abstract:
We study a class of integer bilevel programs with second-order cone constraints at the upper-level and a convex-quadratic objective function and linear constraints at the lower-level. We develop disjunctive cuts (DCs) to separate bilevel-infeasible solutions using a second-order-cone-based cut-generating procedure. We propose DC separation strategies and consider several approaches for removing re…
▽ More
We study a class of integer bilevel programs with second-order cone constraints at the upper-level and a convex-quadratic objective function and linear constraints at the lower-level. We develop disjunctive cuts (DCs) to separate bilevel-infeasible solutions using a second-order-cone-based cut-generating procedure. We propose DC separation strategies and consider several approaches for removing redundant disjunctions and normalization. Using these DCs, we propose a branch-and-cut algorithm for the problem class we study, and a cutting-plane method for the problem variant with only binary variables.
We present an extensive computational study on a diverse set of instances, including instances with binary and with integer variables, and instances with a single and with multiple linking constraints. Our computational study demonstrates that the proposed enhancements of our solution approaches are effective for improving the performance. Moreover, both of our approaches outperform a state-of-the-art generic solver for mixed-integer bilevel linear programs that is able to solve a linearized version of our binary instances.
△ Less
Submitted 8 January, 2023; v1 submitted 11 July, 2022;
originally announced July 2022.
-
Reinforcement Learning Approaches for the Orienteering Problem with Stochastic and Dynamic Release Dates
Authors:
Yuanyuan Li,
Claudia Archetti,
Ivana Ljubic
Abstract:
In this paper, we study a sequential decision-making problem faced by e-commerce carriers related to when to send out a vehicle from the central depot to serve customer requests, and in which order to provide the service, under the assumption that the time at which parcels arrive at the depot is stochastic and dynamic. The objective is to maximize the expected number of parcels that can be deliver…
▽ More
In this paper, we study a sequential decision-making problem faced by e-commerce carriers related to when to send out a vehicle from the central depot to serve customer requests, and in which order to provide the service, under the assumption that the time at which parcels arrive at the depot is stochastic and dynamic. The objective is to maximize the expected number of parcels that can be delivered during service hours. We propose two reinforcement learning (RL) approaches for solving this problem. These approaches rely on a look-ahead strategy in which future release dates are sampled in a Monte-Carlo fashion and a batch approach is used to approximate future routes. Both RL approaches are based on value function approximation - one combines it with a consensus function (VFA-CF) and the other one with a two-stage stochastic integer linear programming model (VFA-2S). VFA-CF and VFA-2S do not need extensive training as they are based on very few hyper-parameters and make good use of integer linear programming (ILP) and branch-and-cut-based exact methods to improve the quality of decisions. We also establish sufficient conditions for partial characterization of optimal policy and integrate them into VFA-CF/VFA-2S. In an empirical study, we conduct a competitive analysis using upper bounds with perfect information. We also show that VFA-CF and VFA-2S greatly outperform alternative approaches that: 1) do not rely on future information, or 2) are based on point estimation of future information, or 3) employ heuristics rather than exact methods, or 4) use exact evaluations of future rewards.
△ Less
Submitted 27 May, 2024; v1 submitted 2 July, 2022;
originally announced July 2022.
-
Benders Adaptive-Cuts Method for Two-Stage Stochastic Programs
Authors:
Cristian Ramírez-Pico,
Ivana Ljubić,
Eduardo Moreno
Abstract:
Benders decomposition is one of the most applied methods to solve two-stage stochastic problems (TSSP) with a large number of scenarios. The main idea behind the Benders decomposition is to solve a large problem by replacing the values of the second-stage subproblems with individual variables, and progressively forcing those variables to reach the optimal value of the subproblems, dynamically inse…
▽ More
Benders decomposition is one of the most applied methods to solve two-stage stochastic problems (TSSP) with a large number of scenarios. The main idea behind the Benders decomposition is to solve a large problem by replacing the values of the second-stage subproblems with individual variables, and progressively forcing those variables to reach the optimal value of the subproblems, dynamically inserting additional valid constraints, known as Benders cuts. Most traditional implementations add a cut for each scenario (multi-cut) or a single-cut that includes all scenarios. In this paper we present a novel Benders adaptive-cuts method, where the Benders cuts are aggregated according to a partition of the scenarios, which is dynamically refined using the LP-dual information of the subproblems. This scenario aggregation/disaggregation is based on the Generalized Adaptive Partitioning Method (GAPM), which has been successfully applied to TSSPs. We formalize this hybridization of Benders decomposition and the GAPM, by providing sufficient conditions under which an optimal solution of the deterministic equivalent can be obtained in a finite number of iterations. Our new method can be interpreted as a compromise between the Benders single-cuts and multi-cuts methods, drawing on the advantages of both sides, by rendering the initial iterations faster (as for the single-cuts Benders) and ensuring the overall faster convergence (as for the multi-cuts Benders). Computational experiments on three TSSPs validate these statements, showing that the new method outperforms the other implementations of Benders method, as well as other standard methods for solving TSSPs, in particular when the number of scenarios is very large.
△ Less
Submitted 22 November, 2022; v1 submitted 1 March, 2022;
originally announced March 2022.
-
An Exact Method for Fortification Games
Authors:
Markus Leitner,
Ivana Ljubić,
Michele Monaci,
Markus Sinnl,
Kübra Tanınmış
Abstract:
A fortification game (FG) is a three-level, two-player Stackelberg game, also known as defender-attacker-defender game, in which at the uppermost level, the defender selects some assets to be protected from potential malicious attacks. At the middle level, the attacker solves an interdiction game by depreciating unprotected assets, i.e., reducing the values of such assets for the defender, while a…
▽ More
A fortification game (FG) is a three-level, two-player Stackelberg game, also known as defender-attacker-defender game, in which at the uppermost level, the defender selects some assets to be protected from potential malicious attacks. At the middle level, the attacker solves an interdiction game by depreciating unprotected assets, i.e., reducing the values of such assets for the defender, while at the innermost level the defender solves a recourse problem over the surviving or partially damaged assets. Fortification games have applications in various important areas, such as military operations, design of survivable networks, protection of facilities, or power grid protection. In this work, we present an exact solution algorithm for FGs, in which the recourse problems correspond to (possibly NP-hard) combinatorial optimization problems. The algorithm is based on a new generic mixed-integer linear programming reformulation in the natural space of fortification variables. Our new model makes use of fortification cuts that measure the contribution of a given fortification strategy to the objective function value. These cuts are generated on-the-fly by solving separation problems, which correspond to (modified) middle-level interdiction games. We design a branch-and-cut-based solution algorithm based on fortification cuts, their lifted versions, and other speed-up techniques. We present a computational study using the knapsack fortification game and the shortest path fortification game. For the latter one, we include a comparison with a state-of-the-art solution method from the literature. Our algorithm outperforms this method and allows us to solve previously unsolved instances to optimality.
△ Less
Submitted 9 February, 2022; v1 submitted 26 November, 2021;
originally announced November 2021.
-
SOCP-based disjunctive cuts for a class of integer nonlinear bilevel programs
Authors:
Elisabeth Gaar,
Jon Lee,
Ivana Ljubić,
Markus Sinnl,
Kübra Tanınmış
Abstract:
We study a class of bilevel integer programs with second-order cone constraints at the upper level and a convex quadratic objective and linear constraints at the lower level. We develop disjunctive cuts to separate bilevel infeasible points using a second-order-cone-based cut-generating procedure. To the best of our knowledge, this is the first time disjunctive cuts are studied in the context of d…
▽ More
We study a class of bilevel integer programs with second-order cone constraints at the upper level and a convex quadratic objective and linear constraints at the lower level. We develop disjunctive cuts to separate bilevel infeasible points using a second-order-cone-based cut-generating procedure. To the best of our knowledge, this is the first time disjunctive cuts are studied in the context of discrete bilevel optimization. Using these disjunctive cuts, we establish a branch-and-cut algorithm for the problem class we study, and a cutting plane method for the problem variant with only binary variables. We present a preliminary computational study on instances with no second-order cone constraints at the upper level and a single linear constraint at the lower level. Our study demonstrates that both our approaches outperform a state-of-the-art generic solver for mixed-integer bilevel linear programs that is able to solve a linearized version of our test instances, where the non-linearities are linearized in a McCormick fashion.
△ Less
Submitted 8 March, 2022; v1 submitted 12 November, 2021;
originally announced November 2021.
-
The Generalized Reserve Set Covering Problem with Connectivity and Buffer Requirements
Authors:
Eduardo Álvarez-Miranda,
Marcos Goycoolea,
Ivana Ljubić,
Markus Sinnl
Abstract:
The design of nature reserves is becoming, more and more, a crucial task for ensuring the conservation of endangered wildlife. In order to guarantee the preservation of species and a general ecological functioning, the designed reserves must typically verify a series of spatial requirements. Among the required characteristics, practitioners and researchers have pointed out two crucial aspects: (i)…
▽ More
The design of nature reserves is becoming, more and more, a crucial task for ensuring the conservation of endangered wildlife. In order to guarantee the preservation of species and a general ecological functioning, the designed reserves must typically verify a series of spatial requirements. Among the required characteristics, practitioners and researchers have pointed out two crucial aspects: (i) connectivity, so as to avoid spatial fragmentation, and (ii) the design of buffer zones surrounding (or protecting) so-called core areas. In this paper, we introduce the Generalized Reserve Set Covering Problem with Connectivity and Buffer Requirements. This problem extends the classical Reserve Set Covering Problem and allows to address these two requirements simultaneously. A solution framework based on Integer Linear Programming and branch-and-cut is developed. The framework is enhanced by valid inequalities, a construction and a primal heuristic and local branching. An extensive computational study on grid-graph instances and real-life instances based on data from three states of the U.S. and one region of Australia is carried out to assess the suitability of the proposed model to deal with the challenges faced by decision-makers in natural reserve design. The results show, on the one hand, the flexibility of the proposed models to provide solutions according to the decision-makers' requirements, and on the other hand, the effectiveness of the devised algorithm for providing' good solutions in reasonable computing times.
△ Less
Submitted 11 September, 2019; v1 submitted 10 September, 2019;
originally announced September 2019.