-
Real-Time Integrated Learning and Decision-Making for Asset Networks
Authors:
Peter Verleijsdonk,
Collin Drent,
Stella Kapodistria,
Willem van Jaarsveld
Abstract:
Problem definition: Unexpected component failures in industrial assets can lead to significant financial losses and operational disruptions, making preventive maintenance crucial for reducing unplanned downtime. This study focuses on optimizing maintenance for economically coupled assets that degrade according to a compound Poisson process, failing when degradation exceeds a specified threshold. D…
▽ More
Problem definition: Unexpected component failures in industrial assets can lead to significant financial losses and operational disruptions, making preventive maintenance crucial for reducing unplanned downtime. This study focuses on optimizing maintenance for economically coupled assets that degrade according to a compound Poisson process, failing when degradation exceeds a specified threshold. Degradation parameters vary across components and cannot be directly observed, necessitating inference from real-time sensor data, with maintenance limited to scheduled interventions. Academic/practical relevance: We characterize optimal replacement policies that have access to the degradation process parameters, which inform suitable initial policies for our deep reinforcement learning (DRL) algorithm. The effectiveness of the trained policies is demonstrated through a case study on interventional X-ray systems. Methodology: The maintenance optimization problem is modeled as a partially observable Markov decision process (POMDP). To manage the inherent computational complexity, we reformulate it as a Bayesian Markov decision process (BMDP) using conjugate pairs. This allows us to extend the DRL algorithm to BMDPs. Additionally, we propose an open-loop feedback approach to adapt the DRL algorithm to the POMDP setting when a BMDP reformulation is infeasible. Results: Numerical results indicate that our DRL algorithm effectively reduces maintenance costs and unplanned downtime by leveraging real-time data and shared setup costs to learn state-of-the-art complex opportunistic maintenance policies. Managerial implications: Our approach demonstrates how integrating real-time learning and decision-making can significantly lower costs in industrial asset networks, providing a practical solution for improving operational efficiency amidst asset heterogeneity and economic dependence.
△ Less
Submitted 23 October, 2024;
originally announced October 2024.
-
A Holistic Approach for Bitcoin Confirmation Times & Optimal Fee Selection
Authors:
Rowel Gündlach,
Ivo V. Stoepker,
Stella Kapodistria,
Jacques A. C. Resing
Abstract:
Bitcoin is currently subject to a significant pay-for-speed trade-off. This is caused by lengthy and highly variable transaction confirmation times, especially during times of congestion. Users can reduce their transaction confirmation times by increasing their transaction fee. In this paper, based on the inner workings of Bitcoin, we propose a model-based approach (based on the Cramér-Lundberg mo…
▽ More
Bitcoin is currently subject to a significant pay-for-speed trade-off. This is caused by lengthy and highly variable transaction confirmation times, especially during times of congestion. Users can reduce their transaction confirmation times by increasing their transaction fee. In this paper, based on the inner workings of Bitcoin, we propose a model-based approach (based on the Cramér-Lundberg model) that can be used to determine the optimal fee, via, for example, the mean or quantiles, and models accurately the confirmation time distribution for a given fee. The proposed model is highly suitable as it arises as the limiting model for the mempool process (that tracks the unconfirmed transactions), which we rigorously show via a fluid limit and we extend this to the diffusion limit (an approximation of the Cramér-Lundberg model for fast computations in highly congested instances). We also propose methods (incorporating the real-time data) to estimate the model parameters, thereby combining model and data-driven approaches. The model-based approach is validated on real-world data and the resulting transaction fees outperform, in most instances, the data-driven ones.
△ Less
Submitted 27 February, 2024;
originally announced February 2024.
-
Scalable Policies for the Dynamic Traveling Multi-Maintainer Problem with Alerts
Authors:
Peter Verleijsdonk,
Willem van Jaarsveld,
Stella Kapodistria
Abstract:
Downtime of industrial assets such as wind turbines and medical imaging devices is costly. To avoid such downtime costs, companies seek to initiate maintenance just before failure, which is challenging because: (i) Asset failures are notoriously difficult to predict, even in the presence of real-time monitoring devices which signal degradation; and (ii) Limited resources are available to serve a n…
▽ More
Downtime of industrial assets such as wind turbines and medical imaging devices is costly. To avoid such downtime costs, companies seek to initiate maintenance just before failure, which is challenging because: (i) Asset failures are notoriously difficult to predict, even in the presence of real-time monitoring devices which signal degradation; and (ii) Limited resources are available to serve a network of geographically dispersed assets. In this work, we study the dynamic traveling multi-maintainer problem with alerts ($K$-DTMPA) under perfect condition information with the objective to devise scalable solution approaches to maintain large networks with $K$ maintenance engineers. Since such large-scale $K$-DTMPA instances are computationally intractable, we propose an iterative deep reinforcement learning (DRL) algorithm optimizing long-term discounted maintenance costs. The efficiency of the DRL approach is vastly improved by a reformulation of the action space (which relies on the Markov structure of the underlying problem) and by choosing a smart, suitable initial solution. The initial solution is created by extending existing heuristics with a dispatching mechanism. These extensions further serve as compelling benchmarks for tailored instances. We demonstrate through extensive numerical experiments that DRL can solve single maintainer instances up to optimality, regardless of the chosen initial solution. Experiments with hospital networks containing up to $35$ assets show that the proposed DRL algorithm is scalable. Lastly, the trained policies are shown to be robust against network modifications such as removing an asset or an engineer or yield a suitable initial solution for the DRL approach.
△ Less
Submitted 9 January, 2024;
originally announced January 2024.
-
Workload analysis of a two-queue fluid polling model
Authors:
Stella Kapodistria,
Mayank Saxena,
Onno Boxma,
Offer Kella
Abstract:
In this paper, we analyze a two-queue random time-limited Markov modulated polling model. In the first part of the paper, we investigate the fluid version: Fluid arrives at the two queues as two independent flows with deterministic rate. There is a single server that serves both queues at constant speeds. The server spends an exponentially distributed amount of time in each queue. After the comple…
▽ More
In this paper, we analyze a two-queue random time-limited Markov modulated polling model. In the first part of the paper, we investigate the fluid version: Fluid arrives at the two queues as two independent flows with deterministic rate. There is a single server that serves both queues at constant speeds. The server spends an exponentially distributed amount of time in each queue. After the completion of such a visit time to one queue, the server instantly switches to the other queue, i.e., there is no switchover time.
For this model, we first derive the Laplace-Stieltjes Transform (LST) of the stationary marginal fluid content/workload at each queue. Subsequently, we derive a functional equation for the LST of the two-dimensional workload distribution that leads to a Riemann-Hilbert boundary value problem (BVP). After taking a heavy-traffic limit, and restricting ourselves to the symmetric case, the boundary value problem simplifies and can be solved explicitly.
In the second part of the paper, allowing for more general (Lévy) input processes and server switching policies, we investigate the transient process-limit of the joint workload in heavy traffic. Again solving a BVP, we determine the stationary distribution of the limiting process. We show that, in the symmetric case, this distribution coincides with our earlier solution of the BVP, implying that in this case the two limits (stationarity and heavy traffic) commute.
△ Less
Submitted 9 December, 2021;
originally announced December 2021.
-
Policies for the Dynamic Traveling Maintainer Problem with Alerts
Authors:
Paulo da Costa,
Peter Verleijsdonk,
Simon Voorberg,
Alp Akcay,
Stella Kapodistria,
Willem van Jaarsveld,
Yingqian Zhang
Abstract:
Downtime of industrial assets such as wind turbines and medical imaging devices comes at a sharp cost. To avoid such downtime costs, companies seek to initiate maintenance just before failure. Unfortunately, this is challenging for the following two reasons: On the one hand, because asset failures are notoriously difficult to predict, even in the presence of real-time monitoring devices which sign…
▽ More
Downtime of industrial assets such as wind turbines and medical imaging devices comes at a sharp cost. To avoid such downtime costs, companies seek to initiate maintenance just before failure. Unfortunately, this is challenging for the following two reasons: On the one hand, because asset failures are notoriously difficult to predict, even in the presence of real-time monitoring devices which signal early degradation. On the other hand, because the available resources to serve a network of geographically dispersed assets are typically limited. In this paper, we propose a novel dynamic traveling maintainer problem with alerts model that incorporates these two challenges and we provide three solution approaches on how to dispatch the limited resources. Namely, we propose: (i) Greedy heuristic approaches that rank assets on urgency, proximity and economic risk; (ii) A novel traveling maintainer heuristic approach that optimizes short-term costs; and (iii) A deep reinforcement learning (DRL) approach that optimizes long-term costs. Each approach has different requirements concerning the available alert information. Experiments with small asset networks show that all methods can approximate the optimal policy when given access to complete condition information. For larger networks, the proposed methods yield competitive policies, with DRL consistently achieving the lowest costs.
△ Less
Submitted 20 May, 2022; v1 submitted 31 May, 2021;
originally announced May 2021.
-
Unified approach for solving exit problems for additive-increase and multiplicative-decrease processes
Authors:
Remco van der Hofstad,
Stella Kapodistria,
Zbigniew Palmowski,
Seva Shneer
Abstract:
We analyse an additive-increase and multiplicative-decrease (aka growth-collapse) process that grows linearly in time and that experiences downward jumps at Poisson epochs that are (deterministically) proportional to its present position. This process is used for example in modelling of Transmission Control Protocol (TCP) and can be viewed as a particular example of the so-called shot noise model,…
▽ More
We analyse an additive-increase and multiplicative-decrease (aka growth-collapse) process that grows linearly in time and that experiences downward jumps at Poisson epochs that are (deterministically) proportional to its present position. This process is used for example in modelling of Transmission Control Protocol (TCP) and can be viewed as a particular example of the so-called shot noise model, a basic tool in modeling earthquakes, avalanches and neuron firings.
For this process, and also for its reflected versions, we consider one- and two-sided exit problems that concern the identification of the laws of exit times from fixed intervals and half-lines. All proofs are based on a unified first-step analysis approach at the first jump epoch, which allows us to give explicit, yet involved, formulas for their Laplace transforms.
All the eight Laplace transforms can be described in terms of two so-called scale functions $Z_{\uparrow}$ and $L_{\uparrow}$. Here $Z_{\uparrow}$ is described in terms of multiple explicit sums, and $L_{\uparrow}$ in terms of an explicit recursion formula. All other Laplace transforms can be obtained from $Z_{\uparrow}$ and $L_{\uparrow}$ by taking limits, derivatives, integrals and combinations of these.
△ Less
Submitted 31 January, 2021;
originally announced February 2021.
-
Stability of two-dimensional Markov processes, with an application to QBD processes with an infinite number of phases
Authors:
Stella Kapodistria,
Seva Shneer
Abstract:
In this paper, we derive a simple drift condition for the stability of a class of two-dimensional Markov processes, for which one of the coordinates (also referred to as the {\em phase} for convenience) has a well understood behaviour dependent on the other coordinate (also referred as {\em level}). The first (phase) component's transitions may depend on the second component and are only assumed t…
▽ More
In this paper, we derive a simple drift condition for the stability of a class of two-dimensional Markov processes, for which one of the coordinates (also referred to as the {\em phase} for convenience) has a well understood behaviour dependent on the other coordinate (also referred as {\em level}). The first (phase) component's transitions may depend on the second component and are only assumed to be eventually independent. The second (level) component has partially bounded jumps and it is assumed to have a negative drift given that the first one is in its stationary distribution. The results presented in this work can be applied to processes of the QBD (quasi-birth-and-death) type on the quarter- and on the half-plane, where the phase and level are interdependent. Furthermore, they provide an off-the-shelf technique to tackle stability issues for a class of two-dimensional Markov processes. These results set the stepping stones towards closing the existing gap in the literature of deriving easily verifiable conditions/criteria for two-dimensional processes with unbounded jumps and interdependence between the two components.
△ Less
Submitted 30 September, 2020;
originally announced September 2020.
-
Condition based maintenance policies under imperfect maintenance at scheduled and unscheduled opportunities
Authors:
C. Drent,
S. Kapodistria,
J. A. C. Resing
Abstract:
Motivated by the cost savings that can be obtained by sharing resources in a network context, we consider a stylized, yet representative model, for the coordination of maintenance and service logistics for a geographic network of assets. Capital assets, such as wind turbines in a wind park, require maintenance throughout their long lifetimes. Two types of preventive maintenance are considered: pla…
▽ More
Motivated by the cost savings that can be obtained by sharing resources in a network context, we consider a stylized, yet representative model, for the coordination of maintenance and service logistics for a geographic network of assets. Capital assets, such as wind turbines in a wind park, require maintenance throughout their long lifetimes. Two types of preventive maintenance are considered: planned maintenance at periodic, scheduled opportunities, and opportunistic maintenance at unscheduled opportunities. The latter type of maintenance arises due to the network context: when an asset in the network fails, this constitutes an opportunity for preventive maintenance for the other assets in the network.
So as to increase the realism of the model at hand and its applicability to various sectors, we consider the option of not-deferring and of deferring planned maintenance after the occurrence of opportunistic maintenance. We also assume that preventive maintenance may not always restore the condition of the system to `as good as new'. By formulating this problem as a semi-Markov decision process, we characterize the optimal policy as a control limit policy (depending on the remaining time until the next planned maintenance) that indicates on the one hand when it is optimal to perform preventive maintenance and on the other hand when maintenance resources should be shared if an opportunity in the network arises. In order to facilitate managerial insights on the effect of each parameter on the cost, we provide a closed-form expression for the long-run rate of cost for any given control limit policy (depending on the remaining time until the next planned maintenance) and compare the costs (under the optimal policy) to these of sub-optimal policies that neglect the opportunity for resource sharing. We illustrate our findings using data from the wind energy industry.
△ Less
Submitted 16 August, 2019; v1 submitted 25 March, 2019;
originally announced March 2019.
-
Analysis of the shortest relay queue policy in a cooperative random access network with collisions
Authors:
Mayank Saxena,
Ioannis Dimitriou,
Stella Kapodistria
Abstract:
The scope of this work is twofold: On the one hand, strongly motivated by emerging engineering issues in multiple access communication systems, we investigate the performance of a slotted-time relay-assisted cooperative random access wireless network with collisions and with join the shortest queue relay-routing protocol. For this model, we investigate the stability condition, and apply different…
▽ More
The scope of this work is twofold: On the one hand, strongly motivated by emerging engineering issues in multiple access communication systems, we investigate the performance of a slotted-time relay-assisted cooperative random access wireless network with collisions and with join the shortest queue relay-routing protocol. For this model, we investigate the stability condition, and apply different methods to derive the joint equilibrium distribution of the queue lengths. On the other hand, using the cooperative communication system as a vehicle for illustration, we investigate and compare three different approaches for this type of multi-dimensional stochastic processes, namely the compensation approach, the power series algorithm (PSA), and the probability generating function (PGF) approach. We present an extensive numerical comparison of the compensation approach and PSA, and discuss which method performs better in terms of accuracy and computation time. We also provide details on how to compute the PGF in terms of a solution of a Riemann-Hilbert boundary value problem.
△ Less
Submitted 1 October, 2018;
originally announced October 2018.
-
Two queues with random time-limited polling
Authors:
Mayank Saxena,
Onno Boxma,
Stella Kapodistria,
Rudesindo Núñez Queija
Abstract:
In this paper, we analyse a single server polling model with two queues. Customers arrive at the two queues according to two independent Poisson processes. There is a single server that serves both queues with generally distributed service times. The server spends an exponentially distributed amount of time in each queue. After the completion of this residing time, the server instantaneously switc…
▽ More
In this paper, we analyse a single server polling model with two queues. Customers arrive at the two queues according to two independent Poisson processes. There is a single server that serves both queues with generally distributed service times. The server spends an exponentially distributed amount of time in each queue. After the completion of this residing time, the server instantaneously switches to the other queue, i.e., there is no switch-over time. For this polling model we derive the steady-state marginal workload distribution, as well as heavy traffic and heavy tail asymptotic results. Furthermore, we also calculate the joint queue length distribution for the special case of exponentially distributed service times using singular perturbation analysis.
△ Less
Submitted 12 July, 2017; v1 submitted 24 January, 2017;
originally announced January 2017.
-
Condition-based maintenance at both scheduled and unscheduled opportunities
Authors:
Szilard Kalosi,
Stella Kapodistria,
Jacques A. C. Resing
Abstract:
Motivated by original equipment manufacturer (OEM) service and maintenance practices we consider a single component subject to replacements at failure instances and two types of preventive maintenance opportunities: scheduled, which occur due to periodic system reviews of the equipment, and unscheduled, which occur due to failures of other components in the system. Modelling the state of the compo…
▽ More
Motivated by original equipment manufacturer (OEM) service and maintenance practices we consider a single component subject to replacements at failure instances and two types of preventive maintenance opportunities: scheduled, which occur due to periodic system reviews of the equipment, and unscheduled, which occur due to failures of other components in the system. Modelling the state of the component appropriately and incorporating a realistic cost structure for corrective maintenance as well as condition-based maintenance (CBM), we derive the optimal CBM policy. In particular, we show that the optimal long-run average cost policy for the model at hand is a control-limit policy, where the control limit depends on the time until the next scheduled opportunity. Furthermore, we explicitly calculate the long-run average cost for any given control-limit time dependent policy and compare various policies numerically.
△ Less
Submitted 8 July, 2016;
originally announced July 2016.
-
Matrix geometric approach for random walks: stability condition and equilibrium distribution
Authors:
Stella Kapodistria,
Zbigniew Palmowski
Abstract:
In this paper, we analyse a sub-class of two-dimensional homogeneous nearest neighbour (simple) random walk restricted on the lattice using the matrix geometric approach. In particular, we first present an alternative approach for the calculation of the stability condition, extending the result of Neuts drift conditions [30] and connecting it with the result of Fayolle et al. which is based on Lya…
▽ More
In this paper, we analyse a sub-class of two-dimensional homogeneous nearest neighbour (simple) random walk restricted on the lattice using the matrix geometric approach. In particular, we first present an alternative approach for the calculation of the stability condition, extending the result of Neuts drift conditions [30] and connecting it with the result of Fayolle et al. which is based on Lyapunov functions [13]. Furthermore, we consider the sub-class of random walks with equilibrium distributions given as series of product-forms and, for this class of random walks, we calculate the eigenvalues and the corresponding eigenvectors of the infinite matrix $\mathbf{R}$ appearing in the matrix geometric approach. This result is obtained by connecting and extending three existing approaches available for such an analysis: the matrix geometric approach, the compensation approach and the boundary value problem method. In this paper, we also present the spectral properties of the infinite matrix $\mathbf{R}$.
△ Less
Submitted 19 July, 2017; v1 submitted 15 March, 2016;
originally announced March 2016.
-
Approximate performance analysis of generalized join the shortest queue routing
Authors:
Jori Selen,
Ivo J. B. F. Adan,
Stella Kapodistria
Abstract:
In this paper we propose a highly accurate approximate performance analysis of a heterogeneous server system with a processor sharing service discipline and a general job-size distribution under a generalized join the shortest queue (GJSQ) routing protocol. The GJSQ routing protocol is a natural extension of the well-known join the shortest queue routing policy that takes into account the non-iden…
▽ More
In this paper we propose a highly accurate approximate performance analysis of a heterogeneous server system with a processor sharing service discipline and a general job-size distribution under a generalized join the shortest queue (GJSQ) routing protocol. The GJSQ routing protocol is a natural extension of the well-known join the shortest queue routing policy that takes into account the non-identical service rates in addition to the number of jobs at each server. The performance metrics that are of interest here are the equilibrium distribution and the mean and standard deviation of the number of jobs at each server. We show that the latter metrics are near-insensitive to the job-size distribution using simulation experiments. By applying a single queue approximation we model each server as a single server queue with a state-dependent arrival process, independent of other servers in the system, and derive the distribution of the number of jobs at the server. These state-dependent arrival rates are intended to capture the inherent correlation between servers in the original system and behave in a rather atypical way.
△ Less
Submitted 23 November, 2015; v1 submitted 18 September, 2015;
originally announced September 2015.
-
Steady-state analysis of shortest expected delay routing
Authors:
Jori Selen,
Ivo J. B. F. Adan,
Stella Kapodistria,
Johan S. H. van Leeuwaarden
Abstract:
We consider a queueing system consisting of two non-identical exponential servers, where each server has its own dedicated queue and serves the customers in that queue FCFS. Customers arrive according to a Poisson process and join the queue promising the shortest expected delay, which is a natural and near-optimal policy for systems with non-identical servers. This system can be modeled as an inho…
▽ More
We consider a queueing system consisting of two non-identical exponential servers, where each server has its own dedicated queue and serves the customers in that queue FCFS. Customers arrive according to a Poisson process and join the queue promising the shortest expected delay, which is a natural and near-optimal policy for systems with non-identical servers. This system can be modeled as an inhomogeneous random walk in the quadrant. By stretching the boundaries of the compensation approach we prove that the equilibrium distribution of this random walk can be expressed as a series of product-forms that can be determined recursively. The resulting series expression is directly amenable for numerical calculations and it also provides insight in the asymptotic behavior of the equilibrium probabilities as one of the state coordinates tends to infinity.
△ Less
Submitted 14 March, 2016; v1 submitted 11 September, 2015;
originally announced September 2015.
