-
Fuzzy Ensembles of Reinforcement Learning Policies for Robotic Systems with Varied Parameters
Authors:
Abdel Gafoor Haddad,
Mohammed B. Mohiuddin,
Igor Boiko,
Yahya Zweiri
Abstract:
Reinforcement Learning (RL) is an emerging approach to control many dynamical systems for which classical control approaches are not applicable or insufficient. However, the resultant policies may not generalize to variations in the parameters that the system may exhibit. This paper presents a powerful yet simple algorithm in which collaboration is facilitated between RL agents that are trained in…
▽ More
Reinforcement Learning (RL) is an emerging approach to control many dynamical systems for which classical control approaches are not applicable or insufficient. However, the resultant policies may not generalize to variations in the parameters that the system may exhibit. This paper presents a powerful yet simple algorithm in which collaboration is facilitated between RL agents that are trained independently to perform the same task but with different system parameters. The independency among agents allows the exploitation of multi-core processing to perform parallel training. Two examples are provided to demonstrate the effectiveness of the proposed technique. The main demonstration is performed on a quadrotor with slung load tracking problem in a real-time experimental setup. It is shown that integrating the developed algorithm outperforms individual policies by reducing the RMSE tracking error. The robustness of the ensemble is also verified against wind disturbance.
△ Less
Submitted 8 November, 2023;
originally announced November 2023.
-
Reinforcement Learning Generalization for Nonlinear Systems Through Dual-Scale Homogeneity Transformations
Authors:
Abdel Gafoor Haddad,
Igor Boiko,
Yahya Zweiri
Abstract:
Reinforcement learning is an emerging approach to control dynamical systems for which classical approaches are difficult to apply. However, trained agents may not generalize against the variations of system parameters. This paper presents the concept of dual-scale homogeneity, an important property in understating the scaling behavior of nonlinear systems. Furthermore, it also presents an effectiv…
▽ More
Reinforcement learning is an emerging approach to control dynamical systems for which classical approaches are difficult to apply. However, trained agents may not generalize against the variations of system parameters. This paper presents the concept of dual-scale homogeneity, an important property in understating the scaling behavior of nonlinear systems. Furthermore, it also presents an effective yet simple approach to designing a parameter-dependent control law that homogenizes a nonlinear system. The presented approach is applied to two systems, demonstrating its ability to provide a consistent performance irrespective of parameters variations. To demonstrate the practicality of the proposed approach, the control policy is generated by a deep deterministic policy gradient to control the load position of a quadrotor with a slung load. The proposed synergy between the homogeneity transformations and reinforcement learning yields superior performance compared to other recent learning-based control techniques. It achieves a success rate of 96% in bringing the load to its designated target with a 3D RMSE of 0.0253 m. The video that shows the experimental results along with a summary of the paper is available at this link.
△ Less
Submitted 8 November, 2023;
originally announced November 2023.
-
Relay-based identification of Aerodynamic and Delay Sensor Dynamics with applications for Unmanned Aerial Vehicles
Authors:
Anees Peringal,
Mohamad Chehadeh,
Igor Boiko,
Yahya Zweiri
Abstract:
In this paper, we present a real-time system identification method based on relay feedback testing with applications to multirotor unmanned aerial vehicles. The proposed identification method provides an alternative to the expensive lab testing of certain UAV dynamic parameters. Moreover, it has the advantage of identifying the parameters that get changed throughout the operation of the UAV, which…
▽ More
In this paper, we present a real-time system identification method based on relay feedback testing with applications to multirotor unmanned aerial vehicles. The proposed identification method provides an alternative to the expensive lab testing of certain UAV dynamic parameters. Moreover, it has the advantage of identifying the parameters that get changed throughout the operation of the UAV, which requires onboard identification methods. The modified relay feedback test (MRFT) is used to generate stable limit cycles at frequency points that reveal the underlying UAV dynamics. The locus of the perturbed relay system (LPRS) is used to predict the exact amplitude and frequency of these limit cycles. Real-time identification is achieved by using the homogeneity properties of the MRFT and the LPRS which are proven in this paper. The proposed identification method was tested experimentally to estimate the aerodynamic parameters as well as the onboard sensor's time delay parameters. The MRFT testing takes a few seconds to perform, and the identification computations take an average of 0.2 seconds to complete in modern embedded computers. The proposed identification method is compared against state-of-the-art alternatives. Advantages in identification accuracy and quantification of uncertainty in estimated parameters are shown.
△ Less
Submitted 25 March, 2023;
originally announced March 2023.
-
The Role of Time Delay in Sim2real Transfer of Reinforcement Learning for Cyber-Physical Systems
Authors:
Mohamad Chehadeh,
Igor Boiko,
Yahya Zweiri
Abstract:
This paper analyzes the simulation to reality gap in reinforcement learning (RL) cyber-physical systems with fractional delays (i.e. delays that are non-integer multiple of the sampling period). The consideration of fractional delay has important implications on the nature of the cyber-physical system considered. Systems with delays are non-Markovian, and the system state vector needs to be extend…
▽ More
This paper analyzes the simulation to reality gap in reinforcement learning (RL) cyber-physical systems with fractional delays (i.e. delays that are non-integer multiple of the sampling period). The consideration of fractional delay has important implications on the nature of the cyber-physical system considered. Systems with delays are non-Markovian, and the system state vector needs to be extended to make the system Markovian. We show that this is not possible when the delay is in the output, and the problem would always be non-Markovian. Based on this analysis, a sampling scheme is proposed that results in efficient RL training and agents that perform well in realistic multirotor unmanned aerial vehicle simulations. We demonstrate that the resultant agents do not produce excessive oscillations, which is not the case with RL agents that do not consider time delay in the model.
△ Less
Submitted 30 September, 2022;
originally announced September 2022.
-
Analysis of the Effect of Time Delay for Unmanned Aerial Vehicles with Applications to Vision Based Navigation
Authors:
Muhammad Ahmed Humais,
Mohamad Chehadeh,
Igor Boiko,
Yahya Zweiri
Abstract:
In this paper, we analyze the effect of time delay dynamics on controller design for Unmanned Aerial Vehicles (UAVs) with vision based navigation. Time delay is an inevitable phenomenon in cyber-physical systems, and has important implications on controller design and trajectory generation for UAVs. The impact of time delay on UAV dynamics increases with the use of the slower vision based navigati…
▽ More
In this paper, we analyze the effect of time delay dynamics on controller design for Unmanned Aerial Vehicles (UAVs) with vision based navigation. Time delay is an inevitable phenomenon in cyber-physical systems, and has important implications on controller design and trajectory generation for UAVs. The impact of time delay on UAV dynamics increases with the use of the slower vision based navigation stack. We show that the existing models in the literature, which exclude time delay, are unsuitable for controller tuning since a trivial solution for minimizing an error cost functional always exists. The trivial solution that we identify suggests use of infinite controller gains to achieve optimal performance, which contradicts practical findings. We avoid such shortcomings by introducing a novel nonlinear time delay model for UAVs, and then obtain a set of linear decoupled models corresponding to each of the UAV control loops. The cost functional of the linearized time delay model of angular and altitude dynamics is analyzed, and in contrast to the delay-free models, we show the existence of finite optimal controller parameters. Due to the use of time delay models, we experimentally show that the proposed model accurately represents system stability limits. Due to time delay consideration, we achieved a tracking results of RMSE 5.01 cm when tracking a lemniscate trajectory with a peak velocity of 2.09 m/s using visual odometry (VO) based UAV navigation, which is on par with the state-of-the-art.
△ Less
Submitted 5 September, 2022;
originally announced September 2022.
-
Design of Dynamics Invariant LSTM for Touch Based Human-UAV Interaction Detection
Authors:
Anees Peringal,
Mohamad Chehadeh,
Rana Azzam,
Mahmoud Hamandi,
Igor Boiko,
Yahya Zweiri
Abstract:
The field of Unmanned Aerial Vehicles (UAVs) has reached a high level of maturity in the last few years. Hence, bringing such platforms from closed labs, to day-to-day interactions with humans is important for commercialization of UAVs. One particular human-UAV scenario of interest for this paper is the payload handover scheme, where a UAV hands over a payload to a human upon their request. In thi…
▽ More
The field of Unmanned Aerial Vehicles (UAVs) has reached a high level of maturity in the last few years. Hence, bringing such platforms from closed labs, to day-to-day interactions with humans is important for commercialization of UAVs. One particular human-UAV scenario of interest for this paper is the payload handover scheme, where a UAV hands over a payload to a human upon their request. In this scope, this paper presents a novel real-time human-UAV interaction detection approach, where Long short-term memory (LSTM) based neural network is developed to detect state profiles resulting from human interaction dynamics. A novel data pre-processing technique is presented; this technique leverages estimated process parameters of training and testing UAVs to build dynamics invariant testing data. The proposed detection algorithm is lightweight and thus can be deployed in real-time using off the shelf UAV platforms; in addition, it depends solely on inertial and position measurements present on any classical UAV platform. The proposed approach is demonstrated on a payload handover task between multirotor UAVs and humans. Training and testing data were collected using real-time experiments. The detection approach has achieved an accuracy of 96\%, giving no false positives even in the presence of external wind disturbances, and when deployed and tested on two different UAVs.
△ Less
Submitted 12 July, 2022;
originally announced July 2022.
-
Multirotors from Takeoff to Real-Time Full Identification Using the Modified Relay Feedback Test and Deep Neural Networks
Authors:
Abdulla Ayyad,
Mohamad Chehadeh,
Pedro Silva,
Mohamad Wahbah,
Oussama Abdul Hay,
Igor Boiko,
Yahya Zweiri
Abstract:
Low cost real-time identification of multirotor unmanned aerial vehicle (UAV) dynamics is an active area of research supported by the surge in demand and emerging application domains. Such real-time identification capabilities shorten development time and cost, making UAVs' technology more accessible, and enable a wide variety of advanced applications. In this paper, we present a novel comprehensi…
▽ More
Low cost real-time identification of multirotor unmanned aerial vehicle (UAV) dynamics is an active area of research supported by the surge in demand and emerging application domains. Such real-time identification capabilities shorten development time and cost, making UAVs' technology more accessible, and enable a wide variety of advanced applications. In this paper, we present a novel comprehensive approach, called DNN-MRFT, for real-time identification and tuning of multirotor UAVs using the Modified Relay Feedback Test (MRFT) and Deep Neural Networks (DNN). The main contribution is the development of a generalized framework for the application of DNN-MRFT to higher-order systems. One of the notable advantages of DNN-MRFT is the exact estimation of identified process gain, which mitigates the inaccuracies introduced due to the use of the describing function method in approximating the response of Lure's systems. A secondary contribution is a generalized controller based on DNN-MRFT that takes-off a UAV with unknown dynamics and identifies the inner loops dynamics in-flight. Using the developed framework, DNN-MRFT is sequentially applied to the outer translational loops of the UAV utilizing in-flight results obtained for the inner attitude loops. DNN-MRFT takes on average 15 seconds to get the full knowledge of multirotor UAV dynamics and without any further tuning or calibration the UAV would be able to pass through a vertical window, and accurately follow trajectories achieving state-of-the-art performance. Such demonstrated accuracy, speed, and robustness of identification pushes the limits of state-of-the-art in real-time identification of UAVs.
△ Less
Submitted 6 September, 2021; v1 submitted 6 October, 2020;
originally announced October 2020.
