-
Deep Visual Servoing of an Aerial Robot Using Keypoint Feature Extraction
Authors:
Shayan Sepahvand,
Niloufar Amiri,
Farrokh Janabi-Sharifi
Abstract:
The problem of image-based visual servoing (IBVS) of an aerial robot using deep-learning-based keypoint detection is addressed in this article. A monocular RGB camera mounted on the platform is utilized to collect the visual data. A convolutional neural network (CNN) is then employed to extract the features serving as the visual data for the servoing task. This paper contributes to the field by ci…
▽ More
The problem of image-based visual servoing (IBVS) of an aerial robot using deep-learning-based keypoint detection is addressed in this article. A monocular RGB camera mounted on the platform is utilized to collect the visual data. A convolutional neural network (CNN) is then employed to extract the features serving as the visual data for the servoing task. This paper contributes to the field by circumventing not only the challenge stemming from the need for man-made marker detection in conventional visual servoing techniques, but also enhancing the robustness against undesirable factors including occlusion, varying illumination, clutter, and background changes, thereby broadening the applicability of perception-guided motion control tasks in aerial robots. Additionally, extensive physics-based ROS Gazebo simulations are conducted to assess the effectiveness of this method, in contrast to many existing studies that rely solely on physics-less simulations. A demonstration video is available at https://youtu.be/Dd2Her8Ly-E.
△ Less
Submitted 29 March, 2025;
originally announced March 2025.
-
Hybrid Visual Servoing of Tendon-driven Continuum Robots
Authors:
Rana Danesh,
Farrokh Janabi-Sharifi,
Farhad Aghili
Abstract:
This paper introduces a novel Hybrid Visual Servoing (HVS) approach for controlling tendon-driven continuum robots (TDCRs). The HVS system combines Image-Based Visual Servoing (IBVS) with Deep Learning-Based Visual Servoing (DLBVS) to overcome the limitations of each method and improve overall performance. IBVS offers higher accuracy and faster convergence in feature-rich environments, while DLBVS…
▽ More
This paper introduces a novel Hybrid Visual Servoing (HVS) approach for controlling tendon-driven continuum robots (TDCRs). The HVS system combines Image-Based Visual Servoing (IBVS) with Deep Learning-Based Visual Servoing (DLBVS) to overcome the limitations of each method and improve overall performance. IBVS offers higher accuracy and faster convergence in feature-rich environments, while DLBVS enhances robustness against disturbances and offers a larger workspace. By enabling smooth transitions between IBVS and DLBVS, the proposed HVS ensures effective control in dynamic, unstructured environments. The effectiveness of this approach is validated through simulations and real-world experiments, demonstrating that HVS achieves reduced iteration time, faster convergence, lower final error, and smoother performance compared to DLBVS alone, while maintaining DLBVS's robustness in challenging conditions such as occlusions, lighting changes, actuator noise, and physical impacts.
△ Less
Submitted 19 February, 2025;
originally announced February 2025.
-
Backstepping Control of Tendon-Driven Continuum Robots in Large Deflections Using the Cosserat Rod Model
Authors:
Rana Danesh,
Farrokh Janabi-Sharifi
Abstract:
This paper presents a study on the backstepping control of tendon-driven continuum robots for large deflections using the Cosserat rod model. Continuum robots are known for their flexibility and adaptability, making them suitable for various applications. However, modeling and controlling them pose challenges due to their nonlinear dynamics. To model their dynamics, the Cosserat rod method is empl…
▽ More
This paper presents a study on the backstepping control of tendon-driven continuum robots for large deflections using the Cosserat rod model. Continuum robots are known for their flexibility and adaptability, making them suitable for various applications. However, modeling and controlling them pose challenges due to their nonlinear dynamics. To model their dynamics, the Cosserat rod method is employed to account for significant deflections, and a numerical solution method is developed to solve the resulting partial differential equations. Previous studies on controlling tendon-driven continuum robots using Cosserat rod theory focused on sliding mode control and were not tested for large deflections, lacking experimental validation. In this paper, backstepping control is proposed as an alternative to sliding mode control for achieving a significant bending. The numerical results are validated through experiments in this study, demonstrating that the proposed backstepping control approach is a promising solution for achieving large deflections with smoother trajectories, reduced settling time, and lower overshoot. Furthermore, two scenarios involving external forces and disturbances were introduced to further highlight the robustness of the backstepping control approach.
△ Less
Submitted 16 December, 2024;
originally announced December 2024.
-
Dual Agent Learning Based Aerial Trajectory Tracking
Authors:
Shaswat Garg,
Houman Masnavi,
Baris Fidan,
Farrokh Janabi-Sharifi
Abstract:
This paper presents a novel reinforcement learning framework for trajectory tracking of unmanned aerial vehicles in cluttered environments using a dual-agent architecture. Traditional optimization methods for trajectory tracking face significant computational challenges and lack robustness in dynamic environments. Our approach employs deep reinforcement learning (RL) to overcome these limitations,…
▽ More
This paper presents a novel reinforcement learning framework for trajectory tracking of unmanned aerial vehicles in cluttered environments using a dual-agent architecture. Traditional optimization methods for trajectory tracking face significant computational challenges and lack robustness in dynamic environments. Our approach employs deep reinforcement learning (RL) to overcome these limitations, leveraging 3D pointcloud data to perceive the environment without relying on memory-intensive obstacle representations like occupancy grids. The proposed system features two RL agents: one for predicting UAV velocities to follow a reference trajectory and another for managing collision avoidance in the presence of obstacles. This architecture ensures real-time performance and adaptability to uncertainties. We demonstrate the efficacy of our approach through simulated and real-world experiments, highlighting improvements over state-of-the-art RL and optimization-based methods. Additionally, a curriculum learning paradigm is employed to scale the algorithms to more complex environments, ensuring robust trajectory tracking and obstacle avoidance in both static and dynamic scenarios.
△ Less
Submitted 30 October, 2024;
originally announced October 2024.
-
Keypoint Detection Technique for Image-Based Visual Servoing of Manipulators
Authors:
Niloufar Amiri,
Guanghui Wang,
Farrokh Janabi-Sharifi
Abstract:
This paper introduces an innovative keypoint detection technique based on Convolutional Neural Networks (CNNs) to enhance the performance of existing Deep Visual Servoing (DVS) models. To validate the convergence of the Image-Based Visual Servoing (IBVS) algorithm, real-world experiments utilizing fiducial markers for feature detection are conducted before designing the CNN-based feature detector.…
▽ More
This paper introduces an innovative keypoint detection technique based on Convolutional Neural Networks (CNNs) to enhance the performance of existing Deep Visual Servoing (DVS) models. To validate the convergence of the Image-Based Visual Servoing (IBVS) algorithm, real-world experiments utilizing fiducial markers for feature detection are conducted before designing the CNN-based feature detector. To address the limitations of fiducial markers, the novel feature detector focuses on extracting keypoints that represent the corners of a more realistic object compared to fiducial markers. A dataset is generated from sample data captured by the camera mounted on the robot end-effector while the robot operates randomly in the task space. The samples are automatically labeled, and the dataset size is increased by flipping and rotation. The CNN model is developed by modifying the VGG-19 pre-trained on the ImageNet dataset. While the weights in the base model remain fixed, the fully connected layer's weights are updated to minimize the mean absolute error, defined based on the deviation of predictions from the real pixel coordinates of the corners. The model undergoes two modifications: replacing max-pooling with average-pooling in the base model and implementing an adaptive learning rate that decreases during epochs. These changes lead to a 50 percent reduction in validation loss. Finally, the trained model's reliability is assessed through k-fold cross-validation.
△ Less
Submitted 20 September, 2024;
originally announced September 2024.
-
A Master-Follower Teleoperation System for Robotic Catheterization: Design, Characterization, and Tracking Control
Authors:
Ali A. Nazari,
Jeremy Catania,
Soroush Sadeghian,
Amir Jalali,
Houman Masnavi,
Farrokh Janabi-Sharifi,
Kourosh Zareinia
Abstract:
Minimally invasive robotic surgery has gained significant attention over the past two decades. Telerobotic systems, combined with robot-mediated minimally invasive techniques, have enabled surgeons and clinicians to mitigate radiation exposure for medical staff and extend medical services to remote and hard-to-reach areas. To enhance these services, teleoperated robotic surgery systems incorporati…
▽ More
Minimally invasive robotic surgery has gained significant attention over the past two decades. Telerobotic systems, combined with robot-mediated minimally invasive techniques, have enabled surgeons and clinicians to mitigate radiation exposure for medical staff and extend medical services to remote and hard-to-reach areas. To enhance these services, teleoperated robotic surgery systems incorporating master and follower devices should offer transparency, enabling surgeons and clinicians to remotely experience a force interaction similar to the one the follower device experiences with patients' bodies. This paper presents the design and development of a three-degree-of-freedom master-follower teleoperated system for robotic catheterization. To resemble manual intervention by clinicians, the follower device features a grip-insert-release mechanism to eliminate catheter buckling and torsion during operation. The bidirectionally navigable ablation catheter is statically characterized for force-interactive medical interventions. The system's performance is evaluated through approaching and open-loop path tracking over typical circular, infinity-like, and spiral paths. Path tracking errors are presented as mean Euclidean error (MEE) and mean absolute error (MAE). The MEE ranges from 0.64 cm (infinity-like path) to 1.53 cm (spiral path). The MAE also ranges from 0.81 cm (infinity-like path) to 1.92 cm (spiral path). The results indicate that while the system's precision and accuracy with an open-loop controller meet the design targets, closed-loop controllers are necessary to address the catheter's hysteresis and dead zone, and system nonlinearities.
△ Less
Submitted 14 February, 2025; v1 submitted 18 July, 2024;
originally announced July 2024.
-
Differentiable-Optimization Based Neural Policy for Occlusion-Aware Target Tracking
Authors:
Houman Masnavi,
Arun Kumar Singh,
Farrokh Janabi-Sharifi
Abstract:
Tracking a target in cluttered and dynamic environments is challenging but forms a core component in applications like aerial cinematography. The obstacles in the environment not only pose collision risk but can also occlude the target from the field-of-view of the robot. Moreover, the target future trajectory may be unknown and only its current state can be estimated. In this paper, we propose a…
▽ More
Tracking a target in cluttered and dynamic environments is challenging but forms a core component in applications like aerial cinematography. The obstacles in the environment not only pose collision risk but can also occlude the target from the field-of-view of the robot. Moreover, the target future trajectory may be unknown and only its current state can be estimated. In this paper, we propose a learned probabilistic neural policy for safe, occlusion-free target tracking.
The core novelty of our work stems from the structure of our policy network that combines generative modeling based on Conditional Variational Autoencoder (CVAE) with differentiable optimization layers. The role of the CVAE is to provide a base trajectory distribution which is then projected onto a learned feasible set through the optimization layer. Furthermore, both the weights of the CVAE network and the parameters of the differentiable optimization can be learned in an end-to-end fashion through demonstration trajectories.
We improve the state-of-the-art (SOTA) in the following respects. We show that our learned policy outperforms existing SOTA in terms of occlusion/collision avoidance capabilities and computation time. Second, we present an extensive ablation showing how different components of our learning pipeline contribute to the overall tracking task. We also demonstrate the real-time performance of our approach on resource-constrained hardware such as NVIDIA Jetson TX2. Finally, our learned policy can also be viewed as a reactive planner for navigation in highly cluttered environments.
△ Less
Submitted 20 June, 2024;
originally announced June 2024.
-
Image-to-Joint Inverse Kinematic of a Supportive Continuum Arm Using Deep Learning
Authors:
Shayan Sepahvand,
Guanghui Wang,
Farrokh Janabi-Sharifi
Abstract:
In this work, a deep learning-based technique is used to study the image-to-joint inverse kinematics of a tendon-driven supportive continuum arm. An eye-off-hand configuration is considered by mounting a camera at a fixed pose with respect to the inertial frame attached at the arm base. This camera captures an image for each distinct joint variable at each sampling time to construct the training d…
▽ More
In this work, a deep learning-based technique is used to study the image-to-joint inverse kinematics of a tendon-driven supportive continuum arm. An eye-off-hand configuration is considered by mounting a camera at a fixed pose with respect to the inertial frame attached at the arm base. This camera captures an image for each distinct joint variable at each sampling time to construct the training dataset. This dataset is then employed to adapt a feed-forward deep convolutional neural network, namely the modified VGG-16 model, to estimate the joint variable. One thousand images are recorded to train the deep network, and transfer learning and fine-tuning techniques are applied to the modified VGG-16 to further improve the training. Finally, training is also completed with a larger dataset of images that are affected by various types of noises, changes in illumination, and partial occlusion. The main contribution of this research is the development of an image-to-joint network that can estimate the joint variable given an image of the arm, even if the image is not captured in an ideal condition. The key benefits of this research are twofold: 1) image-to-joint mapping can offer a real-time alternative to computationally complex inverse kinematic mapping through analytical models; and 2) the proposed technique can provide robustness against noise, occlusion, and changes in illumination. The dataset is publicly available on Kaggle.
△ Less
Submitted 26 June, 2024; v1 submitted 30 May, 2024;
originally announced May 2024.
-
Deep Direct Visual Servoing of Tendon-Driven Continuum Robots
Authors:
Ibrahim Abdulhafiz,
Ali A. Nazari,
Taha Abbasi-Hashemi,
Amir Jalali,
Kourosh Zareinia,
Sajad Saeedi,
Farrokh Janabi-Sharifi
Abstract:
Vision-based control provides a significant potential for the end-point positioning of continuum robots under physical sensing limitations. Traditional visual servoing requires feature extraction and tracking followed by full or partial pose estimation, limiting the controller's efficiency. We hypothesize that employing deep learning models and implementing direct visual servoing can effectively r…
▽ More
Vision-based control provides a significant potential for the end-point positioning of continuum robots under physical sensing limitations. Traditional visual servoing requires feature extraction and tracking followed by full or partial pose estimation, limiting the controller's efficiency. We hypothesize that employing deep learning models and implementing direct visual servoing can effectively resolve the issue by eliminating such intermediate steps, enabling control of a continuum robot without requiring an exact system model. This paper presents the control of a single-section tendon-driven continuum robot using a modified VGG-16 deep learning network and an eye-in-hand direct visual servoing approach. The proposed algorithm is first developed in Blender software using only one input image of the target and then implemented on a real robot. The convergence and accuracy of the results in normal, shadowed, and occluded scenes demonstrate the effectiveness and robustness of the proposed controller.
△ Less
Submitted 25 March, 2022; v1 submitted 3 November, 2021;
originally announced November 2021.
-
Path Planning and Controlled Crash Landing of a Quadcopter in case of a Rotor Failure
Authors:
Mojtaba Hedayatpour,
Mehran Mehrandezh,
Farrokh Janabi-Sharifi
Abstract:
This paper presents a framework for controlled emergency landing of a quadcopter, experiencing a rotor failure, away from sensitive areas. A complete mathematical model capturing the dynamics of the system is presented that takes the asymmetrical aerodynamic load on the propellers into account. An equilibrium state of the system is calculated around which a linear time-invariant control strategy i…
▽ More
This paper presents a framework for controlled emergency landing of a quadcopter, experiencing a rotor failure, away from sensitive areas. A complete mathematical model capturing the dynamics of the system is presented that takes the asymmetrical aerodynamic load on the propellers into account. An equilibrium state of the system is calculated around which a linear time-invariant control strategy is developed to stabilize the system. By utilizing the proposed model, a specific configuration for a quadcopter is introduced that leads to the minimum power consumption during a yaw-rate-resolved hovering after a rotor failure. Furthermore, given a 3D representation of the environment, an optimal flight trajectory towards a safe crash landing spot, while avoiding collision with obstacles, is developed using an RRT* approach. The cost function for determining the best landing spot consists of: (i) finding the safest landing spot with the largest clearance from the obstacles; and (ii) finding the most energy-efficient trajectory towards the landing spot. The performance of the proposed framework is tested via simulations.
△ Less
Submitted 25 September, 2018;
originally announced September 2018.
-
Optimal-power Configurations for Hover Solutions in Mono-spinners
Authors:
Mojtaba Hedayatpour,
Mehran Mehrandezh,
Farrokh Janabi-Sharifi
Abstract:
Rotary-wing flying machines draw attention within the UAV community for their in-place hovering capability, and recently, holonomic motion over fixed-wings. In this paper, we investigate about the power-optimality in a mono-spinner, i.e., a class of rotary-wing UAVs with one rotor only, whose main body has a streamlined shape for producing additional lift when counter-spinning the rotor. We provid…
▽ More
Rotary-wing flying machines draw attention within the UAV community for their in-place hovering capability, and recently, holonomic motion over fixed-wings. In this paper, we investigate about the power-optimality in a mono-spinner, i.e., a class of rotary-wing UAVs with one rotor only, whose main body has a streamlined shape for producing additional lift when counter-spinning the rotor. We provide a detailed dynamic model of our mono-spinner. Two configurations are studied: (1) a symmetric configuration, in which the rotor is aligned with the fuselage's COM, and (2) an asymmetric configuration, in which the rotor is located with an offset from the fuselage's COM. While the former can generate an in-place hovering flight condition, the latter can achieve trajectory tracking in 3D space by resolving the yaw and precession rates. Furthermore, it is shown that by introducing a tilting angle between the rotor and the fuselage, within the asymmetric design, one can further minimize the power consumption without compromising the overall stability. It is shown that an energy optimal solution can be achieved through the proper aerodynamic design of the mono-spinner for the first time.
△ Less
Submitted 29 August, 2020; v1 submitted 22 September, 2017;
originally announced September 2017.
-
A Unified Approach to Configuration-based Dynamic Analysis of Quadcopters for Optimal Stability
Authors:
Mojtaba Hedayatpour,
Mehran Mehrandezh,
Farrokh Janabi-Sharifi
Abstract:
A special type of rotary-wing Unmanned Aerial Vehicles (UAV), called Quadcopter have prevailed to the civilian use for the past decade. They have gained significant amount of attention within the UAV community for their redundancy and ease of control, despite the fact that they fall under an under-actuated system category. They come in a variety of configurations. The "+" and "x" configurations we…
▽ More
A special type of rotary-wing Unmanned Aerial Vehicles (UAV), called Quadcopter have prevailed to the civilian use for the past decade. They have gained significant amount of attention within the UAV community for their redundancy and ease of control, despite the fact that they fall under an under-actuated system category. They come in a variety of configurations. The "+" and "x" configurations were introduced first. Literature pertinent to these two configurations is vast. However, in this paper, we define 6 additional possible configurations for a Quadcopter that can be built under either "+" or "x" setup. These configurations can be achieved by changing the angle that the axis of rotation for rotors make with the main body, i.e., fuselage. This would also change the location of the COM with respect to the propellers which can add to the overall stability. A comprehensive dynamic model for all these configurations is developed for the first time. The overall stability for these configurations are addressed. In particular, it is shown that one configuration can lead to the most statically-stable platform by adopting damping motion in Roll/Pitch/Yaw, which is described for the first time to the best of our knowledge.
△ Less
Submitted 22 September, 2017;
originally announced September 2017.
