Showing 1–29 of 29 results for author: de Wagter, C

Depth Transfer: Learning to See Like a Simulator for Real-World Drone Navigation

Authors: Hang Yu, Christophe De Wagter, Guido C. H. E de Croon

Abstract: Sim-to-real transfer is a fundamental challenge in robot reinforcement learning. Discrepancies between simulation and reality can significantly impair policy performance, especially if it receives high-dimensional inputs such as dense depth estimates from vision. We propose a novel depth transfer method based on domain adaptation to bridge the visual gap between simulated and real-world depth data… ▽ More Sim-to-real transfer is a fundamental challenge in robot reinforcement learning. Discrepancies between simulation and reality can significantly impair policy performance, especially if it receives high-dimensional inputs such as dense depth estimates from vision. We propose a novel depth transfer method based on domain adaptation to bridge the visual gap between simulated and real-world depth data. A Variational Autoencoder (VAE) is first trained to encode ground-truth depth images from simulation into a latent space, which serves as input to a reinforcement learning (RL) policy. During deployment, the encoder is refined to align stereo depth images with this latent space, enabling direct policy transfer without fine-tuning. We apply our method to the task of autonomous drone navigation through cluttered environments. Experiments in IsaacGym show that our method nearly doubles the obstacle avoidance success rate when switching from ground-truth to stereo depth input. Furthermore, we demonstrate successful transfer to the photo-realistic simulator AvoidBench using only IsaacGym-generated stereo data, achieving superior performance compared to state-of-the-art baselines. Real-world evaluations in both indoor and outdoor environments confirm the effectiveness of our approach, enabling robust and generalizable depth-based navigation across diverse domains. △ Less

Submitted 18 May, 2025; originally announced May 2025.

Self-Supervised Monocular Visual Drone Model Identification through Improved Occlusion Handling

Authors: Stavrow A. Bahnam, Christophe De Wagter, Guido C. H. E. de Croon

Abstract: Ego-motion estimation is vital for drones when flying in GPS-denied environments. Vision-based methods struggle when flight speed increases and close-by objects lead to difficult visual conditions with considerable motion blur and large occlusions. To tackle this, vision is typically complemented by state estimation filters that combine a drone model with inertial measurements. However, these dron… ▽ More Ego-motion estimation is vital for drones when flying in GPS-denied environments. Vision-based methods struggle when flight speed increases and close-by objects lead to difficult visual conditions with considerable motion blur and large occlusions. To tackle this, vision is typically complemented by state estimation filters that combine a drone model with inertial measurements. However, these drone models are currently learned in a supervised manner with ground-truth data from external motion capture systems, limiting scalability to different environments and drones. In this work, we propose a self-supervised learning scheme to train a neural-network-based drone model using only onboard monocular video and flight controller data (IMU and motor feedback). We achieve this by first training a self-supervised relative pose estimation model, which then serves as a teacher for the drone model. To allow this to work at high speed close to obstacles, we propose an improved occlusion handling method for training self-supervised pose estimation models. Due to this method, the root mean squared error of resulting odometry estimates is reduced by an average of 15%. Moreover, the student neural drone model can be successfully obtained from the onboard data. It even becomes more accurate at higher speeds compared to its teacher, the self-supervised vision-based model. We demonstrate the value of the neural drone model by integrating it into a traditional filter-based VIO system (ROVIO), resulting in superior odometry accuracy on aggressive 3D racing trajectories near obstacles. Self-supervised learning of ego-motion estimation represents a significant step toward bridging the gap between flying in controlled, expensive lab environments and real-world drone applications. The fusion of vision and drone models will enable higher-speed flight and improve state estimation, on any drone in any environment. △ Less

Submitted 30 April, 2025; originally announced April 2025.

One Net to Rule Them All: Domain Randomization in Quadcopter Racing Across Different Platforms

Authors: Robin Ferede, Till Blaha, Erin Lucassen, Christophe De Wagter, Guido C. H. E. de Croon

Abstract: In high-speed quadcopter racing, finding a single controller that works well across different platforms remains challenging. This work presents the first neural network controller for drone racing that generalizes across physically distinct quadcopters. We demonstrate that a single network, trained with domain randomization, can robustly control various types of quadcopters. The network relies sol… ▽ More In high-speed quadcopter racing, finding a single controller that works well across different platforms remains challenging. This work presents the first neural network controller for drone racing that generalizes across physically distinct quadcopters. We demonstrate that a single network, trained with domain randomization, can robustly control various types of quadcopters. The network relies solely on the current state to directly compute motor commands. The effectiveness of this generalized controller is validated through real-world tests on two substantially different crafts (3-inch and 5-inch race quadcopters). We further compare the performance of this generalized controller with controllers specifically trained for the 3-inch and 5-inch drone, using their identified model parameters with varying levels of domain randomization (0%, 10%, 20%, 30%). While the generalized controller shows slightly slower speeds compared to the fine-tuned models, it excels in adaptability across different platforms. Our results show that no randomization fails sim-to-real transfer while increasing randomization improves robustness but reduces speed. Despite this trade-off, our findings highlight the potential of domain randomization for generalizing controllers, paving the way for universal AI controllers that can adapt to any platform. △ Less

Submitted 30 April, 2025; originally announced April 2025.

Multi-objective Evolution of Drone Morphology

Authors: Elijah H. W. Ang, Christophe De Wagter, Guido C. H. E. de Croon

Abstract: The design of multicopter drones has remained almost the same since its inception. While conventional designs, such as the quadcopter, work well in many cases, they may not be optimal in specific environments or missions. This paper revisits rotary drone design by exploring which body morphologies are optimal for different objectives and constraints. Specifically, an evolutionary algorithm is used… ▽ More The design of multicopter drones has remained almost the same since its inception. While conventional designs, such as the quadcopter, work well in many cases, they may not be optimal in specific environments or missions. This paper revisits rotary drone design by exploring which body morphologies are optimal for different objectives and constraints. Specifically, an evolutionary algorithm is used to produce optimal drone morphologies for three objectives: (1) high thrust-to-weight ratio, (2) high maneuverability, and (3) small size. To generate a range of optimal drones with performance trade-offs between them, the non-dominated sorting genetic algorithm II, or NSGA-II is used. A randomly sampled population of 600 is evolved over 2000 generations. The NSGA-II algorithm evolved drone bodies that outperform a standard 5-inch 220 mm wheelbase quadcopter in at least one of the three objectives. The three extrema in the Pareto front show improvement of 487.8%, 23.5% and 4.8% in maneuverability, thrust-to-weight ratio and size, respectively. The improvement in maneuverability can be attributed to the tilt angles of the propellers, while the increase in thrust-to-weight ratio is primarily due to the higher number of propellers. The quadcopter is located on the Pareto front for the three objectives. However, our results also show that other designs can be better depending on the objectives. △ Less

Submitted 3 February, 2025; originally announced February 2025.

Comments: 7 pages, 8 figures

Neuromorphic Attitude Estimation and Control

Authors: Stein Stroobants, Christophe de Wagter, Guido C. H. E. De Croon

Abstract: The real-world application of small drones is mostly hampered by energy limitations. Neuromorphic computing promises extremely energy-efficient AI for autonomous flight but is still challenging to train and deploy on real robots. To reap the maximal benefits from neuromorphic computing, it is necessary to perform all autonomy functions end-to-end on a single neuromorphic chip, from low-level attit… ▽ More The real-world application of small drones is mostly hampered by energy limitations. Neuromorphic computing promises extremely energy-efficient AI for autonomous flight but is still challenging to train and deploy on real robots. To reap the maximal benefits from neuromorphic computing, it is necessary to perform all autonomy functions end-to-end on a single neuromorphic chip, from low-level attitude control to high-level navigation. This research presents the first neuromorphic control system using a spiking neural network (SNN) to effectively map a drone's raw sensory input directly to motor commands. We apply this method to low-level attitude estimation and control for a quadrotor, deploying the SNN on a tiny Crazyflie. We propose a modular SNN, separately training and then merging estimation and control sub-networks. The SNN is trained with imitation learning, using a flight dataset of sensory-motor pairs. Post-training, the network is deployed on the Crazyflie, issuing control commands from sensor inputs at 500Hz. Furthermore, for the training procedure we augmented training data by flying a controller with additional excitation and time-shifting the target data to enhance the predictive capabilities of the SNN. On the real drone, the perception-to-control SNN tracks attitude commands with an average error of 3.0 degrees, compared to 2.7 degrees for the regular flight stack. We also show the benefits of the proposed learning modifications for reducing the average tracking error and reducing oscillations. Our work shows the feasibility of performing neuromorphic end-to-end control, laying the basis for highly energy-efficient and low-latency neuromorphic autopilots. △ Less

Submitted 21 March, 2025; v1 submitted 21 November, 2024; originally announced November 2024.

MAVRL: Learn to Fly in Cluttered Environments with Varying Speed

Authors: Hang Yu, Christophe De Wagter, Guido C. H. E de Croon

Abstract: Many existing obstacle avoidance algorithms overlook the crucial balance between safety and agility, especially in environments of varying complexity. In our study, we introduce an obstacle avoidance pipeline based on reinforcement learning. This pipeline enables drones to adapt their flying speed according to the environmental complexity. Moreover, to improve the obstacle avoidance performance in… ▽ More Many existing obstacle avoidance algorithms overlook the crucial balance between safety and agility, especially in environments of varying complexity. In our study, we introduce an obstacle avoidance pipeline based on reinforcement learning. This pipeline enables drones to adapt their flying speed according to the environmental complexity. Moreover, to improve the obstacle avoidance performance in cluttered environments, we propose a novel latent space. The latent space in this representation is explicitly trained to retain memory of previous depth map observations. Our findings confirm that varying speed leads to a superior balance of success rate and agility in cluttered environments. Additionally, our memory-augmented latent representation outperforms the latent representation commonly used in reinforcement learning. Finally, after minimal fine-tuning, we successfully deployed our network on a real drone for enhanced obstacle avoidance. △ Less

Submitted 13 February, 2024; originally announced February 2024.

Comments: 8 pages

End-to-end Reinforcement Learning for Time-Optimal Quadcopter Flight

Authors: Robin Ferede, Christophe De Wagter, Dario Izzo, Guido C. H. E. de Croon

Abstract: Aggressive time-optimal control of quadcopters poses a significant challenge in the field of robotics. The state-of-the-art approach leverages reinforcement learning (RL) to train optimal neural policies. However, a critical hurdle is the sim-to-real gap, often addressed by employing a robust inner loop controller -an abstraction that, in theory, constrains the optimality of the trained controller… ▽ More Aggressive time-optimal control of quadcopters poses a significant challenge in the field of robotics. The state-of-the-art approach leverages reinforcement learning (RL) to train optimal neural policies. However, a critical hurdle is the sim-to-real gap, often addressed by employing a robust inner loop controller -an abstraction that, in theory, constrains the optimality of the trained controller, necessitating margins to counter potential disturbances. In contrast, our novel approach introduces high-speed quadcopter control using end-to-end RL (E2E) that gives direct motor commands. To bridge the reality gap, we incorporate a learned residual model and an adaptive method that can compensate for modeling errors in thrust and moments. We compare our E2E approach against a state-of-the-art network that commands thrust and body rates to an INDI inner loop controller, both in simulated and real-world flight. E2E showcases a significant 1.39-second advantage in simulation and a 0.17-second edge in real-world testing, highlighting end-to-end reinforcement learning's potential. The performance drop observed from simulation to reality shows potential for further improvement, including refining strategies to address the reality gap or exploring offline reinforcement learning with real flight data. △ Less

Submitted 28 November, 2023; originally announced November 2023.

Comments: 6 pages, 6 figures, 1 table

Optimality Principles in Spacecraft Neural Guidance and Control

Authors: Dario Izzo, Emmanuel Blazquez, Robin Ferede, Sebastien Origer, Christophe De Wagter, Guido C. H. E. de Croon

Abstract: Spacecraft and drones aimed at exploring our solar system are designed to operate in conditions where the smart use of onboard resources is vital to the success or failure of the mission. Sensorimotor actions are thus often derived from high-level, quantifiable, optimality principles assigned to each task, utilizing consolidated tools in optimal control theory. The planned actions are derived on t… ▽ More Spacecraft and drones aimed at exploring our solar system are designed to operate in conditions where the smart use of onboard resources is vital to the success or failure of the mission. Sensorimotor actions are thus often derived from high-level, quantifiable, optimality principles assigned to each task, utilizing consolidated tools in optimal control theory. The planned actions are derived on the ground and transferred onboard where controllers have the task of tracking the uploaded guidance profile. Here we argue that end-to-end neural guidance and control architectures (here called G&CNets) allow transferring onboard the burden of acting upon these optimality principles. In this way, the sensor information is transformed in real time into optimal plans thus increasing the mission autonomy and robustness. We discuss the main results obtained in training such neural architectures in simulation for interplanetary transfers, landings and close proximity operations, highlighting the successful learning of optimality principles by the neural model. We then suggest drone racing as an ideal gym environment to test these architectures on real robotic platforms, thus increasing confidence in their utilization on future space exploration missions. Drone racing shares with spacecraft missions both limited onboard computational capabilities and similar control structures induced from the optimality principle sought, but it also entails different levels of uncertainties and unmodelled effects. Furthermore, the success of G&CNets on extremely resource-restricted drones illustrates their potential to bring real-time optimal control within reach of a wider variety of robotic systems, both in space and on Earth. △ Less

Submitted 22 May, 2023; originally announced May 2023.

Guidance & Control Networks for Time-Optimal Quadcopter Flight

Authors: Sebastien Origer, Christophe De Wagter, Robin Ferede, Guido C. H. E. de Croon, Dario Izzo

Abstract: Reaching fast and autonomous flight requires computationally efficient and robust algorithms. To this end, we train Guidance & Control Networks to approximate optimal control policies ranging from energy-optimal to time-optimal flight. We show that the policies become more difficult to learn the closer we get to the time-optimal 'bang-bang' control profile. We also assess the importance of knowing… ▽ More Reaching fast and autonomous flight requires computationally efficient and robust algorithms. To this end, we train Guidance & Control Networks to approximate optimal control policies ranging from energy-optimal to time-optimal flight. We show that the policies become more difficult to learn the closer we get to the time-optimal 'bang-bang' control profile. We also assess the importance of knowing the maximum angular rotor velocity of the quadcopter and show that over- or underestimating this limit leads to less robust flight. We propose an algorithm to identify the current maximum angular rotor velocity onboard and a network that adapts its policy based on the identified limit. Finally, we extend previous work on Guidance & Control Networks by learning to take consecutive waypoints into account. We fly a 4x3m track in similar lap times as the differential-flatness-based minimum snap benchmark controller while benefiting from the flexibility that Guidance & Control Networks offer. △ Less

Submitted 4 May, 2023; originally announced May 2023.

End-to-end Neural Network Based Quadcopter control

Authors: Robin Ferede, Guido C. H. E. de Croon, Christophe De Wagter, Dario Izzo

Abstract: Developing optimal controllers for aggressive high-speed quadcopter flight poses significant challenges in robotics. Recent trends in the field involve utilizing neural network controllers trained through supervised or reinforcement learning. However, the sim-to-real transfer introduces a reality gap, requiring the use of robust inner loop controllers during real flights, which limits the network'… ▽ More Developing optimal controllers for aggressive high-speed quadcopter flight poses significant challenges in robotics. Recent trends in the field involve utilizing neural network controllers trained through supervised or reinforcement learning. However, the sim-to-real transfer introduces a reality gap, requiring the use of robust inner loop controllers during real flights, which limits the network's control authority and flight performance. In this paper, we investigate for the first time, an end-to-end neural network controller, addressing the reality gap issue without being restricted by an inner-loop controller. The networks, referred to as G\&CNets, are trained to learn an energy-optimal policy mapping the quadcopter's state to rpm commands using an optimal trajectory dataset. In hover-to-hover flights, we identified the unmodeled moments as a significant contributor to the reality gap. To mitigate this, we propose an adaptive control strategy that works by learning from optimal trajectories of a system affected by constant external pitch, roll and yaw moments. In real test flights, this model mismatch is estimated onboard and fed to the network to obtain the optimal rpm command. We demonstrate the effectiveness of our method by performing energy-optimal hover-to-hover flights with and without moment feedback. Finally, we compare the adaptive controller to a state-of-the-art differential-flatness-based controller in a consecutive waypoint flight and demonstrate the advantages of our method in terms of energy optimality and robustness. △ Less

Submitted 22 June, 2023; v1 submitted 26 April, 2023; originally announced April 2023.

Comments: 11 pages, 16 figures

Neuromorphic Control using Input-Weighted Threshold Adaptation

Authors: Stein Stroobants, Christophe De Wagter, Guido C. H. E. de Croon

Abstract: Neuromorphic processing promises high energy efficiency and rapid response rates, making it an ideal candidate for achieving autonomous flight of resource-constrained robots. It will be especially beneficial for complex neural networks as are involved in high-level visual perception. However, fully neuromorphic solutions will also need to tackle low-level control tasks. Remarkably, it is currently… ▽ More Neuromorphic processing promises high energy efficiency and rapid response rates, making it an ideal candidate for achieving autonomous flight of resource-constrained robots. It will be especially beneficial for complex neural networks as are involved in high-level visual perception. However, fully neuromorphic solutions will also need to tackle low-level control tasks. Remarkably, it is currently still challenging to replicate even basic low-level controllers such as proportional-integral-derivative (PID) controllers. Specifically, it is difficult to incorporate the integral and derivative parts. To address this problem, we propose a neuromorphic controller that incorporates proportional, integral, and derivative pathways during learning. Our approach includes a novel input threshold adaptation mechanism for the integral pathway. This Input-Weighted Threshold Adaptation (IWTA) introduces an additional weight per synaptic connection, which is used to adapt the threshold of the post-synaptic neuron. We tackle the derivative term by employing neurons with different time constants. We first analyze the performance and limits of the proposed mechanisms and then put our controller to the test by implementing it on a microcontroller connected to the open-source tiny Crazyflie quadrotor, replacing the innermost rate controller. We demonstrate the stability of our bio-inspired algorithm with flights in the presence of disturbances. The current work represents a substantial step towards controlling highly dynamic systems with neuromorphic algorithms, thus advancing neuromorphic processing and robotics. In addition, integration is an important part of any temporal task, so the proposed Input-Weighted Threshold Adaptation (IWTA) mechanism may have implications well beyond control tasks. △ Less

Submitted 18 April, 2023; originally announced April 2023.

Taming Contrast Maximization for Learning Sequential, Low-latency, Event-based Optical Flow

Authors: Federico Paredes-Vallés, Kirk Y. W. Scheper, Christophe De Wagter, Guido C. H. E. de Croon

Abstract: Event cameras have recently gained significant traction since they open up new avenues for low-latency and low-power solutions to complex computer vision problems. To unlock these solutions, it is necessary to develop algorithms that can leverage the unique nature of event data. However, the current state-of-the-art is still highly influenced by the frame-based literature, and usually fails to del… ▽ More Event cameras have recently gained significant traction since they open up new avenues for low-latency and low-power solutions to complex computer vision problems. To unlock these solutions, it is necessary to develop algorithms that can leverage the unique nature of event data. However, the current state-of-the-art is still highly influenced by the frame-based literature, and usually fails to deliver on these promises. In this work, we take this into consideration and propose a novel self-supervised learning pipeline for the sequential estimation of event-based optical flow that allows for the scaling of the models to high inference frequencies. At its core, we have a continuously-running stateful neural model that is trained using a novel formulation of contrast maximization that makes it robust to nonlinearities and varying statistics in the input events. Results across multiple datasets confirm the effectiveness of our method, which establishes a new state of the art in terms of accuracy for approaches trained or optimized without ground truth. △ Less

Submitted 27 September, 2023; v1 submitted 9 March, 2023; originally announced March 2023.

Comments: ICCV 2023. 15 pages, 12 figures, 7 tables

AvoidBench: A high-fidelity vision-based obstacle avoidance benchmarking suite for multi-rotors

Authors: Hang Yu, Guido C. H. E de Croon, Christophe De Wagter

Abstract: Obstacle avoidance is an essential topic in the field of autonomous drone research. When choosing an avoidance algorithm, many different options are available, each with their advantages and disadvantages. As there is currently no consensus on testing methods, it is quite challenging to compare the performance between algorithms. In this paper, we propose AvoidBench, a benchmarking suite which can… ▽ More Obstacle avoidance is an essential topic in the field of autonomous drone research. When choosing an avoidance algorithm, many different options are available, each with their advantages and disadvantages. As there is currently no consensus on testing methods, it is quite challenging to compare the performance between algorithms. In this paper, we propose AvoidBench, a benchmarking suite which can evaluate the performance of vision-based obstacle avoidance algorithms by subjecting them to a series of tasks. Thanks to the high fidelity of multi-rotors dynamics from RotorS and virtual scenes of Unity3D, AvoidBench can realize realistic simulated flight experiments. Compared to current drone simulators, we propose and implement both performance and environment metrics to reveal the suitability of obstacle avoidance algorithms for environments of different complexity. To illustrate AvoidBench's usage, we compare three algorithms: Ego-planner, MBPlanner, and Agile-autonomy. The trends observed are validated with real-world obstacle avoidance experiments. △ Less

Submitted 18 January, 2023; originally announced January 2023.

An Experimental Study of Wind Resistance and Power Consumption in MAVs with a Low-Speed Multi-Fan Wind System

Authors: Diana A. Olejnik, Sunyi Wang, Julien Dupeyroux, Stein Stroobants, Matěj Karásek, Christophe De Wagter, Guido de Croon

Abstract: This paper discusses a low-cost, open-source and open-hardware design and performance evaluation of a low-speed, multi-fan wind system dedicated to micro air vehicle (MAV) testing. In addition, a set of experiments with a flapping wing MAV and rotorcraft is presented, demonstrating the capabilities of the system and the properties of these different types of drones in response to various types of… ▽ More This paper discusses a low-cost, open-source and open-hardware design and performance evaluation of a low-speed, multi-fan wind system dedicated to micro air vehicle (MAV) testing. In addition, a set of experiments with a flapping wing MAV and rotorcraft is presented, demonstrating the capabilities of the system and the properties of these different types of drones in response to various types of wind. We performed two sets of experiments where a MAV is flying into the wake of the fan system, gathering data about states, battery voltage and current. Firstly, we focus on steady wind conditions with wind speeds ranging from 0.5 m/s to 3.4 m/s. During the second set of experiments, we introduce wind gusts, by periodically modulating the wind speed from 1.3 m/s to 3.4 m/s with wind gust oscillations of 0.5 Hz, 0.25 Hz and 0.125 Hz. The "Flapper" flapping wing MAV requires much larger pitch angles to counter wind than the "CrazyFlie" quadrotor. This is due to the Flapper's larger wing surface. In forward flight, its wings do provide extra lift, considerably reducing the power consumption. In contrast, the CrazyFlie's power consumption stays more constant for different wind speeds. The experiments with the varying wind show a quicker gust response by the CrazyFlie compared with the Flapper drone, but both their responses could be further improved. We expect that the proposed wind gust system will provide a useful tool to the community to achieve such improvements. △ Less

Submitted 14 February, 2022; originally announced February 2022.

The Artificial Intelligence behind the winning entry to the 2019 AI Robotic Racing Competition

Authors: Christophe De Wagter, Federico Paredes-Vallés, Nilay Sheth, Guido de Croon

Abstract: Robotics is the next frontier in the progress of Artificial Intelligence (AI), as the real world in which robots operate represents an enormous, complex, continuous state space with inherent real-time requirements. One extreme challenge in robotics is currently formed by autonomous drone racing. Human drone racers can fly through complex tracks at speeds of up to 190 km/h. Achieving similar speeds… ▽ More Robotics is the next frontier in the progress of Artificial Intelligence (AI), as the real world in which robots operate represents an enormous, complex, continuous state space with inherent real-time requirements. One extreme challenge in robotics is currently formed by autonomous drone racing. Human drone racers can fly through complex tracks at speeds of up to 190 km/h. Achieving similar speeds with autonomous drones signifies tackling fundamental problems in AI under extreme restrictions in terms of resources. In this article, we present the winning solution of the first AI Robotic Racing (AIRR) Circuit, a competition consisting of four races in which all participating teams used the same drone, to which they had limited access. The core of our approach is inspired by how human pilots combine noisy observations of the race gates with their mental model of the drone's dynamics to achieve fast control. Our approach has a large focus on gate detection with an efficient deep neural segmentation network and active vision. Further, we make contributions to robust state estimation and risk-based control. This allowed us to reach speeds of ~9.2m/s in the last race, unrivaled by previous autonomous drone race competitions. Although our solution was the fastest and most robust, it still lost against one of the best human pilots, Gab707. The presented approach indicates a promising direction to close the gap with human drone pilots, forming an important step in bringing AI to the real world. △ Less

Submitted 30 September, 2021; originally announced September 2021.

Self-supervised Monocular Multi-robot Relative Localization with Efficient Deep Neural Networks

Authors: Shushuai Li, Christophe De Wagter, Guido C. H. E. de Croon

Abstract: Relative localization is an important ability for multiple robots to perform cooperative tasks in GPS-denied environment. This paper presents a novel autonomous positioning framework for monocular relative localization of multiple tiny flying robots. This approach does not require any groundtruth data from external systems or manual labelling. Instead, the proposed framework is able to label real-… ▽ More Relative localization is an important ability for multiple robots to perform cooperative tasks in GPS-denied environment. This paper presents a novel autonomous positioning framework for monocular relative localization of multiple tiny flying robots. This approach does not require any groundtruth data from external systems or manual labelling. Instead, the proposed framework is able to label real-world images with 3D relative positions between robots based on another onboard relative estimation technology, using ultra-wide band (UWB). After training in this self-supervised manner, the proposed deep neural network (DNN) can predict relative positions of peer robots by purely using a monocular camera. This deep learning-based visual relative localization is scalable, distributed and autonomous. We also built an open-source and light-weight simulation pipeline by using Blender for 3D rendering, which allows synthetic image generation of other robots, and generalized training of the neural network. The proposed localization framework is tested on two real-world Crazyflie2 quadrotors by running the DNN on the onboard AIdeck (a tiny AI chip and monocular camera). All results demonstrate the effectiveness of the self-supervised multi-robot localization method. △ Less

Submitted 22 September, 2021; v1 submitted 26 May, 2021; originally announced May 2021.

Comments: 6+1 pages, submitted to ICRA 2022

The NederDrone: A hybrid lift, hybrid energy hydrogen UAV

Authors: Christophe De Wagter, Bart Remes, Ewoud Smeur, Freek van Tienen, Rick Ruijsink, Kevin van Hecke, Erik van der Horst

Abstract: A lot of UAV applications require vertical take-off and landing (VTOL) combined with very long-range or endurance. Transitioning UAVs have been proposed to combine the VTOL capabilities of helicopters with the efficient long-range flight properties of fixed-wing aircraft. But energy is still a bottleneck for many electric long endurance applications. While solar power technology and battery techno… ▽ More A lot of UAV applications require vertical take-off and landing (VTOL) combined with very long-range or endurance. Transitioning UAVs have been proposed to combine the VTOL capabilities of helicopters with the efficient long-range flight properties of fixed-wing aircraft. But energy is still a bottleneck for many electric long endurance applications. While solar power technology and battery technology have improved a lot, in rougher conditions they still respectively lack the power or total amount of energy required for many real-world situations. In this paper, we introduce the NederDrone, a hybrid lift, hybrid energy hydrogen-powered UAV which can perform vertical take-off and landings using 12 propellers while flying efficiently in forward flight thanks to its fixed wings. The energy is supplied from a mix of hydrogen-driven fuel-cells to store large amounts of energy and battery power for high power situations. The hydrogen is stored in a pressurized cylinder around which the UAV is optimized. This paper analyses the selection of the concept, the implemented safety elements, the electronics and flight control and shows flight data including a 3h38 flight at sea, starting and landing on a small moving ship. △ Less

Submitted 8 November, 2020; originally announced November 2020.

Fault Tolerant Control of Multirotor UAV for Piloted Outdoor Flights

Authors: A. Narasimhan, C. C. de Visser, C. de Wagter, M. Rischmueller

Abstract: This paper aims to develop a Fault Tolerant Control (FTC) architecture, for the case of a damaged actuator for a multirotor UAV that can be applied across multirotor platforms based on their Attainable Virtual Control Set (AVCS). The research is aimed to study the AVCS and identify the parameters that limit the controllability of multirotor UAV post an actuator failure. Based on the study of contr… ▽ More This paper aims to develop a Fault Tolerant Control (FTC) architecture, for the case of a damaged actuator for a multirotor UAV that can be applied across multirotor platforms based on their Attainable Virtual Control Set (AVCS). The research is aimed to study the AVCS and identify the parameters that limit the controllability of multirotor UAV post an actuator failure. Based on the study of controllability, the requirements for a FTC is laid out. The implemented control solution will be tested on a quadrotor, Intel Shooting Star UAV platform in indoor and outdoor flights using only the onboard sensors. The attitude control solution is implemented with reduced attitude control, and the control allocation is performed with pseudo-inverse based model inversion with sequential desaturation to ensure tilt priority. The model is identified with an offline Ordinary Least Squares routine and subsequently updated with the Recursive Least Squares method. An offline calibration routine is implemented to correct IMU offset distance from the centre of rotation to correct for accelerometer bias caused by the high-speed spin after failure in a quadrotor. △ Less

Submitted 1 November, 2020; originally announced November 2020.

Comments: 30 pages with appendix

MSC Class: 70E60

Onboard Ranging-based Relative Localization and Stability for Lightweight Aerial Swarms

Authors: Shushuai Li, Feng Shan, Jiangpeng Liu, Mario Coppola, Christophe de Wagter, Guido C. H. E. de Croon

Abstract: Lightweight aerial swarms have potential applications in scenarios where larger drones fail to operate efficiently. The primary foundation for lightweight aerial swarms is efficient relative localization, which enables cooperation and collision avoidance. Computing the real-time position is challenging due to extreme resource constraints. This paper presents an autonomous relative localization tec… ▽ More Lightweight aerial swarms have potential applications in scenarios where larger drones fail to operate efficiently. The primary foundation for lightweight aerial swarms is efficient relative localization, which enables cooperation and collision avoidance. Computing the real-time position is challenging due to extreme resource constraints. This paper presents an autonomous relative localization technique for lightweight aerial swarms without infrastructure by fusing ultra-wideband wireless distance measurements and the shared state information (e.g., velocity, yaw rate, height) from neighbors. This is the first fully autonomous, tiny, fast, and accurate relative localization scheme implemented on a team of 13 lightweight (33 grams) and resource-constrained (168MHz MCU with 192 KB memory) aerial vehicles. The proposed resource-constrained swarm ranging protocol is scalable, and a surprising theoretical result is discovered: the unobservability poses no issues because the state drift leads to control actions that make the state observable again. By experiment, less than 0.2m position error is achieved at the frequency of 16Hz for as many as 13 drones. The code is open-sourced, and the proposed technique is relevant not only for tiny drones but can be readily applied to many other resource-restricted robots. Video and code can be found at \textnormal{\url{https://shushuai3.github.io/autonomous-swarm/}}. △ Less

Submitted 28 September, 2024; v1 submitted 12 March, 2020; originally announced March 2020.

Comments: Project link: https://shushuai3.github.io/autonomous-swarm/

Aggressive Online Control of a Quadrotor via Deep Network Representations of Optimality Principles

Authors: Shuo Li, Ekin Ozturk, Christophe De Wagter, Guido C. H. E. de Croon, Dario Izzo

Abstract: Optimal control holds great potential to improve a variety of robotic applications. The application of optimal control on-board limited platforms has been severely hindered by the large computational requirements of current state of the art implementations. In this work, we make use of a deep neural network to directly map the robot states to control actions. The network is trained offline to imit… ▽ More Optimal control holds great potential to improve a variety of robotic applications. The application of optimal control on-board limited platforms has been severely hindered by the large computational requirements of current state of the art implementations. In this work, we make use of a deep neural network to directly map the robot states to control actions. The network is trained offline to imitate the optimal control computed by a time consuming direct nonlinear method. A mixture of time optimality and power optimality is considered with a continuation parameter used to select the predominance of each objective. We apply our networks (termed G\&CNets) to aggressive quadrotor control, first in simulation and then in the real world. We give insight into the factors that influence the `reality gap' between the quadrotor model used by the offline optimal control method and the real quadrotor. Furthermore, we explain how we set up the model and the control structure on-board of the real quadrotor to successfully close this gap and perform time-optimal maneuvers in the real world. Finally, G\&CNet's performance is compared to state-of-the-art differential-flatness-based optimal control methods. We show, in the experiments, that G\&CNets lead to significantly faster trajectory execution due to, in part, the less restrictive nature of the allowed state-to-input mappings. △ Less

Submitted 15 December, 2019; originally announced December 2019.

Comments: 6 pages, 8 figures

Visual Model-predictive Localization for Computationally Efficient Autonomous Racing of a 72-gram Drone

Authors: Shuo Li, Erik van der Horst, Philipp Duernay, Christophe De Wagter, Guido C. H. E. de Croon

Abstract: Drone racing is becoming a popular e-sport all over the world, and beating the best human drone race pilots has quickly become a new major challenge for artificial intelligence and robotics. In this paper, we propose a strategy for autonomous drone racing which is computationally more efficient than navigation methods like visual inertial odometry and simultaneous localization and mapping. This fa… ▽ More Drone racing is becoming a popular e-sport all over the world, and beating the best human drone race pilots has quickly become a new major challenge for artificial intelligence and robotics. In this paper, we propose a strategy for autonomous drone racing which is computationally more efficient than navigation methods like visual inertial odometry and simultaneous localization and mapping. This fast light-weight vision-based navigation algorithm estimates the position of the drone by fusing race gate detections with model dynamics predictions. Theoretical analysis and simulation results show the clear advantage compared to Kalman filtering when dealing with the relatively low frequency visual updates and occasional large outliers that occur in fast drone racing. Flight tests are performed on a tiny racing quadrotor named "Trashcan", which was equipped with a Jevois smart-camera for a total of 72g. The test track consists of 3 laps around a 4-gate racing track. The gates spaced 4 meters apart and can be displaced from their supposed position. An average speed of 2m/s is achieved while the maximum speed is 2.6m/s. To the best of our knowledge, this flying platform is the smallest autonomous racing drone in the world and is 6 times lighter than the existing lightest autonomous racing drone setup (420g), while still being one of the fastest autonomous racing drones in the world. △ Less

Submitted 15 December, 2019; v1 submitted 24 May, 2019; originally announced May 2019.

Autonomous drone race: A computationally efficient vision-based navigation and control strategy

Authors: S. Li, M. M. O. I. Ozo, C. De Wagter, G. C. H. E. de Croon

Abstract: Drone racing is becoming a popular sport where human pilots have to control their drones to fly at high speed through complex environments and pass a number of gates in a pre-defined sequence. In this paper, we develop an autonomous system for drones to race fully autonomously using only onboard resources. Instead of commonly used visual navigation methods, such as simultaneous localization and ma… ▽ More Drone racing is becoming a popular sport where human pilots have to control their drones to fly at high speed through complex environments and pass a number of gates in a pre-defined sequence. In this paper, we develop an autonomous system for drones to race fully autonomously using only onboard resources. Instead of commonly used visual navigation methods, such as simultaneous localization and mapping and visual inertial odometry, which are computationally expensive for micro aerial vehicles (MAVs), we developed the highly efficient snake gate detection algorithm for visual navigation, which can detect the gate at 20HZ on a Parrot Bebop drone. Then, with the gate detection result, we developed a robust pose estimation algorithm which has better tolerance to detection noise than a state-of-the-art perspective-n-point method. During the race, sometimes the gates are not in the drone's field of view. For this case, a state prediction-based feed-forward control strategy is developed to steer the drone to fly to the next gate. Experiments show that the drone can fly a half-circle with 1.5m radius within 2 seconds with only 30cm error at the end of the circle without any position feedback. Finally, the whole system is tested in a complex environment (a showroom in the faculty of Aerospace Engineering, TU Delft). The result shows that the drone can complete the track of 15 gates with a speed of 1.5m/s which is faster than the speeds exhibited at the 2016 and 2017 IROS autonomous drone races. △ Less

Submitted 15 December, 2019; v1 submitted 16 September, 2018; originally announced September 2018.

MSC Class: I.2.9 ACM Class: I.2.9

Learning what is above and what is below: horizon approach to monocular obstacle detection

Authors: Guido de Croon, Christophe De Wagter

Abstract: A novel approach is proposed for monocular obstacle detection, which relies on self-supervised learning to discriminate everything above the horizon line from everything below. Obstacles on the path of a robot that keeps moving at the same height, will appear both above and under the horizon line. This implies that classifying obstacle pixels will be inherently uncertain. Hence, in the proposed ap… ▽ More A novel approach is proposed for monocular obstacle detection, which relies on self-supervised learning to discriminate everything above the horizon line from everything below. Obstacles on the path of a robot that keeps moving at the same height, will appear both above and under the horizon line. This implies that classifying obstacle pixels will be inherently uncertain. Hence, in the proposed approach the classifier's uncertainty is used for obstacle detection. The (preliminary) results show that this approach can indeed work in different environments. On the well-known KITTI data set, the self-supervised learning scheme clearly segments the road and sky, while application to a flying data set leads to the segmentation of the flight arena's floor. △ Less

Submitted 20 June, 2018; originally announced June 2018.

Design, Control and Visual Navigation of the DelftaCopter

Authors: Christophe De Wagter, Rick Ruijsink, Ewoud Smeur, Kevin van Hecke, Freek van Tienen, Erik van der Horst, Bart Remes

Abstract: To participate in the Outback Medical Express UAV Challenge 2016, a vehicle was designed and tested that can hover precisely, take-off and land vertically, fly fast forward efficiently and use computer vision to locate a person and a suitable landing location. A rotor blade was designed that can deliver sufficient thrust in hover, while still being efficient in fast forward flight. Energy measurem… ▽ More To participate in the Outback Medical Express UAV Challenge 2016, a vehicle was designed and tested that can hover precisely, take-off and land vertically, fly fast forward efficiently and use computer vision to locate a person and a suitable landing location. A rotor blade was designed that can deliver sufficient thrust in hover, while still being efficient in fast forward flight. Energy measurements and windtunnel tests were performed. A rotor-head and corresponding control algorithms were developed to allow transitioning flight with the non-conventional rotor dynamics. Dedicated electronics were designed that meet vehicle needs and regulations to allow safe flight beyond visual line of sight. Vision based search and guidance algorithms were developed and tested. Flight tests and a competition participation illustrate the applicability of the DelftaCopter concept. △ Less

Submitted 3 January, 2017; originally announced January 2017.

First free-flight flow visualisation of a flapping-wing robot

Authors: Matěj Karásek, Mustafa Percin, Torbjørn Cunis, Bas W. van Oudheusden, Christophe De Wagter, Bart D. W. Remes, Guido C. H. E. de Croon

Abstract: Flow visualisations are essential to better understand the unsteady aerodynamics of flapping wing flight. The issues inherent to animal experiments, such as poor controllability and unnatural flapping when tethered, can be avoided by using robotic flyers. Such an approach holds a promise for a more systematic and repeatable methodology for flow visualisation, through a better controlled flight. Su… ▽ More Flow visualisations are essential to better understand the unsteady aerodynamics of flapping wing flight. The issues inherent to animal experiments, such as poor controllability and unnatural flapping when tethered, can be avoided by using robotic flyers. Such an approach holds a promise for a more systematic and repeatable methodology for flow visualisation, through a better controlled flight. Such experiments require high precision position control, however, and until now this was not possible due to the challenging flight dynamics and payload restrictions of flapping wing Micro Air Vehicles (FWMAV). Here, we present a new FWMAV-specific control approach that, by employing an external motion tracking system, achieved autonomous wind tunnel flight with a maximum root-mean-square position error of 28 mm at low speeds (0.8 - 1.2 m/s) and 75 mm at high speeds (2 - 2.4 m/s). This allowed the first free-flight flow visualisation experiments to be conducted with an FWMAV. Time-resolved stereoscopic Particle Image Velocimetry (PIV) was used to reconstruct the 3D flow patterns of the FWMAV wake. A good qualitative match was found in comparison to a tethered configuration at similar conditions, suggesting that the obtained free-flight measurements are reliable and meaningful. △ Less

Submitted 22 December, 2016; originally announced December 2016.

Comments: 30 pages, 12 figures

Efficient Optical flow and Stereo Vision for Velocity Estimation and Obstacle Avoidance on an Autonomous Pocket Drone

Authors: Kimberly McGuire, Guido de Croon, Christophe De Wagter, Karl Tuyls, Hilbert Kappen

Abstract: Miniature Micro Aerial Vehicles (MAV) are very suitable for flying in indoor environments, but autonomous navigation is challenging due to their strict hardware limitations. This paper presents a highly efficient computer vision algorithm called Edge-FS for the determination of velocity and depth. It runs at 20 Hz on a 4 g stereo camera with an embedded STM32F4 microprocessor (168 MHz, 192 kB) and… ▽ More Miniature Micro Aerial Vehicles (MAV) are very suitable for flying in indoor environments, but autonomous navigation is challenging due to their strict hardware limitations. This paper presents a highly efficient computer vision algorithm called Edge-FS for the determination of velocity and depth. It runs at 20 Hz on a 4 g stereo camera with an embedded STM32F4 microprocessor (168 MHz, 192 kB) and uses feature histograms to calculate optical flow and stereo disparity. The stereo-based distance estimates are used to scale the optical flow in order to retrieve the drone's velocity. The velocity and depth measurements are used for fully autonomous flight of a 40 g pocket drone only relying on on-board sensors. The method allows the MAV to control its velocity and avoid obstacles. △ Less

Submitted 14 March, 2017; v1 submitted 20 December, 2016; originally announced December 2016.

Comments: 7 pages, 10 figures, Published at IEEE Robotics and Automation Letters

Journal ref: IEEE Robotics and Automation Letters, 2017, 2, 1070-1076

Obstacle Avoidance Strategy using Onboard Stereo Vision on a Flapping Wing MAV

Authors: Sjoerd Tijmons, Guido de Croon, Bart Remes, Christophe De Wagter, Max Mulder

Abstract: The development of autonomous lightweight MAVs, capable of navigating in unknown indoor environments, is one of the major challenges in robotics. The complexity of this challenge comes from constraints on weight and power consumption of onboard sensing and processing devices. In this paper we propose the "Droplet" strategy, an avoidance strategy based on stereo vision inputs that outperforms react… ▽ More The development of autonomous lightweight MAVs, capable of navigating in unknown indoor environments, is one of the major challenges in robotics. The complexity of this challenge comes from constraints on weight and power consumption of onboard sensing and processing devices. In this paper we propose the "Droplet" strategy, an avoidance strategy based on stereo vision inputs that outperforms reactive avoidance strategies by allowing constant speed maneuvers while being computationally extremely efficient, and which does not need to store previous images or maps. The strategy deals with nonholonomic motion constraints of most fixed and flapping wing platforms, and with the limited field-of-view of stereo camera systems. It guarantees obstacle-free flight in the absence of sensor and motor noise. We first analyze the strategy in simulation, and then show its robustness in real-world conditions by implementing it on a 20-gram flapping wing MAV. △ Less

Submitted 2 January, 2017; v1 submitted 4 April, 2016; originally announced April 2016.

Local Histogram Matching for Efficient Optical Flow Computation Applied to Velocity Estimation on Pocket Drones

Authors: Kimberly McGuire, Guido de Croon, Christophe de Wagter, Bart Remes, Karl Tuyls, Hilbert Kappen

Abstract: Autonomous flight of pocket drones is challenging due to the severe limitations on on-board energy, sensing, and processing power. However, tiny drones have great potential as their small size allows maneuvering through narrow spaces while their small weight provides significant safety advantages. This paper presents a computationally efficient algorithm for determining optical flow, which can be… ▽ More Autonomous flight of pocket drones is challenging due to the severe limitations on on-board energy, sensing, and processing power. However, tiny drones have great potential as their small size allows maneuvering through narrow spaces while their small weight provides significant safety advantages. This paper presents a computationally efficient algorithm for determining optical flow, which can be run on an STM32F4 microprocessor (168 MHz) of a 4 gram stereo-camera. The optical flow algorithm is based on edge histograms. We propose a matching scheme to determine local optical flow. Moreover, the method allows for sub-pixel flow determination based on time horizon adaptation. We demonstrate velocity measurements in flight and use it within a velocity control-loop on a pocket drone. △ Less

Submitted 14 March, 2017; v1 submitted 24 March, 2016; originally announced March 2016.

Comments: 7 pages, 10 figures, Changes: format changed one column to two columns, used url package for links

Journal ref: 2016 IEEE International Conference on Robotics and Automation (ICRA), 3255 - 3260,

Optical-Flow based Self-Supervised Learning of Obstacle Appearance applied to MAV Landing

Authors: H. W. Ho, C. De Wagter, B. D. W. Remes, G. C. H. E. de Croon

Abstract: Monocular optical flow has been widely used to detect obstacles in Micro Air Vehicles (MAVs) during visual navigation. However, this approach requires significant movement, which reduces the efficiency of navigation and may even introduce risks in narrow spaces. In this paper, we introduce a novel setup of self-supervised learning (SSL), in which optical flow cues serve as a scaffold to learn the… ▽ More Monocular optical flow has been widely used to detect obstacles in Micro Air Vehicles (MAVs) during visual navigation. However, this approach requires significant movement, which reduces the efficiency of navigation and may even introduce risks in narrow spaces. In this paper, we introduce a novel setup of self-supervised learning (SSL), in which optical flow cues serve as a scaffold to learn the visual appearance of obstacles in the environment. We apply it to a landing task, in which initially 'surface roughness' is estimated from the optical flow field in order to detect obstacles. Subsequently, a linear regression function is learned that maps appearance features represented by texton distributions to the roughness estimate. After learning, the MAV can detect obstacles by just analyzing a still image. This allows the MAV to search for a landing spot without moving. We first demonstrate this principle to work with offline tests involving images captured from an on-board camera, and then demonstrate the principle in flight. Although surface roughness is a property of the entire flow field in the global image, the appearance learning even allows for the pixel-wise segmentation of obstacles. △ Less

Submitted 17 August, 2017; v1 submitted 4 September, 2015; originally announced September 2015.

Comments: This manuscript is submitted to Robotics and Autonomous Systems