-
CropNav: a Framework for Autonomous Navigation in Real Farms
Authors:
Mateus Valverde Gasparino,
Vitor Akihiro Hisano Higuti,
Arun Narenthiran Sivakumar,
Andres Eduardo Baquero Velasquez,
Marcelo Becker,
Girish Chowdhary
Abstract:
Small robots that can operate under the plant canopy can enable new possibilities in agriculture. However, unlike larger autonomous tractors, autonomous navigation for such under canopy robots remains an open challenge because Global Navigation Satellite System (GNSS) is unreliable under the plant canopy. We present a hybrid navigation system that autonomously switches between different sets of se…
▽ More
Small robots that can operate under the plant canopy can enable new possibilities in agriculture. However, unlike larger autonomous tractors, autonomous navigation for such under canopy robots remains an open challenge because Global Navigation Satellite System (GNSS) is unreliable under the plant canopy. We present a hybrid navigation system that autonomously switches between different sets of sensing modalities to enable full field navigation, both inside and outside of crop. By choosing the appropriate path reference source, the robot can accommodate for loss of GNSS signal quality and leverage row-crop structure to autonomously navigate. However, such switching can be tricky and difficult to execute over scale. Our system provides a solution by automatically switching between an exteroceptive sensing based system, such as Light Detection And Ranging (LiDAR) row-following navigation and waypoints path tracking. In addition, we show how our system can detect when the navigate fails and recover automatically extending the autonomous time and mitigating the necessity of human intervention. Our system shows an improvement of about 750 m per intervention over GNSS-based navigation and 500 m over row following navigation.
△ Less
Submitted 17 November, 2024;
originally announced November 2024.
-
Fed-EC: Bandwidth-Efficient Clustering-Based Federated Learning For Autonomous Visual Robot Navigation
Authors:
Shreya Gummadi,
Mateus V. Gasparino,
Deepak Vasisht,
Girish Chowdhary
Abstract:
Centralized learning requires data to be aggregated at a central server, which poses significant challenges in terms of data privacy and bandwidth consumption. Federated learning presents a compelling alternative, however, vanilla federated learning methods deployed in robotics aim to learn a single global model across robots that works ideally for all. But in practice one model may not be well su…
▽ More
Centralized learning requires data to be aggregated at a central server, which poses significant challenges in terms of data privacy and bandwidth consumption. Federated learning presents a compelling alternative, however, vanilla federated learning methods deployed in robotics aim to learn a single global model across robots that works ideally for all. But in practice one model may not be well suited for robots deployed in various environments. This paper proposes Federated-EmbedCluster (Fed-EC), a clustering-based federated learning framework that is deployed with vision based autonomous robot navigation in diverse outdoor environments. The framework addresses the key federated learning challenge of deteriorating model performance of a single global model due to the presence of non-IID data across real-world robots. Extensive real-world experiments validate that Fed-EC reduces the communication size by 23x for each robot while matching the performance of centralized learning for goal-oriented navigation and outperforms local learning. Fed-EC can transfer previously learnt models to new robots that join the cluster.
△ Less
Submitted 6 November, 2024;
originally announced November 2024.
-
AdaCropFollow: Self-Supervised Online Adaptation for Visual Under-Canopy Navigation
Authors:
Arun N. Sivakumar,
Federico Magistri,
Mateus V. Gasparino,
Jens Behley,
Cyrill Stachniss,
Girish Chowdhary
Abstract:
Under-canopy agricultural robots can enable various applications like precise monitoring, spraying, weeding, and plant manipulation tasks throughout the growing season. Autonomous navigation under the canopy is challenging due to the degradation in accuracy of RTK-GPS and the large variability in the visual appearance of the scene over time. In prior work, we developed a supervised learning-based…
▽ More
Under-canopy agricultural robots can enable various applications like precise monitoring, spraying, weeding, and plant manipulation tasks throughout the growing season. Autonomous navigation under the canopy is challenging due to the degradation in accuracy of RTK-GPS and the large variability in the visual appearance of the scene over time. In prior work, we developed a supervised learning-based perception system with semantic keypoint representation and deployed this in various field conditions. A large number of failures of this system can be attributed to the inability of the perception model to adapt to the domain shift encountered during deployment. In this paper, we propose a self-supervised online adaptation method for adapting the semantic keypoint representation using a visual foundational model, geometric prior, and pseudo labeling. Our preliminary experiments show that with minimal data and fine-tuning of parameters, the keypoint prediction model trained with labels on the source domain can be adapted in a self-supervised manner to various challenging target domains onboard the robot computer using our method. This can enable fully autonomous row-following capability in under-canopy robots across fields and crops without requiring human intervention.
△ Less
Submitted 16 October, 2024;
originally announced October 2024.
-
Learning to Turn: Diffusion Imitation for Robust Row Turning in Under-Canopy Robots
Authors:
Arun N. Sivakumar,
Pranay Thangeda,
Yixiao Fang,
Mateus V. Gasparino,
Jose Cuaran,
Melkior Ornik,
Girish Chowdhary
Abstract:
Under-canopy agricultural robots require robust navigation capabilities to enable full autonomy but struggle with tight row turning between crop rows due to degraded GPS reception, visual aliasing, occlusion, and complex vehicle dynamics. We propose an imitation learning approach using diffusion policies to learn row turning behaviors from demonstrations provided by human operators or privileged c…
▽ More
Under-canopy agricultural robots require robust navigation capabilities to enable full autonomy but struggle with tight row turning between crop rows due to degraded GPS reception, visual aliasing, occlusion, and complex vehicle dynamics. We propose an imitation learning approach using diffusion policies to learn row turning behaviors from demonstrations provided by human operators or privileged controllers. Simulation experiments in a corn field environment show potential in learning this task with only visual observations and velocity states. However, challenges remain in maintaining control within rows and handling varied initial conditions, highlighting areas for future improvement.
△ Less
Submitted 6 August, 2024;
originally announced August 2024.
-
W-RIZZ: A Weakly-Supervised Framework for Relative Traversability Estimation in Mobile Robotics
Authors:
Andre Schreiber,
Arun N. Sivakumar,
Peter Du,
Mateus V. Gasparino,
Girish Chowdhary,
Katherine Driggs-Campbell
Abstract:
Successful deployment of mobile robots in unstructured domains requires an understanding of the environment and terrain to avoid hazardous areas, getting stuck, and colliding with obstacles. Traversability estimation--which predicts where in the environment a robot can travel--is one prominent approach that tackles this problem. Existing geometric methods may ignore important semantic consideratio…
▽ More
Successful deployment of mobile robots in unstructured domains requires an understanding of the environment and terrain to avoid hazardous areas, getting stuck, and colliding with obstacles. Traversability estimation--which predicts where in the environment a robot can travel--is one prominent approach that tackles this problem. Existing geometric methods may ignore important semantic considerations, while semantic segmentation approaches involve a tedious labeling process. Recent self-supervised methods reduce labeling tedium, but require additional data or models and tend to struggle to explicitly label untraversable areas. To address these limitations, we introduce a weakly-supervised method for relative traversability estimation. Our method involves manually annotating the relative traversability of a small number of point pairs, which significantly reduces labeling effort compared to traditional segmentation-based methods and avoids the limitations of self-supervised methods. We further improve the performance of our method through a novel cross-image labeling strategy and loss function. We demonstrate the viability and performance of our method through deployment on a mobile robot in outdoor environments.
△ Less
Submitted 4 June, 2024;
originally announced June 2024.
-
Lessons from Deploying CropFollow++: Under-Canopy Agricultural Navigation with Keypoints
Authors:
Arun N. Sivakumar,
Mateus V. Gasparino,
Michael McGuire,
Vitor A. H. Higuti,
M. Ugur Akcal,
Girish Chowdhary
Abstract:
We present a vision-based navigation system for under-canopy agricultural robots using semantic keypoints. Autonomous under-canopy navigation is challenging due to the tight spacing between the crop rows ($\sim 0.75$ m), degradation in RTK-GPS accuracy due to multipath error, and noise in LiDAR measurements from the excessive clutter. Our system, CropFollow++, introduces modular and interpretable…
▽ More
We present a vision-based navigation system for under-canopy agricultural robots using semantic keypoints. Autonomous under-canopy navigation is challenging due to the tight spacing between the crop rows ($\sim 0.75$ m), degradation in RTK-GPS accuracy due to multipath error, and noise in LiDAR measurements from the excessive clutter. Our system, CropFollow++, introduces modular and interpretable perception architecture with a learned semantic keypoint representation. We deployed CropFollow++ in multiple under-canopy cover crop planting robots on a large scale (25 km in total) in various field conditions and we discuss the key lessons learned from this.
△ Less
Submitted 26 April, 2024;
originally announced April 2024.
-
WayFASTER: a Self-Supervised Traversability Prediction for Increased Navigation Awareness
Authors:
Mateus Valverde Gasparino,
Arun Narenthiran Sivakumar,
Girish Chowdhary
Abstract:
Accurate and robust navigation in unstructured environments requires fusing data from multiple sensors. Such fusion ensures that the robot is better aware of its surroundings, including areas of the environment that are not immediately visible but were visible at a different time. To solve this problem, we propose a method for traversability prediction in challenging outdoor environments using a s…
▽ More
Accurate and robust navigation in unstructured environments requires fusing data from multiple sensors. Such fusion ensures that the robot is better aware of its surroundings, including areas of the environment that are not immediately visible but were visible at a different time. To solve this problem, we propose a method for traversability prediction in challenging outdoor environments using a sequence of RGB and depth images fused with pose estimations. Our method, termed WayFASTER (Waypoints-Free Autonomous System for Traversability with Enhanced Robustness), uses experience data recorded from a receding horizon estimator to train a self-supervised neural network for traversability prediction, eliminating the need for heuristics. Our experiments demonstrate that our method excels at avoiding obstacles, and correctly detects that traversable terrains, such as tall grass, can be navigable. By using a sequence of images, WayFASTER significantly enhances the robot's awareness of its surroundings, enabling it to predict the traversability of terrains that are not immediately visible. This enhanced awareness contributes to better navigation performance in environments where such predictive capabilities are essential.
△ Less
Submitted 9 March, 2024; v1 submitted 1 February, 2024;
originally announced February 2024.
-
Unmatched uncertainty mitigation through neural network supported model predictive control
Authors:
Mateus V. Gasparino,
Prabhat K. Mishra,
Girish Chowdhary
Abstract:
This paper presents a deep learning based model predictive control (MPC) algorithm for systems with unmatched and bounded state-action dependent uncertainties of unknown structure. We utilize a deep neural network (DNN) as an oracle in the underlying optimization problem of learning based MPC (LBMPC) to estimate unmatched uncertainties. Generally, non-parametric oracles such as DNN are considered…
▽ More
This paper presents a deep learning based model predictive control (MPC) algorithm for systems with unmatched and bounded state-action dependent uncertainties of unknown structure. We utilize a deep neural network (DNN) as an oracle in the underlying optimization problem of learning based MPC (LBMPC) to estimate unmatched uncertainties. Generally, non-parametric oracles such as DNN are considered difficult to employ with LBMPC due to the technical difficulties associated with estimation of their coefficients in real time. We employ a dual-timescale adaptation mechanism, where the weights of the last layer of the neural network are updated in real time while the inner layers are trained on a slower timescale using the training data collected online and selectively stored in a buffer. Our results are validated through a numerical experiment on the compression system model of jet engine. These results indicate that the proposed approach is implementable in real time and carries the theoretical guarantees of LBMPC.
△ Less
Submitted 22 April, 2023;
originally announced April 2023.
-
Deep Model Predictive Control
Authors:
Prabhat K. Mishra,
Mateus V. Gasparino,
Andres E. B. Velasquez,
Girish Chowdhary
Abstract:
This paper presents a deep learning based model predictive control algorithm for control affine nonlinear discrete time systems with matched and bounded state-dependent uncertainties of unknown structure. Since the structure of uncertainties is not known, a deep neural network (DNN) is employed to approximate the disturbances. In order to avoid any unwanted behavior during the learning phase, a tu…
▽ More
This paper presents a deep learning based model predictive control algorithm for control affine nonlinear discrete time systems with matched and bounded state-dependent uncertainties of unknown structure. Since the structure of uncertainties is not known, a deep neural network (DNN) is employed to approximate the disturbances. In order to avoid any unwanted behavior during the learning phase, a tube based model predictive controller is employed, which ensures satisfaction of constraints and input-to-state stability of the closed-loop states.
△ Less
Submitted 27 February, 2023;
originally announced February 2023.
-
WayFAST: Navigation with Predictive Traversability in the Field
Authors:
Mateus Valverde Gasparino,
Arun Narenthiran Sivakumar,
Yixiao Liu,
Andres Eduardo Baquero Velasquez,
Vitor Akihiro Hisano Higuti,
John Rogers,
Huy Tran,
Girish Chowdhary
Abstract:
We present a self-supervised approach for learning to predict traversable paths for wheeled mobile robots that require good traction to navigate. Our algorithm, termed WayFAST (Waypoint Free Autonomous Systems for Traversability), uses RGB and depth data, along with navigation experience, to autonomously generate traversable paths in outdoor unstructured environments. Our key inspiration is that t…
▽ More
We present a self-supervised approach for learning to predict traversable paths for wheeled mobile robots that require good traction to navigate. Our algorithm, termed WayFAST (Waypoint Free Autonomous Systems for Traversability), uses RGB and depth data, along with navigation experience, to autonomously generate traversable paths in outdoor unstructured environments. Our key inspiration is that traction can be estimated for rolling robots using kinodynamic models. Using traction estimates provided by an online receding horizon estimator, we are able to train a traversability prediction neural network in a self-supervised manner, without requiring heuristics utilized by previous methods. We demonstrate the effectiveness of WayFAST through extensive field testing in varying environments, ranging from sandy dry beaches to forest canopies and snow covered grass fields. Our results clearly demonstrate that WayFAST can learn to avoid geometric obstacles as well as untraversable terrain, such as snow, which would be difficult to avoid with sensors that provide only geometric data, such as LiDAR. Furthermore, we show that our training pipeline based on online traction estimates is more data-efficient than other heuristic-based methods.
△ Less
Submitted 1 August, 2022; v1 submitted 22 March, 2022;
originally announced March 2022.
-
Learned Visual Navigation for Under-Canopy Agricultural Robots
Authors:
Arun Narenthiran Sivakumar,
Sahil Modi,
Mateus Valverde Gasparino,
Che Ellis,
Andres Eduardo Baquero Velasquez,
Girish Chowdhary,
Saurabh Gupta
Abstract:
We describe a system for visually guided autonomous navigation of under-canopy farm robots. Low-cost under-canopy robots can drive between crop rows under the plant canopy and accomplish tasks that are infeasible for over-the-canopy drones or larger agricultural equipment. However, autonomously navigating them under the canopy presents a number of challenges: unreliable GPS and LiDAR, high cost of…
▽ More
We describe a system for visually guided autonomous navigation of under-canopy farm robots. Low-cost under-canopy robots can drive between crop rows under the plant canopy and accomplish tasks that are infeasible for over-the-canopy drones or larger agricultural equipment. However, autonomously navigating them under the canopy presents a number of challenges: unreliable GPS and LiDAR, high cost of sensing, challenging farm terrain, clutter due to leaves and weeds, and large variability in appearance over the season and across crop types. We address these challenges by building a modular system that leverages machine learning for robust and generalizable perception from monocular RGB images from low-cost cameras, and model predictive control for accurate control in challenging terrain. Our system, CropFollow, is able to autonomously drive 485 meters per intervention on average, outperforming a state-of-the-art LiDAR based system (286 meters per intervention) in extensive field testing spanning over 25 km.
△ Less
Submitted 6 July, 2021;
originally announced July 2021.
-
Multi-Sensor Fusion based Robust Row Following for Compact Agricultural Robots
Authors:
Andres Eduardo Baquero Velasquez,
Vitor Akihiro Hisano Higuti,
Mateus Valverde Gasparino,
Arun Narenthiran Sivakumar,
Marcelo Becker,
Girish Chowdhary
Abstract:
This paper presents a state-of-the-art LiDAR based autonomous navigation system for under-canopy agricultural robots. Under-canopy agricultural navigation has been a challenging problem because GNSS and other positioning sensors are prone to significant errors due to attentuation and multi-path caused by crop leaves and stems. Reactive navigation by detecting crop rows using LiDAR measurements is…
▽ More
This paper presents a state-of-the-art LiDAR based autonomous navigation system for under-canopy agricultural robots. Under-canopy agricultural navigation has been a challenging problem because GNSS and other positioning sensors are prone to significant errors due to attentuation and multi-path caused by crop leaves and stems. Reactive navigation by detecting crop rows using LiDAR measurements is a better alternative to GPS but suffers from challenges due to occlusion from leaves under the canopy. Our system addresses this challenge by fusing IMU and LiDAR measurements using an Extended Kalman Filter framework on low-cost hardwware. In addition, a local goal generator is introduced to provide locally optimal reference trajectories to the onboard controller. Our system is validated extensively in real-world field environments over a distance of 50.88~km on multiple robots in different field conditions across different locations. We report state-of-the-art distance between intervention results, showing that our system is able to safely navigate without interventions for 386.9~m on average in fields without significant gaps in the crop rows, 56.1~m in production fields and 47.5~m in fields with gaps (space of 1~m without plants in both sides of the row).
△ Less
Submitted 28 June, 2021;
originally announced June 2021.
-
Deep Model Predictive Control with Stability Guarantees
Authors:
Prabhat K. Mishra,
Mateus V. Gasparino,
Andres E. B. Velsasquez,
Girish Chowdhary
Abstract:
This paper presents a deep learning based model predictive control algorithm for control affine nonlinear discrete time systems with matched and bounded state dependent uncertainties of unknown structure. Since the structure of uncertainties is not known, a deep learning based adaptive mechanism is utilized to mitigate disturbances. In order to avoid any unwanted behavior during the learning phase…
▽ More
This paper presents a deep learning based model predictive control algorithm for control affine nonlinear discrete time systems with matched and bounded state dependent uncertainties of unknown structure. Since the structure of uncertainties is not known, a deep learning based adaptive mechanism is utilized to mitigate disturbances. In order to avoid any unwanted behavior during the learning phase, a tube based model predictive controller is employed, which ensures satisfaction of constraints and input-to-state stability of the closed-loop states. In addition, the proposed approach guarantees the convergence of states to origin under certain verifiable conditions. To ensure stability and undesirable learning transients, a dual-timescale adaptation mechanism is proposed, where the weights of the last layer of the neural network are updated each time instant while the inner layers are trained on a slower timescale using training data collected online and selectively stored in a buffer on the basis of singular value maximization criterion. Our results are validated through numerical experiments on wing-rock dynamics. These results indicate that the proposed deep-MPC architecture is effective in learning to control safety critical systems without suffering instability drawbacks.
△ Less
Submitted 24 September, 2021; v1 submitted 14 April, 2021;
originally announced April 2021.
