-
Neural-Augmented Kelvinlet: Real-Time Soft Tissue Deformation with Multiple Graspers
Authors:
Ashkan Shahbazi,
Kyvia Pereira,
Jon S. Heiselman,
Elaheh Akbari,
Annie C. Benson,
Sepehr Seifi,
Xinyuan Liu,
Garrison L. Johnston,
Erwin Terpstra,
Anne Draaisma,
Jan-Jaap Severes,
Jie Ying Wu,
Nabil Simaan,
Michael L. Miga,
Soheil Kolouri
Abstract:
Fast and accurate simulation of soft tissue deformation is a critical factor for surgical robotics and medical training. In this paper, we introduce a novel physics-informed neural simulator that approximates soft tissue deformations in a realistic and real-time manner. Our framework integrates Kelvinlet-based priors into neural simulators, making it the first approach to leverage Kelvinlets for r…
▽ More
Fast and accurate simulation of soft tissue deformation is a critical factor for surgical robotics and medical training. In this paper, we introduce a novel physics-informed neural simulator that approximates soft tissue deformations in a realistic and real-time manner. Our framework integrates Kelvinlet-based priors into neural simulators, making it the first approach to leverage Kelvinlets for residual learning and regularization in data-driven soft tissue modeling. By incorporating large-scale Finite Element Method (FEM) simulations of both linear and nonlinear soft tissue responses, our method improves neural network predictions across diverse architectures, enhancing accuracy and physical consistency while maintaining low latency for real-time performance. We demonstrate the effectiveness of our approach by performing accurate surgical maneuvers that simulate the use of standard laparoscopic tissue grasping tools with high fidelity. These results establish Kelvinlet-augmented learning as a powerful and efficient strategy for real-time, physics-aware soft tissue simulation in surgical applications.
△ Less
Submitted 6 June, 2025;
originally announced June 2025.
-
A Modal-Space Formulation for Momentum Observer Contact Estimation and Effects of Uncertainty for Continuum Robots
Authors:
Garrison L. H. Johnston,
Neel Shihora,
Nabil Simaan
Abstract:
Contact detection for continuum and soft robots has been limited in past works to statics or kinematics-based methods with assumed circular bending curvature or known bending profiles. In this paper, we adapt the generalized momentum observer contact estimation method to continuum robots. This is made possible by leveraging recent results for real-time shape sensing of continuum robots along with…
▽ More
Contact detection for continuum and soft robots has been limited in past works to statics or kinematics-based methods with assumed circular bending curvature or known bending profiles. In this paper, we adapt the generalized momentum observer contact estimation method to continuum robots. This is made possible by leveraging recent results for real-time shape sensing of continuum robots along with a modal-space representation of the robot dynamics. In addition to presenting an approach for estimating the generalized forces due to contact via a momentum observer, we present a constrained optimization method to identify the wrench imparted on the robot during contact. We also present an approach for investigating the effects of unmodeled deviations in the robot's dynamic state on the contact detection method and we validate our algorithm by simulations and experiments. We also compare the performance of the momentum observer to the joint force deviation method, a direct estimation approach using the robot's full dynamic model. We also demonstrate a basic extension of the method to multisegment continuum robots. Results presented in this work extend dynamic contact detection to the domain of continuum and soft robots and can be used to improve the safety of large-scale continuum robots for human-robot collaboration.
△ Less
Submitted 5 May, 2025;
originally announced May 2025.
-
Multi-Modal Gesture Recognition from Video and Surgical Tool Pose Information via Motion Invariants
Authors:
Jumanh Atoum,
Garrison L. H. Johnston,
Nabil Simaan,
Jie Ying Wu
Abstract:
Recognizing surgical gestures in real-time is a stepping stone towards automated activity recognition, skill assessment, intra-operative assistance, and eventually surgical automation. The current robotic surgical systems provide us with rich multi-modal data such as video and kinematics. While some recent works in multi-modal neural networks learn the relationships between vision and kinematics d…
▽ More
Recognizing surgical gestures in real-time is a stepping stone towards automated activity recognition, skill assessment, intra-operative assistance, and eventually surgical automation. The current robotic surgical systems provide us with rich multi-modal data such as video and kinematics. While some recent works in multi-modal neural networks learn the relationships between vision and kinematics data, current approaches treat kinematics information as independent signals, with no underlying relation between tool-tip poses. However, instrument poses are geometrically related, and the underlying geometry can aid neural networks in learning gesture representation. Therefore, we propose combining motion invariant measures (curvature and torsion) with vision and kinematics data using a relational graph network to capture the underlying relations between different data streams. We show that gesture recognition improves when combining invariant signals with tool position, achieving 90.3\% frame-wise accuracy on the JIGSAWS suturing dataset. Our results show that motion invariant signals coupled with position are better representations of gesture motion compared to traditional position and quaternion representations. Our results highlight the need for geometric-aware modeling of kinematics for gesture recognition.
△ Less
Submitted 19 March, 2025;
originally announced March 2025.
-
A Feasibility Study of a Soft, Low-Cost, 6-Axis Load Cell for Haptics
Authors:
Madison Veliky,
Garrison L. H. Johnston,
Ahmet Yildiz,
Nabil Simaan
Abstract:
Haptic devices have shown to be valuable in supplementing surgical training, especially when providing haptic feedback based on user performance metrics such as wrench applied by the user on the tool. However, current 6-axis force/torque sensors are prohibitively expensive. This paper presents the design and calibration of a low-cost, six-axis force/torque sensor specially designed for laparoscopi…
▽ More
Haptic devices have shown to be valuable in supplementing surgical training, especially when providing haptic feedback based on user performance metrics such as wrench applied by the user on the tool. However, current 6-axis force/torque sensors are prohibitively expensive. This paper presents the design and calibration of a low-cost, six-axis force/torque sensor specially designed for laparoscopic haptic training applications. The proposed design uses Hall-effect sensors to measure the change in the position of magnets embedded in a silicone layer that results from an applied wrench to the device. Preliminary experimental validation demonstrates that these sensors can achieve an accuracy of 0.45 N and 0.014 Nm, and a theoretical XY range of +/-50N, Z range of +/-20N, and torque range of +/-0.2Nm. This study indicates that the proposed low-cost 6-axis force/torque sensor can accurately measure user force and provide useful feedback during laparoscopic training on a haptic device.
△ Less
Submitted 4 October, 2024;
originally announced October 2024.
-
Design Considerations for 3RRR Parallel Robots with Lightweight, Approximate Static-Balancing
Authors:
Giuseppe Del Giudice,
Garrison L. H. Johnston,
Nabil Simaan
Abstract:
Balancing parallel robots throughout their workspace while avoiding the use of balancing masses and respecting design practicality constraints is difficult. Medical robots demand such compact and lightweight designs. This paper considers the difficult task of achieving optimal approximate balancing of a parallel robot throughout a desired task-based dexterous workspace using balancing springs only…
▽ More
Balancing parallel robots throughout their workspace while avoiding the use of balancing masses and respecting design practicality constraints is difficult. Medical robots demand such compact and lightweight designs. This paper considers the difficult task of achieving optimal approximate balancing of a parallel robot throughout a desired task-based dexterous workspace using balancing springs only. While it is possible to achieve perfect balancing in a path, only approximate balancing may be achieved without the addition of balancing masses. Design considerations for optimal robot base placement and the effects of placement of torsional balancing springs are presented. Using a modal representation for the balancing torque requirements, we use recent results on the design of wire-wrapped cam mechanisms to achieve balancing throughout a task-based workspace. A simulation study shows that robot base placement can have a detrimental effect on the attainability of a practical design solution for static balancing. We also show that optimal balancing using torsional springs is best achieved when all springs are at the actuated joints and that the wire-wrapped cam design can significantly improve the performance of static balancing. The methodology presented in this paper provides practical design solutions that yield simple, lightweight and compact designs suitable for medical applications where such traits are paramount.
△ Less
Submitted 22 June, 2023;
originally announced June 2023.
-
Task and Configuration Space Compliance of Continuum Robots via Lie Group and Modal Shape Formulations
Authors:
Andrew L. Orekhov,
Garrison L. H. Johnston,
Nabil Simaan
Abstract:
Continuum robots suffer large deflections due to internal and external forces. Accurate modeling of their passive compliance is necessary for accurate environmental interaction, especially in scenarios where direct force sensing is not practical. This paper focuses on deriving analytic formulations for the compliance of continuum robots that can be modeled as Kirchhoff rods. Compared to prior work…
▽ More
Continuum robots suffer large deflections due to internal and external forces. Accurate modeling of their passive compliance is necessary for accurate environmental interaction, especially in scenarios where direct force sensing is not practical. This paper focuses on deriving analytic formulations for the compliance of continuum robots that can be modeled as Kirchhoff rods. Compared to prior works, the approach presented herein is not subject to the constant-curvature assumptions to derive the configuration space compliance, and we do not rely on computationally-expensive finite difference approximations to obtain the task space compliance. Using modal approximations over curvature space and Lie group integration, we obtain closed-form expressions for the task and configuration space compliance matrices of continuum robots, thereby bridging the gap between constant-curvature analytic formulations of configuration space compliance and variable curvature task space compliance. We first present an analytic expression for the compliance of a single Kirchhoff rod. We then extend this formulation for computing both the task space and configuration space compliance of a tendon-actuated continuum robot. We then use our formulation to study the tradeoffs between computation cost and modeling accuracy as well as the loss in accuracy from neglecting the Jacobian derivative term in the compliance model. Finally, we experimentally validate the model on a tendon-actuated continuum segment, demonstrating the model's ability to predict passive deflections with error below 11.5\% percent of total arc length.
△ Less
Submitted 22 June, 2023;
originally announced June 2023.
-
Design Considerations and Robustness to Parameter Uncertainty in Wire-Wrapped Cam Mechanisms
Authors:
Garrison L. H. Johnston,
Andrew L. Orekhov,
Nabil Simaan
Abstract:
Collaborative robots must simultaneously be safe enough to operate in close proximity to human operators and powerful enough to assist users in industrial tasks such as lifting heavy equipment. The requirement for safety necessitates that collaborative robots are designed with low-powered actuators. However, some industrial tasks may require the robot to have high payload capacity and/or long reac…
▽ More
Collaborative robots must simultaneously be safe enough to operate in close proximity to human operators and powerful enough to assist users in industrial tasks such as lifting heavy equipment. The requirement for safety necessitates that collaborative robots are designed with low-powered actuators. However, some industrial tasks may require the robot to have high payload capacity and/or long reach. For collaborative robot designs to be successful, they must find ways of addressing these conflicting design requirements. One promising strategy for navigating this tradeoff is through the use of static balancing mechanisms to offset the robot's self weight, thus enabling the selection of lower-powered actuators. In this paper, we introduce a novel, 2 degree of freedom static balancing mechanism based on spring-loaded, wire-wrapped cams. We also present an optimization-based cam design method that guarantees the cams stay convex, ensures the springs stay below their extensions limits, and minimizes sensitivity to unmodeled deviations from the nominal spring constant. Additionally, we present a model of the effect of friction between the wire and the cam. Lastly, we show experimentally that the torque generated by the cam mechanism matches the torque predicted in our modeling approach. Our results also suggest that the effects of wire-cam friction are significant for non-circular cams.
△ Less
Submitted 22 June, 2023;
originally announced June 2023.
-
Long Horizon Planning through Contact using Discrete Search and Continuous Optimization
Authors:
Ramkumar Natarajan,
Garrison L. H. Johnston,
Nabil Simaan,
Maxim Likhachev,
Howie Choset
Abstract:
Robots often have to perform manipulation tasks in close proximity to people. As such, it is desirable to use a robot arm that has limited joint torques to not injure the nearby person and interacts with the environment to explore new possibilities for completing a task. By bracing against the environment, robots can expand their reachable workspace, which would otherwise be inaccessible due to ex…
▽ More
Robots often have to perform manipulation tasks in close proximity to people. As such, it is desirable to use a robot arm that has limited joint torques to not injure the nearby person and interacts with the environment to explore new possibilities for completing a task. By bracing against the environment, robots can expand their reachable workspace, which would otherwise be inaccessible due to exceeding actuator torque limits, and accomplish tasks beyond their design specifications. However, motion planning for complex contact-rich tasks requires reasoning through the permutations of different possible contact modes and bracing locations, which grow exponentially with the number of contact points and links in the robot. To address this combinatorial problem, we developed INSAT, which interleaves graph search to explore the manipulator joint configuration and the contact mode space with incremental trajectory optimizations seeded by neighborhood solutions to find a dynamically feasible trajectory through contact. In this paper, we present recent additions to the INSAT algorithm that improve its runtime performance. In particular, we propose Lazy INSAT with reduced optimization rejection that systematically procrastinates its calls to trajectory optimization while reusing feasible solutions that violate boundary constraints. The algorithm is evaluated on a heavy payload transportation task in simulation and on physical hardware. In simulation, we show that Lazy INSAT can discover solutions for tasks that cannot be accomplished within its design limits and without interacting with the environment. In comparison to executing the same trajectory without environment support, we demonstrate that the utilization of bracing contacts reduces the overall torque required to execute the trajectory.
△ Less
Submitted 16 January, 2024; v1 submitted 16 October, 2022;
originally announced October 2022.
-
Kinematic Modeling and Compliance Modulation of Redundant Manipulators Under Bracing Constraints
Authors:
Garrison L. H. Johnston,
Andrew L. Orekhov,
Nabil Simaan
Abstract:
Collaborative robots should ideally use low torque actuators for passive safety reasons. However, some applications require these collaborative robots to reach deep into confined spaces while assisting a human operator in physically demanding tasks. In this paper, we consider the use of in-situ collaborative robots (ISCRs) that balance the conflicting demands of passive safety dictating low torque…
▽ More
Collaborative robots should ideally use low torque actuators for passive safety reasons. However, some applications require these collaborative robots to reach deep into confined spaces while assisting a human operator in physically demanding tasks. In this paper, we consider the use of in-situ collaborative robots (ISCRs) that balance the conflicting demands of passive safety dictating low torque actuation and the need to reach into deep confined spaces. We consider the judicious use of bracing as a possible solution to these conflicting demands and present a modeling framework that takes into account the constrained kinematics and the effect of bracing on the end-effector compliance. We then define a redundancy resolution framework that minimizes the directional compliance of the end-effector while maximizing end-effector dexterity. Kinematic simulation results show that the redundancy resolution strategy successfully decreases compliance and improves kinematic conditioning while satisfying the constraints imposed by the bracing task. Applications of this modeling framework can support future research on the choice of bracing locations and support the formation of an admittance control framework for collaborative control of ISCRs under bracing constraints. Such robots can benefit workers in the future by reducing the physiological burdens that contribute to musculoskeletal injury.
△ Less
Submitted 11 August, 2020;
originally announced August 2020.
