Showing 1–50 of 57 results for author: Chakraborty, N

Adaptive Stress Testing Black-Box LLM Planners

Authors: Neeloy Chakraborty, John Pohovey, Melkior Ornik, Katherine Driggs-Campbell

Abstract: Large language models (LLMs) have recently demonstrated success in generalizing across decision-making tasks including planning, control and prediction, but their tendency to hallucinate unsafe and undesired outputs poses risks. We argue that detecting such failures is necessary, especially in safety-critical scenarios. Existing black-box methods often detect hallucinations by identifying inconsis… ▽ More Large language models (LLMs) have recently demonstrated success in generalizing across decision-making tasks including planning, control and prediction, but their tendency to hallucinate unsafe and undesired outputs poses risks. We argue that detecting such failures is necessary, especially in safety-critical scenarios. Existing black-box methods often detect hallucinations by identifying inconsistencies across multiple samples. Many of these approaches typically introduce prompt perturbations like randomizing detail order or generating adversarial inputs, with the intuition that a confident model should produce stable outputs. We first perform a manual case study showing that other forms of perturbations (e.g., adding noise, removing sensor details) cause LLMs to hallucinate in a driving environment. We then propose a novel method for efficiently searching the space of prompt perturbations using Adaptive Stress Testing (AST) with Monte-Carlo Tree Search (MCTS). Our AST formulation enables discovery of scenarios and prompts that cause language models to act with high uncertainty. By generating MCTS prompt perturbation trees across diverse scenarios, we show that offline analyses can be used at runtime to automatically generate prompts that influence model uncertainty, and to inform real-time trust assessments of an LLM. △ Less

Submitted 8 May, 2025; originally announced May 2025.

Comments: 26 pages, 16 figures, 4 tables

SKALD: Scalable K-Anonymisation for Large Datasets

Authors: Kailash Reddy, Novoneel Chakraborty, Amogh Dharmavaram, Anshoo Tandon

Abstract: Data privacy and anonymisation are critical concerns in today's data-driven society, particularly when handling personal and sensitive user data. Regulatory frameworks worldwide recommend privacy-preserving protocols such as k-anonymisation to de-identify releases of tabular data. Available hardware resources provide an upper bound on the maximum size of dataset that can be processed at a time. La… ▽ More Data privacy and anonymisation are critical concerns in today's data-driven society, particularly when handling personal and sensitive user data. Regulatory frameworks worldwide recommend privacy-preserving protocols such as k-anonymisation to de-identify releases of tabular data. Available hardware resources provide an upper bound on the maximum size of dataset that can be processed at a time. Large datasets with sizes exceeding this upper bound must be broken up into smaller data chunks for processing. In these cases, standard k-anonymisation tools such as ARX can only operate on a per-chunk basis. This paper proposes SKALD, a novel algorithm for performing k-anonymisation on large datasets with limited RAM. Our SKALD algorithm offers multi-fold performance improvement over standard k-anonymisation methods by extracting and combining sufficient statistics from each chunk during processing to ensure successful k-anonymisation while providing better utility. △ Less

Submitted 6 May, 2025; originally announced May 2025.

Comments: 7 pages, 3 figures, 3 tables

Unifying Complementarity Constraints and Control Barrier Functions for Safe Whole-Body Robot Control

Authors: Rafael I. Cabral Muchacho, Riddhiman Laha, Florian T. Pokorny, Luis F. C. Figueredo, Nilanjan Chakraborty

Abstract: Safety-critical whole-body robot control demands reactive methods that ensure collision avoidance in real-time. Complementarity constraints and control barrier functions (CBF) have emerged as core tools for ensuring such safety constraints, and each represents a well-developed field. Despite addressing similar problems, their connection remains largely unexplored. This paper bridges this gap by fo… ▽ More Safety-critical whole-body robot control demands reactive methods that ensure collision avoidance in real-time. Complementarity constraints and control barrier functions (CBF) have emerged as core tools for ensuring such safety constraints, and each represents a well-developed field. Despite addressing similar problems, their connection remains largely unexplored. This paper bridges this gap by formally proving the equivalence between these two methodologies for sampled-data, first-order systems, considering both single and multiple constraint scenarios. By demonstrating this equivalence, we provide a unified perspective on these techniques. This unification has theoretical and practical implications, facilitating the cross-application of robustness guarantees and algorithmic improvements between complementarity and CBF frameworks. We discuss these synergistic benefits and motivate future work in the comparison of the methods in more general cases. △ Less

Submitted 24 April, 2025; originally announced April 2025.

Transferring Kinesthetic Demonstrations across Diverse Objects for Manipulation Planning

Authors: Dibyendu Das, Aditya Patankar, Nilanjan Chakraborty, C. R. Ramakrishnan, I. V. Ramakrishnan

Abstract: Given a demonstration of a complex manipulation task such as pouring liquid from one container to another, we seek to generate a motion plan for a new task instance involving objects with different geometries. This is non-trivial since we need to simultaneously ensure that the implicit motion constraints are satisfied (glass held upright while moving), the motion is collision-free, and that the ta… ▽ More Given a demonstration of a complex manipulation task such as pouring liquid from one container to another, we seek to generate a motion plan for a new task instance involving objects with different geometries. This is non-trivial since we need to simultaneously ensure that the implicit motion constraints are satisfied (glass held upright while moving), the motion is collision-free, and that the task is successful (e.g. liquid is poured into the target container). We solve this problem by identifying positions of critical locations and associating a reference frame (called motion transfer frames) on the manipulated object and the target, selected based on their geometries and the task at hand. By tracking and transferring the path of the motion transfer frames, we generate motion plans for arbitrary task instances with objects of different geometries and poses. We show results from simulation as well as robot experiments on physical objects to evaluate the effectiveness of our solution. △ Less

Submitted 13 March, 2025; originally announced March 2025.

An Expert Ensemble for Detecting Anomalous Scenes, Interactions, and Behaviors in Autonomous Driving

Authors: Tianchen Ji, Neeloy Chakraborty, Andre Schreiber, Katherine Driggs-Campbell

Abstract: As automated vehicles enter public roads, safety in a near-infinite number of driving scenarios becomes one of the major concerns for the widespread adoption of fully autonomous driving. The ability to detect anomalous situations outside of the operational design domain is a key component in self-driving cars, enabling us to mitigate the impact of abnormal ego behaviors and to realize trustworthy… ▽ More As automated vehicles enter public roads, safety in a near-infinite number of driving scenarios becomes one of the major concerns for the widespread adoption of fully autonomous driving. The ability to detect anomalous situations outside of the operational design domain is a key component in self-driving cars, enabling us to mitigate the impact of abnormal ego behaviors and to realize trustworthy driving systems. On-road anomaly detection in egocentric videos remains a challenging problem due to the difficulties introduced by complex and interactive scenarios. We conduct a holistic analysis of common on-road anomaly patterns, from which we propose three unsupervised anomaly detection experts: a scene expert that focuses on frame-level appearances to detect abnormal scenes and unexpected scene motions; an interaction expert that models normal relative motions between two road participants and raises alarms whenever anomalous interactions emerge; and a behavior expert which monitors abnormal behaviors of individual objects by future trajectory prediction. To combine the strengths of all the modules, we propose an expert ensemble (Xen) using a Kalman filter, in which the final anomaly score is absorbed as one of the states and the observations are generated by the experts. Our experiments employ a novel evaluation protocol for realistic model performance, demonstrate superior anomaly detection performance than previous methods, and show that our framework has potential in classifying anomaly types using unsupervised learning on a large-scale on-road anomaly dataset. △ Less

Submitted 22 February, 2025; originally announced February 2025.

Comments: Accepted by International Journal of Robotics Research (IJRR)

Synthesizing Grasps and Regrasps for Complex Manipulation Tasks

Authors: Aditya Patankar, Dasharadhan Mahalingam, Nilanjan Chakraborty

Abstract: In complex manipulation tasks, e.g., manipulation by pivoting, the motion of the object being manipulated has to satisfy path constraints that can change during the motion. Therefore, a single grasp may not be sufficient for the entire path, and the object may need to be regrasped. Additionally, geometric data for objects from a sensor are usually available in the form of point clouds. The problem… ▽ More In complex manipulation tasks, e.g., manipulation by pivoting, the motion of the object being manipulated has to satisfy path constraints that can change during the motion. Therefore, a single grasp may not be sufficient for the entire path, and the object may need to be regrasped. Additionally, geometric data for objects from a sensor are usually available in the form of point clouds. The problem of computing grasps and regrasps from point-cloud representation of objects for complex manipulation tasks is a key problem in endowing robots with manipulation capabilities beyond pick-and-place. In this paper, we formalize the problem of grasping/regrasping for complex manipulation tasks with objects represented by (partial) point clouds and present an algorithm to solve it. We represent a complex manipulation task as a sequence of constant screw motions. Using a manipulation plan skeleton as a sequence of constant screw motions, we use a grasp metric to find graspable regions on the object for every constant screw segment. The overlap of the graspable regions for contiguous screws are then used to determine when and how many times the object needs to be regrasped. We present experimental results on point cloud data collected from RGB-D sensors to illustrate our approach. △ Less

Submitted 29 January, 2025; originally announced January 2025.

Privacy-Preserving Data Quality Assessment for Time-Series IoT Sensors

Authors: Novoneel Chakraborty, Abhay Sharma, Jyotirmoy Dutta, Hari Dilip Kumar

Abstract: Data from Internet of Things (IoT) sensors has emerged as a key contributor to decision-making processes in various domains. However, the quality of the data is crucial to the effectiveness of applications built on it, and assessment of the data quality is heavily context-dependent. Further, preserving the privacy of the data during quality assessment is critical in domains where sensitive data is… ▽ More Data from Internet of Things (IoT) sensors has emerged as a key contributor to decision-making processes in various domains. However, the quality of the data is crucial to the effectiveness of applications built on it, and assessment of the data quality is heavily context-dependent. Further, preserving the privacy of the data during quality assessment is critical in domains where sensitive data is prevalent. This paper proposes a novel framework for automated, objective, and privacy-preserving data quality assessment of time-series data from IoT sensors deployed in smart cities. We leverage custom, autonomously computable metrics that parameterise the temporal performance and adherence to a declarative schema document to achieve objectivity. Additionally, we utilise a trusted execution environment to create a "data-blind" model that ensures individual privacy, eliminates assessee bias, and enhances adaptability across data types. This paper describes this data quality assessment methodology for IoT sensors, emphasising its relevance within the smart-city context while addressing the growing need for privacy in the face of extensive data collection practices. △ Less

Submitted 13 January, 2025; originally announced January 2025.

Comments: 7 pages, 4 figures, 1 table, published - IoTaIS 2024 Conference Proceedings

Journal ref: 2024 IEEE International Conference on Internet of Things and Intelligence Systems (IoTaIS), Bali, Indonesia, 2024, pp. 51-57

Building a Privacy Web with SPIDEr -- Secure Pipeline for Information De-Identification with End-to-End Encryption

Authors: Novoneel Chakraborty, Anshoo Tandon, Kailash Reddy, Kaushal Kirpekar, Bryan Paul Robert, Hari Dilip Kumar, Abhilash Venkatesh, Abhay Sharma

Abstract: Data de-identification makes it possible to glean insights from data while preserving user privacy. The use of Trusted Execution Environments (TEEs) allow for the execution of de-identification applications on the cloud without the need for a user to trust the third-party application provider. In this paper, we present \textit{SPIDEr - Secure Pipeline for Information De-Identification with End-to-… ▽ More Data de-identification makes it possible to glean insights from data while preserving user privacy. The use of Trusted Execution Environments (TEEs) allow for the execution of de-identification applications on the cloud without the need for a user to trust the third-party application provider. In this paper, we present \textit{SPIDEr - Secure Pipeline for Information De-Identification with End-to-End Encryption}, our implementation of an end-to-end encrypted data de-identification pipeline. SPIDEr supports classical anonymisation techniques such as suppression, pseudonymisation, generalisation, and aggregation, as well as techniques that offer a formal privacy guarantee such as k-anonymisation and differential privacy. To enable scalability and improve performance on constrained TEE hardware, we enable batch processing of data for differential privacy computations. We present our design of the control flows for end-to-end secure execution of de-identification operations within a TEE. As part of the control flow for running SPIDEr within the TEE, we perform attestation, a process that verifies that the software binaries were properly instantiated on a known, trusted platform. △ Less

Submitted 12 December, 2024; originally announced December 2024.

Comments: 3 pages, 2 figures

HEIGHT: Heterogeneous Interaction Graph Transformer for Robot Navigation in Crowded and Constrained Environments

Authors: Shuijing Liu, Haochen Xia, Fatemeh Cheraghi Pouria, Kaiwen Hong, Neeloy Chakraborty, Zichao Hu, Joydeep Biswas, Katherine Driggs-Campbell

Abstract: We study the problem of robot navigation in dense and interactive crowds with environmental constraints such as corridors and furniture. Previous methods fail to consider all types of interactions among agents and obstacles, leading to unsafe and inefficient robot paths. In this article, we leverage a graph-based representation of crowded and constrained scenarios and propose a structured framewor… ▽ More We study the problem of robot navigation in dense and interactive crowds with environmental constraints such as corridors and furniture. Previous methods fail to consider all types of interactions among agents and obstacles, leading to unsafe and inefficient robot paths. In this article, we leverage a graph-based representation of crowded and constrained scenarios and propose a structured framework to learn robot navigation policies with deep reinforcement learning. We first split the representations of different components in the environment and propose a heterogeneous spatio-temporal (st) graph to model distinct interactions among humans, robots, and obstacles. Based on the heterogeneous st-graph, we propose HEIGHT, a novel navigation policy network architecture with different components to capture heterogeneous interactions among entities through space and time. HEIGHT utilizes attention mechanisms to prioritize important interactions and a recurrent network to track changes in the dynamic scene over time, encouraging the robot to avoid collisions adaptively. Through extensive simulation and real-world experiments, we demonstrate that HEIGHT outperforms state-of-the-art baselines in terms of success and efficiency in challenging navigation scenarios. Furthermore, we demonstrate that our pipeline achieves better zero-shot generalization capability than previous works when the densities of humans and obstacles change. More videos are available at https://sites.google.com/view/crowdnav-height/home. △ Less

Submitted 1 May, 2025; v1 submitted 18 November, 2024; originally announced November 2024.

On the Synthesis of Reactive Collision-Free Whole-Body Robot Motions: A Complementarity-based Approach

Authors: Haowen Yao, Riddhiman Laha, Anirban Sinha, Jonas Hall, Luis F. C. Figueredo, Nilanjan Chakraborty, Sami Haddadin

Abstract: This paper is about generating motion plans for high degree-of-freedom systems that account for collisions along the entire body. A particular class of mathematical programs with complementarity constraints become useful in this regard. Optimization-based planners can tackle confined-space trajectory planning while being cognizant of robot constraints. However, introducing obstacles in this settin… ▽ More This paper is about generating motion plans for high degree-of-freedom systems that account for collisions along the entire body. A particular class of mathematical programs with complementarity constraints become useful in this regard. Optimization-based planners can tackle confined-space trajectory planning while being cognizant of robot constraints. However, introducing obstacles in this setting transforms the formulation into a non-convex problem (oftentimes with ill-posed bilinear constraints), which is non-trivial in a real-time setting. To this end, we present the FLIQC (Fast LInear Quadratic Complementarity based) motion planner. Our planner employs a novel motion model that captures the entire rigid robot as well as the obstacle geometry and ensures non-penetration between the surfaces due to the imposed constraint. We perform thorough comparative studies with the state-of-the-art, which demonstrate improved performance. Extensive simulation and hardware experiments validate our claim of generating continuous and reactive motion plans at 1 kHz for modern collaborative robots with constant minimal parameters. △ Less

Submitted 29 October, 2024; originally announced October 2024.

Screw Geometry Meets Bandits: Incremental Acquisition of Demonstrations to Generate Manipulation Plans

Authors: Dibyendu Das, Aditya Patankar, Nilanjan Chakraborty, C. R. Ramakrishnan, I. V. Ramakrishnan

Abstract: In this paper, we study the problem of methodically obtaining a sufficient set of kinesthetic demonstrations, one at a time, such that a robot can be confident of its ability to perform a complex manipulation task in a given region of its workspace. Although Learning from Demonstrations has been an active area of research, the problems of checking whether a set of demonstrations is sufficient, and… ▽ More In this paper, we study the problem of methodically obtaining a sufficient set of kinesthetic demonstrations, one at a time, such that a robot can be confident of its ability to perform a complex manipulation task in a given region of its workspace. Although Learning from Demonstrations has been an active area of research, the problems of checking whether a set of demonstrations is sufficient, and systematically seeking additional demonstrations have remained open. We present a novel approach to address these open problems using (i) a screw geometric representation to generate manipulation plans from demonstrations, which makes the sufficiency of a set of demonstrations measurable; (ii) a sampling strategy based on PAC-learning from multi-armed bandit optimization to evaluate the robot's ability to generate manipulation plans in a subregion of its task space; and (iii) a heuristic to seek additional demonstration from areas of weakness. Thus, we present an approach for the robot to incrementally and actively ask for new demonstration examples until the robot can assess with high confidence that it can perform the task successfully. We present experimental results on two example manipulation tasks, namely, pouring and scooping, to illustrate our approach. A short video on the method: https://youtu.be/R-qICICdEos △ Less

Submitted 23 October, 2024; originally announced October 2024.

Comments: 8 pages, 6 figures, under review in IEEE Robotics and Automation Letters

A General Formulation for Path Constrained Time-Optimized Trajectory Planning with Environmental and Object Contacts

Authors: Dasharadhan Mahalingam, Aditya Patankar, Riddhiman Laha, Srinivasan Lakshminarayanan, Sami Haddadin, Nilanjan Chakraborty

Abstract: A typical manipulation task consists of a manipulator equipped with a gripper to grasp and move an object with constraints on the motion of the hand-held object, which may be due to the nature of the task itself or from object-environment contacts. In this paper, we study the problem of computing joint torques and grasping forces for time-optimal motion of an object, while ensuring that the grasp… ▽ More A typical manipulation task consists of a manipulator equipped with a gripper to grasp and move an object with constraints on the motion of the hand-held object, which may be due to the nature of the task itself or from object-environment contacts. In this paper, we study the problem of computing joint torques and grasping forces for time-optimal motion of an object, while ensuring that the grasp is not lost and any constraints on the motion of the object, either due to dynamics, environment contact, or no-slip requirements, are also satisfied. We present a second-order cone program (SOCP) formulation of the time-optimal trajectory planning problem that considers nonlinear friction cone constraints at the hand-object and object-environment contacts. Since SOCPs are convex optimization problems that can be solved optimally in polynomial time using interior point methods, we can solve the trajectory optimization problem efficiently. We present simulation results on three examples, including a non-prehensile manipulation task, which shows the generality and effectiveness of our approach. △ Less

Submitted 8 October, 2024; originally announced October 2024.

Provable Methods for Searching with an Imperfect Sensor

Authors: Nilanjan Chakraborty, Prahlad Narasimhan Kasthurirangan, Joseph S. B. Mitchell, Linh Nguyen, Michael Perk

Abstract: Assume that a target is known to be present at an unknown point among a finite set of locations in the plane. We search for it using a mobile robot that has imperfect sensing capabilities. It takes time for the robot to move between locations and search a location; we have a total time budget within which to conduct the search. We study the problem of computing a search path/strategy for the robot… ▽ More Assume that a target is known to be present at an unknown point among a finite set of locations in the plane. We search for it using a mobile robot that has imperfect sensing capabilities. It takes time for the robot to move between locations and search a location; we have a total time budget within which to conduct the search. We study the problem of computing a search path/strategy for the robot that maximizes the probability of detection of the target. Considering non-uniform travel times between points (e.g., based on the distance between them) is crucial for search and rescue applications; such problems have been investigated to a limited extent due to their inherent complexity. In this paper, we describe fast algorithms with performance guarantees for this search problem and some variants, complement them with complexity results, and perform experiments to observe their performance. △ Less

Submitted 8 October, 2024; originally announced October 2024.

Comments: 10 pages, 6 figures, 3 algorithms

Towards Real-Time Generation of Delay-Compensated Video Feeds for Outdoor Mobile Robot Teleoperation

Authors: Neeloy Chakraborty, Yixiao Fang, Andre Schreiber, Tianchen Ji, Zhe Huang, Aganze Mihigo, Cassidy Wall, Abdulrahman Almana, Katherine Driggs-Campbell

Abstract: Teleoperation is an important technology to enable supervisors to control agricultural robots remotely. However, environmental factors in dense crop rows and limitations in network infrastructure hinder the reliability of data streamed to teleoperators. These issues result in delayed and variable frame rate video feeds that often deviate significantly from the robot's actual viewpoint. We propose… ▽ More Teleoperation is an important technology to enable supervisors to control agricultural robots remotely. However, environmental factors in dense crop rows and limitations in network infrastructure hinder the reliability of data streamed to teleoperators. These issues result in delayed and variable frame rate video feeds that often deviate significantly from the robot's actual viewpoint. We propose a modular learning-based vision pipeline to generate delay-compensated images in real-time for supervisors. Our extensive offline evaluations demonstrate that our method generates more accurate images compared to state-of-the-art approaches in our setting. Additionally, ours is one of the few works to evaluate a delay-compensation method in outdoor field environments with complex terrain on data from a real robot in real-time. Resulting videos and code are provided at https://sites.google.com/illinois.edu/comp-teleop. △ Less

Submitted 16 February, 2025; v1 submitted 15 September, 2024; originally announced September 2024.

Comments: Accepted to IEEE ICRA 2025; 8 pages, 4 figures, 3 tables

Lessons in Cooperation: A Qualitative Analysis of Driver Sentiments towards Real-Time Advisory Systems from a Driving Simulator User Study

Authors: Aamir Hasan, Neeloy Chakraborty, Haonan Chen, Cathy Wu, Katherine Driggs-Campbell

Abstract: Real-time Advisory (RTA) systems, such as navigational and eco-driving assistants, are becoming increasingly ubiquitous in vehicles due to their benefits for users and society. Until autonomous vehicles mature, such advisory systems will continue to expand their ability to cooperate with drivers, enabling safer and more eco-friendly driving practices while improving user experience. However, the i… ▽ More Real-time Advisory (RTA) systems, such as navigational and eco-driving assistants, are becoming increasingly ubiquitous in vehicles due to their benefits for users and society. Until autonomous vehicles mature, such advisory systems will continue to expand their ability to cooperate with drivers, enabling safer and more eco-friendly driving practices while improving user experience. However, the interactions between these systems and drivers have not been studied extensively. To this end, we conduct a driving simulator study (N=16) to capture driver reactions to a Cooperative RTA system. Through a case study with a congestion mitigation assistant, we qualitatively analyze the sentiments of drivers towards advisory systems and discuss driver preferences for various aspects of the interaction. We comment on how the advice should be communicated, the effects of the advice on driver trust, and how drivers adapt to the system. We present recommendations to inform the future design of Cooperative RTA systems. △ Less

Submitted 29 June, 2024; originally announced July 2024.

Cooperative Advisory Residual Policies for Congestion Mitigation

Authors: Aamir Hasan, Neeloy Chakraborty, Haonan Chen, Jung-Hoon Cho, Cathy Wu, Katherine Driggs-Campbell

Abstract: Fleets of autonomous vehicles can mitigate traffic congestion through simple actions, thus improving many socioeconomic factors such as commute time and gas costs. However, these approaches are limited in practice as they assume precise control over autonomous vehicle fleets, incur extensive installation costs for a centralized sensor ecosystem, and also fail to account for uncertainty in driver b… ▽ More Fleets of autonomous vehicles can mitigate traffic congestion through simple actions, thus improving many socioeconomic factors such as commute time and gas costs. However, these approaches are limited in practice as they assume precise control over autonomous vehicle fleets, incur extensive installation costs for a centralized sensor ecosystem, and also fail to account for uncertainty in driver behavior. To this end, we develop a class of learned residual policies that can be used in cooperative advisory systems and only require the use of a single vehicle with a human driver. Our policies advise drivers to behave in ways that mitigate traffic congestion while accounting for diverse driver behaviors, particularly drivers' reactions to instructions, to provide an improved user experience. To realize such policies, we introduce an improved reward function that explicitly addresses congestion mitigation and driver attitudes to advice. We show that our residual policies can be personalized by conditioning them on an inferred driver trait that is learned in an unsupervised manner with a variational autoencoder. Our policies are trained in simulation with our novel instruction adherence driver model, and evaluated in simulation and through a user study (N=16) to capture the sentiments of human drivers. Our results show that our approaches successfully mitigate congestion while adapting to different driver behaviors, with up to 20% and 40% improvement as measured by a combination metric of speed and deviations in speed across time over baselines in our simulation tests and user study, respectively. Our user study further shows that our policies are human-compatible and personalize to drivers. △ Less

Submitted 29 June, 2024; originally announced July 2024.

Structured Graph Network for Constrained Robot Crowd Navigation with Low Fidelity Simulation

Authors: Shuijing Liu, Kaiwen Hong, Neeloy Chakraborty, Katherine Driggs-Campbell

Abstract: We investigate the feasibility of deploying reinforcement learning (RL) policies for constrained crowd navigation using a low-fidelity simulator. We introduce a representation of the dynamic environment, separating human and obstacle representations. Humans are represented through detected states, while obstacles are represented as computed point clouds based on maps and robot localization. This r… ▽ More We investigate the feasibility of deploying reinforcement learning (RL) policies for constrained crowd navigation using a low-fidelity simulator. We introduce a representation of the dynamic environment, separating human and obstacle representations. Humans are represented through detected states, while obstacles are represented as computed point clouds based on maps and robot localization. This representation enables RL policies trained in a low-fidelity simulator to deploy in real world with a reduced sim2real gap. Additionally, we propose a spatio-temporal graph to model the interactions between agents and obstacles. Based on the graph, we use attention mechanisms to capture the robot-human, human-human, and human-obstacle interactions. Our method significantly improves navigation performance in both simulated and real-world environments. Video demonstrations can be found at https://sites.google.com/view/constrained-crowdnav/home. △ Less

Submitted 27 May, 2024; v1 submitted 27 May, 2024; originally announced May 2024.

Hallucination Detection in Foundation Models for Decision-Making: A Flexible Definition and Review of the State of the Art

Authors: Neeloy Chakraborty, Melkior Ornik, Katherine Driggs-Campbell

Abstract: Autonomous systems are soon to be ubiquitous, spanning manufacturing, agriculture, healthcare, entertainment, and other industries. Most of these systems are developed with modular sub-components for decision-making, planning, and control that may be hand-engineered or learning-based. While these approaches perform well under the situations they were specifically designed for, they can perform esp… ▽ More Autonomous systems are soon to be ubiquitous, spanning manufacturing, agriculture, healthcare, entertainment, and other industries. Most of these systems are developed with modular sub-components for decision-making, planning, and control that may be hand-engineered or learning-based. While these approaches perform well under the situations they were specifically designed for, they can perform especially poorly in out-of-distribution scenarios that will undoubtedly arise at test-time. The rise of foundation models trained on multiple tasks with impressively large datasets has led researchers to believe that these models may provide "common sense" reasoning that existing planners are missing, bridging the gap between algorithm development and deployment. While researchers have shown promising results in deploying foundation models to decision-making tasks, these models are known to hallucinate and generate decisions that may sound reasonable, but are in fact poor. We argue there is a need to step back and simultaneously design systems that can quantify the certainty of a model's decision, and detect when it may be hallucinating. In this work, we discuss the current use cases of foundation models for decision-making tasks, provide a general definition for hallucinations with examples, discuss existing approaches to hallucination detection and mitigation with a focus on decision problems, present guidelines, and explore areas for further research in this exciting field. △ Less

Submitted 11 February, 2025; v1 submitted 25 March, 2024; originally announced March 2024.

Comments: Accepted to ACM Computing Surveys; 55 pages, 5 tables, 3 figures

Mean Estimation with User-Level Privacy for Spatio-Temporal IoT Datasets

Authors: V. Arvind Rameshwar, Anshoo Tandon, Prajjwal Gupta, Aditya Vikram Singh, Novoneel Chakraborty, Abhay Sharma

Abstract: This paper considers the problem of the private release of sample means of speed values from traffic datasets. Our key contribution is the development of user-level differentially private algorithms that incorporate carefully chosen parameter values to ensure low estimation errors on real-world datasets, while ensuring privacy. We test our algorithms on ITMS (Intelligent Traffic Management System)… ▽ More This paper considers the problem of the private release of sample means of speed values from traffic datasets. Our key contribution is the development of user-level differentially private algorithms that incorporate carefully chosen parameter values to ensure low estimation errors on real-world datasets, while ensuring privacy. We test our algorithms on ITMS (Intelligent Traffic Management System) data from an Indian city, where the speeds of different buses are drawn in a potentially non-i.i.d. manner from an unknown distribution, and where the number of speed samples contributed by different buses is potentially different. We then apply our algorithms to large synthetic datasets, generated based on the ITMS data. Here, we provide theoretical justification for the observed performance trends, and also provide recommendations for the choices of algorithm subroutines that result in low estimation errors. Finally, we characterize the best performance of pseudo-user creation-based algorithms on worst-case datasets via a minimax approach; this then gives rise to a novel procedure for the creation of pseudo-users, which optimizes the worst-case total estimation error. The algorithms discussed in the paper are readily applicable to general spatio-temporal IoT datasets for releasing a differentially private mean of a desired value. △ Less

Submitted 25 April, 2024; v1 submitted 29 January, 2024; originally announced January 2024.

Comments: 14 pages, 5 figures, submitted to the ACM for possible publication

Containerized Vertical Farming Using Cobots

Authors: Dasharadhan Mahalingam, Aditya Patankar, Khiem Phi, Nilanjan Chakraborty, Ryan McGann, IV Ramakrishnan

Abstract: Containerized vertical farming is a type of vertical farming practice using hydroponics in which plants are grown in vertical layers within a mobile shipping container. Space limitations within shipping containers make the automation of different farming operations challenging. In this paper, we explore the use of cobots (i.e., collaborative robots) to automate two key farming operations, namely,… ▽ More Containerized vertical farming is a type of vertical farming practice using hydroponics in which plants are grown in vertical layers within a mobile shipping container. Space limitations within shipping containers make the automation of different farming operations challenging. In this paper, we explore the use of cobots (i.e., collaborative robots) to automate two key farming operations, namely, the transplantation of saplings and the harvesting of grown plants. Our method uses a single demonstration from a farmer to extract the motion constraints associated with the tasks, namely, transplanting and harvesting, and can then generalize to different instances of the same task. For transplantation, the motion constraint arises during insertion of the sapling within the growing tube, whereas for harvesting, it arises during extraction from the growing tube. We present experimental results to show that using RGBD camera images (obtained from an eye-in-hand configuration) and one demonstration for each task, it is feasible to perform transplantation of saplings and harvesting of leafy greens using a cobot, without task-specific programming. △ Less

Submitted 23 October, 2023; originally announced October 2023.

MONOVAB : An Annotated Corpus for Bangla Multi-label Emotion Detection

Authors: Sumit Kumar Banshal, Sajal Das, Shumaiya Akter Shammi, Narayan Ranjan Chakraborty, Aulia Luqman Aziz, Mohammed Aljuaid, Fazla Rabby, Rohit Bansal

Abstract: In recent years, Sentiment Analysis (SA) and Emotion Recognition (ER) have been increasingly popular in the Bangla language, which is the seventh most spoken language throughout the entire world. However, the language is structurally complicated, which makes this field arduous to extract emotions in an accurate manner. Several distinct approaches such as the extraction of positive and negative sen… ▽ More In recent years, Sentiment Analysis (SA) and Emotion Recognition (ER) have been increasingly popular in the Bangla language, which is the seventh most spoken language throughout the entire world. However, the language is structurally complicated, which makes this field arduous to extract emotions in an accurate manner. Several distinct approaches such as the extraction of positive and negative sentiments as well as multiclass emotions, have been implemented in this field of study. Nevertheless, the extraction of multiple sentiments is an almost untouched area in this language. Which involves identifying several feelings based on a single piece of text. Therefore, this study demonstrates a thorough method for constructing an annotated corpus based on scrapped data from Facebook to bridge the gaps in this subject area to overcome the challenges. To make this annotation more fruitful, the context-based approach has been used. Bidirectional Encoder Representations from Transformers (BERT), a well-known methodology of transformers, have been shown the best results of all methods implemented. Finally, a web application has been developed to demonstrate the performance of the pre-trained top-performer model (BERT) for multi-label ER in Bangla. △ Less

Submitted 5 May, 2025; v1 submitted 27 September, 2023; originally announced September 2023.

Task-Oriented Grasping with Point Cloud Representation of Objects

Authors: Aditya Patankar, Khiem Phi, Dasharadhan Mahalingam, Nilanjan Chakraborty, IV Ramakrishnan

Abstract: In this paper, we study the problem of task-oriented grasp synthesis from partial point cloud data using an eye-in-hand camera configuration. In task-oriented grasp synthesis, a grasp has to be selected so that the object is not lost during manipulation, and it is also ensured that adequate force/moment can be applied to perform the task. We formalize the notion of a gross manipulation task as a c… ▽ More In this paper, we study the problem of task-oriented grasp synthesis from partial point cloud data using an eye-in-hand camera configuration. In task-oriented grasp synthesis, a grasp has to be selected so that the object is not lost during manipulation, and it is also ensured that adequate force/moment can be applied to perform the task. We formalize the notion of a gross manipulation task as a constant screw motion (or a sequence of constant screw motions) to be applied to the object after grasping. Using this notion of task, and a corresponding grasp quality metric developed in our prior work, we use a neural network to approximate a function for predicting the grasp quality metric on a cuboid shape. We show that by using a bounding box obtained from the partial point cloud of an object, and the grasp quality metric mentioned above, we can generate a good grasping region on the bounding box that can be used to compute an antipodal grasp on the actual object. Our algorithm does not use any manually labeled data or grasping simulator, thus making it very efficient to implement and integrate with screw linear interpolation-based motion planners. We present simulation as well as experimental results that show the effectiveness of our approach. △ Less

Submitted 20 September, 2023; originally announced September 2023.

PeRP: Personalized Residual Policies For Congestion Mitigation Through Co-operative Advisory Systems

Authors: Aamir Hasan, Neeloy Chakraborty, Haonan Chen, Jung-Hoon Cho, Cathy Wu, Katherine Driggs-Campbell

Abstract: Intelligent driving systems can be used to mitigate congestion through simple actions, thus improving many socioeconomic factors such as commute time and gas costs. However, these systems assume precise control over autonomous vehicle fleets, and are hence limited in practice as they fail to account for uncertainty in human behavior. Piecewise Constant (PC) Policies address these issues by structu… ▽ More Intelligent driving systems can be used to mitigate congestion through simple actions, thus improving many socioeconomic factors such as commute time and gas costs. However, these systems assume precise control over autonomous vehicle fleets, and are hence limited in practice as they fail to account for uncertainty in human behavior. Piecewise Constant (PC) Policies address these issues by structurally modeling the likeness of human driving to reduce traffic congestion in dense scenarios to provide action advice to be followed by human drivers. However, PC policies assume that all drivers behave similarly. To this end, we develop a co-operative advisory system based on PC policies with a novel driver trait conditioned Personalized Residual Policy, PeRP. PeRP advises drivers to behave in ways that mitigate traffic congestion. We first infer the driver's intrinsic traits on how they follow instructions in an unsupervised manner with a variational autoencoder. Then, a policy conditioned on the inferred trait adapts the action of the PC policy to provide the driver with a personalized recommendation. Our system is trained in simulation with novel driver modeling of instruction adherence. We show that our approach successfully mitigates congestion while adapting to different driver behaviors, with 4 to 22% improvement in average speed over baselines. △ Less

Submitted 15 August, 2023; v1 submitted 1 August, 2023; originally announced August 2023.

Comments: Accepted to ITSC 2023. Additional material and code is available at the project webpage: https://sites.google.com/illinois.edu/perp

Enhanced Read Resolution in Reconfigurable Memristive Synapses for Spiking Neural Networks

Authors: Hritom Das, Nishith N. Chakraborty, Catherine Schuman, Garrett S. Rose

Abstract: Synapse is a key element of any neuromorphic computing system which is mostly constructed with memristor devices. A memristor is a two-terminal analog memory device. Memristive synapse suffers from various challenges such as forming at high voltage, SET, RESET failure, and READ margin or resolution issue between two weights. Enhanced READ resolution is very important to make a memristive synapse f… ▽ More Synapse is a key element of any neuromorphic computing system which is mostly constructed with memristor devices. A memristor is a two-terminal analog memory device. Memristive synapse suffers from various challenges such as forming at high voltage, SET, RESET failure, and READ margin or resolution issue between two weights. Enhanced READ resolution is very important to make a memristive synapse functionally reliable. Usually, the READ resolution is very small for a memristive synapse with 4-bit data precision. This work considers a step-by-step analysis to enhance the READ current resolution for a current-controlled memristor-based synapse. An empirical model is used to characterize the HfO2-based memristive device. 1st and 2nd stage device of our proposed synapse can be scaled to enhance the READ current margin up to ~ 4.3x and ~ 21% respectively. Moreover, READ current resolution can be enhanced with run-time adaptation features such as READ voltage scaling and body biasing. The READ voltage scaling and body biasing can improve the READ current resolution by about 46% and 15% respectively. TENNLabs' neuromorphic computing framework is leveraged to evaluate the effect of READ current resolution on classification applications. Higher READ current resolution shows better accuracy than lower resolution with different percentages of read noise scenarios. △ Less

Submitted 23 June, 2023; originally announced June 2023.

Journal ref: Scientific Reports 2024

An Efficient and Accurate Memristive Memory for Array-based Spiking Neural Networks

Authors: Hritom Das, Rocco D. Febbo, SNB Tushar, Nishith N. Chakraborty, Maximilian Liehr, Nathaniel Cady, Garrett S. Rose

Abstract: Memristors provide a tempting solution for weighted synapse connections in neuromorphic computing due to their size and non-volatile nature. However, memristors are unreliable in the commonly used voltage-pulse-based programming approaches and require precisely shaped pulses to avoid programming failure. In this paper, we demonstrate a current-limiting-based solution that provides a more predictab… ▽ More Memristors provide a tempting solution for weighted synapse connections in neuromorphic computing due to their size and non-volatile nature. However, memristors are unreliable in the commonly used voltage-pulse-based programming approaches and require precisely shaped pulses to avoid programming failure. In this paper, we demonstrate a current-limiting-based solution that provides a more predictable analog memory behavior when reading and writing memristive synapses. With our proposed design READ current can be optimized by about 19x compared to the 1T1R design. Moreover, our proposed design saves about 9x energy compared to the 1T1R design. Our 3T1R design also shows promising write operation which is less affected by the process variation in MOSFETs and the inherent stochastic behavior of memristors. Memristors used for testing are hafnium oxide based and were fabricated in a 65nm hybrid CMOS-memristor process. The proposed design also shows linear characteristics between the voltage applied and the resulting resistance for the writing operation. The simulation and measured data show similar patterns with respect to voltage pulse-based programming and current compliance-based programming. We further observed the impact of this behavior on neuromorphic-specific applications such as a spiking neural network △ Less

Submitted 6 September, 2023; v1 submitted 10 June, 2023; originally announced June 2023.

Journal ref: IEEE Transactions on Circuits and Systems I: Regular Papers, 10 August 2023

Optimizations for a Current-Controlled Memristor-based Neuromorphic Synapse Design

Authors: Hritom Das, Rocco D. Febbo, Charlie P. Rizzo, Nishith N. Chakraborty, James S. Plank, Garrett S. Rose

Abstract: The synapse is a key element of neuromorphic computing in terms of efficiency and accuracy. In this paper, an optimized current-controlled memristive synapse circuit is proposed. Our proposed synapse demonstrates reliability in the face of process variation and the inherent stochastic behavior of memristors. Up to an 82% energy optimization can be seen during the SET operation over prior work. In… ▽ More The synapse is a key element of neuromorphic computing in terms of efficiency and accuracy. In this paper, an optimized current-controlled memristive synapse circuit is proposed. Our proposed synapse demonstrates reliability in the face of process variation and the inherent stochastic behavior of memristors. Up to an 82% energy optimization can be seen during the SET operation over prior work. In addition, the READ process shows up to 54% energy savings. Our current-controlled approach also provides more reliable programming over traditional programming methods. This design is demonstrated with a 4-bit memory precision configuration. Using a spiking neural network (SNN), a neuromorphic application analysis was performed with this precision configuration. Our optimized design showed up to 82% improvement in control applications and a 2.7x improvement in classification applications compared with other design cases. △ Less

Submitted 7 September, 2023; v1 submitted 25 May, 2023; originally announced May 2023.

Journal ref: IEEE Journal on Emerging and Selected Topics in Circuits and Systems, 2023

Towards Co-operative Congestion Mitigation

Authors: Aamir Hasan, Neeloy Chakraborty, Cathy Wu, Katherine Driggs-Campbell

Abstract: The effects of traffic congestion are widespread and are an impedance to everyday life. Piecewise constant driving policies have shown promise in helping mitigate traffic congestion in simulation environments. However, no works currently test these policies in situations involving real human users. Thus, we propose to evaluate these policies through the use of a shared control framework in a colla… ▽ More The effects of traffic congestion are widespread and are an impedance to everyday life. Piecewise constant driving policies have shown promise in helping mitigate traffic congestion in simulation environments. However, no works currently test these policies in situations involving real human users. Thus, we propose to evaluate these policies through the use of a shared control framework in a collaborative experiment with the human driver and the driving policy aiming to co-operatively mitigate congestion. We intend to use the CARLA simulator alongside the Flow framework to conduct user studies to evaluate the affect of piecewise constant driving policies. As such, we present our in-progress work in building our framework and discuss our proposed plan on evaluating this framework through a human-in-the-loop simulation user study. △ Less

Submitted 17 February, 2023; originally announced February 2023.

Comments: Presented at the ICRA 2022 Workshop on Shared Autonomy in Physical Human-Robot Interaction: Adaptability and Trust

A Data-Efficient Visual-Audio Representation with Intuitive Fine-tuning for Voice-Controlled Robots

Authors: Peixin Chang, Shuijing Liu, Tianchen Ji, Neeloy Chakraborty, Kaiwen Hong, Katherine Driggs-Campbell

Abstract: A command-following robot that serves people in everyday life must continually improve itself in deployment domains with minimal help from its end users, instead of engineers. Previous methods are either difficult to continuously improve after the deployment or require a large number of new labels during fine-tuning. Motivated by (self-)supervised contrastive learning, we propose a novel represent… ▽ More A command-following robot that serves people in everyday life must continually improve itself in deployment domains with minimal help from its end users, instead of engineers. Previous methods are either difficult to continuously improve after the deployment or require a large number of new labels during fine-tuning. Motivated by (self-)supervised contrastive learning, we propose a novel representation that generates an intrinsic reward function for command-following robot tasks by associating images with sound commands. After the robot is deployed in a new domain, the representation can be updated intuitively and data-efficiently by non-experts without any hand-crafted reward functions. We demonstrate our approach on various sound types and robotic tasks, including navigation and manipulation with raw sensor inputs. In simulated and real-world experiments, we show that our system can continually self-improve in previously unseen scenarios given fewer new labeled data, while still achieving better performance over previous methods. △ Less

Submitted 16 October, 2023; v1 submitted 23 January, 2023; originally announced January 2023.

Comments: Published at Conference on Robot Learning (CoRL), 2023

Structural Attention-Based Recurrent Variational Autoencoder for Highway Vehicle Anomaly Detection

Authors: Neeloy Chakraborty, Aamir Hasan, Shuijing Liu, Tianchen Ji, Weihang Liang, D. Livingston McPherson, Katherine Driggs-Campbell

Abstract: In autonomous driving, detection of abnormal driving behaviors is essential to ensure the safety of vehicle controllers. Prior works in vehicle anomaly detection have shown that modeling interactions between agents improves detection accuracy, but certain abnormal behaviors where structured road information is paramount are poorly identified, such as wrong-way and off-road driving. We propose a no… ▽ More In autonomous driving, detection of abnormal driving behaviors is essential to ensure the safety of vehicle controllers. Prior works in vehicle anomaly detection have shown that modeling interactions between agents improves detection accuracy, but certain abnormal behaviors where structured road information is paramount are poorly identified, such as wrong-way and off-road driving. We propose a novel unsupervised framework for highway anomaly detection named Structural Attention-Based Recurrent VAE (SABeR-VAE), which explicitly uses the structure of the environment to aid anomaly identification. Specifically, we use a vehicle self-attention module to learn the relations among vehicles on a road, and a separate lane-vehicle attention module to model the importance of permissible lanes to aid in trajectory prediction. Conditioned on the attention modules' outputs, a recurrent encoder-decoder architecture with a stochastic Koopman operator-propagated latent space predicts the next states of vehicles. Our model is trained end-to-end to minimize prediction loss on normal vehicle behaviors, and is deployed to detect anomalies in (ab)normal scenarios. By combining the heterogeneous vehicle and lane information, SABeR-VAE and its deterministic variant, SABeR-AE, improve abnormal AUPR by 18% and 25% respectively on the simulated MAAD highway dataset over STGAE-KDE. Furthermore, we show that the learned Koopman operator in SABeR-VAE enforces interpretable structure in the variational latent space. The results of our method indeed show that modeling environmental factors is essential to detecting a diverse set of anomalies in deployment. For code implementation, please visit https://sites.google.com/illinois.edu/saber-vae. △ Less

Submitted 23 February, 2023; v1 submitted 9 January, 2023; originally announced January 2023.

Comments: 11 pages, 5 figures; Published as a full paper in IFAAMAS International Conference on Autonomous Agents and Multiagent Systems (AAMAS), 2023; Added appendix and discussion of Att-LSTM-VAE ablation

Human-Guided Planning for Complex Manipulation Tasks Using the Screw Geometry of Motion

Authors: Dasharadhan Mahalingam, Nilanjan Chakraborty

Abstract: In this paper, we present a novel method of motion planning for performing complex manipulation tasks by using human demonstration and exploiting the screw geometry of motion. We consider complex manipulation tasks where there are constraints on the motion of the end effector of the robot. Examples of such tasks include opening a door, opening a drawer, transferring granular material from one cont… ▽ More In this paper, we present a novel method of motion planning for performing complex manipulation tasks by using human demonstration and exploiting the screw geometry of motion. We consider complex manipulation tasks where there are constraints on the motion of the end effector of the robot. Examples of such tasks include opening a door, opening a drawer, transferring granular material from one container to another with a spoon, and loading dishes to a dishwasher. Our approach consists of two steps: First, using the fact that a motion in the task space of the robot can be approximated by using a sequence of constant screw motions, we segment a human demonstration into a sequence of constant screw motions. Second, we use the segmented screws to generate motion plans via screw-linear interpolation for other instances of the same task. The use of screw segmentation allows us to capture the invariants of the demonstrations in a coordinate-free fashion, thus allowing us to plan for different task instances from just one example. We present extensive experimental results on a variety of manipulation scenarios showing that our method can be used across a wide range of manipulation tasks. △ Less

Submitted 24 September, 2022; v1 submitted 12 September, 2022; originally announced September 2022.

Coordinate Invariant User-Guided Constrained Path Planning with Reactive Rapidly Expanding Plane-Oriented Escaping Trees

Authors: Riddhiman Laha, Ruiai Sun, Wenxi Wu, Dasharadhan Mahalingam, Nilanjan Chakraborty, Luis F. C. Figueredo, Sami Haddadin

Abstract: As collaborative robots move closer to human environments, motion generation and reactive planning strategies that allow for elaborate task execution with minimal easy-to-implement guidance whilst coping with changes in the environment is of paramount importance. In this paper, we present a novel approach for generating real-time motion plans for point-to-point tasks using a single successful huma… ▽ More As collaborative robots move closer to human environments, motion generation and reactive planning strategies that allow for elaborate task execution with minimal easy-to-implement guidance whilst coping with changes in the environment is of paramount importance. In this paper, we present a novel approach for generating real-time motion plans for point-to-point tasks using a single successful human demonstration. Our approach is based on screw linear interpolation,which allows us to respect the underlying geometric constraints that characterize the task and are implicitly present in the demonstration. We also integrate an original reactive collision avoidance approach with our planner. We present extensive experimental results to demonstrate that with our approach,by using a single demonstration of moving one block, we can generate motion plans for complex tasks like stacking multiple blocks (in a dynamic environment). Analogous generalization abilities are also shown for tasks like pouring and loading shelves. For the pouring task, we also show that a demonstration given for one-armed pouring can be used for planning pouring with a dual-armed manipulator of different kinematic structure. △ Less

Submitted 20 March, 2022; originally announced March 2022.

Comments: 8 pages, 7 figures, to be published in IEEE International Conference on Robotics and Automation (ICRA) 2022, Philadelphia (PA) USA

Intention Aware Robot Crowd Navigation with Attention-Based Interaction Graph

Authors: Shuijing Liu, Peixin Chang, Zhe Huang, Neeloy Chakraborty, Kaiwen Hong, Weihang Liang, D. Livingston McPherson, Junyi Geng, Katherine Driggs-Campbell

Abstract: We study the problem of safe and intention-aware robot navigation in dense and interactive crowds. Most previous reinforcement learning (RL) based methods fail to consider different types of interactions among all agents or ignore the intentions of people, which results in performance degradation. To learn a safe and efficient robot policy, we propose a novel recurrent graph neural network with at… ▽ More We study the problem of safe and intention-aware robot navigation in dense and interactive crowds. Most previous reinforcement learning (RL) based methods fail to consider different types of interactions among all agents or ignore the intentions of people, which results in performance degradation. To learn a safe and efficient robot policy, we propose a novel recurrent graph neural network with attention mechanisms to capture heterogeneous interactions among agents through space and time. To encourage longsighted robot behaviors, we infer the intentions of dynamic agents by predicting their future trajectories for several timesteps. The predictions are incorporated into a model-free RL framework to prevent the robot from intruding into the intended paths of other agents. We demonstrate that our method enables the robot to achieve good navigation performance and non-invasiveness in challenging crowd navigation scenarios. We successfully transfer the policy learned in simulation to a real-world TurtleBot 2i. Our code and videos are available at https://sites.google.com/view/intention-aware-crowdnav/home. △ Less

Submitted 24 April, 2023; v1 submitted 3 March, 2022; originally announced March 2022.

Comments: Published as a conference paper in IEEE International Conference on Robotics and Automation (ICRA), 2023

Blockchain-based Covid Vaccination Registration and Monitoring

Authors: Shirajus Salekin Nabil, Md. Sabbir Alam Pran, Ali Abrar Al Haque, Narayan Ranjan Chakraborty, Mohammad Jabed Morshed Chowdhury, Md Sadek Ferdous

Abstract: Covid-19 (SARS-CoV-2) has changed almost all the aspects of our living. Governments around the world have imposed lockdown to slow down the transmissions. In the meantime, researchers worked hard to find the vaccine. Fortunately, we have found the vaccine, in fact a good number of them. However, managing the testing and vaccination process of the total population is a mammoth job. There are multip… ▽ More Covid-19 (SARS-CoV-2) has changed almost all the aspects of our living. Governments around the world have imposed lockdown to slow down the transmissions. In the meantime, researchers worked hard to find the vaccine. Fortunately, we have found the vaccine, in fact a good number of them. However, managing the testing and vaccination process of the total population is a mammoth job. There are multiple government and private sector organisations that are working together to ensure proper testing and vaccination. However, there is always delay or data silo problems in multi-organisational works. Therefore, streamlining this process is vital to improve the efficiency and save more lives. It is already proved that technology has a significant impact on the health sector, including blockchain. Blockchain provides a distributed system along with greater privacy, transparency and authenticity. In this article, we have presented a blockchain-based system that seamlessly integrates testing and vaccination system, allowing the system to be transparent. The instant verification of any tamper-proof result and a transparent and efficient vaccination system have been exhibited and implemented in the research. We have also implemented the system as "Digital Vaccine Passport" (DVP) and analysed its performance. △ Less

Submitted 20 September, 2021; originally announced September 2021.

Comments: 12 pages

Learning to Navigate Intersections with Unsupervised Driver Trait Inference

Authors: Shuijing Liu, Peixin Chang, Haonan Chen, Neeloy Chakraborty, Katherine Driggs-Campbell

Abstract: Navigation through uncontrolled intersections is one of the key challenges for autonomous vehicles. Identifying the subtle differences in hidden traits of other drivers can bring significant benefits when navigating in such environments. We propose an unsupervised method for inferring driver traits such as driving styles from observed vehicle trajectories. We use a variational autoencoder with rec… ▽ More Navigation through uncontrolled intersections is one of the key challenges for autonomous vehicles. Identifying the subtle differences in hidden traits of other drivers can bring significant benefits when navigating in such environments. We propose an unsupervised method for inferring driver traits such as driving styles from observed vehicle trajectories. We use a variational autoencoder with recurrent neural networks to learn a latent representation of traits without any ground truth trait labels. Then, we use this trait representation to learn a policy for an autonomous vehicle to navigate through a T-intersection with deep reinforcement learning. Our pipeline enables the autonomous vehicle to adjust its actions when dealing with drivers of different traits to ensure safety and efficiency. Our method demonstrates promising performance and outperforms state-of-the-art baselines in the T-intersection scenario. △ Less

Submitted 28 February, 2022; v1 submitted 14 September, 2021; originally announced September 2021.

Comments: Published as a conference paper in IEEE International Conference on Robotics and Automation (ICRA), 2022

Computing a Task-Dependent Grasp Metric Using Second Order Cone Programs

Authors: Amin Fakhari, Aditya Patankar, Jiayin Xie, Nilanjan Chakraborty

Abstract: Evaluating a grasp generated by a set of hand-object contact locations is a key component of many grasp planning algorithms. In this paper, we present a novel second order cone program (SOCP) based optimization formulation for evaluating a grasps' ability to apply wrenches to generate a linear motion along a given direction and/or an angular motion about the given direction. Our quality measure ca… ▽ More Evaluating a grasp generated by a set of hand-object contact locations is a key component of many grasp planning algorithms. In this paper, we present a novel second order cone program (SOCP) based optimization formulation for evaluating a grasps' ability to apply wrenches to generate a linear motion along a given direction and/or an angular motion about the given direction. Our quality measure can be computed efficiently, since the SOCP is a convex optimization problem, which can be solved optimally with interior point methods. A key feature of our approach is that we can consider the effect of contact wrenches from any contact of the object with the environment. This is different from the extant literature where only the effect of finger-object contacts is considered. Exploiting the environmental contact is useful in many manipulation scenarios either to enhance the dexterity of simple hands or improve the payload capability of the manipulator. In contrast to most existing approaches, our approach also takes into account the practical constraint that the maximum contact force that can be applied at a finger-object contact can be different for each contact. We can also include the effect of external forces like gravity, as well as the joint torque constraints of the fingers/manipulators. Furthermore, for a given motion path as a constant screw motion or a sequence of constant screw motions, we can discretize the path and compute a global grasp metric to accomplish the whole task with a chosen set of finger-object contact locations. △ Less

Submitted 25 April, 2021; originally announced April 2021.

Comments: 8 pages, Pre-print, Submitted to IROS 2021

Task Space Planning with Complementarity Constraint-based Obstacle Avoidance

Authors: Anirban Sinha, Anik Sarker, Nilanjan Chakraborty

Abstract: In this paper, we present a task space-based local motion planner that incorporates collision avoidance and constraints on end-effector motion during the execution of a task. Our key technical contribution is the development of a novel kinematic state evolution model of the robot where the collision avoidance is encoded as a complementarity constraint. We show that the kinematic state evolution wi… ▽ More In this paper, we present a task space-based local motion planner that incorporates collision avoidance and constraints on end-effector motion during the execution of a task. Our key technical contribution is the development of a novel kinematic state evolution model of the robot where the collision avoidance is encoded as a complementarity constraint. We show that the kinematic state evolution with collision avoidance can be represented as a Linear Complementarity Problem (LCP). Using the LCP model along with Screw Linear Interpolation (ScLERP) in SE(3), we show that it may be possible to compute a path between two given task space poses by directly moving from the start to the goal pose, even if there are potential collisions with obstacles. The scalability of the planner is demonstrated with experiments using a physical robot. We present simulation and experimental results with both collision avoidance and task constraints to show the efficacy of our approach. △ Less

Submitted 15 April, 2021; originally announced April 2021.

Motion and Force Planning for Manipulating Heavy Objects by Pivoting

Authors: Amin Fakhari, Aditya Patankar, Nilanjan Chakraborty

Abstract: Manipulation of objects by exploiting their contact with the environment can enhance both the dexterity and payload capability of robotic manipulators. A common way to manipulate heavy objects beyond the payload capability of a robot is to use a sequence of pivoting motions, wherein, an object is moved while some contact points between the object and a support surface are kept fixed. The goal of t… ▽ More Manipulation of objects by exploiting their contact with the environment can enhance both the dexterity and payload capability of robotic manipulators. A common way to manipulate heavy objects beyond the payload capability of a robot is to use a sequence of pivoting motions, wherein, an object is moved while some contact points between the object and a support surface are kept fixed. The goal of this paper is to develop an algorithmic approach for automated plan generation for object manipulation with a sequence of pivoting motions. A plan for manipulating a heavy object consists of a sequence of joint angles of the manipulator, the corresponding object poses, as well as the joint torques required to move the object. The constraint of maintaining object contact with the ground during manipulation results in nonlinear constraints in the configuration space of the robot, which is challenging for motion planning algorithms. Exploiting the fact that pivoting motion corresponds to movements in a subgroup of the group of rigid body motions, SE(3), we present a novel task-space based planning approach for computing a motion plan for both the manipulator and the object while satisfying contact constraints. We also combine our motion planning algorithm with a grasping force synthesis algorithm to ensure that friction constraints at the contacts and actuator torque constraints are satisfied. We present simulation results with a dual-armed Baxter robot to demonstrate our approach. △ Less

Submitted 25 April, 2021; v1 submitted 10 December, 2020; originally announced December 2020.

Comments: 8 pages, Pre-print, Submitted to IROS 2021

Decentralized Structural-RNN for Robot Crowd Navigation with Deep Reinforcement Learning

Authors: Shuijing Liu, Peixin Chang, Weihang Liang, Neeloy Chakraborty, Katherine Driggs-Campbell

Abstract: Safe and efficient navigation through human crowds is an essential capability for mobile robots. Previous work on robot crowd navigation assumes that the dynamics of all agents are known and well-defined. In addition, the performance of previous methods deteriorates in partially observable environments and environments with dense crowds. To tackle these problems, we propose decentralized structura… ▽ More Safe and efficient navigation through human crowds is an essential capability for mobile robots. Previous work on robot crowd navigation assumes that the dynamics of all agents are known and well-defined. In addition, the performance of previous methods deteriorates in partially observable environments and environments with dense crowds. To tackle these problems, we propose decentralized structural-Recurrent Neural Network (DS-RNN), a novel network that reasons about spatial and temporal relationships for robot decision making in crowd navigation. We train our network with model-free deep reinforcement learning without any expert supervision. We demonstrate that our model outperforms previous methods in challenging crowd navigation scenarios. We successfully transfer the policy learned in the simulator to a real-world TurtleBot 2i. For more information, please visit the project website at https://sites.google.com/view/crowdnav-ds-rnn/home. △ Less

Submitted 27 January, 2025; v1 submitted 9 November, 2020; originally announced November 2020.

Comments: Published as a conference paper in IEEE International Conference on Robotics and Automation (ICRA), 2021

Dynamic Simulation-Guided Design of Tumbling Magnetic Microrobots

Authors: Jiayin Xie, Chenghao Bi, David J. Cappelleri, Nilanjan Chakraborty

Abstract: Design of robots at the small scale is a trial-and-error based process, which is costly and time-consuming. There are few dynamic simulation tools available to accurately predict the motion or performance of untethered microrobots as they move over a substrate. At smaller length scales, the influence of adhesion and friction, which scales with surface area, becomes more pronounced. Thus, rigid bod… ▽ More Design of robots at the small scale is a trial-and-error based process, which is costly and time-consuming. There are few dynamic simulation tools available to accurately predict the motion or performance of untethered microrobots as they move over a substrate. At smaller length scales, the influence of adhesion and friction, which scales with surface area, becomes more pronounced. Thus, rigid body dynamic simulators, which implicitly assume that contact between two bodies can be modeled as point contact are not suitable. In this paper, we present techniques for simulating the motion of microrobots where there can be intermittent and non-point contact between the robot and the substrate. We use these techniques to study the motion of tumbling microrobots of different shapes and select shapes that are optimal for improving locomotion performance. Simulation results are verified using experimental data on linear velocity, maximum climbable incline angle, and microrobot trajectory. Microrobots with improved geometry were fabricated, but limitations in the fabrication process resulted in unexpected manufacturing errors and material/size scale adjustments. The developed simulation model is able to incorporate these limitations and emulate their effect on the microrobot's motion, reproducing the experimental behavior of the tumbling microrobots, further showcasing the effectiveness of having such a dynamic model. △ Less

Submitted 5 October, 2020; originally announced October 2020.

Comments: arXiv admin note: substantial text overlap with arXiv:1907.12699

Modeling and Prediction of Rigid Body Motion with Planar Non-Convex Contact

Authors: Jiayin Xie, Nilanjan Chakraborty

Abstract: We present a principled method for motion prediction via dynamic simulation for rigid bodies in intermittent contact with each other where the contact region is a planar non-convex contact patch. Such methods are useful in planning and control for robotic manipulation. The planar non-convex contact patch can either be a topologically connected set or disconnected set. Most work in rigid body dyn… ▽ More We present a principled method for motion prediction via dynamic simulation for rigid bodies in intermittent contact with each other where the contact region is a planar non-convex contact patch. Such methods are useful in planning and control for robotic manipulation. The planar non-convex contact patch can either be a topologically connected set or disconnected set. Most work in rigid body dynamic simulation assume that the contact between objects is a point contact, which may not be valid in many applications. In this paper, by using the convex hull of the contact patch, we build on our recent work on simulating rigid bodies with convex contact patches for simulating motion of objects with planar non-convex contact patches. We formulate a discrete-time mixed complementarity problem where we solve the contact detection and integration of the equations of motion simultaneously. We solve for the equivalent contact point (ECP) and contact impulse of each contact patch simultaneously along with the state, i.e., configuration and velocity of the objects. We prove that although we are representing a patch contact by an equivalent point, our model for enforcing non-penetration constraints ensure that there is no artificial penetration between the contacting rigid bodies. We provide empirical evidence to show that our method can seamlessly capture transition among different contact modes like patch contact, multiple or single point contact. △ Less

Submitted 5 October, 2020; originally announced October 2020.

Comments: arXiv admin note: substantial text overlap with arXiv:1904.07723

Rigid Body Dynamic Simulation with Line and Surface Contact

Authors: Jiayin Xie, Nilanjan Chakraborty

Abstract: In this paper, we develop a principled method to model line and surface contact with point contact (we call this point, equivalent contact point) that is consistent with physics-based models of surface (line) contact. Assuming that the set of contact points form a convex set, we solve the contact detection and dynamic simulation step simultaneously by formulating the problem as a mixed nonlinear c… ▽ More In this paper, we develop a principled method to model line and surface contact with point contact (we call this point, equivalent contact point) that is consistent with physics-based models of surface (line) contact. Assuming that the set of contact points form a convex set, we solve the contact detection and dynamic simulation step simultaneously by formulating the problem as a mixed nonlinear complementarity problem. This allows us to simultaneously compute the equivalent contact point as well as the wrenches (forces and moments) at the equivalent contact point (consistent with the friction model) along with the configuration and velocities of the rigid objects. Furthermore, we prove that the contact constraints of no inter-penetration between the objects is also satisfied. We present a geometrically implicit time-stepping scheme for dynamic simulation for contacts between two bodies with convex contact area, which includes line contact and surface contact. We prove that for surface and line contact, for any value of the velocity of center of mass of the object, there is a unique solution for contact point and contact wrench that satisfies the discrete-time equations of motion. Simulation examples are shown to demonstrate the validity of our approach and show that with our approach we can seamlessly transition between point, line, and surface contact. △ Less

Submitted 7 October, 2020; v1 submitted 5 October, 2020; originally announced October 2020.

A Framework for Causal Discovery in non-intervenable systems

Authors: Peter Jan van Leeuwen, Michael DeCaria, Nachiketa Chakaborty, Manuel Pulido

Abstract: Many frameworks exist to infer cause and effect relations in complex nonlinear systems but a complete theory is lacking. A new framework is presented that is fully nonlinear, provides a complete information theoretic disentanglement of causal processes, allows for nonlinear interactions between causes, identifies the causal strength of missing or unknown processes, and can analyze systems that can… ▽ More Many frameworks exist to infer cause and effect relations in complex nonlinear systems but a complete theory is lacking. A new framework is presented that is fully nonlinear, provides a complete information theoretic disentanglement of causal processes, allows for nonlinear interactions between causes, identifies the causal strength of missing or unknown processes, and can analyze systems that cannot be represented on Directed Acyclic Graphs. The basic building blocks are information theoretic measures such as (conditional) mutual information and a new concept called certainty that monotonically increases with the information available about the target process. The framework is presented in detail and compared with other existing frameworks, and the treatment of confounders is discussed. While there are systems with structures that the framework cannot disentangle, it is argued that any causal framework that is based on integrated quantities will miss out potentially important information of the underlying probability density functions. The framework is tested on several highly simplified stochastic processes to demonstrate how blocking and gateways are handled, and on the chaotic Lorentz 1963 system. We show that the framework provides information on the local dynamics, but also reveals information on the larger scale structure of the underlying attractor. Furthermore, by applying it to real observations related to the El-Nino-Southern-Oscillation system we demonstrate its power and advantage over other methodologies. △ Less

Submitted 27 September, 2021; v1 submitted 5 October, 2020; originally announced October 2020.

Comments: 45 pages, 9 figures

Robust Relative Hand Placement For Bi-Manual Tasks

Authors: Anirban Sinha, Nilanjan Chakraborty

Abstract: In many bi-manual robotic tasks, like peg-in-a-hole assembly, the success of the task execution depends on the error in achieving the desired relative pose between the peg and the hole in a pre-insertion configuration. Random actuation errors in the joint space usually prevent the two arms from reaching their desired task space poses, which in turn results in a random error in relative pose betwee… ▽ More In many bi-manual robotic tasks, like peg-in-a-hole assembly, the success of the task execution depends on the error in achieving the desired relative pose between the peg and the hole in a pre-insertion configuration. Random actuation errors in the joint space usually prevent the two arms from reaching their desired task space poses, which in turn results in a random error in relative pose between the two hands. This random error varies from trial to trial, and thus depending on the tolerance between the peg and the hole, the outcome of the assembly task may be random (sometimes the task execution succeeds and sometimes it fails). In general, since the relative pose has $6$ degrees-of-freedom, there are infinite numbers of joint space solutions for the two arms that correspond to the same task space relative pose. However, in the presence of actuation errors, the joint space solutions are not all identical since they map the joint space error sets differently to the task space. Thus, the goal of this paper is to develop a methodical approach to compute a joint space solution such that the maximum task space error is below a (specified) threshold with high probability. Such a solution is called a robust inverse kinematics solution for the bi-manual robot. Our proposed method also allows the robot to self-evaluate whether it can perform a given bi-manual task reliably. We use a square peg-in-a-hole assembly scenario on the dual-arm Baxter robot for numerical simulations that shows the utility of our approach. △ Less

Submitted 25 June, 2020; originally announced June 2020.

Computing Robust Inverse Kinematics Under Uncertainty

Authors: Anirban Sinha, Nilanjan Chakraborty

Abstract: Robotic tasks, like reaching a pre-grasp configuration, are specified in the end effector space or task space, whereas, robot motion is controlled in joint space. Because of inherent actuation errors in joint space, robots cannot achieve desired configurations in task space exactly. Furthermore, different inverse kinematics (IK) solutions map joint space error set to task space differently. Thus f… ▽ More Robotic tasks, like reaching a pre-grasp configuration, are specified in the end effector space or task space, whereas, robot motion is controlled in joint space. Because of inherent actuation errors in joint space, robots cannot achieve desired configurations in task space exactly. Furthermore, different inverse kinematics (IK) solutions map joint space error set to task space differently. Thus for a given task with a prescribed error tolerance, all IK solutions will not be guaranteed to successfully execute the task. Any IK solution that is guaranteed to execute a task (possibly with high probability) irrespective of the realization of the joint space error is called a robust IK solution. In this paper we formulate and solve the robust inverse kinematics problem for redundant manipulators with actuation uncertainties (errors). We also present simulation and experimental results on a $7$-DoF redundant manipulator for two applications, namely, a pre-grasp positioning and a pre-insertion positioning scenario. Our results show that the robust IK solutions result in higher success rates and also allows the robot to self-evaluate how successful it might be in any application scenario. △ Less

Submitted 23 October, 2019; originally announced October 2019.

Towards Dynamic Simulation Guided Optimal Design of Tumbling Microrobots

Authors: Jiayin Xie, Chenghao Bi, David J. Cappelleri, Nilanjan Chakraborty

Abstract: Design of robots at the small scale is a trial-and-error based process, which is costly and time-consuming. There are no good dynamic simulation tools to predict the motion or performance of a microrobot as it moves against a substrate. At smaller length scales, the influence of adhesion and friction, which scales with surface area, becomes more pronounced. Thus, rigid body dynamic simulators, whi… ▽ More Design of robots at the small scale is a trial-and-error based process, which is costly and time-consuming. There are no good dynamic simulation tools to predict the motion or performance of a microrobot as it moves against a substrate. At smaller length scales, the influence of adhesion and friction, which scales with surface area, becomes more pronounced. Thus, rigid body dynamic simulators, which implicitly assume that contact between two bodies can be modeled as point contact are not suitable. In this paper, we present techniques for simulating the motion of microrobots where there can be intermittent and non-point contact between the robot and the substrate. We use this simulator to study the motion of microrobots of different shapes and select shapes that are most promising for performing a given task. △ Less

Submitted 29 July, 2019; originally announced July 2019.

Introduction to Neural Network based Approaches for Question Answering over Knowledge Graphs

Authors: Nilesh Chakraborty, Denis Lukovnikov, Gaurav Maheshwari, Priyansh Trivedi, Jens Lehmann, Asja Fischer

Abstract: Question answering has emerged as an intuitive way of querying structured data sources, and has attracted significant advancements over the years. In this article, we provide an overview over these recent advancements, focusing on neural network based question answering systems over knowledge graphs. We introduce readers to the challenges in the tasks, current paradigms of approaches, discuss nota… ▽ More Question answering has emerged as an intuitive way of querying structured data sources, and has attracted significant advancements over the years. In this article, we provide an overview over these recent advancements, focusing on neural network based question answering systems over knowledge graphs. We introduce readers to the challenges in the tasks, current paradigms of approaches, discuss notable advancements, and outline the emerging trends in the field. Through this article, we aim to provide newcomers to the field with a suitable entry point, and ease their process of making informed decisions while creating their own QA system. △ Less

Submitted 22 July, 2019; originally announced July 2019.

Comments: Preprint, under review. The first four authors contributed equally to this paper, and should be regarded as co-first authors

Rigid Body Motion Prediction with Planar Non-convex Contact Patch

Authors: Jiayin Xie, Nilanjan Chakraborty

Abstract: We present a principled method for motion prediction via dynamic simulation for rigid bodies in intermittent contact with each other where the contact is assumed to be a planar non-convex contact patch. The planar non-convex contact patch can either be a topologically connected set or disconnected set. Such algorithms are useful in planning and control for robotic manipulation. Most works in rigid… ▽ More We present a principled method for motion prediction via dynamic simulation for rigid bodies in intermittent contact with each other where the contact is assumed to be a planar non-convex contact patch. The planar non-convex contact patch can either be a topologically connected set or disconnected set. Such algorithms are useful in planning and control for robotic manipulation. Most works in rigid body dynamic simulation assume that the contact between objects is a point contact, which may not be valid in many applications. In this paper, by using the convex hull of the contact patch, we build on our recent work on simulating rigid bodies with convex contact patches, for simulating the motion of objects with planar non-convex contact patches. We formulate a discrete-time mixed complementarity problem where we solve the contact detection and integration of the equations of motion simultaneously. Thus, our method is a geometrically-implicit method and we prove that in our formulation, there is no artificial penetration between the contacting rigid bodies. We solve for the equivalent contact point (ECP) and contact impulse of each contact patch simultaneously along with the state, i.e., configuration and velocity of the objects. We provide empirical evidence to show that our method can seamlessly capture the transition between different contact modes like patch contact to multiple or single point contact during the simulation. △ Less

Submitted 15 April, 2019; originally announced April 2019.

Comments: arXiv admin note: text overlap with arXiv:1809.05555

Journal ref: 2019 IEEE International Conference on Robotics and Automation

Translating Natural Language to SQL using Pointer-Generator Networks and How Decoding Order Matters

Authors: Denis Lukovnikov, Nilesh Chakraborty, Jens Lehmann, Asja Fischer

Abstract: Translating natural language to SQL queries for table-based question answering is a challenging problem and has received significant attention from the research community. In this work, we extend a pointer-generator and investigate the order-matters problem in semantic parsing for SQL. Even though our model is a straightforward extension of a general-purpose pointer-generator, it outperforms early… ▽ More Translating natural language to SQL queries for table-based question answering is a challenging problem and has received significant attention from the research community. In this work, we extend a pointer-generator and investigate the order-matters problem in semantic parsing for SQL. Even though our model is a straightforward extension of a general-purpose pointer-generator, it outperforms early works for WikiSQL and remains competitive to concurrently introduced, more complex models. Moreover, we provide a deeper investigation of the potential order-matters problem that could arise due to having multiple correct decoding paths, and investigate the use of REINFORCE as well as a dynamic oracle in this context. △ Less

Submitted 13 November, 2018; originally announced November 2018.

Learning to Rank Query Graphs for Complex Question Answering over Knowledge Graphs

Authors: Gaurav Maheshwari, Priyansh Trivedi, Denis Lukovnikov, Nilesh Chakraborty, Asja Fischer, Jens Lehmann

Abstract: In this paper, we conduct an empirical investigation of neural query graph ranking approaches for the task of complex question answering over knowledge graphs. We experiment with six different ranking models and propose a novel self-attention based slot matching model which exploits the inherent structure of query graphs, our logical form of choice. Our proposed model generally outperforms the oth… ▽ More In this paper, we conduct an empirical investigation of neural query graph ranking approaches for the task of complex question answering over knowledge graphs. We experiment with six different ranking models and propose a novel self-attention based slot matching model which exploits the inherent structure of query graphs, our logical form of choice. Our proposed model generally outperforms the other models on two QA datasets over the DBpedia knowledge graph, evaluated in different settings. In addition, we show that transfer learning from the larger of those QA datasets to the smaller dataset yields substantial improvements, effectively offsetting the general lack of training data. △ Less

Submitted 2 November, 2018; originally announced November 2018.

Rigid Body Dynamic Simulation with Multiple Convex Contact Patches

Authors: Jiayin Xie, Nilanjan Chakraborty

Abstract: We present a principled method for dynamic simulation of rigid bodies in intermittent contact with each other where the contact is assumed to be a non-convex contact patch that can be modeled as a union of convex patches. The prevalent assumption in simulating rigid bodies undergoing intermittent contact with each other is that the contact is a point contact. In recent work, we introduced an appro… ▽ More We present a principled method for dynamic simulation of rigid bodies in intermittent contact with each other where the contact is assumed to be a non-convex contact patch that can be modeled as a union of convex patches. The prevalent assumption in simulating rigid bodies undergoing intermittent contact with each other is that the contact is a point contact. In recent work, we introduced an approach to simulate contacting rigid bodies with convex contact patches (line and surface contact). In this paper, for non-convex contact patches modeled as a union of convex patches, we formulate a discrete-time mixed complementarity problem where we solve the contact detection and integration of the equations of motion simultaneously. Thus, our method is a geometrically-implicit method and we prove that in our formulation, there is no artificial penetration between the contacting rigid bodies. We solve for the equivalent contact point (ECP) and contact impulse of each contact patch simultaneously along with the state, i.e., configuration and velocity of the objects. We provide empirical evidence to show that if the number of contact patches between two objects is less than or equal to three, the state evolution of the bodies is unique, although the contact impulses and ECP may not be unique. We also present simulation results showing that our method can seamlessly capture transition between different contact modes like non-convex patch to point (or line contact) and vice-versa during simulation. △ Less

Submitted 14 September, 2018; originally announced September 2018.