-
Can Large Language Models Learn Formal Logic? A Data-Driven Training and Evaluation Framework
Authors:
Yuan Xia,
Akanksha Atrey,
Fadoua Khmaissia,
Kedar S. Namjoshi
Abstract:
This paper investigates the logical reasoning capabilities of large language models (LLMs). For a precisely defined yet tractable formulation, we choose the conceptually simple but technically complex task of constructing proofs in Boolean logic. A trained LLM receives as input a set of assumptions and a goal, and produces as output a proof that formally derives the goal from the assumptions. Inco…
▽ More
This paper investigates the logical reasoning capabilities of large language models (LLMs). For a precisely defined yet tractable formulation, we choose the conceptually simple but technically complex task of constructing proofs in Boolean logic. A trained LLM receives as input a set of assumptions and a goal, and produces as output a proof that formally derives the goal from the assumptions. Incorrect proofs are caught by an automated proof checker. A critical obstacle for training is the scarcity of real-world proofs. We propose an efficient, randomized procedure for synthesizing valid proofs and introduce Template Transformation, a data augmentation technique that enhances the model's ability to handle complex logical expressions. The central evaluation question is whether an LLM has indeed learned to reason. We propose tests to measure the reasoning ability of a black-box LLM. By these measures, experiments demonstrate strong reasoning capabilities for assertions with short proofs, which decline with proof complexity. Notably, template transformation improves accuracy even for smaller models, suggesting its effectiveness across model scales.
△ Less
Submitted 28 April, 2025;
originally announced April 2025.
-
W4-Groups: Modeling the Who, What, When and Where of Group Behavior via Mobility Sensing
Authors:
Akanksha Atrey,
Camellia Zakaria,
Rajesh Balan,
Prashant Shenoy
Abstract:
Human social interactions occur in group settings of varying sizes and locations, depending on the type of social activity. The ability to distinguish group formations based on their purposes transforms how group detection mechanisms function. Not only should such tools support the effective detection of serendipitous encounters, but they can derive categories of relation types among users. Determ…
▽ More
Human social interactions occur in group settings of varying sizes and locations, depending on the type of social activity. The ability to distinguish group formations based on their purposes transforms how group detection mechanisms function. Not only should such tools support the effective detection of serendipitous encounters, but they can derive categories of relation types among users. Determining who is involved, what activity is performed, and when and where the activity occurs are critical to understanding group processes in greater depth, including supporting goal-oriented applications (e.g., performance, productivity, and mental health) that require sensing social factors. In this work, we propose W4-Groups that captures the functional perspective of variability and repeatability when automatically constructing short-term and long-term groups via multiple data sources (e.g., WiFi and location check-in data). We design and implement W4-Groups to detect and extract all four group features who-what-when-where from the user's daily mobility patterns. We empirically evaluate the framework using two real-world WiFi datasets and a location check-in dataset, yielding an average of 92% overall accuracy, 96% precision, and 94% recall. Further, we supplement two case studies to demonstrate the application of W4-Groups for next-group activity prediction and analyzing changes in group behavior at a longitudinal scale, exemplifying short-term and long-term occurrences.
△ Less
Submitted 8 January, 2024; v1 submitted 22 December, 2023;
originally announced December 2023.
-
SODA: Protecting Proprietary Information in On-Device Machine Learning Models
Authors:
Akanksha Atrey,
Ritwik Sinha,
Saayan Mitra,
Prashant Shenoy
Abstract:
The growth of low-end hardware has led to a proliferation of machine learning-based services in edge applications. These applications gather contextual information about users and provide some services, such as personalized offers, through a machine learning (ML) model. A growing practice has been to deploy such ML models on the user's device to reduce latency, maintain user privacy, and minimize…
▽ More
The growth of low-end hardware has led to a proliferation of machine learning-based services in edge applications. These applications gather contextual information about users and provide some services, such as personalized offers, through a machine learning (ML) model. A growing practice has been to deploy such ML models on the user's device to reduce latency, maintain user privacy, and minimize continuous reliance on a centralized source. However, deploying ML models on the user's edge device can leak proprietary information about the service provider. In this work, we investigate on-device ML models that are used to provide mobile services and demonstrate how simple attacks can leak proprietary information of the service provider. We show that different adversaries can easily exploit such models to maximize their profit and accomplish content theft. Motivated by the need to thwart such attacks, we present an end-to-end framework, SODA, for deploying and serving on edge devices while defending against adversarial usage. Our results demonstrate that SODA can detect adversarial usage with 89% accuracy in less than 50 queries with minimal impact on service performance, latency, and storage.
△ Less
Submitted 22 December, 2023;
originally announced December 2023.
-
Preserving Privacy in Personalized Models for Distributed Mobile Services
Authors:
Akanksha Atrey,
Prashant Shenoy,
David Jensen
Abstract:
The ubiquity of mobile devices has led to the proliferation of mobile services that provide personalized and context-aware content to their users. Modern mobile services are distributed between end-devices, such as smartphones, and remote servers that reside in the cloud. Such services thrive on their ability to predict future contexts to pre-fetch content or make context-specific recommendations.…
▽ More
The ubiquity of mobile devices has led to the proliferation of mobile services that provide personalized and context-aware content to their users. Modern mobile services are distributed between end-devices, such as smartphones, and remote servers that reside in the cloud. Such services thrive on their ability to predict future contexts to pre-fetch content or make context-specific recommendations. An increasingly common method to predict future contexts, such as location, is via machine learning (ML) models. Recent work in context prediction has focused on ML model personalization where a personalized model is learned for each individual user in order to tailor predictions or recommendations to a user's mobile behavior. While the use of personalized models increases efficacy of the mobile service, we argue that it increases privacy risk since a personalized model encodes contextual behavior unique to each user. To demonstrate these privacy risks, we present several attribute inference-based privacy attacks and show that such attacks can leak privacy with up to 78% efficacy for top-3 predictions. We present Pelican, a privacy-preserving personalization system for context-aware mobile services that leverages both device and cloud resources to personalize ML models while minimizing the risk of privacy leakage for users. We evaluate Pelican using real world traces for location-aware mobile services and show that Pelican can substantially reduce privacy leakage by up to 75%.
△ Less
Submitted 21 April, 2021; v1 submitted 14 January, 2021;
originally announced January 2021.
-
New Frontiers in IoT: Networking, Systems, Reliability, and Security Challenges
Authors:
Saurabh Bagchi,
Tarek F. Abdelzaher,
Ramesh Govindan,
Prashant Shenoy,
Akanksha Atrey,
Pradipta Ghosh,
Ran Xu
Abstract:
The field of IoT has blossomed and is positively influencing many application domains. In this paper, we bring out the unique challenges this field poses to research in computer systems and networking. The unique challenges arise from the unique characteristics of IoT systems such as the diversity of application domains where they are used and the increasingly demanding protocols they are being ca…
▽ More
The field of IoT has blossomed and is positively influencing many application domains. In this paper, we bring out the unique challenges this field poses to research in computer systems and networking. The unique challenges arise from the unique characteristics of IoT systems such as the diversity of application domains where they are used and the increasingly demanding protocols they are being called upon to run (such as, video and LIDAR processing) on constrained resources (on-node and network). We show how these open challenges can benefit from foundations laid in other areas, such as, 5G cellular protocols, ML model reduction, and device-edge-cloud offloading. We then discuss the unique challenges for reliability, security, and privacy posed by IoT systems due to their salient characteristics which include heterogeneity of devices and protocols, dependence on the physical environment, and the close coupling with humans. We again show how the open research challenges benefit from reliability, security, and privacy advancements in other areas. We conclude by providing a vision for a desirable end state for IoT systems.
△ Less
Submitted 14 May, 2020;
originally announced May 2020.
-
Exploratory Not Explanatory: Counterfactual Analysis of Saliency Maps for Deep Reinforcement Learning
Authors:
Akanksha Atrey,
Kaleigh Clary,
David Jensen
Abstract:
Saliency maps are frequently used to support explanations of the behavior of deep reinforcement learning (RL) agents. However, a review of how saliency maps are used in practice indicates that the derived explanations are often unfalsifiable and can be highly subjective. We introduce an empirical approach grounded in counterfactual reasoning to test the hypotheses generated from saliency maps and…
▽ More
Saliency maps are frequently used to support explanations of the behavior of deep reinforcement learning (RL) agents. However, a review of how saliency maps are used in practice indicates that the derived explanations are often unfalsifiable and can be highly subjective. We introduce an empirical approach grounded in counterfactual reasoning to test the hypotheses generated from saliency maps and assess the degree to which they correspond to the semantics of RL environments. We use Atari games, a common benchmark for deep RL, to evaluate three types of saliency maps. Our results show the extent to which existing claims about Atari games can be evaluated and suggest that saliency maps are best viewed as an exploratory tool rather than an explanatory tool.
△ Less
Submitted 20 February, 2020; v1 submitted 9 December, 2019;
originally announced December 2019.
-
Measuring and Characterizing Generalization in Deep Reinforcement Learning
Authors:
Sam Witty,
Jun Ki Lee,
Emma Tosch,
Akanksha Atrey,
Michael Littman,
David Jensen
Abstract:
Deep reinforcement-learning methods have achieved remarkable performance on challenging control tasks. Observations of the resulting behavior give the impression that the agent has constructed a generalized representation that supports insightful action decisions. We re-examine what is meant by generalization in RL, and propose several definitions based on an agent's performance in on-policy, off-…
▽ More
Deep reinforcement-learning methods have achieved remarkable performance on challenging control tasks. Observations of the resulting behavior give the impression that the agent has constructed a generalized representation that supports insightful action decisions. We re-examine what is meant by generalization in RL, and propose several definitions based on an agent's performance in on-policy, off-policy, and unreachable states. We propose a set of practical methods for evaluating agents with these definitions of generalization. We demonstrate these techniques on a common benchmark task for deep RL, and we show that the learned networks make poor decisions for states that differ only slightly from on-policy states, even though those states are not selected adversarially. Taken together, these results call into question the extent to which deep Q-networks learn generalized representations, and suggest that more experimentation and analysis is necessary before claims of representation learning can be supported.
△ Less
Submitted 11 December, 2018; v1 submitted 6 December, 2018;
originally announced December 2018.
-
An Efficient and Secure Routing Protocol for Mobile Ad-Hoc Networks
Authors:
N. Ch. Sriman Narayana Iyengar,
Syed Mohammad Ansar Sachin kumar,
Piyush Nagar,
Siddharth Sharma,
Akshay Atrey
Abstract:
Efficiency and simplicity of random algorithms have made them a lucrative alternative for solving complex problems in the domain of communication networks. This paper presents a random algorithm for handling the routing problem in Mobile Ad hoc Networks [MANETS].The performance of most existing routing protocols for MANETS degrades in terms of packet delay and congestion caused as the number of mo…
▽ More
Efficiency and simplicity of random algorithms have made them a lucrative alternative for solving complex problems in the domain of communication networks. This paper presents a random algorithm for handling the routing problem in Mobile Ad hoc Networks [MANETS].The performance of most existing routing protocols for MANETS degrades in terms of packet delay and congestion caused as the number of mobile nodes increases beyond a certain level or their speed passes a certain level. As the network becomes more and more dynamic, congestion in network increases due to control packets generated by the routing protocols in the process of route discovery and route maintenance. Most of this congestion is due to flooding mechanism used in protocols like AODV and DSDV for the purpose of route discovery and route maintenance or for route discovery as in the case of DSR protocol. This paper introduces the concept of random routing algorithm that neither maintains a routing table nor floods the entire network as done by various known protocols thereby reducing the load on network in terms of number of control packets in a highly dynamic scenario. This paper calculates the expected run time of the designed random algorithm.
△ Less
Submitted 11 May, 2010;
originally announced May 2010.
