-
What Sensors See, What People Feel: Exploring Subjective Collaboration Perception in Mixed Reality
Authors:
Yasra Chandio,
Diana Romero,
Salma Elmalaki,
Fatima Anwar
Abstract:
Mixed Reality (MR) enables rich, embodied collaboration, yet it's uncertain if sensor and system-logged behavioral signals capture how users experience that collaboration. This disconnect stems from a fundamental gap: behavioral signals are observable and continuous, while collaboration is interpreted subjectively, shaped by internal states like presence, cognitive availability, and social awarene…
▽ More
Mixed Reality (MR) enables rich, embodied collaboration, yet it's uncertain if sensor and system-logged behavioral signals capture how users experience that collaboration. This disconnect stems from a fundamental gap: behavioral signals are observable and continuous, while collaboration is interpreted subjectively, shaped by internal states like presence, cognitive availability, and social awareness. Our core insight is that sensor signals serve as observable manifestations of subjective experiences in MR collaboration, and they can be captured through sensor data such as shared gaze, speech, spatial movement, and other system-logged performance metrics. We propose the Sensor-to-Subjective (S2S) Mapping Framework, a conceptual model that links observable interaction patterns to users' subjective perceptions of collaboration and internal cognitive states through sensor-based indicators and task performance metrics. To validate this model, we conducted a study with 48 participants across 12 MR groups engaged in a collaborative image-sorting task. Our findings show a correlation between sensed behavior and perceived collaboration, particularly through shared attention and proximity.
△ Less
Submitted 22 April, 2025;
originally announced April 2025.
-
PluralLLM: Pluralistic Alignment in LLMs via Federated Learning
Authors:
Mahmoud Srewa,
Tianyu Zhao,
Salma Elmalaki
Abstract:
Ensuring Large Language Models (LLMs) align with diverse human preferences while preserving privacy and fairness remains a challenge. Existing methods, such as Reinforcement Learning from Human Feedback (RLHF), rely on centralized data collection, making them computationally expensive and privacy-invasive. We introduce PluralLLM a federated learning-based approach that enables multiple user groups…
▽ More
Ensuring Large Language Models (LLMs) align with diverse human preferences while preserving privacy and fairness remains a challenge. Existing methods, such as Reinforcement Learning from Human Feedback (RLHF), rely on centralized data collection, making them computationally expensive and privacy-invasive. We introduce PluralLLM a federated learning-based approach that enables multiple user groups to collaboratively train a transformer-based preference predictor without sharing sensitive data, which can also serve as a reward model for aligning LLMs. Our method leverages Federated Averaging (FedAvg) to aggregate preference updates efficiently, achieving 46% faster convergence, a 4% improvement in alignment scores, and nearly the same group fairness measure as in centralized training. Evaluated on a Q/A preference alignment task, PluralLLM demonstrates that federated preference learning offers a scalable and privacy-preserving alternative for aligning LLMs with diverse human values.
△ Less
Submitted 12 March, 2025;
originally announced March 2025.
-
MoCoMR: A Collaborative MR Simulator with Individual Behavior Modeling
Authors:
Diana Romero,
Fatima Anwar,
Salma Elmalaki
Abstract:
Studying collaborative behavior in Mixed Reality (MR) often requires extensive, challenging data collection. This paper introduces MoCoMR, a novel simulator designed to address this by generating synthetic yet realistic collaborative MR data. MoCoMR captures individual behavioral modalities such as speaking, gaze, and locomotion during a collaborative image-sorting task with 48 participants to ide…
▽ More
Studying collaborative behavior in Mixed Reality (MR) often requires extensive, challenging data collection. This paper introduces MoCoMR, a novel simulator designed to address this by generating synthetic yet realistic collaborative MR data. MoCoMR captures individual behavioral modalities such as speaking, gaze, and locomotion during a collaborative image-sorting task with 48 participants to identify distinct behavioral patterns. MoCoMR simulates individual actions and interactions within a virtual space, enabling researchers to investigate the impact of individual behaviors on group dynamics and task performance. This simulator facilitates the development of more effective and human-centered MR applications by providing insights into user behavior and interaction patterns. The simulator's API allows for flexible configuration and data analysis, enabling researchers to explore various scenarios and generate valuable insights for optimizing collaborative MR experiences.
△ Less
Submitted 12 March, 2025;
originally announced March 2025.
-
FinP: Fairness-in-Privacy in Federated Learning by Addressing Disparities in Privacy Risk
Authors:
Tianyu Zhao,
Mahmoud Srewa,
Salma Elmalaki
Abstract:
Ensuring fairness in machine learning, particularly in human-centric applications, extends beyond algorithmic bias to encompass fairness in privacy, specifically the equitable distribution of privacy risk. This is critical in federated learning (FL), where decentralized data necessitates balanced privacy preservation across clients. We introduce FinP, a framework designed to achieve fairness in pr…
▽ More
Ensuring fairness in machine learning, particularly in human-centric applications, extends beyond algorithmic bias to encompass fairness in privacy, specifically the equitable distribution of privacy risk. This is critical in federated learning (FL), where decentralized data necessitates balanced privacy preservation across clients. We introduce FinP, a framework designed to achieve fairness in privacy by mitigating disproportionate exposure to source inference attacks (SIA). FinP employs a dual approach: (1) server-side adaptive aggregation to address unfairness in client contributions in global model, and (2) client-side regularization to reduce client vulnerability. This comprehensive strategy targets both the symptoms and root causes of privacy unfairness. Evaluated on the Human Activity Recognition (HAR) and CIFAR-10 datasets, FinP demonstrates ~20% improvement in fairness in privacy on HAR with minimal impact on model utility, and effectively mitigates SIA risks on CIFAR-10, showcasing its ability to provide fairness in privacy in FL systems without compromising performance.
△ Less
Submitted 24 February, 2025;
originally announced February 2025.
-
AutoLike: Auditing Social Media Recommendations through User Interactions
Authors:
Hieu Le,
Salma Elmalaki,
Zubair Shafiq,
Athina Markopoulou
Abstract:
Modern social media platforms, such as TikTok, Facebook, and YouTube, rely on recommendation systems to personalize content for users based on user interactions with endless streams of content, such as "For You" pages. However, these complex algorithms can inadvertently deliver problematic content related to self-harm, mental health, and eating disorders. We introduce AutoLike, a framework to audi…
▽ More
Modern social media platforms, such as TikTok, Facebook, and YouTube, rely on recommendation systems to personalize content for users based on user interactions with endless streams of content, such as "For You" pages. However, these complex algorithms can inadvertently deliver problematic content related to self-harm, mental health, and eating disorders. We introduce AutoLike, a framework to audit recommendation systems in social media platforms for topics of interest and their sentiments. To automate the process, we formulate the problem as a reinforcement learning problem. AutoLike drives the recommendation system to serve a particular type of content through interactions (e.g., liking). We apply the AutoLike framework to the TikTok platform as a case study. We evaluate how well AutoLike identifies TikTok content automatically across nine topics of interest; and conduct eight experiments to demonstrate how well it drives TikTok's recommendation system towards particular topics and sentiments. AutoLike has the potential to assist regulators in auditing recommendation systems for problematic content. (Warning: This paper contains qualitative examples that may be viewed as offensive or harmful.)
△ Less
Submitted 12 February, 2025;
originally announced February 2025.
-
GroupBeaMR: Analyzing Collaborative Group Behavior in Mixed Reality Through Passive Sensing and Sociometry
Authors:
Diana Romero,
Yasra Chandio,
Fatima Anwar,
Salma Elmalaki
Abstract:
Understanding group behavior is crucial for enhancing collaboration and productivity in mixed reality (MR). This paper introduces a framework for group behavior analysis in MR, or GroupBeaMR for short for analyzing group behavior in MR. GroupBeaMR leverages MR headsets' sensors to analyze group behavior through conversation, shared attention, and proximity, identifying cohesive, fragmented, and co…
▽ More
Understanding group behavior is crucial for enhancing collaboration and productivity in mixed reality (MR). This paper introduces a framework for group behavior analysis in MR, or GroupBeaMR for short for analyzing group behavior in MR. GroupBeaMR leverages MR headsets' sensors to analyze group behavior through conversation, shared attention, and proximity, identifying cohesive, fragmented, and competitive interaction patterns. Using social network analysis, GroupBeaMR provides quantitative assessments of group dynamics, offering insights into collaboration structures. A user study with 48 participants across 12 groups validates the framework's ability to distinguish interaction patterns in MR environments. Our analyses show that group behavior is independent of task performance, emphasizing the significance of social interaction patterns. Our group-type assignments indicate that sensor-based assessments in MR can provide meaningful insights into collaborative experiences, supporting the design of systems that adapt and optimize group behaviors.
△ Less
Submitted 10 February, 2025; v1 submitted 7 November, 2024;
originally announced November 2024.
-
PEaRL: Personalized Privacy of Human-Centric Systems using Early-Exit Reinforcement Learning
Authors:
Mojtaba Taherisadr,
Salma Elmalaki
Abstract:
In the evolving landscape of human-centric systems, personalized privacy solutions are becoming increasingly crucial due to the dynamic nature of human interactions. Traditional static privacy models often fail to meet the diverse and changing privacy needs of users. This paper introduces PEaRL, a system designed to enhance privacy preservation by tailoring its approach to individual behavioral pa…
▽ More
In the evolving landscape of human-centric systems, personalized privacy solutions are becoming increasingly crucial due to the dynamic nature of human interactions. Traditional static privacy models often fail to meet the diverse and changing privacy needs of users. This paper introduces PEaRL, a system designed to enhance privacy preservation by tailoring its approach to individual behavioral patterns and preferences. While incorporating reinforcement learning (RL) for its adaptability, PEaRL primarily focuses on employing an early-exit strategy that dynamically balances privacy protection and system utility. This approach addresses the challenges posed by the variability and evolution of human behavior, which static privacy models struggle to handle effectively. We evaluate PEaRL in two distinct contexts: Smart Home environments and Virtual Reality (VR) Smart Classrooms. The empirical results demonstrate PEaRL's capability to provide a personalized tradeoff between user privacy and application utility, adapting effectively to individual user preferences. On average, across both systems, PEaRL enhances privacy protection by 31%, with a corresponding utility reduction of 24%.
△ Less
Submitted 12 November, 2024; v1 submitted 9 March, 2024;
originally announced March 2024.
-
GaitGuard: Towards Private Gait in Mixed Reality
Authors:
Diana Romero,
Ruchi Jagdish Patel,
Athina Markopoulou,
Salma Elmalaki
Abstract:
Augmented/Mixed Reality (AR/MR) technologies usher in a new era of immersive, collective experiences, differentiating them from traditional mobile systems. As these technologies evolve, prioritizing privacy and security is critical. This paper focuses on gait privacy, where gait, the way a person walks, can reveal sensitive information such as age, ethnicity, or disorders. We present GaitGuard, a…
▽ More
Augmented/Mixed Reality (AR/MR) technologies usher in a new era of immersive, collective experiences, differentiating them from traditional mobile systems. As these technologies evolve, prioritizing privacy and security is critical. This paper focuses on gait privacy, where gait, the way a person walks, can reveal sensitive information such as age, ethnicity, or disorders. We present GaitGuard, a real-time system that protects gait privacy against video-based gait extraction attacks in MR environments. GaitGuard leverages a multi-threaded framework to efficiently process video frames, incorporating dedicated modules for stream capture, body detection and tracking, and privacy leak mitigation. We compare and combine multiple mitigation techniques, offering guidance to navigate the privacy-utility tradeoff. Through extensive experiments covering 248 settings across mitigation regions, types, and tunable parameters, we assess the impact of these techniques on privacy, video quality, and system performance. GaitGuard reduces the confidence of video-based gait extraction attacks by introducing a substantial distribution shift (Jensen-Shannon Divergence of 0.63, indicating highly altered gait features) and a decrease in identification risks by up to 68%, while maintaining 29 FPS and preserving video clarity. GaitGuard provides a practical real-time solution for privacy-preserving MR applications without affecting the MR user experience based on 20 subjective user surveys.
△ Less
Submitted 29 April, 2025; v1 submitted 7 December, 2023;
originally announced December 2023.
-
FinA: Fairness of Adverse Effects in Decision-Making of Human-Cyber-Physical-System
Authors:
Tianyu Zhao,
Salma Elmalaki
Abstract:
Ensuring fairness in decision-making systems within Human-Cyber-Physical-Systems (HCPS) is a pressing concern, particularly when diverse individuals, each with varying behaviors and expectations, coexist within the same application space, influenced by a shared set of control actions in the system. The long-term adverse effects of these actions further pose the challenge, as historical experiences…
▽ More
Ensuring fairness in decision-making systems within Human-Cyber-Physical-Systems (HCPS) is a pressing concern, particularly when diverse individuals, each with varying behaviors and expectations, coexist within the same application space, influenced by a shared set of control actions in the system. The long-term adverse effects of these actions further pose the challenge, as historical experiences and interactions shape individual perceptions of fairness. This paper addresses the challenge of fairness from an equity perspective of adverse effects, taking into account the dynamic nature of human behavior and evolving preferences while recognizing the lasting impact of adverse effects. We formally introduce the concept of Fairness-in-Adverse-Effects (FinA) within the HCPS context. We put forth a comprehensive set of five formulations for FinA, encompassing both the instantaneous and long-term aspects of adverse effects. To empirically validate the effectiveness of our FinA approach, we conducted an evaluation within the domain of smart homes, a pertinent HCPS application. The outcomes of our evaluation demonstrate that the adoption of FinA significantly enhances the overall perception of fairness among individuals, yielding an average improvement of 66.7% when compared to the state-of-the-art method.
△ Less
Submitted 6 November, 2023;
originally announced November 2023.
-
Towards Fairness-aware Crowd Management System and Surge Prevention in Smart Cities
Authors:
Yixin Zhang,
Tianyu Zhao,
Salma Elmalaki
Abstract:
Instances of casualties resulting from large crowds persist, highlighting the existing limitations of current crowd management practices in Smart Cities. One notable drawback is the insufficient provision for disadvantaged individuals who may require additional time to evacuate due to their slower running speed. Moreover, the existing escape strategies may fall short of ensuring the safety of all…
▽ More
Instances of casualties resulting from large crowds persist, highlighting the existing limitations of current crowd management practices in Smart Cities. One notable drawback is the insufficient provision for disadvantaged individuals who may require additional time to evacuate due to their slower running speed. Moreover, the existing escape strategies may fall short of ensuring the safety of all individuals during a crowd surge. To address these pressing concerns, this paper proposes two crowd management methodologies. Firstly, we advocate for implementing a fair evacuation strategy following a surge event, which considers the diverse needs of all individuals, ensuring inclusivity and mitigating potential risks. Secondly, we propose a preventative approach involving the adjustment of attraction locations and switching between stage performances in large-crowded events to minimize the occurrence of surges and enhance crowd dispersion. We used high-fidelity crowd management simulators to assess the effectiveness of our proposals. Our findings demonstrate the positive impact of the fair evacuation strategy on safety measures and inclusivity, which increases fairness by 41.8% on average. Furthermore, adjusting attraction locations and stage performances has shown a significant reduction in surges by 34% on average, enhancing overall crowd safety.
△ Less
Submitted 22 April, 2024; v1 submitted 3 November, 2023;
originally announced November 2023.
-
BehaVR: User Identification Based on VR Sensor Data
Authors:
Ismat Jarin,
Yu Duan,
Rahmadi Trimananda,
Hao Cui,
Salma Elmalaki,
Athina Markopoulou
Abstract:
Virtual reality (VR) platforms enable a wide range of applications, however, pose unique privacy risks. In particular, VR devices are equipped with a rich set of sensors that collect personal and sensitive information (e.g., body motion, eye gaze, hand joints, and facial expression). The data from these newly available sensors can be used to uniquely identify a user, even in the absence of explici…
▽ More
Virtual reality (VR) platforms enable a wide range of applications, however, pose unique privacy risks. In particular, VR devices are equipped with a rich set of sensors that collect personal and sensitive information (e.g., body motion, eye gaze, hand joints, and facial expression). The data from these newly available sensors can be used to uniquely identify a user, even in the absence of explicit identifiers. In this paper, we seek to understand the extent to which a user can be identified based solely on VR sensor data, within and across real-world apps from diverse genres. We consider adversaries with capabilities that range from observing APIs available within a single app (app adversary) to observing all or selected sensor measurements across multiple apps on the VR device (device adversary). To that end, we introduce BehaVR, a framework for collecting and analyzing data from all sensor groups collected by multiple apps running on a VR device. We use BehaVR to collect data from real users that interact with 20 popular real-world apps. We use that data to build machine learning models for user identification within and across apps, with features extracted from available sensor data. We show that these models can identify users with an accuracy of up to 100%, and we reveal the most important features and sensor groups, depending on the functionality of the app and the adversary. To the best of our knowledge, BehaVR is the first to analyze user identification in VR comprehensively, i.e., considering all sensor measurements available on consumer VR devices, collected by multiple real-world, as opposed to custom-made, apps.
△ Less
Submitted 23 September, 2024; v1 submitted 14 August, 2023;
originally announced August 2023.
-
FAIRO: Fairness-aware Adaptation in Sequential-Decision Making for Human-in-the-Loop Systems
Authors:
Tianyu Zhao,
Mojtaba Taherisadr,
Salma Elmalaki
Abstract:
Achieving fairness in sequential-decision making systems within Human-in-the-Loop (HITL) environments is a critical concern, especially when multiple humans with different behavior and expectations are affected by the same adaptation decisions in the system. This human variability factor adds more complexity since policies deemed fair at one point in time may become discriminatory over time due to…
▽ More
Achieving fairness in sequential-decision making systems within Human-in-the-Loop (HITL) environments is a critical concern, especially when multiple humans with different behavior and expectations are affected by the same adaptation decisions in the system. This human variability factor adds more complexity since policies deemed fair at one point in time may become discriminatory over time due to variations in human preferences resulting from inter- and intra-human variability. This paper addresses the fairness problem from an equity lens, considering human behavior variability, and the changes in human preferences over time. We propose FAIRO, a novel algorithm for fairness-aware sequential-decision making in HITL adaptation, which incorporates these notions into the decision-making process. In particular, FAIRO decomposes this complex fairness task into adaptive sub-tasks based on individual human preferences through leveraging the Options reinforcement learning framework. We design FAIRO to generalize to three types of HITL application setups that have the shared adaptation decision problem. Furthermore, we recognize that fairness-aware policies can sometimes conflict with the application's utility. To address this challenge, we provide a fairness-utility tradeoff in FAIRO, allowing system designers to balance the objectives of fairness and utility based on specific application requirements. Extensive evaluations of FAIRO on the three HITL applications demonstrate its generalizability and effectiveness in promoting fairness while accounting for human variability. On average, FAIRO can improve fairness compared with other methods across all three applications by 35.36%.
△ Less
Submitted 6 November, 2023; v1 submitted 11 July, 2023;
originally announced July 2023.
-
ERUDITE: Human-in-the-Loop IoT for an Adaptive Personalized Learning System
Authors:
Mojtaba Taherisadr,
Mohammad Abdullah Al Faruque,
Salma Elmalaki
Abstract:
Thanks to the rapid growth in wearable technologies and recent advancement in machine learning and signal processing, monitoring complex human contexts becomes feasible, paving the way to develop human-in-the-loop IoT systems that naturally evolve to adapt to the human and environment state autonomously. Nevertheless, a central challenge in designing many of these IoT systems arises from the requi…
▽ More
Thanks to the rapid growth in wearable technologies and recent advancement in machine learning and signal processing, monitoring complex human contexts becomes feasible, paving the way to develop human-in-the-loop IoT systems that naturally evolve to adapt to the human and environment state autonomously. Nevertheless, a central challenge in designing many of these IoT systems arises from the requirement to infer the human mental state, such as intention, stress, cognition load, or learning ability. While different human contexts can be inferred from the fusion of different sensor modalities that can correlate to a particular mental state, the human brain provides a richer sensor modality that gives us more insights into the required human context. This paper proposes ERUDITE, a human-in-the-loop IoT system for the learning environment that exploits recent wearable neurotechnology to decode brain signals. Through insights from concept learning theory, ERUDITE can infer the human state of learning and understand when human learning increases or declines. By quantifying human learning as an input sensory signal, ERUDITE can provide adequate personalized feedback to humans in a learning environment to enhance their learning experience. ERUDITE is evaluated across $15$ participants and showed that by using the brain signals as a sensor modality to infer the human learning state and providing personalized adaptation to the learning environment, the participants' learning performance increased on average by $26\%$. Furthermore, we showed that ERUDITE can be deployed on an edge-based prototype to evaluate its practicality and scalability.
△ Less
Submitted 20 November, 2023; v1 submitted 7 March, 2023;
originally announced March 2023.
-
adaPARL: Adaptive Privacy-Aware Reinforcement Learning for Sequential-Decision Making Human-in-the-Loop Systems
Authors:
Mojtaba Taherisadr,
Stelios Andrew Stavroulakis,
Salma Elmalaki
Abstract:
Reinforcement learning (RL) presents numerous benefits compared to rule-based approaches in various applications. Privacy concerns have grown with the widespread use of RL trained with privacy-sensitive data in IoT devices, especially for human-in-the-loop systems. On the one hand, RL methods enhance the user experience by trying to adapt to the highly dynamic nature of humans. On the other hand,…
▽ More
Reinforcement learning (RL) presents numerous benefits compared to rule-based approaches in various applications. Privacy concerns have grown with the widespread use of RL trained with privacy-sensitive data in IoT devices, especially for human-in-the-loop systems. On the one hand, RL methods enhance the user experience by trying to adapt to the highly dynamic nature of humans. On the other hand, trained policies can leak the user's private information. Recent attention has been drawn to designing privacy-aware RL algorithms while maintaining an acceptable system utility. A central challenge in designing privacy-aware RL, especially for human-in-the-loop systems, is that humans have intrinsic variability and their preferences and behavior evolve. The effect of one privacy leak mitigation can be different for the same human or across different humans over time. Hence, we can not design one fixed model for privacy-aware RL that fits all. To that end, we propose adaPARL, an adaptive approach for privacy-aware RL, especially for human-in-the-loop IoT systems. adaPARL provides a personalized privacy-utility trade-off depending on human behavior and preference. We validate the proposed adaPARL on two IoT applications, namely (i) Human-in-the-Loop Smart Home and (ii) Human-in-the-Loop Virtual Reality (VR) Smart Classroom. Results obtained on these two applications validate the generality of adaPARL and its ability to provide a personalized privacy-utility trade-off. On average, for the first application, adaPARL improves the utility by $57\%$ over the baseline and by $43\%$ over randomization. adaPARL also reduces the privacy leak by $23\%$ on average. For the second application, adaPARL decreases the privacy leak to $44\%$ before the utility drops by $15\%$.
△ Less
Submitted 7 March, 2023;
originally announced March 2023.
-
AutoFR: Automated Filter Rule Generation for Adblocking
Authors:
Hieu Le,
Salma Elmalaki,
Athina Markopoulou,
Zubair Shafiq
Abstract:
Adblocking relies on filter lists, which are manually curated and maintained by a community of filter list authors. Filter list curation is a laborious process that does not scale well to a large number of sites or over time. In this paper, we introduce AutoFR, a reinforcement learning framework to fully automate the process of filter rule creation and evaluation for sites of interest. We design a…
▽ More
Adblocking relies on filter lists, which are manually curated and maintained by a community of filter list authors. Filter list curation is a laborious process that does not scale well to a large number of sites or over time. In this paper, we introduce AutoFR, a reinforcement learning framework to fully automate the process of filter rule creation and evaluation for sites of interest. We design an algorithm based on multi-arm bandits to generate filter rules that block ads while controlling the trade-off between blocking ads and avoiding visual breakage. We test AutoFR on thousands of sites and we show that it is efficient: it takes only a few minutes to generate filter rules for a site of interest. AutoFR is effective: it generates filter rules that can block 86% of the ads, as compared to 87% by EasyList, while achieving comparable visual breakage. Furthermore, AutoFR generates filter rules that generalize well to new sites. We envision that AutoFR can assist the adblocking community in filter rule generation at scale.
△ Less
Submitted 7 March, 2023; v1 submitted 25 February, 2022;
originally announced February 2022.
-
VindiCo: Privacy Safeguard Against Adaptation Based Spyware in Human-in-the-Loop IoT
Authors:
Salma Elmalaki,
Bo-Jhang Ho,
Moustafa Alzantot,
Yasser Shoukry,
Mani Srivastava
Abstract:
Personalized IoT adapts their behavior based on contextual information, such as user behavior and location. Unfortunately, the fact that personalized IoT adapts to user context opens a side-channel that leaks private information about the user. To that end, we start by studying the extent to which a malicious eavesdropper can monitor the actions taken by an IoT system and extract users' private in…
▽ More
Personalized IoT adapts their behavior based on contextual information, such as user behavior and location. Unfortunately, the fact that personalized IoT adapts to user context opens a side-channel that leaks private information about the user. To that end, we start by studying the extent to which a malicious eavesdropper can monitor the actions taken by an IoT system and extract users' private information. In particular, we show two concrete instantiations (in the context of mobile phones and smart homes) of a new category of spyware which we refer to as Context-Aware Adaptation Based Spyware (SpyCon). Experimental evaluations show that the developed SpyCon can predict users' daily behavior with an accuracy of 90.3%. The rest of this paper is devoted to introducing VindiCo, a software mechanism designed to detect and mitigate possible SpyCon. Being new spyware with no known prior signature or behavior, traditional spyware detection that is based on code signature or app behavior is not adequate to detect SpyCon. Therefore, VindiCo proposes a novel information-based detection engine along with several mitigation techniques to restrain the ability of the detected SpyCon to extract private information. By having general detection and mitigation engines, VindiCo is agnostic to the inference algorithm used by SpyCon. Our results show that VindiCo reduces the ability of SpyCon to infer user context from 90.3% to the baseline accuracy (accuracy based on random guesses) with negligible execution overhead.
△ Less
Submitted 2 February, 2022;
originally announced February 2022.
-
Future of Smart Classroom in the Era of Wearable Neurotechnology
Authors:
Mojtaba Taherisadr,
Berken Utku Demirel,
Mohammad Abdullah Al Faruque,
Salma Elmalaki
Abstract:
Interdisciplinary research among engineering, computer science, and neuroscience to understand and utilize the human brain signals resulted in advances and widespread applicability of wearable neurotechnology in adaptive human-in-the-loop smart systems. Considering these advances, we envision that future education will exploit the advances in wearable neurotechnology and move toward more personali…
▽ More
Interdisciplinary research among engineering, computer science, and neuroscience to understand and utilize the human brain signals resulted in advances and widespread applicability of wearable neurotechnology in adaptive human-in-the-loop smart systems. Considering these advances, we envision that future education will exploit the advances in wearable neurotechnology and move toward more personalized smart classrooms where instructions and interactions are tailored towards. students' individual strengths and needs. In this paper, we discuss the future of smart classrooms and how advances in neuroscience, machine learning, and embedded systems as key enablers will provide the infrastructure for envisioned smart classrooms and personalized education along with open challenges that are required to be addressed.
△ Less
Submitted 21 October, 2021;
originally announced October 2021.
-
FaiR-IoT: Fairness-aware Human-in-the-Loop Reinforcement Learning for Harnessing Human Variability in Personalized IoT
Authors:
Salma Elmalaki
Abstract:
Thanks to the rapid growth in wearable technologies, monitoring complex human context becomes feasible, paving the way to develop human-in-the-loop IoT systems that naturally evolve to adapt to the human and environment state autonomously. Nevertheless, a central challenge in designing such personalized IoT applications arises from human variability. Such variability stems from the fact that diffe…
▽ More
Thanks to the rapid growth in wearable technologies, monitoring complex human context becomes feasible, paving the way to develop human-in-the-loop IoT systems that naturally evolve to adapt to the human and environment state autonomously. Nevertheless, a central challenge in designing such personalized IoT applications arises from human variability. Such variability stems from the fact that different humans exhibit different behaviors when interacting with IoT applications (intra-human variability), the same human may change the behavior over time when interacting with the same IoT application (inter-human variability), and human behavior may be affected by the behaviors of other people in the same environment (multi-human variability). To that end, we propose FaiR-IoT, a general reinforcement learning-based framework for adaptive and fairness-aware human-in-the-loop IoT applications. In FaiR-IoT, three levels of reinforcement learning agents interact to continuously learn human preferences and maximize the system's performance and fairness while taking into account the intra-, inter-, and multi-human variability. We validate the proposed framework on two applications, namely (i) Human-in-the-Loop Automotive Advanced Driver Assistance Systems and (ii) Human-in-the-Loop Smart House. Results obtained on these two applications validate the generality of FaiR-IoT and its ability to provide a personalized experience while enhancing the system's performance by 40%-60% compared to non-personalized systems and enhancing the fairness of the multi-human systems by 1.5 orders of magnitude.
△ Less
Submitted 29 March, 2021;
originally announced March 2021.
