Showing 1–8 of 8 results for author: Klatzky, R

Realistic Modeling of Human Timings for Wearable Cognitive Assistance

Authors: Manuel O. J. Olguín Muñoz, Vishnu N. Moothedath, Jaya Prakash Champati, Roberta Klatzky, Mahadev Satyanarayanan, James Gross

Abstract: Wearable Cognitive Assistance (WCA) applications present a challenge to benchmark and characterize due to their human-in-the-loop nature. Employing user testing to optimize system parameters is generally not feasible, given the scope of the problem and the number of observations needed to detect small but important effects in controlled experiments. Considering the intended mass-scale deployment o… ▽ More Wearable Cognitive Assistance (WCA) applications present a challenge to benchmark and characterize due to their human-in-the-loop nature. Employing user testing to optimize system parameters is generally not feasible, given the scope of the problem and the number of observations needed to detect small but important effects in controlled experiments. Considering the intended mass-scale deployment of WCA applications in the future, there exists a need for tools enabling human-independent benchmarking. We present in this paper the first model for the complete end-to-end emulation of humans in WCA. We build this model through statistical analysis of data collected from previous work in this field, and demonstrate its utility by studying application task durations. Compared to first-order approximations, our model shows a ~36% larger gap between step execution times at high system impairment versus low. We further introduce a novel framework for stochastic optimization of resource consumption-responsiveness tradeoffs in WCA, and show that by combining this framework with our realistic model of human behavior, significant reductions of up to 50% in number processed frame samples and 20% in energy consumption can be achieved with respect to the state-of-the-art. △ Less

Submitted 12 December, 2022; originally announced December 2022.

Comments: 16 total pages. 12 figures, 2 tables, 1 appendix. Main document body by Manuel Olguín Muñoz and Vishnu N. Moothedath; appendix by Vishu N. Moothedath and Jaya Prakash Champati; editing and feedback by all authors; funding by James Gross and Mahadev Satyanarayanan. Submitted to IEEE Transactions on Mobile Computing

Ajalon: Simplifying the Authoring of Wearable Cognitive Assistants

Authors: Truong An Pham, Junjue Wang, Yu Xiao, Padmanabhan Pillai, Roger Iyengar, Roberta Klatzky, Mahadev Satyanarayanan

Abstract: Wearable Cognitive Assistance (WCA) amplifies human cognition in real time through a wearable device and low-latency wireless access to edge computing infrastructure. It is inspired by, and broadens, the metaphor of GPS navigation tools that provide real-time step-by-step guidance, with prompt error detection and correction. WCA applications are likely to be transformative in education, health car… ▽ More Wearable Cognitive Assistance (WCA) amplifies human cognition in real time through a wearable device and low-latency wireless access to edge computing infrastructure. It is inspired by, and broadens, the metaphor of GPS navigation tools that provide real-time step-by-step guidance, with prompt error detection and correction. WCA applications are likely to be transformative in education, health care, industrial troubleshooting, manufacturing, and many other areas. Today, WCA application development is difficult and slow, requiring skills in areas such as machine learning and computer vision that are not widespread among software developers. This paper describes Ajalon, an authoring toolchain for WCA applications that reduces the skill and effort needed at each step of the development pipeline. Our evaluation shows that Ajalon significantly reduces the effort needed to create new WCA applications. △ Less

Submitted 14 January, 2021; originally announced January 2021.

Impact of delayed response on Wearable Cognitive Assistance

Authors: M. Olguín Muñoz, R. Klatzky, J. Wang, P. Pillai, M. Satyanarayanan, J. Gross

Abstract: Wearable Cognitive Assistants (WCA) are anticipated to become a widely-used application class, in conjunction with emerging network infrastructures like 5G that incorporate edge computing capabilities. While prototypical studies of such applications exist today, the relationship between infrastructure service provisioning and its implication for WCA usability is largely unexplored despite the rele… ▽ More Wearable Cognitive Assistants (WCA) are anticipated to become a widely-used application class, in conjunction with emerging network infrastructures like 5G that incorporate edge computing capabilities. While prototypical studies of such applications exist today, the relationship between infrastructure service provisioning and its implication for WCA usability is largely unexplored despite the relevance that these applications have for future networks. This paper presents an experimental study assessing how WCA users react to varying end-to-end delays induced by the application pipeline or infrastructure. Participants interacted directly with an instrumented task-guidance WCA as delays were introduced into the system in a controllable fashion. System and task state were tracked in real time, and biometric data from wearable sensors on the participants were recorded. Our results show that periods of extended system delay cause users to correspondingly (and substantially) slow down in their guided task execution, an effect that persists for a time after the system returns to a more responsive state. Furthermore, the slow-down in task execution is correlated with a personality trait, neuroticism, associated with intolerance for time delays. We show that our results implicate impaired cognitive planning, as contrasted with resource depletion or emotional arousal, as the reason for slowed user task executions under system delay. The findings have several implications for the design and operation of WCA applications as well as computational and communication infrastructure, and additionally for the development of performance analysis tools for WCA. △ Less

Submitted 4 November, 2020; originally announced November 2020.

Comments: 27 pages, 14 figures. Submitted to PLOS ONE on October 15, 2020

ACM Class: H.5.2

Building a navigable fine texture design space

Authors: Rebecca Fenton Friesen, Roberta L. Klatzky, Michael A. Peshkin, J. Edward Colgate

Abstract: Friction modulation technology enables the creation of textural effects on flat haptic displays. However, an intuitive and manageably small design space for construction of such haptic textures remains an unfulfilled goal for user interface designers. In this paper, we explore perceptually relevant features of fine texture for use in texture construction and modification. Beginning with simple sin… ▽ More Friction modulation technology enables the creation of textural effects on flat haptic displays. However, an intuitive and manageably small design space for construction of such haptic textures remains an unfulfilled goal for user interface designers. In this paper, we explore perceptually relevant features of fine texture for use in texture construction and modification. Beginning with simple sinusoidal patterns of friction force that vary in frequency and amplitude, we define irregularity as a third building block of a texture pattern and show it to be a scalable feature distinct from the others using multidimensional scaling. Additionally, subjects' verbal descriptions of this 3-dimensional design space provide insight into their intuitive interpretation of the physical parameter changes. △ Less

Submitted 12 June, 2020; originally announced June 2020.

Localizable Button Click Rendering via Active Lateral Force Feedback

Authors: Heng Xu, Roberta L. Klatzky, Michael A. Peshkin, J. Edward Colgate

Abstract: We have developed a novel button click rendering mechanism based on active lateral force feedback. The effect can be localized because electroadhesion between a finger and a surface can be localized. Psychophysical experiments were conducted to evaluate the quality of a rendered button click, which subjects judged to be acceptable. Both the experiment results and the subjects' comments confirm tha… ▽ More We have developed a novel button click rendering mechanism based on active lateral force feedback. The effect can be localized because electroadhesion between a finger and a surface can be localized. Psychophysical experiments were conducted to evaluate the quality of a rendered button click, which subjects judged to be acceptable. Both the experiment results and the subjects' comments confirm that this button click rendering mechanism has the ability to generate a range of realistic button click sensations that could match subjects' different preferences. We can thus generate a button click on a flat surface without macroscopic motion of the surface in the lateral or normal direction, and we can localize this haptic effect to an individual finger. △ Less

Submitted 28 April, 2020; v1 submitted 18 December, 2019; originally announced December 2019.

Comments: Accepted, IEEE Transactions on Haptics

Learning efficient haptic shape exploration with a rigid tactile sensor array

Authors: Sascha Fleer, Alexandra Moringen, Roberta L. Klatzky, Helge Ritter

Abstract: Haptic exploration is a key skill for both robots and humans to discriminate and handle unknown objects or to recognize familiar objects. Its active nature is evident in humans who from early on reliably acquire sophisticated sensory-motor capabilities for active exploratory touch and directed manual exploration that associates surfaces and object properties with their spatial locations. This is i… ▽ More Haptic exploration is a key skill for both robots and humans to discriminate and handle unknown objects or to recognize familiar objects. Its active nature is evident in humans who from early on reliably acquire sophisticated sensory-motor capabilities for active exploratory touch and directed manual exploration that associates surfaces and object properties with their spatial locations. This is in stark contrast to robotics. In this field, the relative lack of good real-world interaction models - along with very restricted sensors and a scarcity of suitable training data to leverage machine learning methods - has so far rendered haptic exploration a largely underdeveloped skill. In the present work, we connect recent advances in recurrent models of visual attention with previous insights about the organisation of human haptic search behavior, exploratory procedures and haptic glances for a novel architecture that learns a generative model of haptic exploration in a simulated three-dimensional environment. The proposed algorithm simultaneously optimizes main perception-action loop components: feature extraction, integration of features over time, and the control strategy, while continuously acquiring data online. We perform a multi-module neural network training, including a feature extractor and a recurrent neural network module aiding pose control for storing and combining sequential sensory data. The resulting haptic meta-controller for the rigid $16 \times 16$ tactile sensor array moving in a physics-driven simulation environment, called the Haptic Attention Model, performs a sequence of haptic glances, and outputs corresponding force measurements. The resulting method has been successfully tested with four different objects. It achieved results close to $100 \%$ while performing object contour exploration that has been optimized for its own sensor morphology. △ Less

Submitted 26 January, 2020; v1 submitted 20 February, 2019; originally announced February 2019.

Journal ref: PLOS ONE 15(1): e0226880 (2020)

P-Reverb: Perceptual Characterization of Early and Late Reflections for Auditory Displays

Authors: Atul Rungta, Nicholas Rewkowski, Roberta Klatzky, Dinesh Manocha

Abstract: We introduce a novel, perceptually derived metric (P-Reverb) that relates the just-noticeable difference (JND) of the early sound field(also called early reflections) to the late sound field (known as late reflections or reverberation). Early and late reflections are crucial components of the sound field and provide multiple perceptual cues for auditory displays. We conduct two extensive user eval… ▽ More We introduce a novel, perceptually derived metric (P-Reverb) that relates the just-noticeable difference (JND) of the early sound field(also called early reflections) to the late sound field (known as late reflections or reverberation). Early and late reflections are crucial components of the sound field and provide multiple perceptual cues for auditory displays. We conduct two extensive user evaluations that relate the JNDs of early reflections and late reverberation in terms of the mean-free path of the environment and present a novel P-Reverb metric. Our metric is used to estimate dynamic reverberation characteristics efficiently in terms of important parameters like reverberation time (RT60). We show the numerical accuracy of our P-Reverb metric in estimating RT60. Finally, we use our metric to design an interactive sound propagation algorithm and demonstrate its effectiveness on various benchmarks. △ Less

Submitted 18 February, 2019; originally announced February 2019.

Effects of virtual acoustics on dynamic auditory distance perception

Authors: Atul Rungta, Nicholas Rewkowski, Roberta Klatzky, Ming Lin, Dinesh Manocha

Abstract: Sound propagation encompasses various acoustic phenomena including reverberation. Current virtual acoustic methods, ranging from parametric filters to physically-accurate solvers, can simulate reverberation with varying degrees of fidelity. We investigate the effects of reverberant sounds generated using different propagation algorithms on acoustic distance perception, i.e., how faraway humans per… ▽ More Sound propagation encompasses various acoustic phenomena including reverberation. Current virtual acoustic methods, ranging from parametric filters to physically-accurate solvers, can simulate reverberation with varying degrees of fidelity. We investigate the effects of reverberant sounds generated using different propagation algorithms on acoustic distance perception, i.e., how faraway humans perceive a sound source. In particular, we evaluate two classes of methods for real-time sound propagation in dynamic scenes based on parametric filters and ray tracing. Our study shows that the more accurate method shows less distance compression as compared to the approximate, filter-based method. This suggests that accurate reverberation in VR results in a better reproduction of acoustic distances. We also quantify the levels of distance compression introduced by different propagation methods in a virtual environment. △ Less

Submitted 20 April, 2017; originally announced April 2017.

Comments: 8 Pages, 7 figures