-
Effect of Duration and Delay on the Identifiability of VR Motion
Authors:
Mark Roman Miller,
Vivek Nair,
Eugy Han,
Cyan DeVeaux,
Christian Rack,
Rui Wang,
Brandon Huang,
Marc Erich Latoschik,
James F. O'Brien,
Jeremy N. Bailenson
Abstract:
Social virtual reality is an emerging medium of communication. In this medium, a user's avatar (virtual representation) is controlled by the tracked motion of the user's headset and hand controllers. This tracked motion is a rich data stream that can leak characteristics of the user or can be effectively matched to previously-identified data to identify a user. To better understand the boundaries…
▽ More
Social virtual reality is an emerging medium of communication. In this medium, a user's avatar (virtual representation) is controlled by the tracked motion of the user's headset and hand controllers. This tracked motion is a rich data stream that can leak characteristics of the user or can be effectively matched to previously-identified data to identify a user. To better understand the boundaries of motion data identifiability, we investigate how varying training data duration and train-test delay affects the accuracy at which a machine learning model can correctly classify user motion in a supervised learning task simulating re-identification. The dataset we use has a unique combination of a large number of participants, long duration per session, large number of sessions, and a long time span over which sessions were conducted. We find that training data duration and train-test delay affect identifiability; that minimal train-test delay leads to very high accuracy; and that train-test delay should be controlled in future experiments.
△ Less
Submitted 26 August, 2024; v1 submitted 25 July, 2024;
originally announced July 2024.
-
Effect of Data Degradation on Motion Re-Identification
Authors:
Vivek Nair,
Mark Roman Miller,
Rui Wang,
Brandon Huang,
Christian Rack,
Marc Erich Latoschik,
James F. O'Brien
Abstract:
The use of virtual and augmented reality devices is increasing, but these sensor-rich devices pose risks to privacy. The ability to track a user's motion and infer the identity or characteristics of the user poses a privacy risk that has received significant attention. Existing deep-network-based defenses against this risk, however, require significant amounts of training data and have not yet bee…
▽ More
The use of virtual and augmented reality devices is increasing, but these sensor-rich devices pose risks to privacy. The ability to track a user's motion and infer the identity or characteristics of the user poses a privacy risk that has received significant attention. Existing deep-network-based defenses against this risk, however, require significant amounts of training data and have not yet been shown to generalize beyond specific applications. In this work, we study the effect of signal degradation on identifiability, specifically through added noise, reduced framerate, reduced precision, and reduced dimensionality of the data. Our experiment shows that state-of-the-art identification attacks still achieve near-perfect accuracy for each of these degradations. This negative result demonstrates the difficulty of anonymizing this motion data and gives some justification to the existing data- and compute-intensive deep-network based methods.
△ Less
Submitted 25 July, 2024;
originally announced July 2024.
-
Deep Motion Masking for Secure, Usable, and Scalable Real-Time Anonymization of Virtual Reality Motion Data
Authors:
Vivek Nair,
Wenbo Guo,
James F. O'Brien,
Louis Rosenberg,
Dawn Song
Abstract:
Virtual reality (VR) and "metaverse" systems have recently seen a resurgence in interest and investment as major technology companies continue to enter the space. However, recent studies have demonstrated that the motion tracking "telemetry" data used by nearly all VR applications is as uniquely identifiable as a fingerprint scan, raising significant privacy concerns surrounding metaverse technolo…
▽ More
Virtual reality (VR) and "metaverse" systems have recently seen a resurgence in interest and investment as major technology companies continue to enter the space. However, recent studies have demonstrated that the motion tracking "telemetry" data used by nearly all VR applications is as uniquely identifiable as a fingerprint scan, raising significant privacy concerns surrounding metaverse technologies. Although previous attempts have been made to anonymize VR motion data, we present in this paper a state-of-the-art VR identification model that can convincingly bypass known defensive countermeasures. We then propose a new "deep motion masking" approach that scalably facilitates the real-time anonymization of VR telemetry data. Through a large-scale user study (N=182), we demonstrate that our method is significantly more usable and private than existing VR anonymity systems.
△ Less
Submitted 8 November, 2023;
originally announced November 2023.
-
Berkeley Open Extended Reality Recordings 2023 (BOXRR-23): 4.7 Million Motion Capture Recordings from 105,852 Extended Reality Device Users
Authors:
Vivek Nair,
Wenbo Guo,
Rui Wang,
James F. O'Brien,
Louis Rosenberg,
Dawn Song
Abstract:
Extended reality (XR) devices such as the Meta Quest and Apple Vision Pro have seen a recent surge in attention, with motion tracking "telemetry" data lying at the core of nearly all XR and metaverse experiences. Researchers are just beginning to understand the implications of this data for security, privacy, usability, and more, but currently lack large-scale human motion datasets to study. The B…
▽ More
Extended reality (XR) devices such as the Meta Quest and Apple Vision Pro have seen a recent surge in attention, with motion tracking "telemetry" data lying at the core of nearly all XR and metaverse experiences. Researchers are just beginning to understand the implications of this data for security, privacy, usability, and more, but currently lack large-scale human motion datasets to study. The BOXRR-23 dataset contains 4,717,215 motion capture recordings, voluntarily submitted by 105,852 XR device users from over 50 countries. BOXRR-23 is over 200 times larger than the largest existing motion capture research dataset and uses a new, highly efficient purpose-built XR Open Recording (XROR) file format.
△ Less
Submitted 30 September, 2023;
originally announced October 2023.
-
Truth in Motion: The Unprecedented Risks and Opportunities of Extended Reality Motion Data
Authors:
Vivek Nair,
Louis Rosenberg,
James F. O'Brien,
Dawn Song
Abstract:
Motion tracking "telemetry" data lies at the core of nearly all modern extended reality (XR) and metaverse experiences. While generally presumed innocuous, recent studies have demonstrated that motion data actually has the potential to profile and deanonymize XR users, posing a significant threat to security and privacy in the metaverse.
Motion tracking "telemetry" data lies at the core of nearly all modern extended reality (XR) and metaverse experiences. While generally presumed innocuous, recent studies have demonstrated that motion data actually has the potential to profile and deanonymize XR users, posing a significant threat to security and privacy in the metaverse.
△ Less
Submitted 10 June, 2023;
originally announced June 2023.
-
Inferring Private Personal Attributes of Virtual Reality Users from Head and Hand Motion Data
Authors:
Vivek Nair,
Christian Rack,
Wenbo Guo,
Rui Wang,
Shuixian Li,
Brandon Huang,
Atticus Cull,
James F. O'Brien,
Marc Latoschik,
Louis Rosenberg,
Dawn Song
Abstract:
Motion tracking "telemetry" data lies at the core of nearly all modern virtual reality (VR) and metaverse experiences. While generally presumed innocuous, recent studies have demonstrated that motion data actually has the potential to uniquely identify VR users. In this study, we go a step further, showing that a variety of private user information can be inferred just by analyzing motion data rec…
▽ More
Motion tracking "telemetry" data lies at the core of nearly all modern virtual reality (VR) and metaverse experiences. While generally presumed innocuous, recent studies have demonstrated that motion data actually has the potential to uniquely identify VR users. In this study, we go a step further, showing that a variety of private user information can be inferred just by analyzing motion data recorded from VR devices. We conducted a large-scale survey of VR users (N=1,006) with dozens of questions ranging from background and demographics to behavioral patterns and health information. We then obtained VR motion samples of each user playing the game "Beat Saber," and attempted to infer their survey responses using just their head and hand motion patterns. Using simple machine learning models, over 40 personal attributes could be accurately and consistently inferred from VR motion data alone. Despite this significant observed leakage, there remains limited awareness of the privacy implications of VR motion data, highlighting the pressing need for privacy-preserving mechanisms in multi-user VR applications.
△ Less
Submitted 10 June, 2023; v1 submitted 30 May, 2023;
originally announced May 2023.
-
Results of the 2023 Census of Beat Saber Users: Virtual Reality Gaming Population Insights and Factors Affecting Virtual Reality E-Sports Performance
Authors:
Vivek Nair,
Viktor Radulov,
James F. O'Brien
Abstract:
The emergence of affordable standalone virtual reality (VR) devices has allowed VR technology to reach mass-market adoption in recent years, driven primarily by the popularity of VR gaming applications such as Beat Saber. However, despite being the top-grossing VR application to date and the most popular VR e-sport, the population of over 6 million Beat Saber users has not yet been widely studied.…
▽ More
The emergence of affordable standalone virtual reality (VR) devices has allowed VR technology to reach mass-market adoption in recent years, driven primarily by the popularity of VR gaming applications such as Beat Saber. However, despite being the top-grossing VR application to date and the most popular VR e-sport, the population of over 6 million Beat Saber users has not yet been widely studied. In this report, we present a large-scale comprehensive survey of Beat Saber players (N=1,006) that sheds light on several important aspects of this population, including their background, biometrics, demographics, health information, behavioral patterns, and technical device specifications. We further provide insights into the emerging field of VR e-sports by analyzing correlations between responses and an authoritative measure of in-game performance.
△ Less
Submitted 30 May, 2023; v1 submitted 23 May, 2023;
originally announced May 2023.
-
KBody: Towards general, robust, and aligned monocular whole-body estimation
Authors:
Nikolaos Zioulis,
James F. O'Brien
Abstract:
KBody is a method for fitting a low-dimensional body model to an image. It follows a predict-and-optimize approach, relying on data-driven model estimates for the constraints that will be used to solve for the body's parameters. Acknowledging the importance of high quality correspondences, it leverages ``virtual joints" to improve fitting performance, disentangles the optimization between the pose…
▽ More
KBody is a method for fitting a low-dimensional body model to an image. It follows a predict-and-optimize approach, relying on data-driven model estimates for the constraints that will be used to solve for the body's parameters. Acknowledging the importance of high quality correspondences, it leverages ``virtual joints" to improve fitting performance, disentangles the optimization between the pose and shape parameters, and integrates asymmetric distance fields to strike a balance in terms of pose and shape capturing capacity, as well as pixel alignment. We also show that generative model inversion offers a strong appearance prior that can be used to complete partial human images and used as a building block for generalized and robust monocular body fitting. Project page: https://zokin.github.io/KBody.
△ Less
Submitted 15 June, 2024; v1 submitted 23 April, 2023;
originally announced April 2023.
-
Animating Fracture
Authors:
James F. O'Brien,
Jessica K. Hodgins
Abstract:
We have developed a simulation technique that uses non-linear finite element analysis and elastic fracture mechanics to compute physically plausible motion for three-dimensional, solid objects as they break, crack, or tear. When these objects deform beyond their mechanical limits, the system automatically determines where fractures should begin and in what directions they should propagate. The sys…
▽ More
We have developed a simulation technique that uses non-linear finite element analysis and elastic fracture mechanics to compute physically plausible motion for three-dimensional, solid objects as they break, crack, or tear. When these objects deform beyond their mechanical limits, the system automatically determines where fractures should begin and in what directions they should propagate. The system allows fractures to propagate in arbitrary directions by dynamically restructuring the elements of a tetrahedral mesh. Because cracks are not limited to the original element boundaries, the objects can form irregularly shaped shards and edges as they shatter. The result is realistic fracture patterns such as the ones shown in our examples. This paper presents an overview of the fracture algorithm, the details are presented in our ACM SIGGRAPH 1999 and 2002 papers.
△ Less
Submitted 19 March, 2023;
originally announced March 2023.
-
Combining Active and Passive Simulations for Secondary Motion
Authors:
James F. O'Brien,
Victor B. Zordan,
Jessica K. Hodgins
Abstract:
Objects that move in response to the actions of a main character often make an important contribution to the visual richness of an animated scene. We use the term "secondary motion" to refer to passive motions generated in response to the movements of characters and other objects or environmental forces. Secondary motions aren't normally the mail focus of an animated scene, yet their absence can d…
▽ More
Objects that move in response to the actions of a main character often make an important contribution to the visual richness of an animated scene. We use the term "secondary motion" to refer to passive motions generated in response to the movements of characters and other objects or environmental forces. Secondary motions aren't normally the mail focus of an animated scene, yet their absence can distract or disturb the viewer, destroying the illusion of reality created by the scene. We describe how to generate secondary motion by coupling physically based simulations of passive objects to actively controlled characters.
△ Less
Submitted 18 March, 2023;
originally announced March 2023.
-
Animating Explosions
Authors:
Gary D. Yngve,
James F. O'Brien,
Jessica K. Hodgins
Abstract:
In this paper, we introduce techniques for animating explosions and their effects. The primary effect of an explosion is a disturbance that causes a shock wave to propagate through the surrounding medium. This disturbance determines the behavior of nearly all other secondary effects seen in explosions. We simulate the propagation of an explosion through the surrounding air using a computational fl…
▽ More
In this paper, we introduce techniques for animating explosions and their effects. The primary effect of an explosion is a disturbance that causes a shock wave to propagate through the surrounding medium. This disturbance determines the behavior of nearly all other secondary effects seen in explosions. We simulate the propagation of an explosion through the surrounding air using a computational fluid dynamics model based on the equations for compressible, viscous flow. To model the numerically stable formulation of shocks along blast wave fronts, we employ an integration method that can handle steep gradients without introducing inappropriate damping. The system includes two-way coupling between solid objects and surrounding fluid. Using this technique, we can generate a variety of effects including shaped explosive charges, a projectile propelled from a chamber by an explosion, and objects damaged by a blast. With appropriate rendering techniques, our explosion model can be used to create such visual effects such as fireballs, dust clouds, and the refraction of light caused by a blast wave.
△ Less
Submitted 18 March, 2023;
originally announced March 2023.
-
Automatic Joint Parameter Estimation from Magnetic Motion Capture Data
Authors:
James F. O'Brien,
Robert E. Bodenheimer,
Gabriel J. Brostow,
Jessica K. Hodgins
Abstract:
This paper describes a technique for using magnetic motion capture data to determine the joint parameters of an articulated hierarchy. This technique makes it possible to determine limb lengths, joint locations, and sensor placement for a human subject without external measurements. Instead, the joint parameters are inferred with high accuracy from the motion data acquired during the capture sessi…
▽ More
This paper describes a technique for using magnetic motion capture data to determine the joint parameters of an articulated hierarchy. This technique makes it possible to determine limb lengths, joint locations, and sensor placement for a human subject without external measurements. Instead, the joint parameters are inferred with high accuracy from the motion data acquired during the capture session. The parameters are computed by performing a linear least squares fit of a rotary joint model to the input data. A hierarchical structure for the articulated model can also be determined in situations where the topology of the model is not known. Once the system topology and joint parameters have been recovered, the resulting model can be used to perform forward and inverse kinematic procedures. We present the results of using the algorithm on human motion capture data, as well as validation results obtained with data from a simulation and a wooden linkage of known dimensions.
△ Less
Submitted 18 March, 2023;
originally announced March 2023.
-
Shape Transformation Using Variational Implicit Functions
Authors:
Greg Turk,
James F. O'Brien
Abstract:
Traditionally, shape transformation using implicit functions is performed in two distinct steps: 1) creating two implicit functions, and 2) interpolating between these two functions. We present a new shape transformation method that combines these two tasks into a single step. We create a transformation between two N-dimensional objects by casting this as a scattered data interpolation problem in…
▽ More
Traditionally, shape transformation using implicit functions is performed in two distinct steps: 1) creating two implicit functions, and 2) interpolating between these two functions. We present a new shape transformation method that combines these two tasks into a single step. We create a transformation between two N-dimensional objects by casting this as a scattered data interpolation problem in N + 1 dimensions. For the case of 2D shapes, we place all of our data constraints within two planes, one for each shape. These planes are placed parallel to one another in 3D. Zero-valued constraints specify the locations of shape boundaries and positive-valued constraints are placed along the normal direction in towards the center of the shape. We then invoke a variational interpolation technique (the 3D generalization of thin-plate interpolation), and this yields a single implicit function in 3D. Intermediate shapes are simply the zero-valued contours of 2D slices through this 3D function. Shape transformation between 3D shapes can be performed similarly by solving a 4D interpolation problem. To our knowledge, ours is the first shape transformation method to unify the tasks of implicit function creation and interpolation. The transformations produced by this method appear smooth and natural, even between objects of differing topologies. If desired, one or more additional shapes may be introduced that influence the intermediate shapes in a sequence. Our method can also reconstruct surfaces from multiple slices that are not restricted to being parallel to one another.
△ Less
Submitted 6 March, 2023;
originally announced March 2023.
-
Graphical Modeling and Animation of Brittle Fracture
Authors:
James F. O'Brien,
Jessica K. Hodgins
Abstract:
In this paper, we augment existing techniques for simulating flexible objects to include models for crack initiation and propagation in three-dimensional volumes. By analyzing the stress tensors computed over a finite element model, the simulation determines where cracks should initiate and in what directions they should propagate. We demonstrate our results with animations of breaking bowls, crac…
▽ More
In this paper, we augment existing techniques for simulating flexible objects to include models for crack initiation and propagation in three-dimensional volumes. By analyzing the stress tensors computed over a finite element model, the simulation determines where cracks should initiate and in what directions they should propagate. We demonstrate our results with animations of breaking bowls, cracking walls, and objects that fracture when they collide. By varying the shape of the objects, the material properties, and the initial conditions of the simulations, we can create strikingly different effects ranging from a wall that shatters when it is hit by a wrecking ball to a bowl that breaks in two when it is dropped on edge. This paper received the SIGGRAPH 99 Impact Award.
△ Less
Submitted 6 March, 2023;
originally announced March 2023.
-
Unique Identification of 50,000+ Virtual Reality Users from Head & Hand Motion Data
Authors:
Vivek Nair,
Wenbo Guo,
Justus Mattern,
Rui Wang,
James F. O'Brien,
Louis Rosenberg,
Dawn Song
Abstract:
With the recent explosive growth of interest and investment in virtual reality (VR) and the so-called "metaverse," public attention has rightly shifted toward the unique security and privacy threats that these platforms may pose. While it has long been known that people reveal information about themselves via their motion, the extent to which this makes an individual globally identifiable within v…
▽ More
With the recent explosive growth of interest and investment in virtual reality (VR) and the so-called "metaverse," public attention has rightly shifted toward the unique security and privacy threats that these platforms may pose. While it has long been known that people reveal information about themselves via their motion, the extent to which this makes an individual globally identifiable within virtual reality has not yet been widely understood. In this study, we show that a large number of real VR users (N=55,541) can be uniquely and reliably identified across multiple sessions using just their head and hand motion relative to virtual objects. After training a classification model on 5 minutes of data per person, a user can be uniquely identified amongst the entire pool of 50,000+ with 94.33% accuracy from 100 seconds of motion, and with 73.20% accuracy from just 10 seconds of motion. This work is the first to truly demonstrate the extent to which biomechanics may serve as a unique identifier in VR, on par with widely used biometrics such as facial or fingerprint recognition.
△ Less
Submitted 17 February, 2023;
originally announced February 2023.
-
Animating Sand, Mud, and Snow
Authors:
Robert W. Sumner,
James F. O'Brien,
Jessica K. Hodgins
Abstract:
Computer animations often lack the subtle environmental changes that should occur due to the actions of the characters. Squealing car tires usually leave no skid marks, airplanes rarely leave jet trails in the sky, and most runners leave no footprints. In this paper, we describe a simulation model of ground surfaces that can be deformed by the impact of rigid body models of animated characters. To…
▽ More
Computer animations often lack the subtle environmental changes that should occur due to the actions of the characters. Squealing car tires usually leave no skid marks, airplanes rarely leave jet trails in the sky, and most runners leave no footprints. In this paper, we describe a simulation model of ground surfaces that can be deformed by the impact of rigid body models of animated characters. To demonstrate the algorithms, we show footprints made by a runner in sand, mud, and snow as well as bicycle tire tracks, a bicycle crash, and a falling runner. The shapes of the footprints in the three surfaces are quite different, but the effects were controlled through only five essentially independent parameters. To assess the realism of the resulting motion, we compare the simulated footprints to human footprints in sand.
△ Less
Submitted 21 February, 2023; v1 submitted 16 February, 2023;
originally announced February 2023.
-
Perception of Human Motion with Different Geometric Models
Authors:
Jessica K. Hodgins,
James F. O'Brien,
Jack Tumblin
Abstract:
Human figures have been animated using a variety of geometric models including stick figures, polygonal models, and NURBS-based models with muscles, flexible skin, or clothing. This paper reports on experimental results indicating that a viewer's perception of motion characteristics is affected by the geometric model used for rendering. Subjects were shown a series of paired motion sequences and a…
▽ More
Human figures have been animated using a variety of geometric models including stick figures, polygonal models, and NURBS-based models with muscles, flexible skin, or clothing. This paper reports on experimental results indicating that a viewer's perception of motion characteristics is affected by the geometric model used for rendering. Subjects were shown a series of paired motion sequences and asked if the two motions in each pair were the same or different. The motion sequences in each pair were rendered using the same geometric model. For the three types of motion variation tested, sensitivity scores indicate that subjects were better able to observe changes with the polygonal model than they were with the stick figure model.
△ Less
Submitted 20 February, 2023; v1 submitted 15 February, 2023;
originally announced February 2023.
-
Animating Human Athletics
Authors:
Jessica K. Hodgins,
Wayne L. Wooten,
David C. Brogan,
James F. O'Brien
Abstract:
This paper describes algorithms for the animation of men and women performing three dynamic athletic behaviors: running, bicycling, and vaulting. We animate these behaviors using control algorithms that cause a physically realistic model to perform the desired maneuver. For example, control algorithms allow the simulated humans to maintain balance while moving their arms, to run or bicycle at a va…
▽ More
This paper describes algorithms for the animation of men and women performing three dynamic athletic behaviors: running, bicycling, and vaulting. We animate these behaviors using control algorithms that cause a physically realistic model to perform the desired maneuver. For example, control algorithms allow the simulated humans to maintain balance while moving their arms, to run or bicycle at a variety of speeds, and to perform a handspring vault. Algorithms for group behaviors allow a number of simulated bicyclists to ride as a group while avoiding simple patterns of obstacles. We add secondary motion to the animations with spring-mass simulations of clothing driven by the rigid-body motion of the simulated human. For each simulation, we compare the computed motion to that of humans performing similar maneuvers both qualitatively through the comparison of real and simulated video images and quantitatively through the comparison of simulated and biomechanical data.
△ Less
Submitted 13 February, 2023;
originally announced February 2023.
-
Dynamic Simulation of Splashing Fluids
Authors:
James F. O'Brien,
Jessica K. Hodgins
Abstract:
In this paper we describe a method for modeling the dynamic behavior of splashing fluids. The model simulates the behavior of a fluid when objects impact or float on its surface. The forces generated by the objects create waves and splashes on the surface of the fluid. To demonstrate the realism and limitations of the model, images from a computer-generated animation are presented and compared wit…
▽ More
In this paper we describe a method for modeling the dynamic behavior of splashing fluids. The model simulates the behavior of a fluid when objects impact or float on its surface. The forces generated by the objects create waves and splashes on the surface of the fluid. To demonstrate the realism and limitations of the model, images from a computer-generated animation are presented and compared with video frames of actual splashes occurring under similar initial conditions.
△ Less
Submitted 12 February, 2023;
originally announced February 2023.
-
Exploring the Privacy Risks of Adversarial VR Game Design
Authors:
Vivek Nair,
Gonzalo Munilla Garrido,
Dawn Song,
James F. O'Brien
Abstract:
Fifty study participants playtested an innocent-looking "escape room" game in virtual reality (VR). Within just a few minutes, an adversarial program had accurately inferred over 25 of their personal data attributes, from anthropometrics like height and wingspan to demographics like age and gender. As notoriously data-hungry companies become increasingly involved in VR development, this experiment…
▽ More
Fifty study participants playtested an innocent-looking "escape room" game in virtual reality (VR). Within just a few minutes, an adversarial program had accurately inferred over 25 of their personal data attributes, from anthropometrics like height and wingspan to demographics like age and gender. As notoriously data-hungry companies become increasingly involved in VR development, this experimental scenario may soon represent a typical VR user experience. Since the Cambridge Analytica scandal of 2018, adversarially designed gamified elements have been known to constitute a significant privacy threat in conventional social platforms. In this work, we present a case study of how metaverse environments can similarly be adversarially constructed to covertly infer dozens of personal data attributes from seemingly anonymous users. While existing VR privacy research largely focuses on passive observation, we argue that because individuals subconsciously reveal personal information via their motion in response to specific stimuli, active attacks pose an outsized risk in VR environments.
△ Less
Submitted 13 December, 2023; v1 submitted 26 July, 2022;
originally announced July 2022.