Showing 1–47 of 47 results for author: Goldman, D

Robust control for multi-legged elongate robots in noisy environments

Authors: Baxi Chong, Juntao He, Daniel Irvine, Tianyu Wang, Esteban Flores, Daniel Soto, Jianfeng Lin, Zhaochen Xu, Vincent R Nienhusser, Grigoriy Blekherman, Daniel I. Goldman

Abstract: Modern two and four legged robots exhibit impressive mobility on complex terrain, largely attributed to advancement in learning algorithms. However, these systems often rely on high-bandwidth sensing and onboard computation to perceive/respond to terrain uncertainties. Further, current locomotion strategies typically require extensive robot-specific training, limiting their generalizability across… ▽ More Modern two and four legged robots exhibit impressive mobility on complex terrain, largely attributed to advancement in learning algorithms. However, these systems often rely on high-bandwidth sensing and onboard computation to perceive/respond to terrain uncertainties. Further, current locomotion strategies typically require extensive robot-specific training, limiting their generalizability across platforms. Building on our prior research connecting robot-environment interaction and communication theory, we develop a new paradigm to construct robust and simply controlled multi-legged elongate robots (MERs) capable of operating effectively in cluttered, unstructured environments. In this framework, each leg-ground contact is thought of as a basic active contact (bac), akin to bits in signal transmission. Reliable locomotion can be achieved in open-loop on "noisy" landscapes via sufficient redundancy in bacs. In such situations, robustness is achieved through passive mechanical responses. We term such processes as those displaying mechanical intelligence (MI) and analogize these processes to forward error correction (FEC) in signal transmission. To augment MI, we develop feedback control schemes, which we refer to as computational intelligence (CI) and such processes analogize automatic repeat request (ARQ) in signal transmission. Integration of these analogies between locomotion and communication theory allow analysis, design, and prediction of embodied intelligence control schemes (integrating MI and CI) in MERs, showing effective and reliable performance (approximately half body lengths per cycle) on complex landscapes with terrain "noise" over twice the robot's height. Our work provides a foundation for systematic development of MER control, paving the way for terrain-agnostic, agile, and resilient robotic systems capable of operating in extreme environments. △ Less

Submitted 18 June, 2025; originally announced June 2025.

Fault-Tolerant Multi-Robot Coordination with Limited Sensing within Confined Environments

Authors: Kehinde O. Aina, Hosain Bagheri, Daniel I. Goldman

Abstract: As robots are increasingly deployed to collaborate on tasks within shared workspaces and resources, the failure of an individual robot can critically affect the group's performance. This issue is particularly challenging when robots lack global information or direct communication, relying instead on social interaction for coordination and to complete their tasks. In this study, we propose a novel… ▽ More As robots are increasingly deployed to collaborate on tasks within shared workspaces and resources, the failure of an individual robot can critically affect the group's performance. This issue is particularly challenging when robots lack global information or direct communication, relying instead on social interaction for coordination and to complete their tasks. In this study, we propose a novel fault-tolerance technique leveraging physical contact interactions in multi-robot systems, specifically under conditions of limited sensing and spatial confinement. We introduce the "Active Contact Response" (ACR) method, where each robot modulates its behavior based on the likelihood of encountering an inoperative (faulty) robot. Active robots are capable of collectively repositioning stationary and faulty peers to reduce obstructions and maintain optimal group functionality. We implement our algorithm in a team of autonomous robots, equipped with contact-sensing and collision-tolerance capabilities, tasked with collectively excavating cohesive model pellets. Experimental results indicate that the ACR method significantly improves the system's recovery time from robot failures, enabling continued collective excavation with minimal performance degradation. Thus, this work demonstrates the potential of leveraging local, social, and physical interactions to enhance fault tolerance and coordination in multi-robot systems operating in constrained and extreme environments. △ Less

Submitted 20 May, 2025; originally announced May 2025.

Comments: 15 pages, 4 figures. Accepted to DARS 2024 (Distributed Autonomous Robotic Systems), to appear in Springer Proceedings in Advanced Robotics

Toward Task Capable Active Matter: Learning to Avoid Clogging in Confined Collectives via Collisions

Authors: Kehinde O. Aina, Ram Avinery, Hui-Shun Kuan, Meredith D. Betterton, Michael A. D. Goodisman, Daniel I. Goldman

Abstract: Social organisms which construct nests consisting of tunnels and chambers necessarily navigate confined and crowded conditions. Unlike low-density collectives like bird flocks and insect swarms, in which hydrodynamic and statistical phenomena dominate, the physics of glasses and supercooled fluids is important to understand clogging behaviors in high-density collectives. Our previous work revealed… ▽ More Social organisms which construct nests consisting of tunnels and chambers necessarily navigate confined and crowded conditions. Unlike low-density collectives like bird flocks and insect swarms, in which hydrodynamic and statistical phenomena dominate, the physics of glasses and supercooled fluids is important to understand clogging behaviors in high-density collectives. Our previous work revealed that fire ants flowing in confined tunnels utilize diverse behaviors like unequal workload distributions, spontaneous direction reversals, and limited interaction times to mitigate clogging and jamming and thus maintain functional flow; implementation of similar rules in a small robophysical swarm led to high performance through spontaneous dissolution of clogs and clusters. However, how the insects learn such behaviors, and how we can develop "task capable" active matter in such regimes, remains a challenge in part because interaction dynamics are dominated by local, time-consuming collisions and no single agent can guide the entire collective. Here, we hypothesized that effective flow and clog mitigation could emerge purely through local learning. We tasked small groups of robots with pellet excavation in a narrow tunnel, allowing them to modify reversal probabilities over time. Initially, robots had equal probabilities and clogs were common. Reversals improved flow. When reversal probabilities adapted via collisions and noisy tunnel length estimates, workload inequality and performance improved. Our robophysical study of an excavating swarm shows that, despite the seeming complexity and difficulty of the task, simple learning rules can mitigate or leverage unavoidable features in task-capable dense active matter, leading to hypotheses for dense biological and robotic swarms. △ Less

Submitted 20 May, 2025; originally announced May 2025.

Comments: 13 pages, 9 figures. Published in Frontiers in Physics, Social Physics section. Includes experimental and simulation analysis of multi-robot excavation using decentralized learning

Journal ref: Front. Phys., 10:73566 (2022)

Collision Induced Binding and Transport of Shape Changing Robot Pairs

Authors: Akash Vardhan, Ram Avinery, Hosain Bagheri, Velin Kojohourav, Shengkai Li, Hridesh Kedia, Tianyu Wang, Daniel Soto, Kurt Wiesenfeld, Daniel I. Goldman

Abstract: We report in experiment and simulation the spontaneous formation of dynamically bound pairs of shape changing robots undergoing locally repulsive collisions. These physical `gliders' robustly emerge from an ensemble of individually undulating three-link two-motor robots and can remain bound for hundreds of undulations and travel for multiple robot dimensions. Gliders occur in two distinct binding… ▽ More We report in experiment and simulation the spontaneous formation of dynamically bound pairs of shape changing robots undergoing locally repulsive collisions. These physical `gliders' robustly emerge from an ensemble of individually undulating three-link two-motor robots and can remain bound for hundreds of undulations and travel for multiple robot dimensions. Gliders occur in two distinct binding symmetries and form over a wide range of angular oscillation extent. This parameter sets the maximal concavity which influences formation probability and translation characteristics. Analysis of dynamics in simulation reveals the mechanism of effective dynamical attraction -- a result of the emergent interplay of appropriately oriented and timed repulsive interactions. Tactile sensing stabilizes the short-lived conformation via concavity modulation. △ Less

Submitted 5 June, 2025; v1 submitted 19 April, 2025; originally announced April 2025.

Comments: 7 pages, 6 figures, submitted to PRL

Tactile sensing enables vertical obstacle negotiation for elongate many-legged robots

Authors: Juntao He, Baxi Chong, Massimiliano Iaschi, Vincent R. Nienhusser, Sehoon Ha, Daniel I. Goldman

Abstract: Many-legged elongated robots show promise for reliable mobility on rugged landscapes. However, most studies on these systems focus on planar motion planning without addressing rapid vertical motion. Despite their success on mild rugged terrains, recent field tests reveal a critical need for 3D behaviors (e.g., climbing or traversing tall obstacles). The challenges of 3D motion planning partially l… ▽ More Many-legged elongated robots show promise for reliable mobility on rugged landscapes. However, most studies on these systems focus on planar motion planning without addressing rapid vertical motion. Despite their success on mild rugged terrains, recent field tests reveal a critical need for 3D behaviors (e.g., climbing or traversing tall obstacles). The challenges of 3D motion planning partially lie in designing sensing and control for a complex high-degree-of-freedom system, typically with over 25 degrees of freedom. To address the first challenge regarding sensing, we propose a tactile antenna system that enables the robot to probe obstacles to gather information about their structure. Building on this sensory input, we develop a control framework that integrates data from the antenna and foot contact sensors to dynamically adjust the robot's vertical body undulation for effective climbing. With the addition of simple, low-bandwidth tactile sensors, a robot with high static stability and redundancy exhibits predictable climbing performance in complex environments using a simple feedback controller. Laboratory and outdoor experiments demonstrate the robot's ability to climb obstacles up to five times its height. Moreover, the robot exhibits robust climbing capabilities on obstacles covered with shifting, robot-sized random items and those characterized by rapidly changing curvatures. These findings demonstrate an alternative solution to perceive the environment and facilitate effective response for legged robots, paving ways towards future highly capable, low-profile many-legged robots. △ Less

Submitted 21 April, 2025; v1 submitted 11 April, 2025; originally announced April 2025.

Comments: Accepted by RSS 2025

Probabilistic approach to feedback control enhances multi-legged locomotion on rugged landscapes

Authors: Juntao He, Baxi Chong, Jianfeng Lin, Zhaochen Xu, Hosain Bagheri, Esteban Flores, Daniel I. Goldman

Abstract: Achieving robust legged locomotion on complex terrains poses challenges due to the high uncertainty in robot-environment interactions. Recent advances in bipedal and quadrupedal robots demonstrate good mobility on rugged terrains but rely heavily on sensors for stability due to low static stability from a high center of mass and a narrow base of support. We hypothesize that a multi-legged robotic… ▽ More Achieving robust legged locomotion on complex terrains poses challenges due to the high uncertainty in robot-environment interactions. Recent advances in bipedal and quadrupedal robots demonstrate good mobility on rugged terrains but rely heavily on sensors for stability due to low static stability from a high center of mass and a narrow base of support. We hypothesize that a multi-legged robotic system can leverage morphological redundancy from additional legs to minimize sensing requirements when traversing challenging terrains. Studies suggest that a multi-legged system with sufficient legs can reliably navigate noisy landscapes without sensing and control, albeit at a low speed of up to 0.1 body lengths per cycle (BLC). However, the control framework to enhance speed on challenging terrains remains underexplored due to the complex environmental interactions, making it difficult to identify the key parameters to control in these high-degree-of-freedom systems. Here, we present a bio-inspired vertical body undulation wave as a novel approach to mitigate environmental disturbances affecting robot speed, supported by experiments and probabilistic models. Finally, we introduce a control framework which monitors foot-ground contact patterns on rugose landscapes using binary foot-ground contact sensors to estimate terrain rugosity. The controller adjusts the vertical body wave based on the deviation of the limb's averaged actual-to-ideal foot-ground contact ratio, achieving a significant enhancement of up to 0.235 BLC on rugose laboratory terrain. We observed a $\sim$ 50\% increase in speed and a $\sim$ 40\% reduction in speed variance compared to the open-loop controller. Additionally, the controller operates in complex terrains outside the lab, including pine straw, robot-sized rocks, mud, and leaves. △ Less

Submitted 30 April, 2025; v1 submitted 11 November, 2024; originally announced November 2024.

Comments: Submitted to IEEE Transactions on Robotics (T-RO)

Effective self-righting strategies for elongate multi-legged robots

Authors: Erik Teder, Baxi Chong, Juntao He, Tianyu Wang, Massimiliano Iaschi, Daniel Soto, Daniel I Goldman

Abstract: Centipede-like robots offer an effective and robust solution to navigation over complex terrain with minimal sensing. However, when climbing over obstacles, such multi-legged robots often elevate their center-of-mass into unstable configurations, where even moderate terrain uncertainty can cause tipping over. Robust mechanisms for such elongate multi-legged robots to self-right remain unstudied. H… ▽ More Centipede-like robots offer an effective and robust solution to navigation over complex terrain with minimal sensing. However, when climbing over obstacles, such multi-legged robots often elevate their center-of-mass into unstable configurations, where even moderate terrain uncertainty can cause tipping over. Robust mechanisms for such elongate multi-legged robots to self-right remain unstudied. Here, we developed a comparative biological and robophysical approach to investigate self-righting strategies. We first released \textit{S. polymorpha} upside down from a 10 cm height and recorded their self-righting behaviors using top and side view high-speed cameras. Using kinematic analysis, we hypothesize that these behaviors can be prescribed by two traveling waves superimposed in the body lateral and vertical planes, respectively. We tested our hypothesis on an elongate robot with static (non-actuated) limbs, and we successfully reconstructed these self-righting behaviors. We further evaluated how wave parameters affect self-righting effectiveness. We identified two key wave parameters: the spatial frequency, which characterizes the sequence of body-rolling, and the wave amplitude, which characterizes body curvature. By empirically obtaining a behavior diagram of spatial frequency and amplitude, we identify effective and versatile self-righting strategies for general elongate multi-legged robots, which greatly enhances these robots' mobility and robustness in practical applications such as agricultural terrain inspection and search-and-rescue. △ Less

Submitted 1 October, 2024; originally announced October 2024.

Steering Elongate Multi-legged Robots By Modulating Body Undulation Waves

Authors: Esteban Flores, Baxi Chong, Daniel Soto, Daniel I. Goldman

Abstract: Centipedes exhibit great maneuverability in diverse environments due to their many legs and body-driven control. By leveraging similar morphologies and control strategies, their robotic counterparts also demonstrate effective terrestrial locomotion. However, the success of these multi-legged robots is largely limited to forward locomotion; steering is substantially less studied, in part because of… ▽ More Centipedes exhibit great maneuverability in diverse environments due to their many legs and body-driven control. By leveraging similar morphologies and control strategies, their robotic counterparts also demonstrate effective terrestrial locomotion. However, the success of these multi-legged robots is largely limited to forward locomotion; steering is substantially less studied, in part because of the difficulty in coordinating a high degree-of-freedom robot to follow predictable, planar trajectories. To resolve these challenges, we take inspiration from control schemes based on geometric mechanics(GM) in elongate system's locomotion through highly damped environments. We model the elongate, multi-legged system as a ``terrestrial swimmer" in highly frictional environments and implement steering schemes derived from low-order templates of elongate, limbless systems. We identify an effective turning strategy by superimposing two traveling waves of lateral body undulation and further explore variations of the ``turning wave" to enable a spectrum of arc-following steering primitives. We test our hypothesized modulation scheme on a robophysical model and validate steering trajectories against theoretically predicted displacements. We then apply our control framework to Ground Control Robotics' elongate multi-legged robot, Major Tom, using these motion primitives to construct planar motion and in closed-loop control on different terrains. Our work creates a systematic framework for controlling these highly mobile devices in the plane using a low-order model based on sequences of body shape changes. △ Less

Submitted 2 June, 2025; v1 submitted 1 October, 2024; originally announced October 2024.

Addition of a peristaltic wave improves multi-legged locomotion performance on complex terrains

Authors: Massimiliano Iaschi, Baxi Chong, Tianyu Wang, Jianfeng Lin, Juntao He, Daniel Soto, Zhaochen Xu, Daniel I Goldman

Abstract: Characterized by their elongate bodies and relatively simple legs, multi-legged robots have the potential to locomote through complex terrains for applications such as search-and-rescue and terrain inspection. Prior work has developed effective and reliable locomotion strategies for multi-legged robots by propagating the two waves of lateral body undulation and leg stepping, which we will refer to… ▽ More Characterized by their elongate bodies and relatively simple legs, multi-legged robots have the potential to locomote through complex terrains for applications such as search-and-rescue and terrain inspection. Prior work has developed effective and reliable locomotion strategies for multi-legged robots by propagating the two waves of lateral body undulation and leg stepping, which we will refer to as the two-wave template. However, these robots have limited capability to climb over obstacles with sizes comparable to their heights. We hypothesize that such limitations stem from the two-wave template that we used to prescribe the multi-legged locomotion. Seeking effective alternative waves for obstacle-climbing, we designed a five-segment robot with static (non-actuated) legs, where each cable-driven joint has a rotational degree-of-freedom (DoF) in the sagittal plane (vertical wave) and a linear DoF (peristaltic wave). We tested robot locomotion performance on a flat terrain and a rugose terrain. While the benefit of peristalsis on flat-ground locomotion is marginal, the inclusion of a peristaltic wave substantially improves the locomotion performance in rugose terrains: it not only enables obstacle-climbing capabilities with obstacles having a similar height as the robot, but it also significantly improves the traversing capabilities of the robot in such terrains. Our results demonstrate an alternative actuation mechanism for multi-legged robots, paving the way towards all-terrain multi-legged robots. △ Less

Submitted 1 October, 2024; originally announced October 2024.

AquaMILR+: Design of an untethered limbless robot for complex aquatic terrain navigation

Authors: Matthew Fernandez, Tianyu Wang, Galen Tunnicliffe, Donoven Dortilus, Peter Gunnarson, John O. Dabiri, Daniel I. Goldman

Abstract: This paper presents AquaMILR+, an untethered limbless robot designed for agile navigation in complex aquatic environments. The robot features a bilateral actuation mechanism that models musculoskeletal actuation in many anguilliform swimming organisms which propagates a moving wave from head to tail allowing open fluid undulatory swimming. This actuation mechanism employs mechanical intelligence,… ▽ More This paper presents AquaMILR+, an untethered limbless robot designed for agile navigation in complex aquatic environments. The robot features a bilateral actuation mechanism that models musculoskeletal actuation in many anguilliform swimming organisms which propagates a moving wave from head to tail allowing open fluid undulatory swimming. This actuation mechanism employs mechanical intelligence, enhancing the robot's maneuverability when interacting with obstacles. AquaMILR+ also includes a compact depth control system inspired by the swim bladder and lung structures of eels and sea snakes. The mechanism, driven by a syringe and telescoping leadscrew, enables depth and pitch control-capabilities that are difficult for most anguilliform swimming robots to achieve. Additional structures, such as fins and a tail, further improve stability and propulsion efficiency. Our tests in both open water and indoor 2D and 3D heterogeneous aquatic environments highlight AquaMILR+'s capabilities and suggest a promising system for complex underwater tasks such as search and rescue and deep-sea exploration. △ Less

Submitted 26 September, 2024; originally announced September 2024.

Learning to enhance multi-legged robot on rugged landscapes

Authors: Juntao He, Baxi Chong, Zhaochen Xu, Sehoon Ha, Daniel I. Goldman

Abstract: Navigating rugged landscapes poses significant challenges for legged locomotion. Multi-legged robots (those with 6 and greater) offer a promising solution for such terrains, largely due to their inherent high static stability, resulting from a low center of mass and wide base of support. Such systems require minimal effort to maintain balance. Recent studies have shown that a linear controller, wh… ▽ More Navigating rugged landscapes poses significant challenges for legged locomotion. Multi-legged robots (those with 6 and greater) offer a promising solution for such terrains, largely due to their inherent high static stability, resulting from a low center of mass and wide base of support. Such systems require minimal effort to maintain balance. Recent studies have shown that a linear controller, which modulates the vertical body undulation of a multi-legged robot in response to shifts in terrain roughness, can ensure reliable mobility on challenging terrains. However, the potential of a learning-based control framework that adjusts multiple parameters to address terrain heterogeneity remains underexplored. We posit that the development of an experimentally validated physics-based simulator for this robot can rapidly advance capabilities by allowing wide parameter space exploration. Here we develop a MuJoCo-based simulator tailored to this robotic platform and use the simulation to develop a reinforcement learning-based control framework that dynamically adjusts horizontal and vertical body undulation, and limb stepping in real-time. Our approach improves robot performance in simulation, laboratory experiments, and outdoor tests. Notably, our real-world experiments reveal that the learning-based controller achieves a 30\% to 50\% increase in speed compared to a linear controller, which only modulates vertical body waves. We hypothesize that the superior performance of the learning-based controller arises from its ability to adjust multiple parameters simultaneously, including limb stepping, horizontal body wave, and vertical body wave. △ Less

Submitted 14 September, 2024; originally announced September 2024.

Comments: Submitted to ICRA 2025

AquaMILR: Mechanical intelligence simplifies control of undulatory robots in cluttered fluid environments

Authors: Tianyu Wang, Nishanth Mankame, Matthew Fernandez, Velin Kojouharov, Daniel I. Goldman

Abstract: While undulatory swimming of elongate limbless robots has been extensively studied in open hydrodynamic environments, less research has been focused on limbless locomotion in complex, cluttered aquatic environments. Motivated by the concept of mechanical intelligence, where controls for obstacle navigation can be offloaded to passive body mechanics in terrestrial limbless locomotion, we hypothesiz… ▽ More While undulatory swimming of elongate limbless robots has been extensively studied in open hydrodynamic environments, less research has been focused on limbless locomotion in complex, cluttered aquatic environments. Motivated by the concept of mechanical intelligence, where controls for obstacle navigation can be offloaded to passive body mechanics in terrestrial limbless locomotion, we hypothesize that principles of mechanical intelligence can be extended to cluttered hydrodynamic regimes. To test this, we developed an untethered limbless robot capable of undulatory swimming on water surfaces, utilizing a bilateral cable-driven mechanism inspired by organismal muscle actuation morphology to achieve programmable anisotropic body compliance. We demonstrated through robophysical experiments that, similar to terrestrial locomotion, an appropriate level of body compliance can facilitate emergent swim through complex hydrodynamic environments under pure open-loop control. Moreover, we found that swimming performance depends on undulation frequency, with effective locomotion achieved only within a specific frequency range. This contrasts with highly damped terrestrial regimes, where inertial effects can often be neglected. Further, to enhance performance and address the challenges posed by nondeterministic obstacle distributions, we incorporated computational intelligence by developing a real-time body compliance tuning controller based on cable tension feedback. This controller improves the robot's robustness and overall speed in heterogeneous hydrodynamic environments. △ Less

Submitted 26 September, 2024; v1 submitted 1 July, 2024; originally announced July 2024.

Generative AI for 2D Character Animation

Authors: Jaime Guajardo, Ozgun Bursalioglu, Dan B Goldman

Abstract: In this pilot project, we teamed up with artists to develop new workflows for 2D animation while producing a short educational cartoon. We identified several workflows to streamline the animation process, bringing the artists' vision to the screen more effectively. In this pilot project, we teamed up with artists to develop new workflows for 2D animation while producing a short educational cartoon. We identified several workflows to streamline the animation process, bringing the artists' vision to the screen more effectively. △ Less

Submitted 21 May, 2024; v1 submitted 17 May, 2024; originally announced May 2024.

Comments: Project page: https://genai-2d-character-animation.github.io/

BoLD: Fast and Cheap Dispute Resolution

Authors: Mario M. Alvarez, Henry Arneson, Ben Berger, Lee Bousfield, Chris Buckland, Yafah Edelman, Edward W. Felten, Daniel Goldman, Raul Jordan, Mahimna Kelkar, Akaki Mamageishvili, Harry Ng, Aman Sanghi, Victor Shoup, Terence Tsao

Abstract: BoLD is a new dispute resolution protocol that is designed to replace the originally deployed Arbitrum dispute resolution protocol. Unlike that protocol, BoLD is resistant to delay attacks. It achieves this resistance without a significant increase in onchain computation costs and with reduced staking costs. BoLD is a new dispute resolution protocol that is designed to replace the originally deployed Arbitrum dispute resolution protocol. Unlike that protocol, BoLD is resistant to delay attacks. It achieves this resistance without a significant increase in onchain computation costs and with reduced staking costs. △ Less

Submitted 16 April, 2024; originally announced April 2024.

Inpaint3D: 3D Scene Content Generation using 2D Inpainting Diffusion

Authors: Kira Prabhu, Jane Wu, Lynn Tsai, Peter Hedman, Dan B Goldman, Ben Poole, Michael Broxton

Abstract: This paper presents a novel approach to inpainting 3D regions of a scene, given masked multi-view images, by distilling a 2D diffusion model into a learned 3D scene representation (e.g. a NeRF). Unlike 3D generative methods that explicitly condition the diffusion model on camera pose or multi-view information, our diffusion model is conditioned only on a single masked 2D image. Nevertheless, we sh… ▽ More This paper presents a novel approach to inpainting 3D regions of a scene, given masked multi-view images, by distilling a 2D diffusion model into a learned 3D scene representation (e.g. a NeRF). Unlike 3D generative methods that explicitly condition the diffusion model on camera pose or multi-view information, our diffusion model is conditioned only on a single masked 2D image. Nevertheless, we show that this 2D diffusion model can still serve as a generative prior in a 3D multi-view reconstruction problem where we optimize a NeRF using a combination of score distillation sampling and NeRF reconstruction losses. Predicted depth is used as additional supervision to encourage accurate geometry. We compare our approach to 3D inpainting methods that focus on object removal. Because our method can generate content to fill any 3D masked region, we additionally demonstrate 3D object completion, 3D object replacement, and 3D scene completion. △ Less

Submitted 6 December, 2023; originally announced December 2023.

Readout Guidance: Learning Control from Diffusion Features

Authors: Grace Luo, Trevor Darrell, Oliver Wang, Dan B Goldman, Aleksander Holynski

Abstract: We present Readout Guidance, a method for controlling text-to-image diffusion models with learned signals. Readout Guidance uses readout heads, lightweight networks trained to extract signals from the features of a pre-trained, frozen diffusion model at every timestep. These readouts can encode single-image properties, such as pose, depth, and edges; or higher-order properties that relate multiple… ▽ More We present Readout Guidance, a method for controlling text-to-image diffusion models with learned signals. Readout Guidance uses readout heads, lightweight networks trained to extract signals from the features of a pre-trained, frozen diffusion model at every timestep. These readouts can encode single-image properties, such as pose, depth, and edges; or higher-order properties that relate multiple images, such as correspondence and appearance similarity. Furthermore, by comparing the readout estimates to a user-defined target, and back-propagating the gradient through the readout head, these estimates can be used to guide the sampling process. Compared to prior methods for conditional generation, Readout Guidance requires significantly fewer added parameters and training samples, and offers a convenient and simple recipe for reproducing different forms of conditional control under a single framework, with a single architecture and sampling procedure. We showcase these benefits in the applications of drag-based manipulation, identity-consistent generation, and spatially aligned control. Project page: https://readout-guidance.github.io. △ Less

Submitted 2 April, 2024; v1 submitted 4 December, 2023; originally announced December 2023.

Comments: CVPR 2024

Learning manipulation of steep granular slopes for fast Mini Rover turning

Authors: Deniz Kerimoglu, Daniel Soto, Malone Lincoln Hemsley, Joseph Brunner, Sehoon Ha, Tingnan Zhang, Daniel I. Goldman

Abstract: Future planetary exploration missions will require reaching challenging regions such as craters and steep slopes. Such regions are ubiquitous and present science-rich targets potentially containing information regarding the planet's internal structure. Steep slopes consisting of low-cohesion regolith are prone to flow downward under small disturbances, making it very challenging for autonomous rov… ▽ More Future planetary exploration missions will require reaching challenging regions such as craters and steep slopes. Such regions are ubiquitous and present science-rich targets potentially containing information regarding the planet's internal structure. Steep slopes consisting of low-cohesion regolith are prone to flow downward under small disturbances, making it very challenging for autonomous rovers to traverse. Moreover, the navigation trajectories of rovers are heavily limited by the terrain topology and future systems will need to maneuver on flowable surfaces without getting trapped, allowing them to further expand their reach and increase mission efficiency. In this work, we used a laboratory-scale rover robot and performed maneuvering experiments on a steep granular slope of poppy seeds to explore the rover's turning capabilities. The rover is capable of lifting, sweeping, and spinning its wheels, allowing it to execute leg-like gait patterns. The high-dimensional actuation capabilities of the rover facilitate effective manipulation of the underlying granular surface. We used Bayesian Optimization (BO) to gain insight into successful turning gaits in high dimensional search space and found strategies such as differential wheel spinning and pivoting around a single sweeping wheel. We then used these insights to further fine-tune the turning gait, enabling the rover to turn 90 degrees at just above 4 seconds with minimal slip. Combining gait optimization and human-tuning approaches, we found that fast turning is empowered by creating anisotropic torques with the sweeping wheel. △ Less

Submitted 2 October, 2023; originally announced October 2023.

Comments: 6 pages, 6 figures, conference paper submission for ICRA2024

Anisotropic body compliance facilitates robotic sidewinding in complex environments

Authors: Velin Kojouharov, Tianyu Wang, Matthew Fernandez, Jiyeon Maeng, Daniel I. Goldman

Abstract: Sidewinding, a locomotion strategy characterized by the coordination of lateral and vertical body undulations, is frequently observed in rattlesnakes and has been successfully reconstructed by limbless robotic systems for effective movement across diverse terrestrial terrains. However, the integration of compliant mechanisms into sidewinding limbless robots remains less explored, posing challenges… ▽ More Sidewinding, a locomotion strategy characterized by the coordination of lateral and vertical body undulations, is frequently observed in rattlesnakes and has been successfully reconstructed by limbless robotic systems for effective movement across diverse terrestrial terrains. However, the integration of compliant mechanisms into sidewinding limbless robots remains less explored, posing challenges for navigation in complex, rheologically diverse environments. Inspired by a notable control simplification via mechanical intelligence in lateral undulation, which offloads feedback control to passive body mechanics and interactions with the environment, we present an innovative design of a mechanically intelligent limbless robot for sidewinding. This robot features a decentralized bilateral cable actuation system that resembles organismal muscle actuation mechanisms. We develop a feedforward controller that incorporates programmable body compliance into the sidewinding gait template. Our experimental results highlight the emergence of mechanical intelligence when the robot is equipped with an appropriate level of body compliance. This allows the robot to 1) locomote more energetically efficiently, as evidenced by a reduced cost of transport, and 2) navigate through terrain heterogeneities, all achieved in an open-loop manner, without the need for environmental awareness. △ Less

Submitted 23 September, 2023; originally announced September 2023.

Robust self-propulsion in sand using simply controlled vibrating cubes

Authors: Bangyuan Liu, Tianyu Wang, Velin Kojouharov, Frank L. Hammond III, Daniel I. Goldman

Abstract: Much of the Earth and many surfaces of extraterrestrial bodies are composed of in-cohesive particle matter. Locomoting on granular terrain is challenging for common robotic devices, either wheeled or legged. In this work, we discover a robust alternative locomotion mechanism on granular media -- generating movement via self-vibration. To demonstrate the effectiveness of this locomotion mechanism,… ▽ More Much of the Earth and many surfaces of extraterrestrial bodies are composed of in-cohesive particle matter. Locomoting on granular terrain is challenging for common robotic devices, either wheeled or legged. In this work, we discover a robust alternative locomotion mechanism on granular media -- generating movement via self-vibration. To demonstrate the effectiveness of this locomotion mechanism, we develop a cube-shaped robot with an embedded vibratory motor and conduct systematic experiments on diverse granular terrains of various particle properties. We investigate how locomotion changes as a function of vibration frequency/intensity on granular terrains. Compared to hard surfaces, we find such a vibratory locomotion mechanism enables the robot to move faster, and more stable on granular surfaces, facilitated by the interaction between the body and surrounding granules. The simplicity in structural design and controls of this robotic system indicates that vibratory locomotion can be a valuable alternative way to produce robust locomotion on granular terrains. We further demonstrate that such cube-shape robots can be used as modular units for morphologically structured vibratory robots with capabilities of maneuverable forward and turning motions, showing potential practical scenarios for robotic systems. △ Less

Submitted 22 September, 2023; originally announced September 2023.

Multi-legged matter transport: a framework for locomotion on noisy landscapes

Authors: Baxi Chong, Juntao He, Daniel Soto, Tianyu Wang, Daniel Irvine, Grigoriy Blekherman, Daniel I. Goldman

Abstract: While the transport of matter by wheeled vehicles or legged robots can be guaranteed in engineered landscapes like roads or rails, locomotion prediction in complex environments like collapsed buildings or crop fields remains challenging. Inspired by principles of information transmission which allow signals to be reliably transmitted over noisy channels, we develop a ``matter transport" framework… ▽ More While the transport of matter by wheeled vehicles or legged robots can be guaranteed in engineered landscapes like roads or rails, locomotion prediction in complex environments like collapsed buildings or crop fields remains challenging. Inspired by principles of information transmission which allow signals to be reliably transmitted over noisy channels, we develop a ``matter transport" framework demonstrating that non-inertial locomotion can be provably generated over ``noisy" rugose landscapes (heterogeneities on the scale of locomotor dimensions). Experiments confirm that sufficient spatial redundancy in the form of serially-connected legged robots leads to reliable transport on such terrain without requiring sensing and control. Further analogies from communication theory coupled to advances in gaits (coding) and sensor-based feedback control (error detection/correction) can lead to agile locomotion in complex terradynamic regimes. △ Less

Submitted 8 May, 2023; originally announced May 2023.

Mechanical Intelligence Simplifies Control in Terrestrial Limbless Locomotion

Authors: Tianyu Wang, Christopher Pierce, Velin Kojouharov, Baxi Chong, Kelimar Diaz, Hang Lu, Daniel I. Goldman

Abstract: Limbless locomotors, from microscopic worms to macroscopic snakes, traverse complex, heterogeneous natural environments typically using undulatory body wave propagation. Theoretical and robophysical models typically emphasize body kinematics and active neural/electronic control. However, we contend that because such approaches often neglect the role of passive, mechanically controlled processes (t… ▽ More Limbless locomotors, from microscopic worms to macroscopic snakes, traverse complex, heterogeneous natural environments typically using undulatory body wave propagation. Theoretical and robophysical models typically emphasize body kinematics and active neural/electronic control. However, we contend that because such approaches often neglect the role of passive, mechanically controlled processes (those involving "mechanical intelligence"), they fail to reproduce the performance of even the simplest organisms. To uncover principles of how mechanical intelligence aids limbless locomotion in heterogeneous terradynamic regimes, here we conduct a comparative study of locomotion in a model of heterogeneous terrain (lattices of rigid posts). We used a model biological system, the highly studied nematode worm Caenorhabditis elegans, and a robophysical device whose bilateral actuator morphology models that of limbless organisms across scales. The robot's kinematics quantitatively reproduced the performance of the nematodes with purely open-loop control; mechanical intelligence simplified control of obstacle navigation and exploitation by reducing the need for active sensing and feedback. An active behavior observed in C. elegans, undulatory wave reversal upon head collisions, robustified locomotion via exploitation of the systems' mechanical intelligence. Our study provides insights into how neurally simple limbless organisms like nematodes can leverage mechanical intelligence via appropriately tuned bilateral actuation to locomote in complex environments. These principles likely apply to neurally more sophisticated organisms and also provide a design and control paradigm for limbless robots for applications like search and rescue and planetary exploration. △ Less

Submitted 1 February, 2024; v1 submitted 17 April, 2023; originally announced April 2023.

Comments: Published in Science Robotics

Journal ref: Sci. Robot. 8, eadi2243 (2023)

Gait design for limbless obstacle aided locomotion using geometric mechanics

Authors: Baxi Chong, Tianyu Wang, Daniel Irvine, Velin Kojouharov, Bo Lin, Howie Choset, Daniel I. Goldman, Grigoriy Blekherman

Abstract: Limbless robots have the potential to maneuver through cluttered environments that conventional robots cannot traverse. As illustrated in their biological counterparts such as snakes and nematodes, limbless locomotors can benefit from interactions with obstacles, yet such obstacle-aided locomotion (OAL) requires properly coordinated high-level self-deformation patterns (gait templates) as well as… ▽ More Limbless robots have the potential to maneuver through cluttered environments that conventional robots cannot traverse. As illustrated in their biological counterparts such as snakes and nematodes, limbless locomotors can benefit from interactions with obstacles, yet such obstacle-aided locomotion (OAL) requires properly coordinated high-level self-deformation patterns (gait templates) as well as low-level body adaptation to environments. Most prior work on OAL utilized stereotyped traveling-wave gait templates and relied on local body deformations (e.g., passive body mechanics or decentralized controller parameter adaptation based on force feedback) for obstacle navigation, while gait template design for OAL remains less studied. In this paper, we explore novel gait templates for OAL based on tools derived from geometric mechanics (GM), which thus far has been limited to homogeneous environments. Here, we expand the scope of GM to obstacle-rich environments. Specifically, we establish a model that maps the presence of an obstacle to directional constraints in optimization. In doing so, we identify novel gait templates suitable for sparsely and densely distributed obstacle-rich environments respectively. Open-loop robophysical experiments verify the effectiveness of our identified OAL gaits in obstacle-rich environments. We posit that when such OAL gait templates are augmented with appropriate sensing and feedback controls, limbless locomotors will gain robust function in obstacle rich environments. △ Less

Submitted 13 February, 2023; originally announced February 2023.

Geometry of contact: contact planning for multi-legged robots via spin models duality

Authors: Baxi Chong, Di Luo, Tianyu Wang, Gabriel Margolis, Juntao He, Pulkit Agrawal, Marin Soljačić, Daniel I. Goldman

Abstract: Contact planning is crucial in locomoting systems.Specifically, appropriate contact planning can enable versatile behaviors (e.g., sidewinding in limbless locomotors) and facilitate speed-dependent gait transitions (e.g., walk-trot-gallop in quadrupedal locomotors). The challenges of contact planning include determining not only the sequence by which contact is made and broken between the locomoto… ▽ More Contact planning is crucial in locomoting systems.Specifically, appropriate contact planning can enable versatile behaviors (e.g., sidewinding in limbless locomotors) and facilitate speed-dependent gait transitions (e.g., walk-trot-gallop in quadrupedal locomotors). The challenges of contact planning include determining not only the sequence by which contact is made and broken between the locomotor and the environments, but also the sequence of internal shape changes (e.g., body bending and limb shoulder joint oscillation). Most state-of-art contact planning algorithms focused on conventional robots (e.g.biped and quadruped) and conventional tasks (e.g. forward locomotion), and there is a lack of study on general contact planning in multi-legged robots. In this paper, we show that using geometric mechanics framework, we can obtain the global optimal contact sequence given the internal shape changes sequence. Therefore, we simplify the contact planning problem to a graph optimization problem to identify the internal shape changes. Taking advantages of the spatio-temporal symmetry in locomotion, we map the graph optimization problem to special cases of spin models, which allows us to obtain the global optima in polynomial time. We apply our approach to develop new forward and sidewinding behaviors in a hexapod and a 12-legged centipede. We verify our predictions using numerical and robophysical models, and obtain novel and effective locomotion behaviors. △ Less

Submitted 7 February, 2023; v1 submitted 6 February, 2023; originally announced February 2023.

Comments: SI video: https://doi.org/10.5281/zenodo.7608693

Report number: MIT-CTP/5526

A robophysical model of spacetime dynamics

Authors: Shengkai Li, Hussain N. Gynai, Steven Tarr, Emily Alicea-Muñoz, Pablo Laguna, Gongjie Li, Daniel I. Goldman

Abstract: Systems consisting of spheres rolling on elastic membranes have been used to introduce a core conceptual idea of General Relativity (GR): how curvature guides the movement of matter. However, such schemes cannot accurately represent relativistic dynamics in the laboratory because of the dominance of dissipation and external gravitational fields. Here we demonstrate that an ``active" object (a whee… ▽ More Systems consisting of spheres rolling on elastic membranes have been used to introduce a core conceptual idea of General Relativity (GR): how curvature guides the movement of matter. However, such schemes cannot accurately represent relativistic dynamics in the laboratory because of the dominance of dissipation and external gravitational fields. Here we demonstrate that an ``active" object (a wheeled robot), which moves in a straight line on level ground and can alter its speed depending on the curvature of the deformable terrain it moves on, can exactly capture dynamics in curved relativistic spacetimes. Via the systematic study of the robot's dynamics in the radial and orbital directions, we develop a mapping of the emergent trajectories of a wheeled vehicle on a spandex membrane to the motion in a curved spacetime. Our mapping demonstrates how the driven robot's dynamics mix space and time in a metric, and shows how active particles do not necessarily follow geodesics in the real space but instead follow geodesics in a fiducial spacetime. The mapping further reveals how parameters such as the membrane elasticity and instantaneous speed allow the programming of a desired spacetime, such as the Schwarzschild metric near a non-rotating blackhole. Our mapping and framework facilitate creation of a robophysical analog to a general relativistic system in the laboratory at low cost that can provide insights into active matter in deformable environments and robot exploration in complex landscapes. △ Less

Submitted 6 October, 2023; v1 submitted 10 February, 2022; originally announced February 2022.

Journal ref: Scientific Reports 13, 21589 (2023)

Generalized Omega Turn Gait Enables Agile Limbless Robot Turning in Complex Environments

Authors: Tianyu Wang, Baxi Chong, Yuelin Deng, Ruijie Fu, Howie Choset, Daniel I. Goldman

Abstract: Reorientation (turning in plane) plays a critical role for all robots in any field application, especially those that in confined spaces. While important, reorientation remains a relatively unstudied problem for robots, including limbless mechanisms, often called snake robots. Instead of looking at snakes, we take inspiration from observations of the turning behavior of tiny nematode worms C. eleg… ▽ More Reorientation (turning in plane) plays a critical role for all robots in any field application, especially those that in confined spaces. While important, reorientation remains a relatively unstudied problem for robots, including limbless mechanisms, often called snake robots. Instead of looking at snakes, we take inspiration from observations of the turning behavior of tiny nematode worms C. elegans. Our previous work presented an in-place and in-plane turning gait for limbless robots, called an omega turn, and prescribed it using a novel two-wave template. In this work, we advance omega turn-inspired controllers in three aspects: 1) we use geometric methods to vary joint angle amplitudes and forward wave spatial frequency in our turning equation to establish a wide and precise amplitude modulation and frequency modulation on omega turn; 2) we use this new relationship to enable robots with fewer internal degrees of freedom (i.e., fewer joints in the body) to achieve desirable performance, and 3) we apply compliant control methods to this relationship to handle unmodelled effects in the environment. We experimentally validate our approach on a limbless robot that the omega turn can produce effective and robust turning motion in various types of environments, such as granular media and rock pile. △ Less

Submitted 3 March, 2022; v1 submitted 3 February, 2022; originally announced February 2022.

Comments: Accepted to ICRA 2022

A general locomotion control framework for multi-legged locomotors

Authors: Baxi Chong, Yasemin O. Aydin, Jennifer M. Rieser, Guillaume Sartoretti, Tianyu Wang, Julian Whitman, Abdul Kaba, Enes Aydin, Ciera McFarland, Kelimar Diaz Cruz, Jeffery W. Rankin, Krijn B Michel, Alfredo Nicieza, John R Hutchinson, Howie Choset, Daniel I. Goldman

Abstract: Serially connected robots are promising candidates for performing tasks in confined spaces such as search-and-rescue in large-scale disasters. Such robots are typically limbless, and we hypothesize that the addition of limbs could improve mobility. However, a challenge in designing and controlling such devices lies in the coordination of high-dimensional redundant modules in a way that improves mo… ▽ More Serially connected robots are promising candidates for performing tasks in confined spaces such as search-and-rescue in large-scale disasters. Such robots are typically limbless, and we hypothesize that the addition of limbs could improve mobility. However, a challenge in designing and controlling such devices lies in the coordination of high-dimensional redundant modules in a way that improves mobility. Here we develop a general framework to control serially connected multi-legged robots. Specifically, we combine two approaches to build a general shape control scheme which can provide baseline patterns of self-deformation ("gaits") for effective locomotion in diverse robot morphologies. First, we take inspiration from a dimensionality reduction and a biological gait classification scheme to generate cyclic patterns of body deformation and foot lifting/lowering, which facilitate generation of arbitrary substrate contact patterns. Second, we use geometric mechanics methods to facilitates identification of optimal phasing of these undulations to maximize speed and/or stability. Our scheme allows the development of effective gaits in multi-legged robots locomoting on flat frictional terrain with diverse number of limbs (4, 6, 16, and even 0 limbs) and body actuation capabilities (including sidewinding gaits on limbless devices). By properly coordinating the body undulation and the leg placement, our framework combines the advantages of both limbless robots (modularity) and legged robots (mobility). We expect that our framework can provide general control schemes for the rapid deployment of general multi-legged robots, paving the ways toward machines that can traverse complex environments under real-life conditions. △ Less

Submitted 3 February, 2022; v1 submitted 1 December, 2021; originally announced December 2021.

HyperNeRF: A Higher-Dimensional Representation for Topologically Varying Neural Radiance Fields

Authors: Keunhong Park, Utkarsh Sinha, Peter Hedman, Jonathan T. Barron, Sofien Bouaziz, Dan B Goldman, Ricardo Martin-Brualla, Steven M. Seitz

Abstract: Neural Radiance Fields (NeRF) are able to reconstruct scenes with unprecedented fidelity, and various recent works have extended NeRF to handle dynamic scenes. A common approach to reconstruct such non-rigid scenes is through the use of a learned deformation field mapping from coordinates in each input image into a canonical template coordinate space. However, these deformation-based approaches st… ▽ More Neural Radiance Fields (NeRF) are able to reconstruct scenes with unprecedented fidelity, and various recent works have extended NeRF to handle dynamic scenes. A common approach to reconstruct such non-rigid scenes is through the use of a learned deformation field mapping from coordinates in each input image into a canonical template coordinate space. However, these deformation-based approaches struggle to model changes in topology, as topological changes require a discontinuity in the deformation field, but these deformation fields are necessarily continuous. We address this limitation by lifting NeRFs into a higher dimensional space, and by representing the 5D radiance field corresponding to each individual input image as a slice through this "hyper-space". Our method is inspired by level set methods, which model the evolution of surfaces as slices through a higher dimensional surface. We evaluate our method on two tasks: (i) interpolating smoothly between "moments", i.e., configurations of the scene, seen in the input images while maintaining visual plausibility, and (ii) novel-view synthesis at fixed moments. We show that our method, which we dub HyperNeRF, outperforms existing methods on both tasks. Compared to Nerfies, HyperNeRF reduces average error rates by 4.1% for interpolation and 8.6% for novel-view synthesis, as measured by LPIPS. Additional videos, results, and visualizations are available at https://hypernerf.github.io. △ Less

Submitted 10 September, 2021; v1 submitted 24 June, 2021; originally announced June 2021.

Comments: SIGGRAPH Asia 2021, Project page: https://hypernerf.github.io/

Neural RGB-D Surface Reconstruction

Authors: Dejan Azinović, Ricardo Martin-Brualla, Dan B Goldman, Matthias Nießner, Justus Thies

Abstract: Obtaining high-quality 3D reconstructions of room-scale scenes is of paramount importance for upcoming applications in AR or VR. These range from mixed reality applications for teleconferencing, virtual measuring, virtual room planing, to robotic applications. While current volume-based view synthesis methods that use neural radiance fields (NeRFs) show promising results in reproducing the appeara… ▽ More Obtaining high-quality 3D reconstructions of room-scale scenes is of paramount importance for upcoming applications in AR or VR. These range from mixed reality applications for teleconferencing, virtual measuring, virtual room planing, to robotic applications. While current volume-based view synthesis methods that use neural radiance fields (NeRFs) show promising results in reproducing the appearance of an object or scene, they do not reconstruct an actual surface. The volumetric representation of the surface based on densities leads to artifacts when a surface is extracted using Marching Cubes, since during optimization, densities are accumulated along the ray and are not used at a single sample point in isolation. Instead of this volumetric representation of the surface, we propose to represent the surface using an implicit function (truncated signed distance function). We show how to incorporate this representation in the NeRF framework, and extend it to use depth measurements from a commodity RGB-D sensor, such as a Kinect. In addition, we propose a pose and camera refinement technique which improves the overall reconstruction quality. In contrast to concurrent work on integrating depth priors in NeRF which concentrates on novel view synthesis, our approach is able to reconstruct high-quality, metrical 3D reconstructions. △ Less

Submitted 14 March, 2022; v1 submitted 9 April, 2021; originally announced April 2021.

Comments: CVPR'22; Project page: https://dazinovic.github.io/neural-rgbd-surface-reconstruction/ Video: https://youtu.be/iWuSowPsC3g

Low rattling: A predictive principle for self-organization in active collectives

Authors: Pavel Chvykov, Thomas A. Berrueta, Akash Vardhan, William Savoie, Alexander Samland, Todd D. Murphey, Kurt Wiesenfeld, Daniel I. Goldman, Jeremy L. England

Abstract: Self-organization is frequently observed in active collectives, from ant rafts to molecular motor assemblies. General principles describing self-organization away from equilibrium have been challenging to identify. We offer a unifying framework that models the behavior of complex systems as largely random, while capturing their configuration-dependent response to external forcing. This allows deri… ▽ More Self-organization is frequently observed in active collectives, from ant rafts to molecular motor assemblies. General principles describing self-organization away from equilibrium have been challenging to identify. We offer a unifying framework that models the behavior of complex systems as largely random, while capturing their configuration-dependent response to external forcing. This allows derivation of a Boltzmann-like principle for understanding and manipulating driven self-organization. We validate our predictions experimentally in shape-changing robotic active matter, and outline a methodology for controlling collective behavior. Our findings highlight how emergent order depends sensitively on the matching between external patterns of forcing and internal dynamical response properties, pointing towards future approaches for design and control of active particle mixtures and metamaterials. △ Less

Submitted 3 January, 2021; originally announced January 2021.

Journal ref: Science, Vol. 371, Issue 6524, pp. 90-95 (2021)

Reconstruction of Backbone Curves for Snake Robots

Authors: Tianyu Wang, Bo Lin, Baxi Chong, Julian Whitman, Matthew Travers, Daniel I. Goldman, Greg Blekherman, Howie Choset

Abstract: Snake robots composed of alternating single-axis pitch and yaw joints have many internal degrees of freedom, which make them capable of versatile three-dimensional locomotion. In motion planning process, snake robot motions are often designed kinematically by a chronological sequence of continuous backbone curves that capture desired macroscopic shapes of the robot. However, as the geometric arran… ▽ More Snake robots composed of alternating single-axis pitch and yaw joints have many internal degrees of freedom, which make them capable of versatile three-dimensional locomotion. In motion planning process, snake robot motions are often designed kinematically by a chronological sequence of continuous backbone curves that capture desired macroscopic shapes of the robot. However, as the geometric arrangement of single-axis rotary joints creates constraints on the rotations in the robot, it is challenging for the robot to reconstruct an arbitrary 3D curve. When the robot configuration does not accurately achieve the desired shapes defined by these backbone curves, the robot can have unexpected contacts with the environment, such that the robot does not achieve the desired motion. In this work, we propose a method for snake robots to reconstruct desired backbone curves by posing an optimization problem that exploits the robot's geometric structure. We verified that our method enables fast and accurate curve-configuration conversions through its applications to commonly used 3D gaits. We also demonstrated via robot experiments that 1) our method results in smooth locomotion on the robot; 2) our method allows the robot to approach the numerically predicted locomotive performance of a sequence of continuous backbone curve. △ Less

Submitted 17 February, 2021; v1 submitted 8 December, 2020; originally announced December 2020.

Comments: Accepted for publication in IEEE RA-L

Nerfies: Deformable Neural Radiance Fields

Authors: Keunhong Park, Utkarsh Sinha, Jonathan T. Barron, Sofien Bouaziz, Dan B Goldman, Steven M. Seitz, Ricardo Martin-Brualla

Abstract: We present the first method capable of photorealistically reconstructing deformable scenes using photos/videos captured casually from mobile phones. Our approach augments neural radiance fields (NeRF) by optimizing an additional continuous volumetric deformation field that warps each observed point into a canonical 5D NeRF. We observe that these NeRF-like deformation fields are prone to local mini… ▽ More We present the first method capable of photorealistically reconstructing deformable scenes using photos/videos captured casually from mobile phones. Our approach augments neural radiance fields (NeRF) by optimizing an additional continuous volumetric deformation field that warps each observed point into a canonical 5D NeRF. We observe that these NeRF-like deformation fields are prone to local minima, and propose a coarse-to-fine optimization method for coordinate-based models that allows for more robust optimization. By adapting principles from geometry processing and physical simulation to NeRF-like models, we propose an elastic regularization of the deformation field that further improves robustness. We show that our method can turn casually captured selfie photos/videos into deformable NeRF models that allow for photorealistic renderings of the subject from arbitrary viewpoints, which we dub "nerfies." We evaluate our method by collecting time-synchronized data using a rig with two mobile phones, yielding train/validation images of the same pose at different viewpoints. We show that our method faithfully reconstructs non-rigidly deforming scenes and reproduces unseen views with high fidelity. △ Less

Submitted 9 September, 2021; v1 submitted 25 November, 2020; originally announced November 2020.

Comments: ICCV 2021, Project page with videos: https://nerfies.github.io/

Programming Active Cohesive Granular Matter with Mechanically Induced Phase Changes

Authors: Shengkai Li, Bahnisikha Dutta, Sarah Cannon, Joshua J. Daymude, Ram Avinery, Enes Aydin, Andréa W. Richa, Daniel I. Goldman, Dana Randall

Abstract: Active matter physics and swarm robotics have provided powerful tools for the study and control of ensembles driven by internal sources. At the macroscale, controlling swarms typically utilizes significant memory, processing power, and coordination unavailable at the microscale, e.g., for colloidal robots, which could be useful for fighting disease, fabricating intelligent textiles, and designing… ▽ More Active matter physics and swarm robotics have provided powerful tools for the study and control of ensembles driven by internal sources. At the macroscale, controlling swarms typically utilizes significant memory, processing power, and coordination unavailable at the microscale, e.g., for colloidal robots, which could be useful for fighting disease, fabricating intelligent textiles, and designing nanocomputers. To develop principles that that can leverage physics of interactions and thus can be utilized across scales, we take a two-pronged approach: a theoretical abstraction of self-organizing particle systems and an experimental robot system of active cohesive granular matter that intentionally lacks digital electronic computation and communication, using minimal (or no) sensing and control, to test theoretical predictions. We consider the problems of aggregation, dispersion, and collective transport. As predicted by the theory, as a parameter representing interparticle attraction increases, the robots transition from a dispersed phase to an aggregated one, forming a dense, compact collective. When aggregated, the collective can transport non-robot "impurities" in their environment, thus performing an emergent task driven by the physics underlying the transition. These results point to a fruitful interplay between algorithm design and active matter robophysics that can result in new nonequilibrium physics and principles for programming collectives without the need for complex algorithms or capabilities. △ Less

Submitted 2 February, 2021; v1 submitted 11 September, 2020; originally announced September 2020.

Journal ref: Science Advances 7(17), eabe8494, 2021

GeLaTO: Generative Latent Textured Objects

Authors: Ricardo Martin-Brualla, Rohit Pandey, Sofien Bouaziz, Matthew Brown, Dan B Goldman

Abstract: Accurate modeling of 3D objects exhibiting transparency, reflections and thin structures is an extremely challenging problem. Inspired by billboards and geometric proxies used in computer graphics, this paper proposes Generative Latent Textured Objects (GeLaTO), a compact representation that combines a set of coarse shape proxies defining low frequency geometry with learned neural textures, to enc… ▽ More Accurate modeling of 3D objects exhibiting transparency, reflections and thin structures is an extremely challenging problem. Inspired by billboards and geometric proxies used in computer graphics, this paper proposes Generative Latent Textured Objects (GeLaTO), a compact representation that combines a set of coarse shape proxies defining low frequency geometry with learned neural textures, to encode both medium and fine scale geometry as well as view-dependent appearance. To generate the proxies' textures, we learn a joint latent space allowing category-level appearance and geometry interpolation. The proxies are independently rasterized with their corresponding neural texture and composited using a U-Net, which generates an output photorealistic image including an alpha map. We demonstrate the effectiveness of our approach by reconstructing complex objects from a sparse set of views. We show results on a dataset of real images of eyeglasses frames, which are particularly challenging to reconstruct using classical methods. We also demonstrate that these coarse proxies can be handcrafted when the underlying object geometry is easy to model, like eyeglasses, or generated using a neural network for more complex categories, such as cars. △ Less

Submitted 11 August, 2020; originally announced August 2020.

Comments: ECCV 2020 Spotlight. Project website: https://gelato-paper.github.io

Journal ref: European Conference on Computer Vision 2020

State of the Art on Neural Rendering

Authors: Ayush Tewari, Ohad Fried, Justus Thies, Vincent Sitzmann, Stephen Lombardi, Kalyan Sunkavalli, Ricardo Martin-Brualla, Tomas Simon, Jason Saragih, Matthias Nießner, Rohit Pandey, Sean Fanello, Gordon Wetzstein, Jun-Yan Zhu, Christian Theobalt, Maneesh Agrawala, Eli Shechtman, Dan B Goldman, Michael Zollhöfer

Abstract: Efficient rendering of photo-realistic virtual worlds is a long standing effort of computer graphics. Modern graphics techniques have succeeded in synthesizing photo-realistic images from hand-crafted scene representations. However, the automatic generation of shape, materials, lighting, and other aspects of scenes remains a challenging problem that, if solved, would make photo-realistic computer… ▽ More Efficient rendering of photo-realistic virtual worlds is a long standing effort of computer graphics. Modern graphics techniques have succeeded in synthesizing photo-realistic images from hand-crafted scene representations. However, the automatic generation of shape, materials, lighting, and other aspects of scenes remains a challenging problem that, if solved, would make photo-realistic computer graphics more widely accessible. Concurrently, progress in computer vision and machine learning have given rise to a new approach to image synthesis and editing, namely deep generative models. Neural rendering is a new and rapidly emerging field that combines generative machine learning techniques with physical knowledge from computer graphics, e.g., by the integration of differentiable rendering into network training. With a plethora of applications in computer graphics and vision, neural rendering is poised to become a new area in the graphics community, yet no survey of this emerging field exists. This state-of-the-art report summarizes the recent trends and applications of neural rendering. We focus on approaches that combine classic computer graphics techniques with deep generative models to obtain controllable and photo-realistic outputs. Starting with an overview of the underlying computer graphics and machine learning concepts, we discuss critical aspects of neural rendering approaches. This state-of-the-art report is focused on the many important use cases for the described algorithms such as novel view synthesis, semantic photo manipulation, facial and body reenactment, relighting, free-viewpoint video, and the creation of photo-realistic avatars for virtual and augmented reality telepresence. Finally, we conclude with a discussion of the social implications of such technology and investigate open research problems. △ Less

Submitted 8 April, 2020; originally announced April 2020.

Comments: Eurographics 2020 survey paper

Field-mediated locomotor dynamics on highly deformable surfaces

Authors: Shengkai Li, Yasemin Ozkan Aydin, Charles Xiao, Gabriella Small, Hussain N. Gynai, Gongjie Li, Jennifer M. Rieser, Pablo Laguna, Daniel I. Goldman

Abstract: In many systems motion occurs on deformed and deformable surfaces, setting up the possibility for dynamical interactions solely mediated by the coupling of the entities with their environment. Here we study the "two-body" dynamics of robot locomotion on a highly deformable spandex membrane in two scenarios: one in which a robot orbits a large central depression and the other where the two robots a… ▽ More In many systems motion occurs on deformed and deformable surfaces, setting up the possibility for dynamical interactions solely mediated by the coupling of the entities with their environment. Here we study the "two-body" dynamics of robot locomotion on a highly deformable spandex membrane in two scenarios: one in which a robot orbits a large central depression and the other where the two robots affect each other's motion solely through mutual environmental deformations. Inspired by the resemblance of the orbits of the single robot with those of general relativistic orbits around black holes, we recast the vehicle plus membrane dynamics in physical space into the geodesic motion of a "test particle" in a fiducial curved space-time and demonstrate how this framework facilitates understanding the observed dynamics. The two-robot problem also exhibits a resemblance with Einstein's general relativistic view of gravity, which in the words of Wheeler: "spacetime tells matter how to move; matter tells spacetime how to curve." We generalize this case the mapping to include a reciprocal coupling that translates into robotic curvature-based control schemes which modify interaction (promoting avoidance or aggregation) without long-range sensing. Our work provides a starting point for developing a mechanical analog gravity system as well as develops a framework that can provide insights into active matter in deformable environments and robot exploration in complex landscapes. △ Less

Submitted 3 August, 2021; v1 submitted 7 April, 2020; originally announced April 2020.

Journal ref: Proceedings of the National Academy of Sciences 119 (30), e2113912119, 2022

Text-based Editing of Talking-head Video

Authors: Ohad Fried, Ayush Tewari, Michael Zollhöfer, Adam Finkelstein, Eli Shechtman, Dan B Goldman, Kyle Genova, Zeyu Jin, Christian Theobalt, Maneesh Agrawala

Abstract: Editing talking-head video to change the speech content or to remove filler words is challenging. We propose a novel method to edit talking-head video based on its transcript to produce a realistic output video in which the dialogue of the speaker has been modified, while maintaining a seamless audio-visual flow (i.e. no jump cuts). Our method automatically annotates an input talking-head video wi… ▽ More Editing talking-head video to change the speech content or to remove filler words is challenging. We propose a novel method to edit talking-head video based on its transcript to produce a realistic output video in which the dialogue of the speaker has been modified, while maintaining a seamless audio-visual flow (i.e. no jump cuts). Our method automatically annotates an input talking-head video with phonemes, visemes, 3D face pose and geometry, reflectance, expression and scene illumination per frame. To edit a video, the user has to only edit the transcript, and an optimization strategy then chooses segments of the input corpus as base material. The annotated parameters corresponding to the selected segments are seamlessly stitched together and used to produce an intermediate video representation in which the lower half of the face is rendered with a parametric face model. Finally, a recurrent video generation network transforms this representation to a photorealistic video that matches the edited transcript. We demonstrate a large variety of edits, such as the addition, removal, and alteration of words, as well as convincing language translation and full sentence synthesis. △ Less

Submitted 4 June, 2019; originally announced June 2019.

Comments: A version with higher resolution images can be downloaded from the authors' website

Neural Rerendering in the Wild

Authors: Moustafa Meshry, Dan B Goldman, Sameh Khamis, Hugues Hoppe, Rohit Pandey, Noah Snavely, Ricardo Martin-Brualla

Abstract: We explore total scene capture -- recording, modeling, and rerendering a scene under varying appearance such as season and time of day. Starting from internet photos of a tourist landmark, we apply traditional 3D reconstruction to register the photos and approximate the scene as a point cloud. For each photo, we render the scene points into a deep framebuffer, and train a neural network to learn t… ▽ More We explore total scene capture -- recording, modeling, and rerendering a scene under varying appearance such as season and time of day. Starting from internet photos of a tourist landmark, we apply traditional 3D reconstruction to register the photos and approximate the scene as a point cloud. For each photo, we render the scene points into a deep framebuffer, and train a neural network to learn the mapping of these initial renderings to the actual photos. This rerendering network also takes as input a latent appearance vector and a semantic mask indicating the location of transient objects like pedestrians. The model is evaluated on several datasets of publicly available images spanning a broad range of illumination conditions. We create short videos demonstrating realistic manipulation of the image viewpoint, appearance, and semantic labeling. We also compare results with prior work on scene reconstruction from internet photos. △ Less

Submitted 8 April, 2019; originally announced April 2019.

Comments: To be presented at CVPR 2019 (oral). Supplementary video available at http://youtu.be/E1crWQn_kmY

LookinGood: Enhancing Performance Capture with Real-time Neural Re-Rendering

Authors: Ricardo Martin-Brualla, Rohit Pandey, Shuoran Yang, Pavel Pidlypenskyi, Jonathan Taylor, Julien Valentin, Sameh Khamis, Philip Davidson, Anastasia Tkach, Peter Lincoln, Adarsh Kowdle, Christoph Rhemann, Dan B Goldman, Cem Keskin, Steve Seitz, Shahram Izadi, Sean Fanello

Abstract: Motivated by augmented and virtual reality applications such as telepresence, there has been a recent focus in real-time performance capture of humans under motion. However, given the real-time constraint, these systems often suffer from artifacts in geometry and texture such as holes and noise in the final rendering, poor lighting, and low-resolution textures. We take the novel approach to augmen… ▽ More Motivated by augmented and virtual reality applications such as telepresence, there has been a recent focus in real-time performance capture of humans under motion. However, given the real-time constraint, these systems often suffer from artifacts in geometry and texture such as holes and noise in the final rendering, poor lighting, and low-resolution textures. We take the novel approach to augment such real-time performance capture systems with a deep architecture that takes a rendering from an arbitrary viewpoint, and jointly performs completion, super resolution, and denoising of the imagery in real-time. We call this approach neural (re-)rendering, and our live system "LookinGood". Our deep architecture is trained to produce high resolution and high quality images from a coarse rendering in real-time. First, we propose a self-supervised training method that does not require manual ground-truth annotation. We contribute a specialized reconstruction error that uses semantic information to focus on relevant parts of the subject, e.g. the face. We also introduce a salient reweighing scheme of the loss function that is able to discard outliers. We specifically design the system for virtual and augmented reality headsets where the consistency between the left and right eye plays a crucial role in the final user experience. Finally, we generate temporally stable results by explicitly minimizing the difference between two consecutive frames. We tested the proposed system in two different scenarios: one involving a single RGB-D sensor, and upper body reconstruction of an actor, the second consisting of full body 360 degree capture. Through extensive experimentation, we demonstrate how our system generalizes across unseen sequences and subjects. The supplementary video is available at http://youtu.be/Md3tdAKoLGU. △ Less

Submitted 12 November, 2018; originally announced November 2018.

Comments: The supplementary video is available at: http://youtu.be/Md3tdAKoLGU To be presented at SIGGRAPH Asia 2018

Locomoting robots composed of immobile robots

Authors: Ross Warkentin, William Savoie, Daniel I. Goldman

Abstract: Robotic materials are multi-robot systems formulated to leverage the low-order computation and actuation of the constituents to manipulate the high-order behavior of the entire material. We study the behaviors of ensembles composed of smart active particles, smarticles. Smarticles are small, low cost robots equipped with basic actuation and sensing abilities that are individually incapable of rota… ▽ More Robotic materials are multi-robot systems formulated to leverage the low-order computation and actuation of the constituents to manipulate the high-order behavior of the entire material. We study the behaviors of ensembles composed of smart active particles, smarticles. Smarticles are small, low cost robots equipped with basic actuation and sensing abilities that are individually incapable of rotating or displacing. We demonstrate that a "supersmarticle", composed of many smarticles constrained within a bounding membrane, can harness the internal collisions of the robotic material among the constituents and the membrane to achieve diffusive locomotion. The emergent diffusion can be directed by modulating the robotic material properties in response to a light source, analogous to biological phototaxis. The light source introduces asymmetries within the robotic material, resulting in modified populations of interaction modes and dynamics which ultimately result in supersmarticle biased locomotion. We present experimental methods and results for the robotic material which moves with a directed displacement in response to a light source. △ Less

Submitted 14 February, 2018; originally announced February 2018.

Comments: 4 pages 4 figures IRC 2018 conference paper

The dynamics of scattering in undulatory active collisions

Authors: Jennifer M. Rieser, Perrin E. Schiebel, Arman Pazouki, Feifei Qian, Zachary Goddard, Andrew Zangwill, Dan Negrut, Daniel I. Goldman

Abstract: Natural and artificial self-propelled systems must manage environmental interactions during movement. Such interactions, which we refer to as active collisions, are fundamentally different from momentum-conserving interactions studied in classical physics, largely because the internal driving of the locomotor can lead to persistent contact with heterogeneities. Here, we experimentally and numerica… ▽ More Natural and artificial self-propelled systems must manage environmental interactions during movement. Such interactions, which we refer to as active collisions, are fundamentally different from momentum-conserving interactions studied in classical physics, largely because the internal driving of the locomotor can lead to persistent contact with heterogeneities. Here, we experimentally and numerically study the effects of active collisions on a laterally-undulating sensory-deprived robophysical model, whose dynamics are applicable to self-propelled systems across length scales and environments. The robot moves via spatial undulation of body segments, with a nearly-linear center-of-geometry trajectory. Interactions with a single rigid post scatter the robot, and these deflections are proportional to the head-post contact duration. The distribution of scattering angles is smooth and strongly-peaked directly behind the post. Interactions with a single row of evenly-spaced posts (with inter-post spacing $d$) produce distributions reminiscent of far-field diffraction patterns: as $d$ decreases, distinct secondary peaks emerge as large deflections become more likely. Surprisingly, we find that the presence of multiple posts does not change the nature of individual collisions; instead, multi-modal scattering patterns arise from multiple posts altering the likelihood of individual collisions to occur. As $d$ decreases, collisions near the leading edges of the posts become more probable, and we find that these interactions are associated with larger deflections. Our results, which highlight the surprising dynamics that can occur during active collisions of self-propelled systems, can inform control principles for locomotors in complex terrain and facilitate design of task-capable active matter. △ Less

Submitted 11 June, 2018; v1 submitted 30 November, 2017; originally announced December 2017.

Comments: 32 pages, 20 main figures, 8 supplemental figures

Journal ref: Phys. Rev. E 99, 022606 (2019)

Phototactic Supersmarticles

Authors: Sarah Cannon, Joshua J. Daymude, William Savoie, Ross Warkentin, Shengkai Li, Daniel I. Goldman, Dana Randall, Andrea W. Richa

Abstract: Smarticles, or smart active particles, are small robots equipped with only basic movement and sensing abilities that are incapable of rotating or displacing individually. We study the ensemble behavior of smarticles, i.e., the behavior a collective of these very simple computational elements can achieve, and how such behavior can be implemented using minimal programming. We show that an ensemble o… ▽ More Smarticles, or smart active particles, are small robots equipped with only basic movement and sensing abilities that are incapable of rotating or displacing individually. We study the ensemble behavior of smarticles, i.e., the behavior a collective of these very simple computational elements can achieve, and how such behavior can be implemented using minimal programming. We show that an ensemble of smarticles constrained to remain close to one another (which we call a supersmarticle), achieves directed locomotion toward or away from a light source, a phenomenon known as phototaxing. We present experimental and theoretical models of phototactic supersmarticles that collectively move with a directed displacement in response to light. The motion of the supersmarticle is approximately Brownian, and is a result of chaotic interactions among smarticles. The system can be directed by introducing asymmetries among the individual smarticle's behavior, in our case by varying activity levels in response to light, resulting in supersmarticle biased motion. △ Less

Submitted 3 November, 2017; originally announced November 2017.

Journal ref: Artificial Life and Robotics 23:4 (2018) 459-468

Semantic Facial Expression Editing using Autoencoded Flow

Authors: Raymond Yeh, Ziwei Liu, Dan B Goldman, Aseem Agarwala

Abstract: High-level manipulation of facial expressions in images --- such as changing a smile to a neutral expression --- is challenging because facial expression changes are highly non-linear, and vary depending on the appearance of the face. We present a fully automatic approach to editing faces that combines the advantages of flow-based face manipulation with the more recent generative capabilities of V… ▽ More High-level manipulation of facial expressions in images --- such as changing a smile to a neutral expression --- is challenging because facial expression changes are highly non-linear, and vary depending on the appearance of the face. We present a fully automatic approach to editing faces that combines the advantages of flow-based face manipulation with the more recent generative capabilities of Variational Autoencoders (VAEs). During training, our model learns to encode the flow from one expression to another over a low-dimensional latent space. At test time, expression editing can be done simply using latent vector arithmetic. We evaluate our methods on two applications: 1) single-image facial expression editing, and 2) facial expression interpolation between two images. We demonstrate that our method generates images of higher perceptual quality than previous VAE and flow-based methods. △ Less

Submitted 29 November, 2016; originally announced November 2016.

Towards a Drone Cinematographer: Guiding Quadrotor Cameras using Visual Composition Principles

Authors: Niels Joubert, Jane L. E, Dan B Goldman, Floraine Berthouzoz, Mike Roberts, James A. Landay, Pat Hanrahan

Abstract: We present a system to capture video footage of human subjects in the real world. Our system leverages a quadrotor camera to automatically capture well-composed video of two subjects. Subjects are tracked in a large-scale outdoor environment using RTK GPS and IMU sensors. Then, given the tracked state of our subjects, our system automatically computes static shots based on well-established visual… ▽ More We present a system to capture video footage of human subjects in the real world. Our system leverages a quadrotor camera to automatically capture well-composed video of two subjects. Subjects are tracked in a large-scale outdoor environment using RTK GPS and IMU sensors. Then, given the tracked state of our subjects, our system automatically computes static shots based on well-established visual composition principles and canonical shots from cinematography literature. To transition between these static shots, we calculate feasible, safe, and visually pleasing transitions using a novel real-time trajectory planning algorithm. We evaluate the performance of our tracking system, and experimentally show that RTK GPS significantly outperforms conventional GPS in capturing a variety of canonical shots. Lastly, we demonstrate our system guiding a consumer quadrotor camera autonomously capturing footage of two subjects in a variety of use cases. This is the first end-to-end system that enables people to leverage the mobility of quadrotors, as well as the knowledge of expert filmmakers, to autonomously capture high-quality footage of people in the real world. △ Less

Submitted 5 October, 2016; originally announced October 2016.

A review on locomotion robophysics: the study of movement at the intersection of robotics, soft matter and dynamical systems

Authors: Jeffrey Aguilar, Tingnan Zhang, Feifei Qian, Mark Kingsbury, Benjamin McInroe, Nicole Mazouchova, Chen Li, Ryan Maladen, Chaohui Gong, Matt Travers, Ross L. Hatton, Howie Choset, Paul B. Umbanhowar, Daniel I. Goldman

Abstract: In this review we argue for the creation of a physics of moving systems -- a locomotion "robophysics" -- which we define as the pursuit of the discovery of principles of self generated motion. Robophysics can provide an important intellectual complement to the discipline of robotics, largely the domain of researchers from engineering and computer science. The essential idea is that we must complem… ▽ More In this review we argue for the creation of a physics of moving systems -- a locomotion "robophysics" -- which we define as the pursuit of the discovery of principles of self generated motion. Robophysics can provide an important intellectual complement to the discipline of robotics, largely the domain of researchers from engineering and computer science. The essential idea is that we must complement study of complex robots in complex situations with systematic study of simplified robophysical devices in controlled laboratory settings and simplified theoretical models. We must thus use the methods of physics to examine successful and failed locomotion in simplified (abstracted) devices using parameter space exploration, systematic control, and techniques from dynamical systems. Using examples from our and other's research, we will discuss how such robophysical studies have begun to aid engineers in the creation of devices that begin to achieve life-like locomotor abilities on and within complex environments, have inspired interesting physics questions in low dimensional dynamical systems, geometric mechanics and soft matter physics, and have been useful to develop models for biological locomotion in complex terrain. The rapidly decreasing cost of constructing sophisticated robot models with easy access to significant computational power bodes well for scientists and engineers to engage in a discipline which can readily integrate experiment, theory and computation. △ Less

Submitted 12 February, 2016; originally announced February 2016.

Comments: 61 pages, 18 figures, IOPScience journal: Reports on Progress in Physics

Interactive 3D Face Stylization Using Sculptural Abstraction

Authors: Jan Jachnik, Dan B Goldman, Linjie Luo, Andrew J. Davison

Abstract: Sculptors often deviate from geometric accuracy in order to enhance the appearance of their sculpture. These subtle stylizations may emphasize anatomy, draw the viewer's focus to characteristic features of the subject, or symbolize textures that might not be accurately reproduced in a particular sculptural medium, while still retaining fidelity to the unique proportions of an individual. In this w… ▽ More Sculptors often deviate from geometric accuracy in order to enhance the appearance of their sculpture. These subtle stylizations may emphasize anatomy, draw the viewer's focus to characteristic features of the subject, or symbolize textures that might not be accurately reproduced in a particular sculptural medium, while still retaining fidelity to the unique proportions of an individual. In this work we demonstrate an interactive system for enhancing face geometry using a class of stylizations based on visual decomposition into abstract semantic regions, which we call sculptural abstraction. We propose an interactive two-scale optimization framework for stylization based on sculptural abstraction, allowing real-time adjustment of both global and local parameters. We demonstrate this system's effectiveness in enhancing physical 3D prints of scans from various sources. △ Less

Submitted 6 February, 2015; originally announced February 2015.

Comments: 9 pages, 16 figures. For associated video and to download some of the results, see the project webpage at http://wp.doc.ic.ac.uk/robotvision/project/face-stylization/

Lift-off dynamics in a simple jumping robot

Authors: Jeffrey Aguilar, Alex Lesov, Kurt Wiesenfeld, Daniel I. Goldman

Abstract: We study vertical jumping in a simple robot comprising an actuated mass-spring arrangement. The actuator frequency and phase are systematically varied to find optimal performance. Optimal jumps occur above and below (but not at) the robot's resonant frequency $f_0$. Two distinct jumping modes emerge: a simple jump which is optimal above $f_0$ is achievable with a squat maneuver, and a peculiar stu… ▽ More We study vertical jumping in a simple robot comprising an actuated mass-spring arrangement. The actuator frequency and phase are systematically varied to find optimal performance. Optimal jumps occur above and below (but not at) the robot's resonant frequency $f_0$. Two distinct jumping modes emerge: a simple jump which is optimal above $f_0$ is achievable with a squat maneuver, and a peculiar stutter jump which is optimal below $f_0$ is generated with a counter-movement. A simple dynamical model reveals how optimal lift-off results from non-resonant transient dynamics. △ Less

Submitted 29 August, 2012; originally announced August 2012.

Comments: 4 pages, 4 figures, Physical Review Letters, in press (2012)

Crowdsourcing Gaze Data Collection

Authors: Dmitry Rudoy, Dan B. Goldman, Eli Shechtman, Lihi Zelnik-Manor

Abstract: Knowing where people look is a useful tool in many various image and video applications. However, traditional gaze tracking hardware is expensive and requires local study participants, so acquiring gaze location data from a large number of participants is very problematic. In this work we propose a crowdsourced method for acquisition of gaze direction data from a virtually unlimited number of part… ▽ More Knowing where people look is a useful tool in many various image and video applications. However, traditional gaze tracking hardware is expensive and requires local study participants, so acquiring gaze location data from a large number of participants is very problematic. In this work we propose a crowdsourced method for acquisition of gaze direction data from a virtually unlimited number of participants, using a robust self-reporting mechanism (see Figure 1). Our system collects temporally sparse but spatially dense points-of-attention in any visual information. We apply our approach to an existing video data set and demonstrate that we obtain results similar to traditional gaze tracking. We also explore the parameter ranges of our method, and collect gaze tracking data for a large set of YouTube videos. △ Less

Submitted 16 April, 2012; originally announced April 2012.

Comments: Presented at Collective Intelligence conference, 2012 (arXiv:1204.2991)

Report number: CollectiveIntelligence/2012/106