Showing 1–50 of 51 results for author: Allen, R

"Salt is the Soul of Hakka Baked Chicken": Reimagining Traditional Chinese Culinary ICH for Modern Contexts Without Losing Tradition

Authors: Sijia Liu, XiaoKe Zeng, Fengyihan Wu, Shu Ye, Bowen Liu, Sidney Cheung, Richard William Allen, Ray Lc

Abstract: Intangible Cultural Heritage (ICH) like traditional culinary practices face increasing pressure to adapt to globalization while maintaining their cultural authenticity. Centuries-old traditions in Chinese cuisine are subject to rapid changes for adaptation to contemporary tastes and dietary preferences. The preservation of these cultural practices requires approaches that can enable ICH practition… ▽ More Intangible Cultural Heritage (ICH) like traditional culinary practices face increasing pressure to adapt to globalization while maintaining their cultural authenticity. Centuries-old traditions in Chinese cuisine are subject to rapid changes for adaptation to contemporary tastes and dietary preferences. The preservation of these cultural practices requires approaches that can enable ICH practitioners to reimagine and recreate ICH for modern contexts. To address this, we created workshops where experienced practitioners of traditional Chinese cuisine co-created recipes using GenAI tools and realized the dishes. We found that GenAI inspired ICH practitioners to innovate recipes based on traditional workflows for broader audiences and adapt to modern dining contexts. However, GenAI-inspired co-creation posed challenges in maintaining the accuracy of original ICH workflows and preserving traditional flavors in the culinary outcomes. This study offers implications for designing human-AI collaborative processes for safeguarding and enhancing culinary ICH. △ Less

Submitted 5 May, 2025; originally announced May 2025.

Ember: A Compiler for Efficient Embedding Operations on Decoupled Access-Execute Architectures

Authors: Marco Siracusa, Olivia Hsu, Victor Soria-Pardos, Joshua Randall, Arnaud Grasset, Eric Biscondi, Doug Joseph, Randy Allen, Fredrik Kjolstad, Miquel Moretó Planas, Adrià Armejach

Abstract: Irregular embedding lookups are a critical bottleneck in recommender models, sparse large language models, and graph learning models. In this paper, we first demonstrate that, by offloading these lookups to specialized access units, Decoupled Access-Execute (DAE) processors achieve 2.6$\times$ higher performance and 6.4$\times$ higher performance/watt than GPUs on end-to-end models. Then, we propo… ▽ More Irregular embedding lookups are a critical bottleneck in recommender models, sparse large language models, and graph learning models. In this paper, we first demonstrate that, by offloading these lookups to specialized access units, Decoupled Access-Execute (DAE) processors achieve 2.6$\times$ higher performance and 6.4$\times$ higher performance/watt than GPUs on end-to-end models. Then, we propose the Ember compiler for automatically generating optimized DAE code from PyTorch and TensorFlow. Conversely from other DAE compilers, Ember features multiple intermediate representations specifically designed for different optimization levels. In this way, Ember can implement all optimizations to match the performance of hand-written code, unlocking the full potential of DAE architectures at scale. △ Less

Submitted 14 April, 2025; originally announced April 2025.

Comments: 14 pages, 19 figures, under review

ACM Class: C.1.2; C.1.3; D.3.4

"Becoming My Own Audience": How Dancers React to Avatars Unlike Themselves in Motion Capture-Supported Live Improvisational Performance

Authors: Fan Zhang, Molin Li, Xiaoyu Chang, Kexue Fu, Richard William Allen, RAY LC

Abstract: The use of motion capture in live dance performances has created an emerging discipline enabling dancers to play different avatars on the digital stage. Unlike classical workflows, avatars enable performers to act as different characters in customized narratives, but research has yet to address how movement, improvisation, and perception change when dancers act as avatars. We created five avatars… ▽ More The use of motion capture in live dance performances has created an emerging discipline enabling dancers to play different avatars on the digital stage. Unlike classical workflows, avatars enable performers to act as different characters in customized narratives, but research has yet to address how movement, improvisation, and perception change when dancers act as avatars. We created five avatars representing differing genders, shapes, and body limitations, and invited 15 dancers to improvise with each in practice and performance settings. Results show that dancers used avatars to distance themselves from their own habitual movements, exploring new ways of moving through differing physical constraints. Dancers explored using gender-stereotyped movements like powerful or feminine actions, experimenting with gender identity. However, focusing on avatars can coincide with a lack of continuity in improvisation. This work shows how emerging practices with performance technology enable dancers to improvise with new constraints, stepping outside the classical stage. △ Less

Submitted 24 March, 2025; originally announced March 2025.

Comments: 24 pages, 18 figures, accepted to CHI25

In Pursuit of Predictive Models of Human Preferences Toward AI Teammates

Authors: Ho Chit Siu, Jaime D. Peña, Yutai Zhou, Ross E. Allen

Abstract: We seek measurable properties of AI agents that make them better or worse teammates from the subjective perspective of human collaborators. Our experiments use the cooperative card game Hanabi -- a common benchmark for AI-teaming research. We first evaluate AI agents on a set of objective metrics based on task performance, information theory, and game theory, which are measurable without human int… ▽ More We seek measurable properties of AI agents that make them better or worse teammates from the subjective perspective of human collaborators. Our experiments use the cooperative card game Hanabi -- a common benchmark for AI-teaming research. We first evaluate AI agents on a set of objective metrics based on task performance, information theory, and game theory, which are measurable without human interaction. Next, we evaluate subjective human preferences toward AI teammates in a large-scale (N=241) human-AI teaming experiment. Finally, we correlate the AI-only objective metrics with the human subjective preferences. Our results refute common assumptions from prior literature on reinforcement learning, revealing new correlations between AI behaviors and human preferences. We find that the final game score a human-AI team achieves is less predictive of human preferences than esoteric measures of AI action diversity, strategic dominance, and ability to team with other AI. In the future, these correlations may help shape reward functions for training human-collaborative AI. △ Less

Submitted 31 January, 2025; originally announced March 2025.

More Information is Not Always Better: Connections between Zero-Sum Local Nash Equilibria in Feedback and Open-Loop Information Patterns

Authors: Kushagra Gupta, Ross Allen, David Fridovich-Keil, Ufuk Topcu

Abstract: Non-cooperative dynamic game theory provides a principled approach to modeling sequential decision-making among multiple noncommunicative agents. A key focus has been on finding Nash equilibria in two-agent zero-sum dynamic games under various information structures. A well-known result states that in linear-quadratic games, unique Nash equilibria under feedback and open-loop information structure… ▽ More Non-cooperative dynamic game theory provides a principled approach to modeling sequential decision-making among multiple noncommunicative agents. A key focus has been on finding Nash equilibria in two-agent zero-sum dynamic games under various information structures. A well-known result states that in linear-quadratic games, unique Nash equilibria under feedback and open-loop information structures yield identical trajectories. Motivated by two key perspectives -- (i) many real-world problems extend beyond linear-quadratic settings and lack unique equilibria, making only local Nash equilibria computable, and (ii) local open-loop Nash equilibria (OLNE) are easier to compute than local feedback Nash equilibria (FBNE) -- it is natural to ask whether a similar result holds for local equilibria in zero-sum games. To this end, we establish that for a broad class of zero-sum games with potentially nonconvex-nonconcave objectives and nonlinear dynamics: (i) the state/control trajectory of a local FBNE satisfies local OLNE first-order optimality conditions, and vice versa, (ii) a local FBNE trajectory satisfies local OLNE second-order necessary conditions, (iii) a local FBNE trajectory satisfying feedback sufficiency conditions also constitutes a local OLNE, and (iv) with additional hard constraints on agents' actuations, a local FBNE where strict complementarity holds also satisfies local OLNE first-order optimality conditions, and vice versa. △ Less

Submitted 19 March, 2025; originally announced March 2025.

Comments: 6 pages

Syndeo: Portable Ray Clusters with Secure Containerization

Authors: William Li, Rodney S. Lafuente Mercado, Jaime D. Pena, Ross E. Allen

Abstract: We present Syndeo: a software framework for container orchestration of Ray on Slurm. In general the idea behind Syndeo is to write code once and deploy anywhere. Specifically, Syndeo is designed to addresses the issues of portability, scalability, and security for parallel computing. The design is portable because the containerized Ray code can be re-deployed on Amazon Web Services, Microsoft Azur… ▽ More We present Syndeo: a software framework for container orchestration of Ray on Slurm. In general the idea behind Syndeo is to write code once and deploy anywhere. Specifically, Syndeo is designed to addresses the issues of portability, scalability, and security for parallel computing. The design is portable because the containerized Ray code can be re-deployed on Amazon Web Services, Microsoft Azure, Google Cloud, or Alibaba Cloud. The process is scalable because we optimize for multi-node, high-throughput computing. The process is secure because users are forced to operate with unprivileged profiles meaning administrators control the access permissions. We demonstrate Syndeo's portable, scalable, and secure design by deploying containerized parallel workflows on Slurm for which Ray does not officially support. △ Less

Submitted 25 September, 2024; originally announced September 2024.

Comments: IEEE HPEC 2024. 6 Pages

Neural Assets: 3D-Aware Multi-Object Scene Synthesis with Image Diffusion Models

Authors: Ziyi Wu, Yulia Rubanova, Rishabh Kabra, Drew A. Hudson, Igor Gilitschenski, Yusuf Aytar, Sjoerd van Steenkiste, Kelsey R. Allen, Thomas Kipf

Abstract: We address the problem of multi-object 3D pose control in image diffusion models. Instead of conditioning on a sequence of text tokens, we propose to use a set of per-object representations, Neural Assets, to control the 3D pose of individual objects in a scene. Neural Assets are obtained by pooling visual representations of objects from a reference image, such as a frame in a video, and are train… ▽ More We address the problem of multi-object 3D pose control in image diffusion models. Instead of conditioning on a sequence of text tokens, we propose to use a set of per-object representations, Neural Assets, to control the 3D pose of individual objects in a scene. Neural Assets are obtained by pooling visual representations of objects from a reference image, such as a frame in a video, and are trained to reconstruct the respective objects in a different image, e.g., a later frame in the video. Importantly, we encode object visuals from the reference image while conditioning on object poses from the target frame. This enables learning disentangled appearance and pose features. Combining visual and 3D pose representations in a sequence-of-tokens format allows us to keep the text-to-image architecture of existing models, with Neural Assets in place of text tokens. By fine-tuning a pre-trained text-to-image diffusion model with this information, our approach enables fine-grained 3D pose and placement control of individual objects in a scene. We further demonstrate that Neural Assets can be transferred and recomposed across different scenes. Our model achieves state-of-the-art multi-object editing results on both synthetic 3D scene datasets, as well as two real-world video datasets (Objectron, Waymo Open). △ Less

Submitted 28 October, 2024; v1 submitted 13 June, 2024; originally announced June 2024.

Comments: Additional details and video results are available at https://neural-assets-paper.github.io/

Second-Order Algorithms for Finding Local Nash Equilibria in Zero-Sum Games

Authors: Kushagra Gupta, Xinjie Liu, Ross Allen, Ufuk Topcu, David Fridovich-Keil

Abstract: Zero-sum games arise in a wide variety of problems, including robust optimization and adversarial learning. However, algorithms deployed for finding a local Nash equilibrium in these games often converge to non-Nash stationary points. This highlights a key challenge: for any algorithm, the stability properties of its underlying dynamical system can cause non-Nash points to be potential attractors.… ▽ More Zero-sum games arise in a wide variety of problems, including robust optimization and adversarial learning. However, algorithms deployed for finding a local Nash equilibrium in these games often converge to non-Nash stationary points. This highlights a key challenge: for any algorithm, the stability properties of its underlying dynamical system can cause non-Nash points to be potential attractors. To overcome this challenge, algorithms must account for subtleties involving the curvatures of players' costs. To this end, we leverage dynamical system theory and develop a second-order algorithm for finding a local Nash equilibrium in the smooth, possibly nonconvex-nonconcave, zero-sum game setting. First, we prove that this novel method guarantees convergence to only local Nash equilibria with a local linear convergence rate. We then interpret a version of this method as a modified Gauss-Newton algorithm with local superlinear convergence to the neighborhood of a point that satisfies first-order local Nash equilibrium conditions. In comparison, current related state-of-the-art methods do not offer convergence rate guarantees. Furthermore, we show that this approach naturally generalizes to settings with convex and potentially coupled constraints while retaining earlier guarantees of convergence to only local (generalized) Nash equilibria. △ Less

Submitted 3 October, 2024; v1 submitted 5 June, 2024; originally announced June 2024.

Distributed Online Planning for Min-Max Problems in Networked Markov Games

Authors: Alexandros E. Tzikas, Jinkyoo Park, Mykel J. Kochenderfer, Ross E. Allen

Abstract: Min-max problems are important in multi-agent sequential decision-making because they improve the performance of the worst-performing agent in the network. However, solving the multi-agent min-max problem is challenging. We propose a modular, distributed, online planning-based algorithm that is able to approximate the solution of the min-max objective in networked Markov games, assuming that the a… ▽ More Min-max problems are important in multi-agent sequential decision-making because they improve the performance of the worst-performing agent in the network. However, solving the multi-agent min-max problem is challenging. We propose a modular, distributed, online planning-based algorithm that is able to approximate the solution of the min-max objective in networked Markov games, assuming that the agents communicate within a network topology and the transition and reward functions are neighborhood-dependent. This set-up is encountered in the multi-robot setting. Our method consists of two phases at every planning step. In the first phase, each agent obtains sample returns based on its local reward function, by performing online planning. Using the samples from online planning, each agent constructs a concave approximation of its underlying local return as a function of only the action of its neighborhood at the next planning step. In the second phase, the agents deploy a distributed optimization framework that converges to the optimal immediate next action for each agent, based on the function approximations of the first phase. We demonstrate our algorithm's performance through formation control simulations. △ Less

Submitted 29 May, 2024; originally announced May 2024.

Comments: Accepted to appear in the IEEE Robotics and Automation Letters

Gemini & Physical World: Large Language Models Can Estimate the Intensity of Earthquake Shaking from Multi-Modal Social Media Posts

Authors: S. Mostafa Mousavi, Marc Stogaitis, Tajinder Gadh, Richard M Allen, Alexei Barski, Robert Bosch, Patrick Robertson, Nivetha Thiruverahan, Youngmin Cho, Aman Raj

Abstract: This paper presents a novel approach to extract scientifically valuable information about Earth's physical phenomena from unconventional sources, such as multi-modal social media posts. Employing a state-of-the-art large language model (LLM), Gemini 1.5 Pro (Reid et al. 2024), we estimate earthquake ground shaking intensity from these unstructured posts. The model's output, in the form of Modified… ▽ More This paper presents a novel approach to extract scientifically valuable information about Earth's physical phenomena from unconventional sources, such as multi-modal social media posts. Employing a state-of-the-art large language model (LLM), Gemini 1.5 Pro (Reid et al. 2024), we estimate earthquake ground shaking intensity from these unstructured posts. The model's output, in the form of Modified Mercalli Intensity (MMI) values, aligns well with independent observational data. Furthermore, our results suggest that LLMs, trained on vast internet data, may have developed a unique understanding of physical phenomena. Specifically, Google's Gemini models demonstrate a simplified understanding of the general relationship between earthquake magnitude, distance, and MMI intensity, accurately describing observational data even though it's not identical to established models. These findings raise intriguing questions about the extent to which Gemini's training has led to a broader understanding of the physical world and its phenomena. The ability of Generative AI models like Gemini to generate results consistent with established scientific knowledge highlights their potential to augment our understanding of complex physical phenomena like earthquakes. The flexible and effective approach proposed in this study holds immense potential for enriching our understanding of the impact of physical phenomena and improving resilience during natural disasters. This research is a significant step toward harnessing the power of social media and AI for natural disaster mitigation, opening new avenues for understanding the emerging capabilities of Generative AI and LLMs for scientific applications. △ Less

Submitted 14 June, 2024; v1 submitted 28 May, 2024; originally announced May 2024.

Learning rigid-body simulators over implicit shapes for large-scale scenes and vision

Authors: Yulia Rubanova, Tatiana Lopez-Guevara, Kelsey R. Allen, William F. Whitney, Kimberly Stachenfeld, Tobias Pfaff

Abstract: Simulating large scenes with many rigid objects is crucial for a variety of applications, such as robotics, engineering, film and video games. Rigid interactions are notoriously hard to model: small changes to the initial state or the simulation parameters can lead to large changes in the final state. Recently, learned simulators based on graph networks (GNNs) were developed as an alternative to h… ▽ More Simulating large scenes with many rigid objects is crucial for a variety of applications, such as robotics, engineering, film and video games. Rigid interactions are notoriously hard to model: small changes to the initial state or the simulation parameters can lead to large changes in the final state. Recently, learned simulators based on graph networks (GNNs) were developed as an alternative to hand-designed simulators like MuJoCo and PyBullet. They are able to accurately capture dynamics of real objects directly from real-world observations. However, current state-of-the-art learned simulators operate on meshes and scale poorly to scenes with many objects or detailed shapes. Here we present SDF-Sim, the first learned rigid-body simulator designed for scale. We use learned signed-distance functions (SDFs) to represent the object shapes and to speed up distance computation. We design the simulator to leverage SDFs and avoid the fundamental bottleneck of the previous simulators associated with collision detection. For the first time in literature, we demonstrate that we can scale the GNN-based simulators to scenes with hundreds of objects and up to 1.1 million nodes, where mesh-based approaches run out of memory. Finally, we show that SDF-Sim can be applied to real world scenes by extracting SDFs from multi-view images. △ Less

Submitted 22 May, 2024; originally announced May 2024.

Scaling Face Interaction Graph Networks to Real World Scenes

Authors: Tatiana Lopez-Guevara, Yulia Rubanova, William F. Whitney, Tobias Pfaff, Kimberly Stachenfeld, Kelsey R. Allen

Abstract: Accurately simulating real world object dynamics is essential for various applications such as robotics, engineering, graphics, and design. To better capture complex real dynamics such as contact and friction, learned simulators based on graph networks have recently shown great promise. However, applying these learned simulators to real scenes comes with two major challenges: first, scaling learne… ▽ More Accurately simulating real world object dynamics is essential for various applications such as robotics, engineering, graphics, and design. To better capture complex real dynamics such as contact and friction, learned simulators based on graph networks have recently shown great promise. However, applying these learned simulators to real scenes comes with two major challenges: first, scaling learned simulators to handle the complexity of real world scenes which can involve hundreds of objects each with complicated 3D shapes, and second, handling inputs from perception rather than 3D state information. Here we introduce a method which substantially reduces the memory required to run graph-based learned simulators. Based on this memory-efficient simulation model, we then present a perceptual interface in the form of editable NeRFs which can convert real-world scenes into a structured representation that can be processed by graph network simulator. We show that our method uses substantially less memory than previous graph-based simulators while retaining their accuracy, and that the simulators learned in synthetic environments can be applied to real world scenes captured from multiple camera angles. This paves the way for expanding the application of learned simulators to settings where only perceptual information is available at inference time. △ Less

Submitted 22 January, 2024; originally announced January 2024.

Comments: 16 pages, 12 figures

Learning 3D Particle-based Simulators from RGB-D Videos

Authors: William F. Whitney, Tatiana Lopez-Guevara, Tobias Pfaff, Yulia Rubanova, Thomas Kipf, Kimberly Stachenfeld, Kelsey R. Allen

Abstract: Realistic simulation is critical for applications ranging from robotics to animation. Traditional analytic simulators sometimes struggle to capture sufficiently realistic simulation which can lead to problems including the well known "sim-to-real" gap in robotics. Learned simulators have emerged as an alternative for better capturing real-world physical dynamics, but require access to privileged g… ▽ More Realistic simulation is critical for applications ranging from robotics to animation. Traditional analytic simulators sometimes struggle to capture sufficiently realistic simulation which can lead to problems including the well known "sim-to-real" gap in robotics. Learned simulators have emerged as an alternative for better capturing real-world physical dynamics, but require access to privileged ground truth physics information such as precise object geometry or particle tracks. Here we propose a method for learning simulators directly from observations. Visual Particle Dynamics (VPD) jointly learns a latent particle-based representation of 3D scenes, a neural simulator of the latent particle dynamics, and a renderer that can produce images of the scene from arbitrary views. VPD learns end to end from posed RGB-D videos and does not require access to privileged information. Unlike existing 2D video prediction models, we show that VPD's 3D structure enables scene editing and long-term predictions. These results pave the way for downstream applications ranging from video editing to robotic planning. △ Less

Submitted 8 December, 2023; originally announced December 2023.

A Comparative Analysis of Text-to-Image Generative AI Models in Scientific Contexts: A Case Study on Nuclear Power

Authors: Veda Joynt, Jacob Cooper, Naman Bhargava, Katie Vu, O Hwang Kwon, Todd R. Allen, Aditi Verma, Majdi I. Radaideh

Abstract: In this work, we propose and assess the potential of generative artificial intelligence (AI) to generate public engagement around potential clean energy sources. Such an application could increase energy literacy -- an awareness of low-carbon energy sources among the public therefore leading to increased participation in decision-making about the future of energy systems. We explore the use of gen… ▽ More In this work, we propose and assess the potential of generative artificial intelligence (AI) to generate public engagement around potential clean energy sources. Such an application could increase energy literacy -- an awareness of low-carbon energy sources among the public therefore leading to increased participation in decision-making about the future of energy systems. We explore the use of generative AI to communicate technical information about low-carbon energy sources to the general public, specifically in the realm of nuclear energy. We explored 20 AI-powered text-to-image generators and compared their individual performances on general and scientific nuclear-related prompts. Of these models, DALL-E, DreamStudio, and Craiyon demonstrated promising performance in generating relevant images from general-level text related to nuclear topics. However, these models fall short in three crucial ways: (1) they fail to accurately represent technical details of energy systems; (2) they reproduce existing biases surrounding gender and work in the energy sector; and (3) they fail to accurately represent indigenous landscapes -- which have historically been sites of resource extraction and waste deposition for energy industries. This work is performed to motivate the development of specialized generative tools and their captions to improve energy literacy and effectively engage the public with low-carbon energy sources. △ Less

Submitted 2 December, 2023; originally announced December 2023.

Comments: 26 pages, 11 figures, 9 tables, submitted to review

Using Causal Threads to Explain Changes in a Dynamic System

Authors: Robert B. Allen

Abstract: We explore developing rich semantic models of systems. Specifically, we consider structured causal explanations about state changes in those systems. Essentially, we are developing process-based dynamic knowledge graphs. As an example, we construct a model of the causal threads for geological changes proposed by the Snowball Earth theory. Further, we describe an early prototype of a graphical inte… ▽ More We explore developing rich semantic models of systems. Specifically, we consider structured causal explanations about state changes in those systems. Essentially, we are developing process-based dynamic knowledge graphs. As an example, we construct a model of the causal threads for geological changes proposed by the Snowball Earth theory. Further, we describe an early prototype of a graphical interface to present the explanations. Unlike statistical approaches to summarization and explanation such as Large Language Models (LLMs), our approach of direct representation can be inspected and verified directly. △ Less

Submitted 19 November, 2023; originally announced November 2023.

Comments: 2023 ICADL

Safe Neural Control for Non-Affine Control Systems with Differentiable Control Barrier Functions

Authors: Wei Xiao, Ross Allen, Daniela Rus

Abstract: This paper addresses the problem of safety-critical control for non-affine control systems. It has been shown that optimizing quadratic costs subject to state and control constraints can be sub-optimally reduced to a sequence of quadratic programs (QPs) by using Control Barrier Functions (CBFs). Our recently proposed High Order CBFs (HOCBFs) can accommodate constraints of arbitrary relative degree… ▽ More This paper addresses the problem of safety-critical control for non-affine control systems. It has been shown that optimizing quadratic costs subject to state and control constraints can be sub-optimally reduced to a sequence of quadratic programs (QPs) by using Control Barrier Functions (CBFs). Our recently proposed High Order CBFs (HOCBFs) can accommodate constraints of arbitrary relative degree. The main challenges in this approach are that it requires affine control dynamics and the solution of the CBF-based QP is sub-optimal since it is solved point-wise. To address these challenges, we incorporate higher-order CBFs into neural ordinary differential equation-based learning models as differentiable CBFs to guarantee safety for non-affine control systems. The differentiable CBFs are trainable in terms of their parameters, and thus, they can address the conservativeness of CBFs such that the system state will not stay unnecessarily far away from safe set boundaries. Moreover, the imitation learning model is capable of learning complex and optimal control policies that are usually intractable online. We illustrate the effectiveness of the proposed framework on LiDAR-based autonomous driving and compare it with existing methods. △ Less

Submitted 6 September, 2023; originally announced September 2023.

Comments: 8 pages, accepted in IEEE CDC 2023

Learned Risk Metric Maps for Kinodynamic Systems

Authors: Ross Allen, Wei Xiao, Daniela Rus

Abstract: We present Learned Risk Metric Maps (LRMM) for real-time estimation of coherent risk metrics of high dimensional dynamical systems operating in unstructured, partially observed environments. LRMM models are simple to design and train -- requiring only procedural generation of obstacle sets, state and control sampling, and supervised training of a function approximator -- which makes them broadly a… ▽ More We present Learned Risk Metric Maps (LRMM) for real-time estimation of coherent risk metrics of high dimensional dynamical systems operating in unstructured, partially observed environments. LRMM models are simple to design and train -- requiring only procedural generation of obstacle sets, state and control sampling, and supervised training of a function approximator -- which makes them broadly applicable to arbitrary system dynamics and obstacle sets. In a parallel autonomy setting, we demonstrate the model's ability to rapidly infer collision probabilities of a fast-moving car-like robot driving recklessly in an obstructed environment; allowing the LRMM agent to intervene, take control of the vehicle, and avoid collisions. In this time-critical scenario, we show that LRMMs can evaluate risk metrics 20-100x times faster than alternative safety algorithms based on control barrier functions (CBFs) and Hamilton-Jacobi reachability (HJ-reach), leading to 5-15\% fewer obstacle collisions by the LRMM agent than CBFs and HJ-reach. This performance improvement comes in spite of the fact that the LRMM model only has access to local/partial observation of obstacles, whereas the CBF and HJ-reach agents are granted privileged/global information. We also show that our model can be equally well trained on a 12-dimensional quadrotor system operating in an obstructed indoor environment. The LRMM codebase is provided at https://github.com/mit-drl/pyrmm. △ Less

Submitted 28 February, 2023; originally announced February 2023.

Learning rigid dynamics with face interaction graph networks

Authors: Kelsey R. Allen, Yulia Rubanova, Tatiana Lopez-Guevara, William Whitney, Alvaro Sanchez-Gonzalez, Peter Battaglia, Tobias Pfaff

Abstract: Simulating rigid collisions among arbitrary shapes is notoriously difficult due to complex geometry and the strong non-linearity of the interactions. While graph neural network (GNN)-based models are effective at learning to simulate complex physical dynamics, such as fluids, cloth and articulated bodies, they have been less effective and efficient on rigid-body physics, except with very simple sh… ▽ More Simulating rigid collisions among arbitrary shapes is notoriously difficult due to complex geometry and the strong non-linearity of the interactions. While graph neural network (GNN)-based models are effective at learning to simulate complex physical dynamics, such as fluids, cloth and articulated bodies, they have been less effective and efficient on rigid-body physics, except with very simple shapes. Existing methods that model collisions through the meshes' nodes are often inaccurate because they struggle when collisions occur on faces far from nodes. Alternative approaches that represent the geometry densely with many particles are prohibitively expensive for complex shapes. Here we introduce the Face Interaction Graph Network (FIGNet) which extends beyond GNN-based methods, and computes interactions between mesh faces, rather than nodes. Compared to learned node- and particle-based methods, FIGNet is around 4x more accurate in simulating complex shape interactions, while also 8x more computationally efficient on sparse, rigid meshes. Moreover, FIGNet can learn frictional dynamics directly from real-world data, and can be more accurate than analytical solvers given modest amounts of training data. FIGNet represents a key step forward in one of the few remaining physical domains which have seen little competition from learned simulators, and offers allied fields such as robotics, graphics and mechanical design a new tool for simulation and model-based planning. △ Less

Submitted 7 December, 2022; originally announced December 2022.

Quantum and Classical Algorithms for Bounded Distance Decoding

Authors: Richard Allen, Ratip Emin Berker, Sílvia Casacuberta, Michael Gul

Abstract: In this paper, we provide a comprehensive overview of a recent debate over the quantum versus classical solvability of bounded distance decoding (BDD). Specifically, we review the work of Eldar and Hallgren [EH22], [Hal21] demonstrating a quantum algorithm solving $λ_1 2^{-Ω(\sqrt{k \log q})}$-BDD in polynomial time for lattices of periodicity $q$, finite group rank $k$, and shortest lattice vecto… ▽ More In this paper, we provide a comprehensive overview of a recent debate over the quantum versus classical solvability of bounded distance decoding (BDD). Specifically, we review the work of Eldar and Hallgren [EH22], [Hal21] demonstrating a quantum algorithm solving $λ_1 2^{-Ω(\sqrt{k \log q})}$-BDD in polynomial time for lattices of periodicity $q$, finite group rank $k$, and shortest lattice vector length $λ_1$. Subsequently, we prove the results of [DvW21a], [DvW21b] with far greater detail and elaboration than in the original work. Namely, we show that there exists a deterministic, classical algorithm achieving the same result. △ Less

Submitted 17 February, 2022; originally announced March 2022.

Physical Design using Differentiable Learned Simulators

Authors: Kelsey R. Allen, Tatiana Lopez-Guevara, Kimberly Stachenfeld, Alvaro Sanchez-Gonzalez, Peter Battaglia, Jessica Hamrick, Tobias Pfaff

Abstract: Designing physical artifacts that serve a purpose - such as tools and other functional structures - is central to engineering as well as everyday human behavior. Though automating design has tremendous promise, general-purpose methods do not yet exist. Here we explore a simple, fast, and robust approach to inverse design which combines learned forward simulators based on graph neural networks with… ▽ More Designing physical artifacts that serve a purpose - such as tools and other functional structures - is central to engineering as well as everyday human behavior. Though automating design has tremendous promise, general-purpose methods do not yet exist. Here we explore a simple, fast, and robust approach to inverse design which combines learned forward simulators based on graph neural networks with gradient-based design optimization. Our approach solves high-dimensional problems with complex physical dynamics, including designing surfaces and tools to manipulate fluid flows and optimizing the shape of an airfoil to minimize drag. This framework produces high-quality designs by propagating gradients through trajectories of hundreds of steps, even when using models that were pre-trained for single-step predictions on data substantially different from the design tasks. In our fluid manipulation tasks, the resulting designs outperformed those found by sampling-based optimization techniques. In airfoil design, they matched the quality of those obtained with a specialized solver. Our results suggest that despite some remaining challenges, machine learning-based simulators are maturing to the point where they can support general-purpose design optimization across a variety of domains. △ Less

Submitted 1 February, 2022; originally announced February 2022.

Comments: First three authors contributed equally

Any-Play: An Intrinsic Augmentation for Zero-Shot Coordination

Authors: Keane Lucas, Ross E. Allen

Abstract: Cooperative artificial intelligence with human or superhuman proficiency in collaborative tasks stands at the frontier of machine learning research. Prior work has tended to evaluate cooperative AI performance under the restrictive paradigms of self-play (teams composed of agents trained together) and cross-play (teams of agents trained independently but using the same algorithm). Recent work has… ▽ More Cooperative artificial intelligence with human or superhuman proficiency in collaborative tasks stands at the frontier of machine learning research. Prior work has tended to evaluate cooperative AI performance under the restrictive paradigms of self-play (teams composed of agents trained together) and cross-play (teams of agents trained independently but using the same algorithm). Recent work has indicated that AI optimized for these narrow settings may make for undesirable collaborators in the real-world. We formalize an alternative criteria for evaluating cooperative AI, referred to as inter-algorithm cross-play, where agents are evaluated on teaming performance with all other agents within an experiment pool with no assumption of algorithmic similarities between agents. We show that existing state-of-the-art cooperative AI algorithms, such as Other-Play and Off-Belief Learning, under-perform in this paradigm. We propose the Any-Play learning augmentation -- a multi-agent extension of diversity-based intrinsic rewards for zero-shot coordination (ZSC) -- for generalizing self-play-based algorithms to the inter-algorithm cross-play setting. We apply the Any-Play learning augmentation to the Simplified Action Decoder (SAD) and demonstrate state-of-the-art performance in the collaborative card game Hanabi. △ Less

Submitted 28 January, 2022; originally announced January 2022.

Comments: Accepted to AAMAS 2022. Code will be made available at https://github.com/mit-ll/hanabi_AnyPlay (may take several weeks after posting of this pre-print)

ACM Class: I.2.11

Detecting Damage Building Using Real-time Crowdsourced Images and Transfer Learning

Authors: Gaurav Chachra, Qingkai Kong, Jim Huang, Srujay Korlakunta, Jennifer Grannen, Alexander Robson, Richard Allen

Abstract: After significant earthquakes, we can see images posted on social media platforms by individuals and media agencies owing to the mass usage of smartphones these days. These images can be utilized to provide information about the shaking damage in the earthquake region both to the public and research community, and potentially to guide rescue work. This paper presents an automated way to extract th… ▽ More After significant earthquakes, we can see images posted on social media platforms by individuals and media agencies owing to the mass usage of smartphones these days. These images can be utilized to provide information about the shaking damage in the earthquake region both to the public and research community, and potentially to guide rescue work. This paper presents an automated way to extract the damaged building images after earthquakes from social media platforms such as Twitter and thus identify the particular user posts containing such images. Using transfer learning and ~6500 manually labelled images, we trained a deep learning model to recognize images with damaged buildings in the scene. The trained model achieved good performance when tested on newly acquired images of earthquakes at different locations and ran in near real-time on Twitter feed after the 2020 M7.0 earthquake in Turkey. Furthermore, to better understand how the model makes decisions, we also implemented the Grad-CAM method to visualize the important locations on the images that facilitate the decision. △ Less

Submitted 15 November, 2021; v1 submitted 12 October, 2021; originally announced October 2021.

Journal ref: Sci Rep 12, 8968 (2022)

Evaluation of Human-AI Teams for Learned and Rule-Based Agents in Hanabi

Authors: Ho Chit Siu, Jaime D. Pena, Edenna Chen, Yutai Zhou, Victor J. Lopez, Kyle Palko, Kimberlee C. Chang, Ross E. Allen

Abstract: Deep reinforcement learning has generated superhuman AI in competitive games such as Go and StarCraft. Can similar learning techniques create a superior AI teammate for human-machine collaborative games? Will humans prefer AI teammates that improve objective team performance or those that improve subjective metrics of trust? In this study, we perform a single-blind evaluation of teams of humans an… ▽ More Deep reinforcement learning has generated superhuman AI in competitive games such as Go and StarCraft. Can similar learning techniques create a superior AI teammate for human-machine collaborative games? Will humans prefer AI teammates that improve objective team performance or those that improve subjective metrics of trust? In this study, we perform a single-blind evaluation of teams of humans and AI agents in the cooperative card game Hanabi, with both rule-based and learning-based agents. In addition to the game score, used as an objective metric of the human-AI team performance, we also quantify subjective measures of the human's perceived performance, teamwork, interpretability, trust, and overall preference of AI teammate. We find that humans have a clear preference toward a rule-based AI teammate (SmartBot) over a state-of-the-art learning-based AI teammate (Other-Play) across nearly all subjective metrics, and generally view the learning-based agent negatively, despite no statistical difference in the game score. This result has implications for future AI design and reinforcement learning benchmarking, highlighting the need to incorporate subjective metrics of human-AI teaming rather than a singular focus on objective task performance. △ Less

Submitted 21 October, 2021; v1 submitted 15 July, 2021; originally announced July 2021.

Comments: Accepted for publication at NeurIPS 2021

Prediction of soft proton intensities in the near-Earth space using machine learning

Authors: Elena A. Kronberg, Tanveer Hannan, Jens Huthmacher, Marcus Münzer, Florian Peste, Ziyang Zhou, Max Berrendorf, Evgeniy Faerman, Fabio Gastaldello, Simona Ghizzardi, Philippe Escoubet, Stein Haaland, Artem Smirnov, Nithin Sivadas, Robert C. Allen, Andrea Tiengo, Raluca Ilie

Abstract: The spatial distribution of energetic protons contributes towards the understanding of magnetospheric dynamics. Based upon 17 years of the Cluster/RAPID observations, we have derived machine learning-based models to predict the proton intensities at energies from 28 to 1,885 keV in the 3D terrestrial magnetosphere at radial distances between 6 and 22 RE. We used the satellite location and indices… ▽ More The spatial distribution of energetic protons contributes towards the understanding of magnetospheric dynamics. Based upon 17 years of the Cluster/RAPID observations, we have derived machine learning-based models to predict the proton intensities at energies from 28 to 1,885 keV in the 3D terrestrial magnetosphere at radial distances between 6 and 22 RE. We used the satellite location and indices for solar, solar wind and geomagnetic activity as predictors. The results demonstrate that the neural network (multi-layer perceptron regressor) outperforms baseline models based on the k-Nearest Neighbors and historical binning on average by ~80% and ~33\%, respectively. The average correlation between the observed and predicted data is about 56%, which is reasonable in light of the complex dynamics of fast-moving energetic protons in the magnetosphere. In addition to a quantitative analysis of the prediction results, we also investigate parameter importance in our model. The most decisive parameters for predicting proton intensities are related to the location: ZGSE direction and the radial distance. Among the activity indices, the solar wind dynamic pressure is the most important. The results have a direct practical application, for instance, for assessing the contamination particle background in the X-Ray telescopes for X-ray astronomy orbiting above the radiation belts. To foster reproducible research and to enable the community to build upon our work we publish our complete code, the data, as well as weights of trained models. Further description can be found in the GitHub project at https://github.com/Tanveer81/deep_horizon. △ Less

Submitted 11 May, 2021; originally announced May 2021.

Learning Emergent Discrete Message Communication for Cooperative Reinforcement Learning

Authors: Sheng Li, Yutai Zhou, Ross Allen, Mykel J. Kochenderfer

Abstract: Communication is a important factor that enables agents work cooperatively in multi-agent reinforcement learning (MARL). Most previous work uses continuous message communication whose high representational capacity comes at the expense of interpretability. Allowing agents to learn their own discrete message communication protocol emerged from a variety of domains can increase the interpretability… ▽ More Communication is a important factor that enables agents work cooperatively in multi-agent reinforcement learning (MARL). Most previous work uses continuous message communication whose high representational capacity comes at the expense of interpretability. Allowing agents to learn their own discrete message communication protocol emerged from a variety of domains can increase the interpretability for human designers and other agents.This paper proposes a method to generate discrete messages analogous to human languages, and achieve communication by a broadcast-and-listen mechanism based on self-attention. We show that discrete message communication has performance comparable to continuous message communication but with much a much smaller vocabulary size.Furthermore, we propose an approach that allows humans to interactively send discrete messages to agents. △ Less

Submitted 24 February, 2021; originally announced February 2021.

Semantic Modeling with SUMO

Authors: Robert B. Allen

Abstract: We explore using the Suggested Upper Merged Ontology (SUMO) to develop a semantic simulation. We provide two proof-of-concept demonstrations modeling transitions in a simulated gasoline engine using a general-purpose programming language. Rather than focusing on computationally highly intensive techniques, we explore a less computationally intensive approach related to familiar software engineerin… ▽ More We explore using the Suggested Upper Merged Ontology (SUMO) to develop a semantic simulation. We provide two proof-of-concept demonstrations modeling transitions in a simulated gasoline engine using a general-purpose programming language. Rather than focusing on computationally highly intensive techniques, we explore a less computationally intensive approach related to familiar software engineering testing procedures. In addition, we propose structured representations of terms based on linguistic approaches to lexicography. △ Less

Submitted 12 January, 2021; v1 submitted 31 December, 2020; originally announced December 2020.

Deep Implicit Coordination Graphs for Multi-agent Reinforcement Learning

Authors: Sheng Li, Jayesh K. Gupta, Peter Morales, Ross Allen, Mykel J. Kochenderfer

Abstract: Multi-agent reinforcement learning (MARL) requires coordination to efficiently solve certain tasks. Fully centralized control is often infeasible in such domains due to the size of joint action spaces. Coordination graph based formalization allows reasoning about the joint action based on the structure of interactions. However, they often require domain expertise in their design. This paper introd… ▽ More Multi-agent reinforcement learning (MARL) requires coordination to efficiently solve certain tasks. Fully centralized control is often infeasible in such domains due to the size of joint action spaces. Coordination graph based formalization allows reasoning about the joint action based on the structure of interactions. However, they often require domain expertise in their design. This paper introduces the deep implicit coordination graph (DICG) architecture for such scenarios. DICG consists of a module for inferring the dynamic coordination graph structure which is then used by a graph neural network based module to learn to implicitly reason about the joint actions or values. DICG allows learning the tradeoff between full centralization and decentralization via standard actor-critic methods to significantly improve coordination for domains with large number of agents. We apply DICG to both centralized-training-centralized-execution and centralized-training-decentralized-execution regimes. We demonstrate that DICG solves the relative overgeneralization pathology in predatory-prey tasks as well as outperforms various MARL baselines on the challenging StarCraft II Multi-agent Challenge (SMAC) and traffic junction environments. △ Less

Submitted 3 February, 2021; v1 submitted 19 June, 2020; originally announced June 2020.

Definitions and Semantic Simulations Based on Object-Oriented Analysis and Modeling

Authors: Robert B. Allen

Abstract: We have proposed going beyond traditional ontologies to use rich semantics implemented in programming languages for modeling. In this paper, we discuss the application of executable semantic models to two examples, first a structured definition of a waterfall and second the cardiopulmonary system. We examine the components of these models and the way those components interact. Ultimately, such mod… ▽ More We have proposed going beyond traditional ontologies to use rich semantics implemented in programming languages for modeling. In this paper, we discuss the application of executable semantic models to two examples, first a structured definition of a waterfall and second the cardiopulmonary system. We examine the components of these models and the way those components interact. Ultimately, such models should provide the basis for direct representation. △ Less

Submitted 31 December, 2019; originally announced December 2019.

Health-Informed Policy Gradients for Multi-Agent Reinforcement Learning

Authors: Ross E. Allen, Jayesh K. Gupta, Jaime Pena, Yutai Zhou, Javona White Bear, Mykel J. Kochenderfer

Abstract: This paper proposes a definition of system health in the context of multiple agents optimizing a joint reward function. We use this definition as a credit assignment term in a policy gradient algorithm to distinguish the contributions of individual agents to the global reward. The health-informed credit assignment is then extended to a multi-agent variant of the proximal policy optimization algori… ▽ More This paper proposes a definition of system health in the context of multiple agents optimizing a joint reward function. We use this definition as a credit assignment term in a policy gradient algorithm to distinguish the contributions of individual agents to the global reward. The health-informed credit assignment is then extended to a multi-agent variant of the proximal policy optimization algorithm and demonstrated on particle and multiwalker robot environments that have characteristics such as system health, risk-taking, semi-expendable agents, continuous action spaces, and partial observability. We show significant improvement in learning performance compared to policy gradient methods that do not perform multi-agent credit assignment. △ Less

Submitted 4 January, 2021; v1 submitted 2 August, 2019; originally announced August 2019.

Rapid trial-and-error learning with simulation supports flexible tool use and physical reasoning

Authors: Kelsey R. Allen, Kevin A. Smith, Joshua B. Tenenbaum

Abstract: Many animals, and an increasing number of artificial agents, display sophisticated capabilities to perceive and manipulate objects. But human beings remain distinctive in their capacity for flexible, creative tool use -- using objects in new ways to act on the world, achieve a goal, or solve a problem. To study this type of general physical problem solving, we introduce the Virtual Tools game. In… ▽ More Many animals, and an increasing number of artificial agents, display sophisticated capabilities to perceive and manipulate objects. But human beings remain distinctive in their capacity for flexible, creative tool use -- using objects in new ways to act on the world, achieve a goal, or solve a problem. To study this type of general physical problem solving, we introduce the Virtual Tools game. In this game, people solve a large range of challenging physical puzzles in just a handful of attempts. We propose that the flexibility of human physical problem solving rests on an ability to imagine the effects of hypothesized actions, while the efficiency of human search arises from rich action priors which are updated via observations of the world. We instantiate these components in the "Sample, Simulate, Update" (SSUP) model and show that it captures human performance across 30 levels of the Virtual Tools game. More broadly, this model provides a mechanism for explaining how people condense general physical knowledge into actionable, task-specific plans to achieve flexible and efficient physical problem-solving. △ Less

Submitted 29 June, 2020; v1 submitted 22 July, 2019; originally announced July 2019.

Comments: This manuscript is in press at PNAS. It is an extended version of a paper "Rapid Trial-and-Error Learning in Physical Problem Solving" accepted for oral presentation at the 41st Annual Meeting of the Cognitive Science Society (2019). It represents ongoing work on the part of the authors

Few-Shot Bayesian Imitation Learning with Logical Program Policies

Authors: Tom Silver, Kelsey R. Allen, Alex K. Lew, Leslie Pack Kaelbling, Josh Tenenbaum

Abstract: Humans can learn many novel tasks from a very small number (1--5) of demonstrations, in stark contrast to the data requirements of nearly tabula rasa deep learning methods. We propose an expressive class of policies, a strong but general prior, and a learning algorithm that, together, can learn interesting policies from very few examples. We represent policies as logical combinations of programs d… ▽ More Humans can learn many novel tasks from a very small number (1--5) of demonstrations, in stark contrast to the data requirements of nearly tabula rasa deep learning methods. We propose an expressive class of policies, a strong but general prior, and a learning algorithm that, together, can learn interesting policies from very few examples. We represent policies as logical combinations of programs drawn from a domain-specific language (DSL), define a prior over policies with a probabilistic grammar, and derive an approximate Bayesian inference algorithm to learn policies from demonstrations. In experiments, we study five strategy games played on a 2D grid with one shared DSL. After a few demonstrations of each game, the inferred policies generalize to new game instances that differ substantially from the demonstrations. Our policy learning is 20--1,000x more data efficient than convolutional and fully convolutional policy learning and many orders of magnitude more computationally efficient than vanilla program induction. We argue that the proposed method is an apt choice for tasks that have scarce training data and feature significant, structured variation between task instances. △ Less

Submitted 16 November, 2019; v1 submitted 12 April, 2019; originally announced April 2019.

Comments: AAAI 2020

Infinite Mixture Prototypes for Few-Shot Learning

Authors: Kelsey R. Allen, Evan Shelhamer, Hanul Shin, Joshua B. Tenenbaum

Abstract: We propose infinite mixture prototypes to adaptively represent both simple and complex data distributions for few-shot learning. Our infinite mixture prototypes represent each class by a set of clusters, unlike existing prototypical methods that represent each class by a single cluster. By inferring the number of clusters, infinite mixture prototypes interpolate between nearest neighbor and protot… ▽ More We propose infinite mixture prototypes to adaptively represent both simple and complex data distributions for few-shot learning. Our infinite mixture prototypes represent each class by a set of clusters, unlike existing prototypical methods that represent each class by a single cluster. By inferring the number of clusters, infinite mixture prototypes interpolate between nearest neighbor and prototypical representations, which improves accuracy and robustness in the few-shot regime. We show the importance of adaptive capacity for capturing complex data distributions such as alphabets, with 25% absolute accuracy improvements over prototypical networks, while still maintaining or improving accuracy on the standard Omniglot and mini-ImageNet benchmarks. In clustering labeled and unlabeled data by the same clustering rule, infinite mixture prototypes achieves state-of-the-art semi-supervised accuracy. As a further capability, we show that infinite mixture prototypes can perform purely unsupervised clustering, unlike existing prototypical methods. △ Less

Submitted 12 February, 2019; originally announced February 2019.

Earthquake Early Warning and Beyond: Systems Challenges in Smartphone-based Seismic Network

Authors: Qingkai Kong, Qin Lv, Richard M. Allen

Abstract: Earthquake Early Warning (EEW) systems can effectively reduce fatalities, injuries, and damages caused by earthquakes. Current EEW systems are mostly based on traditional seismic and geodetic networks, and exist only in a few countries due to the high cost of installing and maintaining such systems. The MyShake system takes a different approach and turns people's smartphones into portable seismic… ▽ More Earthquake Early Warning (EEW) systems can effectively reduce fatalities, injuries, and damages caused by earthquakes. Current EEW systems are mostly based on traditional seismic and geodetic networks, and exist only in a few countries due to the high cost of installing and maintaining such systems. The MyShake system takes a different approach and turns people's smartphones into portable seismic sensors to detect earthquake-like motions. However, to issue EEW messages with high accuracy and low latency in the real world, we need to address a number of challenges related to mobile computing. In this paper, we first summarize our experience building and deploying the MyShake system, then focus on two key challenges for smartphone-based EEW (sensing heterogeneity and user/system dynamics) and some preliminary exploration. We also discuss other challenges and new research directions associated with smartphone-based seismic network. △ Less

Submitted 19 January, 2019; originally announced January 2019.

Comments: 6 pages, conference paper, already accepted at hotmobile 2019

Journal ref: HotMobile 2019: 57-62

Issues for Using Semantic Modeling to Represent Mechanisms

Authors: Robert B Allen

Abstract: Mechanisms are a fundamental concept in many areas of science. Nonetheless, there has been little effort to develop structures to represent mechanisms. We explore the issues in developing a basic semantic modeling framework for describing some types of mechanisms. We draw together threads from a number of different approaches and then consider two examples. From this survey, we propose a rich Sema… ▽ More Mechanisms are a fundamental concept in many areas of science. Nonetheless, there has been little effort to develop structures to represent mechanisms. We explore the issues in developing a basic semantic modeling framework for describing some types of mechanisms. We draw together threads from a number of different approaches and then consider two examples. From this survey, we propose a rich Semantic Modeling Framework (SMF) based on Transitionals and hierarchies of Aggregates and Mechanisms, which could be implemented with the XFO programming environment. Potentially, the framework will be useful for developing direct-representation scientific research reports and community models. △ Less

Submitted 29 December, 2018; originally announced December 2018.

Coordinating and Integrating Faceted Classification with Rich Semantic Modeling

Authors: Robert B. Allen, Jaihyun Park

Abstract: Faceted classifications define dimensions for the types of entities included. In effect, the facets provide an "ontological commitment". We compare a faceted thesaurus, the Art and Architecture Thesaurus (AAT), with ontologies derived from the Basic Formal Ontology (BFO2), which is an upper (or formal) ontology widely used to describe entities in biomedicine. We consider how the AAT and BFO2-based… ▽ More Faceted classifications define dimensions for the types of entities included. In effect, the facets provide an "ontological commitment". We compare a faceted thesaurus, the Art and Architecture Thesaurus (AAT), with ontologies derived from the Basic Formal Ontology (BFO2), which is an upper (or formal) ontology widely used to describe entities in biomedicine. We consider how the AAT and BFO2-based ontologies could be coordinated and integrated into a Human Activity and Infrastructure Foundry (HAIF). To extend the AAT to enable this coordination and integration, we describe how a wider range of relationships among its terms could be introduced. Using these extensions, we explore richer modeling of topics from AAT that deal with Technology. Finally, we consider how ontology-based frames and semantic role frames can be integrated to make rich semantic statements about changes in the world. △ Less

Submitted 25 September, 2018; originally announced September 2018.

Relational inductive bias for physical construction in humans and machines

Authors: Jessica B. Hamrick, Kelsey R. Allen, Victor Bapst, Tina Zhu, Kevin R. McKee, Joshua B. Tenenbaum, Peter W. Battaglia

Abstract: While current deep learning systems excel at tasks such as object classification, language processing, and gameplay, few can construct or modify a complex system such as a tower of blocks. We hypothesize that what these systems lack is a "relational inductive bias": a capacity for reasoning about inter-object relations and making choices over a structured description of a scene. To test this hypot… ▽ More While current deep learning systems excel at tasks such as object classification, language processing, and gameplay, few can construct or modify a complex system such as a tower of blocks. We hypothesize that what these systems lack is a "relational inductive bias": a capacity for reasoning about inter-object relations and making choices over a structured description of a scene. To test this hypothesis, we focus on a task that involves gluing pairs of blocks together to stabilize a tower, and quantify how well humans perform. We then introduce a deep reinforcement learning agent which uses object- and relation-centric scene and policy representations and apply it to the task. Our results show that these structured representations allow the agent to outperform both humans and more naive approaches, suggesting that relational inductive bias is an important component in solving structured reasoning problems and for building more intelligent, flexible machines. △ Less

Submitted 4 June, 2018; originally announced June 2018.

Comments: In Proceedings of the Annual Meeting of the Cognitive Science Society (CogSci 2018)

XFO: Toward Programming Rich Semantic Models

Authors: Robert B. Allen, Teryn K. Jones

Abstract: We have proposed that ontologies and programming languages should be more closely aligned. Specifically, we have argued that the Basic Formal Ontology (BFO2) has many features that are consistent with object-oriented analysis, design, and modeling. Here, we describe the eXtended Formal Ontology (XFO), a programming environment we developed to support semantic modeling. We then use XFO to implement… ▽ More We have proposed that ontologies and programming languages should be more closely aligned. Specifically, we have argued that the Basic Formal Ontology (BFO2) has many features that are consistent with object-oriented analysis, design, and modeling. Here, we describe the eXtended Formal Ontology (XFO), a programming environment we developed to support semantic modeling. We then use XFO to implement a Traffic Light Microworld and discuss more complex applications. △ Less

Submitted 30 April, 2018; originally announced May 2018.

Semantic Modeling with Foundries

Authors: Robert B. Allen, Yoonhwan Kim

Abstract: We analyze challenges for the development of the Human Activities and Infrastructures Foundry. We explore a rich semantic modeling approach to describe two Korean ceramic water droppers used to mix ink for calligraphy, how they were produced and the reasons for their differing aesthetic. Our modeling supports schema and allows for transitions of Entities based on the relationships to other Entitie… ▽ More We analyze challenges for the development of the Human Activities and Infrastructures Foundry. We explore a rich semantic modeling approach to describe two Korean ceramic water droppers used to mix ink for calligraphy, how they were produced and the reasons for their differing aesthetic. Our modeling supports schema and allows for transitions of Entities based on the relationships to other Entities with which they are associated. We explore the similarity of our approach to object-oriented analysis and modeling. △ Less

Submitted 21 January, 2018; v1 submitted 29 December, 2017; originally announced January 2018.

A Foundry of Human Activities and Infrastructures

Authors: Robert B. Allen, Eunsang Yang, Tatsawan Timakum

Abstract: Direct representation knowledgebases can enhance and even provide an alternative to document-centered digital libraries. Here we consider realist semantic modeling of everyday activities and infrastructures in such knowledgebases. Because we want to integrate a wide variety of topics, a collection of ontologies (a foundry) and a range of other knowledge resources are needed. We first consider mode… ▽ More Direct representation knowledgebases can enhance and even provide an alternative to document-centered digital libraries. Here we consider realist semantic modeling of everyday activities and infrastructures in such knowledgebases. Because we want to integrate a wide variety of topics, a collection of ontologies (a foundry) and a range of other knowledge resources are needed. We first consider modeling the routine procedures that support human activities and technologies. Next, we examine the interactions of technologies with aspects of social organization. Then, we consider approaches and issues for developing and validating explanations of the relationships among various entities. △ Less

Submitted 31 October, 2017; originally announced November 2017.

Rich Semantic Models and Knowledgebases for Highly-Structured Scientific Communication

Authors: Robert B. Allen

Abstract: Rather than using text for scientific research reports, we have proposed developing highly-structured reports with rich semantic models. In this paper, we consider detailed structures for the components of research reports using a modeling framework based on a rigorous upper ontology. For instance, we consider the use of structured descriptions of Research Designs to support evaluation of internal… ▽ More Rather than using text for scientific research reports, we have proposed developing highly-structured reports with rich semantic models. In this paper, we consider detailed structures for the components of research reports using a modeling framework based on a rigorous upper ontology. For instance, we consider the use of structured descriptions of Research Designs to support evaluation of internal and external validity. In addition, collections of highly-structured scientific research reports would be the key component of a set of evolving and interlocking highly-structured scientific knowledgebases. △ Less

Submitted 28 August, 2017; originally announced August 2017.

From Ontology to Structured Applied Epistemology

Authors: Robert B. Allen

Abstract: Developing and organizing new knowledge is a core activity for scholars. Recently, ontologies have been introduced as an approach for organizing knowledge. However, most ontologies do not readily support the development and organization of new knowledge. By comparison, to ontology, epistemology is the study of what can be known. Aspects of epistemology include the acquisition of and justification… ▽ More Developing and organizing new knowledge is a core activity for scholars. Recently, ontologies have been introduced as an approach for organizing knowledge. However, most ontologies do not readily support the development and organization of new knowledge. By comparison, to ontology, epistemology is the study of what can be known. Aspects of epistemology include the acquisition of and justification for new knowledge. Thus, we need to coordinate ontology with epistemology. Because we are developing frameworks for capturing knowledge across several scholarly domains, we describe the work in this paper as exploring structured applied epistemology. Unlike other recent proposals for new approaches to scholarly publishing, we propose an integrated and comprehensive approach. We have explored direct representation based on the rigorous Basic Formal Ontology and in this paper, we consider how epistemology can be incorporated with that. In addition to highly-structured scientific research reports, we also consider how to develop highly-structured descriptions of historical events on which historical analyses can be based. △ Less

Submitted 31 December, 2016; v1 submitted 23 October, 2016; originally announced October 2016.

Incremental Voronoi Diagrams

Authors: Sarah R. Allen, Luis Barba, John Iacono, Stefan Langerman

Abstract: We study the amortized number of combinatorial changes (edge insertions and removals) needed to update the graph structure of the Voronoi diagram $\mathcal{V}(S)$ (and several variants thereof) of a set $S$ of $n$ sites in the plane as sites are added. We define a general update operation for planar graphs modeling the incremental construction of several variants of Voronoi diagrams as well as t… ▽ More We study the amortized number of combinatorial changes (edge insertions and removals) needed to update the graph structure of the Voronoi diagram $\mathcal{V}(S)$ (and several variants thereof) of a set $S$ of $n$ sites in the plane as sites are added. We define a general update operation for planar graphs modeling the incremental construction of several variants of Voronoi diagrams as well as the incremental construction of an intersection of halfspaces in $\mathbb{R}^3$. We show that the amortized number of edge insertions and removals needed to add a new site is $O(\sqrt{n})$. A matching $Ω(\sqrt{n})$ combinatorial lower bound is shown, even in the case where the graph of the diagram is a tree. This contrasts with the $O(\log{n})$ upper bound of Aronov et al. (2006) for farthest-point Voronoi diagrams when the points are inserted in order along their convex hull. We present a semi-dynamic data structure that maintains the Voronoi diagram of a set $S$ of $n$ sites in convex position. This structure supports the insertion of a new site $p$ and finds the asymptotically minimal number $K$ of edge insertions and removals needed to obtain the diagram of $S \cup \{p\}$ from the diagram of $S$, in time $O(K\,\mathrm{polylog}\ n)$ worst case, which is $O(\sqrt{n}\;\mathrm{polylog}\ n)$ amortized by the aforementioned result. The most distinctive feature of this data structure is that the graph of the Voronoi diagram is maintained at all times and can be traversed in the natural way; this contrasts with other known data structures supporting nearest neighbor queries. Our data structure supports general search operations on the current Voronoi diagram, which can, for example, be used to perform point location queries in the cells of the current Voronoi diagram in $O(\log n)$ time, or to determine whether two given sites are neighbors in the Delaunay triangulation. △ Less

Submitted 28 March, 2016; originally announced March 2016.

Comments: 19 pages, SoCG 2016

Repositories with Direct Representation

Authors: Robert Burnell Allen

Abstract: A new generation of digital repositories could be based on direct representation of the contents with rich semantics and models rather than be collections of documents. The contents of such repositories would be highly structured which should help users to focus on meaningful relationships of the contents. These repositories would implement earlier proposals for model-oriented information organiza… ▽ More A new generation of digital repositories could be based on direct representation of the contents with rich semantics and models rather than be collections of documents. The contents of such repositories would be highly structured which should help users to focus on meaningful relationships of the contents. These repositories would implement earlier proposals for model-oriented information organization by extending current work on ontologies to cover state changes, instances, and scenarios. They could also apply other approaches such as object-oriented design and frame semantics. In addition to semantics, the representation needs to allow for discourse and repository knowledge-support services and policies. For instance, the knowledge base would need to be systematically updated as new findings and theories reshape it. △ Less

Submitted 30 December, 2015; originally announced December 2015.

How to refute a random CSP

Authors: Sarah R. Allen, Ryan O'Donnell, David Witmer

Abstract: Let $P$ be a $k$-ary predicate over a finite alphabet. Consider a random CSP$(P)$ instance $I$ over $n$ variables with $m$ constraints. When $m \gg n$ the instance $I$ will be unsatisfiable with high probability, and we want to find a refutation - i.e., a certificate of unsatisfiability. When $P$ is the $3$-ary OR predicate, this is the well studied problem of refuting random $3$-SAT formulas, and… ▽ More Let $P$ be a $k$-ary predicate over a finite alphabet. Consider a random CSP$(P)$ instance $I$ over $n$ variables with $m$ constraints. When $m \gg n$ the instance $I$ will be unsatisfiable with high probability, and we want to find a refutation - i.e., a certificate of unsatisfiability. When $P$ is the $3$-ary OR predicate, this is the well studied problem of refuting random $3$-SAT formulas, and an efficient algorithm is known only when $m \gg n^{3/2}$. Understanding the density required for refutation of other predicates is important in cryptography, proof complexity, and learning theory. Previously, it was known that for a $k$-ary predicate, having $m \gg n^{\lceil k/2 \rceil}$ constraints suffices for refutation. We give a criterion for predicates that often yields efficient refutation algorithms at much lower densities. Specifically, if $P$ fails to support a $t$-wise uniform distribution, then there is an efficient algorithm that refutes random CSP$(P)$ instances $I$ whp when $m \gg n^{t/2}$. Indeed, our algorithm will "somewhat strongly" refute $I$, certifying $\mathrm{Opt}(I) \leq 1-Ω_k(1)$, if $t = k$ then we get the strongest possible refutation, certifying $\mathrm{Opt}(I) \leq \mathrm{E}[P] + o(1)$. This last result is new even in the context of random $k$-SAT. Regarding the optimality of our $m \gg n^{t/2}$ requirement, prior work on SDP hierarchies has given some evidence that efficient refutation of random CSP$(P)$ may be impossible when $m \ll n^{t/2}$. Thus there is an indication our algorithm's dependence on $m$ is optimal for every $P$, at least in the context of SDP hierarchies. Along these lines, we show that our refutation algorithm can be carried out by the $O(1)$-round SOS SDP hierarchy. Finally, as an application of our result, we falsify assumptions used to show hardness-of-learning results in recent work of Daniely, Linial, and Shalev-Shwartz. △ Less

Submitted 27 July, 2015; v1 submitted 17 May, 2015; originally announced May 2015.

Structured Descriptions of Roles, Activities,and Procedures in the Roman Constitution

Authors: Yoonmi Chu, Robert B. Allen

Abstract: A highly structured description of entities and events in histories can support flexible exploration of those histories by users and, ultimately, support richly-linked full-text digital libraries. Here, we apply the Basic Formal Ontology (BFO) to structure a passage about the Roman Constitution from Gibbon's Decline and Fall of the Roman Empire. Specifically, we consider the specification of Roles… ▽ More A highly structured description of entities and events in histories can support flexible exploration of those histories by users and, ultimately, support richly-linked full-text digital libraries. Here, we apply the Basic Formal Ontology (BFO) to structure a passage about the Roman Constitution from Gibbon's Decline and Fall of the Roman Empire. Specifically, we consider the specification of Roles such as Consuls, Activities associated with those Roles, and Procedures for accomplishing those Activities. △ Less

Submitted 13 February, 2015; originally announced February 2015.

Comments: 6 pages, 2 figures, Presented at the Italian Research Conference on Digital Libraries (IRCDL 2015), Bozen-Bolzano, Italy, 29-30 January, 2015

Improving Access to Digitized Historical Newspapers with Text Mining, Coordinated Models, and Formative User Interface Design

Authors: Robert B. Allen

Abstract: Most tools for accessing digitized historical newspapers emphasize relatively simple search; but, as increasing numbers of digitized historical newspapers and other historical resources become available we can consider much richer modes of interaction with these collections. For instance, users might use exploratory search for looking at larger issues and events such as elections and campaigns or… ▽ More Most tools for accessing digitized historical newspapers emphasize relatively simple search; but, as increasing numbers of digitized historical newspapers and other historical resources become available we can consider much richer modes of interaction with these collections. For instance, users might use exploratory search for looking at larger issues and events such as elections and campaigns or to get a sense of "the texture of the city" or "what the city was thinking". To take full advantage of rich interface tools, the content of the newspapers needs to be described systematically and accurately. Moreover, collections of multiple newspapers need to be richly cross-indexed across titles and even with historical resources beyond the newspapers. △ Less

Submitted 13 February, 2015; originally announced February 2015.

Comments: IFLA Newspaper Section IFLA Newspaper Meeting, New Delhi, February 2010

Conditioning and covariance on caterpillars

Authors: Sarah R. Allen, Ryan O'Donnell

Abstract: Let $X_1, \dots, X_n$ be joint $\{ \pm 1\}$-valued random variables. It is known that conditioning on a random subset of $O(1/ε^2)$ of them reduces their average pairwise covariance to below $ε$ (in expectation). We conjecture that $O(1/ε^2)$ can be improved to $O(1/ε)$. The motivation for the problem and our conjectured improvement comes from the theory of global correlation rounding for convex r… ▽ More Let $X_1, \dots, X_n$ be joint $\{ \pm 1\}$-valued random variables. It is known that conditioning on a random subset of $O(1/ε^2)$ of them reduces their average pairwise covariance to below $ε$ (in expectation). We conjecture that $O(1/ε^2)$ can be improved to $O(1/ε)$. The motivation for the problem and our conjectured improvement comes from the theory of global correlation rounding for convex relaxation hierarchies. We suggest attempting the conjecture in the case that $X_1, \dots, X_n$ are the leaves of an information flow tree. We prove the conjecture in the case that the information flow tree is a caterpillar graph (similar to a two-state hidden Markov model). △ Less

Submitted 16 July, 2014; originally announced July 2014.

Evaluation of DNF Formulas

Authors: Sarah R. Allen, Lisa Hellerstein, Devorah Kletenik, Tonguç Ünlüyurt

Abstract: Stochastic Boolean Function Evaluation (SBFE) is the problem of determining the value of a given Boolean function $f$ on an unknown input $x$, when each bit of $x_i$ of $x$ can only be determined by paying a given associated cost $c_i$. Further, $x$ is drawn from a given product distribution: for each $x_i$, $Prob[x_i=1] = p_i$, and the bits are independent. The goal is to minimize the expected co… ▽ More Stochastic Boolean Function Evaluation (SBFE) is the problem of determining the value of a given Boolean function $f$ on an unknown input $x$, when each bit of $x_i$ of $x$ can only be determined by paying a given associated cost $c_i$. Further, $x$ is drawn from a given product distribution: for each $x_i$, $Prob[x_i=1] = p_i$, and the bits are independent. The goal is to minimize the expected cost of evaluation. Stochastic Boolean Function Evaluation (SBFE) is the problem of determining the value of a given Boolean function $f$ on an unknown input $x$, when each bit of $x_i$ of $x$ can only be determined by paying a given associated cost $c_i$. Further, $x$ is drawn from a given product distribution: for each $x_i$, $Prob[x_i=1] = p_i$, and the bits are independent. The goal is to minimize the expected cost of evaluation. In this paper, we study the complexity of the SBFE problem for classes of DNF formulas. We consider both exact and approximate versions of the problem for subclasses of DNF, for arbitrary costs and product distributions, and for unit costs and/or the uniform distribution. △ Less

Submitted 8 October, 2014; v1 submitted 14 October, 2013; originally announced October 2013.

Toward an Interactive Directory for Norfolk, Nebraska: 1899-1900

Authors: Robert B. Allen

Abstract: We describe steps toward an interactive directory for the town of Norfolk, Nebraska for the years 1899 and 1900. This directory would extend the traditional city directory by including a wider range of entities being described, much richer information about the entities mentioned and linkages to mentions of the entities in material such as digitized historical newspapers. Such a directory would be… ▽ More We describe steps toward an interactive directory for the town of Norfolk, Nebraska for the years 1899 and 1900. This directory would extend the traditional city directory by including a wider range of entities being described, much richer information about the entities mentioned and linkages to mentions of the entities in material such as digitized historical newspapers. Such a directory would be useful to readers who browse the historical newspapers by providing structured summaries of the entities mentioned. We describe the occurrence of entities in two years of the Norfolk Weekly News, focusing on several individuals to better understand the types of information which can be gleaned from historical newspapers and other historical materials. We also describe a prototype program which coordinates information about entities from the traditional city directories, the federal census, and from newspapers. We discuss the structured coding for these entities, noting that richer coding would increasingly include descriptions of events and scenarios. We propose that rich content about individuals and communities could eventually be modeled with agents and woven into historical narratives. △ Less

Submitted 25 August, 2013; originally announced August 2013.

Journal ref: IFLA Newspaper and Genealogy Section Meeting, Singapore, Aug 2013

Packing identical simple polygons is NP-hard

Authors: Sarah R. Allen, John Iacono

Abstract: Given a small polygon S, a big simple polygon B and a positive integer k, it is shown to be NP-hard to determine whether k copies of the small polygon (allowing translation and rotation) can be placed in the big polygon without overlap. Previous NP-hardness results were only known in the case where the big polygon is allowed to be non-simple. A novel reduction from Planar-Circuit-SAT is presented… ▽ More Given a small polygon S, a big simple polygon B and a positive integer k, it is shown to be NP-hard to determine whether k copies of the small polygon (allowing translation and rotation) can be placed in the big polygon without overlap. Previous NP-hardness results were only known in the case where the big polygon is allowed to be non-simple. A novel reduction from Planar-Circuit-SAT is presented where a small polygon is constructed to encode the entire circuit. △ Less

Submitted 24 September, 2012; originally announced September 2012.