-
Explaining Decisions of Agents in Mixed-Motive Games
Authors:
Maayan Orner,
Oleg Maksimov,
Akiva Kleinerman,
Charles Ortiz,
Sarit Kraus
Abstract:
In recent years, agents have become capable of communicating seamlessly via natural language and navigating in environments that involve cooperation and competition, a fact that can introduce social dilemmas. Due to the interleaving of cooperation and competition, understanding agents' decision-making in such environments is challenging, and humans can benefit from obtaining explanations. However,…
▽ More
In recent years, agents have become capable of communicating seamlessly via natural language and navigating in environments that involve cooperation and competition, a fact that can introduce social dilemmas. Due to the interleaving of cooperation and competition, understanding agents' decision-making in such environments is challenging, and humans can benefit from obtaining explanations. However, such environments and scenarios have rarely been explored in the context of explainable AI. While some explanation methods for cooperative environments can be applied in mixed-motive setups, they do not address inter-agent competition, cheap-talk, or implicit communication by actions. In this work, we design explanation methods to address these issues. Then, we proceed to establish generality and demonstrate the applicability of the methods to three games with vastly different properties. Lastly, we demonstrate the effectiveness and usefulness of the methods for humans in two mixed-motive games. The first is a challenging 7-player game called no-press Diplomacy. The second is a 3-player game inspired by the prisoner's dilemma, featuring communication in natural language.
△ Less
Submitted 27 January, 2025; v1 submitted 21 July, 2024;
originally announced July 2024.
-
RSMM: A Framework to Assess Maturity of Research Software Project
Authors:
Deekshitha,
Rena Bakhshi,
Jason Maassen,
Carlos Martinez Ortiz,
Rob van Nieuwpoort,
Slinger Jansen
Abstract:
The organizations and researchers producing research software face a common problem of making their software sustainable beyond funding provided by a single research project. This is addressed by research software engineers through building communities around their software, providing appropriate licensing, creating reliable and reproducible research software, making it sustainable and impactful,…
▽ More
The organizations and researchers producing research software face a common problem of making their software sustainable beyond funding provided by a single research project. This is addressed by research software engineers through building communities around their software, providing appropriate licensing, creating reliable and reproducible research software, making it sustainable and impactful, promoting, and ensuring that the research software is easy to adopt in research workflows, etc. As a result, numerous practices and guidelines exist to enhance research software quality, reusability, and sustainability. However, there is a lack of a unified framework to systematically integrate these practices and help organizations and research software developers refine their development and management processes. Our paper aims at bridging this gap by introducing a novel framework: RSMM. It is designed through systematic literature review and insights from interviews with research software project experts. In short, RSMM offers a structured pathway for evaluating and refining research software project management by categorizing 79 best practices into 17 capabilities across 4 focus areas. From assessing code quality and security to measuring impact, sustainability, and reproducibility, the model provides a complete evaluation of a research software project maturity. With RSMM, individuals as well as organizations involved in research software development gain a systematic approach to tackling various research software engineering challenges. By utilizing RSMM as a comprehensive checklist, organizations can systematically evaluate and refine their project management practices and organizational structure.
△ Less
Submitted 3 June, 2024;
originally announced June 2024.
-
Unpacking Human Teachers' Intentions For Natural Interactive Task Learning
Authors:
Preeti Ramaraj,
Charles L. Ortiz, Jr.,
Shiwali Mohan
Abstract:
Interactive Task Learning (ITL) is an emerging research agenda that studies the design of complex intelligent robots that can acquire new knowledge through natural human teacher-robot learner interactions. ITL methods are particularly useful for designing intelligent robots whose behavior can be adapted by humans collaborating with them. Various research communities are contributing methods for IT…
▽ More
Interactive Task Learning (ITL) is an emerging research agenda that studies the design of complex intelligent robots that can acquire new knowledge through natural human teacher-robot learner interactions. ITL methods are particularly useful for designing intelligent robots whose behavior can be adapted by humans collaborating with them. Various research communities are contributing methods for ITL and a large subset of this research is \emph{robot-centered} with a focus on developing algorithms that can learn online, quickly. This paper studies the ITL problem from a \emph{human-centered} perspective to provide guidance for robot design so that human teachers can naturally teach ITL robots. In this paper, we present 1) a qualitative bidirectional analysis of an interactive teaching study (N=10) through which we characterize various aspects of actions intended and executed by human teachers when teaching a robot; 2) an in-depth discussion of the teaching approach employed by two participants to understand the need for personal adaptation to individual teaching styles; and 3) requirements for ITL robot design based on our analyses and informed by a computational theory of collaborative interactions, SharedPlans.
△ Less
Submitted 2 July, 2021; v1 submitted 12 February, 2021;
originally announced February 2021.
-
Low Earth Orbit Satellites Provide Continuous Enterprise Data Connectivity
Authors:
Sean Batir,
Nicholas Humann,
Carmel Ortiz,
David Bettinger,
Susanne Heger,
Bennie Vorster
Abstract:
A critical problem in global telecommunication is the drastic increase in data volume transmitted and received across the world. To address the need for scalable telecommunication solutions in light of growing data volume, the BMW Group and Low Earth Orbit (LEO) satellite provider OneWeb pursue a proof of concept demo that assesses the potential of LEO networks for enterprise connectivity. Our res…
▽ More
A critical problem in global telecommunication is the drastic increase in data volume transmitted and received across the world. To address the need for scalable telecommunication solutions in light of growing data volume, the BMW Group and Low Earth Orbit (LEO) satellite provider OneWeb pursue a proof of concept demo that assesses the potential of LEO networks for enterprise connectivity. Our results suggest that LEO satellite networks can enable the hybrid connectivity needed for continuous data transmission without interruption or loss of signal to enable the future of work and premium mobility. This is a proof of concept experiment to show throughput, latency, and key applications including handover to 4G and the use of a VPN while running cloud applications. Across three tests (i.e. entertainment and business productivity streaming services), the researchers demonstrate a 2-3x faster ping rate (ms), 4-5x faster download rates (Mb/s), and 30-60x (Mb/s) faster upload rates.
△ Less
Submitted 24 July, 2020;
originally announced July 2020.
-
Intersectional Bias in Hate Speech and Abusive Language Datasets
Authors:
Jae Yeon Kim,
Carlos Ortiz,
Sarah Nam,
Sarah Santiago,
Vivek Datta
Abstract:
Algorithms are widely applied to detect hate speech and abusive language in social media. We investigated whether the human-annotated data used to train these algorithms are biased. We utilized a publicly available annotated Twitter dataset (Founta et al. 2018) and classified the racial, gender, and party identification dimensions of 99,996 tweets. The results showed that African American tweets w…
▽ More
Algorithms are widely applied to detect hate speech and abusive language in social media. We investigated whether the human-annotated data used to train these algorithms are biased. We utilized a publicly available annotated Twitter dataset (Founta et al. 2018) and classified the racial, gender, and party identification dimensions of 99,996 tweets. The results showed that African American tweets were up to 3.7 times more likely to be labeled as abusive, and African American male tweets were up to 77% more likely to be labeled as hateful compared to the others. These patterns were statistically significant and robust even when party identification was added as a control variable. This study provides the first systematic evidence on intersectional bias in datasets of hate speech and abusive language.
△ Less
Submitted 28 May, 2020; v1 submitted 12 May, 2020;
originally announced May 2020.
-
Dialogue Act Classification in Group Chats with DAG-LSTMs
Authors:
Ozan İrsoy,
Rakesh Gosangi,
Haimin Zhang,
Mu-Hsin Wei,
Peter Lund,
Duccio Pappadopulo,
Brendan Fahy,
Neophytos Nephytou,
Camilo Ortiz
Abstract:
Dialogue act (DA) classification has been studied for the past two decades and has several key applications such as workflow automation and conversation analytics. Researchers have used, to address this problem, various traditional machine learning models, and more recently deep neural network models such as hierarchical convolutional neural networks (CNNs) and long short-term memory (LSTM) networ…
▽ More
Dialogue act (DA) classification has been studied for the past two decades and has several key applications such as workflow automation and conversation analytics. Researchers have used, to address this problem, various traditional machine learning models, and more recently deep neural network models such as hierarchical convolutional neural networks (CNNs) and long short-term memory (LSTM) networks. In this paper, we introduce a new model architecture, directed-acyclic-graph LSTM (DAG-LSTM) for DA classification. A DAG-LSTM exploits the turn-taking structure naturally present in a multi-party conversation, and encodes this relation in its model structure. Using the STAC corpus, we show that the proposed method performs roughly 0.8% better in accuracy and 1.2% better in macro-F1 score when compared to existing methods. The proposed method is generic and not limited to conversation applications.
△ Less
Submitted 2 August, 2019;
originally announced August 2019.
-
Empirical Methodology for Crowdsourcing Ground Truth
Authors:
Anca Dumitrache,
Oana Inel,
Benjamin Timmermans,
Carlos Ortiz,
Robert-Jan Sips,
Lora Aroyo,
Chris Welty
Abstract:
The process of gathering ground truth data through human annotation is a major bottleneck in the use of information extraction methods for populating the Semantic Web. Crowdsourcing-based approaches are gaining popularity in the attempt to solve the issues related to volume of data and lack of annotators. Typically these practices use inter-annotator agreement as a measure of quality. However, in…
▽ More
The process of gathering ground truth data through human annotation is a major bottleneck in the use of information extraction methods for populating the Semantic Web. Crowdsourcing-based approaches are gaining popularity in the attempt to solve the issues related to volume of data and lack of annotators. Typically these practices use inter-annotator agreement as a measure of quality. However, in many domains, such as event detection, there is ambiguity in the data, as well as a multitude of perspectives of the information examples. We present an empirically derived methodology for efficiently gathering of ground truth data in a diverse set of use cases covering a variety of domains and annotation tasks. Central to our approach is the use of CrowdTruth metrics that capture inter-annotator disagreement. We show that measuring disagreement is essential for acquiring a high quality ground truth. We achieve this by comparing the quality of the data aggregated with CrowdTruth metrics with majority vote, over a set of diverse crowdsourcing tasks: Medical Relation Extraction, Twitter Event Identification, News Event Extraction and Sound Interpretation. We also show that an increased number of crowd workers leads to growth and stabilization in the quality of annotations, going against the usual practice of employing a small number of annotators.
△ Less
Submitted 24 September, 2018;
originally announced September 2018.
-
Nanopublications: A Growing Resource of Provenance-Centric Scientific Linked Data
Authors:
Tobias Kuhn,
Albert Meroño-Peñuela,
Alexander Malic,
Jorrit H. Poelen,
Allen H. Hurlbert,
Emilio Centeno Ortiz,
Laura I. Furlong,
Núria Queralt-Rosinach,
Christine Chichester,
Juan M. Banda,
Egon Willighagen,
Friederike Ehrhart,
Chris Evelo,
Tareq B. Malas,
Michel Dumontier
Abstract:
Nanopublications are a Linked Data format for scholarly data publishing that has received considerable uptake in the last few years. In contrast to the common Linked Data publishing practice, nanopublications work at the granular level of atomic information snippets and provide a consistent container format to attach provenance and metadata at this atomic level. While the nanopublications format i…
▽ More
Nanopublications are a Linked Data format for scholarly data publishing that has received considerable uptake in the last few years. In contrast to the common Linked Data publishing practice, nanopublications work at the granular level of atomic information snippets and provide a consistent container format to attach provenance and metadata at this atomic level. While the nanopublications format is domain-independent, the datasets that have become available in this format are mostly from Life Science domains, including data about diseases, genes, proteins, drugs, biological pathways, and biotic interactions. More than 10 million such nanopublications have been published, which now form a valuable resource for studies on the domain level of the given Life Science domains as well as on the more technical levels of provenance modeling and heterogeneous Linked Data. We provide here an overview of this combined nanopublication dataset, show the results of some overarching analyses, and describe how it can be accessed and queried.
△ Less
Submitted 18 September, 2018;
originally announced September 2018.
-
Mining Massive Hierarchical Data Using a Scalable Probabilistic Graphical Model
Authors:
Khalifeh AlJadda,
Mohammed Korayem,
Camilo Ortiz,
Trey Grainger,
John A. Miller,
Khaled Rasheed,
Krys J. Kochut,
William S. York,
Rene Ranzinger,
Melody Porterfield
Abstract:
Probabilistic Graphical Models (PGM) are very useful in the fields of machine learning and data mining. The crucial limitation of those models,however, is the scalability. The Bayesian Network, which is one of the most common PGMs used in machine learning and data mining, demonstrates this limitation when the training data consists of random variables, each of them has a large set of possible valu…
▽ More
Probabilistic Graphical Models (PGM) are very useful in the fields of machine learning and data mining. The crucial limitation of those models,however, is the scalability. The Bayesian Network, which is one of the most common PGMs used in machine learning and data mining, demonstrates this limitation when the training data consists of random variables, each of them has a large set of possible values. In the big data era, one would expect new extensions to the existing PGMs to handle the massive amount of data produced these days by computers, sensors and other electronic devices. With hierarchical data - data that is arranged in a treelike structure with several levels - one would expect to see hundreds of thousands or millions of values distributed over even just a small number of levels. When modeling this kind of hierarchical data across large data sets, Bayesian Networks become infeasible for representing the probability distributions. In this paper we introduce an extension to Bayesian Networks to handle massive sets of hierarchical data in a reasonable amount of time and space. The proposed model achieves perfect precision of 1.0 and high recall of 0.93 when it is used as multi-label classifier for the annotation of mass spectrometry data. On another data set of 1.5 billion search logs provided by CareerBuilder.com the model was able to predict latent semantic relationships between search keywords with accuracy up to 0.80.
△ Less
Submitted 28 December, 2015;
originally announced December 2015.
-
Methods of Class Field Theory to Separate Logics over Finite Residue Classes and Circuit Complexity
Authors:
Argimiro Arratia,
Carlos E. Ortiz
Abstract:
Separations among the first order logic ${\cal R}ing(0,+,*)$ of finite residue class rings, its extensions with generalized quantifiers, and in the presence of a built-in order are shown, using algebraic methods from class field theory. These methods include classification of spectra of sentences over finite residue classes as systems of congruences, and the study of their $h$-densities over the s…
▽ More
Separations among the first order logic ${\cal R}ing(0,+,*)$ of finite residue class rings, its extensions with generalized quantifiers, and in the presence of a built-in order are shown, using algebraic methods from class field theory. These methods include classification of spectra of sentences over finite residue classes as systems of congruences, and the study of their $h$-densities over the set of all prime numbers, for various functions $h$ on the natural numbers. Over ordered structures the logic of finite residue class rings and extensions are known to capture DLOGTIME-uniform circuit complexity classes ranging from $AC^0$ to $TC^0$. Separating these circuit complexity classes is directly related to classifying the $h$-density of spectra of sentences in the corresponding logics of finite residue classes. We further give general conditions under which a logic over the finite residue class rings has a sentence whose spectrum has no $h$-density. One application of this result is that in ${\cal R}ing(0,+,*,<) + M$, the logic of finite residue class rings with built-in order and extended with the majority quantifier $M$, there are sentences whose spectrum have no exponential density.
△ Less
Submitted 6 November, 2015;
originally announced November 2015.
-
Augmenting recommendation systems using a model of semantically-related terms extracted from user behavior
Authors:
Khalifeh AlJadda,
Mohammed Korayem,
Camilo Ortiz,
Chris Russell,
David Bernal,
Lamar Payson,
Scott Brown,
Trey Grainger
Abstract:
Common difficulties like the cold-start problem and a lack of sufficient information about users due to their limited interactions have been major challenges for most recommender systems (RS). To overcome these challenges and many similar ones that result in low accuracy (precision and recall) recommendations, we propose a novel system that extracts semantically-related search keywords based on th…
▽ More
Common difficulties like the cold-start problem and a lack of sufficient information about users due to their limited interactions have been major challenges for most recommender systems (RS). To overcome these challenges and many similar ones that result in low accuracy (precision and recall) recommendations, we propose a novel system that extracts semantically-related search keywords based on the aggregate behavioral data of many users. These semantically-related search keywords can be used to substantially increase the amount of knowledge about a specific user's interests based upon even a few searches and thus improve the accuracy of the RS. The proposed system is capable of mining aggregate user search logs to discover semantic relationships between key phrases in a manner that is language agnostic, human understandable, and virtually noise-free. These semantically related keywords are obtained by looking at the links between queries of similar users which, we believe, represent a largely untapped source for discovering latent semantic relationships between search terms.
△ Less
Submitted 8 September, 2014;
originally announced September 2014.
-
PGMHD: A Scalable Probabilistic Graphical Model for Massive Hierarchical Data Problems
Authors:
Khalifeh AlJadda,
Mohammed Korayem,
Camilo Ortiz,
Trey Grainger,
John A. Miller,
William S. York
Abstract:
In the big data era, scalability has become a crucial requirement for any useful computational model. Probabilistic graphical models are very useful for mining and discovering data insights, but they are not scalable enough to be suitable for big data problems. Bayesian Networks particularly demonstrate this limitation when their data is represented using few random variables while each random var…
▽ More
In the big data era, scalability has become a crucial requirement for any useful computational model. Probabilistic graphical models are very useful for mining and discovering data insights, but they are not scalable enough to be suitable for big data problems. Bayesian Networks particularly demonstrate this limitation when their data is represented using few random variables while each random variable has a massive set of values. With hierarchical data - data that is arranged in a treelike structure with several levels - one would expect to see hundreds of thousands or millions of values distributed over even just a small number of levels. When modeling this kind of hierarchical data across large data sets, Bayesian networks become infeasible for representing the probability distributions for the following reasons: i) Each level represents a single random variable with hundreds of thousands of values, ii) The number of levels is usually small, so there are also few random variables, and iii) The structure of the network is predefined since the dependency is modeled top-down from each parent to each of its child nodes, so the network would contain a single linear path for the random variables from each parent to each child node. In this paper we present a scalable probabilistic graphical model to overcome these limitations for massive hierarchical data. We believe the proposed model will lead to an easily-scalable, more readable, and expressive implementation for problems that require probabilistic-based solutions for massive amounts of hierarchical data. We successfully applied this model to solve two different challenging probabilistic-based problems on massive hierarchical data sets for different domains, namely, bioinformatics and latent semantic discovery over search logs.
△ Less
Submitted 19 August, 2014; v1 submitted 21 July, 2014;
originally announced July 2014.
