-
Optimal Embedding Guided Negative Sample Generation for Knowledge Graph Link Prediction
Authors:
Makoto Takamoto,
Daniel Oñoro-Rubio,
Wiem Ben Rim,
Takashi Maruyama,
Bhushan Kotnis
Abstract:
Knowledge graph embedding (KGE) models encode the structural information of knowledge graphs to predicting new links. Effective training of these models requires distinguishing between positive and negative samples with high precision. Although prior research has shown that improving the quality of negative samples can significantly enhance model accuracy, identifying high-quality negative samples…
▽ More
Knowledge graph embedding (KGE) models encode the structural information of knowledge graphs to predicting new links. Effective training of these models requires distinguishing between positive and negative samples with high precision. Although prior research has shown that improving the quality of negative samples can significantly enhance model accuracy, identifying high-quality negative samples remains a challenging problem. This paper theoretically investigates the condition under which negative samples lead to optimal KG embedding and identifies a sufficient condition for an effective negative sample distribution. Based on this theoretical foundation, we propose \textbf{E}mbedding \textbf{MU}tation (\textsc{EMU}), a novel framework that \emph{generates} negative samples satisfying this condition, in contrast to conventional methods that focus on \emph{identifying} challenging negative samples within the training data. Importantly, the simplicity of \textsc{EMU} ensures seamless integration with existing KGE models and negative sampling methods. To evaluate its efficacy, we conducted comprehensive experiments across multiple datasets. The results consistently demonstrate significant improvements in link prediction performance across various KGE models and negative sampling methods. Notably, \textsc{EMU} enables performance improvements comparable to those achieved by models with embedding dimension five times larger. An implementation of the method and experiments are available at https://github.com/nec-research/EMU-KG.
△ Less
Submitted 4 April, 2025;
originally announced April 2025.
-
What Makes a Good Paraphrase: Do Automated Evaluations Work?
Authors:
Anna Moskvina,
Bhushan Kotnis,
Chris Catacata,
Michael Janz,
Nasrin Saef
Abstract:
Paraphrasing is the task of expressing an essential idea or meaning in different words. But how different should the words be in order to be considered an acceptable paraphrase? And can we exclusively use automated metrics to evaluate the quality of a paraphrase? We attempt to answer these questions by conducting experiments on a German data set and performing automatic and expert linguistic evalu…
▽ More
Paraphrasing is the task of expressing an essential idea or meaning in different words. But how different should the words be in order to be considered an acceptable paraphrase? And can we exclusively use automated metrics to evaluate the quality of a paraphrase? We attempt to answer these questions by conducting experiments on a German data set and performing automatic and expert linguistic evaluation.
△ Less
Submitted 27 July, 2023;
originally announced July 2023.
-
Human-Centric Research for NLP: Towards a Definition and Guiding Questions
Authors:
Bhushan Kotnis,
Kiril Gashteovski,
Julia Gastinger,
Giuseppe Serra,
Francesco Alesiani,
Timo Sztyler,
Ammar Shaker,
Na Gong,
Carolin Lawrence,
Zhao Xu
Abstract:
With Human-Centric Research (HCR) we can steer research activities so that the research outcome is beneficial for human stakeholders, such as end users. But what exactly makes research human-centric? We address this question by providing a working definition and define how a research pipeline can be split into different stages in which human-centric components can be added. Additionally, we discus…
▽ More
With Human-Centric Research (HCR) we can steer research activities so that the research outcome is beneficial for human stakeholders, such as end users. But what exactly makes research human-centric? We address this question by providing a working definition and define how a research pipeline can be split into different stages in which human-centric components can be added. Additionally, we discuss existing NLP with HCR components and define a series of guiding questions, which can serve as starting points for researchers interested in exploring human-centric research approaches. We hope that this work would inspire researchers to refine the proposed definition and to pose other questions that might be meaningful for achieving HCR.
△ Less
Submitted 10 July, 2022;
originally announced July 2022.
-
A Human-Centric Assessment Framework for AI
Authors:
Sascha Saralajew,
Ammar Shaker,
Zhao Xu,
Kiril Gashteovski,
Bhushan Kotnis,
Wiem Ben Rim,
Jürgen Quittek,
Carolin Lawrence
Abstract:
With the rise of AI systems in real-world applications comes the need for reliable and trustworthy AI. An essential aspect of this are explainable AI systems. However, there is no agreed standard on how explainable AI systems should be assessed. Inspired by the Turing test, we introduce a human-centric assessment framework where a leading domain expert accepts or rejects the solutions of an AI sys…
▽ More
With the rise of AI systems in real-world applications comes the need for reliable and trustworthy AI. An essential aspect of this are explainable AI systems. However, there is no agreed standard on how explainable AI systems should be assessed. Inspired by the Turing test, we introduce a human-centric assessment framework where a leading domain expert accepts or rejects the solutions of an AI system and another domain expert. By comparing the acceptance rates of provided solutions, we can assess how the AI system performs compared to the domain expert, and whether the AI system's explanations (if provided) are human-understandable. This setup -- comparable to the Turing test -- can serve as a framework for a wide range of human-centric AI system assessments. We demonstrate this by presenting two instantiations: (1) an assessment that measures the classification accuracy of a system with the option to incorporate label uncertainties; (2) an assessment where the usefulness of provided explanations is determined in a human-centric manner.
△ Less
Submitted 1 July, 2022; v1 submitted 25 May, 2022;
originally announced May 2022.
-
milIE: Modular & Iterative Multilingual Open Information Extraction
Authors:
Bhushan Kotnis,
Kiril Gashteovski,
Daniel Oñoro Rubio,
Vanesa Rodriguez-Tembras,
Ammar Shaker,
Makoto Takamoto,
Mathias Niepert,
Carolin Lawrence
Abstract:
Open Information Extraction (OpenIE) is the task of extracting (subject, predicate, object) triples from natural language sentences. Current OpenIE systems extract all triple slots independently. In contrast, we explore the hypothesis that it may be beneficial to extract triple slots iteratively: first extract easy slots, followed by the difficult ones by conditioning on the easy slots, and theref…
▽ More
Open Information Extraction (OpenIE) is the task of extracting (subject, predicate, object) triples from natural language sentences. Current OpenIE systems extract all triple slots independently. In contrast, we explore the hypothesis that it may be beneficial to extract triple slots iteratively: first extract easy slots, followed by the difficult ones by conditioning on the easy slots, and therefore achieve a better overall extraction. Based on this hypothesis, we propose a neural OpenIE system, milIE, that operates in an iterative fashion. Due to the iterative nature, the system is also modular -- it is possible to seamlessly integrate rule based extraction systems with a neural end-to-end system, thereby allowing rule based systems to supply extraction slots which milIE can leverage for extracting the remaining slots. We confirm our hypothesis empirically: milIE outperforms SOTA systems on multiple languages ranging from Chinese to Arabic. Additionally, we are the first to provide an OpenIE test dataset for Arabic and Galician.
△ Less
Submitted 25 April, 2022; v1 submitted 15 October, 2021;
originally announced October 2021.
-
AnnIE: An Annotation Platform for Constructing Complete Open Information Extraction Benchmark
Authors:
Niklas Friedrich,
Kiril Gashteovski,
Mingying Yu,
Bhushan Kotnis,
Carolin Lawrence,
Mathias Niepert,
Goran Glavaš
Abstract:
Open Information Extraction (OIE) is the task of extracting facts from sentences in the form of relations and their corresponding arguments in schema-free manner. Intrinsic performance of OIE systems is difficult to measure due to the incompleteness of existing OIE benchmarks: the ground truth extractions do not group all acceptable surface realizations of the same fact that can be extracted from…
▽ More
Open Information Extraction (OIE) is the task of extracting facts from sentences in the form of relations and their corresponding arguments in schema-free manner. Intrinsic performance of OIE systems is difficult to measure due to the incompleteness of existing OIE benchmarks: the ground truth extractions do not group all acceptable surface realizations of the same fact that can be extracted from a sentence. To measure performance of OIE systems more realistically, it is necessary to manually annotate complete facts (i.e., clusters of all acceptable surface realizations of the same fact) from input sentences. We propose AnnIE: an interactive annotation platform that facilitates such challenging annotation tasks and supports creation of complete fact-oriented OIE evaluation benchmarks. AnnIE is modular and flexible in order to support different use case scenarios (i.e., benchmarks covering different types of facts). We use AnnIE to build two complete OIE benchmarks: one with verb-mediated facts and another with facts encompassing named entities. Finally, we evaluate several OIE systems on our complete benchmarks created with AnnIE. Our results suggest that existing incomplete benchmarks are overly lenient, and that OIE systems are not as robust as previously reported. We publicly release AnnIE under non-restrictive license.
△ Less
Submitted 13 April, 2022; v1 submitted 15 September, 2021;
originally announced September 2021.
-
BenchIE: A Framework for Multi-Faceted Fact-Based Open Information Extraction Evaluation
Authors:
Kiril Gashteovski,
Mingying Yu,
Bhushan Kotnis,
Carolin Lawrence,
Mathias Niepert,
Goran Glavaš
Abstract:
Intrinsic evaluations of OIE systems are carried out either manually -- with human evaluators judging the correctness of extractions -- or automatically, on standardized benchmarks. The latter, while much more cost-effective, is less reliable, primarily because of the incompleteness of the existing OIE benchmarks: the ground truth extractions do not include all acceptable variants of the same fact…
▽ More
Intrinsic evaluations of OIE systems are carried out either manually -- with human evaluators judging the correctness of extractions -- or automatically, on standardized benchmarks. The latter, while much more cost-effective, is less reliable, primarily because of the incompleteness of the existing OIE benchmarks: the ground truth extractions do not include all acceptable variants of the same fact, leading to unreliable assessment of the models' performance. Moreover, the existing OIE benchmarks are available for English only. In this work, we introduce BenchIE: a benchmark and evaluation framework for comprehensive evaluation of OIE systems for English, Chinese, and German. In contrast to existing OIE benchmarks, BenchIE is fact-based, i.e., it takes into account informational equivalence of extractions: our gold standard consists of fact synsets, clusters in which we exhaustively list all acceptable surface forms of the same fact. Moreover, having in mind common downstream applications for OIE, we make BenchIE multi-faceted; i.e., we create benchmark variants that focus on different facets of OIE evaluation, e.g., compactness or minimality of extractions. We benchmark several state-of-the-art OIE systems using BenchIE and demonstrate that these systems are significantly less effective than indicated by existing OIE benchmarks. We make BenchIE (data and evaluation code) publicly available on https://github.com/gkiril/benchie.
△ Less
Submitted 13 April, 2022; v1 submitted 14 September, 2021;
originally announced September 2021.
-
Answering Complex Queries in Knowledge Graphs with Bidirectional Sequence Encoders
Authors:
Bhushan Kotnis,
Carolin Lawrence,
Mathias Niepert
Abstract:
Representation learning for knowledge graphs (KGs) has focused on the problem of answering simple link prediction queries. In this work we address the more ambitious challenge of predicting the answers of conjunctive queries with multiple missing entities. We propose Bi-Directional Query Embedding (BIQE), a method that embeds conjunctive queries with models based on bi-directional attention mechan…
▽ More
Representation learning for knowledge graphs (KGs) has focused on the problem of answering simple link prediction queries. In this work we address the more ambitious challenge of predicting the answers of conjunctive queries with multiple missing entities. We propose Bi-Directional Query Embedding (BIQE), a method that embeds conjunctive queries with models based on bi-directional attention mechanisms. Contrary to prior work, bidirectional self-attention can capture interactions among all the elements of a query graph. We introduce a new dataset for predicting the answer of conjunctive query and conduct experiments that show BIQE significantly outperforming state of the art baselines.
△ Less
Submitted 4 February, 2021; v1 submitted 6 April, 2020;
originally announced April 2020.
-
Attending to Future Tokens For Bidirectional Sequence Generation
Authors:
Carolin Lawrence,
Bhushan Kotnis,
Mathias Niepert
Abstract:
Neural sequence generation is typically performed token-by-token and left-to-right. Whenever a token is generated only previously produced tokens are taken into consideration. In contrast, for problems such as sequence classification, bidirectional attention, which takes both past and future tokens into consideration, has been shown to perform much better. We propose to make the sequence generatio…
▽ More
Neural sequence generation is typically performed token-by-token and left-to-right. Whenever a token is generated only previously produced tokens are taken into consideration. In contrast, for problems such as sequence classification, bidirectional attention, which takes both past and future tokens into consideration, has been shown to perform much better. We propose to make the sequence generation process bidirectional by employing special placeholder tokens. Treated as a node in a fully connected graph, a placeholder token can take past and future tokens into consideration when generating the actual output token. We verify the effectiveness of our approach experimentally on two conversational tasks where the proposed bidirectional model outperforms competitive baselines by a large margin.
△ Less
Submitted 17 September, 2019; v1 submitted 16 August, 2019;
originally announced August 2019.
-
Analysis of the Impact of Negative Sampling on Link Prediction in Knowledge Graphs
Authors:
Bhushan Kotnis,
Vivi Nastase
Abstract:
Knowledge graphs are large, useful, but incomplete knowledge repositories. They encode knowledge through entities and relations which define each other through the connective structure of the graph. This has inspired methods for the joint embedding of entities and relations in continuous low-dimensional vector spaces, that can be used to induce new edges in the graph, i.e., link prediction in know…
▽ More
Knowledge graphs are large, useful, but incomplete knowledge repositories. They encode knowledge through entities and relations which define each other through the connective structure of the graph. This has inspired methods for the joint embedding of entities and relations in continuous low-dimensional vector spaces, that can be used to induce new edges in the graph, i.e., link prediction in knowledge graphs. Learning these representations relies on contrasting positive instances with negative ones. Knowledge graphs include only positive relation instances, leaving the door open for a variety of methods for selecting negative examples. In this paper we present an empirical study on the impact of negative sampling on the learned embeddings, assessed through the task of link prediction. We use state-of-the-art knowledge graph embeddings -- \rescal , TransE, DistMult and ComplEX -- and evaluate on benchmark datasets -- FB15k and WN18. We compare well known methods for negative sampling and additionally propose embedding based sampling methods. We note a marked difference in the impact of these sampling methods on the two datasets, with the "traditional" corrupting positives method leading to best results on WN18, while embedding based methods benefiting the task on FB15k.
△ Less
Submitted 2 March, 2018; v1 submitted 22 August, 2017;
originally announced August 2017.
-
Learning Knowledge Graph Embeddings with Type Regularizer
Authors:
Bhushan Kotnis,
Vivi Nastase
Abstract:
Learning relations based on evidence from knowledge bases relies on processing the available relation instances. Many relations, however, have clear domain and range, which we hypothesize could help learn a better, more generalizing, model. We include such information in the RESCAL model in the form of a regularization factor added to the loss function that takes into account the types (categories…
▽ More
Learning relations based on evidence from knowledge bases relies on processing the available relation instances. Many relations, however, have clear domain and range, which we hypothesize could help learn a better, more generalizing, model. We include such information in the RESCAL model in the form of a regularization factor added to the loss function that takes into account the types (categories) of the entities that appear as arguments to relations in the knowledge base. We note increased performance compared to the baseline model in terms of mean reciprocal rank and hits@N, N = 1, 3, 10. Furthermore, we discover scenarios that significantly impact the effectiveness of the type regularizer.
△ Less
Submitted 2 March, 2018; v1 submitted 28 June, 2017;
originally announced June 2017.
-
Cost Effective Campaigning in Social Networks
Authors:
Bhushan Kotnis,
Joy Kuri
Abstract:
Campaigners are increasingly using online social networking platforms for promoting products, ideas and information. A popular method of promoting a product or even an idea is incentivizing individuals to evangelize the idea vigorously by providing them with referral rewards in the form of discounts, cash backs, or social recognition. Due to budget constraints on scarce resources such as money and…
▽ More
Campaigners are increasingly using online social networking platforms for promoting products, ideas and information. A popular method of promoting a product or even an idea is incentivizing individuals to evangelize the idea vigorously by providing them with referral rewards in the form of discounts, cash backs, or social recognition. Due to budget constraints on scarce resources such as money and manpower, it may not be possible to provide incentives for the entire population, and hence incentives need to be allocated judiciously to appropriate individuals for ensuring the highest possible outreach size. We aim to do the same by formulating and solving an optimization problem using percolation theory. In particular, we compute the set of individuals that are provided incentives for minimizing the expected cost while ensuring a given outreach size. We also solve the problem of computing the set of individuals to be incentivized for maximizing the outreach size for given cost budget. The optimization problem turns out to be non trivial; it involves quantities that need to be computed by numerically solving a fixed point equation. Our primary contribution is, that for a fairly general cost structure, we show that the optimization problems can be solved by solving a simple linear program. We believe that our approach of using percolation theory to formulate an optimization problem is the first of its kind.
△ Less
Submitted 16 June, 2016;
originally announced June 2016.
-
Incentivized Campaigning in Social Networks
Authors:
Bhushan Kotnis,
Albert Sunny,
Joy Kuri
Abstract:
Campaigners, advertisers and activists are increasingly turning to social recommendation mechanisms, provided by social media, for promoting their products, services, brands and even ideas. However, many times, such social network based campaigns perform poorly in practice because the intensity of the recommendations drastically reduces beyond a few hops from the source. A natural strategy for mai…
▽ More
Campaigners, advertisers and activists are increasingly turning to social recommendation mechanisms, provided by social media, for promoting their products, services, brands and even ideas. However, many times, such social network based campaigns perform poorly in practice because the intensity of the recommendations drastically reduces beyond a few hops from the source. A natural strategy for maintaining the intensity is to provide incentives. In this paper, we address the problem of minimizing the cost incurred by the campaigner for incentivizing a fraction of individuals in the social network, while ensuring that the campaign message reaches a given expected fraction of individuals. We also address the dual problem of maximizing the campaign penetration for a resource constrained campaigner. To help us understand and solve the above mentioned problems, we use percolation theory to formally state them as optimization problems. These problems are not amenable to traditional approaches because of a fixed point equation that needs to be solved numerically. However, we use results from reliability theory to establish some key properties of the fixed point, which in turn enables us to solve these problems using algorithms that are linearithmic in maximum node degree. Furthermore, we evaluate the efficacy of the analytical solution by performing simulations on real world networks.
△ Less
Submitted 16 June, 2016;
originally announced June 2016.
-
Game Theoretic Analysis of Tree Based Referrals for Crowd Sensing Social Systems with Passive Rewards
Authors:
Kundan Kandhway,
Bhushan Kotnis
Abstract:
Participatory crowd sensing social systems rely on the participation of large number of individuals. Since humans are strategic by nature, effective incentive mechanisms are needed to encourage participation. A popular mechanism to recruit individuals is through referrals and passive incentives such as geometric incentive mechanisms used by the winning team in the 2009 DARPA Network Challenge and…
▽ More
Participatory crowd sensing social systems rely on the participation of large number of individuals. Since humans are strategic by nature, effective incentive mechanisms are needed to encourage participation. A popular mechanism to recruit individuals is through referrals and passive incentives such as geometric incentive mechanisms used by the winning team in the 2009 DARPA Network Challenge and in multi level marketing schemes. The effect of such recruitment schemes on the effort put in by recruited strategic individuals is not clear. This paper attempts to fill this gap. Given a referral tree and the direct and passive reward mechanism, we formulate a network game where agents compete for finishing crowd sensing tasks. We characterize the Nash equilibrium efforts put in by the agents and derive closed form expressions for the same. We discover free riding behavior among nodes who obtain large passive rewards. This work has implications on designing effective recruitment mechanisms for crowd sourced tasks. For example, usage of geometric incentive mechanisms to recruit large number of individuals may not result in proportionate effort because of free riding.
△ Less
Submitted 27 January, 2016;
originally announced January 2016.
-
Cost Effective Rumor Containment in Social Networks
Authors:
Bhushan Kotnis,
Joy Kuri
Abstract:
The spread of rumors through social media and online social networks can not only disrupt the daily lives of citizens but also result in loss of life and property. A rumor spreads when individuals, who are unable decide the authenticity of the information, mistake the rumor as genuine information and pass it on to their acquaintances. We propose a solution where a set of individuals (based on thei…
▽ More
The spread of rumors through social media and online social networks can not only disrupt the daily lives of citizens but also result in loss of life and property. A rumor spreads when individuals, who are unable decide the authenticity of the information, mistake the rumor as genuine information and pass it on to their acquaintances. We propose a solution where a set of individuals (based on their degree) in the social network are trained and provided resources to help them distinguish a rumor from genuine information. By formulating an optimization problem we calculate the optimum set of individuals, who must undergo training, and the quality of training that minimizes the expected training cost and ensures an upper bound on the size of the rumor outbreak. Our primary contribution is that although the optimization problem turns out to be non convex, we show that the problem is equivalent to solving a set of linear programs. This result also allows us to solve the problem of minimizing the size of rumor outbreak for a given cost budget. The optimum solution displays an interesting pattern which can be implemented as a heuristic. These results can prove to be very useful for social planners and law enforcement agencies for preventing dangerous rumors and misinformation epidemics.
△ Less
Submitted 27 April, 2014; v1 submitted 25 March, 2014;
originally announced March 2014.
