Showing 1–1 of 1 results for author: Boiardi, A

Top-Rank-Focused Adaptive Vote Collection for the Evaluation of Domain-Specific Semantic Models

Authors: Pierangelo Lombardo, Alessio Boiardi, Luca Colombo, Angelo Schiavone, Nicolò Tamagnone

Abstract: The growth of domain-specific applications of semantic models, boosted by the recent achievements of unsupervised embedding learning algorithms, demands domain-specific evaluation datasets. In many cases, content-based recommenders being a prime example, these models are required to rank words or texts according to their semantic relatedness to a given concept, with particular focus on top ranks.… ▽ More The growth of domain-specific applications of semantic models, boosted by the recent achievements of unsupervised embedding learning algorithms, demands domain-specific evaluation datasets. In many cases, content-based recommenders being a prime example, these models are required to rank words or texts according to their semantic relatedness to a given concept, with particular focus on top ranks. In this work, we give a threefold contribution to address these requirements: (i) we define a protocol for the construction, based on adaptive pairwise comparisons, of a relatedness-based evaluation dataset tailored on the available resources and optimized to be particularly accurate in top-rank evaluation; (ii) we define appropriate metrics, extensions of well-known ranking correlation coefficients, to evaluate a semantic model via the aforementioned dataset by taking into account the greater significance of top ranks. Finally, (iii) we define a stochastic transitivity model to simulate semantic-driven pairwise comparisons, which confirms the effectiveness of the proposed dataset construction protocol. △ Less

Submitted 9 October, 2020; originally announced October 2020.

Comments: This is a pre-print of an article published in the proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)

MSC Class: 68T50 (Primary); 62P99; 60G15 (Secondary) ACM Class: I.2.7; G.3; I.6.3; I.2.1; J.4

Journal ref: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) (pp. 3081-3093)