-
A Blueprint of IR Evaluation Integrating Task and User Characteristics: Test Collection and Evaluation Metrics
Authors:
Kal Jarvelin,
Eero Sormunen
Abstract:
Relevance is generally understood as a multi-level and multi-dimensional relationship between an information need and an information object. However, traditional IR evaluation metrics naively assume mono-dimensionality. We ask: How to deal with multidimensional and graded relevance assessments in IR evaluation? Moreover, search result evaluation metrics neglect document overlaps and naively assume…
▽ More
Relevance is generally understood as a multi-level and multi-dimensional relationship between an information need and an information object. However, traditional IR evaluation metrics naively assume mono-dimensionality. We ask: How to deal with multidimensional and graded relevance assessments in IR evaluation? Moreover, search result evaluation metrics neglect document overlaps and naively assume gains piling up as the searcher examines the ranked list into greater length. Consequently, we examine: How to deal with document overlap in IR evaluation? The usability of a document for a person-in-need also depends on document usability attributes beyond relevance. Therefore, we ask: How to deal with usability attributes, and how to combine this with multidimensional relevance assessments in IR evaluation? Finally, we ask how to define a formal model, which deals with multidimensional graded relevance assessments, document overlaps, and document usability attributes in a coherent framework serving IR evaluation?
△ Less
Submitted 1 May, 2023;
originally announced May 2023.
-
Characteristics of LIS Research Articles Affecting Their Citation Impact
Authors:
Kalervo Jarvelin,
Yu-Wei Chang,
Pertti Vakkari
Abstract:
The paper analyses the citation impact of Library and Information Science, LIS for short, research articles published in 31 leading international LIS journals in 2015. The main research question is: to what degree do authors' disciplinary composition in association with other content characteristics of LIS articles affect their citation impact? The impact is analysed in terms of the number of cita…
▽ More
The paper analyses the citation impact of Library and Information Science, LIS for short, research articles published in 31 leading international LIS journals in 2015. The main research question is: to what degree do authors' disciplinary composition in association with other content characteristics of LIS articles affect their citation impact? The impact is analysed in terms of the number of citations received and their authority, using outlier normalization and subfield normalization. The article characteristics analysed using quantitative content analysis include topic, methodology, type of contribution, and the disciplinary composition of their author teams. The citations received by the articles are traced from 2015 to May 2021. Citing document authority is measured by the citations they had received up to May 2021. The overall finding was that authors' disciplinary composition is significantly associated with citation scores. The differences in citation scores between disciplinary compositions appeared typically within information retrieval and scientific communication. In both topics LIS and computer science jointly received significantly higher citation scores than many disciplines like LIS alone or humanities in information retrieval, or natural sciences, medicine, or social sciences alone in scientific communication. The paper is original in allowing joint analysis of content, authorship composition, and impact.
△ Less
Submitted 10 April, 2023;
originally announced April 2023.
-
Adaptive Distributional Extensions to DFR Ranking
Authors:
Casper Petersen,
Jakob Grue Simonsen,
Kalervo Jarvelin,
Christina Lioma
Abstract:
Divergence From Randomness (DFR) ranking models assume that informative terms are distributed in a corpus differently than non-informative terms. Different statistical models (e.g. Poisson, geometric) are used to model the distribution of non-informative terms, producing different DFR models. An informative term is then detected by measuring the divergence of its distribution from the distribution…
▽ More
Divergence From Randomness (DFR) ranking models assume that informative terms are distributed in a corpus differently than non-informative terms. Different statistical models (e.g. Poisson, geometric) are used to model the distribution of non-informative terms, producing different DFR models. An informative term is then detected by measuring the divergence of its distribution from the distribution of non-informative terms. However, there is little empirical evidence that the distributions of non-informative terms used in DFR actually fit current datasets. Practically this risks providing a poor separation between informative and non-informative terms, thus compromising the discriminative power of the ranking model. We present a novel extension to DFR, which first detects the best-fitting distribution of non-informative terms in a collection, and then adapts the ranking computation to this best-fitting distribution. We call this model Adaptive Distributional Ranking (ADR) because it adapts the ranking to the statistics of the specific dataset being processed each time. Experiments on TREC data show ADR to outperform DFR models (and their extensions) and be comparable in performance to a query likelihood language model (LM).
△ Less
Submitted 4 September, 2016;
originally announced September 2016.
