-
Relevance Judgment Convergence Degree -- A Measure of Inconsistency among Assessors for Information Retrieval
Authors:
Dengya Zhu,
Shastri L Nimmagadda,
Kok Wai Wong,
Torsten Reiners
Abstract:
Relevance judgment of human assessors is inherently subjective and dynamic when evaluation datasets are created for Information Retrieval (IR) systems. However, a small group of experts' relevance judgment results are usually taken as ground truth to "objectively" evaluate the performance of the IR systems. Recent trends intend to employ a group of judges, such as outsourcing, to alleviate the pot…
▽ More
Relevance judgment of human assessors is inherently subjective and dynamic when evaluation datasets are created for Information Retrieval (IR) systems. However, a small group of experts' relevance judgment results are usually taken as ground truth to "objectively" evaluate the performance of the IR systems. Recent trends intend to employ a group of judges, such as outsourcing, to alleviate the potentially biased judgment results stemmed from using only a single expert's judgment. Nevertheless, different judges may have different opinions and may not agree with each other, and the inconsistency in human relevance judgment may affect the IR system evaluation results. In this research, we introduce a Relevance Judgment Convergence Degree (RJCD) to measure the quality of queries in the evaluation datasets. Experimental results reveal a strong correlation coefficient between the proposed RJCD score and the performance differences between the two IR systems.
△ Less
Submitted 8 August, 2022;
originally announced August 2022.
-
An Integrated Search Framework for Leveraging the Knowledge-Based Web Ecosystem
Authors:
Dengya Zhu,
Shastri Lakshman Nimmagadda,
Torsten Reiners,
Amit Rudra
Abstract:
The explosion of information constrains the judgement of search terms associated with Knowledge-Based Web Ecosystem (KBWE), making the retrieval of relevant information and its knowledge management challenging. The existing information retrieval (IR) tools and their fusion in a framework need attention, in which search results can effectively be managed. In this article, we demonstrate the effecti…
▽ More
The explosion of information constrains the judgement of search terms associated with Knowledge-Based Web Ecosystem (KBWE), making the retrieval of relevant information and its knowledge management challenging. The existing information retrieval (IR) tools and their fusion in a framework need attention, in which search results can effectively be managed. In this article, we demonstrate the effective use of information retrieval services by a variety of users and agents in various KBWE scenarios. An innovative Integrated Search Framework (ISF) is proposed, which utilises crawling strategies, web search technologies and traditional database search methods. Besides, ISF offers comprehensive, dynamic, personalized, and organization-oriented information retrieval services, ranging from the Internet, extranet, intranet, to personal desktop. In this empirical research, experiments are carried out demonstrating the improvements in the search process, as discerned in the conceptual ISF. The experimental results show improved precision compared with other popular search engines.
△ Less
Submitted 9 December, 2020;
originally announced December 2020.
-
The Lived User Experience of Virtual Environments: Initial Steps of a Phenomenological Analysis in a Safety Training Setting
Authors:
Marko Teräs,
Hanna Teräs,
Torsten Reiners
Abstract:
Virtual environments (VEs) are making their way into various sectors of life to enhance and support human activity, including learning. VEs have been used in various contexts for training, and in many cases they are designed to model or simulate - as accurately and authentically as possible - a specific work context. In striving for authenticity, visual and representative realism tends to receive…
▽ More
Virtual environments (VEs) are making their way into various sectors of life to enhance and support human activity, including learning. VEs have been used in various contexts for training, and in many cases they are designed to model or simulate - as accurately and authentically as possible - a specific work context. In striving for authenticity, visual and representative realism tends to receive most of the development input, despite of several studies that challenge its importance. New training avenues have raised the importance of rigorous phenomenological descriptions for a deeper understanding of user experience in the actual context of use. This paper reports the preliminary steps in a phenomenological analysis of how employees working in actual hazardous settings experience virtual safety training environments. Such open-ended research project can reveal new aspects of user experience that can advice the development and evaluation of human-computer interaction in digital technology-enhanced training contexts.
△ Less
Submitted 4 June, 2016;
originally announced June 2016.
