-
Annotating Scientific Uncertainty: A comprehensive model using linguistic patterns and comparison with existing approaches
Authors:
Panggih Kusuma Ningrum,
Philipp Mayr,
Nina Smirnova,
Iana Atanassova
Abstract:
UnScientify, a system designed to detect scientific uncertainty in scholarly full text. The system utilizes a weakly supervised technique to identify verbally expressed uncertainty in scientific texts and their authorial references. The core methodology of UnScientify is based on a multi-faceted pipeline that integrates span pattern matching, complex sentence analysis and author reference checking…
▽ More
UnScientify, a system designed to detect scientific uncertainty in scholarly full text. The system utilizes a weakly supervised technique to identify verbally expressed uncertainty in scientific texts and their authorial references. The core methodology of UnScientify is based on a multi-faceted pipeline that integrates span pattern matching, complex sentence analysis and author reference checking. This approach streamlines the labeling and annotation processes essential for identifying scientific uncertainty, covering a variety of uncertainty expression types to support diverse applications including information retrieval, text mining and scientific document processing. The evaluation results highlight the trade-offs between modern large language models (LLMs) and the UnScientify system. UnScientify, which employs more traditional techniques, achieved superior performance in the scientific uncertainty detection task, attaining an accuracy score of 0.808. This finding underscores the continued relevance and efficiency of UnScientify's simple rule-based and pattern matching strategy for this specific application. The results demonstrate that in scenarios where resource efficiency, interpretability, and domain-specific adaptability are critical, traditional methods can still offer significant advantages.
△ Less
Submitted 14 March, 2025;
originally announced March 2025.
-
Utilizing Large Language Models for Named Entity Recognition in Traditional Chinese Medicine against COVID-19 Literature: Comparative Study
Authors:
Xu Tong,
Nina Smirnova,
Sharmila Upadhyaya,
Ran Yu,
Jack H. Culbert,
Chao Sun,
Wolfgang Otto,
Philipp Mayr
Abstract:
Objective: To explore and compare the performance of ChatGPT and other state-of-the-art LLMs on domain-specific NER tasks covering different entity types and domains in TCM against COVID-19 literature. Methods: We established a dataset of 389 articles on TCM against COVID-19, and manually annotated 48 of them with 6 types of entities belonging to 3 domains as the ground truth, against which the NE…
▽ More
Objective: To explore and compare the performance of ChatGPT and other state-of-the-art LLMs on domain-specific NER tasks covering different entity types and domains in TCM against COVID-19 literature. Methods: We established a dataset of 389 articles on TCM against COVID-19, and manually annotated 48 of them with 6 types of entities belonging to 3 domains as the ground truth, against which the NER performance of LLMs can be assessed. We then performed NER tasks for the 6 entity types using ChatGPT (GPT-3.5 and GPT-4) and 4 state-of-the-art BERT-based question-answering (QA) models (RoBERTa, MiniLM, PubMedBERT and SciBERT) without prior training on the specific task. A domain fine-tuned model (GSAP-NER) was also applied for a comprehensive comparison. Results: The overall performance of LLMs varied significantly in exact match and fuzzy match. In the fuzzy match, ChatGPT surpassed BERT-based QA models in 5 out of 6 tasks, while in exact match, BERT-based QA models outperformed ChatGPT in 5 out of 6 tasks but with a smaller F-1 difference. GPT-4 showed a significant advantage over other models in fuzzy match, especially on the entity type of TCM formula and the Chinese patent drug (TFD) and ingredient (IG). Although GPT-4 outperformed BERT-based models on entity type of herb, target, and research method, none of the F-1 scores exceeded 0.5. GSAP-NER, outperformed GPT-4 in terms of F-1 by a slight margin on RM. ChatGPT achieved considerably higher recalls than precisions, particularly in the fuzzy match. Conclusions: The NER performance of LLMs is highly dependent on the entity type, and their performance varies across application scenarios. ChatGPT could be a good choice for scenarios where high recall is favored. However, for knowledge acquisition in rigorous scenarios, neither ChatGPT nor BERT-based QA models are off-the-shelf tools for professional practitioners.
△ Less
Submitted 24 August, 2024;
originally announced August 2024.
-
Automatically detecting scientific political science texts from a large general document index
Authors:
Nina Smirnova
Abstract:
This technical report outlines the filtering approach applied to the collection of the Bielefeld Academic Search Engine (BASE) data to extract articles from the political science domain. We combined hard and soft filters to address entries with different available metadata, e.g. title, abstract or keywords. The hard filter is a weighted keyword-based approach. The soft filter uses a multilingual B…
▽ More
This technical report outlines the filtering approach applied to the collection of the Bielefeld Academic Search Engine (BASE) data to extract articles from the political science domain. We combined hard and soft filters to address entries with different available metadata, e.g. title, abstract or keywords. The hard filter is a weighted keyword-based approach. The soft filter uses a multilingual BERT-based classification model, trained to detect scientific articles from the political science domain. We evaluated both approaches using an annotated dataset, consisting of scientific articles from different scientific domains. The weighted keyword-based approach achieved the highest total accuracy of 0.88. The multilingual BERT-based classification model was fine-tuned using a dataset of 14,178 abstracts from scientific articles and reached the highest total accuracy of 0.98. The proposed filtering approach can be applied for filtering metadata from other scientific domains and therefore improve the overview of the domain-related literature and facilitate efficiency in research.
△ Less
Submitted 24 June, 2024; v1 submitted 5 June, 2024;
originally announced June 2024.
-
The rise of Indo-German collaborative research: 1990-2022
Authors:
Aasif Ahmad Mir,
Nina Smirnova,
Jeyshankar Ramalingam,
Philipp Mayr
Abstract:
The study aims to highlight the growth and development of Indo-German collaborative research over the past three decades. Moreover, this study encompasses an in-depth examination of funding acknowledgements to gain valuable insights into the financial support that underpins these collaborative endeavors. Together with this paper, we provide an openly accessible dataset of Indo-German research arti…
▽ More
The study aims to highlight the growth and development of Indo-German collaborative research over the past three decades. Moreover, this study encompasses an in-depth examination of funding acknowledgements to gain valuable insights into the financial support that underpins these collaborative endeavors. Together with this paper, we provide an openly accessible dataset of Indo-German research articles for further and reproducible research activities (the "Indo-German Literature Dataset"). The data were retrieved from the Web of Science (WoS) database from the year 1990 till the 30th of November 2022. A total of 36,999 records were retrieved against the employed query. Acknowledged entities were extracted using a NER model specifically trained for this task. Interrelations between the extracted entities and scientific domains, lengths of acknowledgement texts, number of authors and affiliations, number of citations, and gender of the first author, as well as collaboration patterns between Indian and German funders were examined. The study brings to light that Physics, Chemistry, Materials Science, Astronomy and Astrophysics, and Engineering prominently dominate the Indo-German collaborative research. The United States, followed by England and France, are the most active collaborators in Indian and German research. Additionally, relations between entity, entity type, and scientific domain, were discovered. The study highlights a deeper understanding of the composition of the Indo-German collaborative research landscape of the last 30 years and its significance in advancing scientific knowledge and fostering international partnerships. Furthermore, we provide an open version of the original WoS dataset. The Indo-German Literature Dataset consists of 22,844 articles from OpenAlex and is available for related studies like literature studies and Scientometrics.
△ Less
Submitted 26 April, 2024;
originally announced April 2024.
-
Technical Report: Incorporating Blogs in Pollux
Authors:
Tobias Holtdirk,
Nina Smirnova
Abstract:
This technical report describes the incorporation of political blogs into Pollux, the Specialised Information Service (FID) for Political Science in Germany. Considering the widespread use of political blogs in political science research, we decided to include them in the Pollux search system to enhance the available information infrastructure. We describe the crawling and analyzing of the blogs a…
▽ More
This technical report describes the incorporation of political blogs into Pollux, the Specialised Information Service (FID) for Political Science in Germany. Considering the widespread use of political blogs in political science research, we decided to include them in the Pollux search system to enhance the available information infrastructure. We describe the crawling and analyzing of the blogs and the pipeline that integrates them into the Pollux system. To demonstrate the content of the incorporated blogs, we also provide a visualization of the topics covered by the blog posts during the first three months following integration.
△ Less
Submitted 26 March, 2024;
originally announced March 2024.
-
The Emergence of Preprints: Comparing Publishing Behaviour in the Global South and the Global North
Authors:
Kristin Biesenbender,
Nina Smirnova,
Philipp Mayr,
Isabella Peters
Abstract:
Purpose: The recent proliferation of preprints could be a way for researchers worldwide to increase the availability and visibility of their research findings. Against the background of rising publication costs caused by the increasing prevalence of article processing fees, the search for other ways to publish research results besides traditional journal publication may increase. This could be esp…
▽ More
Purpose: The recent proliferation of preprints could be a way for researchers worldwide to increase the availability and visibility of their research findings. Against the background of rising publication costs caused by the increasing prevalence of article processing fees, the search for other ways to publish research results besides traditional journal publication may increase. This could be especially true for lower-income countries. Design/methodology/approach: Therefore, we are interested in the experiences and attitudes towards posting and using preprints in the Global South as opposed to the Global North. To explore whether motivations and concerns about posting preprints differ, we adopted a mixed-methods approach, combining a quantitative survey of researchers with focus group interviews. Findings: We found that respondents from the Global South were more likely to agree to adhere to policies and to emphasise that mandates could change publishing behaviour towards open access. They were also more likely to agree posting preprints has a positive impact. Respondents from the Global South and the Global North emphasised the importance of peer-reviewed research for career advancement. Originality: The study has identified a wide range of experiences with and attitudes towards posting preprints among researchers in the Global South and the Global North. To our knowledge, this has hardly been studied before, which is also because preprints only have emerged lately in many disciplines and countries.
△ Less
Submitted 8 August, 2023;
originally announced August 2023.
-
Embedding Models for Supervised Automatic Extraction and Classification of Named Entities in Scientific Acknowledgements
Authors:
Nina Smirnova,
Philipp Mayr
Abstract:
Acknowledgments in scientific papers may give an insight into aspects of the scientific community, such as reward systems, collaboration patterns, and hidden research trends. The aim of the paper is to evaluate the performance of different embedding models for the task of automatic extraction and classification of acknowledged entities from the acknowledgment text in scientific papers. We trained…
▽ More
Acknowledgments in scientific papers may give an insight into aspects of the scientific community, such as reward systems, collaboration patterns, and hidden research trends. The aim of the paper is to evaluate the performance of different embedding models for the task of automatic extraction and classification of acknowledged entities from the acknowledgment text in scientific papers. We trained and implemented a named entity recognition (NER) task using the Flair NLP framework. The training was conducted using three default Flair NER models with four differently-sized corpora and different versions of the Flair NLP framework. The Flair Embeddings model trained on the medium corpus with the latest FLAIR version showed the best accuracy of 0.79. Expanding the size of a training corpus from very small to medium size massively increased the accuracy of all training algorithms, but further expansion of the training corpus did not bring further improvement. Moreover, the performance of the model slightly deteriorated. Our model is able to recognize six entity types: funding agency, grant number, individuals, university, corporation, and miscellaneous. The model works more precisely for some entity types than for others; thus, individuals and grant numbers showed a very good F1-Score over 0.9. Most of the previous works on acknowledgment analysis were limited by the manual evaluation of data and therefore by the amount of processed data. This model can be applied for the comprehensive analysis of acknowledgment texts and may potentially make a great contribution to the field of automated acknowledgment analysis.
△ Less
Submitted 25 July, 2023;
originally announced July 2023.
-
A Comprehensive Analysis of Acknowledgement Texts in Web of Science: a case study on four scientific domains
Authors:
Nina Smirnova,
Philipp Mayr
Abstract:
Analysis of acknowledgments is particularly interesting as acknowledgments may give information not only about funding, but they are also able to reveal hidden contributions to authorship and the researcher's collaboration patterns, context in which research was conducted, and specific aspects of the academic work. The focus of the present research is the analysis of a large sample of acknowledgem…
▽ More
Analysis of acknowledgments is particularly interesting as acknowledgments may give information not only about funding, but they are also able to reveal hidden contributions to authorship and the researcher's collaboration patterns, context in which research was conducted, and specific aspects of the academic work. The focus of the present research is the analysis of a large sample of acknowledgement texts indexed in the Web of Science (WoS) Core Collection. Record types 'article' and 'review' from four different scientific domains, namely social sciences, economics, oceanography and computer science, published from 2014 to 2019 in a scientific journal in English were considered. Six types of acknowledged entities, i.e., funding agency, grant number, individuals, university, corporation and miscellaneous, were extracted from the acknowledgement texts using a Named Entity Recognition (NER) tagger and subsequently examined. A general analysis of the acknowledgement texts showed that indexing of funding information in WoS is incomplete. The analysis of the automatically extracted entities revealed differences and distinct patterns in the distribution of acknowledged entities of different types between different scientific domains. A strong association was found between acknowledged entity and scientific domain and acknowledged entity and entity type. Only negligible correlation was found between the number of citations and the number of acknowledged entities. Generally, the number of words in the acknowledgement texts positively correlates with the number of acknowledged funding organizations, universities, individuals and miscellaneous entities. At the same time, acknowledgement texts with the larger number of sentences have more acknowledged individuals and miscellaneous categories.
△ Less
Submitted 18 October, 2022;
originally announced October 2022.
-
Evaluation of Embedding Models for Automatic Extraction and Classification of Acknowledged Entities in Scientific Documents
Authors:
Nina Smirnova,
Philipp Mayr
Abstract:
Acknowledgments in scientific papers may give an insight into aspects of the scientific community, such as reward systems, collaboration patterns, and hidden research trends. The aim of the paper is to evaluate the performance of different embedding models for the task of automatic extraction and classification of acknowledged entities from the acknowledgment text in scientific papers. We trained…
▽ More
Acknowledgments in scientific papers may give an insight into aspects of the scientific community, such as reward systems, collaboration patterns, and hidden research trends. The aim of the paper is to evaluate the performance of different embedding models for the task of automatic extraction and classification of acknowledged entities from the acknowledgment text in scientific papers. We trained and implemented a named entity recognition (NER) task using the Flair NLP-framework. The training was conducted using three default Flair NER models with two differently-sized corpora. The Flair Embeddings model trained on the larger training corpus showed the best accuracy of 0.77. Our model is able to recognize six entity types: funding agency, grant number, individuals, university, corporation and miscellaneous. The model works more precise for some entity types than the others, thus, individuals and grant numbers showed very good F1-Score over 0.9. Most of the previous works on acknowledgement analysis were limited by the manual evaluation of data and therefore by the amount of processed data. This model can be applied for the comprehensive analysis of the acknowledgement texts and may potentially make a great contribution to the field of automated acknowledgement analysis.
△ Less
Submitted 22 June, 2022;
originally announced June 2022.
-
Interactive Duplicate Search in Software Documentation
Authors:
D. V. Luciv,
D. V. Koznov,
A. A. Shelikhovskii,
K. Yu. Romanovsky,
G. A. Chernishev,
A. N. Terekhov,
D. A. Grigoriev,
A. N. Smirnova,
D. V. Borovkov,
A. I. Vasenina
Abstract:
Various software features such as classes, methods, requirements, and tests often have similar functionality. This can lead to emergence of duplicates in their descriptive documentation. Uncontrolled duplicates created via copy/paste hinder the process of documentation maintenance. Therefore, the task of duplicate detection in software documentation is of importance. Solving it makes planned reuse…
▽ More
Various software features such as classes, methods, requirements, and tests often have similar functionality. This can lead to emergence of duplicates in their descriptive documentation. Uncontrolled duplicates created via copy/paste hinder the process of documentation maintenance. Therefore, the task of duplicate detection in software documentation is of importance. Solving it makes planned reuse possible, as well as creating and using templates for unification and automatic generation of documentation. In this paper, we present an interactive process for duplicate detection that involves the user in order to conduct meaningful search. It includes a new formal definition of a near duplicate, a pattern-based, and the proof of its completeness. Moreover, we demonstrate the results of experimenting on a collection of documents of several industrial projects.
△ Less
Submitted 22 August, 2019;
originally announced August 2019.
