-
An extensive analysis of the presence of altmetric data for Web of Science publications across subject fields and research topics
Authors:
Zhichao Fang,
Rodrigo Costas,
Wencan Tian,
Xianwen Wang,
Paul Wouters
Abstract:
Sufficient data presence is one of the key preconditions for applying metrics in practice. Based on both Altmetric.com data and Mendeley data collected up to 2019, this paper presents a state-of-the-art analysis of the presence of 12 kinds of altmetric events for nearly 12.3 million Web of Science publications published between 2012 and 2018. Results show that even though an upward trend of data p…
▽ More
Sufficient data presence is one of the key preconditions for applying metrics in practice. Based on both Altmetric.com data and Mendeley data collected up to 2019, this paper presents a state-of-the-art analysis of the presence of 12 kinds of altmetric events for nearly 12.3 million Web of Science publications published between 2012 and 2018. Results show that even though an upward trend of data presence can be observed over time, except for Mendeley readers and Twitter mentions, the overall presence of most altmetric data is still low. The majority of altmetric events go to publications in the fields of Biomedical and Health Sciences, Social Sciences and Humanities, and Life and Earth Sciences. As to research topics, the level of attention received by research topics varies across altmetric data, and specific altmetric data show different preferences for research topics, on the basis of which a framework for identifying hot research topics is proposed and applied to detect research topics with higher levels of attention garnered on certain altmetric data source. Twitter mentions and policy document citations were selected as two examples to identify hot research topics of interest of Twitter users and policy-makers, respectively, shedding light on the potential of altmetric data in monitoring research trends of specific social attention.
△ Less
Submitted 22 June, 2020;
originally announced June 2020.
-
Social media metrics for new research evaluation
Authors:
Paul Wouters,
Zohreh Zahedi,
Rodrigo Costas
Abstract:
This chapter approaches, both from a theoretical and practical perspective, the most important principles and conceptual frameworks that can be considered in the application of social media metrics for scientific evaluation. We propose conceptually valid uses for social media metrics in research evaluation. The chapter discusses frameworks and uses of these metrics as well as principles and recomm…
▽ More
This chapter approaches, both from a theoretical and practical perspective, the most important principles and conceptual frameworks that can be considered in the application of social media metrics for scientific evaluation. We propose conceptually valid uses for social media metrics in research evaluation. The chapter discusses frameworks and uses of these metrics as well as principles and recommendations for the consideration and application of current (and potentially new) metrics in research evaluation.
△ Less
Submitted 27 June, 2018;
originally announced June 2018.
-
Science and its significant other: Representing the humanities in bibliometric scholarship
Authors:
Thomas Franssen,
Paul Wouters
Abstract:
Bibliometrics offers a particular representation of science. Through bibliometric methods a bibliometrician will always highlight particular elements of publications, and through these elements operationalize particular representations of science, while obscuring other possible representations from view. Understanding bibliometrics as representation implies that a bibliometric analysis is always p…
▽ More
Bibliometrics offers a particular representation of science. Through bibliometric methods a bibliometrician will always highlight particular elements of publications, and through these elements operationalize particular representations of science, while obscuring other possible representations from view. Understanding bibliometrics as representation implies that a bibliometric analysis is always performative: a bibliometric analysis brings a particular representation of science into being that potentially influences the science system itself. In this review we analyze the ways the humanities have been represented throughout the history of bibliometrics, often in comparison to other scientific domains or to a general notion of the sciences. Our review discusses bibliometric scholarship between 1965 and 2016 that studies the humanities empirically. We distinguish between two periods of bibliometric scholarship. The first period, between 1965 and 1989, is characterized by a sociological theoretical framework, the development and use of the Price index, and small samples of journal publications as data sources. The second period, from the mid-1980s up until the present day, is characterized by a new hinterland, that of science policy and research evaluation, in which bibliometric methods become embedded.
△ Less
Submitted 11 October, 2017;
originally announced October 2017.
-
Mendeley readership as a filtering tool to identify highly cited publications
Authors:
Zohreh Zahedi,
Rodrigo Costas,
Paul Wouters
Abstract:
This study presents a large scale analysis of the distribution and presence of Mendeley readership scores over time and across disciplines. We study whether Mendeley readership scores (RS) can identify highly cited publications more effectively than journal citation scores (JCS). Web of Science (WoS) publications with DOIs published during the period 2004-2013 and across 5 major scientific fields…
▽ More
This study presents a large scale analysis of the distribution and presence of Mendeley readership scores over time and across disciplines. We study whether Mendeley readership scores (RS) can identify highly cited publications more effectively than journal citation scores (JCS). Web of Science (WoS) publications with DOIs published during the period 2004-2013 and across 5 major scientific fields have been analyzed. The main result of this study shows that readership scores are more effective (in terms of precision/recall values) than journal citation scores to identify highly cited publications across all fields of science and publication years. The findings also show that 86.5% of all the publications are covered by Mendeley and have at least one reader. Also the share of publications with Mendeley readership scores is increasing from 84% in 2004 to 89% in 2009, and decreasing from 88% in 2010 to 82% in 2013. However, it is noted that publications from 2010 onwards exhibit on average a higher density of readership vs. citation scores. This indicates that compared to citation scores, readership scores are more prevalent for recent publications and hence they could work as an early indicator of research impact. These findings highlight the potential and value of Mendeley as a tool for scientometric purposes and particularly as a relevant tool to identify highly cited publications.
△ Less
Submitted 21 March, 2017;
originally announced March 2017.
-
Professional and Citizen Bibliometrics: Complementarities and ambivalences in the development and use of indicators
Authors:
Loet Leydesdorff,
Paul Wouters,
Lutz Bornmann
Abstract:
Bibliometric indicators such as journal impact factors, h-indices, and total citation counts are algorithmic artifacts that can be used in research evaluation and management. These artifacts have no meaning by themselves, but receive their meaning from attributions in institutional practices. We distinguish four main stakeholders in these practices: (1) producers of bibliometric data and indicator…
▽ More
Bibliometric indicators such as journal impact factors, h-indices, and total citation counts are algorithmic artifacts that can be used in research evaluation and management. These artifacts have no meaning by themselves, but receive their meaning from attributions in institutional practices. We distinguish four main stakeholders in these practices: (1) producers of bibliometric data and indicators; (2) bibliometricians who develop and test indicators; (3) research managers who apply the indicators; and (4) the scientists being evaluated with potentially competing career interests. These different positions may lead to different and sometimes conflicting perspectives on the meaning and value of the indicators. The indicators can thus be considered as boundary objects which are socially constructed in translations among these perspectives. This paper proposes an analytical clarification by listing an informed set of (sometimes unsolved) problems in bibliometrics which can also shed light on the tension between simple but invalid indicators that are widely used (e.g., the h-index) and more sophisticated indicators that are not used or cannot be used in evaluation practices because they are not transparent for users, cannot be calculated, or are difficult to interpret.
△ Less
Submitted 23 September, 2016; v1 submitted 15 September, 2016;
originally announced September 2016.
-
The elephant in the room: The problem of quantifying productivity in evaluative scientometrics
Authors:
Ludo Waltman,
Nees Jan van Eck,
Martijn Visser,
Paul Wouters
Abstract:
In a critical and provocative paper, Abramo and D'Angelo claim that commonly used scientometric indicators such as the mean normalized citation score (MNCS) are completely inappropriate as indicators of scientific performance. Abramo and D'Angelo argue that scientific performance should be quantified using indicators that take into account the productivity of a research unit. We provide a response…
▽ More
In a critical and provocative paper, Abramo and D'Angelo claim that commonly used scientometric indicators such as the mean normalized citation score (MNCS) are completely inappropriate as indicators of scientific performance. Abramo and D'Angelo argue that scientific performance should be quantified using indicators that take into account the productivity of a research unit. We provide a response to Abramo and D'Angelo, indicating where we believe they raise important issues, but also pointing out where we believe their claims to be too extreme.
△ Less
Submitted 8 May, 2016;
originally announced May 2016.
-
How well developed are Altmetrics? Cross-disciplinary analysis of the presence of alternative metrics in scientific publications?
Authors:
Zohreh Zahedi,
Rodrigo Costas,
Paul Wouters
Abstract:
In this paper an analysis of the presence and possibilities of altmetrics for bibliometric and performance analysis is carried out. Using the web based tool Impact Story, we have collected metrics for 20,000 random publications from the Web of Science. We studied the presence and frequency of altmetrics in the set of publications, across fields, document types and also through the years. The main…
▽ More
In this paper an analysis of the presence and possibilities of altmetrics for bibliometric and performance analysis is carried out. Using the web based tool Impact Story, we have collected metrics for 20,000 random publications from the Web of Science. We studied the presence and frequency of altmetrics in the set of publications, across fields, document types and also through the years. The main result of the study is that less than 50% of the publications have some kind of altmetrics. The source that provides most metrics is Mendeley, with metrics on readerships for around 37% of all the publications studied. Other sources only provide marginal information. Possibilities and limitations of these indicators are discussed and future research lines are outlined. We also assessed the accuracy of the data retrieved through Impact Story by focusing on the analysis of the accuracy of data from Mendeley; in a follow up study, the accuracy and validity of other data sources not included here will be assessed.
△ Less
Submitted 8 July, 2015;
originally announced July 2015.
-
Do Mendeley readership counts help to filter highly cited WoS publications better than average citation impact of journals (JCS)?
Authors:
Zohreh Zahedi,
Rodrigo Costas,
Paul Wouters
Abstract:
In this study, the academic status of users of scientific publications in Mendeley is explored in order to analyse the usage pattern of Mendeley users in terms of subject fields, citation and readership impact. The main focus of this study is on studying the filtering capacity of Mendeley readership counts compared to journal citation scores in detecting highly cited WoS publications. Main finding…
▽ More
In this study, the academic status of users of scientific publications in Mendeley is explored in order to analyse the usage pattern of Mendeley users in terms of subject fields, citation and readership impact. The main focus of this study is on studying the filtering capacity of Mendeley readership counts compared to journal citation scores in detecting highly cited WoS publications. Main finding suggests a faster reception of Mendeley readerships as compared to citations across 5 major field of science. The higher correlations of scientific users with citations indicate the similarity between reading and citation behaviour among these users. It is confirmed that Mendeley readership counts filter highly cited publications (PPtop 10%) better than journal citation scores in all subject fields and by most of user types. This result reinforces the potential role that Mendeley readerships could play for informing scientific and alternative impacts.
△ Less
Submitted 8 July, 2015;
originally announced July 2015.
-
How well developed are altmetrics? A cross-disciplinary analysis of the presence of 'alternative metrics' in scientific publications
Authors:
Zohreh Zahedi,
Rodrigo Costas,
Paul Wouters
Abstract:
In this paper an analysis of the presence and possibilities of altmetrics for bibliometric and performance analysis is carried out. Using the web based tool Impact Story, we collected metrics for 20,000 random publications from the Web of Science. We studied both the presence and distribution of altmetrics in the set of publications, across fields, document types and over publication years, as wel…
▽ More
In this paper an analysis of the presence and possibilities of altmetrics for bibliometric and performance analysis is carried out. Using the web based tool Impact Story, we collected metrics for 20,000 random publications from the Web of Science. We studied both the presence and distribution of altmetrics in the set of publications, across fields, document types and over publication years, as well as the extent to which altmetrics correlate with citation indicators. The main result of the study is that the altmetrics source that provides the most metrics is Mendeley, with metrics on readerships for 62.6% of all the publications studied, other sources only provide marginal information. In terms of relation with citations, a moderate spearman correlation (r=0.49) has been found between Mendeley readership counts and citation indicators. Other possibilities and limitations of these indicators are discussed and future research lines are outlined.
△ Less
Submitted 21 February, 2014;
originally announced April 2014.
-
Do altmetrics correlate with citations? Extensive comparison of altmetric indicators with citations from a multidisciplinary perspective
Authors:
Rodrigo Costas,
Zohreh Zahedi,
Paul Wouters
Abstract:
An extensive analysis of the presence of different altmetric indicators provided by Altmetric.com across scientific fields is presented, particularly focusing on their relationship with citations. Our results confirm that the presence and density of social media altmetric counts are still very low and not very frequent among scientific publications, with 15%-24% of the publications presenting some…
▽ More
An extensive analysis of the presence of different altmetric indicators provided by Altmetric.com across scientific fields is presented, particularly focusing on their relationship with citations. Our results confirm that the presence and density of social media altmetric counts are still very low and not very frequent among scientific publications, with 15%-24% of the publications presenting some altmetric activity and concentrating in the most recent publications, although their presence is increasing over time. Publications from the social sciences, humanities and the medical and life sciences show the highest presence of altmetrics, indicating their potential value and interest for these fields. The analysis of the relationships between altmetrics and citations confirms previous claims of positive correlations but relatively weak, thus supporting the idea that altmetrics do not reflect the same concept of impact as citations. Also, altmetric counts do not always present a better filtering of highly cited publications than journal citation scores. Altmetrics scores (particularly mentions in blogs) are able to identify highly cited publications with higher levels of precision than journal citation scores (JCS), but they have a lower level of recall. The value of altmetrics as a complementary tool of citation analysis is highlighted, although more research is suggested to disentangle the potential meaning and value of altmetric indicators for research evaluation.
△ Less
Submitted 17 January, 2014;
originally announced January 2014.
-
Counting publications and citations: Is more always better?
Authors:
Ludo Waltman,
Nees Jan van Eck,
Paul Wouters
Abstract:
Is more always better? We address this question in the context of bibliometric indices that aim to assess the scientific impact of individual researchers by counting their number of highly cited publications. We propose a simple model in which the number of citations of a publication depends not only on the scientific impact of the publication but also on other 'random' factors. Our model indicate…
▽ More
Is more always better? We address this question in the context of bibliometric indices that aim to assess the scientific impact of individual researchers by counting their number of highly cited publications. We propose a simple model in which the number of citations of a publication depends not only on the scientific impact of the publication but also on other 'random' factors. Our model indicates that more need not always be better. It turns out that the most influential researchers may have a systematically lower performance, in terms of highly cited publications, than some of their less influential colleagues. The model also suggests an improved way of counting highly cited publications.
△ Less
Submitted 2 April, 2013; v1 submitted 19 January, 2013;
originally announced January 2013.
-
The Leiden Ranking 2011/2012: Data collection, indicators, and interpretation
Authors:
Ludo Waltman,
Clara Calero-Medina,
Joost Kosten,
Ed C. M. Noyons,
Robert J. W. Tijssen,
Nees Jan van Eck,
Thed N. van Leeuwen,
Anthony F. J. van Raan,
Martijn S. Visser,
Paul Wouters
Abstract:
The Leiden Ranking 2011/2012 is a ranking of universities based on bibliometric indicators of publication output, citation impact, and scientific collaboration. The ranking includes 500 major universities from 41 different countries. This paper provides an extensive discussion of the Leiden Ranking 2011/2012. The ranking is compared with other global university rankings, in particular the Academic…
▽ More
The Leiden Ranking 2011/2012 is a ranking of universities based on bibliometric indicators of publication output, citation impact, and scientific collaboration. The ranking includes 500 major universities from 41 different countries. This paper provides an extensive discussion of the Leiden Ranking 2011/2012. The ranking is compared with other global university rankings, in particular the Academic Ranking of World Universities (commonly known as the Shanghai Ranking) and the Times Higher Education World University Rankings. Also, a detailed description is offered of the data collection methodology of the Leiden Ranking 2011/2012 and of the indicators used in the ranking. Various innovations in the Leiden Ranking 2011/2012 are presented. These innovations include (1) an indicator based on counting a university's highly cited publications, (2) indicators based on fractional rather than full counting of collaborative publications, (3) the possibility of excluding non-English language publications, and (4) the use of stability intervals. Finally, some comments are made on the interpretation of the ranking, and a number of limitations of the ranking are pointed out.
△ Less
Submitted 17 February, 2012;
originally announced February 2012.
-
Multiple Presents: How Search Engines Re-write the Past
Authors:
Iina Hellsten,
Loet Leydesdorff,
Paul Wouters
Abstract:
Internet search engines function in a present which changes continuously. The search engines update their indices regularly, overwriting Web pages with newer ones, adding new pages to the index, and losing older ones. Some search engines can be used to search for information at the internet for specific periods of time. However, these 'date stamps' are not determined by the first occurrence of t…
▽ More
Internet search engines function in a present which changes continuously. The search engines update their indices regularly, overwriting Web pages with newer ones, adding new pages to the index, and losing older ones. Some search engines can be used to search for information at the internet for specific periods of time. However, these 'date stamps' are not determined by the first occurrence of the pages in the Web, but by the last date at which a page was updated or a new page was added, and the search engine's crawler updated this change in the database. This has major implications for the use of search engines in scholarly research as well as theoretical implications for the conceptions of time and temporality. We examine the interplay between the different updating frequencies by using AltaVista and Google for searches at different moments of time. Both the retrieval of the results and the structure of the retrieved information erodes over time.
△ Less
Submitted 18 November, 2009;
originally announced November 2009.
