Showing 1–47 of 47 results for author: Bornmann, L

How to measure research performance of single scientists? A proposal for an index based on scientific prizes: The Prize Winner Index (PWI)

Authors: Lutz Bornmann, Robin Haunschild

Abstract: In this study, we propose a new index for measuring excellence in science which is based on collaborations (co-authorship distances) in science. The index is based on the Erdős number - a number that was introduced several years ago. We propose to focus with the new index on laureates of prestigious prizes in a certain field and to measure co-authorship distances between the laureates and other sc… ▽ More In this study, we propose a new index for measuring excellence in science which is based on collaborations (co-authorship distances) in science. The index is based on the Erdős number - a number that was introduced several years ago. We propose to focus with the new index on laureates of prestigious prizes in a certain field and to measure co-authorship distances between the laureates and other scientists. To exemplify and explain our proposal, we computed the proposed index in the field of quantitative science studies (PWIPM). The Derek de Solla Price Memorial Award (Price Medal, PM) is awarded to outstanding scientists in the field. We tested the convergent validity of the PWIPM. We were interested whether the indicator is related to an established bibliometric indicator: P(top 10%). The results show that the coefficients for the correlation between PWIPM and P(top 10%) are high (in cases when a sufficient number of papers have been considered for a reliable assessment of performance). Therefore, measured by an established indicator for research excellence, the new PWI indicator seems to be convergently valid and, therefore, might be a possible alternative for established (bibliometric) indicators - with a focus on prizes. △ Less

Submitted 31 March, 2023; originally announced March 2023.

Revolutions in science: The proposal of an approach for the identification of most important researchers, institutions, and countries based on Reference Publication Year Spectroscopy (RPYS)

Authors: Lutz Bornmann, Robin Haunschild, Werner Marx

Abstract: RPYS is a bibliometric method originally introduced in order to reveal the historical roots of research topics or fields. RPYS does not identify the most highly cited papers of the publication set being studied (as is usually done by bibliometric analyses in research evaluation), but instead it indicates most frequently referenced publications - each within a specific reference publication year. I… ▽ More RPYS is a bibliometric method originally introduced in order to reveal the historical roots of research topics or fields. RPYS does not identify the most highly cited papers of the publication set being studied (as is usually done by bibliometric analyses in research evaluation), but instead it indicates most frequently referenced publications - each within a specific reference publication year. In this study, we propose to use the method to identify important researchers, institutions and countries in the context of breakthrough research. To demonstrate our approach, we focus on research on physical modeling of Earth's climate and the prediction of global warming as an example. Klaus Hasselmann and Syukuro Manabe were both honored with the Nobel Prize in 2021 for their fundamental contributions to this research. Our results reveal that RPYS is able to identify most important researchers, institutions, and countries. For example, all the relevant authors' institutions are located in the USA. These institutions are either research centers of two US National Research Administrations (NASA and NOAA) or universities: the University of Arizona, Princeton University, the Massachusetts Institute of Technology (MIT), and the University of Stony Brook. △ Less

Submitted 6 May, 2022; originally announced May 2022.

How relevant is climate change research for climate change policy? An empirical analysis based on Overton data

Authors: Lutz Bornmann, Robin Haunschild, Kevin Boyack, Werner Marx, Jan C. Minx

Abstract: Climate change is an ongoing topic in nearly all areas of society since many years. A discussion of climate change without referring to scientific results is not imaginable. This is especially the case for policies since action on the macro scale is required to avoid costly consequences for society. In this study, we deal with the question of how research on climate change and policy are connected… ▽ More Climate change is an ongoing topic in nearly all areas of society since many years. A discussion of climate change without referring to scientific results is not imaginable. This is especially the case for policies since action on the macro scale is required to avoid costly consequences for society. In this study, we deal with the question of how research on climate change and policy are connected. In 2019, the new Overton database of policy documents was released including links to research papers that are cited by policy documents. The use of results and recommendations from research on climate change might be reflected in citations of scientific papers in policy documents. Although we suspect a lot of uncertainty related to the coverage of policy documents in Overton, there seems to be an impact of international climate policy cycles on policy document publication. We observe local peaks in climate policy documents around major decisions in international climate diplomacy. Our results point out that IGOs and think tanks -- with a focus on climate change -- have published more climate change policy documents than expected. We found that climate change papers that are cited in climate change policy documents received significantly more citations on average than climate change papers that are not cited in these documents. Both areas of society (science and policy) focus on similar climate change research fields: biology, earth sciences, engineering, and disease sciences. Based on these and other empirical results in this study, we propose a simple model of policy impact considering a chain of different document types: the chain starts with scientific assessment reports (systematic reviews) that lead via science communication documents (policy briefs, policy reports or plain language summaries) and government reports to legislative documents. △ Less

Submitted 10 March, 2022; originally announced March 2022.

Heat Waves -- a hot topic in climate change research

Authors: Werner Marx, Robin Haunschild, Lutz Bornmann

Abstract: Research on heat waves (periods of excessively hot weather, which may be accompanied by high humidity) is a newly emerging research topic within the field of climate change research with high relevance for the whole of society. In this study, we analyzed the rapidly growing scientific literature dealing with heat waves. No summarizing overview has been published on this literature hitherto. We dev… ▽ More Research on heat waves (periods of excessively hot weather, which may be accompanied by high humidity) is a newly emerging research topic within the field of climate change research with high relevance for the whole of society. In this study, we analyzed the rapidly growing scientific literature dealing with heat waves. No summarizing overview has been published on this literature hitherto. We developed a suitable search query to retrieve the relevant literature covered by the Web of Science (WoS) as complete as possible and to exclude irrelevant literature (n = 8,011 papers). The time-evolution of the publications shows that research dealing with heat waves is a highly dynamic research topic, doubling within about 5 years. An analysis of the thematic content reveals the most severe heat wave events within the recent decades (1995 and 2003), the cities and countries/regions affected (United States, Europe, and Australia), and the ecological and medical impacts (drought, urban heat islands, excess hospital admissions, and mortality). Risk estimation and future strategies for adaptation to hot weather are major political issues. We identified 104 citation classics which include fundamental early works of research on heat waves and more recent works (which are characterized by a relatively strong connection to climate change). △ Less

Submitted 30 July, 2021; v1 submitted 25 June, 2021; originally announced June 2021.

Comments: 40 pages, 2 tables, and 9 figures

Mapping the impact of papers on various status groups: A new excellence mapping tool based on citation and reader scores

Authors: Lutz Bornmann, Ruediger Mutz, Robin Haunschild, Felix de Moya-Anegon, Mirko de Almeida Madeira Clemente, Moritz Stefaner

Abstract: In over five years, Bornmann, Stefaner, de Moya Anegon, and Mutz (2014) and Bornmann, Stefaner, de Moya Anegon, and Mutz (2014, 2015) have published several releases of the www.excellencemapping.net tool revealing (clusters of) excellent institutions worldwide based on citation data. With the new release, a completely revised tool has been published. It is not only based on citation data (bibliome… ▽ More In over five years, Bornmann, Stefaner, de Moya Anegon, and Mutz (2014) and Bornmann, Stefaner, de Moya Anegon, and Mutz (2014, 2015) have published several releases of the www.excellencemapping.net tool revealing (clusters of) excellent institutions worldwide based on citation data. With the new release, a completely revised tool has been published. It is not only based on citation data (bibliometrics), but also Mendeley data (altmetrics). Thus, the institutional impact measurement of the tool has been expanded by focusing on additional status groups besides researchers such as students and librarians. Furthermore, the visualization of the data has been completely updated by improving the operability for the user and including new features such as institutional profile pages. In this paper, we describe the datasets for the current excellencemapping.net tool and the indicators applied. Furthermore, the underlying statistics for the tool and the use of the web application are explained. △ Less

Submitted 18 March, 2021; originally announced March 2021.

Growth rates of modern science: A latent piecewise growth curve approach to model publication numbers from established and new literature databases

Authors: Lutz Bornmann, Robin Haunschild, Ruediger Mutz

Abstract: Growth of science is a prevalent issue in science of science studies. In recent years, two new bibliographic databases have been introduced which can be used to study growth processes in science from centuries back: Dimensions from Digital Science and Microsoft Academic. In this study, we used publication data from these new databases and added publication data from two established databases (Web… ▽ More Growth of science is a prevalent issue in science of science studies. In recent years, two new bibliographic databases have been introduced which can be used to study growth processes in science from centuries back: Dimensions from Digital Science and Microsoft Academic. In this study, we used publication data from these new databases and added publication data from two established databases (Web of Science from Clarivate Analytics and Scopus from Elsevier) to investigate scientific growth processes from the beginning of the modern science system until today. We estimated regression models that included simultaneously the publication counts from the four databases. The results of the unrestricted growth of science calculations show that the overall growth rate amounts to 4.10% with a doubling time of 17.3 years. As the comparison of various segmented regression models in the current study revealed, the model with five segments fits the publication data best. We demonstrated that these segments with different growth rates can be interpreted very well, since they are related to either phases of economic (e.g., industrialization) and / or political developments (e.g., Second World War). In this study, we additionally analyzed scientific growth in two broad fields (Physical and Technical Sciences as well as Life Sciences) and the relationship of scientific and economic growth in UK. The comparison between the two fields revealed only slight differences. The comparison of the British economic and scientific growth rates showed that the economic growth rate is slightly lower than the scientific growth rate. △ Less

Submitted 21 September, 2021; v1 submitted 14 December, 2020; originally announced December 2020.

Convergent validity of several indicators measuring disruptiveness with milestone assignments to physics papers by experts

Authors: Lutz Bornmann, Alexander Tekles

Abstract: This study focuses on a recently introduced type of indicator measuring disruptiveness in science. Disruptive research diverges from current lines of research by opening up new lines. In the current study, we included the initially proposed indicator of this new type (Wu, Wang, & Evans, 2019) and several variants with DI1: DI5, DI1n, DI5n, and DEP. Since indicators should measure what they propose… ▽ More This study focuses on a recently introduced type of indicator measuring disruptiveness in science. Disruptive research diverges from current lines of research by opening up new lines. In the current study, we included the initially proposed indicator of this new type (Wu, Wang, & Evans, 2019) and several variants with DI1: DI5, DI1n, DI5n, and DEP. Since indicators should measure what they propose to measure, we investigated the convergent validity of the indicators. We used a list of milestone papers, selected and published by editors of Physical Review Letters, and investigated whether this human (experts - based list is related to values of the several disruption indicators variants and - if so - which variants show the highest correlation with expert judgements. We used bivariate statistics, multiple regression models, and (coarsened) exact matching (CEM) to investigate the convergent validity of the indicators. The results show that the indicators correlate differently with the milestone paper assignments by the editors. It is not the initially proposed disruption index that performed best (DI1), but the variant DI5 which has been introduced by Bornmann, Devarakonda, Tekles, and Chacko (2019). In the CEM analysis of this study, the DEP variant - introduced by Bu, Waltman, and Huang (2019) - also showed favorable results. △ Less

Submitted 18 June, 2020; originally announced June 2020.

Do disruption index indicators measure what they propose to measure? The comparison of several indicator variants with assessments by peers

Authors: Lutz Bornmann, Sitaram Devarakonda, Alexander Tekles, George Chacko

Abstract: Recently, Wu, Wang, and Evans (2019) and Bu, Waltman, and Huang (2019) proposed a new family of indicators, which measure whether a scientific publication is disruptive to a field or tradition of research. Such disruptive influences are characterized by citations to a focal paper, but not its cited references. In this study, we are interested in the question of convergent validity, i.e., whether t… ▽ More Recently, Wu, Wang, and Evans (2019) and Bu, Waltman, and Huang (2019) proposed a new family of indicators, which measure whether a scientific publication is disruptive to a field or tradition of research. Such disruptive influences are characterized by citations to a focal paper, but not its cited references. In this study, we are interested in the question of convergent validity, i.e., whether these indicators of disruption are able to measure what they propose to measure ('disruptiveness'). We used external criteria of newness to examine convergent validity: in the post-publication peer review system of F1000Prime, experts assess papers whether the reported research fulfills these criteria (e.g., reports new findings). This study is based on 120,179 papers from F1000Prime published between 2000 and 2016. In the first part of the study we discuss the indicators. Based on the insights from the discussion, we propose alternate variants of disruption indicators. In the second part, we investigate the convergent validity of the indicators and the (possibly) improved variants. Although the results of a factor analysis show that the different variants measure similar dimensions, the results of regression analyses reveal that one variant (DI5) performs slightly better than the others. △ Less

Submitted 20 November, 2019; originally announced November 2019.

Does the $h_α$ index reinforce the Matthew effect in science? Agent-based simulations using Stata and R

Authors: Lutz Bornmann, Christian Ganser, Alexander Tekles, Loet Leydesdorff

Abstract: Recently, Hirsch (2019a) proposed a new variant of the h index called the $h_α$ index. He formulated as follows: "we define the $h_α$ index of a scientist as the number of papers in the h-core of the scientist (i.e. the set of papers that contribute to the h-index of the scientist) where this scientist is the $α$-author" (p. 673). The $h_α$ index was criticized by Leydesdorff, Bornmann, and Opthof… ▽ More Recently, Hirsch (2019a) proposed a new variant of the h index called the $h_α$ index. He formulated as follows: "we define the $h_α$ index of a scientist as the number of papers in the h-core of the scientist (i.e. the set of papers that contribute to the h-index of the scientist) where this scientist is the $α$-author" (p. 673). The $h_α$ index was criticized by Leydesdorff, Bornmann, and Opthof (2019). One of their most important points is that the index reinforces the Matthew effect in science. We address this point in the current study using a recently developed Stata command (h_index) and R package (hindex), which can be used to simulate h index and $h_α$index applications in research evaluation. The user can investigate under which conditions $h_α$ reinforces the Matthew effect. The results of our study confirm what Leydesdorff et al. (2019) expected: the $h_α$ index reinforces the Matthew effect. This effect can be intensified if strategic behavior of the publishing scientists and cumulative advantage effects are additionally considered in the simulation. △ Less

Submitted 27 May, 2019; originally announced May 2019.

How well does I3 perform for impact measurement compared to other bibliometric indicators? The convergent validity of several (field-normalized) indicators

Authors: Lutz Bornmann, Alexander Tekles, Loet Leydesdorff

Abstract: Recently, the integrated impact indicator (I3) indicator was introduced where citations are weighted in accordance with the percentile rank class of each publication in a set of publications. I3 can also be used as a field-normalized indicator. Field-normalization is common practice in bibliometrics, especially when institutions and countries are compared. Publication and citation practices are so… ▽ More Recently, the integrated impact indicator (I3) indicator was introduced where citations are weighted in accordance with the percentile rank class of each publication in a set of publications. I3 can also be used as a field-normalized indicator. Field-normalization is common practice in bibliometrics, especially when institutions and countries are compared. Publication and citation practices are so different among fields that citation impact is normalized for cross-field comparisons. In this study, we test the ability of the indicator to discriminate between quality levels of papers as defined by Faculty members at F1000Prime. F1000Prime is a post-publication peer review system for assessing papers in the biomedical area. Thus, we test the convergent validity of I3 (in this study, we test I3/N - the size-independent variant of I3 where I3 is divided by the number of papers) using assessments by peers as baseline and compare its validity with several other (field-normalized) indicators: the mean-normalized citation score (MNCS), relative-citation ratio (RCR), citation score normalized by cited references (CSNCR), characteristic scores and scales (CSS), source-normalized citation score (SNCS), citation percentile, and proportion of papers which belong to the x% most frequently cited papers (PPtop x%). The results show that the PPtop 1% indicator discriminates best among different quality levels. I3 performs similar as (slightly better than) most of the other field-normalized indicators. Thus, the results point out that the indicator could be a valuable alternative to other indicators in bibliometrics. △ Less

Submitted 18 February, 2019; v1 submitted 4 January, 2019; originally announced January 2019.

The value and credits of n-authors publications

Authors: Lutz Bornmann, Antonio Osorio

Abstract: Collaboration among researchers is becoming increasingly common, which raises a large number of scientometrics questions for which there is not a clear and generally accepted answer. For instance, what value should be given to a two-author or three-author publication with respect to a single-author publication? This paper uses axiomatic analysis and proposes a practical method to compute the expec… ▽ More Collaboration among researchers is becoming increasingly common, which raises a large number of scientometrics questions for which there is not a clear and generally accepted answer. For instance, what value should be given to a two-author or three-author publication with respect to a single-author publication? This paper uses axiomatic analysis and proposes a practical method to compute the expected value of an n-authors publication that takes into consideration the added value induced by collaboration in contexts in which there is no prior or ex-ante information about the publication's potential merits or scientific impact. The only information required is the number of authors. We compared the obtained theoretical values with the empirical values based on a large dataset from the Web of Science database. We found that the theoretical values are very close to the empirical values for some disciplines, but not for all. This observation provides support in favor of the method proposed in this paper. We expect that our findings can help researchers and decision-makers to choose more effective and fair counting methods that take into account the benefits of collaboration. △ Less

Submitted 4 July, 2018; v1 submitted 26 May, 2018; originally announced May 2018.

Opium in science and society: Numbers

Authors: Julian N. Marewski, Lutz Bornmann

Abstract: In science and beyond, numbers are omnipresent when it comes to justifying different kinds of judgments. Which scientific author, hiring committee-member, or advisory board panelist has not been confronted with page-long "publication manuals", "assessment reports", "evaluation guidelines", calling for p-values, citation rates, h-indices, or other statistics in order to motivate judgments about the… ▽ More In science and beyond, numbers are omnipresent when it comes to justifying different kinds of judgments. Which scientific author, hiring committee-member, or advisory board panelist has not been confronted with page-long "publication manuals", "assessment reports", "evaluation guidelines", calling for p-values, citation rates, h-indices, or other statistics in order to motivate judgments about the "quality" of findings, applicants, or institutions? Yet, many of those relying on and calling for statistics do not even seem to understand what information those numbers can actually convey, and what not. Focusing on the uninformed usage of bibliometrics as worrysome outgrowth of the increasing quantification of science and society, we place the abuse of numbers into larger historical contexts and trends. These are characterized by a technology-driven bureaucratization of science, obsessions with control and accountability, and mistrust in human intuitive judgment. The ongoing digital revolution increases those trends. We call for bringing sanity back into scientific judgment exercises. Despite all number crunching, many judgments - be it about scientific output, scientists, or research institutions - will neither be unambiguous, uncontroversial, or testable by external standards, nor can they be otherwise validated or objectified. Under uncertainty, good human judgment remains, for the better, indispensable, but it can be aided, so we conclude, by a toolbox of simple judgment tools, called heuristics. In the best position to use those heuristics are research evaluators (1) who have expertise in the to-be-evaluated area of research, (2) who have profound knowledge in bibliometrics, and (3) who are statistically literate. △ Less

Submitted 8 May, 2018; v1 submitted 27 April, 2018; originally announced April 2018.

Can the Journal Impact Factor Be Used as a Criterion for the Selection of Junior Researchers? A Large-Scale Empirical Study Based on ResearcherID Data

Authors: Lutz Bornmann, Richard Williams

Abstract: Early in researchers' careers, it is difficult to assess how good their work is or how important or influential the scholars will eventually be. Hence, funding agencies, academic departments, and others often use the Journal Impact Factor (JIF) of where the authors have published to assess their work and provide resources and rewards for future work. The use of JIFs in this way has been heavily cr… ▽ More Early in researchers' careers, it is difficult to assess how good their work is or how important or influential the scholars will eventually be. Hence, funding agencies, academic departments, and others often use the Journal Impact Factor (JIF) of where the authors have published to assess their work and provide resources and rewards for future work. The use of JIFs in this way has been heavily criticized, however. Using a large data set with many thousands of publication profiles of individual researchers, this study tests the ability of the JIF (in its normalized variant) to identify, at the beginning of their careers, those candidates who will be successful in the long run. Instead of bare JIFs and citation counts, the metrics used here are standardized according to Web of Science subject categories and publication years. The results of the study indicate that the JIF (in its normalized variant) is able to discriminate between researchers who published papers later on with a citation impact above or below average in a field and publication year - not only in the short term, but also in the long term. However, the low to medium effect sizes of the results also indicate that the JIF (in its normalized variant) should not be used as the sole criterion for identifying later success: other criteria, such as the novelty and significance of the specific research, academic distinctions, and the reputation of previous institutions, should also be considered. △ Less

Submitted 20 June, 2017; originally announced June 2017.

The Journal Impact Factor Should Not Be Discarded

Authors: Lutz Bornmann, Alexander I. Pudovkin

Abstract: The Journal Impact Factor (JIF) has been heavily criticized over decades. This opinion piece argues that the JIF should not be demonized. It still can be employed for research evaluation purposes by carefully considering the context and academic environment. The Journal Impact Factor (JIF) has been heavily criticized over decades. This opinion piece argues that the JIF should not be demonized. It still can be employed for research evaluation purposes by carefully considering the context and academic environment. △ Less

Submitted 13 December, 2016; originally announced December 2016.

"Smart Girls" versus "Sleeping Beauties" in the Sciences: The Identification of Instant and Delayed Recognition by Using the Citation Angle

Authors: Fred Y. Ye, Lutz Bornmann

Abstract: In recent years, a number of studies have introduced methods for identifying papers with delayed recognition (so called "sleeping beauties", SBs) or have presented single publications as cases of SBs. Most recently, Ke et al. (2015) proposed the so called "beauty coefficient" (denoted as B) to quantify how much a given paper can be considered as a paper with delayed recognition. In this study, the… ▽ More In recent years, a number of studies have introduced methods for identifying papers with delayed recognition (so called "sleeping beauties", SBs) or have presented single publications as cases of SBs. Most recently, Ke et al. (2015) proposed the so called "beauty coefficient" (denoted as B) to quantify how much a given paper can be considered as a paper with delayed recognition. In this study, the new term "smart girl" (SG) is suggested to differentiate instant credit or "flashes in the pan" from SBs. While SG and SB are qualitatively defined, the dynamic citation angle \b{eta} is introduced in this study as a simple way for identifying SGs and SBs quantitatively - complementing the beauty coefficient B. The citation angles for all articles from 1980 (n=166870) in natural sciences are calculated for identifying SGs and SBs and their extent. We reveal that about 3% of the articles are typical SGs and about 0.1% typical SBs. The potential advantages of the citation angle approach are explained. △ Less

Submitted 27 November, 2016; originally announced November 2016.

Which early works are cited most frequently in climate change research literature? A bibliometric approach based on Reference Publication Year Spectroscopy

Authors: Werner Marx, Robin Haunschild, Andreas Thor, Lutz Bornmann

Abstract: This bibliometric analysis focuses on the general history of climate change research and, more specifically, on the discovery of the greenhouse effect. First, the Reference Publication Year Spectroscopy (RPYS) is applied to a large publication set on climate change of 222,060 papers published between 1980 and 2014. The references cited therein were extracted and analyzed with regard to publication… ▽ More This bibliometric analysis focuses on the general history of climate change research and, more specifically, on the discovery of the greenhouse effect. First, the Reference Publication Year Spectroscopy (RPYS) is applied to a large publication set on climate change of 222,060 papers published between 1980 and 2014. The references cited therein were extracted and analyzed with regard to publications, which are cited most frequently. Second, a new method for establishing a more subject-specific publication set for applying RPYS (based on the co-citations of a marker reference) is proposed (RPYS-CO). The RPYS of the climate change literature focuses on the history of climate change research in total. We identified 35 highly-cited publications across all disciplines, which include fundamental early scientific works of the 19th century (with a weak connection to climate change) and some cornerstones of science with a stronger connection to climate change. By using the Arrhenius (1896) paper as a RPYS-CO marker paper, we selected only publications specifically discussing the discovery of the greenhouse effect and the role of carbon dioxide. Also, we focused on the time period 1800-1850 to reveal the contributions of J.B.J Fourier in terms of cited references. Using different RPYS approaches in this study, we were able to identify the complete range of works of the celebrated icons as well as many less known works relevant for the history of climate change research. The analyses confirmed the potential of the RPYS method for historical studies: Seminal papers are detected on the basis of the references cited by the overall community without any further assumptions. △ Less

Submitted 28 October, 2016; v1 submitted 29 August, 2016; originally announced August 2016.

Comments: in press at Scientometrics

Citation score normalized by cited references (CSNCR): The introduction of a new citation impact indicator

Authors: Lutz Bornmann, Robin Haunschild

Abstract: In this paper, a new field-normalized indicator is introduced, which is rooted in early insights in bibliometrics, and is compared with several established field-normalized indicators (e.g. the mean normalized citation score, MNCS, and indicators based on percentile approaches). Garfield (1979) emphasizes that bare citation counts from different fields cannot be compared for evaluative purposes, b… ▽ More In this paper, a new field-normalized indicator is introduced, which is rooted in early insights in bibliometrics, and is compared with several established field-normalized indicators (e.g. the mean normalized citation score, MNCS, and indicators based on percentile approaches). Garfield (1979) emphasizes that bare citation counts from different fields cannot be compared for evaluative purposes, because the "citation potential" can vary significantly between the fields. Garfield (1979) suggests that "the most accurate measure of citation potential is the average number of references per paper published in a given field". Based on this suggestion, the new indicator is basically defined as follows: the citation count of a focal paper is divided by the mean number of cited references in a field to normalize citations. The new indicator is called citation score normalized by cited references (CSNCR). The theoretical analysis of the CSNCR shows that it has the properties of consistency and homogeneous normalization. The close relation of the new indicator to the MNCS is discussed. The empirical comparison of the CSNCR with other field-normalized indicators shows that it is slightly poorer able to field-normalize citation counts than other cited-side normalized indicators (e.g. the MNCS), but its results are favorable compared to two citing-side indicator variants (SNCS indicators). Taken as a whole, the results of this study confirm the ability of established indicators to field-normalize citations. △ Less

Submitted 4 July, 2016; originally announced July 2016.

Comments: Accepted for publication in the Journal of Informetrics

Efficiency of research performance and the glass researcher

Authors: Lutz Bornmann, Robin Haunschild

Abstract: Abramo and D'Angelo (in press) doubt the validity of established size-independent indicators measuring citation impact and plead in favor of measuring scientific efficiency (by using the Fractional Scientific Strength indicator). This note is intended to comment on some questionable and a few favorable approaches in the paper by Abramo and D'Angelo (in press). Abramo and D'Angelo (in press) doubt the validity of established size-independent indicators measuring citation impact and plead in favor of measuring scientific efficiency (by using the Fractional Scientific Strength indicator). This note is intended to comment on some questionable and a few favorable approaches in the paper by Abramo and D'Angelo (in press). △ Less

Submitted 31 May, 2016; originally announced May 2016.

Comments: Accepted for publication in the Journal of Informetrics

Is collaboration among scientists related to the citation impact of papers because their quality increases with collaboration? An analysis based on data from F1000Prime and normalized citation scores

Authors: Lutz Bornmann

Abstract: In recent years, the relationship of collaboration among scientists and the citation impact of papers have been frequently investigated. Most of the studies show that the two variables are closely related: an increasing collaboration activity (measured in terms of number of authors, number of affiliations, and number of countries) is associated with an increased citation impact. However, it is not… ▽ More In recent years, the relationship of collaboration among scientists and the citation impact of papers have been frequently investigated. Most of the studies show that the two variables are closely related: an increasing collaboration activity (measured in terms of number of authors, number of affiliations, and number of countries) is associated with an increased citation impact. However, it is not clear whether the increased citation impact is based on the higher quality of papers which profit from more than one scientist giving expert input or other (citation-specific) factors. Thus, the current study addresses this question by using two comprehensive datasets with publications (in the biomedical area) including quality assessments by experts (F1000Prime member scores) and citation data for the publications. The study is based on nearly 10,000 papers. Robust regression models are used to investigate the relationship between number of authors, number of affiliations, and number of countries, respectively, and citation impact - controlling for the papers' quality (measured by F1000Prime expert ratings). The results point out that the effect of collaboration activities on impact is largely independent of the papers' quality. The citation advantage is apparently not quality-related; citation specific factors (e.g. self-citations) seem to be important here. △ Less

Submitted 1 February, 2016; originally announced February 2016.

Comments: Accepted for publication in the Journal of the Association for Information Science and Technology

Policy documents as sources for measuring societal impact: How often is climate change research mentioned in policy-related documents?

Authors: Lutz Bornmann, Robin Haunschild, Werner Marx

Abstract: In the current UK Research Excellence Framework (REF) and the Excellence in Research for Australia (ERA) societal impact measurements are inherent parts of the national evaluation systems. In this study, we deal with a relatively new form of societal impact measurements. Recently, Altmetric - a start-up providing publication level metrics - started to make data for publications available which hav… ▽ More In the current UK Research Excellence Framework (REF) and the Excellence in Research for Australia (ERA) societal impact measurements are inherent parts of the national evaluation systems. In this study, we deal with a relatively new form of societal impact measurements. Recently, Altmetric - a start-up providing publication level metrics - started to make data for publications available which have been mentioned in policy documents. We regard this data source as an interesting possibility to specifically measure the (societal) impact of research. Using a comprehensive dataset with publications on climate change as an example, we study the usefulness of the new data source for impact measurement. Only 1.2% (n=2,341) out of 191,276 publications on climate change in the dataset have at least one policy mention. We further reveal that papers published in Nature and Science as well as from the areas "Earth and related environmental sciences" and "Social and economic geography" are especially relevant in the policy context. Given the low coverage of the climate change literature in policy documents, this study can be only a first attempt to study this new source of altmetric data. Further empirical studies are necessary in upcoming years, because mentions in policy documents are of special interest in the use of altmetric data for measuring target-oriented the broader impact of research. △ Less

Submitted 26 August, 2016; v1 submitted 22 December, 2015; originally announced December 2015.

Comments: in press at Scientometrics

Excellence networks in science: A Web-based application based on Bayesian multilevel logistic regression (BMLR) for the identification of institutions collaborating successfully

Authors: Lutz Bornmann, Moritz Stefaner, Felix de Moya Anegon, Ruediger Mutz

Abstract: In this study we present an application which can be accessed via www.excellence-networks.net and which represents networks of scientific institutions worldwide. The application is based on papers (articles, reviews and conference papers) published between 2007 and 2011. It uses (network) data, on which the SCImago Institutions Ranking is based (Scopus data from Elsevier). Using this data, institu… ▽ More In this study we present an application which can be accessed via www.excellence-networks.net and which represents networks of scientific institutions worldwide. The application is based on papers (articles, reviews and conference papers) published between 2007 and 2011. It uses (network) data, on which the SCImago Institutions Ranking is based (Scopus data from Elsevier). Using this data, institutional networks have been estimated with statistical models (Bayesian multilevel logistic regression, BMLR) for a number of Scopus subject areas. Within single subject areas, we have investigated and visualized how successfully overall an institution (reference institution) has collaborated (compared to all the other institutions in a subject area), and with which other institutions (network institutions) a reference institution has collaborated particularly successfully. The "best paper rate" (statistically estimated) was used as an indicator for evaluating the collaboration success of an institution. This gives the proportion of highly cited papers from an institution, and is considered generally as an indicator for measuring impact in bibliometrics. △ Less

Submitted 12 January, 2016; v1 submitted 17 August, 2015; originally announced August 2015.

Comments: accepted for publication in the Journal of Informetrics

Philosophy of science viewed through the lense of "References Publication Years spectrosopy" (RPYS)

Authors: K. Brad Wray, Lutz Bornmann

Abstract: We examine the sub-field of philosophy of science using a new method developed in information science, Referenced Publication Years Spectroscopy (RPYS). RPYS allows us to identify peak years in citations in a field, which promises to help scholars identify the key contributions to a field, and revolutionary discoveries in a field. We discovered that philosophy of science, a sub-field in the humani… ▽ More We examine the sub-field of philosophy of science using a new method developed in information science, Referenced Publication Years Spectroscopy (RPYS). RPYS allows us to identify peak years in citations in a field, which promises to help scholars identify the key contributions to a field, and revolutionary discoveries in a field. We discovered that philosophy of science, a sub-field in the humanities, differs significantly from other fields examined with this method. Books play a more important role in philosophy of science than in the sciences. Further, Einstein's famous 1905 papers created a citation peak in the philosophy of science literature. But rather than being a contribution to the philosophy of science, their importance lies in the fact that they are revolutionary contributions to physics with important implications for philosophy of science. △ Less

Submitted 13 October, 2014; originally announced October 2014.

Measuring impact in research evaluations: A thorough discussion of methods for, effects of, and problems with impact measurements

Authors: Lutz Bornmann

Abstract: Impact of science is one of the most important topics in scientometrics. Recent developments show a fundamental change in impact measurements from impact on science to impact on society. Since impact measurement is currently in a state of far reaching changes, this paper describes recent developments and facing problems in this area. For that the results of key publications (dealing with impact me… ▽ More Impact of science is one of the most important topics in scientometrics. Recent developments show a fundamental change in impact measurements from impact on science to impact on society. Since impact measurement is currently in a state of far reaching changes, this paper describes recent developments and facing problems in this area. For that the results of key publications (dealing with impact measurement) are discussed. The paper discusses how impact is generally measured within science and beyond (section 2), which effects impact measurements have on the science system (section 3), and which problems are associated with impact measurement (section 4). The problems associated with impact measurement constitute the focus of this paper: Science is marked by inequality, random chance, anomalies, the right to make mistakes, unpredictability, and a high significance of extreme events, which might distort impact measurements. Scientometricians as the producer of impact scores and decision makers as their consumers should be aware of these problems and should consider them in the generation and interpretation of bibliometric results, respectively. △ Less

Submitted 17 February, 2016; v1 submitted 6 October, 2014; originally announced October 2014.

Comments: Accepted for publication in Higher Education

Alternative metrics in scientometrics: A meta-analysis of research into three altmetrics

Authors: Lutz Bornmann

Abstract: Alternative metrics are currently one of the most popular research topics in scientometric research. This paper provides an overview of research into three of the most important altmetrics: microblogging (Twitter), online reference managers (Mendeley and CiteULike) and blogging. The literature is discussed in relation to the possible use of altmetrics in research evaluation. Since the research was… ▽ More Alternative metrics are currently one of the most popular research topics in scientometric research. This paper provides an overview of research into three of the most important altmetrics: microblogging (Twitter), online reference managers (Mendeley and CiteULike) and blogging. The literature is discussed in relation to the possible use of altmetrics in research evaluation. Since the research was particularly interested in the correlation between altmetrics counts and citation counts, this overview focuses particularly on this correlation. For each altmetric, a meta-analysis is calculated for its correlation with traditional citation counts. As the results of the meta-analyses show, the correlation with traditional citations for micro-blogging counts is negligible (pooled r=0.003), for blog counts it is small (pooled r=0.12) and for bookmark counts from online reference managers, medium to large (CiteULike pooled r=0.23; Mendeley pooled r=0.51). △ Less

Submitted 5 March, 2015; v1 submitted 30 July, 2014; originally announced July 2014.

Comments: Accepted for publication in Scientometrics

Which of the world's institutions employ the most highly cited researchers? An analysis of the data from highlycited.com

Authors: Lutz Bornmann, Johann Bauer

Abstract: A few weeks ago, Thomson Reuters published a list of the highly cited researchers worldwide (highlycited.com). Since the data is freely available for downloading and includes the names of the researchers' institutions, we produced a ranking of the institutions on the basis of the number of highly cited researchers per institution. This ranking is intended to be a helpful amendment of other availab… ▽ More A few weeks ago, Thomson Reuters published a list of the highly cited researchers worldwide (highlycited.com). Since the data is freely available for downloading and includes the names of the researchers' institutions, we produced a ranking of the institutions on the basis of the number of highly cited researchers per institution. This ranking is intended to be a helpful amendment of other available institutional rankings. △ Less

Submitted 28 July, 2014; v1 submitted 8 July, 2014; originally announced July 2014.

Validity of altmetrics data for measuring societal impact: A study using data from Altmetric and F1000Prime

Authors: Lutz Bornmann

Abstract: Can altmetric data be validly used for the measurement of societal impact? The current study seeks to answer this question with a comprehensive dataset (about 100,000 records) from very disparate sources (F1000, Altmetric, and an in-house database based on Web of Science). In the F1000 peer review system, experts attach particular tags to scientific papers which indicate whether a paper could be o… ▽ More Can altmetric data be validly used for the measurement of societal impact? The current study seeks to answer this question with a comprehensive dataset (about 100,000 records) from very disparate sources (F1000, Altmetric, and an in-house database based on Web of Science). In the F1000 peer review system, experts attach particular tags to scientific papers which indicate whether a paper could be of interest for science or rather for other segments of society. The results show that papers with the tag "good for teaching" do achieve higher altmetric counts than papers without this tag - if the quality of the papers is controlled. At the same time, a higher citation count is shown especially by papers with a tag that is specifically scientifically oriented ("new finding"). The findings indicate that papers tailored for a readership outside the area of research should lead to societal impact. If altmetric data is to be used for the measurement of societal impact, the question arises of its normalization. In bibliometrics, citations are normalized for the papers' subject area and publication year. This study has taken a second analytic step involving a possible normalization of altmetric data. As the results show there are particular scientific topics which are of especial interest for a wide audience. Since these more or less interesting topics are not completely reflected in Thomson Reuters' journal sets, a normalization of altmetric data should not be based on the level of subject categories, but on the level of topics. △ Less

Submitted 16 February, 2015; v1 submitted 30 June, 2014; originally announced June 2014.

Do altmetrics point to the broader impact of research? An overview of benefits and disadvantages of altmetrics

Authors: Lutz Bornmann

Abstract: Today, it is not clear how the impact of research on other areas of society than science should be measured. While peer review and bibliometrics have become standard methods for measuring the impact of research in science, there is not yet an accepted framework within which to measure societal impact. Alternative metrics (called altmetrics to distinguish them from bibliometrics) are considered an… ▽ More Today, it is not clear how the impact of research on other areas of society than science should be measured. While peer review and bibliometrics have become standard methods for measuring the impact of research in science, there is not yet an accepted framework within which to measure societal impact. Alternative metrics (called altmetrics to distinguish them from bibliometrics) are considered an interesting option for assessing the societal impact of research, as they offer new ways to measure (public) engagement with research output. Altmetrics is a term to describe web-based metrics for the impact of publications and other scholarly material by using data from social media platforms (e.g. Twitter or Mendeley). This overview of studies explores the potential of altmetrics for measuring societal impact. It deals with the definition and classification of altmetrics. Furthermore, their benefits and disadvantages for measuring impact are discussed. △ Less

Submitted 10 September, 2014; v1 submitted 27 June, 2014; originally announced June 2014.

Comments: Accepted for publication in the Journal of Informetrics

Inter-rater reliability and convergent validity of F1000Prime peer review

Authors: Lutz Bornmann

Abstract: Peer review is the backbone of modern science. F1000Prime is a post-publication peer review system of the biomedical literature (papers from medical and biological journals). This study is concerned with the inter-rater reliability and convergent validity of the peer recommendations formulated in the F1000Prime peer review system. The study is based on around 100,000 papers with recommendations fr… ▽ More Peer review is the backbone of modern science. F1000Prime is a post-publication peer review system of the biomedical literature (papers from medical and biological journals). This study is concerned with the inter-rater reliability and convergent validity of the peer recommendations formulated in the F1000Prime peer review system. The study is based on around 100,000 papers with recommendations from Faculty members. Even if intersubjectivity plays a fundamental role in science, the analyses of the reliability of the F1000Prime peer review system show a rather low level of agreement between Faculty members. This result is in agreement with most other studies which have been published on the journal peer review system. Logistic regression models are used to investigate the convergent validity of the F1000Prime peer review system. As the results show, the proportion of highly cited papers among those selected by the Faculty members is significantly higher than expected. In addition, better recommendation scores are also connected with better performance of the papers. △ Less

Submitted 3 June, 2014; v1 submitted 31 March, 2014; originally announced April 2014.

Comments: Accepted for publication in the Journal of the Association for Information Science and Technology

Growth rates of modern science: A bibliometric analysis based on the number of publications and cited references

Authors: Lutz Bornmann, Ruediger Mutz

Abstract: Many studies in information science have looked at the growth of science. In this study, we re-examine the question of the growth of science. To do this we (i) use current data up to publication year 2012 and (ii) analyse it across all disciplines and also separately for the natural sciences and for the medical and health sciences. Furthermore, the data are analysed with an advanced statistical te… ▽ More Many studies in information science have looked at the growth of science. In this study, we re-examine the question of the growth of science. To do this we (i) use current data up to publication year 2012 and (ii) analyse it across all disciplines and also separately for the natural sciences and for the medical and health sciences. Furthermore, the data are analysed with an advanced statistical technique - segmented regression analysis - which can identify specific segments with similar growth rates in the history of science. The study is based on two different sets of bibliometric data: (1) The number of publications held as source items in the Web of Science (WoS, Thomson Reuters) per publication year and (2) the number of cited references in the publications of the source items per cited reference year. We have looked at the rate at which science has grown since the mid-1600s. In our analysis of cited references we identified three growth phases in the development of science, which each led to growth rates tripling in comparison with the previous phase: from less than 1% up to the middle of the 18th century, to 2 to 3% up to the period between the two world wars and 8 to 9% to 2012. △ Less

Submitted 8 May, 2014; v1 submitted 19 February, 2014; originally announced February 2014.

Comments: Accepted for publication in the Journal of the Association for Information Science and Technology

On the origins and the historical roots of the Higgs boson research from a bibliometric perspective

Authors: Andreas Barth, Werner Marx, Lutz Bornmann, Ruediger Mutz

Abstract: Subject of our present paper is the analysis of the origins or historical roots of the Higgs boson research from a bibliometric perspective, using a segmented regression analysis in a reference publication year spectroscopy (RPYS). Our analysis is based on the references cited in the Higgs boson publications published since 1974. The objective of our analysis consists of identifying concrete indiv… ▽ More Subject of our present paper is the analysis of the origins or historical roots of the Higgs boson research from a bibliometric perspective, using a segmented regression analysis in a reference publication year spectroscopy (RPYS). Our analysis is based on the references cited in the Higgs boson publications published since 1974. The objective of our analysis consists of identifying concrete individual publications in the Higgs boson research context to which the scientific community frequently had referred to. As a consequence, we are interested in seminal works which contributed to a high extent to the discovery of the Higgs boson. Our results show that researchers in the Higgs boson field preferably refer to more recently published papers - particular papers published since the beginning of the sixties. For example, our analysis reveals seven major contributions which appeared within the sixties: Englert and Brout (1964), Higgs (1964, 2 papers), and Guralnik et al. (1964) on the Higgs mechanism as well as Glashow (1961), Weinberg (1967), and Salam (1968) on the unification of weak and electromagnetic interaction. Even if the Nobel Prize award highlights the outstanding importance of the work of Peter Higgs and Francois Englert, bibliometrics offer the additional possibility of getting hints to other publications in this research field (especially to historical publications), which are of vital importance from the expert point of view. △ Less

Submitted 29 April, 2014; v1 submitted 17 February, 2014; originally announced February 2014.

What is the effect of country-specific characteristics on the research performance of scientific institutions? Using multi-level statistical models to rank and map universities and research-focused institutions worldwide

Authors: Lutz Bornmann, Moritz Stefaner, Felix de Moya Anegon, Ruediger Mutz

Abstract: Bornmann, Stefaner, de Moya Anegon, and Mutz (in press) have introduced a web application (www.excellencemapping.net) which is linked to both academic ranking lists published hitherto (e.g. the Academic Ranking of World Universities) as well as spatial visualization approaches. The web application visualizes institutional performance within specific subject areas as ranking lists and on custom til… ▽ More Bornmann, Stefaner, de Moya Anegon, and Mutz (in press) have introduced a web application (www.excellencemapping.net) which is linked to both academic ranking lists published hitherto (e.g. the Academic Ranking of World Universities) as well as spatial visualization approaches. The web application visualizes institutional performance within specific subject areas as ranking lists and on custom tile-based maps. The new, substantially enhanced version of the web application and the multilevel logistic regression on which it is based are described in this paper. Scopus data were used which have been collected for the SCImago Institutions Ranking. Only those universities and research-focused institutions are considered that have published at least 500 articles, reviews and conference papers in the period 2006 to 2010 in a certain Scopus subject area. In the enhanced version, the effect of single covariates (such as the per capita GDP of a country in which an institution is located) on two performance metrics (best paper rate and best journal rate) is examined and visualized. A covariate-adjusted ranking and mapping of the institutions is produced in which the single covariates are held constant. The results on the performance of institutions can then be interpreted as if the institutions all had the same value (reference point) for the covariate in question. For example, those institutions can be identified worldwide showing a very good performance despite a bad financial situation in the corresponding country. △ Less

Submitted 16 April, 2014; v1 submitted 13 January, 2014; originally announced January 2014.

Sampling Issues in Bibliometric Analysis

Authors: Richard Williams, Lutz Bornmann

Abstract: Bibliometricians face several issues when drawing and analyzing samples of citation records for their research. Drawing samples that are too small may make it difficult or impossible for studies to achieve their goals, while drawing samples that are too large may drain resources that could be better used for other purposes. This paper considers three common situations and offers advice for dealing… ▽ More Bibliometricians face several issues when drawing and analyzing samples of citation records for their research. Drawing samples that are too small may make it difficult or impossible for studies to achieve their goals, while drawing samples that are too large may drain resources that could be better used for other purposes. This paper considers three common situations and offers advice for dealing with each. First, an entire population of records is available for an institution. We argue that, even though all records have been collected, the use of inferential statistics, significance testing, and confidence intervals is both common and desirable. Second, because of limited resources or other factors, a sample of records needs to be drawn. We demonstrate how power analyses can be used to determine in advance how large the sample needs to be to achieve the study's goals. Third, the sample size may already be determined, either because the data have already been collected or because resources are limited. We show how power analyses can again be used to determine how large effects need to be in order to find effects that are statistically significant. Such information can then help bibliometricians to develop reasonable expectations as to what their analysis can accomplish. While we focus on issues of interest to bibliometricians, our recommendations and procedures can easily be adapted for other fields of study. △ Less

Submitted 17 November, 2015; v1 submitted 10 January, 2014; originally announced January 2014.

Tracing the origin of a scientific legend by Reference Publication Year Spectroscopy (RPYS): the legend of the Darwin finches

Authors: Werner Marx, Lutz Bornmann

Abstract: In a previews paper we introduced the quantitative method named Reference Publication Year Spectroscopy (RPYS). With this method one can determine the historical roots of research fields and quantify their impact on current research. RPYS is based on the analysis of the frequency with which references are cited in the publications of a specific research field in terms of the publication years of t… ▽ More In a previews paper we introduced the quantitative method named Reference Publication Year Spectroscopy (RPYS). With this method one can determine the historical roots of research fields and quantify their impact on current research. RPYS is based on the analysis of the frequency with which references are cited in the publications of a specific research field in terms of the publication years of these cited references. In this study, we illustrate that RPYS can also be used to reveal the origin of scientific legends. We selected Darwin finches as an example for illustration. Charles Darwin, the originator of evolutionary theory, was given credit for finches he did not see and for observations and insights about the finches he never made. We have shown that a book published in 1947 is the most-highly cited early reference cited within the relevant literature. This book had already been revealed as the origin of the term Darwin finches by Sulloway through careful historical analysis. △ Less

Submitted 22 November, 2013; originally announced November 2013.

Comments: Accepted for publication in Scientometrics

The Wisdom of Citing Scientists

Authors: Lutz Bornmann, Werner Marx

Abstract: This Brief Communication discusses the benefits of citation analysis in research evaluation based on Galton's "Wisdom of Crowds" (1907). Citations are based on the assessment of many which is why they can be ascribed a certain amount of accuracy. However, we show that citations are incomplete assessments and that one cannot assume that a high number of citations correlate with a high level of usef… ▽ More This Brief Communication discusses the benefits of citation analysis in research evaluation based on Galton's "Wisdom of Crowds" (1907). Citations are based on the assessment of many which is why they can be ascribed a certain amount of accuracy. However, we show that citations are incomplete assessments and that one cannot assume that a high number of citations correlate with a high level of usefulness. Only when one knows that a rarely cited paper has been widely read is it possible to say (strictly speaking) that it was obviously of little use for further research. Using a comparison with 'like' data, we try to determine that cited reference analysis allows a more meaningful analysis of bibliometric data than times-cited analysis. △ Less

Submitted 7 August, 2013; originally announced August 2013.

Comments: Accepted for publication in the Journal of the American Society for Information Science and Technology

The normalization of citation counts based on classification systems

Authors: Lutz Bornmann, Werner Marx, Andreas Barth

Abstract: If we want to assess whether the paper in question has had a particularly high or low citation impact compared to other papers, the standard practice in bibliometrics is to normalize citations in respect of the subject category and publication year. A number of proposals for an improved procedure in the normalization of citation impact have been put forward in recent years. Against the background… ▽ More If we want to assess whether the paper in question has had a particularly high or low citation impact compared to other papers, the standard practice in bibliometrics is to normalize citations in respect of the subject category and publication year. A number of proposals for an improved procedure in the normalization of citation impact have been put forward in recent years. Against the background of these proposals this study describes an ideal solution for the normalization of citation impact: in a first step, the reference set for the publication in question is collated by means of a classification scheme, where every publication is associated with a single principal research field or subfield entry (e. g. via Chemical Abstracts sections) and a publication year. In a second step, percentiles of citation counts are calculated for this set and used to assign the normalized citation impact score to the publications (and also to the publication in question). △ Less

Submitted 29 July, 2013; originally announced July 2013.

Comments: Accepted for publication in the journal publications

Is there currently a scientific revolution in scientometrics?

Authors: Lutz Bornmann

Abstract: The author of this letter to the editor would like to set forth the argument that scientometrics is currently in a phase in which a taxonomic change, and hence a revolution, is taking place. One of the key terms in scientometrics is scientific impact which nowadays is understood to mean not only the impact on science but the impact on every area of society. The author of this letter to the editor would like to set forth the argument that scientometrics is currently in a phase in which a taxonomic change, and hence a revolution, is taking place. One of the key terms in scientometrics is scientific impact which nowadays is understood to mean not only the impact on science but the impact on every area of society. △ Less

Submitted 24 July, 2013; originally announced July 2013.

Comments: Accepted for publication in the Journal of the American Society for Information Science and Technology

How to evaluate individual researchers working in the natural and life sciences meaningfully? A proposal of methods based on percentiles of citations

Authors: Lutz Bornmann, Werner Marx

Abstract: Although bibliometrics has been a separate research field for many years, there is still no uniformity in the way bibliometric analyses are applied to individual researchers. Therefore, this study aims to set up proposals how to evaluate individual researchers working in the natural and life sciences. 2005 saw the introduction of the h index, which gives information about a researcher's productivi… ▽ More Although bibliometrics has been a separate research field for many years, there is still no uniformity in the way bibliometric analyses are applied to individual researchers. Therefore, this study aims to set up proposals how to evaluate individual researchers working in the natural and life sciences. 2005 saw the introduction of the h index, which gives information about a researcher's productivity and the impact of his or her publications in a single number (h is the number of publications with at least h citations); however, it is not possible to cover the multidimensional complexity of research performance and to undertake inter-personal comparisons with this number. This study therefore includes recommendations for a set of indicators to be used for evaluating researchers. Our proposals relate to the selection of data on which an evaluation is based, the analysis of the data and the presentation of the results. △ Less

Submitted 14 October, 2013; v1 submitted 15 February, 2013; originally announced February 2013.

Comments: Accepted for publication in Scientometrics

Ranking and mapping of universities and research-focused institutions worldwide based on highly-cited papers: A visualization of results from multi-level models

Authors: Lutz Bornmann, Moritz Stefaner, Felix de Moya Anegon, Ruediger Mutz

Abstract: The web application presented in this paper allows for an analysis to reveal centres of excellence in different fields worldwide using publication and citation data. Only specific aspects of institutional performance are taken into account and other aspects such as teaching performance or societal impact of research are not considered. Based on data gathered from Scopus, field-specific excellence… ▽ More The web application presented in this paper allows for an analysis to reveal centres of excellence in different fields worldwide using publication and citation data. Only specific aspects of institutional performance are taken into account and other aspects such as teaching performance or societal impact of research are not considered. Based on data gathered from Scopus, field-specific excellence can be identified in institutions where highly-cited papers have been frequently published. The web application combines both a list of institutions ordered by different indicator values and a map with circles visualizing indicator values for geocoded institutions. Compared to the mapping and ranking approaches introduced hitherto, our underlying statistics (multi-level models) are analytically oriented by allowing (1) the estimation of values for the number of excellent papers for an institution which are statistically more appropriate than the observed values; (2) the calculation of confidence intervals as measures of accuracy for the institutional citation impact; (3) the comparison of a single institution with an "average" institution in a subject area, and (4) the direct comparison of at least two institutions. △ Less

Submitted 24 July, 2013; v1 submitted 3 December, 2012; originally announced December 2012.

Comments: Accepted for publication in the journal Online Information Review

How to analyse percentile impact data meaningfully in bibliometrics: The statistical analysis of distributions, percentile rank classes and top-cited papers

Authors: Lutz Bornmann

Abstract: According to current research in bibliometrics, percentiles (or percentile rank classes) are the most suitable method for normalising the citation counts of individual publications in terms of the subject area, the document type and the publication year. Up to now, bibliometric research has concerned itself primarily with the calculation of percentiles. This study suggests how percentiles can be a… ▽ More According to current research in bibliometrics, percentiles (or percentile rank classes) are the most suitable method for normalising the citation counts of individual publications in terms of the subject area, the document type and the publication year. Up to now, bibliometric research has concerned itself primarily with the calculation of percentiles. This study suggests how percentiles can be analysed meaningfully for an evaluation study. Publication sets from four universities are compared with each other to provide sample data. These suggestions take into account on the one hand the distribution of percentiles over the publications in the sets (here: universities) and on the other hand concentrate on the range of publications with the highest citation impact - that is, the range which is usually of most interest in the evaluation of scientific performance. △ Less

Submitted 8 June, 2012; originally announced June 2012.

Mapping (USPTO) Patent Data using Overlays to Google Maps

Authors: Loet Leydesdorff, Lutz Bornmann

Abstract: A technique is developed using patent information available online (at the US Patent and Trademark Office) for the generation of Google Maps. The overlays indicate both the quantity and quality of patents at the city level. This information is relevant for research questions in technology analysis, innovation studies and evolutionary economics, as well as economic geography. The resulting maps can… ▽ More A technique is developed using patent information available online (at the US Patent and Trademark Office) for the generation of Google Maps. The overlays indicate both the quantity and quality of patents at the city level. This information is relevant for research questions in technology analysis, innovation studies and evolutionary economics, as well as economic geography. The resulting maps can also be relevant for technological innovation policies and R&D management, because the US market can be considered the leading market for patenting and patent competition. In addition to the maps, the routines provide quantitative data about the patents for statistical analysis. The cities on the map are colored according to the results of significance tests. The overlays are explored for the Netherlands as a "national system of innovations," and further elaborated in two cases of emerging technologies: "RNA interference" and "nanotechnology." △ Less

Submitted 9 February, 2012; v1 submitted 26 October, 2011; originally announced October 2011.

Comments: Journal of the American Society for Information Science and Technology (in press)

The Anna Karenina principle: A concept for the explanation of success in science

Authors: Lutz Bornmann, Werner Marx

Abstract: The first sentence of Leo Tolstoy's novel Anna Karenina is: "Happy families are all alike; every unhappy family is unhappy in its own way." Here Tolstoy means that for a family to be happy, several key aspects must be given (such as good health of all family members, acceptable financial security, and mutual affection). If there is a deficiency in any one or more of these key aspects, the family w… ▽ More The first sentence of Leo Tolstoy's novel Anna Karenina is: "Happy families are all alike; every unhappy family is unhappy in its own way." Here Tolstoy means that for a family to be happy, several key aspects must be given (such as good health of all family members, acceptable financial security, and mutual affection). If there is a deficiency in any one or more of these key aspects, the family will be unhappy. In this paper we introduce the Anna Karenina principle as a concept that can explain success in science. Here we will refer to three central areas in modern science in which scarce resources will most usually lead to failure: (1) peer review of research grant proposals and manuscripts (money and journal space as scarce resources), (2) citation of publications (reception as a scarce resource), and (3) new scientific discoveries (recognition as a scarce resource). If resources are scarce (journal space, funds, reception, and recognition), there can be success only when several key prerequisites for the allocation of the resources are fulfilled. If any one of these prerequisites is not fulfilled, the grant proposal, manuscript submission, the published paper, or the discovery will not be successful. △ Less

Submitted 29 July, 2011; v1 submitted 5 April, 2011; originally announced April 2011.

Integrated Impact Indicators (I3) compared with Impact Factors (IFs): An alternative research design with policy implications

Authors: Loet Leydesdorff, Lutz Bornmann

Abstract: In bibliometrics, the association of "impact" with central-tendency statistics is mistaken. Impacts add up, and citation curves should therefore be integrated instead of averaged. For example, the journals MIS Quarterly and JASIST differ by a factor of two in terms of their respective impact factors (IF), but the journal with the lower IF has the higher impact. Using percentile ranks (e.g., top-1%… ▽ More In bibliometrics, the association of "impact" with central-tendency statistics is mistaken. Impacts add up, and citation curves should therefore be integrated instead of averaged. For example, the journals MIS Quarterly and JASIST differ by a factor of two in terms of their respective impact factors (IF), but the journal with the lower IF has the higher impact. Using percentile ranks (e.g., top-1%, top-10%, etc.), an integrated impact indicator (I3) can be based on integration of the citation curves, but after normalization of the citation curves to the same scale. The results across document sets can be compared as percentages of the total impact of a reference set. Total number of citations, however, should not be used instead because the shape of the citation curves is then not appreciated. I3 can be applied to any document set and any citation window. The results of the integration (summation) are fully decomposable in terms of journals or instititutional units such as nations, universities, etc., because percentile ranks are determined at the paper level. In this study, we first compare I3 with IFs for the journals in two ISI Subject Categories ("Information Science & Library Science" and "Multidisciplinary Sciences"). The LIS set is additionally decomposed in terms of nations. Policy implications of this possible paradigm shift in citation impact analysis are specified. △ Less

Submitted 13 June, 2011; v1 submitted 27 March, 2011; originally announced March 2011.

Which cities produce excellent papers worldwide more than can be expected? A new mapping approach--using Google Maps--based on statistical significance testing

Authors: Lutz Bornmann, Loet Leydesdorff

Abstract: The methods presented in this paper allow for a statistical analysis revealing centers of excellence around the world using programs that are freely available. Based on Web of Science data, field-specific excellence can be identified in cities where highly-cited papers were published significantly. Compared to the mapping approaches published hitherto, our approach is more analytically oriented by… ▽ More The methods presented in this paper allow for a statistical analysis revealing centers of excellence around the world using programs that are freely available. Based on Web of Science data, field-specific excellence can be identified in cities where highly-cited papers were published significantly. Compared to the mapping approaches published hitherto, our approach is more analytically oriented by allowing the assessment of an observed number of excellent papers for a city (in the sample) against the expected number. Using this test, the approach cannot only identify the top performers in output but the "true jewels." These are cities locating authors who publish significantly more top cited papers than can be expected. As the examples in this paper show for physics, chemistry, and psychology, these cities do not necessarily have a high output of excellent papers. △ Less

Submitted 28 June, 2011; v1 submitted 16 March, 2011; originally announced March 2011.

Turning the tables in citation analysis one more time: Principles for comparing sets of documents

Authors: Loet Leydesdorff, Lutz Bornmann, Rüdiger Mutz, Tobias Opthof

Abstract: We submit newly developed citation impact indicators based not on arithmetic averages of citations but on percentile ranks. Citation distributions are-as a rule-highly skewed and should not be arithmetically averaged. With percentile ranks, the citation of each paper is rated in terms of its percentile in the citation distribution. The percentile ranks approach allows for the formulation of a more… ▽ More We submit newly developed citation impact indicators based not on arithmetic averages of citations but on percentile ranks. Citation distributions are-as a rule-highly skewed and should not be arithmetically averaged. With percentile ranks, the citation of each paper is rated in terms of its percentile in the citation distribution. The percentile ranks approach allows for the formulation of a more abstract indicator scheme that can be used to organize and/or schematize different impact indicators according to three degrees of freedom: the selection of the reference sets, the evaluation criteria, and the choice of whether or not to define the publication sets as independent. Bibliometric data of seven principal investigators (PIs) of the Academic Medical Center of the University of Amsterdam is used as an exemplary data set. We demonstrate that the proposed indicators [R(6), R(100), R(6,k), R(100,k)] are an improvement of averages-based indicators because one can account for the shape of the distributions of citations over papers. △ Less

Submitted 19 June, 2011; v1 submitted 20 January, 2011; originally announced January 2011.

How fractional counting affects the Impact Factor: Normalization in terms of differences in citation potentials among fields of science

Authors: Loet Leydesdorff, Lutz Bornmann

Abstract: The ISI-Impact Factors suffer from a number of drawbacks, among them the statistics-why should one use the mean and not the median?-and the incomparability among fields of science because of systematic differences in citation behavior among fields. Can these drawbacks be counteracted by counting citation weights fractionally instead of using whole numbers in the numerators? (i) Fractional citation… ▽ More The ISI-Impact Factors suffer from a number of drawbacks, among them the statistics-why should one use the mean and not the median?-and the incomparability among fields of science because of systematic differences in citation behavior among fields. Can these drawbacks be counteracted by counting citation weights fractionally instead of using whole numbers in the numerators? (i) Fractional citation counts are normalized in terms of the citing sources and thus would take into account differences in citation behavior among fields of science. (ii) Differences in the resulting distributions can be tested statistically for their significance at different levels of aggregation. (iii) Fractional counting can be generalized to any document set including journals or groups of journals, and thus the significance of differences among both small and large sets can be tested. A list of fractionally counted Impact Factors for 2008 is available online at http://www.leydesdorff.net/weighted_if/weighted_if.xls. The in-between group variance among the thirteen fields of science identified in the U.S. Science and Engineering Indicators is not statistically significant after this normalization. Although citation behavior differs largely between disciplines, the reflection of these differences in fractionally counted citation distributions could not be used as a reliable instrument for the classification. △ Less

Submitted 22 September, 2010; v1 submitted 27 July, 2010; originally announced July 2010.

A Meta-evaluation of Scientific Research Proposals: Different Ways of Comparing Rejected to Awarded Applications

Authors: Lutz Bornmann, Loet Leydesdorff, Peter van den Besselaar

Abstract: Combining different data sets with information on grant and fellowship applications submitted to two renowned funding agencies, we are able to compare their funding decisions (award and rejection) with scientometric performance indicators across two fields of science (life sciences and social sciences). The data sets involve 671 applications in social sciences and 668 applications in life scienc… ▽ More Combining different data sets with information on grant and fellowship applications submitted to two renowned funding agencies, we are able to compare their funding decisions (award and rejection) with scientometric performance indicators across two fields of science (life sciences and social sciences). The data sets involve 671 applications in social sciences and 668 applications in life sciences. In both fields, awarded applicants perform on average better than all rejected applicants. If only the most preeminent rejected applicants are considered in both fields, they score better than the awardees on citation impact. With regard to productivity we find differences between the fields: While the awardees in life sciences outperform on average the most preeminent rejected applicants, the situation is reversed in social sciences. △ Less

Submitted 18 November, 2009; originally announced November 2009.

Citation Environment of Angewandte Chemie

Authors: Lutz Bornmann, Loet Leydesdorff, Werner Marx

Abstract: Recently, aggregated journal-journal citation networks were made accessible from the perspective of each journal included in the Science Citation Index see (http://www.leydesdorff.net/). The local matrices can be used to inspect the relevant citation environment of a journal using statistical analysis and visualization techniques from social network analysis. The inspection gives an answer to th… ▽ More Recently, aggregated journal-journal citation networks were made accessible from the perspective of each journal included in the Science Citation Index see (http://www.leydesdorff.net/). The local matrices can be used to inspect the relevant citation environment of a journal using statistical analysis and visualization techniques from social network analysis. The inspection gives an answer to the question what the local impact of this and other journals in the environment is. In this study the citation environment of Angewandte Chemie was analysed. Angewandte Chemie is one of the prime chemistry journals in the world. Its environment was compared with that of the Journal of the American Chemical Society. The results of the environment analyses give a detailed insight into the field-embeddedness of Angewandte Chemie. The impacts of the German and international editions of this journal are compared. △ Less

Submitted 16 November, 2009; originally announced November 2009.

Journal ref: Lutz Bornmann, Loet Leydesdorff, and Werner Marx, Citation Environment of Angewandte Chemie, CHIMIA 61(3), 104-109, 2007